Evaluating Interfaces for Intelligent Mobile Search
Karen Church, Barry Smyth, Mark T. Keane · 2006 · Proceedings of the 2006 International Cross-Disciplinary Workshop on Web Accessibility (W4A) · doi:10.1145/1133219.1133232
Summary
This paper argues that simply transplanting desktop web search interfaces onto mobile devices creates a poor user experience, and proposes an alternative approach to presenting search results on small screens. The authors first evaluate seven mobile search engines of the era (Google Mobile, Mooobl, Click4WAP, Seek4Wap, WAPAII, WAPly, and Ithaki) across 20 queries, analyzing both result relevance and presentation strategies. They find significant limitations: the pure WML search engines return relevant results for only 20% of queries on average, with relevant results buried at an average position of 17.2 in result lists. Mobile screens (Nokia Series 40 fitting ~6 lines, Series 60 fitting ~7 lines) can display only 1-2 results per screen, making traditional title-plus-snippet presentation formats impractical. The authors propose replacing snippet text with "related queries" — keywords from past queries that led other users to select a particular result — drawn from I-SPY, a community-based collaborative meta-search engine. This approach produces result descriptions averaging only 4 unique terms (73 characters) compared to 35 terms in standard snippets (167 characters), while maintaining informativeness by leveraging community search behavior to provide context about each result.
Key findings
The evaluation of existing mobile search engines revealed that two-thirds presented snippet text despite the severe screen constraints, while the remaining third dropped snippets entirely, leaving users with only uninformative titles. In an offline evaluation using I-SPY search logs (684 result pages, 2,600+ queries), related query terms found the target page in the top 500 results 92-94% of the time, compared to 66-80% for snippet-derived terms. A live user evaluation with 120 participants comparing three interfaces — title only, title plus snippet, and title plus related queries — showed the related query interface performed best overall with 54% of selections across both evaluation questions. When asked specifically which interface provided the best balance of information and screen space, 75% of users preferred the related query interface, while traditional snippet text was preferred by only 8%. ANOVA testing confirmed statistically significant differences (F(2,666) = 169.5, p < 0.001). The related query interface required less than half the screen space of the snippet interface (73 vs. 167 characters per result) while being judged more informative than title-only presentation.
Relevance
While this paper predates the smartphone revolution that would transform mobile search (published when WAP phones and early 3G devices dominated), its core insights about designing for constrained displays remain relevant to accessibility. The challenge of presenting information effectively on small screens parallels accessibility concerns for users who magnify content (effectively creating a small viewport), use screen readers (where verbosity has a cognitive cost similar to screen space constraints), or access the web through alternative devices. The community-driven approach to generating concise, contextual result descriptions anticipates modern techniques in collaborative filtering and personalization. For accessibility practitioners, the paper illustrates an important principle: adapting content for constrained interaction contexts requires fundamental rethinking of information presentation, not simply scaling down desktop designs. This applies equally to mobile accessibility, screen reader optimization, and designing for users with cognitive disabilities who benefit from reduced information density.
Tags: mobile accessibility · mobile search · search interfaces · small screen devices · information retrieval · user evaluation · mobile web