Supporting Older Adults in Locating Mobile Interface Features with Voice Input
Ja Eun Yu, Debaleena Chattopadhyay · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3418044
Summary
This demonstration paper presents a voice assistant system built on Android that helps older adults locate specific features on complex mobile interfaces by speaking natural language queries. The system addresses a well-documented challenge: older adults struggle to find functions in feature-rich mobile applications like Google Maps, Facebook, and Amazon — not because they lack familiarity with the app, but because small screens, dense layouts, partially hidden UI components (horizontal scrolling menus, bottom sheets, scrollable tabs), and frequent app updates that change visual layouts make locating specific buttons and menu items cognitively demanding. Age-related changes including narrowing of the useful field of view (UFOV), difficulty rapidly shifting visual attention, reduced parallel processing of multiple information cues, and inability to ignore competing sensory information compound these challenges. Rather than designing simplified "senior-friendly" apps — which research shows older adults find patronizing and rarely adopt — the system augments mainstream applications with voice-guided feature discovery. The implementation uses Android's Assist API to acquire the complete UI structure (text, view hierarchy, and coordinates) of the currently displayed screen. When a user asks a question like "How can I see the public transit options?", the system converts speech to text via Android's RecognizerIntent, matches query keywords against a dictionary populated from the current screen's UI component labels (augmented with manually created synonyms like "public transportation" mapping to "transit," "bus," and "train"), and visually highlights the matching UI component with guidance on how to reach it.
Key findings
Four scenarios demonstrate the system using Google Maps. In Scenario 1, a user wants to find coffee shops but the "coffee" option is hidden in a partially visible horizontal slider at the top — the system infers the component's location beyond the screen edge and visually guides the user to swipe left. In Scenario 2, the user asks about public transit options and the system highlights the relevant button with an overlay tooltip. In Scenario 3, the user asks about sharing a point-of-interest and the system directs them to the "Share" button. In Scenario 4, the user doesn't know where to type an address for directions, and the system highlights the search text box. The system is generalizable to any Android mobile application since the Assist API provides UI structure data for any foreground app. The approach builds on the discontinued "Now on Tap" / "What's on my screen" Google feature, but focuses specifically on UI components rather than page content. The synonym dictionary is currently manually curated — future work includes using WordNet for automated synonym expansion, exploring additional guidance modalities beyond visual highlighting, and conducting user studies with older adults.
Relevance
This paper offers a practical alternative to the two dominant approaches for older adult mobile accessibility: redesigning apps (which older adults reject as patronizing) and teaching digital skills (which doesn't address the ongoing challenge of app updates). Instead, it adds an assistive layer that works with any existing Android app, preserving dignity and mainstream technology use. For accessibility practitioners, the key insight is that voice input can serve as a bridge between intent ("I want to do X") and interface navigation ("find and tap this specific button"), which is a qualitatively different use of voice than voice commands that execute actions directly. The Assist API approach is elegant because it accesses the actual UI hierarchy rather than relying on screen recognition, making it robust across visual design changes. Limitations include the lack of user evaluation with older adults, the manually curated synonym dictionary, visual-only guidance output (which may not help users with low vision), and the 4-page demo format.
Tags: aging · mobile accessibility · voice interface · user interface design · assistive technology · speech recognition · usability