Dynamically Adapting GUIs to Diverse Input Devices
Scott Carter, Amy Hurst, Jennifer Mankoff, Jack Li · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169000
Summary
This paper presents an analysis framework and demonstration tool (Input Adapter Tool, IAT) for automatically modifying desktop application interfaces to accommodate alternative input devices. Most GUI applications assume keyboard and mouse input, creating barriers for people with motor impairments who may use switches, speech recognition, keyboard-only input, or pointer-only input. Unlike previous work focused on output adaptation (e.g., reformatting web pages for small screens), this research focuses on input adaptation — making applications work with input devices they were never designed for. The authors analyse the design space along two dimensions: navigation (how users find and select interactors) and control (how users manipulate selected interactors). Navigation can be logical (scanning through interactors sequentially or hierarchically) or spatial (moving pixel by pixel or through grid subdivision). Control involves either substituting interactors with alternatives suited to the input device (e.g., replacing a combobox with a filtered text list for keyboard-only use) or augmenting existing interactors with additional capabilities (e.g., adding tooltips showing keyboard shortcuts). The analysis uses O-notation to compare the efficiency of different navigation and control approaches for each input type, providing a principled basis for adaptation decisions.
Key findings
IAT was implemented as a Java tool that works with any Java Swing application without requiring developers to implement special interfaces or use special objects. It automatically traverses an application's interactor hierarchy to determine what adaptations are needed based on the available input devices. The paper demonstrates adaptations for two applications: a paint program adapted for switch input (augmented with a movable dot and directional arrow on the canvas, plus a popup menu for switching between navigation and drawing modes) and a form entry program adapted for keyboard-only input (comboboxes replaced with predictive text fields, navigation augmented with key-binding tooltips). For switch input, binary interactors like buttons and checkboxes only require a single press to activate; menus and listboxes can be controlled by repeatedly scanning through options; text fields use word prediction with an alphabet scanner; and canvases are augmented with movable dots that send mouse events. For keyboard-only input, menus and comboboxes are replaced with filtered text lists, and navigation uses direct key mappings displayed as tooltips. For speech input, canvases are augmented with grid-based selection tools overlaid on the drawing surface. The tool is extensible — users can edit lookup tables to change interactor replacement mappings and add custom interactors to the library.
Relevance
This research addresses a fundamental accessibility challenge: the vast majority of desktop applications are designed exclusively for mouse and keyboard, yet many users must rely on alternative input devices. Rather than requiring each application to be individually redesigned, IAT's approach of automatically adapting interfaces based on available input is a scalable solution. The O-notation analysis of navigation and control techniques provides a rigorous framework for reasoning about which adaptations are most efficient for different input modalities — for example, demonstrating why hierarchical navigation is best for switch users (requiring only O(log(n)) signals) while direct mapping works well for keyboards. The distinction between interactor substitution (replacing a widget entirely) and augmentation (adding capabilities to an existing widget) is a useful design vocabulary for accessibility adaptation. The work anticipates modern interest in automated accessibility repair and adaptation, and the principle that accessibility tools should work with unmodified applications remains essential — developers cannot always be relied upon to build accessible interfaces from the start.
Tags: adaptive interface · alternative input · switch access · input adaptation · GUI adaptation · motor impairment · accessibility toolkit · interaction techniques