Automatically Generating Custom User Interfaces for Users with Physical Disabilities
Krzysztof Z. Gajos, Jing Jing Long, Daniel S. Weld · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169036
Summary
This paper presents SUPPLE, a system that automatically generates custom graphical user interfaces tailored to individual users' physical capabilities. Standard desktop GUIs are optimised for typical users interacting via keyboard, mouse, and standard-sized displays, creating a mismatch for users with motor or vision impairments. Rather than relying on external assistive technologies (which sometimes cannot be avoided) or expecting GUI designers to create custom interfaces for every possible user, SUPPLE takes a functional specification of a UI and an individual's interaction model to automatically render a personalised interface. The system architecture has three components: SUPPLE (the UI generator that combines a functional UI model with a custom interaction model to produce an optimised interface), ARNAULD (a personaliser that elicits user preferences through machine learning), and a device model describing available widgets and screen constraints. SUPPLE casts UI rendering as a constrained optimisation problem, choosing widgets, layout, and navigation structure to minimise the estimated effort for a specific user. This approach is a radical departure from knowledge-based heuristic rules typically used in adaptive UI research, using instead a decision-theoretic optimisation with a personalised utility function induced through max-margin machine learning.
Key findings
The paper demonstrates three automatically generated interfaces for the same stereo system application: (a) a baseline UI for a typical user with small controls and standard layout; (b) an adapted UI for a user with slight motor impairment, replacing small click targets like spinners and sliders with large click targets like radio buttons and checkboxes, and reorganising the interface into tab panes to accommodate the larger widgets; and (c) an adapted UI for a user with slight vision impairment, featuring enlarged fonts and visual cues with higher contrast. The interface generation process takes less than 2 seconds even for complex applications. The ARNAULD personalisation process requires about two dozen basic interactions (taking less than 15 minutes) where users provide feedback on sample concrete UIs, which is sufficient to learn accurate parameters. These learned parameters then generalise to generate interfaces for applications not included in the training set. The system trivially adapts to devices with different screen sizes, and the same optimisation approach can produce fundamentally different interface styles by modifying the objective function parameters in the interaction model.
Relevance
SUPPLE addresses a fundamental scalability problem in accessible interface design: it is unreasonable to expect designers to manually create custom interfaces for every combination of disability, device, and application. The optimisation-based approach — treating UI generation as a mathematical problem of minimising user effort given constraints — is more principled and scalable than hand-crafted heuristic rules. The separation of functional UI specification from rendering means application developers need only describe what their interface does, not how it should look for every possible user. The ARNAULD personalisation component is particularly significant because it allows the system to learn individual needs through brief interactive sessions rather than requiring users to self-classify into disability categories. This research anticipates modern interest in AI-driven personalisation of interfaces and remains relevant as applications increasingly need to adapt to diverse user abilities, devices, and contexts.
Tags: adaptive interface · automatic UI generation · motor impairment · visual impairment · machine learning · personalisation · optimisation · physical disability