A11yExtensions: Accessibility Extensions to Augment Mobile AI Assistive Technology In-Situ
Jaylin Herskovitz, Ellie Seehorn, Ather Jammoa, Jason Meddaugh, Anhong Guo · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3791559
Summary
A11yExtensions introduces the concept of in-situ extensions for mobile AI assistive technology — add-on services that augment existing deployed applications like Seeing AI, Be My AI, and Lookout with additional features, without modifying or replacing those apps. The project was developed through Research through Design (RtD) using four co-design sessions over three months with two blind accessibility professionals who served as co-designers and co-authors. The core problem addressed is the persistent gap between academic accessibility research and deployed commercial products. Research teams regularly develop improvements to AI assistive technology — including camera aiming guidance, AI result verification, and image quality checking — but these features rarely reach commercial apps, which are designed for broad audiences and cannot accommodate the long tail of individual user needs. A11yExtensions proposes mobile automation tools, specifically iOS Shortcuts and App Intents, as a mechanism to bridge this gap by enabling new features to be delivered as add-ons to existing apps. Three extension prototypes were implemented and evaluated: A.1 Camera Aiming, which provides verbal guidance to help users center objects in frame before taking a photo; A.2 Cross Checking, which automates AI result verification by sending a screenshot to multiple AI models and comparing answers; and A.3.M Image Quality, which performs background quality checking on photos, detecting issues like blur, darkness, or poor framing. The paper also develops a design space for mobile accessibility extensions organized around eight dimensions across two themes: Interaction Dimensions (triggers, input, interaction level, output) and Extension Dimensions (time, presence, data captured, function).
Key findings
Co-designers consistently valued the three core add-ons, viewing them as a practical mechanism to bring already-desired features into existing workflows. The add-on approach preserves familiarity: users stay within apps they already know and trust, with extensions invoked as needed rather than requiring full application switches. The image quality checker was rated the most seamless because it ran entirely in the background without disrupting workflow. Preferred triggering methods varied: gesture-based triggers (double-tapping the back of the phone) were preferred for speed and non-verbosity, while verbal menus via Siri provided discoverability for less frequent use. Co-designers stressed the importance of offering multiple trigger options to accommodate different contexts, particularly public settings where speaking to a device may be uncomfortable. Key challenges emerged across three areas. First, onboarding and setup are significant barriers for non-expert users: each Shortcut requires manual installation and first-run permission granting, creating friction that is unlikely to suit novice users. Second, privacy concerns arise from automated data capture, multi-app routing, and uncertainty about where data is processed or stored. Third, VoiceOver bugs in iOS Shortcuts impaired permission dialogs during testing, illustrating how accessibility tooling can itself be inaccessible. Co-designers also envisioned A11yExtensions as a platform for participatory accessibility research, enabling feature testing in everyday contexts without deploying standalone applications — a significant opportunity to close the research-to-practice gap.
Relevance
This paper directly addresses a structural problem in accessibility practice: the gap between what academic research produces and what blind and low-vision users can access in their daily tools. The add-on paradigm offers a practical path for deploying research-grade features — camera guidance, AI verification, image quality checking — without waiting for commercial app updates or maintaining bespoke standalone tools. For organizations like CNIB, A11yExtensions suggests a model where accessibility features developed through user research could be made available as extensions to mainstream apps that clients already use, lowering both deployment barriers and onboarding costs. The co-design methodology, which centred two blind accessibility professionals as co-authors and domain experts across multiple sessions, also models how meaningful participation should be structured in AT research. The paper is notably honest about approachability limitations: add-ons currently suit technically confident power users more than novice users, and this gap must be addressed before broader deployment is viable. Privacy by design and transparent automation — knowing what an add-on does, when, and where data goes — emerge as non-negotiable requirements for user trust.
Tags: blind and low vision · mobile accessibility · AI assistive technology · co-design · automation · in-situ · camera aiming · AI verification · iOS · research through design