Looking Past Screens: Exploring Mixed Reality and Discreet AAC Devices
Humphrey Curtis, Adam D G Jenkins, Seray B Ibrahim, Timothy Neate · 2024 · Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2024) · doi:10.1145/3663548.3675655
Summary
This paper challenges the dominance of tablet-based augmentative and alternative communication (AAC) devices by co-designing mixed reality (MR) and discreet assistive technologies with people living with aphasia following stroke. Current AAC devices are typically fixed-form-factor tablets that are physically obtrusive, visually identifiable, and perpetuate social stigma—qualities particularly problematic for stroke survivors who may also have hemiplegic body paralysis limiting one-handed operation. The researchers conducted two co-design workshops with 11 people with aphasia and speech and language therapists (SLTs), using diegetic prototypes from film and advertising (Star Wars, Iron Man, Minority Report, Spider-Man) alongside real MR technologies (Apple Vision Pro, Humane AI Pin) to spark accessible discussion about future communication technologies. Workshop activities included talking mat scenario grids and "Someone Who Isn't Me" (SWIM) video prompts—narrative techniques that reduce cognitive load and broaden participation for people with language difficulties. From these workshops, the team developed three high-fidelity prototypes: Pico-project AAC (a smartphone with MagSafe pico-projector that projects communication props onto surfaces), Prompt AAC (an iOS app paired with Bluetooth earphones or wearable speaker for private or public audio prompts using Apple's Personal Voice), and Holo AAC (a Microsoft HoloLens 2 application with hand-tracked and head-tracked menus for spatial language support). Focus groups with 7 participants with aphasia and 4 SLTs evaluated all three prototypes through hands-on testing, Likert questionnaires, and group discussion.
Key findings
Prompt AAC received the highest ratings overall, valued for its discreetness, authentic-sounding Personal Voice synthesis, and practical utility as a backup speech option during communication breakdowns. Participants appreciated being able to store phrases and add new prompts easily, seeing it as a safeguard against anxiety in challenging situations like airports or medical appointments. Pico-project AAC was praised for enhancing multimodal communication through projected maps, photos, and communication props—particularly supporting non-verbal storytelling and group conversation that tablet AAC cannot facilitate. The MagSafe projector attachment was considered accessible for people with hemiplegia. Holo AAC generated the most excitement but also the most criticism: participants found HMD interaction gestures prohibitively challenging with hemiplegia, the device too socially prominent and expensive (approximately $4,000), and the private display undermined co-constructed communication between partners. Key barriers to MR adoption included dependence on physical gestures inaccessible to people with bodily paralysis, cognitive overload from visual clutter, surveillance and privacy concerns, antisocial aesthetics of current HMDs, motion sickness, and low user confidence with technology. Critically, participants wanted AAC devices that reflected personal style and fashion identity rather than medicalised aesthetics, with strong opinions about smartglass frame designs, colours, and materials.
Relevance
This research makes a compelling case for moving AAC design beyond screens and toward discreet, wearable, and environmentally integrated communication technologies. For accessibility practitioners, it highlights that current MR hardware—despite its promise—remains largely inaccessible to people with motor impairments, cognitive load sensitivities, and communication difficulties. The finding that participants prioritised aesthetics and personal style in assistive devices challenges the field's tendency toward functional but medicalised designs. The co-design methodology is exemplary: using diegetic prototypes and SWIM narratives made emerging technology concepts accessible to people with significant language impairments who are typically excluded from design research. The three prototypes demonstrate a spectrum of near-term feasibility, with Prompt AAC being immediately deployable using existing iOS features while Holo AAC represents a longer-term vision. A key limitation is the small sample size and controlled evaluation setting; real-world deployment studies would reveal additional challenges around durability, reliability, and social acceptance.
Tags: augmentative and alternative communication · mixed reality · aphasia · co-design · wearable technology · assistive technology · pico projection