Towards Accessible Conversations in a Mobile Context for People who are Deaf and Hard of Hearing
Dhruv Jain, Rachel Franz, Leah Findlater, Jackson Cannon, Raja Kushalnagar, Jon Froehlich · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18) · doi:10.1145/3234695.3236362
Summary
This paper presents two studies examining the communication needs of deaf and hard of hearing (DHH) people in mobile contexts (walking, transit, recreational activities) and the potential for head-mounted display (HMD) captions to address those needs. Prior research on DHH communication challenges and captioning technologies had focused almost exclusively on stationary contexts like meetings and lectures. The authors identify that mobile contexts introduce unique challenges: increased visual attention split between the speaker and the environment (navigating obstacles, traffic), variable background noise, changing lighting conditions, and the impracticality of holding a phone for captions while walking. Study 1 conducted semi-structured interviews with 12 DHH participants about their mobile conversation experiences, coping strategies, and reactions to three captioning form factors (phone, smartwatch, HMD). Participants reported that moving conversations are generally brief and shallow, with limited social connection — nine participants said they miss significant portions of spoken conversations while moving. Study 2 evaluated a proof-of-concept prototype on the Microsoft HoloLens that displayed real-time captions from a professional transcriptionist in the user's field of view, tested with 10 DHH participants during actual walking scenarios in a campus building.
Key findings
In Study 1, eleven of twelve participants preferred the HMD over phone or smartwatch for at least one mobile context, primarily because it would reduce visual attention split by placing captions within the user's gaze. Participants identified both adaptive strategies (asking speakers to repeat, repositioning to face the speaker, controlling conversation flow) and maladaptive strategies (avoiding conversations with strangers, postponing conversations, ignoring conversations entirely). Environmental challenges included visual attention split on uneven terrain, space constraints in narrow areas and vehicles, background noise on transit and outdoors, and variable lighting affecting speechreading. In Study 2, all ten participants reported the HMD prototype helped them understand at least some conversation while walking. Self-reported conversation understanding improved significantly from 2.4 to 3.8 on a 5-point scale (p=.006). However, four participants found captions occasionally distracting, especially on stairs or in bright daylight where the HoloLens display was insufficiently bright. Participants who preferred oral communication used captions supplementarily (to fill gaps), while sign language users depended on captions more heavily. Key design recommendations emerged: captions should auto-align near speakers, adapt color/contrast to lighting, allow customization, display speaker identification and environmental sounds, and include options to toggle or reposition captions.
Relevance
This paper opens an important new research space by extending DHH captioning research from stationary to mobile contexts — addressing how people actually live and communicate throughout their day, not just in formal settings. The finding that mobile conversations for DHH people tend to be superficial and socially limited reveals how communication barriers compound into social isolation in everyday life, beyond structured environments where accommodations are typically provided. The HMD captioning approach anticipates a near-future where lightweight AR glasses could provide always-available captions — a vision increasingly plausible with products like Apple Vision Pro and Meta smart glasses. The paper also honestly addresses concerns from the DHH community about technology burden and audist assumptions (the idea that technology is imposed on deaf people to accommodate hearing communication norms), noting one participant's suggestion that all conversation participants should display captions on their shirts. For practitioners, the design guidelines for mobile HMD captioning — text alignment, adaptive context, user customizability, contextual information beyond speech — provide a valuable framework for emerging AR accessibility applications.
Tags: deaf and hard of hearing · real-time captioning · augmented reality · head-mounted display · mobile accessibility · conversations · HoloLens · visual attention split