Chat in the Hat: A Portable Interpreter for Sign Language Users
Larwan Berke, William Thies, Danielle Bragg · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3373625.3417026
Summary
This paper addresses a fundamental limitation of Video Remote Interpreting (VRI) for Deaf and Hard-of-Hearing (DHH) sign language users: existing smartphone-based VRI requires the DHH person to either hold the phone with one hand (reducing signing to one-handed and causing fatigue) or prop it in a fixed location (preventing mobility and multitasking). These constraints create unequal workplace access, as hearing workers can freely walk, talk, and multitask while DHH workers using VRI cannot. The authors designed and evaluated a hands-free remote interpreting prototype consisting of augmented reality smartglasses (Vuzix Blade) to display the interpreter, a baseball hat with a mounted fisheye camera and speaker/mic to capture signing and handle audio, and a custom 3D-printed mount connecting the components. The hat camera captures a top-down fisheye view of the signer's hands and torso, streamed to the remote interpreter via Zoom, while the interpreter's video is displayed in one lens of the smartglasses. The design went through four iterations — wrist-mounted, belt-mounted, neck-mounted, and the final hat-mounted solution — evaluated against criteria including signing space capture, display clarity, weight, and inconspicuousness. Two studies were conducted: a video interpretability test measuring word-level accuracy across conditions, and a user study with 18 participants (9 DHH, 9 hearing) performing three workplace tasks: stationary chatting, walking and chatting, and playing Pictionary at a whiteboard.
Key findings
The interpretability test showed the hands-free device achieved high accuracy: 96% of signs viewed in the smartglasses were understood correctly, and 83% of signs captured by the hat camera were understood. The smartphone baseline achieved 88-100% depending on conditions. The hat camera's lower accuracy was partly due to signs requiring hand positions near the forehead (e.g., "father") being partially occluded from the top-down view. In the user study, participants preferred the hands-free device over the smartphone for 4 of 5 interaction aspects: attention-getting, eye contact, mobility, and fun. However, both groups slightly preferred the smartphone overall, primarily due to familiarity and conversation ease. The hands-free device's key advantages were clear: DHH participants could sign naturally with both hands, walk around freely, and multitask (e.g., draw on a whiteboard during Pictionary). Hearing participants reported a heightened sense of personal connection because the interpreter became "invisible" — they could make direct eye contact with the DHH person rather than feeling a third-party presence. However, this invisibility created a paradox: as the interaction felt more natural, delays from interpretation became more jarring and unexpected. DHH participants scored the device 7.11/10 on the Net Promoter Score, indicating they were "promoters." Both interpreters preferred the hands-free device for contextual awareness, as the hat camera captured the environment around the signer, enabling them to see objects being referenced.
Relevance
This research tackles a real and largely unaddressed workplace equity issue. While the ADA requires reasonable accommodations, current VRI solutions force DHH workers into constrained interaction modes that hearing colleagues never face. The hands-free prototype demonstrates that the technology exists to address this gap, though refinement is needed. For accessibility practitioners and employers, the paper highlights that providing interpreting access is necessary but not sufficient — the form factor of the access technology matters enormously for genuine workplace inclusion. The finding that hearing participants felt stronger personal connections when the interpreter was invisible has implications for the design of all mediated communication: reducing the visibility of accessibility infrastructure can paradoxically improve social interaction for everyone involved. The authors also raise an important equity question: their solution still places the device burden on the DHH person, and they suggest future work exploring designs where the hearing person wears the device instead.
Tags: deaf and hard of hearing · sign language · video remote interpreting · augmented reality · workplace accessibility · wearable technology · ASL
Standards referenced: ADA