Towards Inclusive External Human-Machine Interface: Exploring the Effects of Visual and Auditory eHMI for Deaf and Hard-of-Hearing People
Wenge Xu, Foroogh Hajiseyedjavadi, Kurtis Weir, Chukwuemeka Eze, Mark Colley · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3790738
Summary
This CHI 2026 paper is a companion to Xu et al. 2026 (10.1145/3772318.3791557) but with a broader, two-stage inclusive-design focus: selecting appropriate visual eHMI concepts for Deaf and Hard-of-Hearing (DHH) pedestrians and then evaluating how visual and auditory eHMIs work together in realistic AV crossing scenarios. The authors note that most eHMI research has excluded the estimated 1.3 billion people globally living with significant disability, and that the 430 million DHH population (projected to reach 700 million by 2050) has been especially overlooked. DHH pedestrians face heightened risk of traffic injury (nearly half of DHH teenagers in one study had been in traffic accidents, 2-3× the rate of hearing peers) because absence of auditory cues impairs direction and distance judgments of approaching vehicles. The work combines two studies. The formative study ran two Microsoft Teams focus groups (one Deaf-focused with BSL interpreters, one HoH-focused) with 6 DHH participants and 6 key stakeholders (eHMI/human-factors researchers, an HMI designer at JLR, accessibility researchers, an assistive technologist, and a DHH-charity professional). Participants evaluated six literature-standard visual eHMI concepts — Abstract Light, Situational Awareness, Text, Symbol, Road-based Projection, and Anthropomorphic — through video presentations, quantitative ratings (Trust, Acceptance, Perceived Safety, Mental Workload, Necessity/Reasonability), and qualitative discussion. The VR user study used a 4×2 mixed design (Visual: No Visual / Abstract Light / Abstract Light + Text / Abstract Light + Symbol; Audio: Without Speech / With Speech; between-subjects Group: 16 Hearing, 16 DHH with hearing losses ranging mild to profound, including four BSL users with an in-person interpreter). Participants used a Varjo XR-4 headset with built-in eye tracking plus a Cyberith Virtualizer Elite 2 treadmill to cross a two-lane urban road 16 times. Measures included NASA-TLX mental workload, Low/High Index of Pupillary Activity (LHIPA), Van der Laan acceptance, Trust in Automation, Perceived Safety, gaze dwell on Light Strip/Display/Whole Vehicle/Active eHMI, step-into-road time, early-step counts, and thematic analysis of semi-structured interviews.
Key findings
Formative study: Abstract Light emerged as the best singular visual eHMI, followed by Text and Symbol; Situational Awareness and Anthropomorphic (smiling-car) concepts were consistently rated worst across trust, safety, usefulness, and satisfying. Participants articulated four design requirements: (1) clearly present key vehicle states and state transitions, (2) avoid "crossing advice" and let pedestrians decide (echoing ISO/TR 23049 and Volvo 360c guidance), (3) provide multi-modal eHMIs for cross-disability coverage, and (4) provide combined-visual communications (e.g., Abstract Light + Text or + Symbol) so pedestrians with different needs can choose a reliable channel. VR study: (1) DHH participants spent significantly more time looking at the AV than Hearing participants (M=4.95s vs. 3.32s) — consistent with prior observations that DHH pedestrians exercise extra caution by visually verifying vehicle behavior. (2) Both visual and auditory eHMIs significantly improved trust, usefulness, satisfying, and perceived safety ratings. Abstract Light + Text and Abstract Light + Symbol consistently outperformed Abstract Light alone and No Visual. (3) Only visual eHMIs affected crossing behavior: Abstract Light + Text/Symbol reduced step-into-road time, reduced time looking at the AV overall, and increased active-visual-eHMI dwell percentage; auditory speech improved subjective experience but did not change objective behavior. DHH valuation differed from Hearing: DHH rated necessity of visual eHMIs significantly higher, and auditory eHMIs showed more mixed utility — only 9/16 DHH participants could hear any auditory eHMI, and 2 BSL users heard nothing at all. Qualitative findings flagged risks: symbols like "Walking Man" were sometimes read as traffic signs implying crossing advice, carrying liability concerns; Abstract Light alone was confusing for some participants on first encounter; participants suggested adding haptic feedback (e.g., via Apple Watch or IoT wearable) as a third modality. Five practical implications: (i) include various populations in eHMI design and evaluation; (ii) display key state and state transitions clearly; (iii) avoid traffic-style symbols to prevent liability/legibility issues; (iv) enable multi-modal eHMI anchored on a visual baseline; (v) enable combined-visual eHMIs (Abstract Light + Text recommended as starting point) to compensate for limitations of any singular visual channel.
Relevance
For accessibility practitioners, this paper is a rare example of proper inclusive-design process applied to automotive HMI: formative involvement of DHH people and key stakeholders, literature-grounded concept selection, and a VR evaluation that accounts for both hearing and deaf users. The finding that DHH pedestrians spend substantially more time visually checking the vehicle — and therefore benefit disproportionately from clear state-transition visual eHMIs — should influence regulatory and industry guidance. The cautionary note against traffic-style symbols on vehicles, and the liability implications of crossing advice, is particularly actionable for automotive designers. Together with the companion background-noise study (Xu et al. 2026), this work outlines a coherent research agenda for inclusive AV-pedestrian communication. Limitations include a UK-only sample, single-crossing scenario without mixed traffic or other pedestrians, no low-vision or blind participants, reliance on Abstract Light as the sole singular visual baseline in the VR study, and a single session per participant precluding longitudinal adaptation effects. The explicit call to work with hearing-technology manufacturers and to investigate haptic modalities (e.g., wrist wearables) points to concrete follow-on work.
Tags: deaf and hard of hearing · external human-machine interface · automated vehicles · pedestrian safety · multimodal interaction · accessibility · virtual reality · focus groups · inclusive design
Standards referenced: ISO/TR 23049:2018 · WHO Environmental Noise Guidelines