Joint Bilingual Navigation of Informed Consent and Intake Forms in American Sign Language and Written English
Matthew Seita, Michaela Usha, Rachel Skwersky, Poorna Kushalnagar, Raja Kushalnagar · 2025 · Proceedings of the 22nd International Web for All Conference (W4A 2025) · doi:10.1145/3744257.3744271
Summary
This paper presents a joint bilingual informed consent and intake form that can be navigated simultaneously in American Sign Language (ASL) and written English, addressing a critical accessibility barrier for the approximately 500,000 Americans who primarily use ASL. Current informed consent processes in healthcare and research are typically available only in written English, which is a second language for many Deaf individuals, leading to comprehension difficulties, reduced participation in research, and health disparities. The system displays ASL video translations of each form section alongside synchronized English transcripts with sentence-by-sentence highlighting. Crucially, it uses Individual Sign Language Recognition (ISLR) via machine learning to allow Deaf users to navigate the form entirely through ASL signs—signing "YES" to advance, "NO" to decline, "AGAIN" to replay, and "CONSENT" to finalize. The sign language recognition uses Google's Mediapipe toolkit with the ASL Citizen Dataset for hand gesture recognition, running client-side in the browser. Two studies were conducted: Study 1 tested the iPad-based form with 23 hearing healthcare professionals (achieving SUS scores of 91.25) and 44 DHH participants across three Deaf community events (average SUS of 71.14). Study 2 tested an updated web-based version with 10 DHH participants at Gallaudet University (SUS of 78.75). The form includes features like a progress bar, adjustable video playback speed, and a time-locked "eye" icon to ensure participants adequately review content before proceeding.
Key findings
Healthcare professionals rated the system's usability as "truly superior" (SUS 91.25), and 81-100% indicated they would use it in their research. Crucially, 93.75% of UMass participants said having the form would make them more comfortable including DHH participants in research. However, only 12.5-42.86% of researchers had ever included DHH sign language users, and only 14.28-18.75% always had funding for accessibility accommodations, revealing systemic barriers beyond technology. DHH participants found the form usable (SUS ~71 in Study 1, 78.75 in Study 2) with all Study 2 participants rating user-friendliness as "Excellent" or "Best Imaginable." Sign recognition accuracy varied: "YES" was recognized 84% of the time, "AGAIN" 80%, but "CONSENT" only 57%, with errors often caused by participants keeping hands near their faces while watching videos. The average completion time was 11.54 minutes. Qualitative feedback revealed four key themes: the need for bilingual language access with matched ASL and English content, model sensitivity calibration to avoid capturing unintentional movements, timing controls so users can review content at their own pace, and user choice in how to access information (ASL video, transcript, or both).
Relevance
This research demonstrates a practical application of sign language recognition technology that addresses a real and urgent accessibility gap. DHH individuals are systematically underrepresented in healthcare research partly because the consent process itself is inaccessible—a barrier that this technology directly addresses. For healthcare organizations and researchers, the finding that the vast majority of professionals would welcome this tool but lack resources for DHH accommodations highlights an infrastructure problem that technology can help solve. The bilingual navigation concept extends beyond healthcare: the same approach could be applied to employment paperwork, legal documents, educational materials, and any form-based process. The use of client-side machine learning for sign recognition, running in the browser without server-side processing, has important privacy implications for sensitive healthcare data. The work also highlights the importance of testing with diverse DHH populations, as participants with language deprivation required significantly more support.
Tags: deaf and hard of hearing · sign language recognition · informed consent · healthcare accessibility · American Sign Language · bilingual interfaces · machine learning
Standards referenced: 45 CFR 46.116 · Civil Rights Act of 1964