Closed ASL Interpreting for Online Videos
Raja Kushalnagar, Matthew Seita, Abraham Glasser · 2017 · Proceedings of the 14th International Web for All Conference (W4A) · doi:10.1145/3058555.3058578
Summary
This paper introduces "closed interpreting," a concept analogous to closed captioning but for sign language interpretation of online videos. While many deaf viewers prefer ASL interpreters over captions (as verbatim captioning speed often exceeds reading abilities, and deaf students learn less from on-screen text than hearing peers), current interpreted videos present a fundamental split-attention problem: viewers must shift their gaze between the video content and the interpreter, missing information from whichever they are not watching. The authors developed an HTML-based tool that overlays an interpreter video on a lecture video with three implementation approaches: (1) static — interpreter fixed beside the lecture video, (2) tracked — interpreter position dynamically follows the relevant content area in the lecture (e.g., moving to align with what the instructor is writing), and (3) customizable — interpreter window can be moved, resized, and made transparent by the viewer. The tracked implementation was manually created by a human who determined the optimal interpreter position for each moment. The study also explored eye-tracking integration to automatically pause the interpreter when the viewer looks away and resume when they look back, with speed-up to catch up to the current point.
Key findings
A user study with 19 deaf and hard of hearing participants compared the three implementations. The customizable version scored highest across all measures: satisfaction (4.36/5), understanding (4.58/5), and ease of viewing (4.42/5). Tracked interpreting scored next: satisfaction (3.84), understanding (4.21), ease of viewing (4.0). Static was lowest: satisfaction (3.16), understanding (3.89), ease of viewing (3.0). Mann-Whitney U tests showed statistically significant differences (p<0.05) between static and tracked for satisfaction and ease of viewing, and between static and customizable for satisfaction, understanding, and ease of viewing. Tracked and customizable were not significantly different from each other. Among the customizable features, the ability to move the interpreter (4.42/5) and resize (4.31/5) were rated highest, while transparency adjustment was less valued (3.42/5). Qualitative feedback revealed key issues: with static interpreting, participants complained about the interpreter being too far from the lecture content, causing them to miss information while shifting gaze. With tracked interpreting, the interpreter window sometimes moved too quickly. Participants valued the customizable version's control, with one noting they could make the interpreter transparent and overlay it directly on the lecture notes.
Relevance
This research addresses an often-overlooked gap in video accessibility: the needs of deaf viewers who prefer sign language over captions. While significant effort has gone into captioning technologies, interpreted video remains poorly studied and poorly implemented. The "closed interpreting" concept — toggleable like closed captions, with user-adjustable presentation — provides a practical framework that video platforms could adopt. The finding that customizable and tracked interpreting significantly outperform static placement has direct implications for how educational institutions and content creators should present interpreted videos. The split-attention problem is particularly acute in educational settings where visual content (slides, whiteboard) competes with the interpreter for the viewer's gaze. The maximum comfortable replay speed for ASL interpreting (approximately 2x normal) is relevant for the eye-tracking pause/resume feature. Future automation of the tracked implementation — automatically determining optimal interpreter placement based on lecture video analysis — could make this approach scalable. The work comes from Gallaudet University (the premier deaf university) and RIT (home to NTID), lending strong domain credibility.
Tags: American Sign Language · ASL · Deaf and hard of hearing · sign language interpreting · video accessibility · closed captioning · online video · user study · personalization · education accessibility · multimedia accessibility · eye tracking · split attention