Caption-Occlusion Severity Judgments across Live-Television Genres from Deaf and Hard-of-Hearing Viewers
Akhter Al Amin, Saad Hassan, Matt Huenerfauth · 2021 · Proceedings of the 18th International Web for All Conference (W4A '21) · doi:10.1145/3430263.3452429
Summary
This paper addresses a largely overlooked dimension of caption quality for Deaf and Hard of Hearing (DHH) viewers: the problem of captions occluding important onscreen visual content during live television broadcasts. While existing caption quality metrics focus primarily on transcription accuracy (e.g., Word Error Rate), they do not account for whether captions block critical visual information such as speakers' faces, onscreen text, or graphical elements. This is particularly problematic for live programming, where captions are typically placed at fixed locations rather than being manually positioned by a human captioner. The researchers conducted two studies involving DHH participants. In the first study, 19 DHH participants (15 deaf, 4 hard of hearing, ages 18-37) rated the importance of 29 different onscreen content regions across 14 layouts spanning 6 live television genres: News, Interviews/Talk Shows, Emergency Announcements, Political Debates, Weather News, and Sports. This produced a dataset of 3,002 importance judgments. In the second study, 23 DHH participants evaluated the quality of caption placement in 11 video stimuli with captions placed at three typical screen locations (upper and lower segments of the lower third, and the upper third), rating satisfaction on a ten-point scale.
Key findings
The importance ratings revealed that DHH viewers' priorities for onscreen content vary significantly across television genres. Of 13 content regions appearing in multiple genres, 7 showed statistically significant differences in importance across genres (Friedman test). The presenter's name was rated most important during Emergency Announcements (4.368) and least important in Weather News (2.807), reflecting viewers' need to assess the authority of information sources during emergencies. Speaker's eyes and mouth were consistently rated highly across all genres, supporting the importance of speechreading (lipreading) for DHH viewers. The topic of discussion or current news story was rated highly across all genres where it appeared (4.4-4.6). Qualitative feedback revealed that DHH viewers value seeing the TV network logo to assess source credibility, and rely on headlines and onscreen text to compensate for caption inaccuracies. Crucially, the severity-weighted, genre-specific caption evaluation metric (using importance weights from the dataset) correlated significantly better with DHH participants' subjective judgments of caption quality (Spearman rho = 0.462) than the baseline metric with uniform weights (rho = 0.374), with the difference being statistically significant (z = 2.08, p = 0.0375).
Relevance
This research has direct implications for broadcasters, captioning service providers, and developers of automatic caption placement systems. The finding that content region importance varies by genre provides empirical evidence against one-size-fits-all caption placement approaches and motivates genre-specific placement algorithms. For accessibility practitioners, the study highlights that caption quality extends beyond text accuracy — the spatial relationship between captions and other visual content is a significant factor in DHH viewers' experience. The publicly released dataset of importance ratings can inform the development of better automated caption evaluation metrics and placement guidelines. The work also underscores the importance of involving DHH users directly in research, as their priorities (such as the importance of seeing faces for speechreading, or needing to verify source credibility during emergencies) would not be apparent without user-centred research methods.
Tags: deaf and hard of hearing · captions · television accessibility · caption placement · media accessibility · user research
Standards referenced: FCC Closed Captioning Quality Report · EIA 608