Visualizing Non-Speech Sounds for the Deaf
Tara Matthews, Janette Fong, Jennifer Mankoff · 2005 · Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '05) · doi:10.1145/1090785.1090797
Summary
This paper investigates peripheral visual displays that help people who are deaf maintain awareness of non-speech sounds in their environment — sounds like phone ringing, doorbells, knocking, fire alarms, children playing, dogs barking, and appliances operating. While commercial products exist for specific alert sounds (doorbell, fire alarm), each requires a separate costly system, and none provides continuous awareness of all ambient sounds. The researchers conducted design interviews with 8 participants (2 profoundly deaf, 2 mostly deaf, 4 hard-of-hearing) to establish visual design preferences and functional requirements. They presented 10 design sketches varying in size (PDA vs. PC screen vs. large wall display), information conveyed (recognized sounds, location, volume, pitch), and visual approach (icons, spectrograms, maps, rings, ambient visualizations). From these interviews, they identified three visual design preferences — ease of interpretation, glance-ability, and appropriate distraction levels — and four functional requirements: ability to identify what sound occurred, view a history of displayed sounds, customize which sounds are shown, and determine the system's confidence in its identification accuracy. They then implemented and evaluated two prototypes: Single Icon (a minimally distracting corner icon with rings for unrecognized sounds) and Spectrograph with Icon (adding a real-time spectrogram for users wanting to learn about sounds).
Key findings
The design interviews revealed that participants wanted awareness of sounds at home, work, and while mobile, with different sounds important in each context — emergency alarms and intruders at home, coworker activities and phone ringing at work, and traffic and people approaching while mobile. Participants strongly preferred designs with icons that identified recognized sounds over abstract visualizations like ambient displays or colored rings, because icons enabled quick glancing without interpretation effort. However, they valued having unrecognized sounds shown as rings (encoding volume by ring count and pitch by color). All four evaluation participants favored the History Display as a stand-alone feature, as it let them maintain awareness without constantly watching the display. The sound recognition system (using Malkin's acoustic event classification) correctly identified phone ringing and speech 100% of the time; door opening/closing were often confused but treated as a single event. The Single Icon display was preferred for its minimal distraction, while Spectrograph with Icon appealed to participants interested in learning about sounds. A key finding was that confidence indication — showing "High," "Medium," or "Low" with varying icon opacity — was important for users to trust the system. Cost was a major concern for all participants, as existing alert systems for individual sounds were already expensive.
Relevance
This research addresses a significant gap in deaf accessibility: while much assistive technology for deaf people focuses on communication (sign language, captioning), maintaining awareness of environmental sounds is equally important for safety, social participation, and quality of life. The design preferences and functional requirements identified provide a valuable framework for anyone building sound awareness tools for deaf users. The emphasis on peripheral, glanceable displays that minimize distraction while still conveying important information is a design principle applicable across many accessibility contexts. For practitioners, the finding that users want customization (choosing which sounds matter), history (reviewing what they missed), and confidence indicators (knowing when to trust the system) highlights that sound visualization is not just a technical recognition problem but a user experience design challenge. This work laid groundwork for the sound awareness applications that have since emerged on smartphones and smartwatches, including features like Apple's Sound Recognition in iOS.
Tags: deaf accessibility · sound visualization · peripheral display · sound recognition · ambient display · environmental sound · deaf and hard of hearing · user-centered design · auditory display