A Personalizable Mobile Sound Detector App Design for Deaf and Hard-of-Hearing Users
Danielle Bragg, Nicholas Huynh, Richard E. Ladner · 2016 · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16) · doi:10.1145/2982142.2982171
Summary
This paper presents the design and evaluation of a personalizable mobile phone app that detects sounds of interest to deaf and hard-of-hearing (DHH) users by learning from training examples recorded by the user themselves. Unlike existing commercial sound detection products — which are typically expensive, specific to individual sounds (separate devices for doorbells, smoke alarms, etc.), and not portable — this app consolidates detection of multiple personally relevant sounds into a single mobile device. The design was informed by a survey of 87 DHH participants (50 deaf, 37 hard-of-hearing, ages 18-99, mean 42) exploring which sounds matter across three contexts (home, work, mobile), how often sounds are missed, current sound awareness techniques, and design criteria for a detector app. The app allows users to create custom sound categories (e.g., "door knock," "microwave beep"), record multiple training examples for each, and the system uses machine learning (Gaussian Mixture Models with MFCC features) to monitor the phone's microphone and alert via vibration and pop-up notification when a trained sound is detected. Visual feedback is central: waveform visualizations during recording and editing help DHH users independently evaluate recording quality without hearing the content, and a real-time sound level display helps locate sound sources.
Key findings
The survey revealed that about 50% of both deaf and hard-of-hearing participants miss sounds more than once per day across all three contexts. Top sounds of interest at home were emergency alarms (90%+ for both groups), knocking on door, intruders, appliance alerts, baby crying, and doorbell. At work: emergency alarms, co-workers trying to get attention, and surrounding conversations. While mobile: sirens, bikes/people coming from behind, and whether blocking another person's path. Key differences between deaf and HoH groups: fewer deaf participants wanted to know about wake-up alarms and phone ringing (already having non-auditory solutions), suggesting deaf people have developed more established workaround systems. For notifications, participants most wanted to know sound identity, location, and urgency; volume and pitch were less important. Deaf participants had higher error tolerance (59% would accept one extra notification/day) than HoH participants (38%). A Wizard-of-Oz usability study with 12 DHH participants (ages 19-60) found all successfully recorded, edited, and organized training examples, with 91.7% strongly agreeing it was easy to record sounds. The waveform visualization was particularly valued — one participant explained that "if a deaf person could not hear it but wants to record whatever the sound is, they could possibly see the repetition and edit it down." The GMM-based sound detection algorithm achieved F-scores of 0.82 for alarms and 0.54 for door knocks (with cleaned training data), with background noise in training examples being the primary accuracy limiter.
Relevance
This research addresses a fundamental information access gap for DHH people: the inability to independently monitor their sonic environment for personally important sounds. For accessibility practitioners, the personalizable approach is key — rather than pre-defining which sounds matter, the system lets users train detection for any sound in their life, accommodating the enormous variability in which sounds are relevant to different people in different contexts. The visual feedback design is a model for how to make audio-centric tasks accessible to DHH users: waveform visualizations transform sound into visual patterns that can be evaluated, edited, and understood without hearing. The survey findings provide valuable data for anyone designing DHH-focused technology: the distinct priorities of deaf versus hard-of-hearing users, the context-dependent nature of sound importance, and the different error tolerance levels between groups. The finding that most participants did not use any mobile sound detection apps despite wanting one highlights a market gap. The study also raises important questions about training data quality — DHH users may not be aware of background noise in their recordings, suggesting that visual noise indicators would improve training outcomes.
Tags: deaf and hard of hearing · mobile accessibility · machine learning · sound detection · personalization · assistive technology · user research