Improving the Efficacy of Automated Sign Language Practice Tools
Helene Brashear · 2007 · SIGACCESS Accessibility and Computing · doi:10.1145/1328567.1328570
Summary
This paper describes dissertation research on improving automatic sign language recognition for CopyCat, an interactive computer game that helps young deaf children practice American Sign Language (ASL). The work addresses a critical need: 90% of deaf children are born to hearing parents who do not know or are not fluent in sign language, making early childhood — a critical period for language acquisition — a particular challenge. CopyCat uses computer vision and wrist-mounted accelerometers to recognize children's signing as they communicate with animated game characters, encouraging the transition from isolated signs to phrase-level signing. The dataset was collected from 9 children ages 8-11 at the Atlanta Area School for the Deaf, who played the game for 5 sessions each over two weeks using a Wizard-of-Oz setup where a hidden human evaluator simulated the recognition system. The game vocabulary included 20 hand-based signs in a Subject+Preposition+Object grammar with sentences of four to six signs. The core thesis is that modeling the disfluencies that naturally occur in children's conversational signing — as opposed to training only on clean, scripted samples — can improve recognition accuracy for real-world use. The paper identifies key challenges in sign language recognition: distinguishing gestures from signs, handling context dependency (directional verbs, inflections), defining the basic unit of modeling, managing transitions between signs (coarticulation and movement epenthesis), and accounting for variable repetition cycles.
Key findings
Preliminary analysis of 16 of 45 total sessions (514 phrases from multiple children) revealed that the majority of signing samples (354 of 514) fell into a "Game Correct but Unclear" category — conversationally correct and intelligible signing that nonetheless contained disfluencies and unusual linguistic artifacts not present in scripted lab datasets. Only 117 phrases were both game-correct and clearly matching textbook form. The data captured a rich variety of naturalistic signing behaviors including self-corrections (children signing "wrong" or "start again" or using an "erase" hand-waving gesture), false starts (beginning with the wrong hand shape then switching), hesitations mid-sign while thinking of the next word, out-of-band comments directed at game characters, and variable pause placement between signs. Prior recognition results using Hidden Markov Models showed user-dependent word accuracies of 90.80-95.65% and user-independent accuracies of 76.90-92.62%, but these were trained on cleaner data. The planned disfluency ontology will classify these phenomena by structure, analyze their impact on recognition, and model select classes to improve accuracy. The paper also notes that sign labeling schemes need to expand beyond simple gesture labels to tag variations due to signer preference, accent, or regional variation.
Relevance
This research highlights a fundamental challenge in building real-world assistive technology: the gap between controlled laboratory data and the messy reality of how people actually use language. For sign language recognition to work in practical educational tools, it must handle the disfluencies, self-corrections, hesitations, and non-sign gestures that are natural parts of conversational signing — especially for children still developing their language skills. The CopyCat approach of collecting naturalistic data through game play rather than scripted sessions was ahead of its time and addresses a bias in ASR training data that persists today. For accessibility practitioners, the work demonstrates that educational technology for deaf children requires deep understanding of both the target language (ASL linguistics) and the target users (children's developmental patterns). The game-based approach also exemplifies effective design for child engagement — making language practice intrinsically motivating by embedding it in a narrative adventure rather than drill-based exercises. The research contributes to the broader goal of making sign language technology that works with the natural variability of human communication rather than requiring users to conform to machine-readable precision.
Tags: sign language · sign language detection · gesture-based interaction · computer vision · machine learning · deaf and hard of hearing · game accessibility · ASL education technology