American Sign Language Recognition in Game Development for Deaf Children
Helene Brashear, Valerie Henderson, Kwang-Hyun Park, Harley Hamilton, Seungyon Lee, Thad Starner · 2006 · Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '06) · doi:10.1145/1168987.1169002
Summary
This paper presents CopyCat, an educational computer game that uses gesture recognition technology to help young deaf children (ages 6-11) practice American Sign Language (ASL) skills. The project addresses a critical language development challenge: 90% of deaf children are born to hearing parents who may not know sign language, meaning these children often lack the language exposure at home that is necessary for developing linguistic skills during the critical period for language acquisition. CopyCat is designed to let children practise signing complete phrases in a game context, with an animated character (Iris the cat) responding to the child's signs. The game uses a Wizard of Oz methodology during development, where a human interpreter behind the scenes evaluates the child's signs while the system appears autonomous. The recognition system combines computer vision (an IEEE 1394 camera tracking colored gloves via HSV color histogram adaptation) with wireless wrist-mounted accelerometers to capture both hand shape and motion data. Children wear small colored gloves that aid computer vision tracking and hold accelerometers that provide x, y, z acceleration data synchronised with video frames. The system trains Hidden Markov Models (HMMs) using the Georgia Tech Gesture Toolkit (GT²K), with a 22-word vocabulary organised into game phrases of three to four signs describing encounters with game characters.
Key findings
The recognition system achieved 93.39% average word accuracy for user-dependent models (trained and tested on the same child's data with 90/10 splits) and 86.28% average word accuracy for user-independent models (leave-one-out validation, training on four children and testing on the fifth). Individual user-independent word accuracies ranged from 73.73% to 92.62%, with the variation reflecting differences in signing clarity and consistency among children. A strong grammar modelling the game's phrase structure was adopted to handle coarticulation effects — the transitional motions between signs in continuous signing — using silence models between words. The dataset of 541 phrase samples and 1,959 individual sign samples was collected from five children (ages 9-11) at the Atlanta Area School for the Deaf playing the Wizard of Oz version over nine days. The data is notable for including the natural disfluencies of children's signing: false starts, fidgeting, hesitations, and varied signing styles, plus real classroom challenges like changing illumination and varied clothing/skin tones. The color histogram adaptation approach proved robust against illumination changes and even handled cases where a child wore a shirt with similar colors to the gloves. Participants who signed clearly and consistently in user-dependent tests also performed well in user-independent models, suggesting the models generalise effectively for consistent signers.
Relevance
CopyCat represents a significant advance in applying sign language recognition to real educational settings with actual deaf children, rather than controlled laboratory conditions with adult signers. The project's focus on encouraging language production — having children actively sign rather than passively receive information — addresses a gap in existing ASL educational software, which typically focuses on receptive language skills. The iterative design process involving the Atlanta Area School for the Deaf ensures the system responds to real educational needs. For accessibility practitioners, the key technical insight is that user-independent sign language recognition is feasible in classroom environments, though significant challenges remain with disfluencies and out-of-game signing. The broader lesson is that game-based approaches can motivate deaf children to practise language skills while simultaneously generating valuable training data for improving recognition systems — a virtuous cycle of educational benefit and technical advancement.
Tags: deaf education · sign language recognition · ASL · gesture recognition · computer vision · educational games · hidden Markov models · children · language acquisition