Designing Gesture-Based Applications for Individuals with Developmental Disabilities: Guidelines from User Studies in India
Sumita Sharma, Blessin Varkey, Krishnaveni Achary, Jaakko Hakulinen, Markku Turunen, Tomi Heimonen, Saurabh Srivastava, Nitendra Rajput · 2018 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/3161710
Summary
This paper presents the design, evaluation, and resulting guidelines from three gesture-based applications created for individuals with developmental disabilities in India, developed through a long-term collaboration with Tamana, an NGO and special education school in New Delhi. The three applications each target different skills: Kirana simulates buying groceries from a local Indian store to teach life skills including mathematical calculation, social interaction, and decision-making; Balloons promotes joint attention through collaborative pointing interactions where two users work together to pop on-screen balloons; and HOPE teaches motor coordination, cognitive skills, and colour/shape matching through grab, drag, and drop gestures with increasing difficulty levels. All three use Microsoft Kinect depth cameras for touchless gesture recognition, allowing full-body interaction without requiring users to hold any device. The research was conducted over multiple years starting in 2013, with participants ranging in age from 8 to 28 years, spanning diagnoses including autism, Down syndrome, cerebral palsy, and intellectual disability at varying functional levels. Participants were divided into groups based on functional ability as assessed by educators, and studies tracked completion scores, performance times, types of assistance needed, and qualitative observations across multiple sessions. The researchers embraced the concept of "inclusion-within" — designing applications that cater to a diverse group of learners within the same school environment, nurturing collaboration rather than competition.
Key findings
The studies yielded 14 design guidelines organized into three categories: application design, research study design, and the Indian context. Key application guidelines include providing clear start and end boundaries for gameplay to reduce confusion, offering multimodal feedback (visual and auditory simultaneously) to accommodate varying sensory abilities, incorporating continuous positive reinforcement including culturally appropriate praise (e.g., "shabash" in Hindi), designing evolving difficulty levels with easy transitions and no locked content, structuring content serially into manageable steps, simulating real-world scenarios with culturally appropriate interactions, and balancing gesture complexity with content complexity — increasing only one dimension at a time. Research guidelines emphasize designing for inclusion-within diverse learner groups, promoting self-efficacy through socially and culturally appropriate gestures, addressing ethical challenges unique to this population (prioritizing participant wellbeing over data collection), and providing physical assistance for gesture fatigue. India-specific guidelines address technology acceptance through stakeholder-inclusive design involving parents, therapists, and educators; addressing technology fears around addiction and radiation; and giving educators control to customize difficulty, content, and learning goals for individual students. Group 2 (higher functioning) participants typically mastered tasks in one session, while Group 1 participants showed steady improvement across three sessions with graduated assistance.
Relevance
This research makes important contributions to accessibility practice by demonstrating how gesture-based technology can serve individuals with developmental disabilities in resource-constrained Global South contexts where assistive technology adoption faces unique cultural, economic, and social barriers. The principle of inclusion-within — designing a single application flexible enough for diverse ability levels within the same group — offers a practical alternative to creating separate specialized tools for each disability category. The guidelines around balancing gesture and content complexity are broadly applicable to any interactive accessibility tool: changing only one variable at a time reduces cognitive overload and supports incremental learning. The emphasis on culturally appropriate design is a valuable reminder that accessibility solutions cannot simply be transplanted from Western contexts; gestures, social interactions, and reinforcement patterns must be locally meaningful. For organizations developing educational technology for people with disabilities, the stakeholder-inclusive approach — involving educators, therapists, and parents throughout — is shown to be essential for technology acceptance and sustained adoption.
Tags: developmental disabilities · gesture interaction · special education · inclusive design · Global South accessibility · autism spectrum disorder · intellectual disability · Kinect · life skills