Motion History to Improve Communication and Switch Access for People with Severe and Multiple Disabilities
Guang Yang, Mamoru Iwabuchi, Rumi Hirabayashi, Kenryu Nakamura, Kimihiko Taniguchi, Syoudai Sano, Takamitsu Aoki · 2014 · Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS) · doi:10.1145/2661334.2661351
Summary
This demo paper presents the application of Motion History — a computer vision technique that visualises a user's movement history as a heat map — to support communication and switch access for people with severe and multiple disabilities. The system is part of OAK (Observation and Access with Kinect), software that uses a Microsoft Kinect to observe user motion without physical contact. Motion History displays movement frequency at each pixel using a six-colour scale (purple through red), where redder colours indicate greater movement at that point during the observation period. This visualisation serves two purposes: first, it helps therapists, teachers, and family members identify and understand voluntary movements that may be subtle and previously unrecognised in children with profound physical and intellectual disabilities. Second, the most active region of the Motion History can be selected to create a virtual "Air Switch" — a contactless switch that is triggered when the system detects movement in that region, automatically establishing an optimal switch fitting based on the individual's actual movement patterns. Seven non-speaking children with severe physical and intellectual disabilities (who had little or no success communicating with family members) participated in case studies. Motion History was recorded in 5-10 second intervals under different conditions (with/without various stimuli) to investigate each child's voluntary movement and cognitive responses. Based on feedback, a new tablet-based version was developed using the built-in camera, making the system more portable and easier to deploy in educational and care settings.
Key findings
The paper presents a detailed case study of a 12-year-old boy with intellectual disability and profound paralysis due to cerebral hypoxia at birth. Before the study, his movements were only subtle, and it was unclear whether they were voluntary. His classroom teacher reported subjective observations (the boy seemed uneasy when she left, relaxed when she looked at him) but could not quantify how much or how often these changes occurred. Motion History visualisation objectively revealed that the boy's movements changed measurably in response to the teacher's presence — his Motion History showed different patterns when the teacher was with him versus absent, providing evidence of both voluntary movement and cognitive awareness of his social environment. This was significant because it moved the assessment of the child's capabilities from subjective caregiver interpretation to objective visual evidence. The contactless nature of the system is crucial for this population: traditional physical switches require precise positioning and sufficient motor control to actuate, which may be impossible for individuals with profound physical disabilities. The Air Switch converts any detectable voluntary movement — however small — into a usable input, with the switch fitting determined by the individual's own movement data rather than a therapist's guess about optimal placement.
Relevance
This work addresses one of the most underserved populations in accessibility research: people with severe and multiple disabilities who have both profound physical impairments and intellectual disabilities, often with no reliable means of communication. For these individuals, the fundamental question is not "how can they use a computer?" but "what voluntary movement do they have, and can it be used for communication?" Motion History provides an assessment tool that makes subtle, previously unrecognised movements visible and quantifiable, potentially changing how caregivers, therapists, and teachers understand an individual's capabilities. The automatic switch fitting — where the system determines the optimal virtual switch location based on observed movement patterns — eliminates the trial-and-error process of physical switch positioning that is time-consuming and often unsuccessful for people with very limited motor control. For accessibility practitioners working with people with severe disabilities, this paper illustrates how computer vision can lower the bar for switch access from "can deliberately press a physical button" to "can make any detectable voluntary movement," dramatically expanding who can benefit from switch-based technology. The progression from Kinect-based to tablet camera-based systems also demonstrates practical attention to deployment constraints in educational and care settings.
Tags: severe disabilities · multiple disabilities · switch access · computer vision · Kinect · communication · AAC · intellectual disability · children