Designing Embodied Musical Interaction for Children with Autism
Grazia Ragone · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3417077
Summary
This paper presents OSMoSIS (Observation of Social Motor Synchrony with an Interactive System), an interactive musical system designed to support embodied musical interaction for children with Autism Spectrum Conditions (ASC) during music therapy sessions. Developed by the author drawing on her experience as a music therapist, OSMoSIS uses a Microsoft Kinect motion capture system to track body movements and transform them into musical sounds in real time, without requiring children to touch any physical instrument or surface. The system maps body-part positions to instrument sounds (guitar, marimba, woodblock, water drops, and flowing water), with different instruments designed to address different motor features — for example, melodic guitar encourages upper body movements while flowing water elicits floor-level movements like rolling. The software is built with ReactJS, NodeJS, and Tone.js, and includes a facilitator GUI that allows toggling between body-to-instrument mappings, customizing sounds, and switching between demo and tracking modes. OSMoSIS can track up to three bodies simultaneously and stores anonymized movement data on a cloud MongoDB instance for later analysis. The system is designed to support observation of Social Motor Synchrony — the degree to which a child's movements synchronize with the facilitator's — as a metric for therapeutic engagement.
Key findings
Preliminary testing with 11 children with autism (2 female, aged 5-11, M=8.27) over four weekly 30-minute sessions using a within-subject design showed promising results. Children displayed a notable increase in engagement and interaction when OSMoSIS was activated compared to sessions without it. In the initial part of each session, when the facilitator invited children to copy movements with no music, children typically showed low engagement. Once the sonification system was switched on, children appeared keener to copy movements and freely explore the sound possibilities of their body movements. One particularly compelling case involved a child with high-level needs who often resisted changes — he attended all four sessions and, while staying in his mother's arms during non-music portions, would jump up and begin interacting when OSMoSIS was activated, especially with the flowing water sound. The system's touchless, non-instrument-dependent design means that children of any motor ability level can participate, removing barriers posed by traditional musical instruments that require fine motor control and musical training.
Relevance
This research sits at the intersection of music therapy, HCI, and autism accessibility, demonstrating how technology can make therapeutic music experiences more inclusive for children who may struggle with traditional instruments. For accessibility practitioners, OSMoSIS illustrates the principle that removing barriers to creative expression — in this case, the requirement to physically manipulate an instrument — can dramatically increase engagement for people with motor and communication differences. The system's use of sonification (converting movement to sound) is a design pattern applicable beyond music therapy, relevant to any context where non-verbal, embodied interaction could provide more accessible alternatives to conventional interfaces. The focus on Social Motor Synchrony as a measurable outcome offers researchers a concrete way to evaluate the therapeutic effectiveness of interactive systems. The study also highlights the value of designing technology that supports rather than replaces the facilitator-child relationship, positioning technology as an enhancement to human therapeutic practice rather than a substitute for it.
Tags: autism · music therapy · embodied interaction · motion capture · sonification · social motor synchrony · children · assistive technology · Microsoft Kinect