Inclusive improvisation through sound and movement mapping: from DMI to ADMI
Alon Ilsar, Gail Kenning · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3416988
Summary
This paper presents a case study of adapting the AirSticks, a gestural Digital Musical Instrument (DMI) designed for expert percussionists, into an Accessible Digital Musical Instrument (ADMI) for people with disabilities. The AirSticks use Razer Hydra virtual reality gaming controllers as motion capture devices, connected through custom MIDI mapping software (CAMS) to Ableton Live digital audio workstation. Performers hold the controllers and move them through the air to trigger and manipulate sounds — position, orientation, and velocity all map to musical parameters through customizable configurations. The paper argues that the key to transforming a DMI into an ADMI is not primarily hardware or software modification, but rather understanding users' needs and customizing the movement-to-sound mappings accordingly. Four pre-designed mappings were created: AirHarp (a five-note harp scale controlled by spatial position), Warmth (a drone with timbre and intensity controlled by both hands), AirRadio (eight spatially-located sound tracks navigated by pointing), and FluidDrum (drum synth sounds morphed by controller direction). These mappings were designed to work with both large and small gestures, accommodate physical tremors, not require musical expertise, and use familiar sound metaphors to create immediate engagement. The instrument has been used in hundreds of live performances by its designer, and more recently in workshops with people with diverse disabilities.
Key findings
Two case studies illustrate the approach. Alessio, a teenage boy with leukodystrophy (a rare progressive genetic disease affecting the brain, spinal cord, and peripheral nerves), engaged with the AirSticks through the Safe in Sound collaboration. With no prior training, he immediately connected with the instrument. Adaptations included limiting all buttons to trigger the same note initially (so he could calibrate his movements), then introducing the FluidDrum mapping whose experimental "glitch aesthetic" appealed to him. Alessio progressed to performing publicly at the Copy That, Copy Cat JOLT Sonic Festival in 2019, demonstrating autonomy and agency. His mappings were personalized "on the fly" by observing how he moved. Violet, a 93-year-old woman with advanced dementia living in residential care, engaged through a more scaffolded approach — demonstration rather than verbal instruction. She immediately grasped the AirHarp mapping, swaying and singing along. When the AirRadio mapping produced sounds she disliked, the facilitator quickly switched to Warmth, which she engaged with vigorously. A collaborative improvisation between Violet and the AirSticks designer produced what the designer described as "a truly magical and spiritual moment" of non-verbal musical communication. The critical insight is that the hardware required minimal changes between able-bodied expert use and accessible use — the transformation happened almost entirely through mapping customization.
Relevance
This paper makes an important argument for the ADMI field: that existing DMI prototypes can be rapidly adapted for inclusive music-making through thoughtful mapping rather than ground-up redesign. This has significant practical implications — rather than building specialized accessible instruments from scratch (which a 2019 systematic review found was the norm), designers can leverage existing, proven instruments and focus effort on understanding individual users' movement capabilities and sound preferences. The emphasis on empowerment over therapy is also significant: for people with disabilities, music engagement is often framed therapeutically, but this work positions it as creative expression, autonomy, and flow. The case studies demonstrate that gestural interfaces, by freeing music-making from screen/keyboard/mouse paradigms, can engage people with very diverse abilities — from a teenager with a progressive neurological condition to a 93-year-old with advanced dementia. Limitations include the small number of case studies (N=2), the reliance on a specific discontinued controller (Razer Hydra), and the need for an expert facilitator to customize mappings in real-time.
Tags: music accessibility · accessible digital musical instruments · gesture interaction · inclusive design · dementia · creative accessibility · assistive technology · empowerment