HoloLearn: Wearable Mixed Reality for People with Neurodevelopmental Disorders (NDD)
Beatrice Aruanno, Franca Garzotto, Emanuele Torelli, Francesco Vona · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2018) · doi:10.1145/3234695.3236351
Summary
This paper presents HoloLearn, a mixed reality application built for Microsoft HoloLens that aims to help people with neurodevelopmental disorders (NDD) learn to perform everyday domestic tasks and improve their autonomy. The system was designed collaboratively with psychologists and therapists at a care center in Milan serving over 80 individuals with NDD. Through focus groups, the therapists identified that many of their educational activities focus on teaching independence in basic daily living tasks — making this an ideal use case for mixed reality, where virtual objects can be overlaid onto real physical environments. HoloLearn implements two activities: laying a table (placing virtual glasses, plates, and cutlery in correct positions on a real table) and garbage collection (sorting virtual trash into the correct recycling bins placed on the real floor). The system uses HoloLens spatial mapping to scan the environment and place holograms contextually. A key design feature is an animated virtual assistant modeled as a Minion character, chosen by the care center residents themselves, which guides users through tasks with three configurable behavior modes: static (feedback only), dynamic (proactive hints), and reactive (responds to voice requests). Therapists can customize task complexity including difficulty level, number of objects, and virtual assistant behavior, allowing the application to be tailored to each individual. The system was built with Unity and C# using the Microsoft Mixed Reality Toolkit.
Key findings
An exploratory study with 20 participants (ages 16-43) across two severity groups revealed several important findings. Device acceptability was high — over 90% of participants in both groups willingly wore the HoloLens headset. However, the interaction paradigm posed significant challenges, particularly for the severe disability group. Coordinating gaze with the air-tap gesture was extremely difficult for many participants, especially those with fine motor impairments. The default white cursor was nearly invisible against light surfaces, prompting the team to replace it with a larger purple hand-shaped cursor with selection feedback, which improved usability. Only 2 of 11 participants in the severe group completed the task fully, compared to 6 of 9 in the moderate group. The virtual assistant was universally engaging — participants immediately recognized and tried to interact with the Minion character. Six participants who attended a second session showed measurable improvement in usability, suggesting a learning effect. A subsequent test with a physical clicker controller (replacing air-tap) showed all 8 participants understood how to use it immediately, indicating that alternative input methods are essential for this population.
Relevance
This study is one of the first to explore wearable mixed reality with people who have neurodevelopmental disorders in a real therapeutic setting, making it a valuable reference point for accessibility practitioners working with emerging technologies. The findings highlight a critical tension in immersive technology design: while the MR environment itself was well-accepted and engaging, the standard interaction mechanisms (gaze plus air-tap) created significant barriers for users with motor and cognitive impairments. The lessons learned — particularly around customizable difficulty, alternative input methods, visible cursor design, and the motivational power of familiar virtual characters — are directly applicable to anyone designing XR experiences for people with disabilities. The study also demonstrates the value of co-designing with therapists and end users, and of iterating on accessibility issues discovered during real-world testing rather than relying solely on lab-based assumptions.
Tags: mixed reality · neurodevelopmental disorders · intellectual disability · HoloLens · wearable technology · virtual assistant · daily living skills · assistive technology