XRMusic4VIP: Enabling Simultaneous Sheet Music Reading and Playing for Visually Impaired Musicians through Extended Reality
Julia Anken, Delia Blaess, Karin Müller · 2025 · ASSETS 2025: 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746339
Summary
This paper addresses a largely unexplored challenge in music accessibility: enabling visually impaired musicians to simultaneously read sheet music and play keyboard instruments. While assistive technologies for visually impaired musicians exist—including magnification tools, digital sheet music apps, braille music notation, and audio learning—none effectively support the dual task of reading music while playing at the same time. Current approaches force musicians to either memorize small sections before playing (a tedious, frustrating process) or rely on magnification tools that require constant manual zooming and scrolling, disrupting fluent playing. The researchers employed a dual-phase human-centered approach with the same group of ten visually impaired participants (four with visual impairments, six severely visually impaired) throughout. Phase I consisted of need-finding interviews exploring current practice challenges and desired system features. Based on these findings, the team developed XRMusic4VIP, a prototype using the Meta Quest 3 headset built with Unity and the MetaXR SDK. The prototype features two scenes: a VR preview scene for getting an overview of a piece (with magnification, color adjustment, and audio playback), and an AR main practice scene where virtual sheet music is displayed as a continuous horizontal band above the real keyboard, scrolling right-to-left at adjustable speed. The system uses video passthrough AR so users can see their real keyboard while reading virtual sheet music. Interaction is entirely hands-free via hand gestures (direct touch, pinch, grab, thumbs-up, stop), keeping hands available for playing. Phase II evaluated the prototype with eight of the original ten participants through guided walkthroughs, free exploration sessions with think-aloud protocol, SUS questionnaires, and semi-structured interviews.
Key findings
The prototype achieved a mean SUS score of 77.81 ("good" usability), with individual scores ranging from 45 to 95. Participant satisfaction averaged 4.13 out of 5. The most highly rated feature was customizability of the sheet music display (M=4.63, SD=0.48), including size, position, color, and contrast adjustments. The color inversion feature was particularly valued for enhancing contrast. The magnification function was rated positively (M=4.13, SD=1.36), though some participants found the horizontal band layout already sufficiently readable without additional magnification. Continuous scrolling movement received mixed ratings (M=3.75, SD=1.29)—some found it enabled fluent practice while others accustomed to memorization-based approaches found it disorienting. Most participants successfully perceived and followed two staves (polyphonic notation) simultaneously, though two participants could only manage one stave at a time due to restricted visual fields. Participants increasingly preferred gesture controls over physical buttons, appreciating the hands-free interaction. Critically, participants reported emotional responses—regaining joy in music-making, with one stating "It could make playing the piano fun again. Honestly, I had given up because you had to learn everything by heart." The prototype was perceived as a significant improvement over telescope magnifiers and traditional methods.
Relevance
This research opens an important new application area for XR accessibility, demonstrating that extended reality can effectively address the specific visual demands of music reading—which differs fundamentally from text reading due to its two-dimensional graphical nature and the need for both detailed and global vision. For accessibility practitioners, the study highlights several transferable design insights: the critical importance of customization options (size, color, contrast, position) for low vision users; the effectiveness of continuous scrolling displays that maintain focus on relevant content; the value of hands-free gesture interaction when hands are occupied with other tasks; and the benefit of environmental adjustment (brightness, contrast, saturation of the real-world passthrough). The dual-phase methodology—using the same participants for both need-finding and evaluation—provides a strong model for user-centered assistive technology development. Limitations include the prototype nature of the system, the small sample size, and the lack of a dedicated familiarization period, which may have affected results for participants new to XR technology.
Tags: extended reality · visual impairment · music accessibility · assistive technology · augmented reality · head-mounted display · keyboard instruments · user-centered design
Standards referenced: WCAG 2.1 · XR Accessibility User Requirements (XAUR)