Multi-touch Exploration and Sonification of Line Segments
Dragan Ahmetovic, Cristian Bernareggi, Sergio Mascetti, Federico Pini · 2021 · Proceedings of the 18th International Web for All Conference (W4A) · doi:10.1145/3430263.3452442
Summary
This paper presents SoundLines, a mobile app designed to help children with visual impairments develop spatial exploration skills through touchscreen interaction coupled with audio (sonification) feedback. The app is structured as a game where a kitten must be guided to find its mother cat by tracing a line connecting them on the touchscreen. Lines can be straight, curved, or broken, and must be followed without exiting their boundaries. The system was designed in collaboration with two experts in accessibility and education for children with visual impairments, following four design goals: inclusive usage by children with and without visual impairments, engaging interactions for prolonged use, caregiver supervision capability, and exploration of both single-touch and multi-touch interaction with diverse sonification. In single-touch mode, users scan the screen with one finger to locate both cats and then trace the line. In multi-touch mode, one finger anchors on the first cat while a second finger explores to find the other cat and trace the connecting line — mirroring the two-handed exploration technique commonly used by people with visual impairments when examining physical tactile materials. Three sonification modalities translate the finger's distance from the line center into non-speech audio: volume modulation (louder near the line edge), pitch modulation (higher pitch near the edge), and period modulation (more frequent pings near the edge, similar to parking sensors). A no-sonification condition relying solely on proprioceptive sensing was also tested.
Key findings
A preliminary user study with 4 visually impaired adults (2 blind, 2 severe low vision) found limited functional differences between single-touch and multi-touch exploration in terms of accuracy and errors. However, single-touch with volume modulation was significantly faster than multi-touch (p < .05). The key insight is that proprioceptive sensing (the body's sense of finger position and the spatial relationship between two fingers) is the predominant information source during multi-touch exploration, while sonification feedback provides more benefit during single-touch exploration. In single-touch mode, period modulation produced the best distance accuracy scores (staying closest to the line center), and was also the preferred and easiest-to-understand sonification for most participants. Most participants preferred single-touch exploration, likely because it aligns with how screen readers on mobile devices work. A practical challenge emerged with no-bezel touchscreen devices: exploration movements near screen edges triggered system gestures (notifications, app switching), disrupting the interaction — a significant accessibility issue for any touchscreen exploration app for visually impaired users.
Relevance
This research contributes to the underexplored area of spatial education for children with visual impairments, who often lack experience with spatial information because it is predominantly conveyed visually. The touchscreen-based approach offers advantages over traditional tactile materials: it is dynamic, customizable, and does not require expensive specialized production. The finding that proprioceptive sensing dominates during multi-touch interaction has design implications for any touchscreen application that uses multi-finger exploration — audio feedback may need to be designed differently (or may be less necessary) when users have a proprioceptive anchor point. The bezel-less touchscreen accessibility issue identified is relevant to all mobile accessibility work, as modern devices increasingly rely on edge gestures that conflict with screen exploration techniques used by visually impaired users. While the study is preliminary (N=4 adults, with the target population being children), it establishes the feasibility of sonification-guided touchscreen exploration and provides a foundation for the educational game design.
Tags: visual impairment · sonification · spatial cognition · mobile accessibility · tactile accessibility · education accessibility · game accessibility