Eyes-free Exploration of Shapes with Invisible Puzzle
Andrea Gerino, Lorenzo Picinali, Cristian Bernareggi, Sergio Mascetti · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2811335
Summary
This paper presents Invisible Puzzle Game, an iPhone application that uses sonification techniques to enable people who are blind or visually impaired to explore shapes on touchscreen devices. Image exploration is fundamental for developing visuospatial skills and STEM education, but touchscreens—being flat glass surfaces—provide no tactile information. The researchers developed multiple sonification modes that convert visual information into audio feedback. The system implements six sonification modes. The "2-D" mode allows free exploration similar to tactile drawings on swell paper, playing a pure tone when the user touches a white pixel. Five "1-D" modes restrict exploration to the vertical axis, simultaneously sonifying all pixels on the horizontal "flush line" being touched and conveying spatial information through spatialized sounds with varying pitch. These 1-D modes are faster than 2-D exploration but require learning different audio rendering techniques. The application serves dual purposes: as an accessible game and as a research instrument for remote evaluation of sonification techniques. The game structure follows Csikszentmihalyi's "Flow theory"—gradually increasing difficulty while introducing new elements (points, lines, polygons). This design was developed using user-centric principles with feedback from multiple users including four people who are blind or visually impaired.
Key findings
Prior testing with the Invisible Puzzle Prototype collected data from approximately 150 subjects, including 18 who are visually impaired or blind. While these results were promising, the researchers identified limitations: the number of representative users (those with visual impairments) remained small, and the short test duration (~15 minutes) limited ability to measure learning effects over time. The Invisible Puzzle Game addresses these limitations through gamification elements designed to encourage sustained engagement. Game features include: performance indicators (0-3 golden stars per level), achievements for milestones like completing chapters, social network sharing to recruit new players, and global leaderboards. Levels are organized into themed chapters, and multiple exploration tasks per level prevent cheating through random guessing. A key innovation is the progressive introduction of sonification modes—new modes are unlocked as rewards for progress, maintaining interest while ensuring minimum training. Players can eventually choose their preferred sonification mode, generating data on both performance metrics and explicit user preferences. This approach enables longitudinal study of how practice affects performance with different sonification techniques.
Relevance
This research addresses a significant gap in touchscreen accessibility: while screen readers can describe text and interface elements, they cannot convey spatial or graphical information. For people who are blind, this limits access to diagrams, charts, maps, and educational graphics—particularly problematic for STEM education where visual representations are common. The gamification approach to accessibility research is notable. Traditional user studies with people with disabilities often struggle with small sample sizes and single-session data. By creating an engaging game that people want to play repeatedly, the researchers can collect longitudinal data from a larger population without requiring supervised sessions. This methodology could be applied to other accessibility research questions. For practitioners, the comparison of multiple sonification modes provides practical guidance: 2-D exploration offers intuitive free-form exploration but is slow, while 1-D modes are faster but require learning. The finding that both sighted and visually impaired users can be studied together (with sighted users playing eyes-free) also suggests inclusive research methodologies where representative and non-representative users contribute to the same evaluation.
Tags: sonification · blindness · visual impairment · touchscreen · image exploration · gamification · STEM education · audio feedback · spatial cognition · remote evaluation