Multimodal Exploration of Mathematical Function Graphs with AudioFunctions.web
Dragan Ahmetovic, Niccolò Cantù, Cristian Bernareggi, João Guerreiro, Sergio Mascetti, Anna Capietto · 2019 · Proceedings of the 16th International Web for All Conference (W4A) · doi:10.1145/3315002.3332438
Summary
This demonstration paper presents AudioFunctions.web, a web-based system designed to make mathematical function graphs accessible to people who are blind or have visual impairments. Function graphs are a fundamental tool in mathematics education and STEM fields, yet they remain largely inaccessible through traditional assistive technologies like screen readers and braille displays. The conventional approach of embossing tactile drawings on paper is limited in the detail it can convey, cannot be dynamically updated, and requires specialized equipment. Prior digital alternatives using haptic force-feedback devices were cumbersome and uncommon, while earlier audio-only techniques could only convey the global trend of a function as a fixed sound sequence without supporting interactive exploration. AudioFunctions.web addresses these limitations by combining sonification with proprioceptive touchscreen exploration and speech synthesis. The system maps function values to sound frequency — higher y-values produce higher-pitched sounds — while sound intensity indicates proximity to the function curve, allowing users to trace the shape of the graph by following the loudest sound. On stereo devices, spatial audio reinforces left-right position awareness. Users can also play the entire function as a left-to-right sound sequence for a quick overview of its shape. The system notifies users of key points of interest such as local minima and maxima, axis intersections, and passage through the origin using earcons or verbal messages. For detailed analytical exploration, users can request spoken information about exact coordinates, first and second derivatives at any point.
Key findings
The system achieves cross-platform accessibility by supporting multiple input interfaces — touchscreen, keyboard, touchpad, and mouse — on both mobile devices and personal computers, adapting to different user abilities and preferences. This is a significant advance over prior one-size-fits-all approaches. AudioFunctions.web is built entirely with standard web technologies (JavaScript, SVG, Web Audio API, Web Speech API via the Tone.js library), meaning it runs in any modern browser without plugins or specialized hardware. A particularly practical feature is its ability to be linked directly from digital documents such as PDFs, ebooks, or web pages using URL parameters that specify the function, coordinate system center, scale, and earcon preferences. This makes it readily integrable into existing teaching materials. The system can even be packaged as a self-contained HTML file for offline use within digital documents. The keyboard interface includes automatic snapping to the function graph and escape-key navigation back to the origin, supporting efficient non-visual exploration.
Relevance
This work demonstrates how standard web technologies can be leveraged to create accessible alternatives to inherently visual mathematical content, directly addressing a major barrier to STEM participation for blind and visually impaired people. The approach of combining multiple audio modalities — continuous sonification for shape perception, earcons for points of interest, and speech synthesis for precise values — provides a model for making other types of data visualizations accessible. The system's design as a linkable web resource that integrates into existing educational documents is particularly practical, lowering adoption barriers for educators. For accessibility practitioners, this paper illustrates the value of multimodal and multi-interface design: rather than forcing users into a single interaction paradigm, offering touchscreen, keyboard, and pointer options accommodates diverse abilities and preferences. The reliance on open web standards also means the approach could be extended to other types of charts and graphs beyond mathematical functions.
Tags: sonification · mathematics accessibility · visual impairment · blindness · STEM accessibility · multimodal interaction · data visualization · assistive technology · web accessibility
Standards referenced: Web Audio API · Web Speech API · SVG · HTML5 · CSS