Translating Color: Sonification as a Method of Sensory Substitution within the Museum
Silvia Dini, Luca Andrea Ludovico, Sergio Mascetti, Maria Joaquina Valero Gisbert · 2023 · Proceedings of the 20th International Web for All Conference (W4A) · doi:10.1145/3587281.3587706
Summary
This extended abstract proposes using sonification — the technique of translating data into sound — to make the chromatic elements of contemporary artworks accessible to people with visual impairments or blindness (VIB). The research addresses a fundamental challenge in museum accessibility: while audio description can effectively verbalize figurative elements of artwork, color and abstract visual compositions cannot be adequately conveyed through words alone. This is particularly problematic for non-figurative works (like Pollock) or works with low figurative density (like Kandinsky), where the color dynamism, spatial organization of hues, and inner rhythm of the painting are central to the artistic experience. The researchers draw on visual semiotics, which treats abstract painting as a language of opposing forms and colors that produce meaning, and position sonification within the broader framework of Accessible Intersemiotic Translation — the process of translating artistic meaning across sensory modalities. The proposed outcome is an assistive technology that combines word, sound, and touch into a simultaneous multimodal and multisensory experience, offering visitors with VIB a way to perceive the compositional and chromatic qualities that verbal description cannot capture.
Key findings
The paper establishes a theoretical framework for color sonification in museum contexts, drawing on the distinction between figurative and non-figurative art and the limitations of verbal audio description for conveying abstract visual elements. The researchers identify that for people who lost their sight during their lifetime, color retains significant experiential meaning that can potentially be evoked through sound associations. For those born blind, sonification can create new perceptual mappings between sound qualities and chromatic properties. The work positions the translator (the person creating accessible content) as both a user of the artwork and a producer of new meanings, operating intersemiotically between visual and auditory channels. The research proposes that effective sonification of paintings must account for the inner rhythm of the composition — how colors relate to each other spatially and dynamically across the canvas — rather than simply mapping individual colors to individual sounds. The paper frames this as ongoing research that will lead to the creation of assistive technologies, though specific implementation details and evaluation results are not yet presented.
Relevance
This research opens an important conversation about the limits of verbal description for art accessibility and explores sonification as a complementary approach. For museum accessibility practitioners, the work highlights that audio description, while essential, cannot fully convey the experience of color-dominant contemporary art — a gap that affects visitor experience for a significant portion of museum collections. The multimodal approach of combining word, sound, and touch aligns with universal design principles and could enhance the experience for all visitors, not just those with visual impairments. As a two-page extended abstract presenting a theoretical framework and research direction rather than implemented results, practitioners should view this as an early-stage exploration. The ideas connect to growing interest in multisensory museum experiences and could inform the design of audio guides, interactive installations, and digital museum accessibility tools.
Tags: sonification · museum accessibility · visual impairment · sensory substitution · art accessibility · multimodal · audio description · cultural accessibility