Touching and Hearing GUIs: Design Issues for the PC-Access System
Christophe Ramstein, Odile Martial, Aude Dufresne, Michel Carignan, Patrick Chassé, Philippe Mabilleau · 1996 · Proceedings of the Second Annual ACM Conference on Assistive Technologies (Assets '96) · doi:10.1145/228347.228349
Summary
This 1996 paper from Canada's Centre for Information Technology Innovation presents PC-Access, a pioneering multimodal system that combines haptic (touch) and auditory feedback to make Microsoft Windows graphical user interfaces accessible to visually impaired users. The core insight is that existing accessibility tools (text-based interfaces with speech or braille output) discarded the spatial and graphical information inherent in GUIs, forcing blind users into a fundamentally different interaction paradigm. PC-Access proposed two configurations: one using an enhanced drawing tablet with sound feedback, and another using a haptic pointing device called the Pantograph — a small, low-cost force-feedback device that could physically push the user's hand to convey interface boundaries and object locations. The system was designed around a three-level model: a semantic level (interface concepts like windows, menus, icons), a syntactic level (fixed graphical/acoustic/haptic elements), and a lexical level (modality-dependent visual, acoustic, or haptic output). This model ensured that the same interface could be rendered through different sensory modalities while preserving its spatial structure.
Key findings
The sound design used four categories of auditory feedback mapped to GUI elements: musical tones (mapped to specific interface objects like icons and windows), verbal feedback (speech synthesis for labels and text), abstract sounds (with no direct physical relationship to objects, serving as abstract indicators), and output channels (stereo panning to convey directionality — e.g., left-to-right panning when resizing a window). Actions like maximizing were conveyed by playing the object's sound with added reverberation and increased amplitude. For haptic modelling, the system used force fields to represent GUI objects: icons, menus, and buttons were modelled as "physical boxes" with attraction forces that pulled the pointer toward them, making it easy to locate and select objects. Different objects had different "depth" parameters — deeper boxes attracted the pointer more strongly, making important objects easier to find. A "physical gutter" (hollow channel) was used along window boundaries so users could follow the edges of windows by feel, with asymmetric design at corners to indicate sides. Dynamic forces modelled as rubber bands connected actions like moving or resizing windows to haptic sensations. A Force Field Editor was developed as a Microsoft Windows application allowing designers to create, modify, and test force profiles using B-spline curves, and save them for use in PC-Access or other applications.
Relevance
PC-Access was remarkably ahead of its time in proposing that blind users should be able to interact with the same spatial GUI that sighted users employ, rather than being relegated to a separate text-based interface. The concept of translating graphical interface elements into haptic objects with physical properties — attraction, boundaries, depth — anticipated research directions that would be explored for decades afterward. For accessibility practitioners, the force field approach to GUI navigation offers a fundamentally different paradigm from the sequential, document-oriented model that screen readers eventually adopted. The multi-level design model (semantic, syntactic, lexical) remains a useful framework for thinking about how to translate interfaces across modalities. The paper's emphasis on enabling "direct manipulation" for blind users — feeling and moving windows, icons, and menus spatially — represents an alternative vision of accessible computing that prioritizes spatial awareness over linearized text access.
Tags: haptic technology · force feedback · multimodal interface · GUI accessibility · blind users · sonification · haptic feedback · accessibility history · assistive technology