μGraph: Haptic Exploration and Editing of 3D Chemical Diagrams
Cristian Bernareggi, Dragan Ahmetovic, Sergio Mascetti · 2019 · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019) · doi:10.1145/3308561.3353811
Summary
This paper presents μGraph (Multimodal Graph), a system that combines haptic force feedback and speech to enable people who are blind or visually impaired (VIB) to explore and edit three-dimensional structural chemical formulae. Chemical diagrams represent molecular structures as graphs of atoms (nodes) and bonds (edges) in 3D space, and their spatial geometry is critical to understanding chemical properties — information that cannot be captured by linear text representations like chemical formulae. Existing assistive approaches either provide only 2D access (embossed images, audio-tactile overlays), cannot be edited (3D prints), or require sighted assistance (physical modelling sets like Molymod). μGraph uses a SensAble Phantom Omni haptic device — a stylus providing force feedback in a 16×12×7cm 3D workspace — selected through a design process involving a blind interaction designer who evaluated three candidate devices. Atoms are represented as haptic points with centripetal force that constrains the stylus, while bonds are represented as haptic lines that let the stylus slide along them with corrective force if it drifts away. Speech feedback announces atom names and bond directions. The system was designed with involvement of four blind participants, three of whom were computer science graduates with no prior haptic interaction experience.
Key findings
Four exploration modalities were implemented: search by name (keyboard entry moves stylus to a specific atom), free exploration (user moves stylus freely, feeling nodes and edges), guided node exploration (stylus automatically visits each atom, pausing for 1 second, ordered by distance), and guided edge exploration (breadth-first traversal along all bonds from the nearest atom). An evaluation with ten legally blind participants (aged 18-40) across three tasks showed strong results. In the exploration task (butane molecule), 5 of 10 participants correctly reported all atoms and bonds, with the remaining making only minor errors. All 9 reported atom positions were correct, and bond angle errors over 15° occurred only twice. In the editing task (creating ethane), participants averaged 4.7 minutes, with 8 of 10 placing all atoms within the 2cm accuracy threshold and all participants defining bonds correctly. In a comparative evaluation against ACD (Accessible Chemical Diagrams, a keyboard/screen-reader-only system), μGraph was slower (5.3 vs 4.4 minutes, p<0.05) but produced significantly more accurate results (p<0.005): all ACD participants misplaced at least one bond, while μGraph users could describe 3D geometric properties — atom positions, bond angles, spatial relationships — that ACD simply cannot convey. Six participants rated guided exploration as the most useful and easy-to-use feature.
Relevance
This work addresses a critical gap in STEM education accessibility: the inability of blind students to independently access and create 3D molecular structures that are fundamental to chemistry. The finding that haptic feedback enables understanding of geometric properties — spatial positions, angles, 3D arrangement — that speech-only solutions cannot provide has implications beyond chemistry for any domain where spatial graph structure matters (flowcharts, automata diagrams, cognitive maps). The trade-off between speed and accuracy is instructive: μGraph is slower because it provides more exploration modalities and richer spatial information, but this additional time yields significantly better comprehension. The editing capability is particularly valuable for educational settings, where students need not only to read but also to create and modify diagrams for assignments and exams. The complementary strengths identified — μGraph’s geometric understanding versus ACD’s hierarchical navigation — suggest that an integrated system combining both approaches could be most effective. A key limitation is the cost and availability of the Phantom Omni haptic device, which the authors identify as motivation for future work exploring commodity haptic hardware.
Tags: haptic technology · STEM accessibility · blind · visual impairment · chemistry education · force feedback · 3D interaction · graph accessibility · multimodal interaction