A Proposal to Adapt the Semiotic Inspection Method to Analyze Screen Reader Mediated Interaction
Lucas Pedroso Carvalho, Raquel Oliveira Prates, André Pimenta Freire · 2019 · Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems (IHC 2019) · doi:10.1145/3357155.3358455
Summary
This paper proposes SIM-SR (Semiotic Inspection Method for Screen Reader mediated interaction), an adaptation of the Semiotic Inspection Method (SIM) from Semiotic Engineering theory to evaluate how screen readers mediate the communication between designers and visually impaired users. In Semiotic Engineering, a computer system's interface is understood as a one-way metacommunication from the designer to users, encoded through three types of signs: metalinguistic (instructions, help text, tooltips), static (buttons, labels, layout elements that represent system state), and dynamic (signs perceived through interaction over time, representing system behavior). The standard SIM has five steps: analyzing metalinguistic signs, static signs, and dynamic signs separately, then contrasting the three resulting metamessages, and finally appraising overall communicability. The key insight of SIM-SR is that the screen reader acts as a translator between the designer's visual metamessage and the blind user, and this translation can introduce communication breakdowns that do not exist for sighted users. The adapted method adds new sub-steps to each sign analysis phase: the evaluator first inspects the signs without a screen reader (the standard analysis), then inspects them with a screen reader, and finally contrasts the two resulting metamessages to identify where the translation introduces problems.
Key findings
The paper illustrates the translation problems through concrete examples from Moodle (an open-source learning platform) using JAWS 18.0. A tooltip button with a question mark icon was read by JAWS as simply "Nao rotulado 1. Botao" (Unlabeled 1. Button), completely losing the metalinguistic information that was designed to help users understand their progress tracking. The tooltip text itself referenced visual characteristics ("a box with a solid border") and mouse-dependent actions ("click on it"), making it incomprehensible even if it could be accessed by screen reader users. A red-colored link intended to signal the creation of a new Wiki page conveyed no special meaning to screen reader users since color information is not narrated. An expand/collapse dynamic sign produced no screen reader feedback about the state change, requiring users to manually navigate back to discover the new state. A modal dialog for file deletion was not captured by the screen reader at all until the user interacted with it. These examples demonstrate that the method can systematically identify three distinct types of problems: communicability issues affecting all users, issues specific to screen reader users (signs that don't translate), and accessibility problems caused by inadequate coding or screen reader limitations.
Relevance
This paper offers a valuable theoretical framework for understanding screen reader accessibility that goes beyond checklist-based compliance testing. While WCAG conformance checks can identify technical failures (missing labels, absent ARIA roles), SIM-SR reveals the deeper communication gap between what a designer intends to convey and what a screen reader user actually perceives. This is particularly useful for practitioners who have achieved WCAG compliance but still receive reports of poor usability from screen reader users. The concept of the screen reader as a "translator" of the designer's message provides an intuitive mental model for explaining to design teams why accessibility requires more than adding alt text — it requires rethinking how meaning is encoded so it survives translation into a non-visual medium. The Moodle examples are especially instructive, showing common real-world failures: visually-dependent instructions, unlabeled interactive elements, color-only signifiers, and missing state-change announcements. A limitation is that the method had not yet been formally applied to a complete system evaluation at the time of publication.
Tags: semiotic engineering · screen readers · communicability · blind users · low vision · accessibility evaluation · metamessage · JAWS · Moodle
Standards referenced: WCAG 2.1 · ISO 9241-171