Evaluating Accessibility-in-Use
Markel Vigo, Simon Harper · 2013 · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461136
Summary
This paper introduces a novel method for evaluating web accessibility that goes beyond traditional guideline conformance testing. The authors argue that standards like WCAG 2.0 capture only about 53% of accessibility problems experienced by users, missing issues such as broken links, unexpected content, slow-loading pages, and information-dense layouts that create real barriers during interaction. They propose the concept of "accessibility-in-use" — defined as how well accessibility metrics predict the actual effects that problems will have on the quality of interaction as perceived by real users working toward real goals on real pages. Rather than treating accessibility as a static property of a web resource, this approach frames it as a quality of the interaction process itself. The method works in three stages: first, researchers observe and catalogue the behavioural tactics users employ when encountering problems on the web; second, these tactics are coded into detection algorithms that can automatically identify when users are struggling; and third, these algorithms are deployed into live websites via browser extensions, proxies, or direct injection. The paper situates this approach within existing evaluation methods including user testing, barrier walkthrough, GOMS modelling, and keystroke-level modelling, arguing that accessibility-in-use evaluation complements rather than replaces these established techniques.
Key findings
Through secondary analysis of datasets containing interactions from 24 blind and low vision users, the researchers identified eight distinct tactics (with 17 implementations) that users employ when facing web accessibility problems: asking for assistance, impulsive clicking, exploration tactics, narrowing down search, gaining orientation, re-doing, not operating or delegating on assistive technology, and giving up. These tactics serve as behavioural markers of underlying cognitive processes indicating problematic situations. The authors implemented several of these tactics as JavaScript detection algorithms, including a global orientation detector based on Mozilla Firefox's Places frecency algorithm that identifies when users retreat to familiar pages after becoming lost. These algorithms were deployed through WebTactics, a Firefox extension that unobtrusively monitors user interaction and reports detected problems to a remote database, capturing the timestamp, URL, and tactic type without interrupting the user's workflow. The tool demonstrated that real-time, unobtrusive detection of accessibility barriers during actual use is feasible.
Relevance
This research challenges the accessibility community to think beyond checklist-based compliance. For practitioners, it highlights that passing WCAG audits does not guarantee an accessible user experience — nearly half of real-world barriers may go undetected by conformance testing alone. The identification of user coping tactics provides a valuable diagnostic framework: if users on your site are repeatedly returning to the homepage, clicking links without clear purpose, or giving up after navigating several pages, these are signals of accessibility failures that no automated checker will flag. While the WebTactics tool itself is a research prototype from 2013, the underlying principle — monitoring real user behaviour to detect accessibility problems — anticipates modern analytics-driven approaches to accessibility. Organizations serious about accessibility should consider combining conformance testing with user behaviour analysis to capture the full spectrum of barriers.
Tags: accessibility evaluation · user interaction · behavioural analysis · screen readers · blind and low vision · automated testing · conformance testing · user experience
Standards referenced: WCAG 2.0