Understanding Users in the Wild
Aitor Apaolaza, Simon Harper, Caroline Jay · 2013 · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461133
Summary
This paper presents a tool for capturing low-level user interaction data from web applications unobtrusively and longitudinally, enabling the study of "accessibility-in-use" — how real users with real goals experience accessibility in real-world settings. The authors argue that traditional laboratory studies suffer from significant limitations for accessibility research: controlled environments exclude unpredictable real-world factors, users behave differently when they know they are being observed (guinea pig effect), and predefined task models prejudge what users will do rather than allowing natural behaviours to emerge. Existing alternatives also fall short — web server logs are too coarse to capture meaningful interaction detail, while approaches requiring software installation are obtrusive. The solution builds on UsaProxy, a proxy-based interaction logging tool previously used in accessibility research, but modifies it to work through simple JavaScript injection rather than proxy configuration. A single script tag added to each page of a web application captures keyboard events, mouse movements (sampled every 150ms), mouse wheel interactions, clicks, and DOM state changes, sending them to a remote NoSQL database. The tool was validated through a formative study comparing two approaches — manual annotation of a web application versus JavaScript injection — deployed on a real application (kupkb.org).
Key findings
The formative study revealed that while manual annotation of web applications provided immediate high-level understanding of site usage, the JavaScript injection approach captured richer data that could deliver the same high-level insights plus much deeper behavioural detail. The authors modified their injection approach to identify specific events using DOM paths, achieving the precision of hard-coded annotation without requiring source code changes. The tool captures DOM state at each click event, storing diffs against a baseline DOM so that the complete state of the interface can be reconstructed at any point — critical for understanding dynamic web applications where state changes occur without HTTP requests. The authors make a key methodological claim: there is no need to collect specific information about users' disabilities because accessibility problems will manifest in the emerging behavioural patterns themselves. Users with similar disabilities or demographics will exhibit similar problematic behaviours, which can be detected through bottom-up analysis of interaction data rather than top-down testing against predefined models.
Relevance
This work addresses a fundamental challenge in accessibility evaluation: the gap between how users interact with web applications in controlled studies versus real life. For accessibility practitioners, the key insight is that conformance testing and lab-based user testing each capture only part of the picture. Real-world, longitudinal observation can reveal accessibility barriers that neither approach detects — patterns like users struggling with specific interface elements over time, developing workarounds, or abandoning tasks entirely. The bottom-up approach of letting problems emerge from behavioural data rather than testing against predefined models is particularly valuable because it can surface unexpected barriers. While the specific tool described is a 2013 research prototype, the principles it embodies — unobtrusive client-side interaction logging, DOM state tracking, and behavioural pattern analysis — are now commonplace in commercial analytics platforms, though rarely applied with an accessibility lens. Organizations could apply similar techniques to their own analytics data to identify accessibility pain points at scale.
Tags: accessibility-in-use · user behaviour analysis · usage mining · in-situ observation · longitudinal study · web analytics · user research · assistive technology