Predictive, Accessible Web Automation: A Longitudinal Study
Yury Puzis, Yevgen Borodin, I.V. Ramakrishnan · 2014 · Proceedings of the 11th Web for All Conference (W4A) · doi:10.1145/2596695.2596721
Summary
This paper presents a six-week longitudinal pilot study of Automation Assistant, a predictive web automation system designed to improve non-visual web browsing for blind users. The core problem is that screen readers force users to navigate web pages sequentially — a fundamentally different and slower workflow than sighted users who can visually scan a two-dimensional page. Traditional macro-based automation is too rigid, requiring users to create, manage, and replay fixed sequences that cannot adapt to page variations. Automation Assistant takes a different approach: it silently observes browsing actions and builds a computational model (using a Sequence Alignment Table) that predicts the most probable next actions. When triggered, it presents a small number of contextually relevant suggestions by filtering the page to show only suggested elements. The model is built continuously and incrementally without requiring any explicit setup from the user. The study used a single expert blind participant who browsed with the Capti Narrator screen reader for three weeks (baseline), then with Automation Assistant integrated into Capti for three additional weeks. The participant used the system for approximately 80% of daily browsing, with the remaining 20% on iPhone with VoiceOver.
Key findings
The participant made 20,019 keystrokes across 285 webpage visits over the six weeks, generating 809 automation instructions (80.5% link/button invocations, 19.5% form value changes). With Automation Assistant enabled, keystrokes per webpage dropped from 74.77 to 67.09, and time per webpage decreased to 91.53% of the baseline (measuring intervals up to 200ms, the most noise-free metric). The participant requested suggestions approximately 3.25 times per page and accepted them 0.54 times per page (including cases where the user briefly explored around the suggestion before accepting). The System Usability Scale (SUS) scores were revealing: Automation Assistant scored 75, Capti screen reader 67.5, JAWS 42.5, and VoiceOver 37.5. In direct comparison questions, the participant preferred Automation Assistant overall, found Capti easiest to use and quickest to learn, rated VoiceOver most complex and cumbersome, and felt most confident with JAWS. The participant reported that perceived time savings were substantial and noted that trust in the system grew as form field auto-completion was consistently verified as correct, suggesting that efficiency gains would increase further over time as verification behaviour decreased.
Relevance
This study explores a compelling alternative to the traditional screen reader model: rather than requiring web content to be made accessible through author-side changes (WCAG compliance, ARIA markup), Automation Assistant works on the user side by learning browsing patterns and reducing the interaction cost of repetitive tasks. The predictive, unsupervised approach — where the system learns silently without requiring the user to create or manage macros — addresses a fundamental usability barrier in non-visual web access. While the single-participant design limits generalisability, the longitudinal format captures something short-term studies miss: how trust develops over time and how automation tools integrate into real daily workflows. For practitioners, the key insight is that even well-built screen readers leave a large efficiency gap compared to visual browsing, and intelligent automation layered on top of screen reader functionality can meaningfully close that gap without requiring any changes to the websites themselves.
Tags: screen readers · web automation · blindness · non-visual web access · adaptive interfaces · machine learning · usability · assistive technology
Standards referenced: ARIA · HTML5