Three web accessibility evaluation perspectives for RIA
Nádia Fernandes, Ana Sofia Batista, Daniel Costa, Carlos Duarte, Luís Carriço · 2013 · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461122
Summary
This paper presents a large-scale comparative study of web accessibility evaluation across three distinct perspectives that reflect how Rich Internet Applications (RIAs) are actually experienced by users. The three evaluation perspectives are: E1, evaluating the raw HTML before browser processing (the traditional approach); E2, evaluating after browser processing including dynamic loading via JavaScript; and E3, evaluating after browser processing while also triggering user interaction events to reach all possible application states. The study used QualWeb, an automated evaluation framework built on PhantomJS that implements 44 WCAG 2.0 techniques (31 HTML and 13 CSS). The researchers crawled over 14,000 URIs, ultimately evaluating 8,282 RIAs that had multiple interaction states. The tool architecture includes a Browser Processing Simulator that renders pages as a real browser would, and an Interaction Simulator that programmatically clicks interactive elements to discover additional application states. The study was motivated by the fundamental observation that modern web pages are no longer static documents — JavaScript and AJAX can dramatically alter page content, structure, and layout without changing the URL, creating a complex graph of states that traditional evaluation tools completely miss.
Key findings
The results demonstrate stark differences across the three evaluation perspectives. The average number of evaluated HTML elements per RIA increased dramatically: 1,152 in E1, 1,666 in E2, and 19,964 in E3 — meaning that interaction-triggered states contained roughly 12 times more elements than the initial browser-processed page. On average, each URI had approximately 12.5 reachable states through user interaction. Critically, the relative growth of failures exceeded the growth of passes when moving from E2 to E3, indicating that dynamically generated content triggered by user interaction tends to be less accessible than the initial page load. The strict rate metric (pass/(pass+fail)) showed consistently low accessibility quality across all perspectives — 0.041 for E2 and 0.039 for E3 — but the slight decrease in E3 confirms that interaction-revealed content has worse accessibility. RIAs with more states also showed a trend toward lower accessibility quality. The study found that regular evaluations (even those considering browser processing but not interaction states) overlook 92% of the states of the pages, meaning the vast majority of accessibility problems in dynamic content go undetected.
Relevance
This research addresses a gap that remains critical today: most accessibility evaluation tools still primarily assess static or initial-load content, missing the accessibility of dynamically generated content that users actually interact with. With modern single-page applications, progressive web apps, and JavaScript-heavy frameworks (React, Angular, Vue), the proportion of content generated dynamically has only increased since this 2013 study. The finding that 92% of application states are missed by conventional evaluation is a sobering statistic for organizations relying solely on automated scanning. For practitioners, this means that automated accessibility testing of SPAs and dynamic applications must include interaction simulation — crawling through clickable elements, form submissions, and state changes — to provide meaningful coverage. The observation that dynamically generated content tends to be less accessible than static content suggests that developers may apply accessibility practices less rigorously to JavaScript-rendered components, a pattern that warrants attention in code review and testing processes.
Tags: automated evaluation · Rich Internet Applications · dynamic content · AJAX · DOM · web accessibility evaluation · browser processing
Standards referenced: WCAG 2.0 · WAI-ARIA · EARL