Comparing Accessibility Evaluation Plug-ins
Tânia Frazão, Carlos Duarte · 2020 · Proceedings of the 17th International Web for All Conference (W4A) · doi:10.1145/3371300.3383346
Summary
This paper from the University of Lisbon systematically compares eight free Chrome browser extensions for automated accessibility evaluation: Microsoft Accessibility Insights, ACCESS Assistant Community, ARC Toolkit, aXe Chrome Plugin, Lighthouse, Tenon Check, TotalValidator, and WAVE. The tools were compared across three dimensions — feature sets, usability, and evaluation results — when applied to ten of the Alexa top websites (Google, YouTube, tmall.com, Wikipedia, Yahoo, Amazon, Reddit, live.com, Netflix, and Blogspot). The study used sandboxed environments to ensure each tool evaluated the same DOM state, and gathered information about which WCAG 2.1 success criteria each tool checks, how violations are classified, and whether tools support manual testing guidance alongside automated checks. The research was motivated by the recognition that while automated tools are essential for scalable accessibility testing, the W3C/WAI itself states that "Web accessibility evaluation tools cannot determine the accessibility of Web sites, they can only assist in doing so."
Key findings
The study revealed significant gaps and inconsistencies across tools. Even combining all eight tools, only 62 of WCAG 2.1's 78 success criteria were tested — meaning 16 criteria have no automated coverage at all. Individual tool coverage varied enormously: ACCESS Assistant Community ran the most checks (252 total), while aXe Chrome had only 44 and Lighthouse 47. Using all tools together provides approximately 10-40% more success criteria coverage than any single tool. The most frequently tested criteria were SC 1.3.1 (Info and Relationships), 1.1.1 (Non-text Content), and 4.1.2 (Name, Role, Value). The Understandable principle (17 success criteria) had the poorest automated coverage, hinting at the fundamental difficulty of automating checks for content comprehensibility. Tools using the same engine (aXe Chrome, Accessibility Insights, and Lighthouse all share aXe-core) produced similar but not identical results, since each wraps the engine differently. Critically, different tools found different violations on the same pages — some found more instances for the same criteria, while others detected violations in entirely different criteria. The most frequently violated success criteria across all tools and sites were 4.1.1, 1.4.3, 1.1.1, and 1.4.3. Usability varied: Accessibility Insights offered the best guided manual testing experience, aXe Chrome had strong documentation and semi-automated checks, WAVE's overlay icons could become confusing on error-heavy pages, and TotalValidator uniquely required installing a separate desktop application.
Relevance
This study provides essential practical guidance for accessibility practitioners: no single automated tool provides adequate WCAG coverage, and using multiple tools with different engines is strongly recommended. The finding that all eight tools combined still miss 16 success criteria reinforces that automated testing must always be complemented with manual evaluation and user testing. For developers choosing testing tools, the detailed feature comparison (which tools support manual test guidance, semi-automatic checks, severity classification, and fix recommendations) helps match tools to workflows. The variance in results between tools — even those sharing the same underlying engine — demonstrates that tool selection meaningfully affects what violations are found, which has implications for compliance claims and audit reliability. The study also highlights that the Understandable principle remains the hardest to automate, a gap that persists today and connects to broader challenges in cognitive accessibility testing. The recommendation to use tools with different engines (e.g., aXe-based plus WAVE plus Tenon) rather than multiple tools sharing the same engine is immediately actionable advice.
Tags: automated testing · accessibility testing · WCAG · web accessibility · evaluation tools · browser extensions
Standards referenced: WCAG 2.1 · Section 508