Developing Hera-FFX for WCAG 2.0
José L. Fuertes, Emmanuelle Gutiérrez, Loïc Martínez · 2011 · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1969289.1969294
Summary
This paper describes the redesign of Hera-FFX, a Mozilla Firefox extension for semi-automatic web accessibility evaluation, to support WCAG 2.0. The original Hera-FFX was built around WCAG 1.0's simpler two-level structure of guidelines and checkpoints. When WCAG 2.0 was published in December 2008, it introduced a substantially more complex hierarchy — principles, guidelines, success criteria, situations, techniques (sufficient and advisory), and failures — that required a complete tool redesign rather than a simple update. The authors first establish a comprehensive set of 13 desirable features for accessibility evaluation tools, including automatic preliminary evaluation, support for manual filling, page presentation modification, annotated code view, local page evaluation, restricted-access page evaluation, rendered-page evaluation, report generation, training support, multi-session capacity, flexibility to integrate other guidelines, and fidelity to the WCAG 2.0 structure. They then survey four existing tools that supported WCAG 2.0 at the time — AChecker, TAW, Total Validator, and Worldspace FireEyes — finding that none provided full coverage of these features. The paper details the specific structural challenges of adapting to WCAG 2.0, including the increased number of hierarchy levels, the complexity of success criteria with their associated techniques and failures grouped by situations, the open nature of the techniques document that allows community contributions, and the need for new aggregation formulas.
Key findings
The authors developed three distinct aggregation algorithms to handle the complexity of WCAG 2.0 evaluation results: a permissive algorithm (where one passing technique is sufficient), a restrictive algorithm (where one failure causes overall failure), and a semi-permissive algorithm (requiring a minimum number of passes). Each technique or failure receives a six-element evaluation array representing fail, N/A, verify, OK, unknown, and partial states. These algorithms propagate results upward through the WCAG 2.0 hierarchy from individual techniques and failures through situations, success criteria, guidelines, and principles to produce a final page-level evaluation. An initial evaluation with four novice students showed that roughly 50% of WCAG 2.0 success criteria yielded better results when using Hera-FFX compared to students from previous courses who used no tool. For certain success criteria (1.1.1, 1.2.1, 1.2.2, 1.4.2, 2.4.2, and 3.2.1), all four students achieved 100% correct results with the tool. However, some success criteria showed worse performance, attributed to limited time with WCAG 2.0 rather than tool deficiency.
Relevance
This paper captures a pivotal moment in accessibility evaluation — the transition from WCAG 1.0 to 2.0 — and the significant tooling challenges that accompanied it. The detailed analysis of desirable evaluation tool features remains a useful benchmark for assessing modern accessibility testing tools. The aggregation algorithms formalized here address a fundamental challenge that persists today: how to combine multiple pass/fail signals across techniques, situations, and success criteria into meaningful overall conformance results. The distinction between permissive and restrictive aggregation reflects WCAG 2.0's nuanced approach where sufficient techniques offer multiple paths to conformance while a single failure can block it. For practitioners, the paper underscores that automated testing alone is insufficient — the tool was explicitly designed to support expert manual evaluation alongside automated checks, a principle that remains central to thorough accessibility assessment. The feature comparison table provides a template for evaluating current-generation tools.
Tags: accessibility evaluation · evaluation tools · WCAG 2.0 · semi-automatic evaluation · browser extensions · manual evaluation
Standards referenced: WCAG 1.0 · WCAG 2.0 · Section 508 · EARL