Certification or Conformance: Making a Successful Commitment to WCAG 2.0
Suzette Keith, Nikolaos Floratos, Gill Whitney · 2012 · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2207016.2207029
Summary
Commissioned by ANEC (the European consumer voice in standardisation), this study examined whether voluntary self-declaration or third-party certification of web accessibility conformance translates into actual compliance with WCAG 2.0. The researchers selected 100 websites from five European countries (Germany, Italy, Netherlands, Spain, UK) that claimed conformance through either certification by national bodies or voluntary declaration with WCAG logos. The sample comprised 76 government/public body websites and 24 commercial sites. Evaluation used SortSite automated testing across 25 pages per site (with over 100 WCAG 2.0 Level A and AA testpoints), followed by manual inspection of 5 pages for sites passing an initial threshold of 10 or fewer automated failures. The study built on earlier European benchmarking research (MeAC studies) which had found that only about 3% of websites passed the full range of Level A automated and manual checkpoints in 2008.
Key findings
The results were starkly disappointing: only 4% of government websites (3 out of 74) passed all automated testpoints at WCAG 2.0 Level A or AA, and after manual inspection only 2 websites passed all Level A checkpoints — both certified by national bodies (PubbliAccesso in Italy and Drempelvrij in Netherlands). No self-declared websites passed manual inspection, and none of the 24 commercial websites passed even the automated threshold. Most websites failed multiple criteria, averaging over 4 failed criteria per site (3.6 for certified, 5 for self-declared). Critically, just 5 Level A success criteria accounted for half of all failures: 1.1.1 Text Equivalence (failed by over half of all sites), 1.3.1 Information and Relationships, 2.4.4 Link Purpose, 3.3.2 Labels, and 4.1.2 Name/Role/Value. The evaluators noted that multimedia and third-party content were major contributors to text alternative failures, while improperly associated form labels and search box design caused many 1.3.1 and 4.1.2 failures. A follow-up survey revealed that inaccessible third-party content, lack of IT personnel know-how, and perceived tension between accessibility and design were the main implementation difficulties.
Relevance
This study provides sobering evidence that accessibility conformance claims — whether certified or self-declared — are frequently unreliable indicators of actual WCAG compliance. For organisations, the finding that just 5 success criteria cause half of all failures is actionable: prioritising text alternatives (1.1.1), proper semantic structure (1.3.1), meaningful link text (2.4.4), form labels (3.3.2), and ARIA name/role/value (4.1.2) would dramatically improve overall compliance. The study also highlights the systemic challenge of third-party content: organisations embedding external multimedia, social media widgets, or third-party services on their pages inherit those components' accessibility failures. For policy-makers, the research suggests that certification alone cannot guarantee accessibility without ongoing monitoring, and that voluntary declaration with genuine developer commitment can be equally effective. The gap between conformance claims and actual compliance remains a significant challenge in the accessibility field, reinforcing the need for continuous evaluation rather than point-in-time certification.
Tags: WCAG compliance · accessibility evaluation · certification · conformance testing · e-government · European accessibility · accessibility policy
Standards referenced: WCAG 2.0 · WCAG 1.0