Testing across the hierarchy

If guarantees do not survive composition, then neither does evidence. Each level of assembly needs its own evidence, of its own kind, and the level at which a claim was tested has to be stated alongside the claim.

Evidence per claim level

What can be automated falls away rapidly as the level rises, and what remains is exactly the part that decides whether the interface is usable.

LevelAutomatableRequires human or AT evidence
PrimitiveRole, name, state, contrast, target size, focusabilityLittle
PatternRequired attributes and relationships, tab order within the widgetKeyboard grammar, announcement wording, AT variation
CompoundDuplicate identifiers, heading sequence, label associationFocus arbitration, announcement ordering, realistic content
RegionLandmark uniqueness, reflow at 320 CSS pixelsReading order coherence, whether the region makes sense heard
PagePage title, single main landmark, bypass mechanism, no duplicate identifiersWhether the whole page is navigable and comprehensible non-visually
ProcessVery littleTask completion, state retention, error recovery, timing, cognitive load

The gradient is the point. Automation is nearly sufficient at the primitive level and nearly worthless at the process level, which is the level at which conformance is actually claimed.

A ten-step method

These steps run in order, and each one exists because it catches a class of failure the previous step cannot see.

  1. Component in isolation. Automated checks, keyboard walk, name and role verification, against the component’s declared guarantees.
  2. Component with hostile content. Long strings, empty values, missing alternatives, translations, user-generated markup, extreme density.
  3. Component in a realistic page. Real heading context, real landmarks, real siblings.
  4. Pairwise composition. Test known collision pairs deliberately: dialog with combobox, grid in tabpanel, form with live-region status, sticky header with in-page links.
  5. Region assembly. Focus arbitration, announcement ordering and coalescing, keyboard grammar resolution.
  6. Page assembly. Landmark uniqueness, tab order across the page, heading outline, reflow at 320 CSS pixels, 200% text, 400% zoom, text-spacing overrides, forced colours.
  7. Process assembly. Task completion by keyboard alone, by screen reader, and by speech input; data retention across steps; error recovery; timing.
  8. Assistive technology matrix. Per browser and screen reader combination, per platform, with results recorded per combination rather than collapsed to a single verdict.
  9. Override and escape-hatch regression. Every documented override retested for the guarantees it is known to touch.
  10. Profile variation. Each supported capability and preference profile exercised, since a variant selected for one profile is untested for another.

Step 4 is the one most likely to be skipped and the one that pays back fastest. The collision pairs are a short, enumerable list, and they are where the compositional bugs actually live.

What the tools reach

The tooling is real and useful. It is also almost entirely concentrated at the levels where the failures are least interesting.

ToolReachesDoes not reach
axe-coreRule-based checks on a rendered tree, at any levelFocus arbitration, announcement ordering, keyboard grammar, comprehension
Playwright and axe-core/playwrightReal browser, keyboard simulation, viewport and zoom variation, per-state scansActual assistive technology behaviour
Storybook test addonPer-component and per-story checks in CIPage and process assembly, by construction
Accessibility Insights and IBM Equal AccessGuided manual assessment, tab-order and landmark visualisationUnattended regression at scale
ARIA-ATStandardised expectations for screen reader behaviour on APG patternsArbitrary compositions and bespoke compounds

ARIA-AT is the most significant of these for the argument in this section, because it is the only project that attempts to make screen reader behaviour testable at all rather than merely reportable. It is also scoped to patterns, which is precisely the level at which composition has not yet happened.

The hard limit

Component-level green is not evidence for the page.

Every component passing every automated check is compatible with a page that is unusable, and this is not a hypothetical. It is the predictable consequence of the eight failure modes: each of them lives in the relationships between components, and automated component-level rules examine components.

This is why a design system has to hold evidence scoped to a level and an environment, and why the level of the claim has to be recorded next to the claim. “This component is accessible” is not a false statement so much as an incomplete one: it omits the level at which it was established, the content it was established with, and the assistive technology it was established on.

References