Evidence and uncertainty

Evidence is a record, not a reassurance

The package format gives evidence standing rather than leaving it to a README. It is one of six artefact roles, defined as a record of observation: what was tested, on which engine and assistive technology, on what date, with what result. The accompanying ownership rule states that an observation of assistive-technology behaviour is owned by an evidence record.

Ownership is the part that bites. A fact that lives only in a generated stylesheet, a design-tool library or a paragraph of prose has left the portable bundle, and the format treats that as non-conformance rather than as untidiness. So a sentence in a README claiming screen reader support is not a weaker form of evidence. Under the ownership rule it is not evidence at all, because prose carries the documentation role and a documentation artefact must not introduce a normative fact of its own.

The specification is unusually direct about why it has to say so. Where prose and contract disagree, the contract wins and the prose is a defect to be corrected, and the reason given is that a reader naturally trusts the readable file over the machine-readable one, and in this format that instinct is wrong. That is worth stating plainly because it inverts a normal instinct about documentation.

What a record has to say

Each record in the sample matrix carries eighteen fields, and they fall into four groups. Identity is id, componentId and claim, so a record states which component and which specific behaviour it concerns rather than a general impression. The environment is engine, engineVersion, browser, browserVersion, at, atVersion, platform, device, startingViewport and zoom. The observation is date, result, observation and tester. The last field, uncertaintyRef, points back at the question the record was made to settle.

Ten fields for the environment looks excessive until the records are read against each other. Two of the nine describe NVDA and JAWS on identical ground: Blink, Chrome, Windows. They differ in one field, and that field can change the result. So a claim qualified only by browser says nothing about which screen reader was present, and a claim qualified only by screen reader says nothing about which engine built the tree it read. The two are independent axes, which is why neither can stand alone.

Two more records use the same device in the other direction. Both describe reflow, both hold engine, browser, platform and device constant, and apart from the identifier the only fields that differ are startingViewport and zoom: one sits at a 320 CSS pixel viewport, the other at 1280 by 1024 with 400% zoom applied. Those two fields exist because a reflow claim without them says nothing. No content is clipped is a different statement under each condition, and a matrix that recorded only the claim would collapse them into one.

The remaining records vary the axes together, because that is how the combinations occur in use. VoiceOver appears on WebKit, Safari and macOS. Orca appears on Gecko, Firefox and Linux. Two records name a speech recognition tool rather than a screen reader, which is a different kind of question again: the claim is that the container introduces no element receiving pointer or voice-driven activation, so voice targeting of its children is unaffected. For a layout primitive that is the only honest form the claim can take, because the primitive exposes no operable target of its own to be addressed by voice. Another record has no assistive technology at all, carrying none in the at field, because its claim is that rem-anchored gaps grow with operating-system font scaling inside an Electron shell. Announcement, rendering, geometry and voice targeting are four different questions, and the record shape holds for all of them.

The tester field deserves its place. An observation has an observer, and a result with no one attached to it is an assertion wearing the costume of a measurement.

A result is one of five values, not a pass or a fail.

Two of those five carry most of the value. partial is the commonest real outcome of assistive-technology testing and the one a boolean has nowhere to put, which is why the vocabulary requires the difference to be described rather than merely flagged. And not-applicable is a different fact from unsupported. A behaviour that cannot occur on a platform is not a failure of that platform, and collapsing the two would manufacture defects and then invite somebody to fix them.

not-applicable also does a second job, and the file now says so rather than leaving it to be inferred. As a result it means the combination cannot exhibit the behaviour. In any other field it means that field does not apply to this record, which is why a zoom level reads not-applicable on a record about screen-reader announcement. Those are different statements, and one value doing two jobs silently is the kind of thing that later gets misread as a finding.

Absent evidence beats invented evidence

Every result in the sample package is not-yet-tested, as is every date, every observation and every tester. The version fields carry the same placeholder, with one exception that shows the vocabulary working as intended: the records with no assistive technology carry not-applicable for the assistive-technology version, because there is no version to state. No record in that file describes an observation that took place. The sample ships a fully formed matrix containing no findings whatsoever, and it says so in its own text.

That is a deliberate choice rather than an unfinished corner, and the reasoning is the load-bearing idea on this page.

Fabricated evidence is worse than absent evidence, because absent evidence is visible as a gap while fabricated evidence looks like a guarantee.

The same file states the principle behind publishing limitations at all, which is that a package showing only its guarantees is advocacy rather than documentation. Two prohibitions follow. A consumer must not treat a placeholder value as a test result. And a consumer must not infer support for a combination merely because the combination appears in the matrix.

The second one is easy to miss and matters more. Listing NVDA, JAWS, VoiceOver and Orca could be read as a support claim by anyone skimming, when the list is a statement about which combinations were judged worth investigating. The matrix is a set of questions. Only the result field turns any of them into an answer.

The package format backs this up at the point where it would be easiest to compromise. The full profile requires evidence records but does not require that they contain results, and a record whose result is not-yet-tested conforms. The specification gives the reason: recording an untested combination is the mechanism by which uncertainty becomes visible, and a profile that demanded results would create pressure to invent them.

The cost of this is worth naming. The sample is useless as a support reference, and it will stay useless until somebody sits down with four screen readers and two speech recognition tools across three operating systems, at two viewport conditions. What it does instead is demonstrate the shape of the record and the discipline of the placeholder, and I would rather ship a package that is honestly empty than one that is plausibly furnished.

Uncertainty is a record type too

A caveats paragraph in a README has no status, no identifier and nothing that would resolve it. It is a gesture at humility. In the component contract, an uncertainty is a structured entry with an id, a subject, a statement, a status and an evidenceRef.

The layout primitive in the sample carries four. The first records that whether any shipping screen reader announces or otherwise exposes the container element itself has not been tested. The second records that the behaviour of rem-anchored gaps under operating-system font scaling inside an Electron shell has not been tested. The third covers reflow at 320 CSS pixels of available inline size and at 400% zoom. The fourth covers whether the container interferes with pointer or voice-driven activation of its children. All four sit at status not-yet-tested, and all four point at the matrix through evidenceRef.

The third is worth dwelling on, because it was not there until I expanded the sample and it exposes something the package had been getting wrong. Reflow was already claimed, as a manual assertion saying no content is clipped at 320 CSS pixels and none at 400% zoom. No observation had been recorded against it. Under the propagation rule that makes it uncertainty, yet it was sitting in the contract as an assertion, so the package was claiming something it held no evidence for. Adding the reflow records forced the question, and the uncertainty entry is the answer.

That pointer is the mechanism, and it is what separates an uncertainty record from an admission of ignorance. Each one names the observation that would settle it, and the matrix names the exact combinations under which the observation would be made. An uncertainty is therefore a work item with an address rather than a shrug, and the pairing runs both ways, since every record in the matrix carries uncertaintyRef back to the question it belongs to.

The division of ownership is precise. Uncertainty is owned by the canonical component contract, alongside the semantic model, the keyboard contract, the Reflow behaviour, the WCAG mapping, the assertions and the non-guarantees. The observation is owned by the evidence record. The reference joins them without either one restating the other, which matters because a fact stated in two places is a fact that can disagree with itself.

One rule gives all of this teeth. Under the project’s rule, a claim whose supporting observation has not been made is uncertainty rather than a guarantee, and the vocabulary carries the rule inside the definition of not-yet-tested itself. The status of the evidence propagates to the status of the claim. That is a real constraint on what a package may say, because untested support cannot be described as a guarantee anywhere in the bundle while the matrix still reads not-yet-tested.

What is not settled about evidence itself

The section above argues that unknowns belong in the record. Applying that to the method itself, the evidence design has four gaps, and none of them is cosmetic.

Ageing is visible, and partly governed, but staleness is not. Because a record states engine, browser and assistive-technology versions alongside a date, a reader can always see how old a result is and what it was scoped to, and a result observed on one screen reader version makes no claim about the next. The versioning rules do cover evidence changing: adding a record that turns uncertainty into a guarantee is a minor payload change, because nothing relied upon has been taken away, while withdrawing an assistive-technology guarantee is a major change even when nothing else moves. What the format does not say is when a result stops counting. Re-test cadence and stale-result marking are open questions, named in the research agenda item on the assistive-technology matrix and again in the item on a minimum matrix per component. So a five-year-old supported record and one from last week are formally indistinguishable, and only the date tells the reader anything.

Coverage is undefined. Which engine, browser and screen reader pairs are mandatory has not been fixed, nor have the pass criteria. Set beside the profile rule above, the consequence is blunt. A package can declare the full profile, carry a matrix in which every single result is not-yet-tested, and conform. That is the right trade against invented results, and it also means the declared profile says nothing whatsoever about how well tested a package is.

The propagation rule is not enforced. The verification algorithm checks digests and schema validation, and requires a consumer to distinguish “checked and passed” from “not checked”, but it does not compare a stated guarantee against the evidence behind it. Both values are machine-readable, so the check is available to be written. It is simply not yet part of what conformance means, which makes the rule a discipline on the author rather than a property of the format.

And the propagation rule has not been applied evenly, even inside the sample that exists to demonstrate it. Two manual assertions still carry no evidence record and no uncertainty entry: one covers text-spacing overrides with a doubled root font size, the other covers visual order matching DOM order. Both are stated as assertions, neither has been observed, and by the project’s own rule both should read as uncertainty until they have been. Widening the matrix closed that gap for reflow and left it open for these two, which is a fair illustration of how the rule fails in practice: not by being rejected, but by being applied wherever attention happened to fall.

None of this is comfortable to publish on the page that argues for recorded uncertainty. It is also the only version of the page that practises what it describes.

References

• In the AFDS package format document, the artefact role definitions, the ownership rule and the status of documentation are clauses 6.1 to 6.3, the verification algorithm is clause 9, the profile rule on evidence without results is clause 12, and the payload-change rules for gaining and withdrawing evidence are clause 13. The evidence and uncertainty records quoted here are evidence/at-matrix.json, evidence/known-limitations.md and the uncertainty block of stack.spec.jsonin the sample package. The result vocabulary is also tabulated in the AFDS user guide, which states the propagation rule as the project’s rule. The open items are E1 and G3 in the research agenda. All are in accessible-by-design.