Why assistive technology fails

Transliteration and its limits

Metaphor, with its relatives metonym and synecdoche, plays a central role in guiding expectation in user interfaces: the desktop metaphor organizes data; traffic-light colours present line quality; a scrollbar presents relative position. Most interfaces yield up a large number of metaphors on inspection, with no guarantee of consistency between applications (“pages” in a browser and a word processor behave quite differently under zoom).

Conventional assistive technology, such as screen readers and zooming tools, relies on automated transliteration between the default presentation and one suited to the user. Its success depends on how well it interprets the default presentation, including content expressed only through metaphor. As Barbosa put it, “the appropriateness and sophistication of interpretations is directly proportional to the expressiveness of the underlying domain models”. A screen reader would need the domain model of the platform and of every application, including each one’s metaphors. In practical terms it can’t have them. So screen readers transliterate little beyond the well-known metaphors of the host platform, and content carried by scalar representation or relative geometry is largely lost.

The platform “accessibility layers” (Windows, macOS, Java) expose instances of presentation elements, with some alternative content. They do not expose a model of the interface, nor the mapping between content and presentation. That mapping is left for the AT vendor to hard-code, version by version.

Now hold Tetris up against that machinery. Nearly everything that matters in the game is the kind of content transliteration loses: the silhouette is relative geometry; the fit between piece and gap is proximal, spatial, and continuous; urgency is carried by animation timing. A screen reader pointed at Tetris can say the score. It cannot say the game.

The conclusion I drew, and the thesis of the wider research, is that accessibility of this kind of content cannot be retrofitted by inspection from outside. The application must be built over an abstract model of itself, with rendering, in whatever design space suits the user, treated as a service that consumes that model. Accessibility becomes an architecture, not a layer.

A vocabulary of game presentation metaphors

The first thing to strike me when dealing with Tetris as a developer was the lack of standardized metaphors for expressing computer games. I was so used to thinking in HTML for the web that I felt almost naked without the familiar document metaphor of headings, paragraphs, tables, and divisions supporting description and rendering. So the first task was to identify common presentation metaphors in games, as they apply to Tetris.

Cockpit and head-up display. Classic Tetris is a cockpit: the playing area surrounded by instruments. A ghost piece overlaid on the playing area is head-up display; a version with both is a combination of the two.

Immersive and observational. In an immersive game the user is at the centre of the action with a restricted field of view, as in a first-person shooter. In an observational game the user is an omnipotent observer of the whole playing area: Tetris, Pac-Man, Manic Miner. Hold this distinction; it becomes the pivot of the entire case study.

Sprite-based animation. Elements perceived as appearing, moving within, and leaving the playing area. I identified eleven non-exclusive aspects of sprite behaviour: game-influenced; user-influenced; sprite-influenced; time-limited; morphing; translucent (the ghost); opaque; synchronized; handshaking (the tessellation of tile against landscape); pregnant (a locked tile separating into its four squares); and borg (those squares merging into the silhouette). A falling Tetris tile is simultaneously game-influenced, user-influenced, sprite-influenced, morphing, opaque, handshaking, pregnant, and borg. Each aspect has properties and potentially its own life-cycle describable as a finite state model; a sprite may take on and shed aspects as it moves through its life. The sprite has a state model, and so may each aspect of its behaviour.

Canvas-based and grid-based playing areas, in 2-D and 3-D. Tetris is grid-based 2-D: sprites jump a perceptible cell at a time, emphasizing the grid.

Gravity. Things fall, rather than merely move toward the bottom of the screen. You could play Tetris rotated ninety degrees, but the metaphor is emphatically falling bricks.

History. The last n actions are remembered; the silhouette is the game’s accumulated history made visible.

Elapsed time. From game start (Tetris) or first move (chess). Chess reminds us there may be several clocks at once.

This taxonomy did real work: it is what the adaptive renderer selects against. In the thesis’s terms, characteristics of the underlying content are matched to Design Language Sets, coordinated groups of interaction modalities analogous to design patterns or web templates, and the taxonomy above is the vocabulary in which Tetris’s characteristics get stated. The frame those sets live inside is the CISNA Model of accessible adaptive hypermedia, the thesis’s core contribution, published at W4A 2008. CISNA rebuilds the Dexter Reference Model of hypertext into five layers: Adaptation (selecting among alternatives by user, device, and context), Navigation (links and traversal), Semantics (composition and meaning, with ontologies and rules), Inventory (the formatted media elements available for use), and External Content(the raw content beneath). The Tetris work was an investigation of how DLSs for a real-time game could be expressed through CISNA: the game’s abstract events live at the Semantics layer, the metaphors of the next part are Inventory, and the choice of which metaphors a given player receives is the Adaptation layer doing its job.