The sonic design space

Inventing the metaphors

With the visual rendering essentially a re-implementation of classic Tetris, the research weight fell on the sonic view. There was no existing vocabulary to borrow: the sonic components and their interaction metaphors were almost entirely new to Tetris, expressing abstract concepts such as gravity, orientation, topography, and relative distance. These are the metaphors I invented for the PhD implementation, together with field notes on how each survived contact with reality. The implementation ran on an ordinary laptop, in Java, with JOAL (OpenAL bindings) providing positional audio, an engine whose limitations themselves shaped the design in instructive ways.

A word on the status of these field notes, because it matters: they are designer introspection, checked against at most one or two informal testers. When I say a metaphor “works surprisingly well”, the fair expansion is worked surprisingly well for me, its designer, who knew what it was trying to say. Nothing in this part has been evaluated in the sense the auditory-display literature means by the word, that is, quantitative task measures across users in the tradition of Brewster’s and Walker’s work; the thesis’s planned formal evaluation was never run. The notes are hypotheses with one data point each.

Aside. Literally an aside: I whisper the type of the next tile, and the content of the hold box, into the player’s right ear. Low-priority peripheral information delivered on a spatially distinct, low-attention channel.

Musical sonar. I needed a way to express the quality of the tessellation between the falling tile and the ground. I play a single note for each column of the tile’s width, in sequence, around the user; the tune repeats every couple of seconds, or when the user moves or rotates the tile. The higher the note, the better the fit. It works surprisingly well, once you get the idea. New listeners took a while to understand it; a training mode belongs in any future version.

Dancing margins. A way to describe the distance of the falling tile from the edges of the playing grid, complicated by the fact that fallen tiles can obstruct movement, so the true “margin” is the distance the piece can actually travel. My solution was to place a sound to the left and right of the user, using 3-D distance to express grid distance. The audio engine’s positional quality let me down. I first made the sounds physically “dance” forwards and backwards to help the ears fix their locations, and when even that proved weak, the dance became a dance in music rather than location; oddly, that made the margin positions clearer. The lesson generalizes: when spatialization is poor, redundant musical encoding can carry what position cannot.

Talking scrollbar. The old idea of speaking text left-to-right so the listener knows how far through it they are, applied to tile position. Again the 3-D audio was not precise enough (the sound jumped perceptibly), so I scaled it back to three coarse locations: the falling tile’s sound plays left, middle, or right in front of the user. Taken together with the dancing margins, it locates the tile in space.

Direction as direction. Orientation of the falling tile: essentially north, south, east, west. I could simply speak it, but I was already whispering in the player’s ear, so I tried animating a sound passing the user in one of four directions. Forward/back motion was unconvincing in the engine, so I rotated the axes 45° to give NW/SE and SW/NE passes; since orientation, not true direction, is what matters, the rotation was harmless. In the end even this failed to earn its keep. The diagonal passes sounded odd and imprecise, and I fell back to speaking the orientation, but in a separate, male voice, distinct from the female voice describing the tile. Two lessons: spatial motion is a fragile carrier for categorical information, and voice identity is itself a usable channel.

Gravity as waterfall. The action of falling, and how far there is left to fall, delivered inside an already busy soundscape. My solution was falling water, with volume and pitch manipulated over time so the water feels nearer as the tile descends. I first implemented it as ambient sound and later as a point source; there was a qualitative difference between tweening the volume and tweening the location, in the point source’s favour. The metaphor asks nothing of the player: everyone knows what approaching water means.

Braided audio. The technique I adapted from the “Audio Hallway” work of navigating large music collections: splice the play-out of several streams into alternating segments. Playing the musical sonar and the dancing margins simultaneously, even from radically different locations, was discordant and distracting, so I serialized them into a braid and used the braid ratio to express priority: two scans of the sonar for every scan of the margins, because tessellation matters more than margins when playing. Braided audio thereby does two jobs at once. It shares a single musical play-out channel, and it encodes relative importance.

The wav assets of the original build tell the same story in miniature: spoken letters and numbers for the asides and scores, water sounds for gravity, and note sets for the sonar. A soundscape barely a dozen samples deep, doing the work of a screenful of pixels.

The voice of the game: third person becomes first person

What I find most interesting in the created audio metaphors is the effective change in the voice of the game. Tetris went from being a third-person observational game to a first-person immersive experience.

And it wasn’t deliberate.

The game became immersive because the player became the centre of all interaction modalities. The tile moves relative to the player; the margins are described relative to the tile the player is steering; gravity ebbs and flows toward the listener; the sonar plays out around them. Realizing I had changed the nature of the game, I went looking for alternative, observational audio metaphors for gravity, tessellation, and relative position. Beyond a screen-reader-style approach with multiple speaking actors, I came up empty. It appears to be in the nature of the sonic design space to be first-person immersive for anything beyond simple linear play-out of content.

The deeper reason emerged when I thought about coordinates. Visual Tetris cheats: it presents a spatial cognitive task as a flat projection on a rectangle, and the player’s visual system does the reconstruction. The moment the game moves to audio it is inherently in a listener-centric polar coordinate system, where everything is defined by angle and distance from the player. I can’t fake a flat rectangle, and I shouldn’t try.

If a third-person observational game naturally becomes first-person immersive under sonic rendering, what should happen to the classic WIMP interface? Windows, macOS, and the Linux desktops are all third-person observational visual interfaces, and what today’s assistive technology provides is the descriptive, spoken, screen-reader approach: an extremely limited set of metaphors. My experience with Tetris suggests a much richer set is waiting to be explored. Exploiting it requires the UI to be described in abstract terms and rendered according to user need, which leads straight back to the architectural model of this case study.

Looking at assistive technology I had seen deployed, the pattern repeats: adaptation seems only ever to travel one direction, toward immersion. Switch scanning is immersive: the user rides a moving play-out, waiting to strike within a time window. Screen magnification is immersive: the user no longer looks down on content but navigates within it. Even page re-ordering for screen reading is first-person, leading the user through content in a chosen order. I cannot think of a first-person modality that becomes third-person observational under adaptation.

There is one place the reverse can happen: adaptation for deaf and hearing-impaired users, where audio’s emotional and off-stage content gets re-presented visually as icons for events outside the field of view. First-person becomes third, and something is lost in the translation, because a static icon carries little of the emotive content the sound carried. If timely or emotive information is lost in adaptation, the interface is not wholly accessible. That places a hard requirement on the quality of the underlying abstract description: in games, content may need to be described in terms as basic as whether the news is good or bad, so that whatever design space renders it can find an expression with equivalent force.