The rhetoric of sound

The seven dimensions of audible meaning

Before deciding how to say something in sound, it is worth being blunt about how much sound can say. Setting aside speech, which is expensive in both time and attention, the dimensions an untrained listener decodes reliably are few:

• Azimuth, the left-right position of a source. The strongest of the lot, and the one that needs no explanation to anybody.
• Pitch, high and low, which maps readily onto up and down, more and less, near and far.
• Loudness, which reads as proximity or as insistence.
• Tempo and pulse rate, which read as urgency almost universally.
• Timbre, the character of a sound, which carries identity and category: this is a different kind of thing from that.
• Consonance and dissonance, which read as resolved and unresolved, settled and unsettled, good and bad.
• Voice identity, a speaker being recognisably a different speaker, which sorts information into streams without a word spent on saying so.

And then the weak ones.

• Elevation is far coarser than people expect: perhaps five distinguishable bands, not the twenty rows of a Tetris well.
• Reverberation and brightness suggest distance and enclosure, but vaguely.
• Front and back are close to unusable over headphones, because the brain resolves them largely through small head movements, and somebody sitting at a screen with headphones on does not make them reliably.

That is roughly seven dependable dimensions, delivered in series, against a visual display offering position in two dimensions, colour, size, shape, texture, motion and persistence, all in parallel and all continuously available. The asymmetry is not a detail. It is the design constraint that generates everything else, and the PhD build makes the point in miniature: the whole soundscape was barely a dozen wav samples deep, doing the work of a screenful of pixels.

Why the shortage forces figures

There is far more to say than there are literal channels to say it in. So transliteration is not available, and one thing has to stand for another. The figure chosen is not decoration laid over the interface. It is the interface, and it is doing compression.

Rhetoric has careful names for the ways one thing stands for another, and the distinctions turn out to be practical rather than academic:

• Metaphor maps across domains on the basis of resemblance. Something in the target behaves like something in the source, and the listener transfers the structure. Falling water for a falling tile is metaphor.
• Metonym works by association and contiguity rather than resemblance. What stands in is not similar to what it represents; it is connected to it, adjacent to it, part of its world. The crown for the monarchy. A different voice for a different category of information. Its close relative synecdoche substitutes a part for the whole, which is what a footstep does when it represents a person walking.
• Allegory is not a single substitution at all. It is a sustained frame in which many substitutions cohere, a whole small world within which the individual mappings make sense together and, importantly, within which the listener can reason.

To those three I would add the case that is not a figure at all, and which turns out to be the most instructive: the literal mapping, where a thing simply is itself. Direction represents direction. Distance represents distance. The auditory-display field has long had a version of this distinction, separating sounds that resemble their source from sounds that stand in by pure convention. What this work suggested to me is that the interesting boundary is not between resemblance and convention. It is between the literal and the figurative.

The seven devices, sorted by figure

Sorting the devices of the previous section by the figure each one uses puts the field notes in a different light:

• Gravity as waterfall is metaphor at its most effective, and it worked outright. It asks nothing of the player, because everybody already knows what approaching water means. That is the tell: the best metaphors are grounded in bodily experience the listener acquired long before meeting the interface.
• Musical sonar is metaphor of a more demanding kind, since sonar is not an experience most people have had. It worked, but new listeners took a while, and a training mode belongs in any future version. Metaphor drawn from something known only by description has to be taught.
• Aside is metonym, and theatrical: a whisper does not resemble low-priority information, it is associated with it, in the way whispers are associated with confidences. Cheap and reliable.
• Talking scrollbar is a figure borrowed from a figure, a scrollbar being already a metaphor for relative position. It half worked, and had to be coarsened to three positions.
• Dancing margins began literal, using real three-dimensional distance for grid distance, and failed. It was rescued only when the dance became a dance in music rather than in location: a symbolic encoding carrying what position could not.
• Direction as direction is literal, as the name concedes, and it failed most completely of all. What replaced it was voice identity, a different speaker meaning a different category, which is metonym and worked immediately and for free.
• Braided audio is not a figure at all. Nothing stands for anything. It is closer to prosody or rhetorical emphasis, being about how attention is allocated across what is said rather than what any part of it means.

The finding: literal mappings failed, figurative ones survived

Lined up that way the pattern is hard to miss. Direction as direction failed outright. Dancing margins failed as literal spatialisation and had to be rescued symbolically. The talking scrollbar had to be coarsened until it was barely spatial. Meanwhile the waterfall, the aside, the substitute voice and the braid all worked, and the sonar worked with teaching.

I do not think this is an accident of a weak 2009 audio engine, though the engine certainly was weak. The literal channels are the ones where sound has the least resolution to offer: fine elevation, fine distance, front and back. Reaching for the literal means asking the medium for accuracy it does not have. Reaching for a figure means asking the listener for an inference they can make effortlessly, using knowledge they already hold. The figure is not a compromise forced by a poor engine. It is the medium’s native strength.

Sequence and proportion as meaning

Braided audio deserves separating out, because it is the one device that operates on a different level from the rest. Playing the sonar and the margins at once, even from radically different locations, was discordant and distracting, so I serialised them into a braid and used the braid ratio to express priority: two sweeps of the sonar for every sweep of the margins, because tessellation matters more than margins while a piece is falling.

It does two jobs at once, sharing a single channel and encoding relative importance in the sharing. The general point is worth stating plainly, because it is a channel most interface designers never think to use: in a serial medium, the order and proportion in which things are said is itself a carrier of meaning.

What the figures are, and are not

A caution about status, because it would be easy to read the ledger above as findings. The seven devices are not the contribution. They are reifications: particular, contingent instances of underlying principles, built so that the principles had something to be tested through. Whether falling water is the best available figure for gravity is an empirical question I have not answered.

The claims are the general ones. That the literal channels are thin and the figurative ones rich. That a serial medium makes sequence and proportion into carriers of meaning. That spending one perceptual dimension twice creates a collision, as the record sets out at my own expense. That sound has no third-person vantage point to offer. The seven examples are how I arrived at those claims rather than the evidence that settles them, and the examples deserve replacing wherever something better is found.