Audio Borealis: sound in, light out.

PresetRainbowStrengthMediumMock voiceRecordingMicrophone

Mock voice

Settings

Every number below is the app’s values, grouped the way the code groups them, and they are live: drag one and the box above changes on the next frame.

ColorsVoiceLobesHills

Hue start

0°

Hue width

360°

Saturation

0.85

Drift

24°

Drift period

12.00s

Back to the app’s values

Take it with you

Everything above runs on @daformat/audio-borealis, which is the app’s driver and painter ported line for line, with tests around them. It has no dependencies and nothing in it is React. The pieces come apart: the driver is a loudness in and a frame out, the painter is a frame onto a context, and the frame carries everything the painter needs, so if you would rather draw it in WebGL or as SVG, the numbers are yours.

Install

Open the repo on Github (and drop a star if you like it!)

npm install @daformat/audio-borealis

Usage

import {
  attachBorealis,
  createMicrophoneSource,
  createMockVoice,
} from "@daformat/audio-borealis";

// A caption box, spoken to by a voice that is not there
const voice = createMockVoice();
const glow = attachBorealis(box, {
  look: "rainbow", // or northernLights, autumn, whiteHaze, or your own
  strength: "medium", // or strong, subtle, or an opacity in 0..1
  source: (dt) => voice.read(dt, isSomeoneTalking()),
});

glow.setLook("northernLights");
glow.configure({ flow: 120, idle: 0.3 });

// Or the real thing, from a click, since the browser will ask
const mic = await createMicrophoneSource();
glow.setSource(() => mic.read());

// Or levels you already have, pushed as they come
glow.feed({ loudness: 0.08, bands: [0.9, 0.6, 0.4, 0.2, 0.1] });

glow.destroy();

What it reads

A loudness and five bands, through four steps.

Whatever feeds the glow hands it two things: a loudness, which for real sound is the root mean square of the waveform, and the level in five bands, low to high, which for real sound is the spectrum cut at 300, 600, 1,200 and 2,400 Hz. The bands are optional, and without them the driver takes five on its own from the loudness.

Both readings go through the same four steps, and each one exists because the one before it was not enough. First the loudness is gained, by a base of 5 and the sensitivity knob, and gated, so the hum of a room does not glow:

raw = loudness × 5 × sensitivity, nothing below 0.06

Second, what clears the gate is rounded off, so a shout tops out instead of clipping. A hard clip is the first thing you see on a meter, but would have looked terrible here:

t = (raw − 0.06) / (1 − 0.06) → level = (1 − e−3t) / (1 − e−3)

Third, automatic gain. Every level is divided by its own running peak, which decays over four seconds and is never allowed under 0.3. A whisper and a shout fill the box the same way, because the glow is there to show that a voice is present rather than how loud it is, and the floor is what stops silence from being amplified into a shout of its own. Turn it off in the knobs and the box reads volume instead.

Fourth, a one-pole follower with a fast attack and a slow release, fifty milliseconds up and two hundred down, the bands fifteen percent slower on the way down than the level. The glow snaps up on a syllable and settles after it, which is the difference between something that reads as listening and something that reads as flickering.

What comes out is the rise, and it is the level to a power:

rise = level0.6

Below one, so it comes up fast at low levels. A caption box wants to say yes, I hear you at the first syllable, not at the loudest one, and 0.6 is where that lands without the glow being pinned to the top for the whole sentence. Everything else, the height, the width, the lift, how fast the lobes slide, is the rise times a knob.

Seven lobes on five bands

Where the light comes from, and why it moves.

The soft part of the glow is seven lobes fanned out from the middle of the bottom edge, each one a radial gradient squashed into an ellipse and each one standing on a band. The center stands on the lows, because that is where a voice’s weight is and the middle is where weight belongs; its two neighbors stand on the mids, the outer pair on the highs, and the far pair, small and low, on the low mids. A voice lights the middle first and the edges on its consonants, which is close enough to how a stage is lit that it reads as one.

A lobe’s height is its band, with a floor under it, so that no lobe ever quite disappears while the glow is up and a loud band more than doubles its own:

height = 0.6 + 0.7 × band

And they move. While a voice is heard the whole fan slides sideways, sixty points a second times the rise, and holds still the moment the voice stops. That one decision is most of why the thing looks alive rather than like a spectrum analyzer: a meter bounces in place, weather drifts. The fan wraps over a span of seven lobe widths, and each lobe fades to nothing over the last few points before the wrap, so the one leaving on the right is gone before it comes back on the left. Without that envelope you get a lobe popping in at the edge once every few seconds, which I did get, and which is the kind of thing you only see after you have stopped looking for it.

Five hills

The sharp part, and the reason it has no seams.

Over the lobes, five hills, one per band from low to high, each a bell standing on the bottom edge, filled from the edge up toward its crest and stroked one pixel along the top. The center-most rests in the middle and the rest alternate outward, the outermost at half the width from the center, and they wander sideways a little as the lobes flow, so the two layers are moving together without being locked together. A hill’s height is its band, with the same kind of floor:

apex = ceiling × (0.15 + 0.85 × band)

The ceiling rises with the glow and never takes more than 55% of the box, because the hills are meant to stand behind the words, never over them. The bell is not quite a Gaussian:

bell(t) = (e−|t/σ|^p − e−(1/σ)^p) / (1 − e−(1/σ)^p)

The subtraction at the end is what matters. A Gaussian is never zero, and a hill filled up to a curve that is never zero has a hairline of fill at both ends of the box where the curve was still a fraction of a pixel above the edge. Subtracting the tail makes the bell exactly zero at t = ±1, so each hill can be clipped and filled on its own and the join is invisible. The profile knob is p: 2 is a bell, higher is flatter on top, and the app sits at 1.75, a little peakier than a bell, because a rounded top read as a hump and a peaked one reads as a wave.

The colors

Seven shares of the wheel, never quite the same twice.

Each lobe has a color, and each hill takes the color of its band. They come from seven shares of the hue wheel, shuffled by hand so that neighbors contrast rather than blend into one wash:

0.94, 0.56, 0.76, 0.40, 0.08, 0.65, 0.49

A preset is where on the wheel those shares are laid, and the package calls it a look. Rainbow takes the whole thing. Northern lights takes the half from green through blue to violet, starting at 100°. Autumn takes a quarter from magenta round through red to orange, starting at 310°. White haze is white, which sounds like the boring option and is the one I use most, because on a busy screen the colors are one more thing moving. Then the whole set drifts, 24° either side of where it started, out and back over twelve seconds on a cosine, so the colors are never quite the same twice and never jump. Zero the drift in the knobs and watch how much staler it gets.

Painting it with what every engine has

Gradients, clips, and two compositing modes. No filters.

The obvious way to draw a soft glow on a canvas is to draw a hard one and run it through filter: blur(). It is also the way to get three different pictures in three browsers and a frame budget spent on blurring, so the painter uses nothing that is not in every 2D context that has ever shipped: gradients, clips, destination-in for the masks and lighter when the hills are set to add. No filters, no OffscreenCanvas, and no roundRect, which Safari got late; the corners are four arcs.

The lobes are painted twice, as two layers, a wide faint one and a tighter brighter one. Each layer is built on a scratch canvas the size of the box: the seven ellipses first, then an elliptical mask laid over them with destination-in, which keeps the lobes only where the mask is and fades them out before the lobes themselves would. Then the scratch is drawn onto the box at the layer’s opacity times the glow. The mask is the whole trick. Without it the lobes are seven blobs; with it they are one shape with seven brighter places in it.

Two things about the canvas itself, both of which cost me an afternoon. The first is that sizing a canvas clears it, and a caption box changes size on every word, so the glow was blinking out for a frame each time a word landed. The fix is to repaint the last frame in the resize observer’s own callback, which runs after layout and before the frame is painted, so there is never a painted frame with an empty canvas in it. The second is that the canvas is sized off the box’s own width and not its width on the page, because on the app’s landing page the box sits inside a screen that is scaled down to fit, and a canvas sized off the page would have been drawn at the wrong resolution and then scaled by the same transform, twice.

One frame loop serves every glow on the page, and it runs only while a glow has something to do: on screen, in a visible tab, and either reading a source or still fading out. A box with nothing to say costs nothing, which matters when the thing is under every caption box on a page, including the ones in the app’s own history stack.

A voice that is not there

What the demo is listening to, since a page has no sound.

The app reads a microphone or the system audio. The landing page has neither, so its demo needed a voice, and a recording would have meant a file, a load, and a loop you could hear the seam in. So the mock voice is arithmetic. Syllables of uneven length and loudness, some stressed, laid end to end with a short gap now and then where a word ends, each with a spectral shape of its own, a vowel low and in the mids and a consonant higher, and a third of them opening on a burst of highs, which is what a sibilant does to a meter.

It is deterministic in time, from a hash, so it repeats and stores nothing: eight syllables fill a cycle of 1.7 seconds and the cycle’s index reseeds them. And it starts over at each stretch of speech, so every line is said the same way, to the same peak, which is a thing the app cannot promise and a demo can. It is close enough to a voice that the meter reads it the same, and I know that because the meter above was the test: if the bars moved like they move on a real voice, the voice was done.

Real sound

An analyser, a root mean square, and four frequencies.

Real sound goes through a Web Audio AnalyserNode. The loudness is the RMS of the waveform, which for a voice at an ordinary microphone gain sits somewhere in 0.02 to 0.1, and with the driver’s gain of fifteen that is exactly the range its gate and its knee are drawn for. The bands are the spectrum in the analyser’s own decibels, cut at the four frequencies above and each averaged into 0..1 between the analyser’s floor and ceiling. None of that is precise, and it does not need to be: the automatic gain scales every reading to its own running peak, so what reaches the glow is the shape of the sound and not its size, and shape is all a glow can show anyway.

The recording in the demo goes through the same analyser, from an audio element rather than a microphone. It is the Mac’s own voice, one sentence at a time, trimmed and laid end to end with a breath between them, and its captions are typed from timings I did not have to measure: a synthesized voice reads at an even pace, so each word’s time is its share of its sentence’s length by letters, with a little extra after a comma. That lands the caption a word behind the voice, which is where the app’s captions land too. A script in the repo makes the file and the timings, and takes a recording of your own in place of the voice.

The microphone button does all of it in one call, from the click, since a browser will only ask on a click, and lets the microphone go the moment you pick something else, so the tab’s recording light goes out with it.

The app draws its glow in Swift, with Core Graphics; the package is a port of its driver and painter with the numbers unchanged, and the landing page’s demo runs on the same code.
The microphone is read in your browser and nowhere else. Nothing on this page records, stores or sends audio.