Audio

Decision status: ADR-0039 is Accepted. ADR-0103, ADR-0104, ADR-0126, and ADR-0127 are Accepted and marked shipped. This page describes the current implementation, including places where it has moved past early ADR examples.

Audio is a native peripheral behind the singleton mailbox aether.audio. Components send symbolic note events or file-backed playback requests; the desktop substrate owns the output device, synth, mixer, and decoded PCM. The mail handler never renders audio itself. It moves control events through a bounded lock-free queue to the callback-owned mixer, while file reads, decode, resampling, and SFZ assembly stay off the real-time path.

Mental model

There are three lanes in one mixer:

• Synth voices come from immediate or scheduled note-on/off events. A voice uses one of the 11 compiled-in patches or a loaded sampled bank.
• Tracks are decoded WAV files. They live beside the voice pool, so note saturation cannot steal background music or ambience.
• Master processing adds a fixed-character reverb return, applies master gain, then soft-clips each output channel with tanh.

The sender in the mail envelope is part of audio state. The audio cap keys it on the proven envelope sender (ctx.sender(), ADR-0230): synth voices are keyed by (sender, instrument id, pitch), sender gain by sender, and tracks by (sender, lane, namespace, path). Each local component is its own sender, so components isolate one another without carrying a sender id in the payload. Session mail shares one key, and MCP mail shares the key of the RPC server that forwards it; callers that share a key should use distinct track lane strings when sharing a path.

AudioCapability is also an addressing marker available with the lightweight audio feature. Native state, cpal, WAV decode, and the worker thread require audio-runtime. A wasm guest can therefore name the capability and its kinds without linking the native stack. See audio/lib.rs and the feature declarations in aether-audio/Cargo.toml.

Public mail surface

Kind names are the wire contract; the Rust names in the second column are payload types. Do not send a Rust type name as a kind name.

The exact fields and schemas live in audio/kinds.rs. The capability has 15 kinds total when the six reply kinds are counted.

Notes and scheduling

NoteOn carries MIDI pitch, MIDI velocity, an instrument_id, and required bipolar pan. Pan 0 is center, -128 is hard left, and 127 is hard right; the mixer uses a constant-power law. NoteOff carries only pitch and instrument because pan is not voice identity.

Repeated note-ons with the same sender, instrument, and pitch stack. Each creates an independent voice. Matching note-offs release the oldest unreleased voice on that key, one at a time; an unmatched note-off is a silent no-op. The runtime does not special-case velocity zero: it still allocates a zero-amplitude voice, so send NoteOff when release is intended. The current behavior is pinned in runtime/synth.rs, even though some older comments describe retrigger or velocity-zero behavior differently.

The active pool is capped at 128 voices. At capacity, a new note moves the quietest current voice into a short forced-release list before entering the pool. An unknown instrument id, or a sampled bank with no region covering the requested pitch and velocity, warn-drops the note.

Schedule carries a vector of ScheduledEvents. Offsets are at_millis relative to the batch's receipt by the audio callback; the callback converts them to its sample clock, preserving simultaneous events and firing events within a render block at the due sample. Validation is atomic:

• the batch must be non-empty;
• it may contain at most 8,192 events;
• no event may be more than 600,000 milliseconds ahead; and
• the entire accepted batch occupies one event-queue slot.

Any validation failure rejects the whole batch. A full event queue also returns ScheduleResult::Err; no prefix is admitted. See runtime/schedule.rs and runtime/handlers.rs.

Instruments, tracks, and files

Built-in instrument ids are append-only and currently map as follows:

0 sine_lead, 1 square_bass, 2 triangle, 3 saw_lead, 4 pluck, 5 piano, 6 electric_piano, 7 pad, 8 kick, 9 hat, 10 snare.

Reordering that table is a wire break. The source of truth is runtime/instrument.rs.

PlayTrack and LoadInstrument address files as namespace plus path and delegate reads to aether.fs; audio does not own a second namespace registry. The v1 decoder accepts 16-bit integer and 32-bit float WAV PCM, averages multichannel files to mono, and linearly resamples once to the device rate. Decode happens on blocking-dispatch work, never in the callback. See runtime/decode.rs and File I/O.

A track has a per-play linear gain and optional looping. Replaying the same full key restarts it rather than stacking it; StopTrack with the same key starts an approximately 5 millisecond fade. Tracks remain centered mono sources and do not use the synth voice pool or per-sender synth gain.

LoadInstrument expects an SFZ file whose samples are WAV files in the same fs namespace. The supported SFZ subset covers control/group/region inheritance, key and velocity ranges, root pitch, sample paths, and loop points/mode; unknown opcodes warn and are ignored. A successful load appends a bank after the built-ins and returns its session-scoped u8 id plus resident PCM bytes. There is no unload or deduplication. Load order therefore matters, and ids do not survive restart. Details are in runtime/sfz.rs and runtime/load.rs.

Gain, pan, and reverb

Master gain clamps to 0.0..=1.0. Reverb send also clamps to 0.0..=1.0 and is dry by default; it feeds a fixed mono Freeverb-style network and adds the wet return equally to left and right. Sender gain clamps to 0.0..=4.0 and affects already-sounding synth voices on the next callback block. It does not create named buses or affect track gain.

Voice output is split left/right by pan. A stereo device receives left on channel 0 and right on channel 1; additional channels receive their average. A mono device receives the pre-pan mono sum, so pan is intentionally inaudible. The implementation is in runtime/voice.rs, runtime/synth.rs, and runtime/reverb.rs.

Invariants and failure modes

• The handler-to-callback queue holds 1,024 AudioEvents. Immediate notes, stops, and several callback registration/control events warn-drop on overflow. Schedule is the exception that reports queue overflow as Err. A sender-forget event warn-drops on a full queue like the other control events; the cap-side in-flight load purge still runs, so no new sound starts even when the forget never reaches the callback.
• The current gain-control handlers ignore a failed queue push and still return the clamped Ok value. Track start and instrument registration can likewise log a full-queue drop after their deferred request later resolves Ok. Treat these replies as validation/load completion, not sample-level acknowledgement by the callback.
• File read, decode, parse, and sampled-bank assembly failures are returned in the corresponding result kind. StopTrack, NoteOff, and fire-and-forget note events have no error reply.
• Loaded PCM and sampled banks are resident for the session. Long tracks are decoded whole; there is no streaming decoder or bank eviction.
• The public float controls are not explicitly rejected for NaN by the handlers. Send finite gain and size values; the documented clamp ranges are meaningful only for finite input.
• The callback owns voice, schedule, gain, bank, track, and reverb state. Do not add blocking work, allocation-heavy transforms, or capability calls to its per-sample path.

Sender lifetime

A sender's close deterministically ends its sound (ADR-0079 §8). The cap watches one monitor per sender on its first state-creating mail (note_on, schedule, set_sender_gain, play_track) and forgets the sender on its MonitorNotice:

• sounding voices release through their envelopes, never cut;
• every track of that sender fades through the same short fade, looping or not, and retires;
• the whole sender's scheduled entries drop, On and Off alike, so no post-close note spawns a voice up to ten minutes later;
• the gain row prunes once nothing still references the sender — a voice, a track, or a scheduled entry keeps it alive through the release tail, so no fading voice jumps to unity — while live senders' idle rows are never touched;
• an in-flight play_track load never starts: its cap-side entry is purged and answered Err.

The shared sender-less key has no actor to depart and is never watched or pruned. Replace fires no notice, so a republished actor keeps its sound; tombstoned ids are never reused, so there is no ABA.

Chassis and boot behavior

The desktop chassis composes the full AudioCapability. It accepts AETHER_AUDIO_OUTPUT / --audio-output — device (the default), null, or disabled — and an optional requested sample rate. null runs the real synth thread and event queue with no device and discards the samples, at the requested rate or 48 kHz; a test that composes AudioCapability on a SubstrateHarness uses it to exercise every handler without audio hardware. Disabled audio, no output device, or an unsupported requested rate is non-fatal to substrate boot: the capability stays registered in nop state. Immediate notes then disappear, while reply-bearing handlers implemented by that capability return an audio-not-initialised error.

The production headless chassis, the minimal hub, and the SubstrateHarness chassis compose no audio actor at all. A chassis composes only the capabilities it serves, so a component that declares depends(AudioCapability) is refused at load there, naming aether.audio as the dependency that is not live (ADR-0232 §6). These current facts are visible in headless/chassis.rs, hub/chassis.rs, and substrate_harness/chassis.rs.

Where to change or extend it

• Change wire vocabulary in audio/kinds.rs, then update handlers, schemas, tests, and the governing ADR. Required fields such as pan are strict on structured input; omission is an encoding error.
• Add a built-in patch only by appending to BUILTINS; never reorder existing ids. Add sampled-bank behavior in sample.rs/sfz.rs, keeping decode off the callback.
• Change scheduling limits or admission in runtime/schedule.rs and runtime/handlers.rs; change sample-clock execution in runtime/synth.rs.
• Add callback DSP only with an explicit architecture decision. ADR-0126 is a narrow exception for one master effect, not an open effect graph.
• Keep desktop and nop-mode behavior aligned when adding a reply-bearing kind. A new handler without a matching nop-mode Err reply otherwise becomes a settlement hang when audio is disabled or no device opens.