Testing

A test earns its place when you can name the bug it would catch and that bug is plausible — a real branch, boundary, invariant, or wire contract a future edit could break without also breaking the test. That is the whole bar. Before you write a test, finish this sentence: "this fails if someone ___." If the only honest ending is "edits the test," the test is junk and CI is poorer for it.

The sharpest form of that question is what logic owned by this crate does the test exercise? Junk tests routinely pass the first question with a plausible story — "it pins the wire name," "it round-trips the kind," "it proves the kind is registered" — while exercising no logic this crate wrote. They run a #[derive] macro, the shared codec, or the inventory registration, all owned and tested once elsewhere, and confirm the value came back. If the only honest answer is "none of ours — it restates a declaration or re-runs machinery another crate already tests," the test is junk no matter how much ceremony surrounds it. Field-by-field assertions, a non-trivial value under test, and a confident doc comment are not evidence of load; they are the camouflage junk hides behind.

Junk tests are not free. Each one spends compile time, run time, and reviewer attention, and every false sense of coverage it adds is a place a real regression can slip through unnoticed. A small suite of load-bearing tests beats a large suite padded with tests that pass no matter what the code does.

What does not clear the bar

These shapes recur. None of them can fail for a reason you care about, so none of them belong in the suite:

  • Mirror tests restate the source as an assertion. assert_eq!(Foo::default().x, 0) sitting next to x: 0 in the Default impl breaks only when someone edits both halves together, which is to say never on its own. The common disguise is the derived constant: assert_eq!(NoteOn::NAME, "aether.audio.note_on") reads like it guards the wire name, but NAME is the #[kind(name = "…")] literal — the assertion's expected value is the same string retyped, with no independent source of truth. A rename edits the attribute, and the test sitting beside it is updated in the same motion. Every real consumer routes on NoteOn::NAME or its hash, so they track the rename for free; the literal in the test is the one copy nobody downstream uses.

  • Round-tripping a derive-only type. decode(encode(x)) == x over a type whose Serialize / Deserialize / Schema are all #[derive]d tests nothing this crate wrote. The roundtrip is symmetric: encode and decode are generated from the same definition, so any change to one changes the other in lockstep and the test still passes. It can fail only if the two disagree — which for a derived type means the derive macro is broken, and that is tested where the macro lives. Building an elaborate value and asserting each field survives does not change this; it confirms the shared codec is an identity function over your struct, which is the codec's invariant, tested once in aether-data. A roundtrip earns its place only when the type has hand-written ser/de, or an invariant the roundtrip actually exercises (a clamp, a normalization, a rejected input) — not when it is plain derives over plain fields.

  • Testing code you do not own. The unit under test must be logic we wrote. A test that exercises a dependency's behavior catches a bug only the dependency's authors can fix, and we keep it green by never upgrading. This covers the standard library and the compiler (pushing three items and asserting len() == 3, checking that #[derive(Clone)] clones), serde and every other third-party crate (does wgpu clear the surface, does tokio schedule the task, does fontdue rasterize), and any generated code whose generator already has its own tests. The codec is the trap here: it is ours, but it is owned once, in aether-data. Testing it means testing it there — re-running it from a consumer crate on a consumer's struct tests the consumer's #[derive]s and the shared codec, neither of which is that crate's logic. When a test fails, the fix should land in the crate the test lives in — if it would land in someone else's, the test was never yours to write.

  • Re-testing shared engine machinery from a consumer. Some logic we own is owned once and already tested where it lives: config resolution through #[derive(Config)] (argv > env > default, whether an env var is set, how the string parses), mail routing, settlement, id and lineage hashing, and everything the Kind / Schema derives emit. A capability test that asserts its own knob picks up AETHER_FOO, or that a missing variable falls back to the default, is testing the Config derive rather than the capability. The same trap wears other masks:

    • Derive-emitted registration. Asserting a kind appears in descriptors::all() guards nothing — #[derive(Kind)] emits the inventory::submit!, so the entry is present by the fact the type derives Kind and the crate is linked. There is no manual registration to forget; a missing entry means the derive is broken (tested elsewhere) or the type was deleted (a compile error, not a test failure).
    • Schema-shape assertions. assert!(matches!(Role::SCHEMA, SchemaType::Enum)) restates the enum keyword through the derive. The derive maps enum → Enum and struct → Struct mechanically; the assertion adds nothing the declaration does not already say.

    Test what the capability does with these — the value it computes, the mail it sends, the input it rejects — and let the machinery's own suite cover that the derive, the codec, and the registry work.

  • Mock theater. A test that stands up so many fakes it only ever exercises the fakes, asserting that a mock returns what the test told it to return. It verifies the setup, not the system.

  • No real assertion. Calling the function and never checking the result; asserting only that it "didn't panic"; or checking the output against a value the test recomputes the same way the code does. An assertion needs a known-good oracle, not a second copy of the implementation.

  • Vacuous bodies. assert!(true), an empty test, a loop that runs zero times, an early return ahead of the assertion, or a guard that skips on every machine that will ever run it.

  • Bulk duplication. Ten near-identical cases driving one branch with different literals. One table-driven case carries the same signal; the other nine are noise.

  • Coverage chasing. A test written to turn a line green rather than because the behavior matters — a trivial getter, a Display impl with no logic, an exhaustiveness arm that can never be reached.

The tripwire that looks like junk

A deliberately boring test can be load-bearing. Pinning the MailboxId lineage hash, a wire format's byte layout, or a KindId's numeric value reads like a mirror test — a flat assertion against a fixed value. The difference is what sits on each side. A tripwire pins a computed value — a hash, a serialized byte layout, a derived id — against an independent constant, so it fails when the logic that produces the value drifts even though the declaration that named it did not. That is a real contract: downstream code depends on the computed value, the value can change invisibly, and the test makes the change loud.

This is exactly what a derived-constant mirror is not. assert_eq!(NoteOn::NAME, "aether.audio.note_on") has the declaration on one side and a copy of it on the other — nothing is computed, nothing can drift on its own. If you want to guard a kind's wire identity, pin the thing consumers actually route on and that is computed from the name: assert_eq!(NoteOn::ID, KindId(0x…)). That fails if the hashing changes or the name changes, both of which move the id without touching any line a reader would notice. Pinning the name string against its own literal guards none of that.

Mark a genuine tripwire so the next reader (and the next sweep) can tell it from junk. A one-line comment naming the invariant and why it is pinned is enough:

#![allow(unused)]
fn main() {
// Tripwire: this byte layout is the wire contract with the hub. A change here
// breaks every connected engine — if this assertion fails, update the protocol
// version, do not just re-bless the bytes.
assert_eq!(frame.as_bytes(), EXPECTED_WIRE_BYTES);
}

The comment is necessary but not sufficient: a comment over a value that cannot drift on its own is a mirror with a story told over the top, and the sweep treats it as junk. The contract is real only when the pinned value is computed.

Where the test goes

Once a test clears the bar, the harness follows from what it checks. Engine-internal correctness goes to SubstrateHarness (aether-harness-substrate) with a concrete assertion (captured, reply, or count_observed over the reports a fixture mails to the harness observer); visual reductions and failure artifacts go through SubstrateHarness Capture (aether-harness-substrate-capture). Behavior over the wire — recipient-name resolution, fleet lifecycle, the RPC boundary — goes to FleetHarness (the aether-harness-fleet crate). FleetHarness is headless, so any rendered-output assertion has to use SubstrateHarness plus its capture extension, and any externally-addressable-over-the-wire assertion has to be FleetHarness.

Journal content goes to BloomeryHarness (the aether-harness-bloomery crate): it seeds a scratch journal from Batches, boots the shipped bloomery chassis over it in process, drives the mounted journal owner and bundle driver by proven reference — call, move_head, and settle, which follows AwaitProcessed → Processed to quiescence rather than sleeping — and asserts the records the loop appended with assert_appended. Expected values are literals the scenario writes, or the handles its seed staging returned; fold runs a view only over the actual journal, never to compute an expected value, so a bug in a fold cannot move both sides together. A crash needs a real process to die, so crash scenarios live in aether-chassis-bloomery's tests, where CARGO_BIN_EXE_aether-bloomery resolves: they fork the binary through FleetHarness with restart supervision armed (start_restarting, spawn_binary, await_restart), and read the journal back through the same expectations on the SeededJournal the engine opened.

For overlay rendering, split structural and raster proof deliberately. Assert exact rectangle geometry, clips, texture coordinates, tint, texture identity, projection space, and commit order through committed_overlay_snapshot (on aether-harness-substrate-capture's RenderHarnessExt); then use CaptureFrame reductions for the smaller set of outcomes that need end-to-end proof through projection, blending, rasterization, and GPU readback. The typed snapshot contains only batches accepted into the recorded draw plan, so missing textures, invalid/empty clips, and an over-budget overlay pass cannot masquerade as rendered work. It localizes a malformed submission, while the rendered capture proves the pipeline actually produced the intended pixels. The snapshot lists batches in commit order, the painter order the frame was recorded in: lineage order between senders and each sender's own send order inside it, never the order the mail arrived in. A harness send has no sender, so its batches come first.

Assert what a capability's work produced — a reply, committed_overlay_snapshot / committed_shape_snapshot, or pixels — and never whether a kind was dispatched to it. A dispatch count re-tests the mail system, and it passes when the work was wrong.

A test that drives neither harness and exercises none of our own pure logic is the case to look at hardest, because there may be no engine behavior under it at all. Our pure logic — the codec, aether-math, schema encode/decode, id and lineage hashing — is load-bearing to test in the crate that owns it (aether-data, aether-math). Re-running it from a consumer crate, on a consumer's derived type, is the junk case above, not a second copy worth keeping.

Within a SubstrateHarness visual test, reach for the narrowest oracle first. A concrete typed observation (reply::<R>, or count_observed over a report a fixture mails to the observer) beats a pixel check whenever the mail already carries the answer. When the behavior is genuinely visual, the aether_harness_substrate_capture::visual frame reductions (not_all_black, differs_from_background, coverage, centroid, bounding_box) turn a captured PNG into a scalar or coordinate assertion — pin a band, not an exact pixel, since GPU / anti-aliasing nondeterminism makes an exact golden image the wrong primary oracle.

Waiting on a pumped slot

A test that owns a pumped actor is that actor's driver, and it waits the way a pumped chassis driver waits (ADR-0161 §Decision 2): the slot's mail wake and each awaited root's settlement feed one channel, the slot drains on each mail wake, and the wait returns when the roots settle. Boot the actor through aether_substrate::testing::PumpedDriver rather than holding the slot:

  • An effect on a chain the test sends. send_tracked (several sends stage on the inbox before any turn runs), then settle on their roots, or send_and_settle for one. A reply precedes its root's settlement, so read replies after settle with try_recv, never as a wait condition.
  • A detached effect that lands on the pumped actor — a departure's MonitorNotice, a worker's self-wake. pump_until over the actor's own state: only the driver's drains change it, so a mail wake is the only thing that can make the condition true.
  • A chain whose actors all dispatch on the pool. There is no slot to pump: send tracked through the chassis and testing::await_settled on the root's settlement receiver, under the same patience a chassis gate gets.
  • What an actor's wire sent. Every birth's wire runs under one held wire root (ADR-0244). Call PassiveChassis::await_boot_settled() for everything the boot's wire hooks sent, or finish an embedder spawn with SpawnBuilder::finish_wire_settled() in place of finish(). For a guest a handler loaded, such as an autoloaded one, or a child a handler staged, call await_wire_settled(path) on the BuiltChassis or PassiveChassis with the actor's path. Each returns once every mail wire sent, and everything it caused, has been handled, and at once for a wire that sends nothing. A hold a handler-staged wire takes gates the chain that caused the birth, so settling that chain covers it.
  • A detached effect that lands on a pooled actor — a MonitorNotice from a close tail, a send_detached send, a child's after_init mail. No root the test holds covers it, so the observing actor signals a test channel and the test waits with testing::await_signal, which names the wait as the signal it is rather than as settlement. When the observed effect carries a value the test asserts on, the test waits with testing::await_event on the channel that carries it. A FIFO tracked barrier (a tracked send of the same kind to the same inbox) fits only mail already in the inbox: an actor's own sends flush onto the pool, so a barrier the test delivers directly can overtake them.
  • An actor's close taking effect. A pooled instanced actor's route drop lands at the registry owner after its closing chain settles, so neither settlement nor its MonitorNotice proves the route is gone. Call PassiveChassis::await_closed(actor): it returns once the close cycle ran and the owner applied the drop, so published_contract answers None. Never poll the route.

Never wait in a loop { drain; check; sleep } against a deadline. That loop is not the production wait, it fails a slow but healthy chain under load, and every test that copies it adds another. A wedged wait through these helpers fails at the chassis settlement cap (AETHER_SETTLEMENT_CAP_SECS) with the gate named.

Running a wasm-gated scenario

Every scenario that loads a component opens with require_wasm("<crate_stem>"), which locates the component's cross-built wasm under target/wasm32-unknown-unknown/<profile>/. <profile> is the one the most recent cargo xtask build-wasm / cargo xtask dist recorded in dist/manifest.json (debug by default, and debug when no manifest exists); the other profile's directory is never read, so a release build from cargo xtask package cannot shadow it. The gate prints loading <path> to stderr, which a failing scenario's output shows. Build that artifact first:

cargo xtask build-wasm

The command discovers the component set structurally — a workspace package with a cdylib target and an aether-actor dependency — so a new component needs no entry anywhere; it is cargo xtask dist without the chassis binaries, and it is what CI's pre-build step runs.

Without the artifact the gate fails the scenario:

SKIPPED (no wasm for aether_kit): run `cargo xtask build-wasm` — set AETHER_ALLOW_WASM_SKIP=1 to ignore (looked for <checkout>/target/wasm32-unknown-unknown/debug/aether_kit.wasm)

That is the whole point of the message. The gate used to return None here and let each scenario return early, which reports test … ok — a green suite that ran none of the code under test, indistinguishable from a green suite that ran all of it. An agent changing a component, running cargo test -p aether-kit, and reporting the change proven was reading that. If you genuinely cannot cross-build wasm, take the skip deliberately with AETHER_ALLOW_WASM_SKIP=1, and do not read the result as proof of anything the wasm would have exercised.

A missing wgpu adapter is the other half of the same gate and answers the other way: require_runtime still skips a visual scenario on a driverless box, because no command the reader can run puts a GPU there. AETHER_REQUIRE_RUNTIME=1 — which CI exports through cargo xtask transform verify.test — makes that a failure too, and overrides AETHER_ALLOW_WASM_SKIP.

Diagnosing a failing visual assertion

A frame reduction that fails leaves only its scalar diagnostic in the test log — the captured pixels it was scored against are gone by the time you read it. Arm aether_harness_substrate_capture::ArtifactGuard around the capture and its checks for a widget- heavy scenario where that scalar isn't enough to see what actually rendered:

#![allow(unused)]
fn main() {
let mut guard = ArtifactGuard::arm(
    "widget_panel_layout",
    png.clone(),
    checks.clone(),
    verdict.results.clone(),
);
// ... assertions on `verdict` that may panic ...
}

The guard is a plain Drop type. A passing test leaves it untouched and it writes nothing; an unwinding panic through its scope best-effort writes target/substrate-harness-artifacts/<id>/actual.png, measurements.json, and one mask_N.png per requested check — each mask rendered from the exact region/background/tolerance partition that scored it, so the artifact can never show a different verdict than the one that failed. A test that detects failure through a Result return rather than a panic calls ArtifactGuard::persist explicitly before returning Err. Attach an already-loaded same-size reference PNG with .with_reference_png(..) to also get reference.png and a pixel-wise difference.png — diagnostics on top of the region/mask read, not a second pass/fail oracle; the frame reductions above stay what decides pass or fail. CI uploads each failed shard's target/substrate-harness-artifacts/** tree for download; a green run uploads nothing.

ArtifactGuard owns visual bytes. For a declarative harness sequence that fails before a capture or reply assertion, use SubstrateHarness::execute_with_diagnostics instead of making ordinary execute a filesystem writer. It retains a failure-only, bounded JSON record under target/substrate-harness-artifacts/execution/<id>/ while returning the original typed ExecutionError; see the harness guide for its exact fields and the visual/non-visual split.