Tracing & settlement

Governing ADRs: ADR-0080 (mail lineage + settlement detection), ADR-0086 (settlement decoupled from the trace stream), ADR-0093 (hold-until-resolve dispatch), ADR-0094 (the owned-dispatch obligation guard). The contract — what "settled" means and what you must uphold for it — is stable. The mechanism that delivers it (the emit-time per-root counter, the per-actor trace rings, the guided walk that rebuilds a tree) is settling, so this page documents the contract and defers the internals to those ADRs.

Send one mail and the handler that receives it may send more, and those handlers more again — a single message fans out into a cascade of downstream work. Settlement is the engine's answer to the question that cascade raises: has everything this mail set in motion finished? Tracing is the same causal lineage seen as a tree — what happened, in what order, and how long each hop took.

The two are built from one body of bookkeeping but serve different needs. An agent driving the engine leans on settlement to know an effect has fully landed before it reads a frame or sends the next thing, and on the trace tree to see why an exchange was slow or where a chain stalled. An author writing a capability or component needs the model because a handler that replies late owes its chain an obligation — miss it and a caller hangs with nothing named.

Why it exists

The engine needs to know a piece of work is fully done only at specific points — when something has to happen atomically, once everything an earlier mail set in motion has finished. A frame can't advance until the tick's effects have played out (the frame lifecycle gates each stage on this); a component swap waits for the old instance to finish draining; a lifecycle step waits for wiring to complete; an agent waits for its mail's effects before it reads the result back. What's being waited on is never one mail but the whole chain of work it set off, and no single mail can report that the chain has closed — the work that matters is everything it triggered.

Without an exact signal, the only way to wait is to approximate — and every approximation of "has this chain closed" races. You could poll a counter until it looks quiet, wait out a fixed window, or send a redundant mail to ride FIFO ordering; each holds up on a quiet machine and breaks under load, because nothing tells the waiter that the last mail in the chain is still on its way. Settlement removes the guesswork: it tracks the causal lineage of every mail, so a gate waits on a fact rather than a timeout. That same lineage, kept anyway, doubles as a complete cause-and-effect graph — the trace tree.

The model: lineage and the settled condition

Every mail belongs to a causal chain, named by its root. A root is a mail sent from outside any handler — the chassis dispatching a Tick, a lifecycle step (init / wire / unwire), a mail injected over MCP or bridged from the hub, a capability's worker thread reacting to a socket read. Mail sent from inside a handler inherits its trigger's root instead of starting a new one. So one external stimulus and the entire cascade it sets off all carry the same root id, and the whole chain is identified by that single originating mail. (Chains that the chassis itself originates are marked by a sender of aether.chassis — the tagged id mbx-aaaa-aaaa-aaaa you'll see at the top of a trace tree.)

A chain is settled when it is closed. Under each root the engine keeps two counts: in_flight — mail sent but not yet finished handling — and held_open — explicit holds, described next. The chain settles the instant both reach zero. Given the hold contract below is honoured, that signal is exact: the counts never transiently dip to zero with work still on the way, so the chain reports settled once, as a fact rather than a hint. That exactness is the whole point — it's what lets a waiter trust the signal outright instead of falling back to a timeout. The invariants page states this as a guarantee; here is the obligation that earns it.

The hold contract

The counts are only exact if every mail that is still coming is already counted. For a synchronous reply that falls out for free: a handler that replies before it returns records the reply's send ahead of its own finish, so the chain can't reach zero with that reply outstanding. Nothing to do.

The gap is deferred work — a handler that kicks off a slow call and replies in a later turn, after it has already returned. In the window between the return and the eventual reply, that handler's mail is finished and nothing new is yet in flight, so the counts would fall to zero and the chain would settle early, reporting "done" while the real answer is still being computed. A waiter would be told the work finished before its reply ever arrived.

A settlement hold closes that gap. A handler with deferred work takes a hold on its root before returning and keeps it until the last send goes out; the hold sits in held_open, so the chain stays open across the window, and releasing it after the reply is what finally settles the chain. You rarely take a hold by hand — the sanctioned offload primitives manage it for you. ctx.dispatch_blocking (and the #[handler(task)] completion sugar) acquire the hold when they spawn the worker and release it when you resolve, reply-then-release in that order so the settle can't beat the reply; spawn_inherit holds for the worker's lifetime. acquire_settlement_hold is the explicit handle for the rare hand-rolled case, such as buffering a request to drain in a later turn. It hands back an Option<SettlementHold>, and None is a real answer: a context with no in-flight root has no chain, so there is nothing to keep open. Work deferred from such a context is outside settlement entirely — no waiter can be told when it finished — which is why the absence is a value you handle rather than a guard that quietly gates nothing. The full offload story — which primitive for which shape, and why a raw std::thread::spawn silently opts work out of settlement — lives on Concurrency & blocking; this page is the why behind the hold.

The obligation guard

There is a second way a chain closes, on the receiving side. A capability that owns an inbox mailbox — one that takes an owned dispatch off a downstream channel — owes that inbound mail its Finished, or the chain never closes. The substrate brackets the two synchronous mailbox shapes automatically — an inline handler, and the standard actor dispatcher at handler exit — so an ordinary actor never has to think about this. The third shape is a mailbox drained by hand, off the pool: a capability that claims a mailbox and drives the drain itself, the way the desktop chassis's window driver does. Window operations have to run on the OS event-loop thread rather than a pool worker, so aether.window is claimed as an inbox the event loop drains between frames (Window covers that mailbox's surface).

A claimed mailbox is settle-safe by construction (ADR-0106). The claim hands the capability a SettlingInbox, not a raw receiver, and its drain methods yield each mail as an InboundMail guard that records Finished and replies along the inbound's causal chain. The guard settles when it falls out of scope — on every arm, including a decode error, an unrecognised kind, an early return, or teardown of the drain with mail still queued — so a leak at this seam is unrepresentable in the consumer's code, the same way the standard dispatcher's unconditional discharge tail keeps an actor author from leaking. Settling on scope exit rather than on payload access is deliberate: a reply's Sent must be recorded before the inbound's Finished, and InboundMail::reply sends inside the guard's scope, so the reply joins the chain before the drop closes it.

Underneath, every owned dispatch still ends one of two ways — discharged (the obligation ends here; I am recording Finished) or transferred (the obligation moves with the work onto a downstream envelope). The framework drain does the discharge; one relay type, built where an inbox channel is opened, moves a dispatch onto an actor's inbox and does the transfer. An owned dispatch dropped while still armed panics at the leaking seam in debug builds, naming the mail_id, kind, and mailbox (ADR-0094); release builds carry no field and no check, so the guard costs nothing. The guard now backs the in-crate relay and park seams rather than the claimed-mailbox drain, which no longer needs it — forget it on a relay you write and you still get an immediate, located failure instead of the silent multi-second hang the chain would otherwise wedge into. Parked mail is unarmed (the obligation guard does not apply), but it is still settled: when the Mailer tears down, every mail still parked on a never-resolving handle receives its Finished, so the upstream chain closes rather than hanging.

Two handles, because there are two counts

Replies bring you into contact with two RAII types, and which one you are holding follows directly from in_flight and held_open.

InboundMail brackets a drained envelope. Its drop records Finished, so it moves in_flight. Replying is a capability it offers along the way — it has the sender and the reply lineage to hand — and reply returns false when the sender asked for nothing.

DeferredReply is the obligation to answer someone. It carries a SettlementHold and a reply target and knows nothing about an envelope, so it moves held_open. TaskDone is the same debt with a worker's output attached; both surrender a bare DeferredReply through IntoDeferredReply. Before its terminal, a debt can forward any number of already-encoded replies with reply_envelope, which borrows it: each goes to the waiting caller under the root the hold keeps open, and the debt stays owed. The fleet proxy relays each reply event a remote engine streams back this way. The typed reply stays the terminal that discharges the debt.

Keeping them apart is what lets both be outstanding on one chain at the same time, which is precisely the deferred case: the handler returns and its inbound settles in_flight, while the hold keeps held_open above zero until the answer goes out. One combined handle would mean one combined count, and the window the hold exists to cover would close early — the settle-before-the-reply failure the hold contract above is there to prevent.

Their drop contracts differ for the same reason. Every drained envelope earns exactly one Finished whether or not anyone wanted an answer, so dropping an InboundMail unreplied is ordinary. A hold is taken only when a caller is waiting, so dropping a DeferredReply unreplied strands that caller forever — which is why its drop releases the hold and then panics, in that order and in every build, so the chain still settles and the scheduler escalates the lost reply through the chassis aborter. An actor closing with debts still owed while the engine keeps running answers them: the engine sends each Held<R> its R::unanswered() and then releases its hold, so Sent precedes Release (ADR-0243 §1), and an actor that keeps a DeferredReply in state answers it in unwire. An engine teardown releases held debts with no reply, because every requester is closing with it. abandon_for_actor_close, the one path that releases without a reply or a failure, survives only for the component host's boot waiters until #7008 removes it.

The trace tree

The same lineage keys that drive settlement — each mail's id, its parent, its root — also reconstruct the full causal graph. A trace tree is a set of mail nodes; each names who sent the mail, who received it, the kind, and a series of timestamps sampled as the mail moves through the engine. You walk a chain by following parent edges down from the root, and the timestamps localize the slow hop in a cascade without instrumenting anything. Here is where each one is taken, for a single mail:

●  t_construct_start    the sender's outbound burst opens (its first buffered send)
│     construct         the rest of the sending handler runs, buffering more sends
●  t_sent               flush — at the handler boundary the burst is routed
│     queued            wakeup + scheduling: waiting for a worker to take the burst
●  t_enqueue            a worker picks up the burst (enters its run cycle)
│     drain             earlier mail in the burst is dispatched ahead of this one
●  t_received           this mail's handler is entered
│     handler           the handler runs
●  t_finished           the handler returns

The two spans you reach for most are queue latency — how long the mail waited before a handler ran it — and handler duration (t_finished − t_received). The finer queued / drain split says why a hop waited: scheduling pressure (no worker free yet) versus a long serial fan-out dispatched ahead of it in the same burst — The scheduler maps each span back to the dispatch machinery. A node still missing t_finished is mail that hasn't finished handling yet.

Where it lives, and how to read it. Trace events aren't kept centrally. Each actor holds its own trace ring — the same per-actor storage the log rings use (ADR-0081 / ADR-0086) — recording the events that passed through it. A tree is rebuilt by a guided walk: start at the root's sender, read its ring (aether.trace.tail, the sibling of the aether.log.tail behind actor_logs), follow each onward Sent to the recipient's ring, and stitch. send_mail_traced runs that walk and hands back the stitched tree — and over MCP that's the surface, since there's no standalone per-actor trace tool the way actor_logs exposes the log rings. The tree it returns carries t_construct_start, t_sent, t_received, and t_finished per node; the t_enqueue pickup point and the ready-queue depth live in the rings but aren't surfaced there, so over MCP queue latency reads as one t_sent → t_received span.

Tracing and settlement are deliberately decoupled (ADR-0086) because they have opposite requirements. Settlement is control-plane: exact, on the frame's critical path, never allowed to lag or settle early. Tracing is observability: best-effort and off the critical path, so it can lose data without ever affecting whether or when a chain settles. The line to hold onto is that the settled signal is never wrong while a trace tree can be honestly incomplete — so don't reach for the tree to decide whether work is done; that's settlement's job. Reach for it to see what happened and how long it took.

Gotcha — a trace is a bounded window, not a durable log. A saturating ring reclaims and grows before it drops a live chain. When it fills, it first reclaims the oldest entries whose chains have already settled — those trees are complete, so dropping them costs nothing — and it grows its capacity, doubling toward a configured ceiling, only to make room for chains still in flight. Past the ceiling, or once every old entry belongs to a still-in-flight chain, the oldest entries are overwritten one at a time. A tree whose entries were partly overwritten still comes back, with the nodes that survive: a node whose Finished was dropped reads as in flight, and truncated names each actor whose ring cut the tree and the sequence it was cut before. A root whose own Sent was dropped is an error, not a tree. So what you tend to lose first is an old, already-settled trace you didn't read promptly, not the in-flight work the ring is actively protecting. A tree self-reports where it was truncated, but old or high-volume chains can come back incomplete or not at all. The starting size and the growth ceiling are tunable (AETHER_ACTOR_TRACE_RING_SIZE and AETHER_ACTOR_TRACE_RING_MAX_SIZE) if you routinely lose chains to traffic spikes; even so, read a trace promptly after the work rather than counting on reconstructing something from minutes ago.

How an agent uses it

send_mail already rides settlement on your behalf. Each item blocks until its chain settles. Its default terminal reply projection keeps the last arrival-ordered reply plus any recognized decoded/kind-name error; none keeps only those errors and all returns the complete stream. A request/reply — mail aether.fs.read, get the bytes — is still a single call with no polling, while a multi-reply driver can opt into the exact amount of history it needs. fire_and_forget opts an item out only when completion and ordering are deliberately unobserved. A handler that emits no reply can still settle normally.

send_mail_traced is the tool for when you want the tree, not just the reply. It dispatches a batch under one shared root and, once that whole chain settles, returns the combined trace tree, the correlated replies, and a status:

  • "settled" — the chain closed. By default mails is null, tree holds one indented line per node (sender → recipient, kind, and handler duration), node_count states how many nodes were rendered, and in_flight reads 0. When a trace ring dropped entries this tree needed, truncated lists each such actor and the sequence its ring was cut before; a whole tree carries no truncated. Pass trace: "nodes" to restore the complete mails nodes with parent edges and all timestamps; that mode omits tree and carries the same node_count.
  • "timeout" — the chain didn't settle within settlement_timeout_millis (default 300 000, clamped to 600 000). A timeout is the bound on a hung chain, and the usual cause is exactly the two failures above: a deferred reply that never held its chain open, or a drain that dropped its finish obligation upstream. The triage path from this status to the offending handler is the Debugging a hung settlement recipe.
  • "dispatched" — only with fire_and_forget; the shared root acked, no settlement wait, and both projections' fields are omitted.

Reach for send_mail_traced over send_mail when you need proof a whole cascade finished rather than just one reply, the timing breakdown of a slow exchange, or an all-or-nothing batch where a bad spec aborts before any mail moves. For independent items where each reply is all you want, plain send_mail is simpler. Both surfaces are covered operationally on The MCP harness.

Where to read more