Recovery

Recover from the symptom outward. Preserve evidence, prefer read-only checks, and make the smallest mutation with clear ownership. A timeout, tool error, and dead engine are different states; do not collapse them into “restart everything.”

First response

For any unexpected result:

• Stop sending additional non-idempotent mail.
• Record the exact tool, arguments, selector/hash, and returned error.
• Call list_engines(show: "alive") with a fresh current hub view.
• If the engine is absent, call list_engines(show: "dead") immediately.
• If it is alive, record heartbeat age before further mutation.
• Collect reachable logs, descriptions, costs, traces, and frames.
• Decide who owns the engine/component before terminating or dropping it.

collect_failure_evidence gathers most of that middle step in one bounded, non-mutating call: pass the original error as primary_error, the engine, and the explicit actors, components, kinds, and window that matter. It records the fleet row, kind schemas, component descriptions, log tails, cost tables, and an optional frame, marking each observation ok, error, timeout, or budget_exhausted rather than failing the bundle. Reach for it before terminate, restart, replace, or retry destroys what is still reachable; it cannot reconstruct evidence already evicted from a ring. The per-tool detail is in Inspect and debug.

Symptom-first matrix

Harness or hub interruption

The tunnel is the stable front. Follow The MCP harness to start it and reconnect missing tools. Prefer a tunnel-admin hub restart when rebuilding the hub; an aether-mcp restart invalidates the MCP session. Restart only with explicit authority over the entire fleet, one coordinator, and no blind retry; see Harness lifecycle.

The MCP RPC client re-dials a restarted hub on a dead connection with bounded retry. A tool call crossing the restart can still fail and should be retried only when it is read-only or known idempotent. Reconnection restores transport, not the old fleet.

That retry can repeat the original envelope after an uncertain transport failure; a lost reply is not proof the first mutation never landed. Reconcile observable state before retrying spawn, lifecycle, or application mail.

After a hub restart:

• old engine ids are not continuity evidence, even if a fresh hub later reuses the same UUID text;
• old loaded component names/ids have no live engine behind them;
• the on-disk binary/component store is intended to remain;
• application state must be rebuilt from an application-owned persistence contract, not inferred from registry persistence.

Hub-only restart also preserves aether-mcp caches keyed by reusable engine ids. If a clean kind/component view is essential, deliberately restart and reconnect aether-mcp; otherwise treat cached descriptions as hints and prefer live probes.

Spawn failure

Separate pre-allocation from post-allocation failures.

If selector resolution fails, no engine id or port was allocated. Inspect list_binaries; upload the chassis if necessary. Do not search the live fleet for a child that was never created.

If an error includes an allocated engine_id, correlate it with the spawn_failed recently-dead row. The proxy kills a child it could not bring up, so the record is evidence, not a still-live engine handle.

Boot components load before the substrate binds its RPC port, so a boot component that fails to load is a startup failure, not a half-booted engine: the substrate exits nonzero naming the component and the host's error, and the spawn fails with a spawn_failed entry whose detail carries that stderr. The child is already gone, so there is nothing to terminate.

Check common boot causes in this order:

1. component was uploaded before selector use;
2. selected module has the requested export, or a sole selectable export when none was named;
3. config JSON matches the selected actor's Config schema;
4. derived replica/name lineages do not conflict;
5. the spawn_failed detail's substrate stderr names the failing boot component and its wasm parse, compile, init, or registration error.

Live engine slow or gone

With hub heartbeats enabled, a rising age means pings are not being confirmed promptly. It does not identify the stuck actor. If queries still land, collect actor logs and cost rows from likely recipients. Avoid adding a traced workload merely to diagnose a saturated engine unless the probe is harmless.

When the engine is absent, the death reason determines the next branch:

• terminated: verify that a known owner requested it. Unexpected termination is an ownership/co-ordination failure.
• crashed: preserve the detail and substrate/host stderr. Spawn a replacement only after capturing the failure evidence.
• evicted: the proxy crossed its heartbeat miss limit and dropped/killed the child. Investigate blocking, CPU starvation, or transport failure; the id is already dead.
• spawn_failed: return to selector/startup evidence rather than application mail debugging.

The death sidecar is bounded. Capture it before repeated failures push it out.

Kind, mailbox, or component failure

For an unknown kind, call describe_kinds with the explicit current engine id and the exact name. The encoder already refreshes once on a cache miss or stale schema encode failure. A repeated miss usually means typo, wrong engine, or a component that never loaded—not a cache that needs endless retries.

For an unresolved component recipient, use the exact lineage name returned by load, not the artifact name, actor namespace, selector, or tagged id rendered as a name. describe_component by lineage enables a live lookup on a cache miss; pair a cache hit with a safe live probe when liveness is the question.

For partial replica spawns, already-spawned instances stay live. The error reports the failed index and how many succeeded, so their addresses follow the counter-keyed sequence (NS:0, NS:1, …) up to that point. It does not return the successful prefix's addresses, and registry list_components lists stored artifacts rather than those live instances. The current MCP surface therefore cannot safely target that prefix for drop unless the addresses were obtained elsewhere.

If the engine belongs to the task, preserve evidence and terminate it. If the engine is shared, stop and report the occupied name prefix to its owner. Do not retry the same base or invent mailbox ids.

Replacement failure

Replacement uses a stable mailbox binding on success, but recovery must trust observation rather than intent. The splice is structural, so there is no drain argument to lengthen or retry. Validation and instantiation errors preserve the old guest untouched, still wired; a later state-save, carried-context, rehydrate, or wire failure reinstalls the old guest, still wired, after its on_dehydrate hook ran. The old guest gets back the state its on_dehydrate saved, through its on_rehydrate, with no second wire; only teardown outside that saved state and outside what wire built stays gone. Nothing the failed candidate sent leaves.

One reinstatement ends differently (ADR-0249 §4). An old guest whose on_rehydrate returns an error for the state it saved has refused its own state, so that instance closes: each reply it held is answered unanswered, the mail queued for it is dropped, and its name is spent. Read its log for the error, and load the component again under a new name or key. A guest trap in on_dehydrate reinstates nothing: it aborts the engine, whose exit reason names the component and the hook.

The phase table and stale-introspection limits are in Replacement failure states.

After an error:

• call describe_component by lineage, while treating a cache hit as a capability snapshot rather than conclusive liveness;
• issue a side-effect-free health/query mail if the component provides one;
• inspect its actor logs and, if useful, cost rows;
• do not drop the component until evidence is preserved;
• prefer a known-good exact content hash for a roll-forward attempt.

If the component is no longer serviceable and the engine is shared, republish its module with publish while it is still live (every live instance of the module's namespaces moves with it), or drop that exact instance and load a differently named one after checking downstream address ownership. A drop retires the name, so a reload under it is refused. If the engine is task-owned, replacing the whole engine is often the cleaner isolation boundary.

Settlement timeout

A timeout is an observer deadline, not cancellation. The original engine chain may still mutate state and eventually finish.

Plain send_mail preserves replies that arrived before its timeout. Save them, especially recognized errors. Its later replies are no longer attached to a pending MCP call. Inspect state and actor logs before considering a resend.

send_mail_traced timeout currently returns no root, replies, tree, or node count. It therefore cannot be used to walk that timed-out chain later. If the operation is safe to reproduce, first correct likely settlement obligations, then reproduce with tracing. For a genuinely hung settlement, follow Debugging a hung settlement.

Never “fix” a timeout by switching to fire-and-forget unless the intended contract is truly unobserved dispatch. That hides the symptom; it does not close the causal chain.

Missing logs or visual evidence

Actor rings contain only in-handler tracing::* events and are bounded. If the expected line was emitted during boot, in a proxy, scheduler, panic hook, or other host context, read process stderr. If truncated_before is present, state the evidence gap explicitly.

For capture problems, confirm the selected binary manifest has the needed chassis/caps, then query describe_handlers. Headless engines can run game logic without a desktop frame. A backgrounded/minimized window may also require the window-specific focus flow described in Window.

Use a new capture_frame.save_path beneath a private task-owned evidence directory when full-resolution bytes matter. Absolute syntax alone is not a safety boundary. A failed host write is reported in the saved block without invalidating the successful engine capture, so inspect both results. See Host evidence files.

Final cleanup and handoff

• Preserve all returned spill-file and capture-file paths needed by the incident record.
• Drop task-owned components in a shared engine, require drop_result, and record the retired names.
• Terminate task-owned engines and verify the live fleet no longer lists them.
• Leave adopted/shared engines alone unless the owner authorizes action.
• Report stored artifact hashes separately from live engine/component ids.
• State evidence gaps: overwritten rings, sparse traced timeout, missing host stderr, or ambiguous fleet ownership.

Source routes

• Recovery-visible tool contracts: crates/aether-mcp/src/tools/mod.rs
• Fleet outcomes: crates/aether-mcp/src/tools/engine.rs and crates/aether-fleet/src/server/runtime.rs
• RPC reconnect and timeout behavior: crates/aether-mcp/src/rpc.rs
• Mail timeout projections: crates/aether-mcp/src/tools/mail.rs
• Live kind/component lookup: crates/aether-mcp/src/tools/state.rs and describe.rs
• Component lifecycle: crates/aether-component/src/component/ and crates/aether-component/src/trampoline/runtime/republish.rs

Return to the Operating overview for the normal loop.