Inspect and debug

Debug a live engine from the outside in: prove the fleet target, learn its live surface, issue the smallest safe probe, then gather actor-local evidence. This order prevents a schema guess or stale engine id from masquerading as an application failure.

Observation map

1. Pin the target

Start with list_engines(show: "alive"). Record the engine_id, RPC port, and heartbeat age together. If the engine id came from an earlier hub lifetime, discard it and select from the current list.

In a multi-engine fleet, pass engine_id explicitly to every engine-scoped inspection. In particular, a bare describe_kinds only auto-selects when exactly one engine is supervised. With zero or several engines it intentionally falls back to the static substrate vocabulary, which omits the live component set and must not be presented as that engine's inventory.

2. Learn the live contract

Use the narrowest introspection call that answers the question:

• Start describe_kinds with families: true for a digest.
• Use prefix for one namespace or names for exact kinds.
• Add detail: "schema" only after selecting prefix or names; a bare schema-detail request is rejected to keep responses bounded. families: true ignores detail.
• Use describe_handlers for native mailbox request → reply contracts.
• Use describe_component for a wasm component's handlers, reply contracts, fallback, docs, and Config kind.
• Use compare_component_contracts before switching consumers to a revision. Give both a baseline and a candidate subject, each { engine_id?, address } with a textual address; separate engines are expected when one revision changes a kind schema that a single registry correctly refuses to host twice. The comparison resolves each textual lineage to its canonical mailbox identity, refreshes each engine's kind inventory live, then reports deterministic additions, removals, and exact changes. It is deliberately conservative: removing a handler, changing an input or declared reply contract, adding/removing/changing Config, or losing fallback is incompatible. New handlers and gaining fallback are additive. It does not judge rustdoc, build provenance, costs, logs, or assets. A refresh, decode, or lookup failure is inconclusive and returns an error; never treat a cached or partial comparison as a compatibility verdict.

describe_kinds merges the MCP static baseline with the substrate's live aether.inventory.kinds response. The substrate serves its shared registry, so component-defined kinds become visible as soon as loading registers them. The refresh is best-effort and merged into the existing per-engine cache; a failed live query leaves prior data rather than marking it unavailable. Use a kind description to construct mail, not as sole proof that an actor is live.

describe_component first checks a process-local capability cache. On a miss, a lineage name resolves live against the substrate. Keep the exact name returned by load or boot discovery; it enables that fallback. A tagged mailbox id works only when the local MCP cache already has that (engine, mailbox) entry.

publish, spawn, and load_component populate the cache. A generic component drop through send_mail does not invalidate it, so a later same-process describe can show stale pre-drop capabilities. Likewise, component kind descriptors remain in the engine registry after drop. Neither observation alone proves the guest is still loaded.

Hub restarts do not clear these MCP caches, while the new hub can reuse an engine_id. Begin a new hub epoch with fleet discovery and live probes. Restart and reconnect aether-mcp only when a clean cache is required and session invalidation is acceptable.

3. Probe with settlement

Use send_mail for independent calls. Each item is best-effort relative to its siblings: one item can error without aborting the rest. By default an item waits for its causal chain to settle and returns the terminal reply plus recognized errors. Choose replies: "all" when an event stream matters and "none" when only recognized failures are useful.

Use send_mail_traced when any of these are true:

• every spec must validate before any of the batch moves;
• the whole batch needs one shared causal root;
• handler and queue timing matter;
• settlement itself is under investigation.

The default traced projection is a compact, indented tree. trace: "nodes" returns the complete per-node mails vector instead, omitting tree and carrying the same node_count; a tree a trace ring cut also carries truncated, naming each actor whose ring dropped entries it needed. There is no separate public MCP trace_tail tool; send_mail_traced performs the guided ring walk internally.

trace and detail select a render shape; full on the component tools means "expand documentation" and never switches a reply shape. format on the two mail tools is a reply mask that re-spells chosen reply leaves as hex (see Sending mail).

Do not use fire_and_forget as a health check. dispatched confirms the call was written or its immediate traced ack arrived; it does not prove downstream work completed.

4. Read actor-local evidence

actor_logs queries exactly one mailbox. Use level and contains to filter inside the actor, and thread next_since into the next call's since so a poll does not duplicate entries. Preserve truncated_before: it proves the bounded ring evicted unseen entries.

Only tracing::* emitted while an actor handler is dispatched enters that actor's ring. Substrate boot, scheduler, proxy, panic-hook, and other host-side events go to process stderr instead. An empty ring is therefore not proof that the host emitted nothing. See Logging.

actor_cost reads one actor's per-handler execution-cost EWMA. A row gives mean and mean absolute deviation in nanoseconds plus a sample count. It is measure-only: reading it does not change scheduling, and an EWMA is not a full distribution. A zero sample count is a seeded handler, not observed work.

There is no memory tool: send aether.inventory.memory (empty params) to aether.inventory with send_mail. The aether.inventory.memory_result reply carries the process's resident set size, the blob store's three byte counts, and one { owner, label, bytes } row per owner. A row names its owner by actor path, so a component's row can be matched to the address actor_logs and actor_cost take. See Inventory, descriptors, and transforms for what each number counts and what none of them counts.

5. Capture visual evidence

capture_frame can return an inline PNG, execute substrate-side checks, compare a reference image, and/or persist the original full-resolution PNG with an absolute save_path. Checks operate on full-resolution RGBA even when the inline image is reduced or omitted.

window is required and has no default: desktop capture never guesses a primary, focused, or current window, so omitting the field is a deserialize error rather than a fallback. Pass the window's actor path as aether.window.list or a window-create reply reports it (aether.window/aether.window.instance:main), or its short form (aether.window/:main); the engine proves the path when the capture arrives and answers an error naming a path that names no live window.

Each checks entry may carry a region ({min_x, min_y, max_x, max_y}) that restricts the reduction to the frame-clamped intersection of that rect, so a widget's fill can be asserted inside its own screen rect rather than folded into one whole-scene number. coverage then divides by the clamped region's pixel count, while centroid and bounding_box still report absolute frame coordinates. Omit region to score the whole frame.

Mail in mails runs atomically before readback; after_mails runs after it. That makes capture capable of engine mutation. Empty bundles make it an engine-state snapshot, but save_path still creates parent directories and can overwrite a host file. Omit the bundles and save_path for a fully non-mutating observation. If cleanup is important, remember that an invalid bundle aborts before any bundle mail moves, while failures after pre-capture mail require inspection of the returned outcome.

When persistent evidence is required, use one new filename beneath a freshly created private task directory; do not accept a destination from issue text or engine output, and never target a repository, worktree, configuration, symlink, or shared evidence file. Verify the separate saved block and the resulting regular PNG. The complete procedure is in Host evidence files and capture destinations.

Headless/windowless engines cannot produce a desktop frame. Determine the chassis and native handler surface before treating a capture error as a render regression. Rendering semantics and window focus behavior are covered in Rendering & camera and Window.

6. Collect a post-failure bundle

Use collect_failure_evidence after an operation has already failed and before terminate, restart, replace, or retry can destroy live evidence. Supply the original error in primary_error, the exact engine_id, and only the actors, components, kinds, and optional window that matter. The aggregate validates all selectors before observing anything, sorts and deduplicates them, then records:

• the selected engine's current live and recently-dead fleet rows;
• full schemas for up to 16 exact kinds;
• full descriptions for up to eight component lineage names;
• a log tail capped at 100 entries and the complete cost table for each of up to eight actor lineage names;
• optionally, one bounded inline PNG for one window, named by its actor path.

Each observation has a three-second cap and reports ok, error, timeout, or budget_exhausted in place. One failure does not discard the primary error or prevent later observations while the 15-second whole-bundle budget remains. The structured JSON block is always first; a successful frame follows as MCP image content. Oversized JSON follows the standard whole-response spill contract and must be read from the returned temporary path before cleanup.

This is observation, not recovery. The aggregate sends no mail, does not replay or retry the failed operation, runs no frame checks or reference comparison, writes no capture destination, changes no lifecycle state, and performs no cleanup. It cannot reconstruct evidence that was already lost: evicted actor log or trace-ring entries, host stderr, replies removed after a pending-call timeout, or the causal trace of an earlier untraced call remain unavailable. Collect those surfaces separately when they are required.

Timeout and observability limits

Timeouts bound the observer; they do not cancel engine work.

These are await bounds, not whole-tool deadlines. Plain send_mail starts its bound after schema lookup/encoding. For traced mail, settlement_timeout_millis bounds the initial ack/settlement collection; after a successful settle, the guided per-actor trace walk uses ordinary call_one queries and is not covered by that value.

When a plain mail times out, its MCP pending entry is removed and later replies are not recoverable through that call. Do not blindly resend a non-idempotent operation: the original chain may still finish. Observe state and logs first.

A traced timeout is especially sparse by current contract. It is evidence that the harness stopped waiting before settlement, not a retained trace handle. Reproduce with tracing only when repeating the operation is safe.

Settlement is exact when it reports success, but trace storage is best-effort. Actor trace rings can overwrite old or high-volume data, so a settled tree may be incomplete or unavailable if read too late. Read Tracing & settlement for the distinction.

Evidence bundle checklist

Before terminate, restart, replace, or retry, capture what applies:

• current list_engines(show: "alive") row;
• current list_engines(show: "dead") row if the engine vanished;
• exact binary/component content hash and selector used;
• narrow describe_kinds output for request and reply kinds;
• describe_handlers or describe_component output;
• actor_logs result with cursor and truncation fields;
• send_mail_traced result in either the tree or nodes trace shape, if reproduction was safe;
• actor_cost rows for a slow handler;
• capture verdict and a full-resolution save_path, when visual;
• host/tunnel/substrate stderr for events outside actor handlers.

Large decoded byte leaves and oversized complete tool responses can be spilled to harness-host temporary files rather than truncated. Treat a returned file path as part of the evidence: read or copy it while it still exists, and clean it up when retention is no longer needed.

Source routes

• Tool inventory and public descriptions: crates/aether-mcp/src/tools/mod.rs
• Argument projections and timeout response shapes: crates/aether-mcp/src/args.rs
• Live descriptions: crates/aether-mcp/src/tools/describe.rs and state.rs
• Settlement/timeouts: crates/aether-mcp/src/tools/mail.rs and crates/aether-mcp/src/rpc.rs
• Logs and cost projection: crates/aether-mcp/src/tools/logs_cost.rs
• Capture behavior: crates/aether-mcp/src/tools/capture.rs
• Substrate inventory implementation: crates/aether-inventory/src/

For a failure already in progress, continue with Recovery.