Offline quality eval

The quality eval is the fleet's ground-truth correctness regression alarm. CI gates syntax, types, lints, and the tests a change ships; it cannot see whether a routing, effort, or model change quietly made the agents worse at solving problems. The dominant failure class the fleet produces — semantic-invariant bugs, wrong edge-case behavior, silently dropped requirements — passes every mechanical gate. The quality eval is the standing signal for that class.

It works by re-implementing recently-landed issues blind and scoring the result against what actually merged. The local manual harness selects samples, runs a fresh agent against each one with no access to the merged solution, and judges each blind candidate diff against the landed "ground truth" diff. Its output is a verdict-rate table grouped by the managed Plan's Size and Implementation model values — the axes a routing change moves. It runs off the merge critical path and never gates a PR.

The probe arc this productizes (2026-07-13/14) already found that effort level barely moves correctness below large work, that Opus-medium beats Sonnet-xhigh on large work, and that the failure class is invisible to CI. The eval turns that one-off finding into a repeatable measurement.

What "ground truth" means

The landed squash commit is the ground truth. It merged, so it passed review and CI and represents an accepted solution to the issue. A blind re-implementation is scored against it: not "does it match byte-for-byte" — a different but functionally equivalent implementation is correct — but "does it solve the issue with the same behavior, free of the checklist's semantic-defect classes".

The manual harness

The harness is four local scripts/ files. The select, run, and judge stages stream newline-delimited JSON to the next, so they compose in a pipe and each is independently runnable from a fixed input; the fourth file unit-tests their pure logic.

1. Select — `scripts/quality-eval-select.mjs`

Enumerates merged PRs from the trailing window (QUALITY_EVAL_WINDOW_DAYS, default 7) that closed a code-bearing issue (the squash commit touched a crate source file). For each, it resolves the closing squash commit and its parent SHA — the pre-merge trunk tip the blind run clones at — and captures, at selection time, the closing issue body plus the Size and Implementation model values parsed from its valid managed Plan. It then deterministically samples QUALITY_EVAL_SAMPLE_SIZE (default 5) and emits one record per line:

{"issue":123,"pr":456,"squash_sha":"…","parent_sha":"…","model":"opus","size":"l","issue_body":"…"}

The sample is deterministic (ordered by a stable hash of each candidate's SHA) so a re-run over the same candidate set reproduces the same sample. The body is the runner's task input and the normalized routing values feed the judge's aggregation. They are captured together so a later issue edit cannot change the meaning of a selected sample.

2. Run — `scripts/quality-eval-run.sh`

The blind runner. Per sample it builds a sealed scratch repo and runs an isolated coding agent against the issue body, then harvests the candidate diff beside the landed ground-truth diff, emitting:

{"issue":123,"candidate_diff":"…","landed_diff":"…","model":"opus","size":"l"}

Blindness is structural, not a matter of trusting the agent:

Single-revision clone. git clone --revision=<parent_sha> --depth=1 (git ≥2.49) makes parent_sha the only reachable commit — the landed squash commit is reachable from no ref in the scratch repo.
Sealed remote. The clone's origin is removed immediately, so the agent cannot git fetch trunk even on a public repo.
Contamination assert. Before the agent runs, git rev-list --all must not contain the squash SHA. A hit fails that sample loudly rather than reporting a possibly-peeked verdict — a contaminated scratch would silently inflate the "correct" rate, the exact failure this design guards against.
Edit-only isolation. The agent runs under an --allowedTools allowlist (Read / Edit / Write / Grep / Glob / Bash for build+test — no WebFetch, no MCP, no gh) with no GitHub token in its env. This is a belt on top of the structural seal, not the primary guarantee.

The no-token / no-gh constraint scopes to the coding-agent invocation only — the driver script itself uses git and the network to clone and to read the landed diff from the full-history checkout.

3. Judge — `scripts/quality-eval-judge.mjs`

Drives a judge model over each record, comparing the candidate diff to the landed diff under a fixed known-failure-mode checklist (semantic-invariant, boundary, error-handling, incomplete, wrong-default, regression). It parses a structured per-sample verdict (correct / defect, with a defect class) and aggregates verdict rates grouped by size and model, printing the rollup markdown. The defect rate is computed over scored (correct + defect) samples — an unparseable verdict is counted but never scores as correct.

Only the pure logic is unit-tested (scripts/quality-eval.test.mjs): the selector's trailing-window filter and deterministic sampler, and the judge's verdict parser and rate aggregation (with a tripwire on the aggregated rates). The git isolation and the claude invocations are I/O, verified by the contamination assert at runtime rather than by a test.

Running it manually

Quality evaluation is a local manual harness, not a hosted merge gate. Run it with git ≥2.49, node, jq, and a CLAUDE_CODE_OAUTH_TOKEN:

GITHUB_TOKEN=$(gh auth token) node scripts/quality-eval-select.mjs > samples.jsonl
QUALITY_EVAL_CLONE_TOKEN=$(gh auth token) bash scripts/quality-eval-run.sh samples.jsonl > judged.jsonl
node scripts/quality-eval-judge.mjs judged.jsonl        # prints the rollup

Reading the verdict rates

The rollup reports, per managed-Plan size × model cell and overall, how many blind re-implementations were judged, how many were correct, how many carried a defect, and the defect rate over the scored samples, followed by a per-defect list naming the issue and its defect class. A defect rate that climbs after a routing / effort / model change — especially on large work, where the probe arc found the model choice matters most — is the signal that the change degraded correctness in a way no other check would surface. A single five-sample run is noisy; repeat runs using the same sampling method over time before treating movement as a trend.

Aether Guide