feat(eval): Phase 4 scaffolding — self-healing agent-quality guard

[ProtocolWarden]

What

Stands up the Phase 4 (EVAL) machinery from HARNESS_TRUST_HARDENING §4 — everything that can be built ahead of the one irreducibly-human step (the operator answer-key signature). The fleet can now grade its own reviewer/worker quality, while being structurally unable to grade its own answer key.

The design in one line

The fleet grades itself, but cannot grade its own answer key. Everything self-heals with no human in the per-correction loop except an operator signature on each ground-truth label — encoded once, offline.

Modules (src/operations_center/eval/)

Module	Role
corpus.py	Append-only, hash-chained case ledger. Editing/deleting any past entry breaks the chain → tamper-evident.
signing.py	Ed25519 operator signatures. A case is graded only when its signature verifies against the constitution public key. Asymmetric so no compute inside the trust boundary can forge a label; the private key stays offline.
replay.py	Deterministic blocking gate — replays input.checks through the pure code-computed verdict (pr_review_watcher.verdict.compute_verdict) and exact-matches the committed answer. No model → zero flakiness → safe to block. Catches a #313-style verdict-bypass regression.
critic.py	Non-blocking different-model-family N-of-M drift monitor (the model extractor is an injected seam).
constitution.py	Monotonic baseline floor + report-only→blocking graduation (D-EVAL-3). Gate can never block before the key is seeded (§0.1 degrade-never-halt).
verify.py	The required CI check (.github/workflows/eval-corpus-integrity.yml) tying chain + signatures + floor together.

The exam / answer-key split

• The fleet may append unsigned candidate cases — scored and reported, never gating.
• Only an operator-signed case counts toward the gate. The signature is made offline; no in-boundary compute can mint one.

What's seeded

7 unsigned candidate cases (#313 verdict-bypass + injection classes, #337 over-flag, plus legitimate concerns). All pass replay; the gate is correctly report-only (0/15 signed). CODEOWNERS pins eval/corpus, eval/constitution, and the workflow to the operator (D-EVAL-2).

Verification

$ python -m operations_center.eval.verify
chain OK: 7 entries, head aee9da2f1594…
operator key: NOT YET ANCHORED — all cases are candidates
cases: 7 total, 0 graded, 7 candidate
gate [report-only]: 0/15 signed cases — gate is report-only until the answer key is seeded
RESULT: PASS

33 unit tests (hash-chain tamper-evidence, signature non-transfer to a swapped input, replay gating, graduation modes, drift majority-vote, end-to-end verify incl. signed blocking + tamper detection). ruff/ty clean; Custodian T7 cleared.

Deferred — the one human step

The operator generates the Ed25519 key offline, commits the public key to eval/constitution/operator_pubkey.ed25519 (currently a placeholder), and signs ≥15 seed cases. At that point the gate graduates from report-only to blocking. Nothing else in EVAL needs a human.

🤖 Generated with Claude Code