Skip to content

Deterministic handoff currency, located+fresh eval output, orchestrator reconciliation (D13–D15) - #88

Merged
fazpu merged 1 commit into
mainfrom
feat/handoff-currency-eval-provenance
Jul 23, 2026
Merged

Deterministic handoff currency, located+fresh eval output, orchestrator reconciliation (D13–D15)#88
fazpu merged 1 commit into
mainfrom
feat/handoff-currency-eval-provenance

Conversation

@fazpu

@fazpu fazpu commented Jul 23, 2026

Copy link
Copy Markdown
Member

Problem

An inner_outer_eval layer can stall after the work is objectively done because the orchestrator is misled by its own durable state: a schema-valid but stale handoff, an advisory eval verdict hand-copied off-channel with no freshness marker, and no instruction to reconcile against live repo/CI state. Post-mortem: design/analysis/inner-outer-eval-nonconvergence-postmortem.md.

The fix — three substrate-determinism changes (opt-in, no new engine gate)

Activated only when a contract declares a typed, validated currency_outputs list; a contract without it is fully inert. Backward compatibility is waived per the operator.

  • D13 / A1 (correctness, hard): _validate_v3_control rejects a terminal goal_met when the accepted handoff snapshot was not re-stamped by the completing attempt — independent of whether handoff_ref is cited (it is optional, so a citation-only check is bypassable). Rejection reuses the existing _reject_v2_control repair path.
  • D13 / A2 (hygiene, soft): a successful handoff-owner finish that did not re-stamp emits a pure handoff_stale diagnostic + a protocol_failures repair note — no success flip, no cap, no counter, no overwrite. Gated on request.success so a crashed attempt never fails against the prior handoff.
  • D13 same-revision re-stamp: relaxed by an exact field-level compare (_is_provenance_only_restamp) — producer is the current owner attempt, updated_at non-decreasing, every other field equal to the accepted snapshot — so a provenance-only re-stamp is legitimate while tamper detection is preserved.
  • D14 (advisory eval, D11-safe): the engine never reads the eval file; it emits at most an eval_missing presence diagnostic. eval-banana (>=0.5.0, --result-out) writes the provenance-stamped eval_results.md by construction — no more off-channel hand-copy. check_runner_roles unchanged.
  • D15 (load-bearing): exposes the session raw root in paths.json to the orchestration role only; the stock outer prompt gains reconcile-against-live-state + anti-over-polish + per-attempt re-stamp guidance, and drops the mandatory-final-eval sentence (D11). inner checks eval provenance before relying on a verdict.

Contract & stock set

CurrencyOutput model + validated currency_outputs on WorkflowSetContract (repo-relative confined unique paths; kind: handoff locked to the canonical handoff identity + declared handoff_owner). Stock inner_outer_eval declares it; the eval runner writes eval_results.md via eval-banana --result-out instead of hand-transcribing.

Design & validation

Binding design: design/designs/deterministic-handoff-and-eval-provenance.md; decisions D13–D15; two-round Codex+Grok design reviews and the implementation review archived under design/analysis/. Full suite green (471+ passed), pyright 0 errors, ruff clean. 21 new contract/behavior tests cover A1 cited/uncited rejection, no-snapshot rejection, re-stamp acceptance, opt-in inertness, A2 purity + cap isolation, non-owner non-diagnosis, crash gating, same-rev vs tamper (incl. timestamp regression), eval_missing, raw_root exposure, and contract validation.

Depends on eval-banana 0.5.0 (--result-out), now published; uv.lock pins it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc

…or reconciliation (D13–D15)

Fixes the inner_outer_eval non-convergence post-mortem: an orchestrator that
stalls after the work is objectively done because it is misled by its own stale
durable state (stale-but-schema-valid handoff, an eval verdict written
off-channel/unversioned, and no path to reconcile against live state).

Engine (opt-in via a contract's typed, validated currency_outputs; backward
compatibility waived — a contract without the declaration is inert):

- D13/A1 (correctness, hard): _validate_v3_control rejects terminal goal_met when
  the accepted handoff snapshot was not re-stamped by the completing attempt,
  independent of whether handoff_ref is cited (handoff_ref is optional, so a
  citation-only check is bypassable). Rejection reuses the proven _reject_v2_control
  repair path. A kind: handoff contract additionally requires the citation.
- D13/A2 (hygiene, soft): a successful handoff-owner finish that did not re-stamp
  emits a pure handoff_stale diagnostic + a protocol_failures repair note — no
  success flip, no cap, no counter, no overwrite. Gated on request.success so a
  crashed attempt never fails against the prior valid handoff.
- D13 same-revision re-stamp: relaxed by an exact field-level compare
  (_is_provenance_only_restamp) — producer is the current owner attempt,
  updated_at non-decreasing, every other field equal to the accepted snapshot —
  so a provenance-only re-stamp is legitimate while tamper detection is preserved.
- D14 (advisory eval): the engine never reads eval output; it emits at most an
  eval_missing presence diagnostic when a declared advisory owner finishes and the
  path is absent. Never fails/respawns/caps (D11). check_runner_roles unchanged.
- D15: exposes the session raw root in paths.json to the orchestration role only.

Contract: CurrencyOutput model + validated currency_outputs on WorkflowSetContract
(repo-relative confined unique paths; kind: handoff locked to the canonical handoff
identity and the declared handoff_owner).

Stock inner_outer_eval: declares currency_outputs; outer prompt gains
reconcile-against-live-state + anti-over-polish + per-attempt re-stamp guidance and
drops the mandatory-final-eval sentence (D11); eval_runner writes eval_results.md via
eval-banana --result-out with observed provenance instead of hand-transcribing;
inner checks eval provenance before relying on a verdict.

Design corpus: analysis post-mortem, the binding design, D13–D15, and the archived
two-round Codex+Grok reviews. eval-banana's --result-out capability is a separate PR.

16 new contract/behavior tests; full suite green (471 passed), pyright + ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
@fazpu
fazpu merged commit 8c94901 into main Jul 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant