Deterministic handoff currency, located+fresh eval output, orchestrator reconciliation (D13–D15) - #88
Merged
Conversation
…or reconciliation (D13–D15) Fixes the inner_outer_eval non-convergence post-mortem: an orchestrator that stalls after the work is objectively done because it is misled by its own stale durable state (stale-but-schema-valid handoff, an eval verdict written off-channel/unversioned, and no path to reconcile against live state). Engine (opt-in via a contract's typed, validated currency_outputs; backward compatibility waived — a contract without the declaration is inert): - D13/A1 (correctness, hard): _validate_v3_control rejects terminal goal_met when the accepted handoff snapshot was not re-stamped by the completing attempt, independent of whether handoff_ref is cited (handoff_ref is optional, so a citation-only check is bypassable). Rejection reuses the proven _reject_v2_control repair path. A kind: handoff contract additionally requires the citation. - D13/A2 (hygiene, soft): a successful handoff-owner finish that did not re-stamp emits a pure handoff_stale diagnostic + a protocol_failures repair note — no success flip, no cap, no counter, no overwrite. Gated on request.success so a crashed attempt never fails against the prior valid handoff. - D13 same-revision re-stamp: relaxed by an exact field-level compare (_is_provenance_only_restamp) — producer is the current owner attempt, updated_at non-decreasing, every other field equal to the accepted snapshot — so a provenance-only re-stamp is legitimate while tamper detection is preserved. - D14 (advisory eval): the engine never reads eval output; it emits at most an eval_missing presence diagnostic when a declared advisory owner finishes and the path is absent. Never fails/respawns/caps (D11). check_runner_roles unchanged. - D15: exposes the session raw root in paths.json to the orchestration role only. Contract: CurrencyOutput model + validated currency_outputs on WorkflowSetContract (repo-relative confined unique paths; kind: handoff locked to the canonical handoff identity and the declared handoff_owner). Stock inner_outer_eval: declares currency_outputs; outer prompt gains reconcile-against-live-state + anti-over-polish + per-attempt re-stamp guidance and drops the mandatory-final-eval sentence (D11); eval_runner writes eval_results.md via eval-banana --result-out with observed provenance instead of hand-transcribing; inner checks eval provenance before relying on a verdict. Design corpus: analysis post-mortem, the binding design, D13–D15, and the archived two-round Codex+Grok reviews. eval-banana's --result-out capability is a separate PR. 16 new contract/behavior tests; full suite green (471 passed), pyright + ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An
inner_outer_evallayer can stall after the work is objectively done because the orchestrator is misled by its own durable state: a schema-valid but stale handoff, an advisory eval verdict hand-copied off-channel with no freshness marker, and no instruction to reconcile against live repo/CI state. Post-mortem:design/analysis/inner-outer-eval-nonconvergence-postmortem.md.The fix — three substrate-determinism changes (opt-in, no new engine gate)
Activated only when a contract declares a typed, validated
currency_outputslist; a contract without it is fully inert. Backward compatibility is waived per the operator._validate_v3_controlrejects a terminalgoal_metwhen the accepted handoff snapshot was not re-stamped by the completing attempt — independent of whetherhandoff_refis cited (it is optional, so a citation-only check is bypassable). Rejection reuses the existing_reject_v2_controlrepair path.handoff_stalediagnostic + aprotocol_failuresrepair note — no success flip, no cap, no counter, no overwrite. Gated onrequest.successso a crashed attempt never fails against the prior handoff._is_provenance_only_restamp) — producer is the current owner attempt,updated_atnon-decreasing, every other field equal to the accepted snapshot — so a provenance-only re-stamp is legitimate while tamper detection is preserved.eval_missingpresence diagnostic. eval-banana (>=0.5.0,--result-out) writes the provenance-stampedeval_results.mdby construction — no more off-channel hand-copy.check_runner_rolesunchanged.paths.jsonto the orchestration role only; the stockouterprompt gains reconcile-against-live-state + anti-over-polish + per-attempt re-stamp guidance, and drops the mandatory-final-eval sentence (D11).innerchecks eval provenance before relying on a verdict.Contract & stock set
CurrencyOutputmodel + validatedcurrency_outputsonWorkflowSetContract(repo-relative confined unique paths;kind: handofflocked to the canonical handoff identity + declaredhandoff_owner). Stockinner_outer_evaldeclares it; the eval runner writeseval_results.mdviaeval-banana --result-outinstead of hand-transcribing.Design & validation
Binding design:
design/designs/deterministic-handoff-and-eval-provenance.md; decisions D13–D15; two-round Codex+Grok design reviews and the implementation review archived underdesign/analysis/. Full suite green (471+ passed), pyright0 errors, ruff clean. 21 new contract/behavior tests cover A1 cited/uncited rejection, no-snapshot rejection, re-stamp acceptance, opt-in inertness, A2 purity + cap isolation, non-owner non-diagnosis, crash gating, same-rev vs tamper (incl. timestamp regression),eval_missing,raw_rootexposure, and contract validation.Depends on eval-banana 0.5.0 (
--result-out), now published;uv.lockpins it.🤖 Generated with Claude Code
https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc