Document that terminal control omits eval_refs (advisory-eval contract) - #83
Merged
Conversation
When the stock v3 contract retired sealed eval receipts (0.9.0,
check_runner_roles: [outer], advisory eval_results.md), the outer and
planner control examples stopped emitting eval_refs but never told the
orchestrator why. Agents kept adding it back, citing eval_results.md or
report paths — and the engine rejected the control ("eval receipt must
be a current-session eval_receipts JSON file") because eval_refs only
accepts eval_receipts/*.json, which this contract never produces.
Observed live: one rejected terminal-control cycle per program in both
the UGM and ultimate-memory-cloud runs. The prompts now state the field
is omitted and eval verdicts are cited via evidence_refs ->
eval_results.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the stock v3 contract retired sealed eval receipts (0.9.0 / #81,
check_runner_roles: [outer], advisoryeval_results.md), the outer and planner control examples stopped emittingeval_refs— but never told the orchestrator why. Agents kept re-adding the field, citingeval_results.mdor a report path, and the engine rejected the whole terminal control:because
eval_refsonly acceptseval_receipts/*.json, which this contract never produces.Observed live: one rejected terminal-control cycle per program in both the UGM and ultimate-memory-cloud runs — each cost a wasted iteration and an operator nudge before the orchestrator self-repaired.
This adds a note under the control JSON in the outer and planner prompts: omit
eval_refs; cite eval verdicts throughevidence_refs→session:/project_state/eval_results.md.Companion to the two earlier live-run hardening fixes on the same prompts (the LayerHandoff schema in #82 and the logical-reference grammar).
Test plan
test_template_contracts.py+test_cli.py: 30 passed. Full suite unaffected (prompt-text-only change).🤖 Generated with Claude Code
https://claude.ai/code/session_01EPYwF6cRYs2mfsfHF1hibc