docs(rfc): RFC-006 — implementation phases for the stack executor - #37
Open
explosivebit wants to merge 1 commit into
Open
docs(rfc): RFC-006 — implementation phases for the stack executor#37explosivebit wants to merge 1 commit into
explosivebit wants to merge 1 commit into
Conversation
…cutor Adds Implementation Phases (5, ordered + shippable), Invariants, Rollback Plan, and a recon-status section to the executor RFC, making it the actionable design for the next focused build session. Records what already exists (Docker up, Half B evaluators + judges, proven CLI recipes) vs the single missing piece (Half A = run harness → patch), and the deliberate task-choice caveat for the first scored slice (be_01 evaluators score-invert per EVID-027). Draft (not activated). Validate PASS (0 MUST errors). Refs: rfc-stack-executor-run-agent-cli-harnesses-in-the-sandbox-candidate-side-patch
explosivebit
added a commit
that referenced
this pull request
Jun 2, 2026
…g_effort=low (#41) * fix(judge): gpt-5-mini reasoning eats the token budget → reasoning_effort=low (proxy-side) Root cause (diagnosed live 2026-06-02): gpt-5-mini is a REASONING model that spent the ENTIRE 2048 max_tokens budget on reasoning_tokens (finish_reason= "length", content len 0) → empty/truncated rubric JSON → _parse_rubric_scores fell back to {'overall': 0.0} → a FALSE 0.0 dragged the panel median + the Krippendorff alpha (went negative). NOT malformed JSON, and NOT fixable by just raising the cap (reasoning would eat that too). Fix: `reasoning_effort: low` on the gpt-5-mini-judge alias in litellm-config.yaml. Bounds reasoning (~640 tok) so the full rubric JSON fits AND the call is cheaper ($0.0019 vs $0.0043). This is the EVID-023 remedy ("cap reasoning, don't raise the cap" — OpenRouter pre-reserves max_tokens×price → HTTP-402 hazard). It lives PROXY-side because inspect_ai 0.3.46 GenerateConfig has no reasoning_effort / extra_body passthrough. Scoped to the JUDGE alias; candidate gpt-5 is unaffected. Validated: the panel's request path (no reasoning_effort in the request) now returns finish_reason="stop" + valid parseable JSON {correctness:9,security:10,…}. - infra/litellm-config.yaml: reasoning_effort: low - scripts/diagnose_judge_json.py: reusable judge-JSON debug tool (finish_reason + raw output + parse + json_mode + reasoning_effort probes) - judge_panel.py: forward-pointer comment to the proxy-side fix Follow-up (separate, noted): a parse-fail should map to DEGRADED (non-vote), not a false 0.0 — defense-in-depth for any future judge garbage. Refs: rfc-006 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(forgeplan): EVID-050 — gpt-5-mini reasoning_effort fix evidence (PASS), informs RFC-006 Diagnosis (reasoning_tokens exhaust max_tokens → empty content → false 0.0) + proxy-side fix (reasoning_effort=low) + live validation. verdict=PASS, CL3, evidence_type=incident_diagnosis. Activated. NOTE: RFC-006.md projection touched by the auto-link; reconcile with PR #37 + the EVID-049 link (scoring branch) at merge — all three just union RFC-006's links array. Refs: evid-gpt-5-mini-judge-reasoning-tokens-exhaust-the-max-tokens-budget-fixed-via Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes RFC-006 the actionable design for the next focused build (the stack executor — run real agent-CLI harnesses in the sandbox → patch → real harness×model numbers).
Adds: Implementation Phases (5, ordered + independently shippable), Invariants, Rollback Plan, and a recon-status section recording what already exists (Docker up, Half B evaluators + judges operational, proven CLI proxy recipes, leaderboard data layer) vs the single missing piece (Half A = run harness → patch). Flags the deliberate task-choice caveat for the first scored slice (be_01 evaluators score-invert, EVID-027).
Draft (not activated). forgeplan-validate PASS (0 MUST errors).
Refs: rfc-006
🤖 Generated with Claude Code