Add single-critic bounce arms H and I - #58
Conversation
C runs a three-model panel, so a win over B cannot be attributed: it could be the panel, or it could be any one critic in it. H (GLM) and I (Kimi) isolate one critic each, which makes C's result readable. The C repair prompt is generalized to take N reviews instead of exactly three, and the critique dispatch now walks an ordered roster shared by the manifest and the dry-run plan, so an arm cannot declare one panel and run another. The canary runner gains --task, and routes a cell by its declared tier instead of assuming every condition is agentic -- F and G previously died in the agentic driver when named in --conditions. Adds a one-task POC suite so a proof-of- concept run reports against a denominator of one rather than of five. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a83c900e02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| TASK="" | ||
| MAX_CLAUDE="" | ||
| DRY_RUN=false | ||
| SUITE=$(code_suite_id) |
There was a problem hiding this comment.
When the planned run-canary --task pallets__flask-5014 command is used without CODE_BENCH_SUITE, this still selects the five-task canary, so the newly added one-task POC suite is never used and downstream validation/site rendering retains a five-task denominator. Add a --suite option (and use it in the POC invocation) or otherwise select swebench-verified-poc; merely naming a task does not change the suite.
Useful? React with 👍 / 👎.
| if [[ -n "$TASK" ]]; then | ||
| jq -e --arg id "$TASK" 'any(.instances[]; .instance_id == $id)' "$subset" >/dev/null \ | ||
| || { code_die "task is outside suite $SUITE: $TASK"; exit 1; } | ||
| TASK_LIMIT=1 |
There was a problem hiding this comment.
When --task is used to retry a provider failure with the same run ID, this block only restricts task selection; the loop still calls prepare-swebench-instance.sh, whose existing-cell guard exits immediately. A partial nine-arm run therefore cannot be resumed with the documented same command and instead reruns already completed, billable cells under a new ID; reuse existing inputs and pass --resume to the workflow, or skip completed cells.
AGENTS.md reference: AGENTS.md:L44-L44
Useful? React with 👍 / 👎.
1 participant
Condition C runs Fable's patch past a three-model panel (Codex, GLM, Kimi) before
Fable repairs it. If C beats B, the result is unattributable: the gain could come
from the panel as a whole or from any one critic in it. H and I isolate one
critic each — GLM for H, Kimi for I — so C becomes readable.
Both are tier
agenticbecause Fable implements and repairs with file tools andtest runs. Their critics are single-shot: GLM and Kimi read the candidate patch
out of the prompt and answer once, with no file access. The conditions table and
the README both say so.
Supporting changes
code_write_repair_promptreplaces the inlined three-reviewer prompt builder,so the same code serves a panel of three and a panel of one. Reviews stay
anonymous and numbered.
criticsroster. That one string drivesthe dispatch loop, the repair prompt, the dry-run plan, and
run-manifest.json,so an arm cannot declare one panel and run another.
run-canary.shgains--taskfor naming a single instance, and routes eachcell by its declared tier. Naming F or G in
--conditionspreviously sent asingle-shot condition into the agentic driver, which refuses it — the arms had
to be run by hand. The routing reads the tier, so a future single-shot arm
routes itself.
swebench-verified-pocsuite pins one task (pallets__flask-5014). Aproof-of-concept run against the 5-task canary would render as "1 / 5"; against
this suite the denominator matches what actually ran.
Verification
bash tests/run-all.sh --jobs 4: 42/42 suites passed in 821s.test-code-bench.shgoes from 29 to 37 assertions, covering the H and I phaseplans and rosters, C's roster, the repair prompt's reviewer sections and its
refusal of an empty review list,
--taskvalidation, and that all nine arms onone task fit a ten-dispatch Fable cap.
🤖 Generated with Claude Code