Land the SWE-bench code benchmark battery on master - #57
Conversation
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Both providers bill reasoning tokens against max_tokens, so the 800-token
critic cap spent the whole budget on reasoning and returned an empty content
string. Every condition-C cell failed its artifact gate before the final
repair. Z.AI rejects disabled thinking and takes reasoning_effort=low; Moonshot
takes thinking:{type:disabled} at temperature 0.6. Both defaults stay unset so
the document seats are unchanged. Critics now retry with a backoff, which
Moonshot's org-concurrency-1 limiter needs.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Kimi missing-key test unset KIMI_API_KEY, but the bouncer reloaded it from the repo's .env.local, so the scenario could never fail closed on a machine that has a real key on disk. CO_EVOLVE_ENV_FILE now selects the file, defaulting to .env.local, which lets the test point at a path that does not exist. The two near-identical loaders collapse into one, and its trailing guard no longer aborts the bouncer without a message when a key line has an empty value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The code battery and the document batch each surfaced harness defects, and neither surface alone answers the pipeline question. One executable plan records every issue's disposition so the executing session starts from the ledger instead of rediscovering fixed problems. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Results on a corpus nobody outside this repo has seen are not comparable and not shareable. The bounce-protocol document benchmark is retired from measurement and all shared reporting (b1 stays archived, internal-only); SWE-bench Verified on the official pinned evaluator is the measurement surface, and future additions must be standardized public suites approved at the Phase 4 gate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Results on a corpus and judge panel that exist only in this repository cannot be compared against anything anyone else has run, so they are not worth publishing. Measurement and every shared surface now use standardized public benchmarks scored by their official evaluators. Batch b1 stays on disk as internal evidence. Nothing is deleted; the runbook remains for reading archived batches. Outstanding doc-suite work (re-judging, sanitizer fix, judge calibration) is cancelled, not deferred. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex 0.144.5 on Windows accepts --sandbox workspace-write and then runs read-only, so condition B's repair step reviewed every patch and changed nothing: its 5/5 was Fable's first draft with a discarded review attached. Gate the mode behind CODE_BENCH_CODEX_SANDBOX and record what each cell actually used, because the mode decides what the treatment is. Add the arms that make the pipeline rows interpretable: codex-solo, and a single-shot tier for GLM and Kimi, which have no agent loop here. Tier is a declared field so a one-prompt answer is never ranked beside a coding agent unlabelled. The single-shot gate applies with --recount: chat models miscount @@ headers while proposing sound edits, and the agentic arms never write a hunk header at all. One status file, one writer, every line tagged - two orchestrators once interleaved into a timeline that described neither run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hand-pinned 5-task canary is one instance per repository, so its score cannot be extrapolated to the full split. Add a uniform random 50-task draw (seed recorded in the subset file) and make the suite selectable via CODE_BENCH_SUITE so existing canary invocations keep working. Also scrub cloned workspace history down to a single synthetic base commit: a full clone carries the upstream fix for the very issue under test, and observed runs read the answer out of `git log` instead of deriving it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05d52f990a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| esac | ||
|
|
||
| patch="$cell/final.patch" | ||
| git -C "$workspace" diff --binary > "$patch" |
There was a problem hiding this comment.
When an agent stages an edit or creates a new source file, this worktree-only git diff omits the staged or untracked content, so the submitted prediction is incomplete or is rejected as empty even though the workspace contains a valid solution. Snapshot the complete delta from HEAD, including new files, before constructing model_patch.
Useful? React with 👍 / 👎.
| repos = {row['instance_id']: row['repo'] for row in subset['instances']} | ||
| lock = read_json(os.path.join(code_dir, 'external-sources.lock.json')) | ||
|
|
||
| latest, superseded = newest_reports(eval_dir) |
There was a problem hiding this comment.
When both declared suites have results under the same evaluation directory, report selection considers only condition and run ID, not whether the report belongs to args.suite. A later random-50 condition-B report can therefore be used while rendering the five-task canary, producing mismatched task cells and even a resolved count greater than the displayed denominator; select reports whose evaluated instances match the requested subset or namespace reports by suite.
Useful? React with 👍 / 👎.
| resolved_ok = all(status_on(rows, c, SEPARATOR_TASK) == 'resolved' for c in UNICODE_DIGITS) | ||
| failed_ok = all(status_on(rows, c, SEPARATOR_TASK) == 'unresolved' for c in ASCII_DIGITS) |
There was a problem hiding this comment.
On a later canary run that happens to reproduce this pass/fail pattern, this guard renders assertions about exact regex edits, critique reasoning, Windows sandbox behavior, and superseded runs even though it checks only task statuses and none of those facts are present in the input JSON. The published page can consequently describe changes or execution history that never occurred; carry the relevant patch and manifest evidence into the aggregate and verify it before rendering this narrative.
Useful? React with 👍 / 👎.
| '<div class="runmeta"><span>fable @ medium</span><span>gpt-5.6-sol @ medium</span>' | ||
| '<span>glm-5.3-flash @ effort:low</span><span>kimi-k3 @ thinking:off</span>' | ||
| '<span>phase timeout 900s</span><span>gold canary %s/%s</span>' |
There was a problem hiding this comment.
When any supported CODE_BENCH_* model, effort, thinking, or timeout override is used, the run manifest records the actual treatment but this header still labels the results with the defaults. That makes an overridden or heterogeneous benchmark appear to have run Fable/Codex at medium effort with a 900-second timeout; aggregate the manifest values and render those values instead of fixed strings.
Useful? React with 👍 / 👎.
| command -v claude >/dev/null 2>&1 || { code_die "claude CLI is required"; exit 1; } | ||
| command -v codex >/dev/null 2>&1 || { code_die "codex CLI is required"; exit 1; } |
There was a problem hiding this comment.
On a machine provisioned for a solo arm, condition A or D is rejected if Codex is absent, while condition E is rejected if Claude is absent, even though those conditions declare zero dispatches for the missing provider. Gate each prerequisite on the selected condition's dispatch counts so the independent comparator arms can run independently.
Useful? React with 👍 / 👎.
1 participant
The code benchmark battery has lived on
codex/code-benchmark-batterysince2026-08-30 while master moved on. This lands it so benchmark work continues in
one place instead of drifting further.
What lands
single-shot drivers, the official-evaluator wrapper, prediction validation,
and the results-site aggregation scripts.
swebench-verified-random50, seed recordedin the subset file). The hand-pinned 5-task canary is one instance per
repository, so its score cannot be extrapolated; a random draw can.
CODE_BENCH_SUITE, defaulting to the canary so everyexisting invocation behaves as before.
carries the upstream fix for the very issue under test, and observed runs
read the answer out of
git loginstead of deriving it.Verification
bash tests/run-all.sh --jobs 4on the merged tree: 42/42 suites passed in809s, including
test-code-bench.shat 29/29 assertions.The merge of master into this branch was clean — no conflicts in
tests/run-all.sh,.github/workflows/ci.yml, or the simulation tests thatPR #55 also touched.
🤖 Generated with Claude Code