Publish light and frontier tier results, each scoped to its own run - #64
Conversation
Scoping a page to run label "base50" also matched "base50-light-2026...", so the frontier page would have rendered light-tier results under frontier model names. The bug is invisible in the output: both are real runs of the same conditions on the same subset, so the page looks correct and reports the wrong experiment. The evaluator's timestamp suffix contains no dash, so the label is everything before the final one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two bugs made a page report an experiment it had not run. Run labels matched by prefix, so scoping a page to "base50" also matched "base50-light-2026...". The frontier page rendered light-tier results under frontier model names. Nothing looks wrong in that output: both are real runs of the same conditions on the same subset. The cell index was global, so two runs whose patches for a task were byte-identical collided on the key and the later one won. A frontier row then read its models and telemetry out of a light-tier cell, which is why the frontier page listed both fable and sonnet as its models. Labels and compositions now name the models that actually ran. "fable-solo" on a page where Sonnet ran was the same class of error as the two above: a true number under a false name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 544fb95c4b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| <header class="masthead"><div class="eyebrow"><span>SWE-bench Verified · 50-task frozen subset</span><span>Official pinned evaluator, Docker</span><span>Built 2026-09-03T02:35:15Z</span><span><a href="poc.html">1-task proof of concept</a></span></div><h1>Co-Evolution Code Battery</h1><p class="standfirst">Does putting a second model in the loop produce better patches than one model working alone? Nine configurations, 50 pinned SWE-bench Verified tasks, every patch scored by the official evaluator in Docker.</p><div class="runmeta"><span>fable @ medium</span><span>gpt-5.6-sol @ medium</span><span>glm-5.3-flash @ effort:low</span><span>kimi-k3 @ thinking:off</span><span>phase timeout 900s</span><span>gold canary 1/1</span><span>harness 9eedc40</span></div></header> | ||
| <div class="callout"><h2>Read this before the table</h2><p>This run is unfinished. Each arm is scored against the tasks it has actually run, not against the full subset, because a task an arm has not reached yet is not a failure. Nothing here is a final score.</p><p>The arms have run different numbers of tasks so far (20 to 23 of 50), so the percentages are not directly comparable to each other. Read the task-by-task table below for any head-to-head comparison; it is the only view here that holds the task set fixed.</p><p>50 tasks is a probe, not a ranking. One task is 2 points, so a one-task gap between two rows is well inside what a 50-task sample produces by chance.</p><p>These numbers are not comparable to published full-500 SWE-bench Verified scores: the subset is fixed and was chosen for the harness, not drawn at random.</p><p>The two tiers are listed separately because they are not the same test. An agentic row had file tools and could run the test suite; a single-shot row got one prompt and answered once.</p></div> | ||
| <div class="tiles"><div class="tile"><span class="k">Configurations</span><span class="v">5 <span class="of">/ 9</span></span><span class="n">measured on this subset</span></div><div class="tile"><span class="k">Tasks per cell</span><span class="v">50</span><span class="n">frozen subset</span></div><div class="tile"><span class="k">Scored cells</span><span class="v">94</span><span class="n">official evaluator, Docker</span></div><div class="tile"><span class="k">Fable dispatches</span><span class="v">43</span><span class="n">across every measured arm</span></div><div class="tile"><span class="k">Wall time</span><span class="v">5h 08m</span><span class="n">model time, all providers</span></div><div class="tile"><span class="k">Cost per resolved</span><span class="v">$1.32</span><span class="n">Claude phases only</span></div></div> | ||
| <header class="masthead"><div class="eyebrow"><span>SWE-bench Verified · 50-task frozen subset</span><span>Official pinned evaluator, Docker</span><span>Built 2026-09-04T12:57:24Z</span><span><a href="frontier.html">frontier tier</a></span><span><a href="poc.html">1-task proof of concept</a></span></div><h1>Co-Evolution Code Battery</h1><p class="standfirst">Does putting a second model in the loop produce better patches than one model working alone? Nine configurations, 50 pinned SWE-bench Verified tasks, every patch scored by the official evaluator in Docker.</p><div class="runmeta"><span>tier: light</span><span>sonnet</span><span>gpt-5.6-terra @ medium</span><span>phase timeout 900s</span><span>gold canary 1/1</span><span>harness 39efd7f</span></div></header> |
There was a problem hiding this comment.
For the newly published light-tier page, this metadata says phase timeout 900s, but the normal light configuration sets CODE_BENCH_PHASE_TIMEOUT to 2400 seconds in benchmarks/code/lib/code-bench-lib.sh:53-56. Because render-page.py hard-codes 900 for every page, readers are given the frontier-tier limit for the Sonnet/Terra experiment; record the effective timeout in the generated data and render that value instead.
Useful? React with 👍 / 👎.
| % (klass, esc(row['condition']), | ||
| esc(retarget(row['label'], data or {})), | ||
| esc(retarget(COMPOSITION.get(row['condition'], row['description']), | ||
| data or {})), |
There was a problem hiding this comment.
When configuration.claude is sonnet, these lines retarget the row to sonnet-solo but leave the table header hard-coded as Fable calls at line 185. The light page consequently presents Sonnet dispatch counts under a Fable label; derive that header from the same configured Claude seat, or use a model-neutral label such as Claude calls.
Useful? React with 👍 / 👎.
| "report_file": "benchmarks/results/code/evaluation/co-evolution-condition-E.base50-20260903T022025Z.json" | ||
| } | ||
| ], | ||
| "caveat": "Frozen five-task probe of SWE-bench Verified, scored by the official evaluator. One task is 20 points; these numbers are not comparable to published full-500 SWE-bench Verified scores." |
There was a problem hiding this comment.
This newly published frontier artifact declares suite.task_count as 50 but its footer caveat calls it a five-task probe and says each task is worth 20 points; for this suite each task is 2 points. The fixed string originates in build-site-data.py:521-523, so generate it from suite['task_count'] and rebuild both 50-task artifacts.
Useful? React with 👍 / 👎.
1 participant
Publishes the completed light-tier run and the refreshed frontier run as two
pages, and fixes two bugs that made a page report an experiment it had not run.
Results
Light tier (
sonnet/gpt-5.6-terra), complete at 50/50 per arm:On the identical 50 tasks the bounce fixed 5 and broke 2 (net +3, p = 0.45), and
beat terra alone by 9 with none lost (p = 0.004). It solved two tasks neither
model solved alone, so it is not acting purely as a selector between them.
Frontier tier (
fable/gpt-5.6-sol), partial — Fable arms stopped at theaccount limit:
Fable resolved every task it attempted, so this subset has no headroom at that
tier and the bounce can only cost. It fixed 0 and broke 2. That is a ceiling
effect, not a finding about bouncing.
Bugs
Run labels matched by prefix. Scoping to
base50also matchedbase50-light-2026…, so the frontier page rendered light-tier results underfrontier model names. Nothing looks wrong in that output — both are real runs of
the same conditions on the same subset — which is what makes it dangerous. It
surfaced only because Fable at 39/50 contradicted the 35/35 in the raw report.
The cell index was global. Two runs whose patch for a task is byte-identical
collided on the index key and the later one won, so a frontier row could read
its models and telemetry out of a light-tier cell. The frontier page listed both
fableandsonnetas its models.Labels named the wrong model. Condition labels are written against the
default seats, so the light page called arm A
fable-soloon a run where Sonnetdid the work. Labels and compositions are now retargeted to the models the run
actually used.
Verification
bash tests/run-all.sh --jobs 4: 42/42 suites in 1773s. Both pages rebuilt andchecked: the light page reports only
sonnetandgpt-5.6-terra, the frontierpage only
fableandgpt-5.6-sol, and each arm's counts match its evaluatorreport.
🤖 Generated with Claude Code