Publish the eval observatory and preserve previous editions - #67
Conversation
B's cost read below A's only because its Codex half was missing: the Codex CLI prints tokens, not dollars, and the page priced only Claude. Rates now live in benchmarks/code/pricing.json with the date they were recorded, and the aggregator prices Codex, GLM and Kimi from the token figures each cell's logs already carry. A total-only Codex log (0.144.x prints one figure) is priced with a recorded assumed split and flagged estimated, with bounds; the driver now captures --json events so future phases carry an exact split. cost_is_complete is true only when every seat that ran is priced, and cost per resolved task is withheld until it is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fifty tasks run once cannot separate two arms whose discordance is a handful of tasks, and repeats recover almost as much power as new tasks because run-to-run variance concentrates in a few unstable tasks. A seed is a repeat index: seed 1 keeps the bare condition name so every existing run stays addressable, seed k>1 gets a .rk suffix on the cell directory, the prediction file, and model_name_or_path, so repeats never collide in one evaluator run and the report name carries the seed. run-canary --repeat K loops the seeds, the compute estimate counts them, resume treats each seed as its own cell, and the aggregator parses the suffix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Verified dataset ships a human time-to-fix estimate per task in four buckets. It is a public field, so it can sit in the frozen subset next to each instance, where a stratified draw and a by-difficulty breakdown can key on it. annotate-difficulty reads it from the pinned dataset revision and records the source and date; the manifest check refuses a label outside the four buckets so no task can drop out of every bucket silently. random50 is 17 / 27 / 5 / 1 across the buckets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nk(UB) Every score on the page now carries its uncertainty, computed in pure Python from the per-task verdicts the JSON already holds: a Wilson interval per arm, an exact two-sided McNemar test and the rescued/broken task lists per pair, a three-level bootstrap over repository, task and seed (paired for the headline delta), and Rank(UB) by interval overlap so no arm is first on the strength of one task. Known values are pinned in tests: 42/50 is 71-92, A vs B on random50 is 2/5 discordant at p = 0.45, and the aggregator reproduces both from the light-tier reports. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An agentic arm that ran every phase and left the tree unchanged has given its answer: no patch, which the evaluator scores as unresolved. The driver now records that as an empty-patch outcome in the cell and exits 3, the batch runner counts it apart from infrastructure failures and keeps it as a zero on resume, and the single-shot driver uses the same exit code for its no-applicable-patch case. A phase that fails outright still exits 1 and is retried by a rerun, because that cell has not produced a result yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The read-only Codex sandbox once produced a B run whose 5/5 was the implementer's first draft with a discarded review attached, and nothing in the tree said so. Every arm with a stage after the implementer now snapshots the implementation patch, hashes it against the final patch, and writes repair.json with repair_inert. The aggregator flags the task and counts inert cells per arm, so the page can report a bounce arm that never bounced instead of listing it beside the arms that did. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The danger-full-access workaround is a caveat on every published row and it is specific to one Codex build, so it needs a retest after every upgrade rather than a permanent note. probe-codex-sandbox makes one one-line edit per mode and records the version, the mode asked for, the mode Codex reported, and whether the file changed; the tests drive it with a stub. Each cell manifest now records the claude and codex CLI versions and the harness commit it ran under, so a row can name the build that produced it instead of trusting the build-time checkout. The upgrade and retest themselves run on the benchmark host, not here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…iewer At equal token spend, repeated sampling with an execution-grounded pick is the baseline a review pass has to beat, or the gain cannot be attributed to the reviewer. select-best-of-k.sh applies each candidate patch to a clean tree, runs the tests the patch touches (or the tests directory beside the files it changes), and keeps the candidate that applied, exited zero, passed most and failed least, earliest on a tie. When no test target can be located it degrades to apply-only and says so in selection.json rather than passing off an untested pick as a test-chosen one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
J reverses B (Codex implements, Fable repairs), K is the Codex-seat self-bounce, L is two Fable implementations chosen by the repository's tests, M/N/O are the mixed-tier crossings with their seats pinned in conditions.json, and P is B with a second review round. Each condition now declares its phases and a prose pipeline; the driver's phase plan is the executable truth and a test asserts every agentic condition's dry-run phases equal the declared ones, so an arm cannot be documented one way and run another. The Phase 1 dry run on random50 (C, D, H, I, F, G, J, K, L) dispatches 550 Claude, 200 Codex, 150 GLM and 150 Kimi over 450 cells, and the tests pin that. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ema 2.0) One JSON, one self-contained page, inline SVG, no chart library, as before; what changes is what the JSON holds. A row is now a configuration, implementer -> reviewer under a run and tier, over one or more seeds, and every score carries a Wilson interval and a Rank(UB) by interval overlap so no arm leads on one task. Cost is priced per seat with a visible "incomplete" flag when a seat has no priced figure and an "est." mark with bounds when a Codex log carried only a total. New sections, all computed in build-site-data.py so the page's JavaScript only selects: a Pareto scatter with a cost / wall / tokens axis toggle and the frontier drawn; a paired-contrast panel with the discordant table, rescued and broken tasks, exact McNemar, bootstrap delta and cost per net flip; a task x configuration heatmap grouped by Verified difficulty and sortable by rescued-by-review; expandable rows with harness commit, evaluator run, sandbox, models, token split, inert-repair count and a reproduce command; provenance badges; tier, run and seed selectors. A methodology page is rendered from the same JSON with the suite draw, stratification, pre-registered contrasts (outcomes filled from the reports, never by hand), the power table, the cost basis and downloads. runs.json registers which runs a page shows and flags the frontier partial run as generated from a dirty tree rather than hiding it; preregistration.json declares the primary contrast per phase. The page is retitled "Co-Evolution: cross-vendor review benchmark for coding agents". The old frontier page is folded into the leaderboard as flagged rows. docs/README.md now says plainly that nothing under docs/ deploys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Untested draft of the headless six-stage phase runbook and its Claude Code adapter. Not wired into code-bench.sh, no Codex adapter in AGENTS.md, no tests yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Add a responsive results site with score intervals, workflow comparisons, task evidence, CSV export, and support for future standardized suites. Preserve the original websites and data as checksum-verified archives, and wire the current page into the existing export and Pages workflows. Validated with 39 automated checks and desktop/mobile browser testing. Co-Authored-By: Codex <noreply@openai.com>
Archive the Code Battery edition from master without changing its bytes, retain existing frontier URLs, and omit the unfinished phase orchestrator from this release. Fix the Bash 3.2 empty-array failure on no-patch resume and require a successful exit in the regression test. Co-Authored-By: Codex <noreply@openai.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Feed the captured help text through a here string so grep can stop after a match without causing printf to fail under pipefail. Co-Authored-By: Codex <noreply@openai.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f45c7366f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for c in contrasts: | ||
| if {c['a_condition'], c['b_condition']} == {primary['a'], primary['b']}: | ||
| observed = c |
There was a problem hiding this comment.
When multiple phases reuse condition IDs, this selects the first contrast solely by condition name, without checking the phase's suite or model tier. The committed methodology consequently reports A@base50-light vs B@base50-light as the outcome of Phase 3 even though that phase specifies a stratified 100/300-task, three-seed suite and has no measured arms. Match both referenced rows against the phase cohort before presenting a pre-registered outcome.
Useful? React with 👍 / 👎.
| $('hero-scores').innerHTML = `<div class="hero-chart-heading"><h2>Observed resolve rate</h2><span>${esc(run)} · ${esc(tier)}</span></div>` + | ||
| (rows.length ? rows.map(r => `<div class="hero-score-row" style="--series:${color(r)}"><div class="hero-score-label"><strong>${esc(title(r))}</strong><b>${(r.score.rate * 100).toFixed(0)}<small>%</small></b></div><div class="hero-track"><div class="hero-fill" style="width:${r.score.rate * 100}%"></div><div class="hero-range" style="left:${r.score.wilson_low * 100}%;width:${(r.score.wilson_high - r.score.wilson_low) * 100}%"></div></div><div class="hero-bottom"><span>${r.score.resolved} / ${r.score.n} resolved</span><span>95% CI ${pct(r.score.wilson_low)}–${pct(r.score.wilson_high)}</span></div></div>`).join('') : '<p class="empty">No completed, publishable results for this selection yet. Inspect available results in the table below.</p>') + |
There was a problem hiding this comment.
When a row has multiple seeds, this still presents the Wilson bounds computed over all task-seed cells as the headline interval. Repeated outcomes from the same task and repository are clustered rather than independent Bernoulli observations, which is why the exporter already computes row.bootstrap over repository, task, and seed and the preregistration specifies that interval for seeded runs. The planned three-seed phase would otherwise publish misleading uncertainty in its hero and results table.
Useful? React with 👍 / 👎.
| a = {t['instance_id']: t['status'] == 'resolved' for t in first['per_task'] | ||
| if t['status'] in ('resolved', 'unresolved', 'no-patch') and t['seed'] == 1} | ||
| b = {t['instance_id']: t['status'] == 'resolved' for t in second['per_task'] | ||
| if t['status'] in ('resolved', 'unresolved', 'no-patch') and t['seed'] == 1} |
There was a problem hiding this comment.
For any --repeat run, these maps discard every outcome except seed 1, so McNemar p-values, gained/lost issue lists, and net task counts ignore seeds 2 through k. The same contrast's bootstrap delta is computed immediately afterward from all seeds, meaning one displayed conclusion combines two different populations and can even disagree in direction. Aggregate each task's seeded outcomes consistently for every paired statistic.
Useful? React with 👍 / 👎.
| seeds = sorted({seed for (cond, seed) in latest if cond == cond_id} | ||
| | {seed for (cond, seed, _) in attempts_index if cond == cond_id}) |
There was a problem hiding this comment.
If evaluation for an intermediate repeat is absent—for example, complete reports exist for seeds 1 and 3 but seed 2 failed—this derives seeds as [1, 3]. build_row then expects only two seeds and can mark the row complete and publishable, even though run-canary --repeat 3 defines the required sequence as seeds 1 through 3. Presence of seed k should imply the full contiguous range through k, or the expected repeat count should come from run metadata.
Useful? React with 👍 / 👎.
| # report file name alone. | ||
| REPORT_NAME_RE = re.compile( | ||
| r'^co-evolution-condition-(?P<cond>[A-Za-z0-9_-]+)\.(?P<run_id>.+)\.json$') | ||
| r'^co-evolution-condition-(?P<cond>[A-Za-z0-9_-]+?)(?:\.r(?P<seed>\d+))?\.(?P<run_id>[^.]+)\.json$') |
There was a problem hiding this comment.
A run ID such as trial.1 is explicitly accepted by run-canary.sh, but the new [^.]+ run-ID group rejects its evaluator filename, so newest_reports silently omits every result from that valid run. Preserve support for dots in the run label while still separating the optional .rN seed suffix.
Useful? React with 👍 / 👎.
2 participants
Summary
Make the current evaluation scores easy to read and compare on the public GitHub Pages site. The new observatory shows the completed SWE-bench Verified results (84%, 78%, 66%) with confidence intervals, cost per resolved task, repository comparisons, paired gains/losses, individual evaluator records, and CSV export. Partial and flagged runs stay out of headline comparisons, and future standardized suites can be registered without redesigning the interface.
The original live Code Battery site is preserved as a complete dated archive, including its frontier and proof-of-concept pages. The subsequent schema 2.0 edition is also preserved. All 16 archived files have pinned checksums, and the existing frontier URLs remain accessible.
Scope and validation