Benchmark expansion Phase 0: priced seats, seeds, difficulty, statistics, site 2.0 - #66
Conversation
B's cost read below A's only because its Codex half was missing: the Codex CLI prints tokens, not dollars, and the page priced only Claude. Rates now live in benchmarks/code/pricing.json with the date they were recorded, and the aggregator prices Codex, GLM and Kimi from the token figures each cell's logs already carry. A total-only Codex log (0.144.x prints one figure) is priced with a recorded assumed split and flagged estimated, with bounds; the driver now captures --json events so future phases carry an exact split. cost_is_complete is true only when every seat that ran is priced, and cost per resolved task is withheld until it is. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Fifty tasks run once cannot separate two arms whose discordance is a handful of tasks, and repeats recover almost as much power as new tasks because run-to-run variance concentrates in a few unstable tasks. A seed is a repeat index: seed 1 keeps the bare condition name so every existing run stays addressable, seed k>1 gets a .rk suffix on the cell directory, the prediction file, and model_name_or_path, so repeats never collide in one evaluator run and the report name carries the seed. run-canary --repeat K loops the seeds, the compute estimate counts them, resume treats each seed as its own cell, and the aggregator parses the suffix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Verified dataset ships a human time-to-fix estimate per task in four buckets. It is a public field, so it can sit in the frozen subset next to each instance, where a stratified draw and a by-difficulty breakdown can key on it. annotate-difficulty reads it from the pinned dataset revision and records the source and date; the manifest check refuses a label outside the four buckets so no task can drop out of every bucket silently. random50 is 17 / 27 / 5 / 1 across the buckets. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…nk(UB) Every score on the page now carries its uncertainty, computed in pure Python from the per-task verdicts the JSON already holds: a Wilson interval per arm, an exact two-sided McNemar test and the rescued/broken task lists per pair, a three-level bootstrap over repository, task and seed (paired for the headline delta), and Rank(UB) by interval overlap so no arm is first on the strength of one task. Known values are pinned in tests: 42/50 is 71-92, A vs B on random50 is 2/5 discordant at p = 0.45, and the aggregator reproduces both from the light-tier reports. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An agentic arm that ran every phase and left the tree unchanged has given its answer: no patch, which the evaluator scores as unresolved. The driver now records that as an empty-patch outcome in the cell and exits 3, the batch runner counts it apart from infrastructure failures and keeps it as a zero on resume, and the single-shot driver uses the same exit code for its no-applicable-patch case. A phase that fails outright still exits 1 and is retried by a rerun, because that cell has not produced a result yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The read-only Codex sandbox once produced a B run whose 5/5 was the implementer's first draft with a discarded review attached, and nothing in the tree said so. Every arm with a stage after the implementer now snapshots the implementation patch, hashes it against the final patch, and writes repair.json with repair_inert. The aggregator flags the task and counts inert cells per arm, so the page can report a bounce arm that never bounced instead of listing it beside the arms that did. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The danger-full-access workaround is a caveat on every published row and it is specific to one Codex build, so it needs a retest after every upgrade rather than a permanent note. probe-codex-sandbox makes one one-line edit per mode and records the version, the mode asked for, the mode Codex reported, and whether the file changed; the tests drive it with a stub. Each cell manifest now records the claude and codex CLI versions and the harness commit it ran under, so a row can name the build that produced it instead of trusting the build-time checkout. The upgrade and retest themselves run on the benchmark host, not here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…iewer At equal token spend, repeated sampling with an execution-grounded pick is the baseline a review pass has to beat, or the gain cannot be attributed to the reviewer. select-best-of-k.sh applies each candidate patch to a clean tree, runs the tests the patch touches (or the tests directory beside the files it changes), and keeps the candidate that applied, exited zero, passed most and failed least, earliest on a tie. When no test target can be located it degrades to apply-only and says so in selection.json rather than passing off an untested pick as a test-chosen one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
J reverses B (Codex implements, Fable repairs), K is the Codex-seat self-bounce, L is two Fable implementations chosen by the repository's tests, M/N/O are the mixed-tier crossings with their seats pinned in conditions.json, and P is B with a second review round. Each condition now declares its phases and a prose pipeline; the driver's phase plan is the executable truth and a test asserts every agentic condition's dry-run phases equal the declared ones, so an arm cannot be documented one way and run another. The Phase 1 dry run on random50 (C, D, H, I, F, G, J, K, L) dispatches 550 Claude, 200 Codex, 150 GLM and 150 Kimi over 450 cells, and the tests pin that. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ema 2.0) One JSON, one self-contained page, inline SVG, no chart library, as before; what changes is what the JSON holds. A row is now a configuration, implementer -> reviewer under a run and tier, over one or more seeds, and every score carries a Wilson interval and a Rank(UB) by interval overlap so no arm leads on one task. Cost is priced per seat with a visible "incomplete" flag when a seat has no priced figure and an "est." mark with bounds when a Codex log carried only a total. New sections, all computed in build-site-data.py so the page's JavaScript only selects: a Pareto scatter with a cost / wall / tokens axis toggle and the frontier drawn; a paired-contrast panel with the discordant table, rescued and broken tasks, exact McNemar, bootstrap delta and cost per net flip; a task x configuration heatmap grouped by Verified difficulty and sortable by rescued-by-review; expandable rows with harness commit, evaluator run, sandbox, models, token split, inert-repair count and a reproduce command; provenance badges; tier, run and seed selectors. A methodology page is rendered from the same JSON with the suite draw, stratification, pre-registered contrasts (outcomes filled from the reports, never by hand), the power table, the cost basis and downloads. runs.json registers which runs a page shows and flags the frontier partial run as generated from a dirty tree rather than hiding it; preregistration.json declares the primary contrast per phase. The page is retitled "Co-Evolution: cross-vendor review benchmark for coding agents". The old frontier page is folded into the leaderboard as flagged rows. docs/README.md now says plainly that nothing under docs/ deploys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d8889c054a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| priced = pricing_mod.price_sidecar(pricing, model, artifact + '.usage.json') | ||
| if priced: | ||
| out[seat + '_cost_usd'] += priced['cost_usd'] | ||
| out[seat + '_precision'].append('exact') |
There was a problem hiding this comment.
For any GLM or Kimi call with a captured usage sidecar, price_sidecar() returns input, cached-input, and output counts, but this branch retains only the dollar cost. The later tokens calculation includes only Claude and Codex, so conditions C/F/G/H/I report zero or understated tokens per task and can incorrectly appear on the token Pareto frontier. Accumulate the sidecar token fields into the cell and row telemetry as well.
Useful? React with 👍 / 👎.
| reset_tree() { | ||
| git -C "$WORKSPACE" checkout -q -- . 2>/dev/null | ||
| git -C "$WORKSPACE" clean -fdq 2>/dev/null | ||
| } |
There was a problem hiding this comment.
When a candidate's test run creates ignored state such as build output, generated caches, or coverage/config artifacts, git clean -fdq preserves it. The next candidate is therefore not evaluated from the same clean repository state and may pass or fail because of its predecessor, causing condition L to select and score the wrong patch. Reset each candidate in an isolated checkout or otherwise remove ignored test artifacts without deleting required environment state.
Useful? React with 👍 / 👎.
| observed = None | ||
| for c in contrasts: | ||
| if {c['a_condition'], c['b_condition']} == {primary['a'], primary['b']}: | ||
| observed = c | ||
| break |
There was a problem hiding this comment.
When the same condition pair has results in multiple batches or model tiers, this selects the first contrast solely by condition IDs and ignores the phase's tier and intended run. For example, A already has light and frontier rows; once M is measured, phase 2 can publish the earlier light-A-versus-M contrast even though its preregistered question explicitly requires frontier A. Select the rows using the phase's tier/run requirements rather than whichever condition-only pair occurs first.
Useful? React with 👍 / 👎.
| REPORT_NAME_RE = re.compile( | ||
| r'^co-evolution-condition-(?P<cond>[A-Za-z0-9_-]+)\.(?P<run_id>.+)\.json$') | ||
| r'^co-evolution-condition-(?P<cond>[A-Za-z0-9_-]+?)(?:\.r(?P<seed>\d+))?\.(?P<run_id>[^.]+)\.json$') |
There was a problem hiding this comment.
For a permitted run label containing a dot, such as experiment.v2, evaluation produces a report named like co-evolution-condition-A.experiment.v2-<timestamp>.json, but run_id is constrained to [^.]+, so the report never matches and aggregation emits no row. Both run-canary and the evaluator accept dots in their filesystem-safe labels, and the previous report regex accepted them, so the report parser should preserve that supported input.
Useful? React with 👍 / 👎.
2 participants
Phase 0 of
.planning/notes/2026-09-04-benchmark-expansion-plan.md(harness fixes, no model calls) plus the section-5 site redesign. Stops at the Phase 1 gate: nothing here spends a model call.What changed (one commit per item)
benchmarks/code/pricing.json(dated list prices). Codex, GLM and Kimi are priced from the token figures in each cell's own logs;cost_is_completeis true only when every seat that ran is priced. A total-only Codex log (0.144.x) is priced with a recorded split and flaggedestimatedwith bounds; the driver now runscodex exec --jsonso future phases carry an exact split.--repeat Konrun-canary; cells keyed(task, condition, seed); seed k>1 gets.rkon the cell dir, prediction file andmodel_name_or_path, so repeats never collide in one evaluator run. Resume treats each seed as its own cell.annotate-difficultywrites the SWE-bench Verified time-to-fix bucket into each subset from the pinned dataset revision;checkrefuses a label outside the four buckets. random50 is 17 / 27 / 5 / 1.benchmarks/site/stats.py: Wilson CI, paired discordance table, exact McNemar, hierarchical bootstrap over repo → task → seed (paired for deltas), Rank(UB) by CI overlap, pass@k / pass^k / flip rate, power table. Tests pin 42/50 = 71–92 and A vs B = 2/5 at p = 0.45; the aggregator reproduces both from the light-tier reports.empty-patchoutcome and exits 3; the batch counts it as a scored zero, keeps it on resume, and moves on. Infra failures still exit 1 and are retried.repair.jsonwithrepair_inert; the site counts inert cells per arm and flags the task.probe-codex-sandbox(tracked, stub-tested) records the Codex version, the mode asked for, the mode Codex reported and whether the file changed. Cell manifests now recordclaude/codexCLI versions and the harness commit/dirty flag. The upgrade to 0.153.2 and the workspace-write retest run on the benchmark host, not here (this session's machine has 0.139.0; the runs come from the runtime worktree on desktop01 at 0.144.5).select-best-of-k.sh: applies each candidate, runs the tests the patch touches (or the tests dir beside the changed files), keeps applied → exit 0 → most passed → fewest failed → earliest; reportsapply-onlywhen no test target exists instead of pretending.conditions.json), P (two-round bounce). Each condition declaresphases; a test asserts the driver's dry-run phases equal the declared ones.implementer → reviewer (tier, run); every score carries a Wilson CI and Rank(UB); cost per task with a visible incomplete flag andest.bounds; wall p50/p90; tokens; provenance badges; expandable row with harness commit, evaluator run, sandbox, models, token split, inert count and a reproduce command; Pareto scatter with cost/wall/tokens toggle; paired-contrast panel; task × config heatmap grouped by difficulty, sortable by rescued-by-review; tier/run/seed selectors; methodology page from the same JSON;runs.jsonregistry (frontier partial run flagged, not hidden);preregistration.json; retitled.docs/README.mdnow says nothing underdocs/deploys.Rebuilt
benchmarks/site/public/(leaderboard, methodology, poc, schema) from the existing light-tier and frontier evaluator reports. The oldfrontier.html/jsonis folded intoleaderboard.jsonas flagged rows. No result on disk was deleted or regenerated.What the priced light tier says now
A vs B: 2 / 5 discordant, exact McNemar p = 0.45, paired delta +6.0 points (bootstrap −7.6 to +18.8), cost per net rescued task $2.16. B now costs more than A, as it should.
Phase 1 gate (needs a go)
Primary pre-registered contrast: B vs D. Run on the benchmark host from the runtime worktree, after merging this PR:
Dry run on this branch: 450 cells, 550 Claude, 200 Codex, 150 GLM, 150 Kimi dispatches (pinned by a test).
Cost estimate, 50 tasks × 1 seed, list price
Basis: measured light tier (Sonnet implement $1.04/task mean over 50 cells; a Codex terra phase $0.10 est., range $0.01–$0.69), plan assumption for a Sonnet repair pass at 0.8× an implement, critic calls at ~30k input + 2.5k output tokens.
Claude is the CLI's list-price figure for work billed to the Max plan; Codex is plan compute priced at list with the total-only caveat; GLM and Kimi are pay-as-you-go. Wall: about 3–5 days at 2-wide. Exit: B-vs-D discordance table published on the leaderboard's contrast panel.
Verification
bash tests/run-all.sh: every suite touched here passes (test-code-bench.sh79/79,test-site-build.sh27/27,docs-sync, etc.). Ten suites fail on this host both on pristineorigin/masterand on this branch, for host reasons (yqabsent forsmoke.sh; scorer sandbox and temp-path failures inscorer-verification,bounce-scorer-verification,pr-emitter,code-proposer,policy-proposer,marker-lifecycle,report-bounce-encoding,bounce-judge,test-judge-lib-extraction); none of them touches the benchmark harness or the site.benchmarks/site/aggregate.shrebuiltleaderboard.json/.html/-methodology.htmlandpoc.*from the existing reports; the page renders in light and dark, and has no horizontal body scroll at 375px or 1265px.🤖 Generated with Claude Code