Static results site for judged benchmark batches - #63
Conversation
The plan-composition suite produced numbers that only existed inside a markdown report. This publishes them: a cost-versus-quality scatter, a sortable leaderboard and a per-task drill-down, republished with one command. benchmarks/export-site-data.sh --batch <id> reduces results/<batch>/ to docs/data/<batch>.json plus an index, reading the same meta.json and bench-pair/1.0 verdict files report.sh reads and running the same win-matrix and Bradley-Terry awk. Keeping that one implementation is what stops the site and the report from disagreeing: every figure the site shows for batch b1 matches benchmarks/reports/b1.md, including the per-judge B-vs-A and B-vs-D counts and the section 6 cost column. The quality axis is the suite's own rubric rather than a metric baked into the page. A pairwise-judged suite scores as win rate, where 50% is parity, so the data file carries a `metric` block with its label, bounds and parity line, and the page reads its axes from that. A suite scored some other way publishes with its own rubric and needs no page change. Only publishable fields leave the results folder: scores, costs, model and condition names, judge tallies, and task ids and difficulty. The verdict files also carry judge reasoning and verbatim quotes from the plans, and the cells carry the plans themselves. Three things keep that out rather than leaving it to review — the export reads a fixed field list and fails on an unexpected one, docs/data/schema.json sets additionalProperties false throughout, and the fixture test asserts no prompt or plan text survives an export. Judges stay separate everywhere, per PREREGISTRATION.md, and the cost axis is labelled a floor because Codex has no token sidecar and the direct GLM and Kimi sidecars report tokens without a price. Coverage is rendered above every number, and it counts pairs that have a verdict but could not be scored — 8 per judge on b1 — rather than calling that full coverage. Deploy wiring is a separate commit: pushing a workflow change needs a token scope this one does not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e7c13e16a0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| regenerates `docs/data/index.json` from every batch file in that folder. Commit | ||
| the result and push to `master`; the `pages` workflow redeploys on any change | ||
| under `docs/index.html`, `docs/assets/` or `docs/data/`. |
There was a problem hiding this comment.
When this commit is pushed to master, the advertised /plans/ site is neither deployed nor triggers a deployment: the checked .github/workflows/pages.yml watches only benchmarks/site/public/** and stages only that directory into _site. Include docs/** in the trigger and copy these files into _site/plans/; otherwise the new site and subsequent exported batches remain unavailable.
Useful? React with 👍 / 👎.
| | unique | join(", ")' "$WORK/out.json") | ||
| [[ -z "$LEAKED" ]] || die "publishability guard: unexpected field(s) in export output: $LEAKED" | ||
|
|
||
| jq . "$WORK/out.json" > "$OUT_FILE" || die "could not write $OUT_FILE" |
There was a problem hiding this comment.
When an external or malformed batch produces a contract-invalid value—for example, an unrecognized verdict that becomes an unsupported task outcome—this command only checks JSON syntax with jq . and still exits successfully. benchmarks/lib/validate-schema.jq is used only by the test suite, so the documented additionalProperties, enums, types, and bounds do not actually protect published exports; invoke the validator against docs/data/schema.json here and fail before replacing the output.
Useful? React with 👍 / 👎.
| file: ("data/" + .batch + ".json") }' "$f" 2>/dev/null >> "$WORK/batches.jsonl" || true | ||
| done < <(find "$DOCS_DIR/data" -maxdepth 1 -name '*.json' -type f 2>/dev/null | LC_ALL=C sort) |
There was a problem hiding this comment.
When callers use the supported --out flag, the batch is written to that path but index generation still scans only $DOCS_DIR/data and constructs data/<batch>.json. A fresh export such as --out /tmp/custom.json therefore succeeds while producing an index whose default batch is absent, and --out docs/data/custom.json produces a link to a different filename. Either derive the index entry from OUT_FILE or require --no-index for outputs outside the canonical location.
Useful? React with 👍 / 👎.
2 participants
Batch
b1's numbers only existed insidebenchmarks/reports/b1.md. This gives them a shareable visual home: a cost-versus-quality scatter, a sortable leaderboard, and a per-task drill-down, republished with one command.The export
Reduces
benchmarks/results/b1/todocs/data/b1.jsonplusdocs/data/index.json. It reads the samemeta.jsonandbench-pair/1.0verdict filesreport.shreads, and runs the same win-matrix and Bradley-Terry awk — one implementation, so the site and the report cannot drift apart.Every number checked against
benchmarks/reports/b1.mdand matching: per-judge B-vs-A (fable 4/1/3/0, codex 7/0/1/0, glm 5/0/3/0) and B-vs-D, the section 6 cost column (A $5.5873, B $9.6163, C $12.6194, D $16.3745), mean words and wall seconds, every win-matrix cell, every Bradley-Terry strength, judge integrity, cross-judge agreement (48/38/30) and the length-bias rates.The metric
The quality axis is the suite's own rubric, not one baked into the page. A pairwise-judged suite scores as win rate, where 50% is parity — the arena-style equivalent of SWE-bench's "% resolved". The data file carries a
metricblock with its label, bounds and parity line, and the page reads its axes from that, so a suite scored some other way publishes with its own rubric and needs no page change.What may be published
Scores, costs, model and condition names, judge tallies, task ids and difficulty. The verdict files also carry judge reasoning and verbatim quotes from the plans, and the cells carry the plans themselves. Three things keep that out rather than leaving it to review:
docs/data/schema.jsonsetsadditionalProperties: falsethroughout;benchmarks/tests/test-site-export.shasserts no prompt or plan text survives an export.Audited on the real output: of the strings in
docs/data/b1.json, zero appear in any corpus prompt or generated plan. The only long strings are the page's own copy and the four condition descriptions, already public inbenchmarks/conditions.yaml.Honesty carried over from the report
PREREGISTRATION.md§4.Tests
benchmarks/tests/test-site-export.sh— 74 checks against a hand-computed fixture batch underbenchmarks/tests/fixtures/site/, registered intests/run-all.sh. It also negative-tests the schema validator, so "output validates" is not a vacuous pass. Bash + jq only, no new dependency.Not in this branch
The deploy wiring — staging
docs/into_site/plans/in.github/workflows/pages.yml, so the SWE-bench code battery keeps the root URL and nothing goes offline. Pushing a workflow change needs a token scope this session does not have; the commit is ready locally.🤖 Generated with Claude Code