Keep the self-judged plan suite off the public site - #65
Conversation
The plan-composition suite scores Co-Evolution using Fable to judge plans that Fable composed, in every condition. That is a useful internal signal and not a publishable result, and the repo is public: b1.json shipped a "Win rate" for the project's own protocol, which borrows credibility from the official-evaluator numbers it would sit beside. Removes the second results site and its published dataset, and points the exporter at the ignored results tree so running it no longer stages a publication. The exporter, its jq libraries, its fixtures and its 74 checks are untouched; the schema stays as the contract they validate against. The README described committing that export to docs/ and a pages workflow that redeployed on changes there. No such trigger existed - pages.yml watches benchmarks/site/public/** and nothing else - so the site was never live and the instructions could not have worked. CLAUDE.md now states which results may be published and why, because the policy lived only in a memory file and a comment inside one script, where the agent that built the second site never saw it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 921361ed90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # internal signal and not a publishable result, so the export lands in the | ||
| # ignored results tree rather than in a directory staged for the public site. | ||
| # Point --docs-dir somewhere else deliberately if you have a reason to. | ||
| DOCS_DIR="${CODE_BENCH_SITE_EXPORT_DIR:-$REPO_ROOT/benchmarks/results/site-export}" |
There was a problem hiding this comment.
After changing the default export location and deleting the docs-site HTML/assets, docs/README.md still claims that site is live, lists the deleted files, and directs users to run this exporter, commit docs/data/*, and preview docs/index.html. With the inspected .github/workflows/pages.yml serving only benchmarks/site/public/, following that retained runbook now previews a broken directory and produces no files to commit; update or remove it as part of retiring the site.
AGENTS.md reference: AGENTS.md:L108-L110
Useful? React with 👍 / 👎.
| Writes `benchmarks/results/site-export/b1.json` and an index beside it, under | ||
| the ignored results tree. |
There was a problem hiding this comment.
The exporter still constructs its output as $DOCS_DIR/data/$BATCH_ID.json and the index as $DOCS_DIR/data/index.json, so this new default writes benchmarks/results/site-export/data/b1.json, not benchmarks/results/site-export/b1.json as stated here. Anyone using the new internal-review runbook will look for or archive the wrong path; either include the data/ component in the documentation or change the output construction.
AGENTS.md reference: AGENTS.md:L108-L110
Useful? React with 👍 / 👎.
1 participant
The repo is public, and
docs/data/b1.jsonshipped a "Win rate" for theCo-Evolution protocol. Every condition in that suite is composed by Fable and
its primary judge is Fable, so the project was scoring itself with the model
under evaluation — and doing it next to SWE-bench numbers from an official
evaluator, which lends the borrowed credibility.
That suite was retired to internal-only on 2026-09-01. The policy lived in a
memory file and a comment inside
aggregate.sh, so the agent that built thesecond site never saw it. That is the actual failure here, and the fix is to put
it where the next agent will read it.
What changes
docs/index.html,docs/assets/site.*) andits published dataset (
docs/data/b1.json,docs/data/index.json).export-site-data.shat the ignoredbenchmarks/results/site-export/instead of a directory staged for publication.
CLAUDE.mdwhich results may be published, and why.What is kept
export-site-data.sh, its jq libraries, its fixtures and its 74 checks areuntouched, and
docs/data/schema.jsonstays as the contract they validateagainst. The suite is worth running and reading; it is not worth publishing.
Nothing is lost either way — the removed files remain in history.
A false instruction, corrected
benchmarks/README.mddescribed committing the export todocs/and apagesworkflow that redeployed on changes under
docs/index.html,docs/assets/ordocs/data/. No such trigger exists:pages.ymlwatchesbenchmarks/site/public/**and nothing else. The site was never live, andfollowing those instructions would never have published anything.
Verification
benchmarks/tests/test-site-export.sh: 74/74 after the default output change.The exporter's own tests pass
--outexplicitly, so they were unaffected.🤖 Generated with Claude Code