Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,24 @@ Co-evolution tools are integrated into GSD workflows:
- Convergence is honest, never forced-silent. A `co-evolve-bouncer.sh` run ends in one of three recorded states (`state.json.convergence_status`): **converged** (markers hit 0 within the configured passes), **adjudicated** (markers survived, so one forced-adjudication pass resolved each one and wrote `adjudication-report.md` mapping every stripped marker to the chosen text + a one-line rationale), or **stuck** (adjudication could not defensibly resolve every marker — the working document is preserved WITH its markers, labeled NOT-final, and fails the scorer gate). A naturally-converging run is byte-identical to the pre-adjudication bouncer; the adjudication pass fires only when markers survive. Exit semantics by design: a stuck document run still exits 0 — the `CO-EVOLVE:STUCK` label and the failing scorer gate carry the signal, not the exit code; only `--execute` refuses a stuck plan, with exit 1.
- Agent-bouncer overwrites the input file in place; orchestrators should back up first

## What May Be Published as a Result

The public results site carries **standardized benchmarks scored by their own
official evaluators** and nothing else. Today that means SWE-bench Verified
through the pinned harness in `benchmarks/code/`.

The plan-composition suite (`benchmarks/`, batches `b1` and successors) is
**internal only**. Every condition in it is composed by Fable and its primary
judge is Fable, so it scores Co-Evolution with the model under evaluation. Run
it, read it, act on it — and do not put it on the site, because publishing it
beside official-evaluator numbers lends it their credibility. Its exporter
writes to the ignored `benchmarks/results/site-export/` for that reason.

One site: `benchmarks/site/`, deployed by `.github/workflows/pages.yml` from
`benchmarks/site/public/`. Before adding a second, check whether that workflow
would actually serve it — a previous results site under `docs/` documented a
deploy trigger that did not exist and was never live.

## Status

- v1.4 npm/MCP publish pending — a human gate, not yet shipped.
Expand Down
33 changes: 20 additions & 13 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,27 +119,34 @@ Writes `results/b1/REPORT.md` and a committed copy under
verdicts, then per-judge matrices, judge-integrity stats, and the exploratory
sections — in that order, matching `PREREGISTRATION.md` section 1.

**6. Publish.**
**6. Export, for internal review only.**

```bash
bash benchmarks/export-site-data.sh --batch b1
```

Writes `docs/data/b1.json` and regenerates `docs/data/index.json`, which is
everything the results site at
<https://alanshurafa.github.io/co-evolution/plans/> renders. Commit those two
JSON files and push to `master`; the `pages` workflow redeploys on any change
under `docs/index.html`, `docs/assets/` or `docs/data/`. That is the whole
republish path — one command, then a commit.
Writes `benchmarks/results/site-export/b1.json` and an index beside it, under
the ignored results tree.
Comment on lines +128 to +129

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Document the actual nested export path

The exporter still constructs its output as $DOCS_DIR/data/$BATCH_ID.json and the index as $DOCS_DIR/data/index.json, so this new default writes benchmarks/results/site-export/data/b1.json, not benchmarks/results/site-export/b1.json as stated here. Anyone using the new internal-review runbook will look for or archive the wrong path; either include the data/ component in the documentation or change the output construction.

AGENTS.md reference: AGENTS.md:L108-L110

Useful? React with 👍 / 👎.


**These numbers are not publishable and this suite is not a lane on the results
site.** Every condition here is composed by Fable and the primary judge is
Fable, so the suite scores Co-Evolution using the model it is evaluating. That
is a useful internal signal and an indefensible public claim, and placing it
next to official-evaluator numbers would lend it their credibility. The public
site carries standardized benchmarks with official evaluators only — see
`benchmarks/code/README.md`.

An earlier version of this document described committing the export to `docs/`
and a `pages` workflow that redeployed on changes there. No such trigger
existed: `pages.yml` watches `benchmarks/site/public/**` and nothing else.

The export reads the same artifacts `report.sh` does and runs the same
win-matrix and Bradley-Terry awk, so the site and the report cannot drift
apart. It carries only publishable fields — scores, costs, model and condition
names, judge tallies, task ids and difficulty — and fails rather than emit a
prompt, a plan, a judge's reasoning or an evidence quote.
`benchmarks/tests/test-site-export.sh` holds it to that against a fixture
batch, and `docs/README.md` covers local preview and the rest of the site's
layout.
apart. It carries only scores, costs, model and condition names, judge tallies,
task ids and difficulty, and fails rather than emit a prompt, a plan, a judge's
reasoning or an evidence quote. `benchmarks/tests/test-site-export.sh` holds it
to that against a fixture batch. That redaction is what makes the export safe to
share internally; it does not make a self-judged win rate safe to publish.

Pass `--batch-dir` when a batch's results live outside `benchmarks/results/`,
for instance in another checkout:
Expand Down
7 changes: 6 additions & 1 deletion benchmarks/export-site-data.sh
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,12 @@ command -v jq >/dev/null 2>&1 || die "jq is required for export-site-data.sh"
BATCH_ID=""
BATCH_DIR=""
RESULTS_ROOT="$SCRIPT_DIR/results"
DOCS_DIR="$REPO_ROOT/docs"
# The plan-composition suite scores Co-Evolution against itself: every condition
# is composed by Fable and the primary judge is Fable. That is a legitimate
# internal signal and not a publishable result, so the export lands in the
# ignored results tree rather than in a directory staged for the public site.
# Point --docs-dir somewhere else deliberately if you have a reason to.
DOCS_DIR="${CODE_BENCH_SITE_EXPORT_DIR:-$REPO_ROOT/benchmarks/results/site-export}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the obsolete docs-site runbook

After changing the default export location and deleting the docs-site HTML/assets, docs/README.md still claims that site is live, lists the deleted files, and directs users to run this exporter, commit docs/data/*, and preview docs/index.html. With the inspected .github/workflows/pages.yml serving only benchmarks/site/public/, following that retained runbook now previews a broken directory and produces no files to commit; update or remove it as part of retiring the site.

AGENTS.md reference: AGENTS.md:L108-L110

Useful? React with 👍 / 👎.

CORPUS_DIR="$SCRIPT_DIR/corpus"
CONDITIONS_FILE="$SCRIPT_DIR/conditions.yaml"
JUDGES_CSV="fable,codex,glm"
Expand Down
279 changes: 0 additions & 279 deletions docs/assets/site.css

This file was deleted.

Loading
Loading