feat: define one public benchmark release contract

.agents/skills/wp-bench-execution-tests/SKILL.md

@@ -151,7 +151,7 @@ The default `run.execution_isolation: reset_per_test` gives every candidate a fr
 
 ## WP-CLI Workflow Tests
 
-Use `wp-cli-v1` for realistic operational tasks that combine discovery, selection,
+Use `wp-cli` for realistic operational tasks that combine discovery, selection,
 changes, reports, and preservation. Prioritize model capability over testing tiny
 WordPress units. This suite targets single-site WordPress 7.1 with SQLite;
 avoid unsupported MySQL operations, network installs, or multisite instead of

.github/workflows/ci.yml

@@ -18,20 +18,27 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
 
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
           cache: pip
 
       - name: Install harness with dev tooling
-        run: pip install -e ./python[dev]
+        run: pip install -e ./python[dev,leaderboard]
 
       - name: Lint (ruff)
-        run: ruff check python
+        run: ruff check python web
 
       - name: Type check (mypy)
-        run: mypy python
+        run: mypy python web
+
+      - name: Benchmark release integrity
+        env:
+          BASE_REF: ${{ github.event.pull_request.base.sha || github.event.before }}
+        run: wp-bench benchmark check --base-ref "$BASE_REF"
 
       - name: Unit tests (pytest)
         run: pytest python -q
@@ -97,12 +104,12 @@ jobs:
 
       - name: WP-CLI workflow reference solutions
         run: >-
-          wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1
+          wp-bench run --config wp-bench.example.yaml --suite wp-cli
           --check-reference-solution --execution-concurrency 4
 
       - name: WP-CLI workflow exploit audit
         run: >-
-          wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1
+          wp-bench run --config wp-bench.example.yaml --suite wp-cli
           --check-exploits --execution-concurrency 4
 
       - name: SQLite database reference solutions

AGENTS.md

@@ -36,3 +36,9 @@ This guide is the fastest path to contributing to WP-Bench. Keep changes minimal
 - Follow Conventional Commits (`feat:`, `refactor:`, etc.) as seen in git history. Scope prefixes are optional.
 - PRs should state intent, key commands run (`ruff`, `mypy`, `pytest`, runtime smoke test), and config changes. Link related issues. Include logs or screenshots for runtime/verifier output when relevant.
 - Keep changes small and focused; update README/AGENTS.md when workflows or flags change. Avoid touching `legacy/` unless explicitly required.
+
+## Benchmark Releases
+- Public versions (`v1`, `v2`, …) identify the entire evaluation; `wp-core` and `wp-cli` are stable component IDs, never independent public versions.
+- Read `benchmark/README.md` before changing the release protocol, category mapping, result provenance, or website export contract.
+- Published `benchmark/releases/v*.json` files are immutable. Batch evaluation changes during development, then freeze the next release. Adding model results and changing presentation do not create benchmark versions.
+- Run `wp-bench benchmark check` and `pytest python/tests/test_benchmark_releases.py -q` after changes to versioning. Dataset/runtime/harness changes intentionally produce unreleased results until a new release is frozen.

README.md

@@ -2,6 +2,19 @@
 
 The official WordPress AI benchmark. Evaluate how well language models understand WordPress development—from core APIs and coding standards to plugin architecture and security best practices.
 
+## Public benchmark version
+
+One public version identifies the complete evaluation: **WP-Bench v1**. The
+`wp-core` and `wp-cli` datasets are stable components of that release. Changes to
+any dataset or score-affecting rules require the next benchmark version; adding
+model results updates the same version. Release manifests, run provenance and
+validated website exports are documented in [benchmark/README.md](benchmark/README.md).
+
+```sh
+wp-bench benchmark show --version v1
+wp-bench benchmark check
+```
+
 ## Overview
 
 WP-Bench measures AI model capabilities on **execution**: code generation tasks graded by static checks and runtime assertions in a real WordPress environment.
@@ -148,7 +161,7 @@ Copy `wp-bench.example.yaml` and customize:
 ```yaml
 dataset:
   source: local              # 'local' or 'huggingface'
-  name: wp-core-v1           # suite name
+  name: wp-core           # suite name
 
 models:
   - name: gpt-4o
@@ -160,7 +173,7 @@ grader:
   setup_timeout_seconds: 600 # hard cap for environment setup
 
 run:
-  suite: wp-core-v1
+  suite: wp-core
   limit: 10                  # limit tests (null = all); seeded stratified selection
   seed: 1337                 # selection seed (same seed = same subset)
   test_ids: []               # optional explicit test IDs to run
@@ -260,12 +273,12 @@ Test suites live in `datasets/suites/<suite-name>/`:
 
 - `execution/` — Code generation tasks with assertions (one JSON file per category)
 
-The default suite `wp-core-v1` contains 350 tasks covering WordPress core APIs,
+The default suite `wp-core` contains 350 tasks covering WordPress core APIs,
 hooks, database operations, and security patterns.
 
 ### WP-CLI workflows
 
-`wp-cli-v1` adds 27 operational tasks across content, taxonomy and metadata,
+`wp-cli` adds 27 operational tasks across content, taxonomy and metadata,
 users and capabilities, options and transients, reporting, maintenance, and migration.
 Tasks combine selection, changes, and preservation: examples include retiring
 an author while reassigning content, consolidating categories, repairing a
@@ -286,10 +299,10 @@ its research basis and source links in metadata. Prompts describe the desired
 WordPress outcome, and equivalent native command workflows are accepted.
 
 ```bash
-wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --execution-concurrency 4
+wp-bench run --config wp-bench.example.yaml --suite wp-cli --execution-concurrency 4
 # Maintainer validation without model calls:
-wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --check-reference-solution
-wp-bench run --config wp-bench.example.yaml --suite wp-cli-v1 --check-exploits
+wp-bench run --config wp-bench.example.yaml --suite wp-cli --check-reference-solution
+wp-bench run --config wp-bench.example.yaml --suite wp-cli --check-exploits
 ```
 
 The model returns a **plain command or Bash script string**, optionally inside
@@ -324,7 +337,7 @@ and repeat runs under `grader.raw.command`. Each output stream is capped at
 output is still recorded. Private container limits also apply, and timeouts
 or output overflow score zero.
 
-The WP-CLI suite is scored separately from `wp-core-v1`; existing PHP task
+The WP-CLI suite is scored separately from `wp-core`; existing PHP task
 contracts and scoring remain unchanged. Compare scores only within the same
 suite and execution profile.
 
@@ -378,11 +391,11 @@ tasks retain specialized assertions for their distinct contracts.
 dataset:
   source: huggingface
   name: WordPress/wp-bench-v1
-  suite: wp-core-v1       # filter a multi-suite export; defaults to run.suite
+  suite: wp-core       # filter a multi-suite export; defaults to run.suite
 ```
 
 An explicit `dataset.suite` selects the rows and their result-suite label;
-`--suite` overrides both. `--suite wp-cli-v1` selects CLI rows without changing
+`--suite` overrides both. `--suite wp-cli` selects CLI rows without changing
 the Hugging Face repository name. The new suite is available locally immediately;
 Hub users need an export that includes it.
 

benchmark/README.md

@@ -0,0 +1,156 @@
+# Public benchmark releases
+
+**WP-Bench v1** identifies one complete, frozen evaluation. `wp-core` and `wp-cli`
+are stable component IDs. They have no independent public versions. The Python
+package, result JSON schema, and scoring formula have internal version numbers;
+those are reproducibility details, not additional leaderboard versions.
+
+`definition.json` is the working evaluation recipe. `releases.json` indexes the
+append-only `releases/vN.json` manifests. A release freezes every dataset file,
+complete test membership, runtime and relevant harness source, WordPress target,
+scoring rules, broad category mapping, approved runtime images, generation
+settings, and required repeats. The manifest fingerprint covers the entire
+recipe. Changing any dataset changes the overall evaluation fingerprint.
+
+| Change | Release handling |
+| --- | --- |
+| Add/remove/edit a dataset task, including a corrected assertion | Next benchmark version |
+| Change grading, task weighting, prompting, or the evaluation protocol | Next benchmark version |
+| Change runtime behavior or the approved runtime images | Next benchmark version |
+| Evaluate another model on the same recipe | Updated results under the same version |
+| Re-run the same models under that recipe | Another results snapshot under the same version |
+| Edit documentation, website layout, or result serialization | Same benchmark version |
+
+Source fingerprinting is deliberately conservative. Even a dataset formatting or
+editorial-only edit changes its frozen file hash. Batch source changes during
+development rather than publishing a version for every commit. A changed checkout
+can run normally, but records `status: unreleased` and no public version. A new
+release is an explicit maintainer action; it is never inferred from directory
+names, a date, or a model list.
+
+## Running a release
+
+Inspect the complete protocol before spending on providers:
+
+```sh
+wp-bench benchmark show --version v1
+wp-bench benchmark check
+```
+
+An explicit assertion in the run config refuses mismatching sources, loaded
+local/Hub tasks, model settings, or runtime settings before model generation:
+
+```yaml
+benchmark:
+  version: v1
+
+dataset:
+  source: local
+  name: wp-core
+
+model:
+  name: YOUR_MODEL_ID
+  temperature: 1
+  max_tokens: 16384
+
+grader:
+  kind: docker
+  # Use one of runtime_image_ids from the frozen manifest, not an unverified tag.
+  image: sha256:6dcab391357137b2ce52d99ef15f342499b6b9f3287e5b96946036628a7dfb1e
+
+run:
+  suite: wp-core
+  execution_isolation: reset_per_test
+```
+
+Run both components and the required independent repeats. A subset remains
+useful for diagnostics but cannot enter the overall leaderboard. With Hugging
+Face, pin `dataset.revision` to the published export commit and select the stable
+suite ID. Loaded task fingerprints must match the frozen release regardless of
+transport. The historical Hub repository name `WordPress/wp-bench-v1` is an
+address, not a separate dataset-version authority; publish the new stable suite
+IDs there before using that export.
+
+Every completed payload carries `metadata.benchmark`; each streamed record also
+carries the benchmark identity, task fingerprint and `run_id`. Multi-model
+payloads retain a separate identity and run ID in each `models` entry. These
+fields survive partial runs, but publication requires actual complete records.
+A `full_suite` flag alone never authorizes a score. Source identity is captured
+before generation, so an edit during a run cannot retroactively relabel it.
+
+## Publishing a new benchmark version
+
+1. Batch intended evaluation changes in `definition.json`, datasets, and code.
+2. Run the normal Python/dataset checks and applicable reference/exploit audits.
+3. Freeze the next version, then inspect the generated manifest:
+
+   ```sh
+   wp-bench benchmark freeze v2
+   wp-bench benchmark check --base-ref origin/trunk
+   ```
+
+4. Commit the new manifest and catalog in the release PR. Existing manifests
+   cannot be edited, removed, or reassigned. CI compares their bytes with the base
+   commit, even if someone recomputes the embedded hashes.
+5. After merging, tag the release commit `benchmark-v2` and retain its approved
+   runtime images. To reproduce an old version, check out its tag; selecting an
+   old version alone does not restore its source into a changed checkout.
+
+If a changed file does not affect evaluation behavior, review that deliberately.
+The default fingerprint policy favors refusing an accidental comparison over
+silently declaring two evaluations equivalent. No file change should be hidden
+by modifying an old manifest.
+
+## Website data contract
+
+Use the harness exporter, rather than recreating release numbers in the website:
+
+```sh
+wp-bench benchmark export --version v1 --output-dir public-data \
+  --input core-model-a-repeat-1.json --input core-model-a-repeat-2.json \
+  --input cli-model-a-repeat-1.json --input cli-model-a-repeat-2.json
+```
+
+Supply the complete set of models desired in the new snapshot. Single- and
+multi-model result JSONs are supported. The exporter checks the release identity
+on both envelopes and individual records, task fingerprints, exact suite
+membership, approved runtime image, model/grader settings, baseline policy,
+unique run IDs, and the required independent repeats. Reference solutions,
+skills variants, errored/partial runs, duplicate repeats and mixed versions fail
+before output is created. A legitimate failed model answer is still a scored
+answer and remains in the denominator.
+
+The shared [website adapter](../web/README.md) turns these validated bundles
+into the Spacefast display data, including category scores and paired generation
+comparisons. Model names/pairs are editorial inputs; versions and categories
+come from the frozen manifest.
+
+The output is a small, stable ingestion contract, not a website theme:
+
+```text
+public-data/
+  results.json                # mutable index, grouped by benchmark version
+  v1/<snapshot-id>.json        # immutable scored-answer snapshots
+  v2/<snapshot-id>.json
+```
+
+Snapshot schema `1.0` contains `benchmark.version`, `benchmark.fingerprint`, the
+release-owned broad category labels, model IDs, per-answer pass/usage data, run
+IDs, input hashes, and an update date. It excludes generated code and raw provider
+responses. Every answer has a stable component ID, test ID and broad category.
+Aggregate pass rates weight every task equally and average the same number of
+repeats per task. Use task-paired uncertainty estimates when comparing models.
+
+The website should expose **one benchmark-version selector** and an **updated
+date**. Snapshot IDs and internal schemas belong in downloads/details. A model
+addition creates a new immutable snapshot under v1; changing any dataset creates
+v2 and its own leaderboard. Preserve v1 scores and files. Never fill missing v2
+rows with v1 scores. Cross-version charts must disclose test changes and cannot
+establish model improvement by themselves.
+
+The first release includes the already-audited four core assertion corrections
+and native read-only `wp db columns` allowance behind the initial public scores.
+Renaming component IDs and removing per-file versions changes no prompts or
+behavioral checks beyond those recorded corrections. Historical files without
+the new identity must be explicitly audited/imported for the first release;
+the exporter never guesses that an old `*-v1` suite name proves compatibility.