Add reproducible validator studies with MateProbe compatibility replay

.github/workflows/ci.yml

@@ -32,6 +32,8 @@ jobs:
           cmp benchmarks/real-mutation-results/evaluation/summary.json /tmp/real-mutations/summary.json
           python scripts/compare_replay.py benchmarks/real-mutation-results/evaluation/cases.json /tmp/real-mutations/cases.json --old-version 0.1.0a2 --new-version 0.1.0a4 --report-path result.report
           python examples/five_minute_demo.py
+      - name: Replay immutable validator studies with MateProbe
+        run: python benchmarks/study_replay.py --output /tmp/mateprobe-study-replay
       - name: Audit existing validators offline
         run: python examples/audit_existing_validator.py --output /tmp/validator-audit
       - run: python -m build && python -m build packages/pytest-mateprobe

.github/workflows/docs.yml

@@ -28,13 +28,15 @@ jobs:
           -r requirements-integrations.txt
       - run: python scripts/check_alpha_install.py --output /tmp/published-docs-evidence
       - run: python scripts/check_launch_package.py --output /tmp/launch-evidence
+      - run: python benchmarks/study_replay.py --output /tmp/published-study-replay
       - uses: actions/upload-artifact@v4
         if: always()
         with:
           name: published-docs-evidence
           path: |
             /tmp/published-docs-evidence
             /tmp/launch-evidence
+            /tmp/published-study-replay
 
   historical-a3-trial:
     runs-on: ubuntu-latest
@@ -46,11 +48,27 @@ jobs:
       - name: Install the historical a3 trial dependencies
         run: python -m pip install narrative-contracts==0.1.0a3 pytest-narrative-contracts==0.1.0a3
       - run: python benchmarks/replay_agent_readiness.py --output /tmp/a3-agent-readiness
+      - name: Replay frozen a3 studies byte-for-byte
+        run: |
+          python benchmarks/validator_study.py replay --prepared benchmarks/validator-study/prepared --captured benchmarks/validator-study/evaluation --output /tmp/a3-controlled
+          python benchmarks/historical_validator_study.py replay --captured benchmarks/historical-validator-results/captured --output /tmp/a3-historical
+          python benchmarks/historical_followup.py replay --captured benchmarks/historical-validator-results/compatibility-followup-v2/captured --output /tmp/a3-followup
+          for file in summary.json audit-reports.json comparisons.json observations.json; do
+            cmp "benchmarks/validator-study/evaluation/$file" "/tmp/a3-controlled/$file"
+          done
+          for file in summary.json audit-reports.json; do
+            cmp "benchmarks/historical-validator-results/evaluation/$file" "/tmp/a3-historical/$file"
+            cmp "benchmarks/historical-validator-results/compatibility-followup-v2/evaluation/$file" "/tmp/a3-followup/$file"
+          done
       - uses: actions/upload-artifact@v4
         if: always()
         with:
-          name: historical-a3-agent-readiness
-          path: /tmp/a3-agent-readiness
+          name: historical-a3-agent-readiness-and-studies
+          path: |
+            /tmp/a3-agent-readiness
+            /tmp/a3-controlled
+            /tmp/a3-historical
+            /tmp/a3-followup
 
   historical-a2-trial:
     runs-on: ubuntu-latest

AGENTS.md

@@ -21,6 +21,14 @@ and `pydantic==2.13.5`, then run `python scripts/check_published_docs.py`;
 an editable checkout cannot verify published compatibility. See
 [the agent integration guide](docs/agent-guide.md) for consumer instructions.
 
+## Project identity
+
+The current project is **MateProbe by Mate4B** (`mateprobe`, `pytest-mateprobe`, a4).
+Frozen a3 study drivers, protocols, captures and hashes retain their original names
+and source bytes. Use `benchmarks/study_replay.py` to replay those studies against
+MateProbe; direct historical commands require an isolated a3 environment.
+See docs/project-name.md and docs/validator-study.md.
+
 ## Version boundary
 
 The current release is `0.1.0a4`, including independent audits and state bindings.

README.md

@@ -244,6 +244,15 @@ trusted Python state transitions, checks branch selection and declarations, and
 valid paraphrases and failures that remain outside the prose guarantee. It is independent of
 LifeCard; this is an executable second integration, not evidence of broad domain generalization.
 
+## Controlled and historical validator evidence
+
+The [a3 validator study](docs/validator-study.md) compares 144 paired cases under eight
+authored profiles. Detailed Python assertions and the audit agree on all 1,152
+classifications; boolean rejection alone loses diagnostic attribution. A separate
+version-pinned study reproduces three public historical fixes across jsonschema and
+Marshmallow, retaining a fourth unsuccessful reproduction and all compatibility
+failures. This is author-run evidence, not external adoption or natural-text accuracy.
+
 ## Reproduce or challenge the results
 
 ```sh

benchmarks/historical-validator-results/README.md

@@ -0,0 +1,29 @@
+# Historical evidence: read the final and initial results together
+
+The final compatibility run is `compatibility-followup-v2/evaluation/summary.json`:
+three reproduced historical fixes across two packages and one non-reproduction.
+There are four issue comparisons, eight changed pairs, and two versions per issue.
+These are independently originated reports reproduced by this project's authors,
+not independent users of Narrative Contracts or newly discovered defects.
+
+- `preflight-v1/`: flawed initial preparation preserved for transparency; not results.
+- `prepared-v2-bootstrap/`: preparation before the macOS venv bootstrap correction.
+- `prepared/`: original executed inputs, adapter and protocol snapshots.
+- `captured/` and `evaluation/`: original run, with one reproduced fix, unavailable
+  jsonschema 4.17.2, missing pkg_resources in old versions and the unchanged IDN case.
+- `compatibility-followup/`: first compatibility run; a conflicting setuptools pin
+  prevented two environments from installing. Other results and the driver are retained.
+- `compatibility-followup-v2/`: corrected, pinned dependency environments, raw
+  outcomes, artifact hashes, locks, and final replay results. Case labels and inputs
+  were never changed to improve results after the original package execution.
+
+The initial screening ledger is purposive, with ten candidates and six exclusions.
+Null versions on excluded entries mean a validation-fix pair was not established.
+Preflight-v1 guesses must not be used as established release information.
+
+See [the study](../../docs/validator-study.md), [screening protocol](../../paper/historical-validator-protocol.md)
+and [follow-up deviations](../../paper/historical-followup-protocol.md). Reproduction
+commands distinguish offline replay from package downloads and fresh execution.
+No package installations or inference calls are needed for CI replay. The original
+and final runs are checked byte-for-byte by the test suite; the intermediate
+compatibility run is an archived failed setup attempt, not the recommended entrypoint.