M18: the front door states the findings, and cannot drift from them

The README said "That may turn out to be false. The benchmark is designed
to be able to say so." It had said so four times and the front door did not
mention it -- not the two refuted predictions, not the retracted result,
not the six flaws found by running the protocols, not the one mechanism
that works.

FINDINGS.md is generated from the same assembled record the HTML paper
renders. A second rendering rather than a second account, so the two cannot
disagree: neither contains a number the other had to be told about. CI
regenerates it on every push and fails on git diff --exit-code. Verified in
both directions -- stable across regeneration, and it catches a one-line
hand edit.

That closes the one place this project was asking to be taken on trust. A
repository whose thesis is take nobody's word for it cannot ask a reader to
take its README's word for it.

The design documents are annotated, not rewritten. docs/04-evaluation.md
staked out the headline prediction in advance -- B4 will capture most of
the gain over B3 -- and a reader had no way to learn it was refuted. It now
carries a note with the intervals and a pointer to the generated record.
The document itself is unchanged, on the same principle that keeps a
superseded protocol on disk: the design is a historical record, and editing
it to match the result would destroy what makes the result meaningful.
docs/00-README.md carries the same warning at the top of the set.

468 tests pass. ruff and mypy clean on 94 source files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: martex-dev

.github/workflows/ci.yml

@@ -221,7 +221,12 @@ jobs:
           enable-cache: true
       - run: uv sync --all-extras --dev
       - name: every protocol and every result, enumerated
-        run: uv run nullius paper build --out paper/index.html
+        run: uv run nullius paper build --out paper/index.html --markdown FINDINGS.md
+      - name: the committed findings are the generated ones
+        # The repository's front door states results. If it can drift from them
+        # it will, and a project whose thesis is "take nobody's word for it"
+        # cannot ask a reader to take its README's word for it.
+        run: git diff --exit-code -- FINDINGS.md
       - uses: actions/upload-artifact@v4
         with:
           name: paper

BUILD_PLAN.md

@@ -4,7 +4,7 @@
 
 This is the executable plan derived from [`docs/`](docs/). The design documents say *what* to build and *why*; this says *in what order*, *with what acceptance test*, and *what changes because of the machine we're actually on*.
 
-**Status:** M0–M17 complete (v5 and v6 ladders pending); M15's v5 ladder is running (mock-driven throughout; the first live run awaits an API key). M12's code-generation half is blocked on both a key and Docker. Nothing below is claimed as done until its acceptance criteria are green in CI.
+**Status:** M0–M18 complete (v5 and v6 ladders pending); M15's v5 ladder is running (mock-driven throughout; the first live run awaits an API key). M12's code-generation half is blocked on both a key and Docker. Nothing below is claimed as done until its acceptance criteria are green in CI.
 
 ---
 
@@ -466,6 +466,19 @@ It lands **under half the true value 8.9% of the time**. When it does, the escal
 
 All six protocols verify and rebuild identically. **v6 has not been run**: at roughly double B8's escalation it is a ~6-hour ladder, and the v5 run has to finish first.
 
+---
+
+### M18 · The front door states the findings, and cannot drift from them ✅
+The README said *"That may turn out to be false. The benchmark is designed to be able to say so."* It had said so, four times, and the front door did not mention it.
+
+**`FINDINGS.md` is generated**, by the same assembled record the HTML paper renders — a second rendering, not a second account, so the two cannot disagree. CI regenerates it on every push and fails on `git diff --exit-code`. The check was verified in both directions: stable across regeneration, and it catches a one-line hand edit.
+
+That closes the one place this project was asking to be taken on trust. A repository whose thesis is *take nobody's word for it* cannot ask a reader to take its README's word for it.
+
+**The design documents are annotated, not rewritten.** `docs/04-evaluation.md` staked out the headline prediction in advance — *B4 will capture most of the gain over B3* — and a reader had no way to learn it was refuted. It now carries a note saying so, with the intervals, and pointing at the generated record. The document itself is unchanged, on the same principle that keeps a superseded protocol on disk: the design is a historical record, and editing it to match the result would destroy the thing that makes the result meaningful.
+
+`docs/00-README.md` carries the same warning at the top of the set.
+
 **A wiring bug worth recording.** The first v4 ladder ran eight arms under a nine-arm protocol: a `ruff format` pass had collapsed the `run_ladder(...)` call onto one line before an edit meant to add `arms=` to it, so the replacement matched nothing and the runner silently used its eight-arm default. It produced a complete-looking results file — seven of seven baseline comparisons, no halted items — and nothing objected except the adjudication, which happened to name the missing arm by id. `score_ladder` now refuses any run whose arms do not match the protocol's, in either direction. The eight completed arms were reused from their checkpoints, so the correction cost one arm's compute rather than nine.
 
 ---

FINDINGS.md

@@ -0,0 +1,170 @@
+# What Nullius found
+
+*Generated from the committed protocols and results by `nullius paper build --markdown`.
+Do not edit by hand: CI regenerates this file and fails if it differs.*
+
+Every number below was produced under the **mock** provider.
+
+## The claim under test
+
+> Institutional structure - preregistration, adversarial challenge, independent replication, and evidence-typed memory - improves the accuracy and calibration of autonomous empirical research relative to an unstructured agent, at a measurable cost in compute and tokens.
+
+Across 6 registered protocols, **2
+predictions were refuted** and **2 upheld**.
+
+## Every registered protocol, in order
+
+| protocol | hash | arms | items | outcome |
+|---|---|---|---|---|
+| v1 | `1d4c76d2561e` | 8 | 20 | upheld |
+| v2 | `254be687163b` | 8 | 60 | refuted |
+| v3 | `9eb8e1e16e79` | 8 | 60 | refuted |
+| v4 | `b46bdef334c9` | 9 | 60 | upheld |
+| v5 | `6bfaa13661c6` | 9 | 60 | registered, not yet run |
+| v6 | `9acad59c27a4` | 10 | 60 | registered, not yet run |
+
+None was edited after registration. Where running one exposed a flaw in it, the fix is a new
+registration and the old protocol stays on disk, still verifying, still wrong in the way it
+was wrong.
+
+### Protocol v1 — upheld
+
+> **Registered prediction.** B4 captures most of the gain over B3 - that is, adding preregistration and the Custodian to a role-decomposed pipeline improves verdict accuracy more than adding the Skeptic, replication, review and memory do on top of it. If true, the finding is that cheap mechanisms beat expensive agents.
+
+**mechanism (B4-B3) = +0.0500; everything else (B6-B4) = -0.0500; prediction upheld**
+
+| arm | | n | acc | coverage | answered | brier | fdr | $/correct |
+|---|---|---|---|---|---|---|---|---|
+| B0 | Oracle-null | 1 | 0.45 | 1.00 | 0.45 | 0.250 | 0.00 | 0.00000 |
+| B1 \*  | Single-shot | 1 | 0.20 | 1.00 | 0.20 | 0.200 | 0.45 | 0.00269 |
+| B2 \*  | Single-agent + loop | 1 | 0.20 | 1.00 | 0.20 | 0.200 | 0.45 | 0.00761 |
+| B3 | Multi-role, no adversary | 1 | 0.90 | 1.00 | 0.90 | 0.283 | 0.00 | 0.00722 |
+| B4 | B3 + preregistration + custodian | 1 | 0.95 | 1.00 | 0.95 | 0.242 | 0.00 | 0.00683 |
+| B5 | B4 + Skeptic | 1 | 0.95 | 1.00 | 0.95 | 0.302 | 0.00 | 0.00683 |
+| B6 | Full institution | 1 | 0.90 | 1.00 | 0.90 | 0.316 | 0.00 | 0.00732 |
+| B7 | Full - memory | 1 | 0.95 | 1.00 | 0.95 | 0.311 | 0.00 | 0.00693 |
+Against the registered baseline `B1`, corrected with
+benjamini-hochberg at alpha 0.05 —
+5 of 7 survive:
+
+- `B0 − B1` = +0.250, 95% CI [-0.100, +0.550] *(model-dependent)*- `B2 − B1` = +0.000, 95% CI [+0.000, +0.000] *(model-dependent)*- `B3 − B1` = +0.700, 95% CI [+0.450, +0.900] *(model-dependent)*- `B4 − B1` = +0.750, 95% CI [+0.550, +0.900] *(model-dependent)*- `B5 − B1` = +0.750, 95% CI [+0.550, +0.900] *(model-dependent)*- `B6 − B1` = +0.700, 95% CI [+0.450, +0.900] *(model-dependent)*- `B7 − B1` = +0.750, 95% CI [+0.500, +0.950] *(model-dependent)*
+\* behaviour dominated by the language model. Under a mock provider these arms describe the
+mock and not a model, and no claim about mechanism rests on them.### Protocol v2 — refuted
+
+> **Registered prediction.** The mechanism contrast B4 - B3 is positive and its 95% interval excludes zero. Adding preregistration and the Custodian to a role-decomposed pipeline improves verdict accuracy by a margin this design can actually resolve. If it holds, cheap mechanism beats expensive agents; if the interval spans zero, the prediction fails regardless of the point estimate.
+
+**mechanism (B4-B3) = +0.0667, 95% CI [-0.0333, +0.1667]; interval does not exclude zero; prediction refuted**
+
+| arm | | n | acc | coverage | answered | brier | fdr | $/correct |
+|---|---|---|---|---|---|---|---|---|
+| B0 | Oracle-null | 1 | 0.45 | 1.00 | 0.45 | — | 0.00 | 0.00000 |
+| B1 \*  | Single-shot | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00293 |
+| B2 \*  | Single-agent + loop | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00830 |
+| B3 | Multi-role, no adversary | 1 | 0.60 | 1.00 | 0.60 | 0.203 | 0.00 | 0.01081 |
+| B4 | B3 + preregistration + custodian | 1 | 0.67 | 1.00 | 0.67 | 0.164 | 0.00 | 0.00973 |
+| B5 | B4 + Skeptic | 1 | 0.62 | 1.00 | 0.62 | 0.151 | 0.00 | 0.01052 |
+| B6 | Full institution | 1 | 0.72 | 1.00 | 0.72 | 0.110 | 0.00 | 0.00919 |
+| B7 | Full - memory | 1 | 0.70 | 1.00 | 0.70 | 0.101 | 0.00 | 0.00940 |
+Against the registered baseline `B0`, corrected with
+benjamini-hochberg at alpha 0.05 —
+4 of 7 survive:
+
+- `B1 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B2 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B3 − B0` = +0.150, 95% CI [-0.067, +0.350]- `B4 − B0` = +0.217, 95% CI [+0.000, +0.417]- `B5 − B0` = +0.167, 95% CI [-0.033, +0.383]- `B6 − B0` = +0.267, 95% CI [+0.050, +0.450]- `B7 − B0` = +0.250, 95% CI [+0.050, +0.467]
+\* behaviour dominated by the language model. Under a mock provider these arms describe the
+mock and not a model, and no claim about mechanism rests on them.### Protocol v3 — refuted
+
+> **Registered prediction.** Separating abstention from finding lowers every arm's verdict accuracy, because v2 credited an arm that could say nothing with having said the right thing whenever the truth happened to be 'inconclusive'. The institutional arms will separate on coverage: B6 answers more of the bank than B3 does, and the interval on that difference excludes zero. If coverage does not separate, the institution's advantage is in what it says and not in how much it is able to say.
+
+**mechanism (B4-B3) = +0.0333, 95% CI [-0.0500, +0.1167]; interval does not exclude zero; prediction refuted**
+
+| arm | | n | acc | coverage | answered | brier | fdr | $/correct |
+|---|---|---|---|---|---|---|---|---|
+| B0 | Oracle-null | 1 | 0.45 | 1.00 | 0.45 | — | 0.00 | 0.00000 |
+| B1 \*  | Single-shot | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00293 |
+| B2 \*  | Single-agent + loop | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00830 |
+| B3 | Multi-role, no adversary | 1 | 0.48 | 0.75 | 0.64 | 0.203 | 0.00 | 0.01341 |
+| B4 | B3 + preregistration + custodian | 1 | 0.52 | 0.77 | 0.67 | 0.142 | 0.00 | 0.01255 |
+| B5 | B4 + Skeptic | 1 | 0.55 | 0.75 | 0.73 | 0.132 | 0.00 | 0.01179 |
+| B6 | Full institution | 1 | 0.57 | 0.78 | 0.72 | 0.117 | 0.00 | 0.01161 |
+| B7 | Full - memory | 1 | 0.57 | 0.73 | 0.77 | 0.118 | 0.00 | 0.01161 |
+Against the registered baseline `B0`, corrected with
+benjamini-hochberg at alpha 0.05 —
+2 of 7 survive:
+
+- `B1 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B2 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B3 − B0` = +0.033, 95% CI [-0.167, +0.233]- `B4 − B0` = +0.067, 95% CI [-0.117, +0.250]- `B5 − B0` = +0.100, 95% CI [-0.100, +0.300]- `B6 − B0` = +0.117, 95% CI [-0.083, +0.317]- `B7 − B0` = +0.117, 95% CI [-0.067, +0.300]
+\* behaviour dominated by the language model. Under a mock provider these arms describe the
+mock and not a model, and no claim about mechanism rests on them.### Protocol v4 — upheld
+
+> **Registered prediction.** Adaptive seeding raises coverage. B8 abstains on fewer bank items than B6 does, and the 95% interval on that difference excludes zero.
+
+**coverage (B8-B6) = +0.1833, 95% CI [+0.0833, +0.3000]; interval excludes zero; prediction upheld**
+
+| arm | | n | acc | coverage | answered | brier | fdr | $/correct |
+|---|---|---|---|---|---|---|---|---|
+| B0 | Oracle-null | 1 | 0.45 | 1.00 | 0.45 | — | 0.00 | 0.00000 |
+| B1 \*  | Single-shot | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00293 |
+| B2 \*  | Single-agent + loop | 1 | 0.18 | 1.00 | 0.18 | 0.197 | 0.45 | 0.00830 |
+| B3 | Multi-role, no adversary | 1 | 0.48 | 0.75 | 0.64 | 0.203 | 0.00 | 0.01341 |
+| B4 | B3 + preregistration + custodian | 1 | 0.62 | 0.80 | 0.77 | 0.155 | 0.00 | 0.01051 |
+| B5 | B4 + Skeptic | 1 | 0.62 | 0.83 | 0.74 | 0.150 | 0.00 | 0.01051 |
+| B6 | Full institution | 1 | 0.55 | 0.75 | 0.73 | 0.101 | 0.00 | 0.01196 |
+| B7 | Full - memory | 1 | 0.55 | 0.75 | 0.73 | 0.118 | 0.00 | 0.01196 |
+| B8 | Full + adaptive seeding | 1 | 0.73 | 0.93 | 0.79 | 0.088 | 0.00 | 0.00944 |
+Against the registered baseline `B0`, corrected with
+benjamini-hochberg at alpha 0.05 —
+3 of 8 survive:
+
+- `B1 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B2 − B0` = -0.267, 95% CI [-0.450, -0.067] *(model-dependent)*- `B3 − B0` = +0.033, 95% CI [-0.167, +0.233]- `B4 − B0` = +0.167, 95% CI [-0.017, +0.350]- `B5 − B0` = +0.167, 95% CI [-0.017, +0.367]- `B6 − B0` = +0.100, 95% CI [-0.100, +0.283]- `B7 − B0` = +0.100, 95% CI [-0.083, +0.283]- `B8 − B0` = +0.283, 95% CI [+0.083, +0.483]
+\* behaviour dominated by the language model. Under a mock provider these arms describe the
+mock and not a model, and no claim about mechanism rests on them.### Protocol v5 — registered, not yet run
+
+> **Registered prediction.** Replication narrows the ladder rather than reordering it. Averaging three custody draws per arm leaves B8 - B6 on coverage positive with an interval still excluding zero, and leaves B4 - B3 on verdict accuracy spanning zero - the contrast that flipped between the v3 and v4 single draws. If B4 - B3 separates under replication, the v4 reading was right and this protocol's caution was wrong.
+
+Registered, not yet run.### Protocol v6 — registered, not yet run
+
+> **Registered prediction.** Sizing the escalation from an upper bound on the noise rather than a point estimate raises coverage. B9 abstains on fewer bank items than B8 does, and the 95% interval on that difference excludes zero. It should also cost more per item, because a bound that errs towards more data buys more data; if cost per correct claim rises without coverage improving, the bound is only expensive.
+
+Registered, not yet run.
+## The question bank
+
+Ground truth is planted, not judged. The oracle measures each item's true effect at forty
+seeds of twenty thousand samples and resolves it to about 0.0008; an experiment gets five
+seeds of two thousand and resolves it to about 0.00348. An item can therefore sit
+inside one *experiment* standard error of a verdict boundary while staying several *oracle*
+standard errors clear of it — hard to answer, and not in doubt.
+
+| bank | items | null | within 1 SE | within 2 SE | metric resolution |
+|---|---|---|---|---|---|
+| v1 | 20 | 45% | 3 | 6 | 0.050 |
+| v2 | 60 | 45% | 20 | 37 | 0.017 |
+
+## What running a protocol found wrong with it
+
+Each was discovered by executing a preregistered plan rather than by reviewing one. This is
+the section a written-up-afterwards paper would not have, because in that genre the flaws are
+fixed before anything is published.
+
+1. **The baseline arm was model-dependent.** v1 registered B1, a single-shot agent, as the arm everything was compared against. Under a mock provider B1's behaviour is a property of the mock, so every comparison in the registered family was uninterpretable as evidence about mechanism. v2 moved the baseline to B0, which answers without looking and cannot depend on a model at all. (M12b)
+2. **The prediction was adjudicated on two point estimates.** v1's rule compared B4 minus B3 against B6 minus B4 and returned 'upheld' for a one-item difference on a twenty-item bank, where one item is 0.05. v2 required the interval to exclude zero, and the same data then refuted the prediction v1 had upheld. (M12b)
+3. **Calibration was scored on a quantity the rubric does not measure.** The confidence rubric measures evidence for an effect, so a correct 'no effect' answer necessarily carries weak evidence and was scored as gross underconfidence. v2 restricted Brier and calibration error to items where the arm asserted an effect, which is the subpopulation where the rubric's quantity and the scored outcome are the same quantity. (M12b)
+4. **Abstention was scored as an answer, and sometimes as a correct one.** One verdict value meant both 'the effect is real and smaller than claimed' and 'the interval is too wide to say anything'. Because the first is a real truth value in this bank, an arm that could say nothing was credited with a correct answer whenever the truth happened to be that value. Every arm's accuracy was inflated, unevenly, by four to nine items in sixty. v3 split the verdict; 'underpowered' is never a truth, so an abstention can no longer be scored correct by accident. (M13)
+5. **A prediction and its adjudication rule described different quantities.** v3 registered a prediction about coverage and inherited a rule that tested accuracy, so the run reported a verdict after measuring something the prediction did not mention. It was right by accident. v4 stores the adjudicated contrast as data — treatment, baseline, quantity, direction — and derives the verdict from it, so the two cannot be edited apart. (M13b)
+6. **A single custody draw cannot support the contrasts being measured.** Arms B0 to B7 ran twice, under v3 and again under v4. The four uncustodied arms returned identical results to three decimals; every custodied arm moved, by up to 0.100 — six times the metric's resolution — because the Custodian derives its evaluation seed from the registration id and draws a fresh holdout each run. One contrast, B4 minus B3, flipped from spanning zero to excluding it on the same bank. v5 replicates every custodied arm three times. (M14b)
+
+## Limitations
+
+- Every result was produced under a mock provider. The institution's machinery — the compiler, the sandbox, the Custodian, the statistics, the confidence rubric — is real and so are the verdicts, but the prose each role emits is canned. Arms B1 and B2 are dominated by that prose and are reported as describing the mock.- The bank is sixty synthetic items from one data generating process. The population these results generalise to is 'questions like these', which is the only population sixty items of one family can speak for.- Cost is measured in real token counts priced as if a named model had produced them, because the mock is free and a cost-per-correct-claim whose numerator is identically zero ranks nothing. Compute cost is not substituted; those seconds were burned.- The comparison holds the science fixed and varies the mechanism. It therefore measures what each mechanism buys given a fixed research design, and not how a mechanism might change the design an institution chooses in the first place.- No result here has been replicated across independent implementations. The replication reported is of runs, not of the system.
+## Provenance
+
+| protocol | hash | bank items | truth lock |
+|---|---|---|---|
+| v1 | `1d4c76d2561e61e3c77998a1` | `c4d90bb633190a86` | `4c2ac5de66e0b751` |
+| v2 | `254be687163bf805ff9573f9` | `d4d1766b0f87c89d` | `8b459a6cc67b41ae` |
+| v3 | `9eb8e1e16e793b64250c5607` | `d4d1766b0f87c89d` | `8b459a6cc67b41ae` |
+| v4 | `b46bdef334c9e6d4f298388a` | `d4d1766b0f87c89d` | `8b459a6cc67b41ae` |
+| v5 | `6bfaa13661c63f3d5aca3c33` | `d4d1766b0f87c89d` | `8b459a6cc67b41ae` |
+| v6 | `9acad59c27a47affc0354911` | `d4d1766b0f87c89d` | `8b459a6cc67b41ae` |
+
+Every figure above is read from a results file whose stored summary re-scores from its own
+per-item rows, and every prediction from a protocol whose hash is in the git history. The
+generator refuses to run if any of that stops checking out.