M15: replicate the ladder, and only the arms that can move

v4 measured why one pass is not enough. Arms B0-B7 ran twice, under v3 and
again under v4, and the custodied arms moved by up to 0.100 -- six times
the metric's 0.017 resolution -- because the Custodian derives its
evaluation seed from the registration id and draws a fresh holdout each
run. A contrast smaller than that cannot be read from a single draw, and
the contrasts this benchmark exists to measure are smaller than that.

Only custodied arms are replicated. `arm.custodian` predicted the drift
exactly: every arm that moved has it set, every arm that did not lacks it.
An uncustodied arm reads the development split, which is fixed by seeds
derived from the item id, so a second pass returns what the first did --
B0 through B3 came back identical to three decimals. Replicating a
deterministic arm reports a spread of zero as though it were evidence of
stability, and costs a full pass over the bank to do it.

Replicates are averaged within a bank item before arms are compared, so
the bootstrap keeps resampling items -- the population the bank can speak
for. Pooling them as independent observations would treat three looks at
one question as three questions and shrink every interval by a factor the
design has not earned. ArmRun.by_item groups them; _rate reduces them; the
paired bootstrap now takes floats instead of booleans so an item can carry
the rate at which an arm got it right.

Identifiers now come from a stream seeded by arm and replicate, which
makes a pass reproducible. Until now the ladder drew random UUIDs, so
registration ids -- and through them the Custodian's evaluation seeds --
differed on every run and no result could be reproduced at all. The
custody guarantee is unaffected: it is that a spec cannot be re-registered
until a flattering holdout appears, and the stream still issues a fresh id
per registration within a pass. What became reproducible is the pass,
which is what a benchmark is supposed to be.

Protocol v5 registers three passes per custodied arm, and predicts that
replication narrows the ladder rather than reordering it: B8-B6 on
coverage stays positive with an interval excluding zero, and B4-B3 on
accuracy keeps spanning zero -- the contrast that flipped between the v3
and v4 single draws. If B4-B3 separates under replication, the v4 reading
was right and this protocol's caution was wrong.

All five protocols verify and rebuild identically.

445 tests pass. ruff and mypy clean on 91 source files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: martex-dev

.github/workflows/ci.yml

@@ -163,7 +163,8 @@ jobs:
           for path in (Path('benchmark/results.lock.json'),
                        Path('benchmark/results.v2.lock.json'),
                        Path('benchmark/results.v3.lock.json'),
-                       Path('benchmark/results.v4.lock.json')):
+                       Path('benchmark/results.v4.lock.json'),
+                       Path('benchmark/results.v5.lock.json')):
               if not path.exists():
                   print(f'{path.name}: not committed yet'); continue
               report, runs = read_results(path)

benchmark/protocol.v5.lock.json

@@ -0,0 +1 @@
+{"protocol":{"arms":[{"adaptive_seeds":false,"adversary":false,"arm_id":"B0","custodian":false,"isolates":"the floor, and any imbalance in the bank","iterations":1,"kind":"constant","label":"Oracle-null","memory":false,"model_dependent":false,"preregistered":false,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":false,"arm_id":"B1","custodian":false,"isolates":"the naive baseline everyone actually ships","iterations":1,"kind":"direct","label":"Single-shot","memory":false,"model_dependent":true,"preregistered":false,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":false,"arm_id":"B2","custodian":false,"isolates":"whether iteration alone helps","iterations":3,"kind":"direct","label":"Single-agent + loop","memory":false,"model_dependent":true,"preregistered":false,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":false,"arm_id":"B3","custodian":false,"isolates":"whether role decomposition alone helps","iterations":1,"kind":"uncustodied","label":"Multi-role, no adversary","memory":false,"model_dependent":false,"preregistered":false,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":false,"arm_id":"B4","custodian":true,"isolates":"how much comes from mechanism rather than from agents","iterations":1,"kind":"institutional","label":"B3 + preregistration + custodian","memory":false,"model_dependent":false,"preregistered":true,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":true,"arm_id":"B5","custodian":true,"isolates":"the value of adversarial challenge","iterations":1,"kind":"institutional","label":"B4 + Skeptic","memory":false,"model_dependent":false,"preregistered":true,"replication":false,"reviewer":false},{"adaptive_seeds":false,"adversary":true,"arm_id":"B6","custodian":true,"isolates":"replication, review and memory on top of challenge","iterations":1,"kind":"institutional","label":"Full institution","memory":true,"model_dependent":false,"preregistered":true,"replication":true,"reviewer":true},{"adaptive_seeds":false,"adversary":true,"arm_id":"B7","custodian":true,"isolates":"institutional memory's own contribution","iterations":1,"kind":"institutional","label":"Full - memory","memory":false,"model_dependent":false,"preregistered":true,"replication":true,"reviewer":true},{"adaptive_seeds":true,"adversary":true,"arm_id":"B8","custodian":true,"isolates":"whether spending seeds where they matter beats spending them evenly","iterations":1,"kind":"institutional","label":"Full + adaptive seeding","memory":true,"model_dependent":false,"preregistered":true,"replication":true,"reviewer":true}],"bank":{"items_hash":"d4d1766b0f87c89d326068010926ee00819d5f738e20374119d588f629dacbf1","n_items":60,"truth_lock_hash":"8b459a6cc67b41ae146207362ea65983835a78c8cbf7687df2e93c0ef71051a6"},"claim":"Institutional structure - preregistration, adversarial challenge, independent replication, and evidence-typed memory - improves the accuracy and calibration of autonomous empirical research relative to an unstructured agent, at a measurable cost in compute and tokens.","confidence_as_probability":{"contested":0.3,"speculative":0.4,"suggestive":0.55,"supported":0.75,"well_supported":0.9},"exclusion_rules":["An item whose lifecycle halts before a verdict counts as incorrect for that arm rather than being dropped, because dropping it would reward an arm for failing to answer the questions it finds hard.","An arm whose behaviour is dominated by the language model is reported with its results labelled model-dependent, and is excluded from any claim about mechanism when the run used a mock provider.","Every arm's outcome is reported, including the ones that make the project look bad. There is no rule under which a result is withheld.","The primary comparison is against B0, the arm that answers without looking. B1 was v1's baseline and is model-dependent, which under a mock provider made every comparison in the registered family uninterpretable for mechanism. B1 and B2 are still reported; nothing is compared against them.","Brier score and calibration error are computed only over items where the arm asserted an effect. The confidence rubric measures evidence *for an effect*, so a correct 'no effect' answer necessarily carries weak evidence and scored as gross underconfidence in v1 - an artefact of the mapping rather than a property of the institution. Restricting to assertions is the subpopulation where the rubric's quantity and the scored outcome are the same quantity.","The registered prediction is adjudicated on an interval, not on a point estimate. v1's rule compared two point estimates and returned 'upheld' for a one-item difference on a twenty-item bank.","An abstention is reported, never dropped. 'underpowered' still counts as incorrect for verdict accuracy, because the first exclusion rule refuses to reward an arm for declining the questions it found hardest. It is also counted separately, because a system that knows what it cannot measure is not the same as one that guesses.","Coverage and assertion accuracy are reported together and neither is the primary metric. An arm can drive assertion accuracy to 1.0 by answering only what it is sure of, and coverage is what stops that reading as a good result.","The adjudicated quantity is named in the protocol and the verdict is computed from it. A protocol whose prediction and whose adjudication rule describe different quantities has not registered anything, however precise either one is on its own.","Adaptive seeding may only spend seeds the registration already declared. The full seed set is derived from seed_root at registration and has length max_seeds; escalation chooses how far down that list to go and never which seeds are on it.","The escalation decision reads the development split only. Deciding how much more data to collect by looking at the quantity the verdict will be computed from is optional stopping, and would buy significance rather than resolution.","Only custodied arms are replicated. An uncustodied arm reads the development split, which is fixed by seeds derived from the item id, and returns identical results however often it runs - measured, not assumed: running the ladder twice left B0 through B3 identical to three decimals while every custodied arm moved. Replicating a deterministic arm reports a spread of zero as though it were evidence of stability.","Replicates are averaged within a bank item before arms are compared, so the bootstrap continues to resample items - the population the bank can speak for. Pooling replicates as independent observations would treat three looks at one question as three questions and shrink every interval by a factor the design has not earned."],"metrics":["verdict_accuracy","null_accuracy","brier","expected_calibration_error","false_discovery_rate","usd_per_correct_claim","effect_size_error","coverage","assertion_accuracy"],"prediction":"Replication narrows the ladder rather than reordering it. Averaging three custody draws per arm leaves B8 - B6 on coverage positive with an interval still excluding zero, and leaves B4 - B3 on verdict accuracy spanning zero - the contrast that flipped between the v3 and v4 single draws. If B4 - B3 separates under replication, the v4 reading was right and this protocol's caution was wrong.","primary_metric":"verdict_accuracy","registered_at":"2026-09-01","statistics":{"adaptive_seed_ceiling":24,"adjudicated":{"baseline":"B6","direction":"greater","quantity":"coverage","treatment":"B8"},"adjudication":"named_contrast","alpha":0.05,"baseline_arm":"B0","calibration_scope":"asserted_effects","confidence_order":["contested","speculative","suggestive","supported","well_supported"],"interval":"percentile bootstrap over items","multiplicity":"benjamini-hochberg","pairing":"paired over bank items","replicated_arms":"custodied","replicates":3,"resamples":2000,"verdict_vocabulary":["supported","refuted","no_effect","conditional","inconclusive","underpowered"]},"version":"5"},"protocol_hash":"6bfaa13661c63f3d5aca3c33462998c5807e39cb70cdfd6355f2c3a398eca48c"}

src/nullius/benchmark/metrics.py

@@ -149,6 +149,14 @@ class ArmMetrics:
     usd_per_correct_claim: float
     effect_size_error: float
 
+    n_replicates: int = 1
+    """Passes over the bank these numbers average.
+
+    Only custodied arms are replicated; the rest return identical results
+    however often they run, which the v3-against-v4 comparison measured rather
+    than assumed.
+    """
+
     n_scored: int = 0
     """Items the calibration metrics were computed over.
 
@@ -164,6 +172,7 @@ def as_dict(self) -> dict[str, Any]:
             "label": self.label,
             "model_dependent": self.model_dependent,
             "n_items": self.n_items,
+            "n_replicates": self.n_replicates,
             "n_scored": self.n_scored,
             "n_correct": self.n_correct,
             "n_halted": self.n_halted,
@@ -220,14 +229,15 @@ def score_arm(run: ArmRun, protocol: Protocol) -> ArmMetrics:
 
     answered = [o for o in outcomes if not o.abstained]
     return ArmMetrics(
+        n_replicates=run.n_replicates,
         n_scored=len(scored),
         coverage=(len(answered) / len(outcomes)) if outcomes else float("nan"),
         assertion_accuracy=_accuracy(answered),
         n_abstained=len(outcomes) - len(answered),
         arm_id=run.arm.arm_id,
         label=run.arm.label,
         model_dependent=run.arm.model_dependent,
-        n_items=len(outcomes),
+        n_items=len(run.by_item()),
         n_correct=n_correct,
         n_halted=sum(1 for o in outcomes if o.halted is not None),
         verdict_accuracy=_accuracy(outcomes),
@@ -294,8 +304,8 @@ def __str__(self) -> str:
 
 
 def _paired_bootstrap(
-    treatment: Sequence[bool],
-    baseline: Sequence[bool],
+    treatment: Sequence[bool] | Sequence[float],
+    baseline: Sequence[bool] | Sequence[float],
     *,
     resamples: int,
     alpha: float,
@@ -311,8 +321,10 @@ def _paired_bootstrap(
     correct. The protocol therefore registered ``percentile bootstrap over
     items``, and this is that.
     """
-    a = np.asarray([1.0 if x else 0.0 for x in treatment], dtype=np.float64)
-    b = np.asarray([1.0 if x else 0.0 for x in baseline], dtype=np.float64)
+    # Floats rather than booleans, because with replicates an item carries the
+    # *rate* at which an arm got it right rather than a single yes or no.
+    a = np.asarray([float(x) for x in treatment], dtype=np.float64)
+    b = np.asarray([float(x) for x in baseline], dtype=np.float64)
     if a.size != b.size:
         raise ValueError("a paired comparison needs both arms to answer the same items")
     if a.size == 0:
@@ -364,17 +376,17 @@ def compare_to_baseline(
             f"comparison"
         )
     base_run = by_id[baseline_id]
-    ordered = {o.item_id: o for o in base_run.outcomes}
+    ordered = base_run.by_item()
 
     comparisons: list[Comparison] = []
     for index, run in enumerate(runs):
         if run.arm.arm_id == baseline_id:
             continue
-        theirs = {o.item_id: o for o in run.outcomes}
+        theirs = run.by_item()
         shared = [item_id for item_id in ordered if item_id in theirs]
         difference, low, high, p_value = _paired_bootstrap(
-            [theirs[i].correct for i in shared],
-            [ordered[i].correct for i in shared],
+            [_rate(theirs[i], QUANTITIES["verdict_accuracy"]) for i in shared],
+            [_rate(ordered[i], QUANTITIES["verdict_accuracy"]) for i in shared],
             resamples=resamples,
             alpha=alpha,
             # Per-arm seeds, derived from position so the whole report is
@@ -546,6 +558,11 @@ def _adjudicate(
 }
 
 
+def _rate(outcomes: Sequence[ArmOutcome], read: Callable[[ArmOutcome], bool]) -> float:
+    """How often an arm got this item right, across its replicates."""
+    return sum(1.0 for o in outcomes if read(o)) / len(outcomes)
+
+
 def _contrast(
     runs: Sequence[ArmRun],
     treatment_id: str,
@@ -560,13 +577,16 @@ def _contrast(
     if treatment_id not in by_id or baseline_id not in by_id:
         return None
     treatment, baseline = by_id[treatment_id], by_id[baseline_id]
-    theirs = {o.item_id: o for o in treatment.outcomes}
-    ours = {o.item_id: o for o in baseline.outcomes}
+    theirs = treatment.by_item()
+    ours = baseline.by_item()
     shared = [i for i in ours if i in theirs]
     read = QUANTITIES[quantity]
+    # Averaged within an item before the arms are compared, so the bootstrap
+    # keeps resampling *items* — the population the bank can speak for — while
+    # extra replicates reduce custody noise instead of inflating the sample.
     difference, low, high, p_value = _paired_bootstrap(
-        [read(theirs[i]) for i in shared],
-        [read(ours[i]) for i in shared],
+        [_rate(theirs[i], read) for i in shared],
+        [_rate(ours[i], read) for i in shared],
         resamples=int(protocol.statistics["resamples"]),
         alpha=float(protocol.statistics["alpha"]),
         seed=seed,

src/nullius/benchmark/protocol.py

@@ -51,6 +51,7 @@
     "V2_PROTOCOL_PATH",
     "V3_PROTOCOL_PATH",
     "V4_PROTOCOL_PATH",
+    "V5_PROTOCOL_PATH",
     "Protocol",
     "ProtocolVerification",
     "build_protocol",
@@ -133,11 +134,26 @@ def _project(arm: Arm, fields: tuple[str, ...]) -> dict[str, Any]:
     return {name: full[name] for name in fields}
 
 
-LATEST_PROTOCOL_VERSION = "4"
+LATEST_PROTOCOL_VERSION = "5"
 
 V2_PROTOCOL_PATH = Path("benchmark/protocol.v2.lock.json")
 V3_PROTOCOL_PATH = Path("benchmark/protocol.v3.lock.json")
 V4_PROTOCOL_PATH = Path("benchmark/protocol.v4.lock.json")
+V5_PROTOCOL_PATH = Path("benchmark/protocol.v5.lock.json")
+
+REPLICATES = 3
+"""Passes over the bank per custodied arm, fixed before the run.
+
+The v4 ladder measured why one is not enough: re-running the same arm moved
+its accuracy by up to 0.100, six times the metric's resolution, because the
+Custodian draws a fresh holdout for every registration. A contrast smaller
+than that cannot be read from a single draw, and the contrasts this benchmark
+exists to measure are smaller than that.
+
+Three rather than more because the expensive arm costs about fifty minutes a
+pass. Registered here so that "we ran it until it looked right" is not
+available afterwards.
+"""
 
 V2_PREDICTION = (
     "The mechanism contrast B4 - B3 is positive and its 95% interval excludes "
@@ -242,7 +258,58 @@ def _project(arm: Arm, fields: tuple[str, ...]) -> dict[str, Any]:
     "than resolution.",
 )
 
+V5_PREDICTION = (
+    "Replication narrows the ladder rather than reordering it. Averaging three "
+    "custody draws per arm leaves B8 - B6 on coverage positive with an interval "
+    "still excluding zero, and leaves B4 - B3 on verdict accuracy spanning zero "
+    "- the contrast that flipped between the v3 and v4 single draws. If B4 - B3 "
+    "separates under replication, the v4 reading was right and this protocol's "
+    "caution was wrong."
+)
+
+V5_ADJUDICATED = {
+    "treatment": "B8",
+    "baseline": "B6",
+    "quantity": "coverage",
+    "direction": "greater",
+}
+
+V5_EXCLUSION_RULES = (
+    *V4_EXCLUSION_RULES,
+    "Only custodied arms are replicated. An uncustodied arm reads the "
+    "development split, which is fixed by seeds derived from the item id, and "
+    "returns identical results however often it runs - measured, not assumed: "
+    "running the ladder twice left B0 through B3 identical to three decimals "
+    "while every custodied arm moved. Replicating a deterministic arm reports "
+    "a spread of zero as though it were evidence of stability.",
+    "Replicates are averaged within a bank item before arms are compared, so "
+    "the bootstrap continues to resample items - the population the bank can "
+    "speak for. Pooling replicates as independent observations would treat "
+    "three looks at one question as three questions and shrink every interval "
+    "by a factor the design has not earned.",
+)
+
 PROTOCOL_VERSIONS: dict[str, dict[str, Any]] = {
+    "5": {
+        "items": BANK_V2,
+        "truth_lock": V2_TRUTH_LOCK_PATH,
+        "path": V5_PROTOCOL_PATH,
+        "baseline_arm": "B0",
+        "prediction": V5_PREDICTION,
+        "exclusion_rules": V5_EXCLUSION_RULES,
+        "metrics": V3_METRICS,
+        "arms": LADDER_V4,
+        "arm_fields": ARM_FIELDS_V4,
+        "extra_statistics": {
+            "calibration_scope": "asserted_effects",
+            "adjudication": "named_contrast",
+            "adjudicated": dict(V5_ADJUDICATED),
+            "verdict_vocabulary": [v.value for v in Verdict],
+            "adaptive_seed_ceiling": ADAPTIVE_SEED_CEILING,
+            "replicates": REPLICATES,
+            "replicated_arms": "custodied",
+        },
+    },
     "4": {
         "items": BANK_V2,
         "truth_lock": V2_TRUTH_LOCK_PATH,