docs: expand the S1 calibration to 140 replays and correct the reading (#18)

* docs: expand the S1 calibration to 140 replays and correct the reading

Four exams at thirty-five replays each, up from three at ten. The larger sample
changes three published claims.

Dispersion was understated on every exam at n=10, because tails need samples.
Leading with max-over-min was the wrong statistic; by coefficient of variation
and p90/p10 the only outlier is the exam where many implementations are
correct. Exam D is the control that separates the explanations: it costs four
times exam A and takes three times the turns, yet disperses barely more. Cost
variance tracks how many shapes a correct answer can take, not how much work it
takes.

Ten replays observed no failures and suggested the exams were uniformly too
easy. At thirty-five a genuine failure appeared, on the moderate exam rather
than either exam built to be hard. It is a true positive: the replay passed its
own fail-to-pass test and broke an existing one by changing code it was not
asked to touch. Only the no-regression suite caught it -- the file footprint was
identical to every passing replay, so a footprint-based scope check would not
have.

The bar is still met on point estimates, but only two of four exams have a 95%
lower bound at or above 90%, and the README now says so. Three of exam D's
replays were lost to permission stalls, which is the sole reason it misses the
floor -- an instrument limitation rather than model behaviour.

* docs: correct the calibration reading after an audit against the raw data

An audit of the published claims against sittings.csv found four errors. Three
were arithmetic or provenance; the first was an overclaim.

The count of exams establishing a 90% floor was two; it is one. Exam C's Wilson
lower bound is 89.848%, which the table rounded to 90% and the prose then
treated as clearing. Rounding a lower bound up is the one direction this
measurement cannot afford. Bounds now print to three decimals.

Exam D's ten-replay dispersion figure was computed over a different window than
A, B and C -- nine scored rows rather than replays n<=10. Corrected from 1.46x
to 1.36x, which strengthens rather than weakens the argument it supported.

Exam D did not exist when the ten-replay reading was published, so its
before-and-after pair is a retrospective comparison and is now labelled as one
rather than presented as a correction.

analyze.py computed none of the confidence bounds or dispersion statistics the
documents published, so 'regenerates with this script' was false and the tables
could drift from the data unnoticed. It now computes every published figure.

Two claims were also too strong for four data points from one repository. 'Cost
variance tracks ambiguity, not effort' is now stated as a hypothesis the
calibration is consistent with, noting that exam C is a single point carrying
most of the contrast. And attributing exam D's missed floor to its three
permission stalls assumed those replays would have passed, which is unknowable:
they produced no verdict, and exam C misses the floor having lost one replay.

Author: JumpTechCode

README.md

@@ -49,11 +49,12 @@ shipping an instrument if it will not hold its own first measurement to the
 standard it asks of everyone else's.
 
 That measurement has now been taken, and the numbers are in [Calibration
-numbers](#calibration-numbers) below. Thirty replays across three exams
-reproduced their deterministic assertions every time, at a median cost of $0.15
-to $0.43 per replay depending on the task. The bar is met on every clause as a
-point estimate — with the caveat, stated there rather than omitted, that ten
-replays cannot establish a 90% floor however they come out.
+numbers](#calibration-numbers) below. Across four exams replayed thirty-five
+times each, deterministic assertions reproduced in 131 of 132 scored replays, at
+a median cost of $0.15 to $0.57 per replay depending on the task. The bar is met
+on every clause as a point estimate. Stated there rather than omitted: only one
+of the four exams has a 95% lower bound at or above 90%, so this establishes a
+high reproduction rate and not a 90% floor for every exam.
 
 The scaffolding that produced them is committed under `spikes/`. It is not
 product code and is held to a lower standard than the rest of this repository,
@@ -218,39 +219,55 @@ recorded because each one is individually tempting to remove.
 
 ## Calibration numbers
 
-Measured 2026-07-25. Three exams, ten replays each, fixed conditions:
-`claude-opus-4-8`, one git worktree per replay at a pinned commit, one
-fail-to-pass assertion plus an 82-file no-regression suite. Method, raw rows,
-and the full reading are under [`spikes/s1/`](spikes/s1/); the table below
-regenerates with `python3 spikes/s1/analyze.py`.
-
-| Exam | Scored | Errors | Assertions reproduced | Cost per replay (median) | Cost spread | Turns |
-| --- | --- | --- | --- | --- | --- | --- |
-| A — bounded fix | 10 | 0 | 10 of 10 | $0.147 | 1.08× | 6 every time |
-| B — feature slice | 9 | 1 | 9 of 9 | $0.234 | 1.28× | 6–8 |
-| C — design latitude | 10 | 0 | 10 of 10 | $0.426 | 2.70× | 10–19 |
-
-Thirty replays cost $8.65 in total. The one error was a permission stall; it is
-excluded from the counts above rather than recorded as a failure.
-
-**What this supports.** No replay failed its deterministic assertions, and the
-file footprint was identical across all thirty. Effort was a different matter:
-the cost spread widened with how much latitude the task allowed, and on the
-most open-ended exam the same task ran from 10 to 19 turns and varied 2.7-fold
-in cost. Outcome-level assertions look stable enough to build on; token-level
-ones do not.
-
-**What this does not support.** Ten replays cannot establish a 90% floor. A
-result of ten out of ten carries a 95% interval of 72% to 100%, so these numbers
-say "no failure observed in ten replays" and not "at least 90%". Separating 90%
-from 99% needs more replays than this measurement bought.
-
-The pre-registered bar in [Status](#status) is met on every clause as a point
-estimate, with that caveat attached. Two consequences already follow: a
-cost band cannot use one global multiple across exams of different latitude, and
-a single session is a weak cost anchor — on exam C the golden session happened
-to fall at the 10th percentile of its own replays, which accounts for most of
-its apparent cost growth.
+Measured 2026-07-25. Four exams, thirty-five replays each, fixed conditions:
+`claude-opus-4-8`, one git worktree per replay at a pinned commit, exam-supplied
+fail-to-pass tests plus an 82-file no-regression suite. Method, raw rows, and
+the full reading are under [`spikes/s1/`](spikes/s1/); the table regenerates
+with `python3 spikes/s1/analyze.py`.
+
+| Exam | Replays | Errors | Scored | Reproduced | Rate | 95% lower bound | Cost per replay |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| A — bounded fix, pointed | 35 | 0 | 35 | 35 | 100% | **90.11%** | $0.147 |
+| B — feature slice, pointed | 35 | 4 | 31 | 30 | 96.8% | 83.81% | $0.243 |
+| C — design latitude | 35 | 1 | 34 | 34 | 100% | 89.85% | $0.328 |
+| D — discovery, unpointed | 35 | 3 | 32 | 32 | 100% | 89.28% | $0.573 |
+
+140 replays cost $48.50 in total. Eight ended in a permission stall and are
+recorded as errors, excluded from the counts rather than charged to the model.
+
+**What this supports.** Deterministic assertions reproduced in 131 of 132 scored
+replays. The one failure is a true positive: that replay passed its own
+fail-to-pass test and broke an existing test by changing code it had not been
+asked to touch. Only the broad no-regression suite caught it — its file
+footprint was identical to every passing replay of the same exam, so a
+footprint-based scope check would have missed it.
+
+**What this does not support.** The pre-registered bar in [Status](#status) is
+met on every clause as a point estimate, but only **one** of the four exams has
+a 95% lower bound at or above 90%. The bounds are given to two decimals because
+C's is 89.85%, which rounds to 90% and is not 90%; rounding a lower bound up is
+the one direction this measurement cannot afford. A point estimate meeting a
+floor is not the same as establishing it. These numbers establish that
+assertions reproduce at a high rate, and only exam A supports the stronger
+claim.
+
+**Cost dispersion appears to track ambiguity more than effort.** Exam C, where
+many different implementations are correct, is the outlier on every measure: a
+coefficient of variation of 0.38 against 0.07–0.15 for the others, and a p90/p10
+cost ratio of 2.58 against 1.20–1.35. Exam D is the control — it costs four
+times exam A and takes three times the turns, and disperses more than A but far
+less than C. Stated as a hypothesis rather than a result: four exams in one
+repository is thin evidence, and C is a single point carrying most of the
+contrast.
+
+Two consequences follow for the design: a baseline-relative cost band cannot use
+one global multiple, and an exam's band does not appear predictable from its
+cost or difficulty, so it has to be calibrated from that exam's own series.
+
+An earlier version of this section reported ten replays of three exams. That
+sample understated dispersion on every exam and observed no failures at all;
+both errors ran optimistic. A ten-replay series is enough to detect a gross
+regression and not enough to characterise a distribution.
 
 The exams were written to be single-goal because no such session existed in the
 available history, so these figures are an optimistic bound. The full list of

spikes/s1/analyze.py

@@ -12,13 +12,19 @@
 counted on their own line: a harness crash, a spend cap, or a broken
 environment is not evidence about the model.
 
+Every statistic published in the README or in `results/summary.md` is computed
+here, so the documents can be regenerated rather than transcribed. Confidence
+bounds print to three decimal places on purpose: a lower bound rounded up to a
+whole percent reads as clearing a floor it does not clear.
+
 Standard library only, by design — this runs anywhere the spike does.
 """
 
 from __future__ import annotations
 
 import csv
 import json
+import math
 import pathlib
 import statistics
 import sys
@@ -27,6 +33,37 @@
 PER_EXAM_FLOOR = 0.90
 ABSOLUTE_FLOOR = 0.70
 COST_MULTIPLE = 2.0
+Z = 1.96
+
+
+def wilson(k: int, n: int) -> tuple[float, float]:
+    """Wilson score interval. Preferred over the normal approximation, which
+    returns a degenerate zero-width interval when every trial passes."""
+    if n == 0:
+        return (0.0, 0.0)
+    p = k / n
+    denom = 1 + Z * Z / n
+    centre = (p + Z * Z / (2 * n)) / denom
+    half = Z * math.sqrt(p * (1 - p) / n + Z * Z / (4 * n * n)) / denom
+    return (max(0.0, centre - half), min(1.0, centre + half))
+
+
+def dispersion(costs: list[float]) -> dict[str, float]:
+    """Robust spread measures. Max-over-min is reported last because a single
+    outlying replay moves it; it is not a description of the distribution.
+
+    Quantiles use the exclusive method (the standard library default). Other
+    conventions give different p90/p10 figures, so the choice is stated rather
+    than assumed."""
+    quart = statistics.quantiles(costs, n=4, method="exclusive")
+    pcts = statistics.quantiles(costs, n=100, method="exclusive")
+    med = statistics.median(costs)
+    return {
+        "cv": statistics.stdev(costs) / statistics.mean(costs),
+        "iqr_med": (quart[2] - quart[0]) / med,
+        "p90_p10": pcts[89] / pcts[9],
+        "max_min": max(costs) / min(costs),
+    }
 
 
 def main() -> int:
@@ -35,57 +72,81 @@ def main() -> int:
         print("no sittings recorded", file=sys.stderr)
         return 2
 
+    exams = sorted({r["exam"] for r in rows})
+    clause_90 = clause_70 = clause_cost = True
+    clearing_floor = []
+
+    print("REPRODUCTION")
     print(
-        f"{'exam':<6}{'n':<4}{'err':<5}{'tier1/2':<10}"
-        f"{'cost med':<11}{'golden':<11}{'ratio':<8}{'spread':<9}{'turns':<9}{'touched'}"
+        f"  {'exam':<6}{'replays':<9}{'err':<5}{'scored':<8}{'passed':<8}"
+        f"{'rate':<9}{'95% lower':<11}{'95% upper'}"
     )
-
-    clause_90 = True
-    clause_70 = True
-    clause_cost = True
-
-    for exam in sorted({r["exam"] for r in rows}):
+    for exam in exams:
         ex = [r for r in rows if r["exam"] == exam]
-        errored = [r for r in ex if r["is_error"] == "1"]
-        # ERROR is not FAIL. An errored sitting produced no verdict, so it
-        # leaves the denominator rather than counting against the model.
+        # ERROR is not FAIL. An errored replay produced no verdict, so it leaves
+        # the denominator rather than counting against the model.
         scored = [r for r in ex if r["is_error"] != "1"]
         passed = [r for r in scored if r["f2p_pass"] == "1" and r["p2p_pass"] == "1"]
-
         rate = len(passed) / len(scored) if scored else 0.0
-        costs = [float(r["cost_usd"]) for r in scored]
-        median = statistics.median(costs) if costs else 0.0
-        golden = json.loads((SPIKE / f"exams/{exam}/golden.json").read_text())["cost_usd"]
-        ratio = median / golden if golden else float("inf")
-        turns = sorted(int(r["num_turns"]) for r in scored)
-        touched = sorted({r["touched_files"] for r in scored})
-        spread = max(costs) / min(costs) if costs and min(costs) else 0.0
-
-        turn_range = f"{turns[0]}" if turns[0] == turns[-1] else f"{turns[0]}-{turns[-1]}"
-        print(
-            f"{exam:<6}{len(ex):<4}{len(errored):<5}{rate:<10.0%}"
-            f"{median:<11.4f}{golden:<11.4f}{ratio:<8.2f}"
-            f"{spread:<9.2f}{turn_range:<9}{','.join(touched)}"
-        )
-
+        lo, hi = wilson(len(passed), len(scored))
+        if lo >= PER_EXAM_FLOOR:
+            clearing_floor.append(exam)
         if rate < PER_EXAM_FLOOR:
             clause_90 = False
         if rate < ABSOLUTE_FLOOR:
             clause_70 = False
+        print(
+            f"  {exam:<6}{len(ex):<9}{len(ex) - len(scored):<5}{len(scored):<8}"
+            f"{len(passed):<8}{rate:<9.1%}{lo:<11.3%}{hi:.3%}"
+        )
+
+    print()
+    print("COST")
+    print(
+        f"  {'exam':<6}{'median':<10}{'golden':<10}{'ratio':<9}"
+        f"{'CV':<9}{'IQR/med':<10}{'p90/p10':<10}{'max/min'}"
+    )
+    for exam in exams:
+        scored = [r for r in rows if r["exam"] == exam and r["is_error"] != "1"]
+        costs = [float(r["cost_usd"]) for r in scored]
+        median = statistics.median(costs)
+        golden = json.loads((SPIKE / f"exams/{exam}/golden.json").read_text())["cost_usd"]
+        ratio = median / golden if golden else float("inf")
         if ratio > COST_MULTIPLE:
             clause_cost = False
+        d = dispersion(costs)
+        print(
+            f"  {exam:<6}{median:<10.4f}{golden:<10.4f}{ratio:<9.2f}"
+            f"{d['cv']:<9.3f}{d['iqr_med']:<10.3f}{d['p90_p10']:<10.2f}{d['max_min']:.2f}"
+        )
 
-    def verdict(ok: bool) -> str:
-        return "MET" if ok else "MISSED"
+    scored_all = [r for r in rows if r["is_error"] != "1"]
+    passed_all = [r for r in scored_all if r["f2p_pass"] == "1" and r["p2p_pass"] == "1"]
+    errors = len(rows) - len(scored_all)
+    total = sum(float(r["cost_usd"]) for r in rows)
 
     print()
-    print(f"  every exam >= 90% ......... {verdict(clause_90)}")
-    print(f"  no exam below 70% ......... {verdict(clause_70)}")
-    print(f"  median cost <= 2x golden .. {verdict(clause_cost)}")
+    print(
+        f"  {len(rows)} replays, {errors} errors ({errors / len(rows):.1%}), "
+        f"{len(passed_all)} of {len(scored_all)} scored reproduced, ${total:.2f} total"
+    )
+
+    print()
+    print("THE PRE-REGISTERED BAR, on point estimates")
+    print(f"  every exam >= 90% ......... {'MET' if clause_90 else 'MISSED'}")
+    print(f"  no exam below 70% ......... {'MET' if clause_70 else 'MISSED'}")
+    print(f"  median cost <= 2x golden .. {'MET' if clause_cost else 'MISSED'}")
     print()
+    print(
+        f"  Exams whose 95% LOWER BOUND also reaches 90%: "
+        f"{len(clearing_floor)} of {len(exams)}"
+        + (f" ({', '.join(clearing_floor)})" if clearing_floor else "")
+    )
+    print("  A point estimate meeting the floor is not the same as establishing it.")
 
+    print()
     if clause_90 and clause_70 and clause_cost:
-        print("BAR MET")
+        print("BAR MET (point estimates)")
         return 0
     print("BAR MISSED — the design pivots before any product code is written")
     return 1