Context
Every published number is a single draw: one eval pool (seed 777), one scoring run per configuration. The infrastructure for multi-seed evaluation already exists — schedule_registry.py defines the recon-benchmark cadence over seeds [7, 13, 42] — but it is paused and has never produced an artifact. Meanwhile the only visible replication (×3 vs ×5 sampling on the same adapter, same pool) already shows ~1.2 R_w points of pure sampling noise and a 4-task swing on PARTIAL_MATCH — the same order of magnitude as the 4.1-point headline margin. Bootstrap CIs capture within-pool resampling uncertainty; they say nothing about across-pool (seed) variance, which is the variance a reader actually cares about.
Evidence
system/.../schedule_registry.py: recon-benchmark, seeds [7,13,42], on-demand/paused.docs/validation/: no multi-seed artifact.- ×3 vs ×5 same adapter, same pool: R_w 0.9128 vs 0.9007; MEDIUM recall 0.8372 vs 0.7984; PARTIAL_MATCH 26 vs 22 caught; FIELD_CORRUPTION 13 vs 10 (
bench-eval-full-740.json vs bench-eval-champion-x5.json).
docs/HANDOFF.md §3 names "the sampling marginal" as the honest variance but never quantifies it.
Task
- Generate 800-task pools for seeds 7, 13, 42 (plus the frozen test seed from the held-out-discipline issue when scoring a final claim). Verify each pool against the training set with the existing exact-hash + MinHash contamination checks; commit the contamination reports.
- Score champion (×5, native prompt) and base model on each pool locally; score DeepSeek on each pool if API budget allows — if not, run DeepSeek on one extra seed and say so.
- Report per-seed R_w/accuracy/precision plus mean ± range across seeds, next to (not instead of) the bootstrap CI. If across-seed spread exceeds the bootstrap CI width, that sentence goes on the card.
- Quantify the sampling marginal on one pool: 3 independent ×5 runs of the champion (different sampling seeds), report R_w spread — this is the number the blog's "not a lucky checkpoint" argument actually needs.
- Unpause/exercise the
recon-benchmark cadence workflow once so the schedule registry entry stops being aspirational, or change its status honestly.
Acceptance criteria
- ≥3 seed pools scored for champion and base; exports + contamination reports committed.
- Card and README show across-seed variance next to bootstrap CIs.
- Sampling-marginal number published; blog's run-variance argument rewritten on top of it.
Dependencies
- Pairs with the held-out-discipline issue (frozen test seed); benefits from the scoring-harness fixes landing first so all new exports are computed with corrected code.
Cost tier
T2-compute — ~3× local MLX eval wall time (~1 h each for ×5) + optional DeepSeek API passes.
Context
Every published number is a single draw: one eval pool (seed 777), one scoring run per configuration. The infrastructure for multi-seed evaluation already exists —
schedule_registry.pydefines therecon-benchmarkcadence over seeds [7, 13, 42] — but it is paused and has never produced an artifact. Meanwhile the only visible replication (×3 vs ×5 sampling on the same adapter, same pool) already shows ~1.2 R_w points of pure sampling noise and a 4-task swing on PARTIAL_MATCH — the same order of magnitude as the 4.1-point headline margin. Bootstrap CIs capture within-pool resampling uncertainty; they say nothing about across-pool (seed) variance, which is the variance a reader actually cares about.Evidence
system/.../schedule_registry.py:recon-benchmark, seeds [7,13,42], on-demand/paused.docs/validation/: no multi-seed artifact.bench-eval-full-740.jsonvsbench-eval-champion-x5.json).docs/HANDOFF.md§3 names "the sampling marginal" as the honest variance but never quantifies it.Task
recon-benchmarkcadence workflow once so the schedule registry entry stops being aspirational, or change its status honestly.Acceptance criteria
Dependencies
Cost tier
T2-compute — ~3× local MLX eval wall time (~1 h each for ×5) + optional DeepSeek API passes.