diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f08b26d..1bcb5ba 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -16,12 +16,18 @@ jobs: - uses: astral-sh/setup-uv@v8.3.0 # no floating v8 major tag exists with: enable-cache: true + # `uv run ` executes the module as a plain script. The suites are + # bare `def test_*` collections with no `__main__` entrypoint, so that + # form imported each file and exited 0 without running a single + # assertion — CI passed vacuously from fdcbfcc until PR #13's review + # caught it. pytest must be the thing invoked, per file so the + # per-suite log grouping still works. - name: Run all offline test suites run: | rc=0 for f in test_*.py; do echo "::group::$f" - if ! uv run "$f"; then + if ! uv run pytest -q "$f"; then echo "FAILED: $f" rc=1 fi diff --git a/HANDOFF.md b/HANDOFF.md index 45bc701..da9aecd 100644 --- a/HANDOFF.md +++ b/HANDOFF.md @@ -1,142 +1,158 @@ # HANDOFF.md — mute-map -_Last updated: 2026-07-29 (M4 brief reviewed; D19–D22 frozen)_ +_Last updated: 2026-07-29 (M4 built, run, and PASSED — every measurement stage closed)_ ## What was just done -**M3 built, run, and PASSED** (2026-07-28) — the specificity matrix, per the -frozen `docs/M3-BRIEF.md` (D15–D18). Landed in one PR with its docs spine: - -- **`m3_matrix.py`** — the M3 runner, cut from `m2_depth.py` (never from the - certified `m0_anchor.py`), with its own byte-frozen `GATE_WORDING`. Holds depth - and dose fixed at the switch's home (late third, λ = 1, k = 1) so every cell - differs from every other in exactly one thing: *which* direction was removed. - 486 cells per subject — 36 `clean` + 432 matrix + 18 out-of-subset - control-extras — with the 108-cell M1 re-certification graded first. -- **`m3_verdict.py`** — the cross-subject AND over 1.5B and 3B, plus the - descriptive package (within- vs cross-category split, row/column profiles, - asymmetry, the printed grid). Refuses any run that is not a result. -- **`test_m3.py`** — 83 cases; the full suite is 272 and green. - -**Results: MATRIX-SPECIFIC at 1.5B AND 3B.** Clause (1) pooled off-diagonal − -diagonal: +0.971 [+0.867, +0.983] at 1.5B (diagonal 0/34 vs off-diagonal -363/374) and +0.881 [+0.731, +0.943] at 3B (3/32 vs 343/352). Clause (2) -within-category: +0.950 [+0.814, +0.978] and +0.891 [+0.730, +0.947]. 0.5B also -MATRIX-SPECIFIC, off-gate. **No subject carries the ON A DAMAGED FLOOR -qualifier** — the collateral floor reads [0.728, 0.963] / [0.851, 0.995] / -[0.843, 0.994] against the pre-registered 0.5, which settles the one question -the brief left genuinely open. Every pre-registered n landed exactly (28/34/32 -gated, 308/374/352 off-diagonal, 96/100/101 within-category, 24/28/29 -restricted). The M1 cross-check re-certified the instrument bit-for-bit on every -run (108/108 cells, `concept_mass` exact, ×3 subjects). No degeneracy fired -anywhere, and the effective-n per-item collapse agrees with the pooled gate on -every subject and both clauses. - -**The three descriptive findings the gate did not ask for:** collateral -concentrates on a few fragile **probes** rather than being caused by damaging -**primes** (at 1.5B all 11 off-diagonal misses land on 4 probes; `silver`, -`Canada`, `China`, `Jupiter` and `Mars` cause zero collateral as primes); -category-block collateral is CI-clean at 0.5B (cross − within +0.105 [+0.032, -+0.196]) and dissolves into noise by 1.5B; and the only non-zero diagonal cells -anywhere are `Egypt` 2/3 and `October` 1/2 at 3B — exactly the S3 leaky-switch -stratum D11 pre-registered. - -**The one pre-registration that inverted.** `silver` entered the subset as the -*non-specific anti-example*, expected to damage everything. Its row damages -nothing at any scale (27/27, 31/31, 31/31); its **column** is the most fragile in -the grid (7/11, 27/33, 6/11). M1's and M2's single control cell had sampled -silver's column and been read as a fact about silver's row. This is a -re-attribution, not a retraction — M1's and M2's numbers stand as published. - -**One carry-forward M3 creates, found at its own adversarial review (PR #9 F1).** -M3's gate is a conjunction, but D17's frozen degeneracy scope names only clause -(1)'s surviving arm (`off_diagonal`) as dispositive — clause (2)'s -(`within_category_off_diagonal`) is guarded nowhere, and at the two smaller -subjects it runs 3–4× the guarded arm's wrong-opening share (0.052 vs 0.016 at -0.5B, 0.030 vs 0.008 at 1.5B; at 3B the ordering inverts, 0.010 vs 0.014). -Nothing here is affected — neither gate-reading arm came within an order of -magnitude of COLLAPSE_SHARE = 0.5, and the one arm that does run close is the -pooled **diagonal** (0.464 at 0.5B), which D17 pre-registers as TAG-only because -it *is* the expected mute signature — and `m3_matrix.GATE_WORDING` was **not** -amended, because it is byte-frozen with three subjects' artifacts and editing a -pre-registration after seeing results is what D9/D10 exist to prevent. Owned in -M3-BRIEF's Honest limits, and carried here: **any later stage whose gate is a -conjunction must put every surviving-side comparison arm on the dispositive -degeneracy list, and its frozen wording must enumerate them explicitly.** - -All four of M2's carry-forwards landed as the brief dispositioned them: PR #7 F2 -(the tier-width caveat, retired structurally and stated affirmatively in D17's -frozen wording), F5 + F4 (D18's two run-time bars, span and ASCII, on the -subject's own tokenizer), and the non-blocking F6 (`m3_matrix.py` validates -before the checkpoint loads and `validate()` returns the parsed M1 artifact). +**M4 built, run, and PASSED** (2026-07-29) — the vocabulary collateral strip, per +the frozen `docs/M4-BRIEF.md` (D19–D22). Landed in one PR with its docs spine: + +- **`m4_strip.py`** — the M4 runner, cut from `m3_matrix.py` (never from the + certified `m0_anchor.py`), with its own byte-frozen `GATE_WORDING` taken + verbatim from D20 including the AS-SCORED ONLY qualifier and both string + templates. Keeps M3's 12 characterized directions as the **primes** and widens + the **probes** to all 60 M1 concepts: 180 `clean` + 12 × 180 = **2,340 cells + per subject**, every one at the identical late third, λ = 1, k = 1. The 255 + M1-recorded and 468 M3-recorded cells — 633 of the 2,340 once the 90-cell + overlap is counted once — are graded FIRST, and any mismatch exits INVALID + before a new cell is read. D22's span and ASCII bars now run over all 60 scored + words plus the 12 direction words. +- **`m4_verdict.py`** — owns `strip_verdict()` (precedence NOT A RESULT > + DEGENERATE > UNDERPOWERED > the level bar, failing label the lineage null + `not shown`) and the cross-subject AND over 1.5B and 3B. The runner imports it, + so exactly one implementation of the pre-committed verdict string exists. +- **`test_m4.py`** — 124 cases (112 at first submission, +12 added at review); + the full suite is 396 and green, in CI as well as locally. + +**Results: VOCAB-SPARING at 1.5B AND 3B — AS-SCORED ONLY.** Of the gated items +whose concept is outside M3's twelve, those surviving **all 12** deletions: +**51/71 = 0.718 [0.605, 0.810]** at 1.5B and **63/84 = 0.750 [0.648, 0.830]** at +3B, both clearing the pre-registered 0.5 bar; 0.5B `not shown` off-gate at +**11/41 = 0.268 [0.157, 0.419]**. Every pre-registered n landed exactly (arms +41/71/84, concepts 23/41/43, ceilings 35/41, 69/71, 82/84 with precisely the +named misses). Both cross-checks re-certified the instrument bit-for-bit on every +run — M1 255/255 and M3 468/468 cells, `concept_mass` exact, ×3 subjects. No +degeneracy fired on the dispositive arm (shares 0.022 / 0.011 / 0.011 against a +0.5 threshold). + +**The qualifier fired at both gate-bearing subjects, and it earned its +existence.** The pre-registered **concept-level collapse** — one binary per +concept instead of per item — reads 0.585 (lower **0.434**) at 1.5B and 0.605 +(lower **0.456**) at 3B: below the bar while the item-level gate clears it. Per +D20's pre-commitment those are **the honest numbers to quote**, and Amendment 1 +(round 4, F20) is exactly why they now ride inside the verdict string instead of +sitting in prose. The residual-conservative read clears at both (49/71 lower +0.575; 62/84 lower 0.635), so only one of the two reads fired. + +**Three descriptive findings the gate did not ask for.** (1) M3's finding 1 +**generalizes out of sample**: collateral concentrates on fragile **probes**, not +damaging **primes** — no prime's row falls below 63/71 at 1.5B or 77/84 at 3B, +while individual columns collapse (`copper` 6/12, `mosquito` 8/12 at 1.5B; +`eagle` 8/12, `platinum` 9/12 at 3B) and 32/53 and 33/55 gated columns take +**zero** collateral. The distribution is bimodal, which is why the item-level and +concept-level statistics diverge. (2) **Category-block collateral is real in the +wider vocabulary and does NOT dissolve with scale** — within 22/29 vs cross +769/823 at 1.5B, 35/53 vs 913/955 at 3B — reversing M3's read, whose +within-category arm was 30 of 34 pairs *countries*. A re-scoping of M3's finding, +not a retraction. (3) **0.5B is the first measured divergence** between the +subset-12's robustness and the wider roster's: 30 of 41 gate-arm items (73%) are +damaged by at least one deletion, against 28% at 1.5B and 25% at 3B. + +**The residual set was bigger in the ablated arm than the clean arm.** The +clean-arm gate-arm residuals were the pre-computed 0 / 2 / 2, but the run +recorded 0 / 27 / 21 residual cells overall (26 / 21 in the gate arm), all on +`beetle`, `butterfly`, `trumpet`. That is exactly what D20's selector was written +for ("in the ablated cells the set is whatever the run records"), and it is why +the conservative read moves the number at all (51 → 49, 63 → 62). A case-exact +selector — the F16 defect — would have found zero of them. + +**Both inherited obligations discharged.** The **conjunction-degeneracy** rule +(PR #9 F1) is satisfied by design: M4's gate is single-clause, so the dispositive +list has exactly one surviving arm, and `m4_strip.GATE_WORDING` enumerates it and +restates the rule for any future conjunction. The **`oracle._BOUNDARY` +boundary-class decision** was not owed: D18's trigger is a stage freezing a +non-ASCII list, and M4 adds no vocabulary — the ASCII bar pins the premise at run +time instead. That trigger still names the banked scope stretch. ## Where things stand -Chain: ~~M0~~ → ~~M1~~ → ~~M2~~ → ~~M3~~ — **the v1 chain per `docs/KICKOFF.md` -is complete.** `docs/DECISIONS.md` now runs D1–D18. S1/S2 stretches are optional -and each needs its own brief. +Chain: ~~M0~~ → ~~M1~~ → ~~M2~~ → ~~M3~~ → ~~M4~~ — **the v1 chain per +`docs/KICKOFF.md` is complete and the Kyle-picked close-out stage has landed.** +`docs/DECISIONS.md` now runs D1–D22. Nothing is pre-committed next. S1/S2 remain +optional stretches, each needing its own brief. ## Immediate next move -**Decided (Kyle, 2026-07-28, session "mute-map post M3 decisions"): run the -vocabulary collateral strip as close-out stage M4, then write-up + -`/seed-hunt`.** The S1/S2 stretches were declined for this repo and banked as -idea #13 in `~/Projects/j-lens-proj-ideas/jlens-followon-backlog.md` — they -compete in the seed-hunt on equal terms, no incumbent's privilege. - -**Where M4 stands: `docs/M4-BRIEF.md` is written, adversarially reviewed (see -the mailbox under `~/.claude/reviews/mute-map/` for the round-by-round -record), and D19–D22 are FROZEN (Kyle, 2026-07-29) — (a) across the board.** -The verdict label was resolved per review F8: the label stays `VOCAB-SPARING` -and the realized survival proportion rides inside the verdict string, the M3 -`ON A DAMAGED FLOOR` pattern applied to a level bar. **Amended post-freeze, -pre-run and ratified by Kyle (round 4, F20):** that string as first frozen -carried only the as-scored proportion, so the two pre-registered reads that -can flip which number is honest (residual-conservative fail-in-place; -concept-level collapse) stayed in prose — the exact failure F8 was adopted to -prevent. D20 now carries a pre-declared **AS-SCORED ONLY** qualifier, attached -by the runner whenever a conservative read's Wilson lower bound falls below -0.5 while the as-scored read's does not. **Amendment 2 (rounds 5–6, -F22/F24–F27), also ratified by Kyle 2026-07-29:** the failing label became the -lineage's null `not shown` rather than an assertive negative, 0.5B was scoped -inside the wording, the qualifier's attachment was restricted to claim-level -verdicts, and both string templates were stated explicitly. The gate, its 0.5 -bar, its arm and its re-certification precondition are unchanged under both -amendments. Full `DECISIONS.md` entries D19–D22 land with the M4 code PR, per -the M0–M3 pattern. - -The stage: 12 subset primes × all 180 M1 items (2,340 cells/subject), gate = a -single-clause VOCAB-SPARING level bar on the non-subset pool (per-item -survives-all-12, Wilson lower bound ≥ 0.5 at 1.5B AND 3B — a *new*, -deliberately lenient, uncalibrated constant on the 12-fold conjunction, ≈ -0.944 per-cell under independence, explicitly not carried from M3's per-cell -floor). Realized ns are known from the recorded gated sets (gate arm 41 / 71 / -84); 486 / 844 / 993 of the pool's 492 / 852 / 1,008 cells are genuinely new, -the rest are M1-recorded and cap the gate arm at 35/41, 69/71, 82/84 before -any forward pass. Design facts found while drafting: five probe clues mention -a prime's spelling under D5's own prefix rule (October→september-2, -silver→flute-1, China→jade-1, October→opal-2, Egypt→beetle-2 — the last -ungated on all three subjects; D21(a) keeps all five in the pool); D9(b)'s owned -span-truncation residual re-enters a gate-bearing arm for the first time since -M1 (0 / 2 / 2 cells, carried by a pre-registered fail-in-place conservative -read); all 60 roster words pass the D18 span bar on all three tokenizers -(checked in advance); and M3's own 0.5B subset already fails M4's bar -in-statistic (19/28, Wilson lower 0.4934), so the off-gate 0.5B floor may -genuinely fail — reportable under the standing frame. No runner code exists -yet; with D19–D22 now frozen, code is the next step, cut from `m3_matrix.py`. +**Write-up, then `/seed-hunt`** — the plan of record since 2026-07-28, unchanged. +Every measurement stage is closed and M3's stated bound ("nothing here shows that +deleting France spares the other 48 M1 concepts") is now measured rather than +assumed. The S1 (7B) and S2 (lexical vs semantic scope) stretches were declined +for this repo and banked as idea #13 in +`~/Projects/j-lens-proj-ideas/jlens-followon-backlog.md`; they compete in the +seed-hunt on equal terms, no incumbent's privilege. + +Two things a write-up must not get wrong, both pre-committed rather than +discovered after the fact: + +- **The headline is scoped.** `VOCAB-SPARING` carries `AS-SCORED ONLY` at both + gate-bearing subjects. The honest one-line summary is "the item-level floor + clears 0.5; the concept-level floor's lower bound does not (0.434 / 0.456)." +- **The claim is about the *measurable* vocabulary.** 25 / 7 / 5 of the 48 + non-subset concepts gate zero items — a competence selection that plausibly + enriches for robust concepts and biases the floor **upward**. Standing constraints unchanged: certified environment = `mps` + torch 2.13.0 + transformers 5.13.1 (off it: NOT A RESULT); `m0_anchor.py` stays certified and un-editable, and `m1_battery.GATE_WORDING` / `m2_depth.GATE_WORDING` / -`m3_matrix.GATE_WORDING` are byte-frozen with their artifacts (editing any forces -a full re-run of that milestone); `oracle.py` is byte-shared by three consumers -and must stay identical in all of them; adversarial review before any merge. +`m3_matrix.GATE_WORDING` / `m4_strip.GATE_WORDING` are byte-frozen with their +artifacts (editing any forces a full re-run of that milestone); `oracle.py` is +byte-shared by **four** consumers now and must stay identical in all of them; +adversarial review before any merge. ## Open questions / blockers -- None blocking. M3 is closed and the v1 chain with it. Two inherited - obligations, both conditional on a later stage existing: **(1)** S2's brief - owes the `oracle._BOUNDARY` boundary-class decision before it freezes any - non-ASCII list; **(2)** any stage whose gate is a **conjunction** must put - every surviving-side comparison arm on the dispositive degeneracy list and - enumerate them in its frozen wording (PR #9 F1, above). +- **None blocking.** Every stage is closed. +- **CI ran zero tests from `fdcbfcc` until 2026-07-29, now fixed** (PR #13 + adversarial review, F1). `.github/workflows/ci.yml` looped `uv run test_.py`, + which executes each file as a plain script; with no `__main__` guard the files + imported, defined their functions and exited 0, so the workflow's `rc` only ever + reflected import errors. It now runs `uv run pytest -q "$f"` per file, keeping + the per-suite log grouping — and the workflow is **green on the real runner** + at `40e7c26` with all 396 cases genuinely collected and run. Any green CI badge + before 2026-07-29 certifies syntax, not behaviour. +- **Making CI real immediately exposed a second defect** (same review, F5): + `test_a_dry_run_never_loads_the_checkpoint` in both `test_m3.py` and + `test_m4.py` passed `--lens lenses/.pt` into the real `main()`, which + `torch.load`s the artifact before the `--dry-run` exit — and those weights are + gitignored by decision K3, so the case passed locally and failed on every clean + checkout. Both now supply the artifact synthetically via monkeypatched + `torch.load`; the guarantee is unchanged (`from_pretrained` still raises if the + checkpoint loads). **Note for a later M3 audit: `test_m3.py` was edited in PR + #13, after M3 PASSED.** Only that one test changed; `m3_matrix.py`, its frozen + `GATE_WORDING` and its published artifacts are untouched — leaving M3's copy + broken would have left the shared CI gate red. +- **Three follow-ups from the same review, all nice-to-have, none fixed:** + **(F3)** `m4_strip.GATE_WORDING["degeneracy"]` — byte-frozen with three + subjects' artifacts, so it cannot be edited — promises per-pair-cell degeneracy + texture "attached to the readout it compromises", but `strip_package()` computes + none and its `tokenizer` parameter is left unused (M3 computed exactly that). + Honouring it would change the JSONs and cost a ~50-minute re-run for a readout + that is pre-declared non-verdict-bearing at n ≤ 3; the write-up owes a + deviations-table row instead, and must not quote that clause as if the field + exists. **(F4)** `m4_strip.main()` re-parses the battery outside the + `try/except` that turns battery drift into a clean `VERDICT: INVALID`, so those + guards would raise a bare traceback rather than exit 2 — unreachable, since the + file cannot change between the two calls in one process. + **(F6)** the CI job is still *named* `offline-suites` and its header comment + still claims no network, but now that pytest really runs, four Qwen2.5 + tokenizer repos are fetched from `huggingface.co` on every push and PR + (`test_the_whole_sixty_word_roster_clears_both_bars_on_the_real_tokenizer` ×3 + plus M3's roster test). 390 of 396 pass with no network at all. **A future red + build there is network, not logic.** The two remedies — cache + `~/.cache/huggingface`, or add a `network` marker and deselect it — trade off + against each other (the marker route stops exercising the real tokenizers, + which is the roster bar's whole point), so it is a workflow design call rather + than a correctness fix. +- **One conditional obligation survives:** S2's brief owes the + `oracle._BOUNDARY` boundary-class decision before it freezes any non-ASCII + list. M4 did not fire that trigger. diff --git a/PROJECT.md b/PROJECT.md index 9137215..8f9081f 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -3,26 +3,26 @@ **One-liner:** Map the late-band J-lens output off-switch found in dim-stage S4b — breadth, localization, dose, specificity — on small local Qwen models. -**Status:** **v1 chain complete — M3 PASSED (2026-07-28)**, the off-switch is -**MATRIX-SPECIFIC at 1.5B and 3B**: over the full 12 × 12 prime × probe grid, -deleting one concept's direction at the late third silences that concept and -leaves the other eleven almost untouched (1.5B: diagonal 0/34 vs off-diagonal -363/374). M2 PASSED (LATE-LOCALIZED at 1.5B and 3B), M1 PASSED -(BREADTH-SPECIFIC at 1.5B and 3B), M0 PASSED (2026-07-27); all re-certified -bit-for-bit on every later run. v1 = M0–M3 per `docs/KICKOFF.md`. **Close-out -stage M4 (the vocabulary collateral strip) is now in flight** — brief written -and reviewed, **decisions D19–D22 frozen 2026-07-29**, no runner code yet. The -S1/S2 stretches were declined for this repo and banked (idea #13). +**Status:** **v1 chain complete and the close-out stage landed — M4 PASSED +(2026-07-29)**, the off-switch is **VOCAB-SPARING at 1.5B and 3B, AS-SCORED +ONLY**: over a 12-prime × 180-item strip, deleting any one of the 12 +characterized directions at the late third spares most of the *measurable wider +vocabulary* — 51/71 = 0.718 [0.605, 0.810] of gated non-subset items survive all +12 deletions at 1.5B and 63/84 = 0.750 [0.648, 0.830] at 3B. Both verdicts carry +the pre-declared **AS-SCORED ONLY** qualifier, because the concept-level collapse +sits below the bar (0.585, lower 0.434; 0.605, lower 0.456). M3 PASSED +(MATRIX-SPECIFIC at 1.5B and 3B), M2 PASSED (LATE-LOCALIZED), M1 PASSED +(BREADTH-SPECIFIC), M0 PASSED (2026-07-27); all re-certified bit-for-bit on every +later run — M4 re-certifies **two** artifact sets at once (M1 255/255 and M3 +468/468 cells, ×3 subjects). v1 = M0–M3 per `docs/KICKOFF.md`; M4 is the +Kyle-picked post-KICKOFF close-out. -**Next action:** build the M4 runner. `docs/M4-BRIEF.md` is written and -adversarially reviewed (PR #10); **D19–D22 were frozen 2026-07-29 — (a) across -the board** — so the next step is code: `m4_strip.py` cut from `m3_matrix.py`, -`m4_verdict.py`, `test_m4.py`, and the D19–D22 entries appended to -`docs/DECISIONS.md` in that same code PR. After M4: write-up + `/seed-hunt`. -The S1 (7B) and S2 (lexical vs semantic scope) stretches were declined for -this repo and banked as idea #13 in -`~/Projects/j-lens-proj-ideas/jlens-followon-backlog.md`; they compete in the -seed-hunt on equal terms. +**Next action:** the write-up, then `/seed-hunt`. Every measurement stage is +closed and M3's stated bound — "nothing here shows that deleting France spares +the other 48 M1 concepts" — is now measured rather than assumed. The S1 (7B) and +S2 (lexical vs semantic scope) stretches were declined for this repo and banked +as idea #13 in `~/Projects/j-lens-proj-ideas/jlens-followon-backlog.md`; they +compete in the seed-hunt on equal terms. **Key facts** - Fact — Anchor: S4b (dim-stage), concept-specific off-switch at 1.5B, +.727 @@ -77,6 +77,36 @@ seed-hunt on equal terms. / 105 (1.5B) / 116 (3B); primed 0/69, 0/105, 12/116; contrast +0.478 / +0.762 / +0.690. Planets and musical instruments go from 0 gated items on every subject to 7/8/15 and 2/8/13 of 18. +- Fact — M4 (2026-07-29): on the 12-prime × 180-item strip at the late third + (λ = 1, k = 1), the gated non-subset items surviving **all 12** deletions are + **51/71 = 0.718 [0.605, 0.810]** at 1.5B and **63/84 = 0.750 [0.648, 0.830]** + at 3B, both clearing the pre-registered 0.5 bar; 0.5B `not shown` off-gate at + 11/41 = 0.268 [0.157, 0.419]. Every pre-registered n and ceiling landed exactly + (arms 41/71/84, ceilings 35/41, 69/71, 82/84 with the named misses). No + degeneracy fired on the dispositive arm. +- Fact — both gate-bearing verdicts carry **AS-SCORED ONLY**: the pre-registered + concept-level collapse reads 0.585 (lower **0.434**) at 1.5B and 0.605 (lower + **0.456**) at 3B — below the bar while the item-level gate clears it — so per + D20 those are the honest numbers to quote. The residual-conservative read + clears at both (49/71 lower 0.575, 62/84 lower 0.635). +- Fact — M4's descriptive map (no gate reads it): finding 1 **generalizes out of + sample** — no prime is a wrecking ball (1.5B rows all 63–67/71, 3B all + 77–83/84), while specific probe columns collapse (`copper` 6/12, `mosquito` + 8/12 at 1.5B; `eagle` 8/12, `platinum` 9/12 at 3B) and 32/53 and 33/55 gated + columns take **zero** collateral. Category-block collateral is real in the + wider vocabulary and does **not** dissolve with scale (within 22/29 vs cross + 769/823 at 1.5B; 35/53 vs 913/955 at 3B). +- Inference — M3's "category collateral dissolves by 1.5B" was an artefact of + **arm composition**, not scale: M3's within-category arm was 30 of 34 pairs + countries, while the strip's samples ten categories. A one-sided sample + inverted the finding; M3's numbers stand, their scope narrows. +- Inference — 0.5B is the first measured divergence between the subset-12's + robustness and the wider roster's: 30 of 41 gate-arm items (73%) are damaged by + at least one deletion, against 28% at 1.5B and 25% at 3B. +- Decision — D19–D22 frozen 2026-07-29 (M4-BRIEF), (a) across the board, with two + post-freeze ratified amendments to the D20 wording package (the AS-SCORED ONLY + mechanism; the `not shown` failing label, 0.5B scoping, bar-level-only + attachment and both string templates). - Decision — K1–K4 at kickoff: slug/visibility, naming-only competence gate, lens provenance (no refits in core), stats ruler ported verbatim. - Fact — Lens artifacts gitignored (70–560MB); sourced from local dim-stage diff --git a/README.md b/README.md index 5694831..d442121 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,30 @@ pre-registered gates frozen as code before any run): **The honest framing:** an effect *found during a replication, characterized here* — the anchor is dim-stage's own recorded result, not a paper claim. -**Status: M3 PASSED 2026-07-28 — the v1 chain is complete; close-out stage M4 -is in flight — brief reviewed, decisions D19–D22 frozen 2026-07-29, runner not -yet written.** On the full 12 × 12 prime × probe matrix at the switch's home +**Status: M4 PASSED 2026-07-29 — every measurement stage is closed.** The +close-out stage widened M3's probe side from 12 concepts to all 60: keep the 12 +characterized directions as the things deleted, and ask *every* item in the +frozen 180-item battery. Deleting any one of those directions at the late third +**spares most of the measurable wider vocabulary** — of the gated items whose +concept is outside M3's twelve, **51/71 = 0.718 [0.605, 0.810]** survive all 12 +deletions at 1.5B and **63/84 = 0.750 [0.648, 0.830]** at 3B, both clearing the +pre-registered 0.5 bar. Both verdicts carry the pre-declared **AS-SCORED ONLY** +qualifier: collapsed to one binary per *concept* rather than per item, the same +statistic reads 0.585 (lower 0.434) and 0.605 (lower 0.456) — below the bar — and +the brief pre-committed those as the honest numbers, so they ride inside the +verdict string rather than sitting in prose. 0.5B reads `not shown` (11/41 = +0.268) off-gate, the first measured divergence between the subset-12's robustness +and the wider roster's. The strip physically contains 255 cells M1 recorded and +468 M3 recorded, and reproduced **all 723 bit-for-bit on every subject** before +reading a single new cell — so every sample size was knowable before the run, and +each landed exactly as pre-registered. What the gate did not ask: collateral +still concentrates on fragile **probes** rather than damaging **primes** (no +prime's row falls below 63/71 at 1.5B, while `copper` takes 6/12 and `mosquito` +8/12), and category-block collateral turns out to be real in the wider vocabulary +and *not* to dissolve with scale — reversing an M3 finding whose within-category +arm was 30/34 countries. + +**M3 PASSED 2026-07-28** — on the full 12 × 12 prime × probe matrix at the switch's home band, deleting one concept's direction silences that concept and leaves the other eleven almost untouched: at 1.5B the diagonal names **0/34** while the pooled off-diagonal names **363/374** (+0.971 [+0.867, +0.983]); at 3B 3/32 vs @@ -75,15 +96,13 @@ deterministic prefix rule on the recorded 3-token span (decision D9b, frozen before any run), M1's published numbers stand untouched, and the re-score is published beside them as a labelled reanalysis (D10a) in which the contrast survives on every subject and the dark categories light up. **The v1 chain -(M0–M3) is now closed.** In progress: close-out stage **M4, the vocabulary -collateral strip** (12 characterized directions × all 60 concepts), which -measures the one thing M3's near-white grid does *not* show — that deleting -France spares the other 48 concepts. Its brief is adversarially reviewed and -its decisions (D19–D22) were frozen 2026-07-29, before any runner code exists -— the lineage's freeze-before-code discipline. The S1 (7B) and S2 (lexical vs -semantic scope) stretches were declined for this repo and banked for a future -seed-hunt. Models: Qwen2.5-0.5B/1.5B/3B-Instruct, local MPS, forward-only; -core chain $0. +(M0–M3) closed 2026-07-28, and the close-out stage M4 landed 2026-07-29** — +measuring the one thing M3's near-white grid did *not* show, that deleting France +spares the other 48 concepts. Every stage's gate wording was frozen as code +before its first run, M4's included, and each stage re-certifies its predecessors +bit-for-bit rather than trusting them. The S1 (7B) and S2 (lexical vs semantic +scope) stretches were declined for this repo and banked for a future seed-hunt. +Models: Qwen2.5-0.5B/1.5B/3B-Instruct, local MPS, forward-only; whole project $0. Full brief: [`docs/KICKOFF.md`](docs/KICKOFF.md). The 12-idea backlog this was picked from: dim-stage diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md index b9dbc61..5a3f3e9 100644 --- a/docs/DECISIONS.md +++ b/docs/DECISIONS.md @@ -639,3 +639,230 @@ by a test that makes `AutoModelForCausalLM.from_pretrained` raise), re-asserts the loaded model's shape against that spec before any trial, and returns the parsed M1 artifact from `validate()` for `main()` to reuse. `m2_depth.py` itself stays untouched, per the certified-predecessor rule. + +## D19 — The strip frame: 12 primes × all 180 items (Kyle, 2026-07-29) + +Option (a): **the full 12 × 180 strip plus a full clean re-run — 2,340 cells per +subject.** M3 measured collateral *among the 12* and said so in its own Honest +limits: "the matrix measures collateral among 12 concepts, not across the +vocabulary … Nothing here shows that deleting France spares the other 48 M1 +concepts." M4 runs exactly that experiment: keep the 12 characterized directions +as the **primes** (the thing deleted), widen the **probes** (the thing asked +about) to all 60 M1 concepts — the whole frozen 180-item battery. Every cell is +the M3 recipe unchanged (late third, λ = 1, k = 1, D9(b) oracle), so cells differ +only in which direction was removed and which item was asked. + +Per subject: `clean` (180) + 12 × 180 = **2,340 cells**. The genuinely new +content is the **non-subset pool** — 492 / 852 / 1,008 cells at 0.5B / 1.5B / 3B, +of which 486 / 844 / 993 have never been measured by any milestone. + +**The re-certification surface is maximal, and two generations deep.** The strip +contains **255** cells per subject recorded in M1's artifacts (`clean` 180, the +12 subset concepts' `primed_late` 36, and the 39 `control_late` cells whose +control direction is a prime) **and 468** recorded in M3's (the 36 subset `clean` +cells + all 432 matrix cells — everything M3 ran except its 18 out-of-subset +control-extras, whose directions are not strip primes). The two surfaces overlap +in 90 cells, so phase 1 of the run grades their union — **633 of the 2,340** — +and both comparisons run on raw recorded strings (`greedy`, `greedy_3`; +`concept_mass` as texture) **before any new cell is read**. Any mismatch on the +certified stack is INVALID. This is D16's pattern applied against two artifact +sets at once. + +Rejected: **(b) 12 primes × the 144 non-subset items only** — saves ~20% of the +run and destroys the M3-overlap re-certification (no subset cells, no diagonal) +plus the in-strip recorded proxies; the one check that has caught nothing yet +*because it runs every time*, broken for one saved coffee break. **(c) the full +60 × 60 matrix** — a different, bigger question ("is *every* direction safe to +delete?") at ~4.7× the cost, with 48 primes no milestone has characterized; that +is a future stage's question, not this close-out's. + +**Consequence worth stating.** Every `clean` cell is M1-recorded and gating is +the deterministic clean arm under a frozen oracle, so the realized gate-arm n is +**fixed before the run** at 41 / 71 / 84. A run that disagrees is an INVALID +cross-check, not a power surprise. + +## D20 — The pre-committed wording package for M4 (Kyle, 2026-07-29) + +Option (a): **a survives-everything level gate on the new pool**, frozen as code +in `m4_strip.GATE_WORDING` before any run and written verbatim into every results +JSON. In brief: + +> **VOCAB-SPARING** iff, per subject: among the gated **non-subset** items, the +> proportion that **survives all 12** subset-direction deletions has its Wilson +> 95% lower bound at or above **0.5**. The bar is read **only when** the 468 +> M3-recorded and 255 M1-recorded cells reproduce their recorded outcomes +> bit-for-bit. The M4 verdict is the AND over 1.5B and 3B; 0.5B runs under its +> standing any-direction-damage frame and is never gate-bearing. Gate-arm +> n < MIN_N = 20 ⇒ UNDERPOWERED. + +Rejected: **(b) an M3-clause-(1)-style ordering gate extended to the strip** — +maximally comparable to M3, and it passes almost by inheritance (the diagonal is +0-to-3 hits at every subject), so it gates nothing; reported beside as +descriptive continuity instead. **(c) the conjunction of (a) AND (b)** — adds +nothing (b) doesn't already concede and re-opens the conjunction-degeneracy +obligation for no inferential gain. + +**Why a level gate at all.** The strip's question is a *level* question — "is the +floor high?" — not an ordering question; M3 already settled the ordering. The +per-item survives-all-12 outcome is a true binary, so the Wilson interval is +exact for it. This deliberately does **not** promote M3's cluster-mean floor +readout to gate-bearing: D17 froze that approximation as "acceptable only because +the qualifier is never dispositive," and M4 keeps *that* rationale intact by +reporting the cluster-mean floor beside, reference line 0.5, never dispositive. + +**The 0.5 constant is new, and owned as new.** M3's 0.5 was a floor on +cluster-collapsed **per-cell** survival; M4's is a bar on a **12-fold +conjunction**. The same digits mean opposite things across the two statistics: +under independence a conjunction of 0.5 corresponds to a per-cell +0.5^(1/12) ≈ **0.944**. Two consequences, stated rather than inherited. **(i) +Status changed** — M3's constant was itself uncalibrated and D17 tolerated that +only because the qualifier it scoped could never create or rescue a claim; M4 +makes a constant of the same value the *single dispositive gate*, so D17's +tolerance does not transfer. **(ii) The deletion count is half the bar** — at a +per-cell rate of 0.971 (M3's recorded 1.5B off-diagonal) the conjunction reads +≈ 0.70 and clears; at 0.94 it reads ≈ 0.48 and fails. So M4's 0.5 is a new, +deliberately lenient, uncalibrated constant, pre-registered before any new cell +was run and fitted to none, with its per-cell equivalence written into +`GATE_WORDING` itself so no write-up can quote it as M3's floor. + +**Why the re-certification clause lives inside the wording.** Every prior stage's +gate compared an intervened arm against another *measured* arm, so a dead +intervention could never pass one. M4's bar is single-clause and reads only the +off-target survival rate — read in isolation, an ablation that did nothing at all +would score ~100% survival and print VOCAB-SPARING. The strip's re-run of M3's +432 matrix cells catches exactly that, but that guarantee lived in D19's design +and an exit code, not in the sentence a write-up quotes. Now it cannot be quoted +out of its own precondition. + +**Why the realized proportion rides inside the verdict string.** `VOCAB-SPARING` +is the strongest possible phrasing of exactly the over-reading M4 exists to +correct, and the bar it names permits real damage: at the 1.5B pass point (44/71) +**27 of 71** gated non-subset items — 38% — are damaged by at least one of the 12 +deletions. Prose owns that, but prose is not what gets quoted; the label is. So +M4 carries the number **inside** the verdict string — M3's `ON A DAMAGED FLOOR` +move, applied to a level bar. + +**Amendment 1 (post-freeze, pre-run, ratified by Kyle 2026-07-29).** The verdict +string as first frozen carried only the **as-scored** proportion, so the two +pre-registered reads that can flip which number is honest stayed in prose — +reproducing the exact failure the realized-proportion resolution was meant to +prevent. The wording now carries a pre-declared **AS-SCORED ONLY** qualifier, +attached *conditionally by the runner* whenever a conservative read's Wilson lower +bound falls below 0.5 while the as-scored read's does not, and names a failing +label the single pass-label template had left unstated. + +**Amendment 2 (post-freeze, pre-run, ratified by Kyle 2026-07-29),** recorded +separately because Amendment 1's ratification quote covers only Amendment 1. +Four changes: **(i)** the failing label is the lineage's pre-committed null +`not shown`, not the assertive `NOT VOCAB-SPARING` Amendment 1 introduced — +failing a Wilson *lower* bound cannot establish the contrary (at 1.5B, k = 40 has +a point estimate of 0.563 *above* the bar with a straddling interval), and all +three predecessor runners emit `not shown`; **(ii)** 0.5B is scoped inside the +wording — the gate verdict is the AND over the two gate-bearing subjects and +0.5B's readout is never a gate claim; **(iii)** the qualifier attaches to a +**claim-level verdict only**, never to `NOT A RESULT` / `DEGENERATE` / +`UNDERPOWERED` — Amendment 1 had attached it to all of them, contradicting the +D17 rule it cites; **(iv)** both string templates are stated explicitly with a +fixed read order. The gate, its 0.5 bar, its arm and its re-certification +precondition are unchanged by both amendments. + +**The two pre-registered conservative reads, never dispositive.** + +1. **The residual-conservative read.** D9(b)'s owned span-truncation residual + re-enters a gate-bearing arm for the first time since M1: `oracle.py`'s frozen + wording closes with "None of the three concepts is in M2's subset," and M4's + wider probe side retires exactly that scope sentence. The gate statistic is + recomputed with every **residual cell** re-scored as a **miss**. A residual + cell is one whose recorded span, after stripping leading whitespace, **equals + the scored concept's spelling with nothing following it, compared + case-insensitively** exactly as `oracle.says_concept_prefix` compares. The + case rule *decides the set*: the recorded spans are `'Beetle'`, `'Butterfly'`, + `'Trumpet'` while the roster spellings are lowercase, so a case-**exact** + comparison would select **zero** cells and silently turn the mitigation into a + no-op; and the rule is *not* "the span fills the 3-token window", which every + recorded cell does by construction. On that reading the gate-arm residual cells + are **0 / 2 / 2** (1.5B `beetle-1`, `butterfly-1`; 3B `trumpet-3`, + `butterfly-1`). **Denominator: fail in place** — the arm stays at 41 / 71 / 84 + and a residual-affected item scores as a *failure*. The alternative (re-score + the clean cell too, so the item un-gates and the arm shrinks) is rejected: it + is the *less* conservative reading at the bar — same numerator, + `wilson(43, 71)` = 0.489 fails while `wilson(43, 69)` = 0.505 passes — and it + would break the power table's pre-registered n. `oracle.py` is not touched; + editing it would force re-runs of three milestones. +2. **The concept-level collapse.** Items cluster three-per-concept on the probe + side, so the same statistic is recomputed collapsed to one binary per + **concept** over the non-subset concepts with ≥ 1 gated item (n = 23 / 41 / 43). + +If the gate passes and either read does not, **that read's numbers are the honest +ones to quote** — and the AS-SCORED ONLY qualifier puts them in the verdict +string rather than leaving them in prose. + +**Degeneracy disposition.** D14/D17's wide-oracle guard, unchanged in mechanism +(pool the first tokens of an arm's non-produced cells only, share against the +arm's full cell count, threshold 0.5). Scope, enumerated — and this enumeration +**discharges PR #9 F1's conjunction-degeneracy obligation**, since M4's gate is +deliberately single-clause and the dispositive list therefore has exactly one +surviving arm: collapse in the pooled **non-subset off-target** arm ⇒ +**DEGENERATE**; collapse in the subset **diagonal** ⇒ **TAG only**; `clean` stays +off the dispositive list (the D14 F3 correction, carried); rows, columns and +per-pair cells are **texture**. + +**Verdict precedence, frozen** in `m4_verdict.strip_verdict()`: NOT A RESULT > +DEGENERATE > UNDERPOWERED > the level bar. Wrong-arm inputs exit INVALID before +the checkpoint loads; `--dry-run` validates and stops; `--limit` is smoke, never a +result; M4 refuses M1 **or** M3 artifacts that were themselves not results. + +## D21 — The five cross-mention cells (Kyle, 2026-07-29) + +Option (a): **keep them in the gate-bearing pool and report them as a named +confound row.** Widening the probe side to all 180 items surfaces a confound M3's +12-concept design never had: five (prime, item) pairs whose clue mentions a +prime's spelling. The list is frozen — **October→september-2, silver→flute-1, +China→jade-1, October→opal-2, Egypt→beetle-2** — and scanned with **D5's own +rule**: no word of the clue may *start with* the string, case-insensitive, plus +that string's `forbidden_forms` entries. The prefix rule is why the list is five +and not four: a whole-word scan misses `Egypt→beetle-2`, whose clue inflects the +prime ("Ancient **Egyptians** carved amulets of the scarab"). In those cells a +miss cannot distinguish collateral damage from "the clue's own text lost a word +it references." + +*Why keep them:* a confounded miss can only **lower** the floor, so the bias runs +**against** the gate — the one direction this project ships owned. Excluding them +would delete only cells that could hurt the claim, the anti-conservative move the +lineage never makes. Four cells of 852 cannot carry a verdict either way; what +they can do is mislead a *reader* of the column profiles, and the named row +prevents that. Four gate at both gate-bearing subjects (only `jade-1` at 0.5B); +`beetle-2` is ungated on all three and carries no gate-bearing cell today — it is +listed so a future re-gate cannot silently acquire one. + +Rejected: **(b) pre-registered exclusion from gate-bearing pools** — evidence +removal in the gate's favour. **(c) drop the five items entirely** — loses their +clean cells and their unconfounded prime cells for no reason. + +## D22 — Run-time instrument bars, widened to the full roster (Kyle, 2026-07-29) + +Option (a): **both D18 bars in `m4_strip.py`'s pre-trial validation, now over +every scored word — all 60 — plus the 12 direction words, with unit tests.** Span +bar: `max(len(tok(w)), len(tok(" " + w))) ≤ SPAN_TOKENS` on the subject's own +tokenizer, else INVALID. ASCII bar: every spelling pure ASCII, else INVALID. +Verified in advance for all 60 words on all three tokenizers (2026-07-28); the +run-time bar still runs, because D18's point was that the premise must hold *at +the moment of measurement*. + +Rejected: **(b) bars over the 12 primes only** (M3's literal scope) — the +soundness premise attaches to every **scored** word, and M4 scores 60, so a bar +checking 12 pins a fifth of the premise. **(c) widen `_BOUNDARY` now** — still +zero live cases, still the unforced version of the mistake D9 exists to prevent +(carried rejection). + +**What the span bar cannot catch, stated so it stays owned:** the D9(b) residual +passes the bar at exactly ≤ 3 tokens, which *is* the residual condition. The +residual is carried by disclosure plus D20's residual-conservative read, never by +a bar. + +**The `oracle._BOUNDARY` boundary-class decision is not owed here.** D18's named +trigger is a stage freezing a **non-ASCII** list; M4 adds no vocabulary — every +probe and every prime comes from M1's frozen 60, all pure-ASCII spellings. The +premise stays *pinned, not assumed*: the ASCII bar runs at run time, unchanged. +`oracle.py` is untouched, and becomes byte-shared by a fourth consumer (a +deviations-table row, on the same D9 rationale as the first three). diff --git a/docs/LEARNING.md b/docs/LEARNING.md index 87d25cf..4a8e4cb 100644 --- a/docs/LEARNING.md +++ b/docs/LEARNING.md @@ -282,3 +282,107 @@ earn their keep by being pre-registered as descriptive, not by being small. two simpler readouts this replaced and the specific way each one lies — and explain why it matters that the qualifier can never turn a "not shown" into a result. + +## M4 — the vocabulary collateral strip (2026-07-29) + +**The same digits can mean opposite things in two statistics.** M3 gated on a +**per-cell** floor of 0.5 — "of all the (deleted concept, probed item) cells, +what fraction still name the word?" M4 gates on a **conjunction**: "what fraction +of items survive *all twelve* deletions?" Those are not the same quantity, and a +0.5 bar on one is nothing like a 0.5 bar on the other: if the twelve deletions +were independent, a conjunction of 0.5 would require a per-cell rate of +0.5^(1/12) ≈ **0.944**. The 0.5B run made this concrete rather than theoretical — +on *the same cells*, the per-cell floor reads 32/41 → [0.633, 0.880], clearing +0.5 comfortably, while the conjunction reads 0.268. Reusing M3's constant because +it was "already pre-registered" would have imported a number that meant something +else. The fix wasn't to change the bar; it was to declare it **new**, own it as +lenient and uncalibrated, and write the per-cell equivalence into the frozen gate +wording so no later write-up can quote it as M3's floor. + +**A qualifier needs a *mechanism*, not just a sentence.** M3's ON A DAMAGED FLOOR +worked because the runner attached it conditionally, from the numbers. M4's first +draft borrowed the shape and dropped the mechanism: it stated in prose that two +conservative reads could be the honest ones, then froze a verdict string that +carried only the as-scored number. The round-4 review caught it, and the fix — +`AS-SCORED ONLY`, attached by the runner whenever a conservative read's lower +bound falls below the bar while the as-scored read's does not — turned out to +matter in the actual run: the concept-level collapse reads 0.585 (lower 0.434) at +1.5B and 0.605 (lower 0.456) at 3B, below the bar at *both* gate-bearing +subjects. Without the amendment, the published label would have been a bare +`VOCAB-SPARING` sitting on top of a number the brief itself had already called +the honest one. + +**Failing a lower bound is not proving the opposite.** A Wilson interval's lower +bound sitting under 0.5 says "this data doesn't establish the floor is above +0.5" — it does not say "the floor is below 0.5". At 1.5B, k = 40 of 71 would have +a *point estimate* of 0.563, above the bar, with an interval straddling it. That +is why the failing label is the lineage's `not shown` rather than a confident +`NOT VOCAB-SPARING`, and why 0.5B's `not shown` is a statement about evidence, +not a claim that 0.5B's vocabulary is unspared. (Its 73%-damaged rate is the +claim; the label is not.) + +**"Fail in place" — keep a mitigation from quietly becoming a rescue.** The D9(b) +oracle can't observe a word's closing boundary when the word fills the recorded +3-token span, so `Beetlejuice` would score as `beetle`. The conservative fix +re-scores every such cell as a miss. But *which denominator?* If you re-score the +clean cell too, the item un-gates and leaves the arm — and with the same +numerator, `wilson(43, 71)` = 0.489 **fails** while `wilson(43, 69)` = 0.505 +**passes**. A "conservative" reading that can raise the number is not +conservative. Fail-in-place — the item stays in the arm and scores as a failure — +keeps the read strictly one-way. + +**A selector rule can be worded so it selects nothing.** The residual rule went +through three drafts, and each broken version failed *silently*. "The span fills +the 3-token window" selects **every** recorded cell, because `greedy_continuation` +always decodes exactly three tokens. A **case-exact** "span equals the spelling" +selects **zero**, because the model emits `'Beetle'` and the roster spells +`beetle` — a mitigation that runs, reports, and does nothing. The rule that works +is "equals the spelling with nothing following it, compared case-insensitively, +exactly as the oracle compares." Both failure directions are now unit tests, +because neither would ever announce itself: one over-fires into noise, the other +under-fires into a no-op. + +**A one-sided sample can invert a finding.** M3 found category-block collateral +dissolving with scale — but M3's within-category arm was 30 of 34 pairs +*countries*. The strip's within-category arm samples ten categories, and the +effect not only survives at 3B, it is the largest structure in the run: 35/53 +within-category survival against 913/955 cross-category. Same lineage, same +operator, opposite conclusion — because the earlier arm's composition, not the +scale, was doing the work. + +**Re-certification is cheap enough to do twice.** M4's strip physically contains +255 cells M1 recorded and 468 M3 recorded, so it re-runs both and demands +bit-for-bit agreement before reading a single new cell — 723 comparisons per +subject, all exact, including every float of `concept_mass`. That has a second +payoff beyond drift-catching: because the 180 `clean` cells are inside the +certified set, and gating is deterministic, **every sample size in the run was +knowable before it started**. The power table stopped being a projection and +became a cross-check — a run that disagreed would have been INVALID, not a +surprise. + +**A single-clause gate needs a liveness check inside its own wording.** Every +earlier gate compared an intervened arm against another *measured* arm, so a +broken intervention could never pass. M4's bar reads only off-target survival — +so an ablation that did nothing at all would score ~100% survival and print +`VOCAB-SPARING`. The re-run of M3's 432 matrix cells is what forecloses that, but +that guarantee lived in a design doc and an exit code, not in the sentence a +write-up quotes. Putting it *inside* `GATE_WORDING` means the claim cannot be +quoted out of its own precondition. + +### Recall questions + +1. M3 and M4 both use a bar of **0.5**, and the M4 brief insists this is a + **new** constant with no provenance from M3. Explain what each 0.5 is a bar + *on*, why 0.5B's own numbers (32/41 → [0.633, 0.880] versus 0.268) are the + cleanest demonstration that the two are unrelated, and why D17's tolerance for + an uncalibrated constant did **not** transfer to M4. +2. Both gate-bearing subjects passed the gate and both verdicts carry + `AS-SCORED ONLY`. Say what fired it, what the honest number to quote is, and + why the qualifier is *mechanically incapable* of appearing beside a + `not shown` — then explain what would have been published had the round-4 + review not restored it. +3. The residual-conservative read re-scores cells as misses and keeps the + denominator at 41/71/84 rather than letting affected items un-gate. Name the + two rejected selector wordings and the opposite way each one fails, and show + with the brief's own numbers why the shrinking denominator would have been the + *less* conservative choice. diff --git a/docs/M4-BRIEF.md b/docs/M4-BRIEF.md index 6ac6e67..c0293d0 100644 --- a/docs/M4-BRIEF.md +++ b/docs/M4-BRIEF.md @@ -645,3 +645,113 @@ from `m3_matrix.py` (never from certified predecessors), verdict in residual is owned in the deviations table and carried by the residual-conservative read (D20), not by a rule change. Never an LLM judge, never free-text parsing (standing guardrail). + +--- + +## Results (2026-07-29) — **VOCAB-SPARING at 1.5B AND 3B, AS-SCORED ONLY** + +Three subjects, 2,340 cells each, ~50 minutes total on MPS, $0. `GATE_WORDING` +was frozen as code and the gates dry-run (six wrong-arm inputs exiting INVALID +with named reasons) before the first real cell. + +**The re-certification, two generations deep, on every subject:** M1 +**255/255** cells and M3 **468/468** cells reproduced bit-for-bit on the raw +recorded strings, with `concept_mass` exact on all 723 comparisons — ×3 subjects. +Because that surface includes all 180 `clean` cells, every realized n below was +knowable before the run, and every one landed exactly as pre-registered. + +| Readout | 0.5B | 1.5B | 3B | +|---|---|---|---| +| Gated items (full roster) | 69 | 105 | 116 | +| **Gate arm** (gated non-subset) | **41** | **71** | **84** | +| **Survives all 12** | **11/41 = 0.268** | **51/71 = 0.718** | **63/84 = 0.750** | +| Wilson 95% | [0.157, 0.419] | **[0.605, 0.810]** | **[0.648, 0.830]** | +| vs the pre-registered bar 0.5 | fails (off-gate) | **clears** | **clears** | +| Residual-conservative (fail in place) | 11/41 = 0.268 | 49/71 = 0.690, lower 0.575 | 62/84 = 0.738, lower 0.635 | +| Concept-level collapse | 4/23 = 0.174 | **24/41 = 0.585, lower 0.434** | **26/43 = 0.605, lower 0.456** | +| Pre-registered ceiling | 35/41 ✓ | 69/71 ✓ | 82/84 ✓ | +| Verdict | `not shown` | **VOCAB-SPARING — AS-SCORED ONLY** | **VOCAB-SPARING — AS-SCORED ONLY** | + +**M4 VERDICT: VOCAB-SPARING at 1.5B AND 3B — AS-SCORED ONLY.** Deleting any one +of the 12 characterized directions at the late third spares most of the +measurable wider vocabulary. M3's stated bound is closed: it is now measured, +not assumed, that deleting France spares the other 48 concepts' items. + +**The qualifier fired, and it earned its existence.** The concept-level collapse +— one binary per concept instead of per item — reads 0.585 (lower **0.434**) at +1.5B and 0.605 (lower **0.456**) at 3B: *below* the bar at both gate-bearing +subjects while the item-level gate clears it. Per D20's pre-commitment those are +**the honest numbers to quote**, and Amendment 1 is exactly why they ride inside +the verdict string instead of sitting in prose. The residual-conservative read +clears the bar at both (49/71, 62/84), so only one of the two reads fired. Had +the qualifier not been restored at round 4, the published label would have been a +bare `VOCAB-SPARING` over a conservative read the brief itself had already named +as the honest one. + +**Every pre-registered number landed.** Gate arms 41/71/84; concept counts +23/41/43; ceilings 35/41, 69/71, 82/84 with exactly the named misses (`july-1`, +`april-1`, `april-3`, `gold-1/2/3` at 0.5B; `july-3`, `venus-3` at 1.5B; +`guitar-2`, `neptune-1` at 3B). No degeneracy fired on the dispositive arm +(shares 0.022 / 0.011 / 0.011 against a 0.5 threshold) and the subset diagonal +carried no collapse tag. + +### The residual set was larger in the ablated arm than in the clean arm + +The clean-arm gate-arm residual cells were the pre-computed **0 / 2 / 2**. The +*run* recorded **0 / 27 / 21** residual cells in total (26 / 21 in the gate arm), +all on `beetle`, `butterfly` and `trumpet` — the three concepts `oracle.py`'s +frozen docstring names. This is precisely the case D20 wrote the selector for +("in the ablated cells the set is whatever the run records"), and it is why the +residual-conservative read moves the number at all: 51 → 49 at 1.5B, 63 → 62 at +3B. A case-exact selector would have found zero of them. + +### Descriptive findings the gate never asked for + +1. **Finding 1 generalizes out of sample — there *are* silver-like columns among + the 48.** Collateral still concentrates on fragile **probes**, not damaging + **primes**. No prime is a wrecking ball: at 1.5B every row lands between 63/71 + and 67/71, at 3B between 77/84 and 83/84. But specific probe columns collapse: + `copper` 6/12 and `mosquito` 8/12 at 1.5B; `eagle` 8/12, `platinum` 9/12 and + `trumpet` 18/36 at 3B. Meanwhile **32 of 53** gated columns at 1.5B and **33 + of 55** at 3B take *zero* collateral across all 12 deletions. The distribution + is bimodal, which is what makes the item-level and concept-level statistics + diverge. +2. **Category-block collateral is real in the wider vocabulary and does NOT + dissolve with scale** — the reverse of what M3 found inside the subset. + Within-category vs cross-category survival: 5/22 vs 382/470 at 0.5B, **22/29 + (0.759) vs 769/823 (0.934)** at 1.5B, **35/53 (0.660) vs 913/955 (0.956)** at + 3B. M3 saw within-category collateral dissolve by 1.5B, but M3's within arm + was 30/34 countries; the strip's within arm samples ten categories. +3. **0.5B is the first measured divergence between the subset's robustness and + the wider roster's.** 30 of its 41 gate-arm items (73%) are damaged by at + least one deletion, against 28% at 1.5B and 25% at 3B. Read under the standing + any-direction-damage frame, never as a gate claim — and consistent in advance + with M3's own 0.5B subset failing this bar in-statistic (19/28, lower 0.4934), + the 0/6 proxy and M1's 33/69 0.5B control cell. +4. **The two statistics disagree by design, and 0.5B shows it starkly.** The + M3-comparable cluster-mean per-cell floor reads **32/41 → [0.633, 0.880]** at + 0.5B — comfortably above 0.5 — while the 12-fold conjunction on the same cells + reads 0.268. Same subject, same cells, opposite side of the same constant. + That is the sharpest possible illustration of why D20 refused to inherit M3's + 0.5 and wrote the per-cell equivalence (0.5^(1/12) ≈ 0.944) into the frozen + wording. +5. **The five cross-mention cells did not carry the verdict, as predicted.** At + 3B all four gate-bearing cells named their concept; at 1.5B three named and + `China→jade-1` missed; `Egypt→beetle-2` remains ungated on all three subjects. + +### Honest limits + +- **The claim is sparing across the *measurable* vocabulary**, said exactly that + way. 25 / 7 / 5 of the 48 non-subset concepts gate zero items — a competence + selection that plausibly enriches for robust concepts and biases the floor + **upward**. +- **The concept-level read is below the bar at both gate-bearing subjects.** The + AS-SCORED ONLY qualifier is not decoration: an honest one-line summary is "the + item-level floor clears 0.5; the concept-level floor's lower bound does not." +- **The 0.5 constant is lenient and uncalibrated**, pre-registered before any new + cell and fitted to none. It is a floor bar, not an effect size; at the realized + 1.5B rate, 20 of 71 measurable items are still damaged by at least one of the + 12 deletions. +- **Prime-side correlation structure is now measured but not modelled** — the + survives-all-12 statistic is conservative under it, and the strip records the + per-pair cells a later stage could model. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index effd632..06e6a2e 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -10,5 +10,6 @@ with the code. | **M1 — breadth** | How much of the measurable vocabulary has an off-switch? | Pooled primed_late < control_late naming, CI-clean at 1.5B AND 3B | **PASSED** (2026-07-28: BREADTH-SPECIFIC at both gate-bearing subjects — 1.5B +0.656 [+0.517,+0.763], 3B +0.636 [+0.443,+0.759]; 0.5B also +0.447 [+0.275,+0.603] off-gate; anchor cross-check re-certified bit-for-bit on all three runs; see M1-BRIEF results) | | **M2 — localization + dose** | Where does the switch live; how much removal does it take? | Late-window effect CI-cleanly > early and middle, pooled | **PASSED** (2026-07-28: LATE-LOCALIZED at both gate-bearing subjects — 1.5B early−late +0.853 [+0.668,+0.936] / middle−late +0.794 [+0.603,+0.897], 3B +0.750 [+0.531,+0.857] / +0.688 [+0.463,+0.812]; 0.5B also LATE-LOCALIZED off-gate but on a raised damage floor; M1 cross-check re-certified bit-for-bit 108/108 cells ×3; dose is a dimmer, not a step; see M2-BRIEF results) | | **M3 — specificity matrix** | Diagonal suppression vs off-diagonal collateral | Diagonal > off-diagonal CI-clean at 1.5B AND 3B (D17: pooled AND within-category) | **PASSED** (2026-07-28: MATRIX-SPECIFIC at both gate-bearing subjects — 1.5B clause(1) +0.971 [+0.867,+0.983] / clause(2) +0.950 [+0.814,+0.978], 3B +0.881 [+0.731,+0.943] / +0.891 [+0.730,+0.947]; 0.5B also MATRIX-SPECIFIC off-gate; collateral floor clear on all three, so no ON A DAMAGED FLOOR qualifier; M1 cross-check re-certified bit-for-bit 108/108 cells ×3; collateral concentrates on fragile *probes* not damaging *primes*; see M3-BRIEF results) | +| **M4 — vocabulary collateral strip** (close-out, post-KICKOFF) | Does deleting one concept spare the *other 48*? | Gated non-subset items surviving all 12 deletions, Wilson lower ≥ 0.5, at 1.5B AND 3B (D20) | **PASSED** (2026-07-29: VOCAB-SPARING at both gate-bearing subjects — 1.5B 51/71 = 0.718 [0.605, 0.810], 3B 63/84 = 0.750 [0.648, 0.830]; **both carry AS-SCORED ONLY**, because the concept-level collapse reads 0.585 (lower 0.434) and 0.605 (lower 0.456) — below the bar; 0.5B `not shown` off-gate at 11/41 = 0.268; M1 **255/255** and M3 **468/468** cells re-certified bit-for-bit ×3; every pre-registered n and ceiling landed exactly; see M4-BRIEF results) | | S1 — scale (stretch) | Does specificity keep sharpening at 7B? | descriptive; 7B lens fit ≤ $15 on rented GPU | optional | | S2 — scope (stretch) | Token mute button or concept mute button? | descriptive; frozen alt-form lists | optional | diff --git a/m4_strip.py b/m4_strip.py new file mode 100644 index 0000000..a7458b2 --- /dev/null +++ b/m4_strip.py @@ -0,0 +1,1710 @@ +"""m4_strip.py — M4: the vocabulary collateral strip (does deleting one of the 12 +characterized directions spare the *wider* vocabulary, not just the other 11?). + +Cut from `m3_matrix.py`, which was cut from `m2_depth.py`, which was cut from +`m1_battery.py`, which was cut from the certified `m0_anchor.py` (untouched +post-gate, D2/M0). The measurement machinery is unchanged — same naming template, +same chat encoding, same k = 1 J-lens projection removal with the runtime +read-back, same frozen band and thirds, same D9(b) oracle, same Wilson/Newcombe +rulers — so the cells this strip shares with M1's *and* M3's recorded artifacts +must reproduce them bit-for-bit, which this runner checks before it reads a +single new cell (D19a). + +What M4 adds, each frozen in `docs/M4-BRIEF.md` before any run: + +- **the 12-prime x 180-item strip (D19a)** — M3 measured collateral among the 12 + subset concepts and stated its own bound plainly: nothing there shows that + deleting France spares the other 48 M1 concepts. M4 keeps the 12 characterized + directions as the PRIMES and widens the PROBES to all 60 M1 concepts — the full + frozen 180-item battery. Per subject: `clean` (180) + 12 x 180 = 2,340 cells. + Depth and dose stay fixed at the switch's home (the late third, lambda = 1, + k = 1), so every cell differs from every other in exactly one thing — which + direction was removed and which item was asked. +- **the level gate on the new pool (D20)** — VOCAB-SPARING iff, among the gated + NON-SUBSET items, the proportion surviving ALL 12 deletions has its Wilson 95% + lower bound at or above 0.5. Single-clause by design (the conjunction-degeneracy + obligation is discharged with exactly one surviving arm on the dispositive + list), with the pre-declared AS-SCORED ONLY qualifier when a pre-registered + conservative read falls below the bar and the as-scored read does not. +- **the re-certification, two generations deep (D19a)** — 255 cells per subject + are recorded in M1's artifacts and 468 in M3's; both sets are graded FIRST and + compared on raw recorded strings, and any mismatch exits INVALID. +- **the run-time instrument bars, widened (D22a)** — D18's span and ASCII bars + now run over every SCORED word (all 60) plus the 12 direction words, because + M4 scores 60 concepts where M3 scored 12. `oracle.py` itself is untouched: it + is byte-shared with `m1_rescore.py`, `m2_depth.py` and `m3_matrix.py` and must + stay identical. + +Owned divergences from `m3_matrix.py` (M4-BRIEF deviations table + dispositions): + +- **no control-extra directions.** M3 ran six out-of-subset control directions to + complete its 108-cell recorded surface. M4 probes all 180 items under all 12 + primes, so every M1 `control_late` cell whose control direction is a subset + member is already a strip cell; the other controls are not strip primes and are + not run. +- **`m3_matrix.GATE_WORDING` is not imported or edited** — M3's wording stays + byte-frozen with M3's artifacts. M4 freezes its own below, verbatim from D20 + including its two post-freeze ratified amendments. +- **the verdict precedence lives in `m4_verdict.strip_verdict()`**, imported here + rather than copied, so exactly one implementation of the pre-committed verdict + string exists (the `oracle.py` rationale, applied to wording). + +Run: uv run python -u m4_strip.py \ + --model-id Qwen/Qwen2.5-1.5B-Instruct \ + --lens lenses/qwen2.5-1.5b-instruct-n100.pt +""" +from __future__ import annotations + +import argparse +import json +import math +import os +import re +import time +from collections import Counter + +import torch +import transformers + +from harness import ( + COLLAPSE_SHARE, + FROZEN_BANDS, + MIN_N, + READBACK_TOL, + degeneracy, + encode_chat, + fail_invalid, + output_logits, + proportional_band, + rate_cell, + token_forms, +) +from intervention import Edit, ablate, jlens_vector +from m1_battery import ( + CLUES_PER_CONCEPT, + EXPECTED_CONCEPTS, + EXPECTED_ITEMS, + ITEMS_PATH, + load_items, + sub_band_thirds, +) +from m4_verdict import SURVIVAL_LOWER_BOUND, strip_verdict +from oracle import ( + ORACLE_WORDING, + SPAN_TOKENS, + says_concept_anywhere, + says_concept_prefix, +) +from stats import excludes_zero, newcombe_diff, wilson +from subject import SubjectModel + +#: D19(a): M3's primes — themselves M2's D11-stratified 12, reused VERBATIM. +#: Copied rather than imported, per the standing cut-from-your-predecessor rule; +#: a test pins it equal to M3's (and so, transitively, to M2's). +SUBSET = ( + "Brazil", "Canada", "China", "Egypt", "France", "Japan", # S1 (+Egypt, S3) + "Jupiter", "Mars", "piano", "violin", # S2 + "October", # S3 + "silver", # S4 +) +SUBSET_STRATA = { + "S1_hard_switch_core": ["Brazil", "Canada", "China", "France", "Japan"], + "S2_readout_unlocked": ["Jupiter", "Mars", "piano", "violin"], + "S3_leaky_switch": ["Egypt", "October"], + "S4_non_specific_anti_example": ["silver"], +} +#: Every word whose direction this run ablates. Unlike M3 there are no +#: control-extra directions: the strip probes all 180 items under all 12 primes, +#: so the recorded control cells that are strip cells are already covered. +PLANNED_DIRECTIONS = SUBSET + +NAMING_Q = ( # verbatim from the certified m0_anchor.py, via m1_battery/m2/m3 + 'Consider this sentence: "{clue}"\n\n' + "Which {noun} is the sentence describing? Answer with just the name." +) +#: M4 holds depth and dose fixed at the switch's home: full removal, k = 1. +STRIP_LAMBDA = 1.0 +#: The M1 conditions the shared cells reproduce, in M1's own names. +M1_SHARED_CONDITIONS = ("clean", "primed_late", "control_late") +#: D19: both cross-checks compare raw recorded strings, so they are +#: oracle-independent — D9 cannot soften them. +CELL_FIELDS = ("greedy", "greedy_3") +#: D20's M3-comparable reference line for the cluster-mean per-cell floor. This +#: is the readout M3 gated on; here it is reported beside and NEVER gating, so +#: D17's "acceptable only because the qualifier is never dispositive" rationale +#: for the binomial-on-bounded-fractions approximation stays intact. +CLUSTER_FLOOR_REFERENCE = 0.5 +P_NAME_FLOOR = 0.85 # paper's verbatim naming threshold — reported, never gating +#: The stack the anchor's bit-for-bit reproduction was certified on (M0, D1). +CERTIFIED_DEVICE = "mps" +CERTIFIED_PINS = {"torch": "2.13.0", "transformers": "5.13.1"} + +#: D21, frozen: the five (prime, item) pairs whose clue mentions a prime's +#: spelling, scanned with D5's OWN rule — no word of the clue may START WITH the +#: string, case-insensitive, plus that string's `forbidden_forms` entries — not +#: the narrower whole-word match, which misses `Egypt -> beetle-2` ("Ancient +#: EgyptIANS carved amulets of the scarab"). Kept in the gate-bearing pool +#: (D21a): a confounded miss can only LOWER the floor, so the bias runs against +#: the gate. `beetle-2` is ungated on all three subjects today and carries no +#: gate-bearing cell; it is listed so a future re-gate cannot silently acquire one. +CROSS_MENTION_PAIRS = ( + ("October", "september-2"), + ("silver", "flute-1"), + ("China", "jade-1"), + ("October", "opal-2"), + ("Egypt", "beetle-2"), +) + +#: D19(a)'s two re-certification surfaces, per subject, computed from the frozen +#: battery in `expected_recorded_cells()` and pinned here as the pre-registered +#: numbers the brief published: 255 M1-recorded cells (180 `clean` + 36 +#: `primed_late` + 39 `control_late` whose control direction is a prime) and 468 +#: M3-recorded cells (the 36 subset `clean` + all 432 matrix cells). +EXPECTED_M1_CELLS = 255 +EXPECTED_M3_CELLS = 468 +EXPECTED_M3_ITEMS = len(SUBSET) * CLUES_PER_CONCEPT # 36 + +#: Every gate's wording, frozen as code before any M4 run (project guardrail: +#: "pre-commit gates as code — wording included"). Written verbatim into the +#: results JSON so prose and code can never drift. M1's, M2's and M3's wordings +#: are NOT imported: each stays byte-frozen with its own artifacts. +GATE_WORDING = { + "oracle": ORACLE_WORDING, + "competence": ( + "D9(b) + D6(a) item-level greedy-span naming-only gate, carried verbatim " + "from M2/M3: an item enters the gated set iff its CLEAN naming span " + "satisfies the D9(b) oracle above. Gating is a property of the clean arm " + "alone, decided once per item and DIRECTION-INDEPENDENT, so every one of " + "this run's 12 prime columns shares ONE gated set — which is what makes " + "the rows, the columns and the pooled arms comparable. Gating is " + "per-subject. M1's first-token gate is computed and recorded beside it as " + "texture, never gating here. The paper-style verbatim-P rate (clean " + "concept mass >= 0.85) is reported alongside as texture, never gating; " + "note it is floor-pinned by construction for concepts whose emitted bare " + "spelling has no single-token form (the mass-channel scope, D13). OWNED, " + "AND THE REASON THE CLAIM SAYS WHAT IT SAYS: 25 / 7 / 5 of the 48 " + "non-subset concepts gate ZERO items at 0.5B / 1.5B / 3B, because the " + "model answers something else, answers correctly behind a modifier the " + "opening-word rule refuses, or misses on morphology — a COMPETENCE " + "SELECTION, not tokenizer geometry (under D9(b) the gate is a prefix on " + "the 3-token span, so it is insensitive to token count up to 3). " + "Confidently-named concepts are plausibly the robust ones, so the " + "selection biases the measured sparing floor UPWARD (F12's " + "selection-enrichment mechanism, now on the probe side). The claim is " + "sparing across the MEASURABLE vocabulary, said exactly that way." + ), + "sparing": ( + "D20(a), pre-committed before any M4 run, with both post-freeze " + "amendments ratified by Kyle 2026-07-29. VOCAB-SPARING iff, per subject: " + "among the gated NON-SUBSET items (concepts outside the 12-concept matrix " + "roster), the proportion that SURVIVES ALL 12 subset-direction deletions " + "— D9(b) naming success in every one of the item's 12 off-target cells — " + "has its Wilson 95% lower bound at or above 0.5. This 0.5 is a bar on the " + "12-FOLD CONJUNCTION, not on per-cell survival, and is NOT M3's per-cell " + "floor: under independence it corresponds to a per-cell survival of " + "0.5^(1/12) ~ 0.944, and its stringency depends on the deletion count " + "(12) as much as on the sparing rate. The bar is read ONLY when the 468 " + "M3-recorded and 255 M1-recorded cells in the strip reproduce their " + "recorded outcomes bit-for-bit; any mismatch is INVALID and there is no " + "verdict. The M4 verdict is the AND over 1.5B and 3B; 0.5B runs and is " + "reported under its standing any-direction-damage frame, never " + "gate-bearing. Gate-arm n < MIN_N = 20 => pre-declared UNDERPOWERED and " + "no claim (realized n = 41 / 71 / 84 from the recorded gated sets). " + "VERDICT STRING, PRE-COMMITTED: the label alone over-reads — clearing a " + "floor bar is compatible with a large minority of measurable items " + "damaged — so the verdict, whichever way it goes, carries its REALIZED " + "SURVIVAL PROPORTION in the same string: VOCAB-SPARING when the bar is " + "cleared and the lineage's pre-committed null 'not shown' when it is not " + "— never an assertive negative, because failing a Wilson LOWER bound does " + "not establish the contrary (m1_battery.py, m2_depth.py, m3_matrix.py all " + "emit 'not shown'). The gate verdict is the AND over the two gate-bearing " + "subjects; 0.5B's readout is reported in the same shape under its " + "standing any-direction-damage frame and is NOT a gate verdict, so a low " + "0.5B reading is never a 'not shown' gate claim. " + "CONSERVATIVE-READ QUALIFIER, PRE-DECLARED: two pre-registered reads can " + "fall below the bar when the as-scored read clears it — the " + "residual-conservative fail-in-place read and the concept-level collapse " + "— and the brief pre-commits that where they diverge, THEIR numbers are " + "the honest ones. So a CLAIM-LEVEL verdict additionally carries the " + "qualifier AS-SCORED ONLY, naming each such read and its number, whenever " + "any pre-registered conservative read's Wilson 95% lower bound is below " + "0.5 while the as-scored read's is not. The qualifier SCOPES A CLAIM AND " + "CAN NEVER CREATE OR RESCUE ONE (D17's rule, carried), so it attaches to " + "a bar-level verdict only and never to NOT A RESULT, DEGENERATE or " + "UNDERPOWERED — precedence has already withheld the claim there, leaving " + "it nothing to scope. THE TWO TEMPLATES, STATED ONCE AND IMPLEMENTED " + "VERBATIM — base: '