From 549d912f1fef0c8f73aad8ba8e449699e746e978 Mon Sep 17 00:00:00 2001 From: ksdisch Date: Wed, 29 Jul 2026 04:32:19 -0500 Subject: [PATCH 1/7] =?UTF-8?q?feat(m4):=20the=20vocabulary=20collateral?= =?UTF-8?q?=20strip=20=E2=80=94=20runner,=20verdict,=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M4 widens M3's probe side from the 12-concept subset to all 60 M1 concepts, keeping the 12 characterized directions as primes: 12 x 180 ablated + 180 clean = 2,340 cells per subject. It closes the bound M3's own Honest limits section stated — nothing so far showed that deleting France spares the other 48 concepts. - m4_strip.py, cut from m3_matrix.py (never from the certified m0_anchor.py), with its own byte-frozen GATE_WORDING taken verbatim from D20 including the AS-SCORED ONLY qualifier and both string templates. The 255 M1-recorded and 468 M3-recorded cells are graded FIRST — 633 cells of the 2,340, the union of the two surfaces — and any mismatch exits INVALID before a new cell is read. D22's span and ASCII bars now run over all 60 scored words plus the 12 direction words. oracle.py is untouched (a fourth byte-shared consumer). - m4_verdict.py owns strip_verdict() — precedence NOT A RESULT > DEGENERATE > UNDERPOWERED > the level bar, the failing label being the lineage null "not shown" — plus the cross-subject AND over 1.5B and 3B. The runner imports it, so exactly one implementation of the pre-committed verdict string exists. - test_m4.py (111 tests) pins the six-round review's hard-won lessons, every one a prose rule that broke when read literally: the residual selector selects exactly 2 / 2 / 0 gate-arm cells case-insensitively (case-exact selects 0; "fills the span" selects every cell), the conservative read fails in place at 41 / 71 / 84, AS-SCORED ONLY attaches at bar level only, and every realized n and ceiling is cross-checked against the recorded M1 artifacts. Gates dry-run clean on all three subjects; six wrong-arm inputs exit INVALID with named reasons; a --limit 18 smoke reproduced its reused cells bit-for-bit and wrote nothing. No real cell has been run yet. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01KkTj11WoNyTF9SGXvrxxVy --- m4_strip.py | 1710 +++++++++++++++++++++++++++++++++++++++++++++++++ m4_verdict.py | 329 ++++++++++ test_m4.py | 1256 ++++++++++++++++++++++++++++++++++++ 3 files changed, 3295 insertions(+) create mode 100644 m4_strip.py create mode 100644 m4_verdict.py create mode 100644 test_m4.py diff --git a/m4_strip.py b/m4_strip.py new file mode 100644 index 0000000..a7458b2 --- /dev/null +++ b/m4_strip.py @@ -0,0 +1,1710 @@ +"""m4_strip.py — M4: the vocabulary collateral strip (does deleting one of the 12 +characterized directions spare the *wider* vocabulary, not just the other 11?). + +Cut from `m3_matrix.py`, which was cut from `m2_depth.py`, which was cut from +`m1_battery.py`, which was cut from the certified `m0_anchor.py` (untouched +post-gate, D2/M0). The measurement machinery is unchanged — same naming template, +same chat encoding, same k = 1 J-lens projection removal with the runtime +read-back, same frozen band and thirds, same D9(b) oracle, same Wilson/Newcombe +rulers — so the cells this strip shares with M1's *and* M3's recorded artifacts +must reproduce them bit-for-bit, which this runner checks before it reads a +single new cell (D19a). + +What M4 adds, each frozen in `docs/M4-BRIEF.md` before any run: + +- **the 12-prime x 180-item strip (D19a)** — M3 measured collateral among the 12 + subset concepts and stated its own bound plainly: nothing there shows that + deleting France spares the other 48 M1 concepts. M4 keeps the 12 characterized + directions as the PRIMES and widens the PROBES to all 60 M1 concepts — the full + frozen 180-item battery. Per subject: `clean` (180) + 12 x 180 = 2,340 cells. + Depth and dose stay fixed at the switch's home (the late third, lambda = 1, + k = 1), so every cell differs from every other in exactly one thing — which + direction was removed and which item was asked. +- **the level gate on the new pool (D20)** — VOCAB-SPARING iff, among the gated + NON-SUBSET items, the proportion surviving ALL 12 deletions has its Wilson 95% + lower bound at or above 0.5. Single-clause by design (the conjunction-degeneracy + obligation is discharged with exactly one surviving arm on the dispositive + list), with the pre-declared AS-SCORED ONLY qualifier when a pre-registered + conservative read falls below the bar and the as-scored read does not. +- **the re-certification, two generations deep (D19a)** — 255 cells per subject + are recorded in M1's artifacts and 468 in M3's; both sets are graded FIRST and + compared on raw recorded strings, and any mismatch exits INVALID. +- **the run-time instrument bars, widened (D22a)** — D18's span and ASCII bars + now run over every SCORED word (all 60) plus the 12 direction words, because + M4 scores 60 concepts where M3 scored 12. `oracle.py` itself is untouched: it + is byte-shared with `m1_rescore.py`, `m2_depth.py` and `m3_matrix.py` and must + stay identical. + +Owned divergences from `m3_matrix.py` (M4-BRIEF deviations table + dispositions): + +- **no control-extra directions.** M3 ran six out-of-subset control directions to + complete its 108-cell recorded surface. M4 probes all 180 items under all 12 + primes, so every M1 `control_late` cell whose control direction is a subset + member is already a strip cell; the other controls are not strip primes and are + not run. +- **`m3_matrix.GATE_WORDING` is not imported or edited** — M3's wording stays + byte-frozen with M3's artifacts. M4 freezes its own below, verbatim from D20 + including its two post-freeze ratified amendments. +- **the verdict precedence lives in `m4_verdict.strip_verdict()`**, imported here + rather than copied, so exactly one implementation of the pre-committed verdict + string exists (the `oracle.py` rationale, applied to wording). + +Run: uv run python -u m4_strip.py \ + --model-id Qwen/Qwen2.5-1.5B-Instruct \ + --lens lenses/qwen2.5-1.5b-instruct-n100.pt +""" +from __future__ import annotations + +import argparse +import json +import math +import os +import re +import time +from collections import Counter + +import torch +import transformers + +from harness import ( + COLLAPSE_SHARE, + FROZEN_BANDS, + MIN_N, + READBACK_TOL, + degeneracy, + encode_chat, + fail_invalid, + output_logits, + proportional_band, + rate_cell, + token_forms, +) +from intervention import Edit, ablate, jlens_vector +from m1_battery import ( + CLUES_PER_CONCEPT, + EXPECTED_CONCEPTS, + EXPECTED_ITEMS, + ITEMS_PATH, + load_items, + sub_band_thirds, +) +from m4_verdict import SURVIVAL_LOWER_BOUND, strip_verdict +from oracle import ( + ORACLE_WORDING, + SPAN_TOKENS, + says_concept_anywhere, + says_concept_prefix, +) +from stats import excludes_zero, newcombe_diff, wilson +from subject import SubjectModel + +#: D19(a): M3's primes — themselves M2's D11-stratified 12, reused VERBATIM. +#: Copied rather than imported, per the standing cut-from-your-predecessor rule; +#: a test pins it equal to M3's (and so, transitively, to M2's). +SUBSET = ( + "Brazil", "Canada", "China", "Egypt", "France", "Japan", # S1 (+Egypt, S3) + "Jupiter", "Mars", "piano", "violin", # S2 + "October", # S3 + "silver", # S4 +) +SUBSET_STRATA = { + "S1_hard_switch_core": ["Brazil", "Canada", "China", "France", "Japan"], + "S2_readout_unlocked": ["Jupiter", "Mars", "piano", "violin"], + "S3_leaky_switch": ["Egypt", "October"], + "S4_non_specific_anti_example": ["silver"], +} +#: Every word whose direction this run ablates. Unlike M3 there are no +#: control-extra directions: the strip probes all 180 items under all 12 primes, +#: so the recorded control cells that are strip cells are already covered. +PLANNED_DIRECTIONS = SUBSET + +NAMING_Q = ( # verbatim from the certified m0_anchor.py, via m1_battery/m2/m3 + 'Consider this sentence: "{clue}"\n\n' + "Which {noun} is the sentence describing? Answer with just the name." +) +#: M4 holds depth and dose fixed at the switch's home: full removal, k = 1. +STRIP_LAMBDA = 1.0 +#: The M1 conditions the shared cells reproduce, in M1's own names. +M1_SHARED_CONDITIONS = ("clean", "primed_late", "control_late") +#: D19: both cross-checks compare raw recorded strings, so they are +#: oracle-independent — D9 cannot soften them. +CELL_FIELDS = ("greedy", "greedy_3") +#: D20's M3-comparable reference line for the cluster-mean per-cell floor. This +#: is the readout M3 gated on; here it is reported beside and NEVER gating, so +#: D17's "acceptable only because the qualifier is never dispositive" rationale +#: for the binomial-on-bounded-fractions approximation stays intact. +CLUSTER_FLOOR_REFERENCE = 0.5 +P_NAME_FLOOR = 0.85 # paper's verbatim naming threshold — reported, never gating +#: The stack the anchor's bit-for-bit reproduction was certified on (M0, D1). +CERTIFIED_DEVICE = "mps" +CERTIFIED_PINS = {"torch": "2.13.0", "transformers": "5.13.1"} + +#: D21, frozen: the five (prime, item) pairs whose clue mentions a prime's +#: spelling, scanned with D5's OWN rule — no word of the clue may START WITH the +#: string, case-insensitive, plus that string's `forbidden_forms` entries — not +#: the narrower whole-word match, which misses `Egypt -> beetle-2` ("Ancient +#: EgyptIANS carved amulets of the scarab"). Kept in the gate-bearing pool +#: (D21a): a confounded miss can only LOWER the floor, so the bias runs against +#: the gate. `beetle-2` is ungated on all three subjects today and carries no +#: gate-bearing cell; it is listed so a future re-gate cannot silently acquire one. +CROSS_MENTION_PAIRS = ( + ("October", "september-2"), + ("silver", "flute-1"), + ("China", "jade-1"), + ("October", "opal-2"), + ("Egypt", "beetle-2"), +) + +#: D19(a)'s two re-certification surfaces, per subject, computed from the frozen +#: battery in `expected_recorded_cells()` and pinned here as the pre-registered +#: numbers the brief published: 255 M1-recorded cells (180 `clean` + 36 +#: `primed_late` + 39 `control_late` whose control direction is a prime) and 468 +#: M3-recorded cells (the 36 subset `clean` + all 432 matrix cells). +EXPECTED_M1_CELLS = 255 +EXPECTED_M3_CELLS = 468 +EXPECTED_M3_ITEMS = len(SUBSET) * CLUES_PER_CONCEPT # 36 + +#: Every gate's wording, frozen as code before any M4 run (project guardrail: +#: "pre-commit gates as code — wording included"). Written verbatim into the +#: results JSON so prose and code can never drift. M1's, M2's and M3's wordings +#: are NOT imported: each stays byte-frozen with its own artifacts. +GATE_WORDING = { + "oracle": ORACLE_WORDING, + "competence": ( + "D9(b) + D6(a) item-level greedy-span naming-only gate, carried verbatim " + "from M2/M3: an item enters the gated set iff its CLEAN naming span " + "satisfies the D9(b) oracle above. Gating is a property of the clean arm " + "alone, decided once per item and DIRECTION-INDEPENDENT, so every one of " + "this run's 12 prime columns shares ONE gated set — which is what makes " + "the rows, the columns and the pooled arms comparable. Gating is " + "per-subject. M1's first-token gate is computed and recorded beside it as " + "texture, never gating here. The paper-style verbatim-P rate (clean " + "concept mass >= 0.85) is reported alongside as texture, never gating; " + "note it is floor-pinned by construction for concepts whose emitted bare " + "spelling has no single-token form (the mass-channel scope, D13). OWNED, " + "AND THE REASON THE CLAIM SAYS WHAT IT SAYS: 25 / 7 / 5 of the 48 " + "non-subset concepts gate ZERO items at 0.5B / 1.5B / 3B, because the " + "model answers something else, answers correctly behind a modifier the " + "opening-word rule refuses, or misses on morphology — a COMPETENCE " + "SELECTION, not tokenizer geometry (under D9(b) the gate is a prefix on " + "the 3-token span, so it is insensitive to token count up to 3). " + "Confidently-named concepts are plausibly the robust ones, so the " + "selection biases the measured sparing floor UPWARD (F12's " + "selection-enrichment mechanism, now on the probe side). The claim is " + "sparing across the MEASURABLE vocabulary, said exactly that way." + ), + "sparing": ( + "D20(a), pre-committed before any M4 run, with both post-freeze " + "amendments ratified by Kyle 2026-07-29. VOCAB-SPARING iff, per subject: " + "among the gated NON-SUBSET items (concepts outside the 12-concept matrix " + "roster), the proportion that SURVIVES ALL 12 subset-direction deletions " + "— D9(b) naming success in every one of the item's 12 off-target cells — " + "has its Wilson 95% lower bound at or above 0.5. This 0.5 is a bar on the " + "12-FOLD CONJUNCTION, not on per-cell survival, and is NOT M3's per-cell " + "floor: under independence it corresponds to a per-cell survival of " + "0.5^(1/12) ~ 0.944, and its stringency depends on the deletion count " + "(12) as much as on the sparing rate. The bar is read ONLY when the 468 " + "M3-recorded and 255 M1-recorded cells in the strip reproduce their " + "recorded outcomes bit-for-bit; any mismatch is INVALID and there is no " + "verdict. The M4 verdict is the AND over 1.5B and 3B; 0.5B runs and is " + "reported under its standing any-direction-damage frame, never " + "gate-bearing. Gate-arm n < MIN_N = 20 => pre-declared UNDERPOWERED and " + "no claim (realized n = 41 / 71 / 84 from the recorded gated sets). " + "VERDICT STRING, PRE-COMMITTED: the label alone over-reads — clearing a " + "floor bar is compatible with a large minority of measurable items " + "damaged — so the verdict, whichever way it goes, carries its REALIZED " + "SURVIVAL PROPORTION in the same string: VOCAB-SPARING when the bar is " + "cleared and the lineage's pre-committed null 'not shown' when it is not " + "— never an assertive negative, because failing a Wilson LOWER bound does " + "not establish the contrary (m1_battery.py, m2_depth.py, m3_matrix.py all " + "emit 'not shown'). The gate verdict is the AND over the two gate-bearing " + "subjects; 0.5B's readout is reported in the same shape under its " + "standing any-direction-damage frame and is NOT a gate verdict, so a low " + "0.5B reading is never a 'not shown' gate claim. " + "CONSERVATIVE-READ QUALIFIER, PRE-DECLARED: two pre-registered reads can " + "fall below the bar when the as-scored read clears it — the " + "residual-conservative fail-in-place read and the concept-level collapse " + "— and the brief pre-commits that where they diverge, THEIR numbers are " + "the honest ones. So a CLAIM-LEVEL verdict additionally carries the " + "qualifier AS-SCORED ONLY, naming each such read and its number, whenever " + "any pre-registered conservative read's Wilson 95% lower bound is below " + "0.5 while the as-scored read's is not. The qualifier SCOPES A CLAIM AND " + "CAN NEVER CREATE OR RESCUE ONE (D17's rule, carried), so it attaches to " + "a bar-level verdict only and never to NOT A RESULT, DEGENERATE or " + "UNDERPOWERED — precedence has already withheld the claim there, leaving " + "it nothing to scope. THE TWO TEMPLATES, STATED ONCE AND IMPLEMENTED " + "VERBATIM — base: '