issue #667 followup: tf-margin gate→behavior DV swap#587
Draft
superkaiba wants to merge 26 commits into
Draft
Conversation
Port i537_contexts.py verbatim from issue-537 (#641 SHA fff2e92); all deps (i406_conditions MARKER_ID/MARKER_TEXT, personas.PERSONAS) already on main, no library drift. The ported registry + frozen P0 inputs reproduce #537's registry_hash f12061d6... exactly (== G_meta pin), validating the context-construction is byte-faithful. gate_chain.py: realized activation gate (B1), rank-one residual + stacked- Delta_V SVD (A3.8), whitened key-query gate + the B3 reduction unit test (A3.9/A3.10), A3.7 source-write reads, partial-Spearman (A3.6, C10), family-clustered bootstrap + shuffled null. Pure-CPU, GPU-free, unit-testable. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Loads #537 adapter as PeftModel (rsLoRA gauge asserts), teacher-forces the frozen base greedy R through base+trained, reads mean-over-response v0/v+ at L14/{7,21}; base-side c_C/c_Cp (all 28 layers) for the whitened gate; t+/t- (A3.7) split from the untagged #537 mix by source-context prompt prefix; fact r_B re-extracted fresh (#594 diff-in-means, absent from #658 r_b.pt). Content-hygiene safe for em harmful rows (counts + activations only, no text). CPU-verified: all 4 probe pools load at correct counts; build_messages produces correct chat structures across all 7 context families. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
B3 reduction unit test gates A3.9/A3.10 (HALT on failure). Reads the per-cell npz store + #537 G_meta (git_commit pin) + #658 sigma_c/r_b (probe_pool_hash pin); writes A3_{6..10}.json with reproducibility metadata. fact r_b read from the store's re-extracted r_b_fact; em/syco from #658. --skip-store-pin runs a synthetic-store CPU smoke (no HF, no model). CPU-verified end-to-end on a structured synthetic store: A3.7 cos_pos=1.0 beats null; A3.8 sigma1_frac=0.996 (rank-one by construction), chance=0.25; A3.6 runs for em (broad_em r_b) + fact (re-extracted); B3 PASS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dispatcher: prefetch (SHA-pins + B3 + rsLoRA parity probe) -> extract
(CVD-pinned per-source-adapter subprocess waves + HF tensor upload) ->
analysis (A3.6-A3.10). Every phase derives its cell subset from the SAME
--behaviors/--sources/--targets filters; smoke IS the sweep scaled down.
Emits [phase=...] + terminal [phase=done] + poll_pipeline-conformant sentinel;
explicit env={**os.environ} + load_dotenv() at main-top.
Phase-0 prefetch CPU-verified end-to-end: B3 PASS, registry_hash f12061d6 ==
G_meta pin, G_meta git_commit 34f2502 pin OK, #658 probe_pool_hash ad687bec
pin OK, rsLoRA parity probe em r=32/a=256/use_rslora=True. Dry-run extract:
CVD pinned per cell, terminal [phase=done], sentinel conforms to
_SENTINEL_REQUIRED_KEYS.
G_meta load: fail-loud (committed in git, NOT on HF; sparse worktrees add it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
26 tests: B3 reduction unit test (multi-dim), whitened-gate self=1 + identity=cosine + diag-anisotropy, realized_gate exact/orthogonal/zero-raises, stacked-ΔV SVD rank-one-recovery + full-rank-chance, A3.7 cos_pos/frac_ctx, partial-Spearman confound removal/signal recovery, family_of grammar, clustered bootstrap, shuffled null. Extract CPU-runnable arithmetic (GPU-bound carve-out item 1): _row_to_messages (em chat / prompt-completion / unknown-raises), _system_signature (question-invariant), mean-resp span resolution + dual-side + c_C all-layers, all against the REAL Qwen-2.5-7B tokenizer + a 2-layer CPU stub (no 7B load). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fix) Root cause of a >2x compute deviation: the extractor generated the frozen base R per-prompt via HF generate (~43 GPU-h projected, 6x the plan's 7.2 — and a CLAUDE.md "use vLLM for generation" violation). Re-architected to Phase A (vLLM batched-generate ALL R for a source's targets x probes from BASE, then reap the engine per gotchas) -> Phase B (HF load base+trained, teacher-force the pre-generated R for mean-resp v0/v+). Projection now ~1.3 GPU-h (0.18x planned). HF greedy stays for the tiny fact-r_B path + CPU-only smoke fallback. 26 gate-chain tests still pass (CPU arithmetic unaffected). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…3.9 full grid
Round-2 code-review fixes (3 BLOCKER + 4 MAJOR + 1 CONCERN):
- BLOCKER a310-oracle-gplus-not-extracted: extract post-FT key/query
(c_C_postft/c_Cp_postft) under the loaded PeftModel; A3.10 computes the
oracle g+=(k+,q+,M0) vs realized, g0 vs realized (distinct), g0 vs oracle,
and key/query drift — no longer a relabel of A3.9.
- BLOCKER rslora-parity-gate-skipped-on-production-extract: phase_extract runs
the NUMERIC diagonal-write parity probe (g_self==1 + write-ratio HALT) as its
mandatory first step, so the production `extract` launch hits the gate.
- BLOCKER cached-artifact-coverage-unverified: fail-loud validators for r_b
columns/recipe/layers, sigma_c shape/keys/layers, G_meta per-cell fields, and
G_tensor cid equality; removed the silent continue on a missing G_meta cell.
- MAJOR a39-ablation-incomplete: A3.9 full 3x3 key{c_C,psi_t,psi_delta} x
metric{I,diag,whitened} grid + shuffled-key/query controls + E0 + predict-mean
baselines.
- MAJOR cosine-baseline-not-cosine: true_cosine helper (both norms); A3.9
reports it as the cosine baseline; the self-normalized I metric kept + renamed.
- MAJOR a36 CI/null mismatch: A3.6 bootstrap + null now on the partial statistic.
- MAJOR lambda sweep: condition-number sweep recorded in A3.9.
- CONCERN a37-icl-source-tpos-tneg-gap: F3-ICL positives matched by demo
role-pattern; A3.7 reports per-source t+/t- coverage + dropped sources.
Tests: 40/40 pass (oracle, validators-fail-loud, partial-bootstrap, lambda
sweep, true-cosine, ICL split). Smoke: Phase-0 prefetch CPU PASS on real
artifacts (pins + standalone validators + parity config); Phase-2 analysis PASS
on synthetic store (full grid + oracle + drift); dispatcher dry-run CVD-pinned +
sentinel + [phase=done].
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-uses-tneg) Round-3 BLOCKER fix. frac_ctx = ||v0(C) - v0(C_neg)|| / ||delta_contra|| (R3-1) was reading t_neg (the negative-persona ANSWER activation that feeds delta_contra) instead of v0(C_neg) (the negative persona's base-CONTEXT vector). - extract: new extract_v0_C_neg() captures the negative-panel base-context activation under each negative persona's prompt (no answer), matched to the v0(C) mean-over-response recipe, panel-averaged; wired into Phase A (neg base R gen via vLLM) + stamped on the primary-layer source payload as v0_C_neg. - analysis: run_a37 reads v0_C_neg from the store for frac_ctx; t_neg still feeds delta_contra only. Absent v0_C_neg -> NaN frac_ctx (non-silent) + v0_cneg_missing / n_sources_with_v0_cneg coverage fields. - tests: regression test on a synthetic store with distinct t_neg vs v0_C_neg proves frac_ctx reads v0(C_neg) exactly (fails pre-fix with the t_neg value); plus an absent-v0_C_neg NaN test. Open CONCERN frozen-r-cache-not-used left as a scope caveat (grep comment near the vLLM R-gen call site). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…subprocess) The rsLoRA NUMERIC parity probe ran in-process in the dispatcher parent, loading the base model + PeftModel and initializing CUDA before the per-cell extract subprocesses fork (vLLM forks its own EngineCore worker). A live CUDA context in the parent poisoned that fork chain -> RuntimeError: Cannot re-initialize CUDA in forked subprocess (bug_class dispatcher_cuda_init_before_subprocess_fork). Fix: the GPU numeric probe now runs in a one-shot subprocess (the new `parity-probe` CLI entrypoint) so the dispatcher parent never touches CUDA. HALT semantics preserved: a non-zero child rc (failed parity assert / crash) or a missing result file re-raises before any extraction wave. The CPU-only smoke path stays in-process (gauge config check, no CUDA, no subprocess). Regression test tests/test_issue667_dispatcher.py pins that the GPU path dispatches a subprocess (parent never calls _numeric_rslora_parity, CUDA stays uninitialized) and the HALT gate fires on rc!=0 / missing result. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ess (gotcha #26) Round-4 fixed the dispatcher CUDA-init-before-fork hazard (parity probe -> subprocess). This closes the OTHER half: issue667_extract.py constructs vllm.LLM() inside vllm_generate_R AFTER main() touches AutoTokenizer, which poisons the EngineCore fork under vLLM V1's default fork() worker method (silent worker death 1-4s after init). Per gotcha #26: - Set os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn") at the extract module top, before any `import vllm`. - Belt-and-suspenders: inject the same var into the dispatcher's per-cell extract subprocess env (guards a future import-reorder). - Regression test: (a) runtime env pin after import, (b) per-cell subprocess env carries spawn + threads to subprocess.run, (c) mechanizable scan that any issue667 extract/dispatch script importing vllm + AutoTokenizer sets the spawn guard above the vllm import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ct fix) The r4 CUDA-isolation refactor moved the rsLoRA parity probe into a one-shot `parity-probe` subprocess: the launch site passes `--behavior <x>` and main() reads `args.behavior`, but the parser only ever registered `--behaviors` (plural, for extract/analysis). The subprocess died deterministically with `AttributeError: 'Namespace' object has no attribute 'behavior'` -> rc=1 -> the dispatcher's fail-loud HALT. Fix: register `--behavior` (singular) as a dedicated parity-probe arg next to the existing `--source`/`--seed`/`--result-out`. Leaves `--behaviors` (plural) untouched. Adds two regression tests that drive main()'s parity-probe argparse end-to-end (GPU work + credential check mocked, CPU-only) — both fail pre-fix with the exact AttributeError and pass post-fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mpleted cells
Fix the fact-behavior extract crash (AttributeError: 'dict' object has no
attribute 'strip' at i537_contexts._casualize): pool_fact_30.json's
ood_framings rows are {"framing","question"} dicts, not strings, so
load_eval_probes("fact") returned a mixed list that every downstream
message-builder threaded into a chat content / _casualize as if it were a
string. New _probe_text normalizer flattens any dict probe (question ->
prompt -> text) at the single load site so all consumers see a flat
list[str].
Add resume-skip (default ON, --no-resume-skip to force a full re-run) to the
extract dispatcher: cells whose .npz tensors already exist under
<TENSORS_DIR>/<behavior>/<source>_seed42 are NOT relaunched. The relaunch
will be FAST — 32/64 cells (all em + all sycophancy) are already on the pod's
disk and will be skipped; only the un-extracted fact cells run.
Tests: _probe_text on the real ood_framings dict shape + defensive variants;
build_messages_for over a dict probe through the _casualize F4 path (no
AttributeError); phase_extract resume-skips a completed cell and re-runs it
under --no-resume-skip; _cell_already_extracted predicate (empty dir != done).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…flag
Round-7's resume-skip predicate `any(cell_dir.glob("*.npz"))` counted any
non-empty cell dir as done, silently accepting a partial dir left by a
mid-cell crash (the extractor writes per-(target,layer) .npz incrementally).
Default-on silent partial-acceptance is the prohibited fail-fast class.
- issue667_extract.write_cell_done_sentinel: atomic `.done` (tempfile +
os.replace) written ONLY after every (target,layer) .npz lands; payload
records targets/layers/expected-count for future validation.
- dispatcher _cell_already_extracted: gate resume-skip on `.done` presence,
never on stray .npz.
- `--backfill-sentinels`: one-shot migration that validates each cell's .npz
complement and writes the sentinel only for complete cells (incomplete ones
reported + re-extracted). Lets the relaunch resume-skip the 32 complete
on-disk cells.
- tests: flip the unsafe predicate test (.npz w/o sentinel = NOT done), add a
partial-cell resume-skip regression + backfill complete/incomplete/idempotent
tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…mpty-acc class) run_extraction now calls assert_full_npz_complement(cell_dir, targets, layers) immediately before write_cell_done_sentinel, mirroring the --backfill-sentinels complement check (issue667_dispatch._expected_npz_for_cell). A target whose probes all return empty responses skips its per-layer .npz write (_extract_one_target's `if not acc[li][0]: continue`); without this gate the unconditional sentinel would stamp a TRUSTED .done over an incomplete cell that the resume-skip then silently treats as complete (round-8 BLOCKER resume-skip-empty-acc-unconditional-sentinel). The gate raises RuntimeError listing the missing files, writes NO sentinel, and leaves the partial .npz on disk so the dispatcher's resume-skip re-extracts the cell. Regression test test_run_extraction_empty_acc_target_raises_and_writes_no_sentinel drives run_extraction with one empty-acc target (CPU-only, heavy callees stubbed) and asserts (a) RAISES, (b) no .done, (c) partial .npz preserved — fails pre-fix, passes post-fix. Plus a direct unit test on the helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…hain inverse/diag caching
…n-space wording, a36/a37 label-clip fix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ing for gate-vs-behavior)
…gate-vs-tf-margin round Same-issue follow-up tf-margin-gate-vs-behavior (plan v6): swap the gate->behavior DV from #537's judged rate G to the continuous teacher-forced fixed +/- completion margin (tf_margin_leak), computed 2-index (16 sources x 30 targets) on the #537 grid. - scripts/issue667_tf_margin_extract.py: vendors score_answer_logprobs_batched + _assistant_suffix_len + build_fixed_pairs BYTE-IDENTICAL from the unmerged issue-722-tf-margin branch (AST-verified), ONE substitution messages_for_instance -> i537_contexts.build_messages via build_messages_for; adds extract_tf_margins_2index (base-margin cache reused across sources). - scripts/issue667_tf_margin_analysis.py: vendors _spearman + clustered_bootstrap_spearman (N_BOOT=2000); run_gate_vs_tf_margin join + g0-correctness gate (reproduce #667's 0.13/0.16/0.40 within +/-0.02, HALT rc=3 on miss) + measurement-validity gate (Spearman(tf,G) point est >0 AND CI excludes zero per behavior). - scripts/issue667_build_fact_pool.py: the one new data artifact -- on-policy fact fixed +/- pool, Sonnet judge >50 pos / <50 neg, drop REFUSAL/malformed both arms, cap 40/side floor-N equalize-down, HF upload. - scripts/issue667_tf_margin_apply_parity.py: Phase-0.5 rsLoRA apply-parity probe (reproduce #537 committed E_plus_source within +/-0.10; HALT rc=3 on drift). - scripts/issue667_tf_margin_dispatch.py: PASS_UNIFIED dispatcher (parity -> fact-pool -> extract -> analysis); wave_size = min(detected device_count, --n-gpus) with a RAISE-on-0-GPU guard (the a36 wave-size bug fix); [phase=...] + [phase=done] + _SENTINEL_REQUIRED_KEYS sentinel. - scripts/issue667_tf_margin_figure.py: hero gate-vs-tf-margin scatter + validation bars. - tests: vendor byte-identity, the 2-index driver + base-cache reuse, both gates on synthetic pass/fail, and the wave-size regression (detected 8 -> 8, clamp, RAISE on 0). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…coverage gate + aggregate deliverables Addresses the round-1 reconciled FAIL (2 BLOCKERs + 2 CONCERNs): - BLOCKER 1 (fact-pool-adapter-generation-base-only): the 'adapter' arm now generates through the loaded `trained` PeftModel via HF .generate() (`_hf_generate_with_adapter`), not the base-only `vllm_generate_R` path; only the base arm keeps vLLM on GPU. Per-arm generation-provenance + adapter path in the pool provenance; hard `model is trained` guard. - BLOCKER 2 (tf-margin-cell-coverage-not-enforced): `run_gate_vs_tf_margin` enforces the full expected off-diagonal key set per behavior BEFORE writing the headline — HARD-FAIL (CellCoverageError -> rc=4 -> epm:failure tf_margin_cell_coverage_incomplete) for em/sycophancy always and for fact unless dropped; SOFT-DROP for a dropped fact arm. - CONCERN 1 (fact-arm-floor-drop-not-enforced): below-floor build writes a DROPPED_FROM_HEADLINE.sentinel; the dispatcher skips the fact extract and the analysis soft-drops fact (via --fact-dropped or the sentinel). - CONCERN 2 (missing-aggregate-tf-margin-deliverables): the analysis writes margins.json + tf_margin_leak.json (plan v6 §6.5 primary_deliverable). Regression tests: tests/test_issue667_build_fact_pool.py (adapter-vs-base labeling + sentinel) + new BLOCKER 2/CONCERN 2 cases in tests/test_issue667_tf_margin_analysis.py. 92 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ale-percell headline carry) BLOCKER fact-arm-drop-stale-percell-still-carried: the fact soft-drop was nested inside the cell-coverage `if missing:` block, so it only fired when the store had missing cells. On a resume (resume_skip=True dispatcher default, #667 multi-run history) a stale-but-COMPLETE per-cell fact store leaves `missing` empty, and fact reached the headline with fact_softdropped=False — contradicting plan v6 §4.3, which mandates the drop as a sentinel invariant. Fix: unconditional `behavior == "fact" and fact_dropped` guard at the TOP of the per-behavior loop, before the correctness/coverage/stat compute; `continue` skips fact entirely. The coverage-gated fact branch is now dead-for-drop but kept harmless. Regression tests fail pre-fix / pass post-fix. Also hardens the _generate_completions adapter-arm guard with a `model is not base` clause (catches a mis-load / loop-wiring drift that returns the base as "trained") + a negative test (Codex round-2 Minor). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…al Betley judge Fix the round-3 sweep-launch false-fail (epm:failure v8 / epm:strategy-pivot v2): the +/-0.10 apply-parity tolerance was mis-calibrated for N_SAMPLES=10 (Wilson-CI half-width ~0.27 at rate 0.7 -> guaranteed false-fail under a perfect apply). - N_SAMPLES 10 -> 100 (Wilson half-width ~0.09 < 0.10 tol; ~3-4 min of judge calls). - Replace the hand-rolled `misaligned` rubric with the project's canonical Betley two-axis EM judge (eval/alignment.format_betley_judge_user_msg + parse_judge_json; aligned<30 AND coherent>=50), drop-never-coerce on REFUSAL/CODE/out-of-range. Addresses deferred CONCERN apply-parity-betley-judge-restatement. - Extract decide_parity() pure helper; record judge_prompt/judge_threshold in the provenance JSON. - Add tests/test_issue667_tf_margin_apply_parity.py (5 tests: Wilson calibration, Betley-judge wiring, known-good PASS, known-bad drift-caught, drop-never-coerce). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…librated floor (import, no shadow) The round-4 fix raised the apply-parity probe module's N_SAMPLES 10->100 (the Wilson-CI floor for the +/-0.10 tolerance), but the tf-margin dispatcher's --n-samples default stayed at 10 and phase_apply_parity forwards it explicitly to the subprocess, SHADOWING the module default. A production `all ...` launch would re-arm the round-3 false-fail. Import N_SAMPLES from the probe module so one module owns the value; extract _build_parser() so a regression test can assert the default >= the calibrated floor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Same-issue follow-up round
tf-margin-gate-vs-behavioron task #667 (source: user-chat 2026-07-01, plan v6 auto-approved at 6 GPU-h).Closes-half task #667.
Recomputes #667's gate→behavior bridge with #722's continuous teacher-forced fixed +/- completion margin as the SECONDARY behavioral DV, on the #537 2-index grid (16 sources × 30 targets, 3 behaviors — em, sycophancy, fact). Everything else inherited from #667's a36 round: same #537 adapters, same 30 target contexts, same read layer L14, same 7-family clustered bootstrap, seed 42, same g0(C') per cell.
New in this round:
scripts/issue667_tf_margin_extract.py— vendors 3 tf-margin helpers from issue-941: artifact-reuse fitness check gains pairwise provenance-coherence item (j) #722 byte-identically + substitutesmessages_for_instance→i537_contexts.build_messages.scripts/issue667_tf_margin_analysis.py— vendors_spearman+clustered_bootstrap_spearmanfrom issue-941: artifact-reuse fitness check gains pairwise provenance-coherence item (j) #722 (N_BOOT=2000).scripts/issue667_build_fact_pool.py— fact fixed pos/neg pool builder (the one new data artifact).Backend: RunPod explicit override, one 8×H100 pod, wave_size=8.