issue-642 round-5: second-behavior-rank-replication (refusal) — directional replication, PARTIAL#483
Open
superkaiba wants to merge 50 commits into
Open
issue-642 round-5: second-behavior-rank-replication (refusal) — directional replication, PARTIAL#483superkaiba wants to merge 50 commits into
superkaiba wants to merge 50 commits into
Conversation
…alysis Adds the cmft (coverage-matched full-FT) arm to #606's sycophancy LoRA-vs-FT bystander-leakage comparison, isolating the module-coverage component from the rank-or-adapter-bundle component at matched s*=0.50. - train_behavior_fullft.py: port of origin/issue-606 trainer + the apply_coverage_match_freeze mask (train only the 7 _proj weights + the q/k/v_proj biases; freeze embed_tokens/lm_head/LayerNorm) gated by --freeze-outside-lora-modules, with 3 fail-loud asserts (84 q/k/v biases, module-set identity vs the #606 adapter_config, 0.80-0.92 trainable fraction). - scripts/issue_642/: dispatcher (cmft arm token, freeze-flag wiring, Phase-0 adapter_config download + reused-#606-cell enumeration, install gate + lr-2e-6 retrain lever, sentinel with reproducibility_card), 3-arm analyze (Delta_rank=cmft-LoRA / Delta_coverage=FT-cmft + additive identity + re-judge of the reused LoRA/FT arms), gen/lora workers, figures (decomposition-bar hero), common (two pinned shas + cmft grid). - configs/accelerate/zero3_4gpu_accum1.yaml: ZeRO-3 config (verbatim port). Smoke: freeze-mask CPU smoke (196 _proj weights + 84 biases, neg tests), Phase-0 against real HF artifacts, full --smoke --dry-run (6 phases, terminal [phase=done]), FT-trainer signature smoke, synthetic 3-arm analysis (designed-gap recovery PASS, both bracket + no_bracket modes), figures render. verify-imports: 51 deferred symbols across 7 files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ight + per-arm fallback + --run-label CLI Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes round-3 CONCERN 642-run-label-local-cache-collision: _ensure_local's local short-circuit keyed on <root>/<behavior>/<rel> omitting cmft_experiment, so a --run-label cmft-lr2e6-retrain run reusing the same --eval-root silently read the stale default-prefix cmft artifacts. New _cmft_cache_root nests a run-label-scoped cmft prefix under _runlabel_<run_label>; the default (HF_EXPERIMENT_NAME) and reused-#606 (PARENT_EXPERIMENT_NAME) prefixes keep the existing flat layout (existing caches still hit; reused path unchanged). The lazy-generation else-branch uses the same scoping. Regression tests added. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…noise_limited; Δ_rank=+0.073 [+0.053,+0.100], Δ_coverage=+0.025 [+0.007,+0.041], additive=+0.099 vs #606 +0.098); fix stacked-bar errorbar negative-yerr crash
…; drop 'arm' from reused-FT row keys
…cy constants Deliverables 1+2 of the onpolicy-matchedlr-rank-isolation follow-up (plan v5 §4.2): (1) i642_lora_train_worker.py gains --lr (+ --run-name-suffix) so the LoRA pole trains at the matched 1e-5 with a custom grid; (2) i642_common.py gains the v4 villain source, MATCHED/ORIG LRs, 4-arm registry + 3 within-villain contrasts, fine + pilot grids, #612/#411 pool/panel/probe pins (sha-verified), and the v4 panel loader + disjointness + byte-identical-negatives + canned-pool splicer helpers. v3 constants untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deliverable 3: i642_v4_splice_canned_pool.py reuses #411's 200 villain canned positives + #612's 500 on-policy negatives (byte-identical), sha-pinning both source pools at fetch and asserting the byte-identical-negatives invariant (the single-variable guarantee for Delta_data). Smoke: 700-row pool built, negatives sha matches #612, exit 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…matched-LR) Deliverable 4: i642_dispatch.py gains a --v4 mode (plan v5 §4). Trains the 4 villain arms (loraOP_lr1e5/cmftOP_lr1e5/cmftOP_lr5e6/cmftCN_lr1e5) at matched LR on on-policy data + reads them on the #612 30-panel; NO #606 reuse. Phase 0 fetches+sha-verifies the #612 on-policy pool + #411 canned pool + panel + eval_60 + #612 adapter_config, splices the canned-cmft pool (byte-identical negatives), runs v4 disjointness asserts. Phase 1 trains each arm at its own LR/pool (LoRA worker --lr / cmft trainer --learning-rate + --run-name-suffix). Per-arm-slug ckpt roots + selection grouping (3 cmft arms don't collapse). --install-pilot (Phase 0.5 short-train gate). v4 reproducibility card with per-arm wandb_run_names + wandb_project=issue642 + wandb_entity (read off the SDK). v3 path untouched. Dry-run smoke: p0->p5->done, exit 0, sentinel + card verified. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…2x rule The CPU phase-0 smoke surfaced that #612's on-policy positives run ~397 tokens (vs #411's short canned templates), tripping the inherited _assert_completion_ lengths 2x-eval-tokens rule. v4 sycophancy is a FREE-GENERATION behavioral eval (512 max_new_tokens is the CLAUDE.md free-gen exception, not the marker/end-of- completion 2x rule), so the relevant guard is TRAIN-time truncation: full prompt+completion must fit max_length=1024. Added _v4_assert_train_truncation; real phase-0 smoke now PASSes (max total 440 <= 1024). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…deferred gap) The plan claimed no analyzer changes, but i642_analyze was v3-wired (3-arm lora/cmft/ft + #606 reuse + additive-to-#606). Adds a parallel v4 path: _v4_analyze_behavior computes Delta_rank_matched / Delta_LR / Delta_data over the 4 NEW villain arms from THIS run's eval_root (NO #606 reuse, NO additive-to- #606 check), on the #612 29-bystander panel, reusing the shared verdict-matrix + _two_arm_gap + bootstrap machinery. v4 decision rule (H_survives / H_artifact / indeterminate). make_synthetic_v4 4-arm fixture + --v4 routing. Synthetic smoke: designed Delta_rank_matched=0.06/Delta_LR=0.04/Delta_data=0.05 recovered within tol 0.06 (verdict H_survives); no_bracket fallback exercised (exit 0). Resolves concern v4-analyzer-decomposition-not-implemented. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ot_gate Fixes BLOCKER install-pilot-not-production-safe (round-1 reconciler FAIL): pilot now trains into ckpts_pilot/<arm> + reads into stage_a_pilot/ with pilot_trajectory.json + selection_pilot.json, so the production full run never resume-skips on pilot artifacts. Adds an embedded p0_5_pilot_gate phase (before p1_train) implementing the plan §7 step-4 dense-collapse + LoRA-monotonic gate that HALTs the production launch on failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses non-blocking concern v4-artifact-pins-incomplete: _phase0_data_v4 now asserts EXPECTED_SHA256 for the #612 30-panel, eval_60 probes, and #612 villain adapter_config (previously only the two training pools were pinned). All 5 pins verified live against the Hub. Shas recorded in data_manifest.json. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…o root Records the #642-r2 trap: Edit/Write absolute file_path must target the .claude/worktrees/issue-<N>/ tree; the bare repo-root absolute path is a separate file on main and silently strands work off-branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dence_only 1e-5 slugs) Item 1 of plan v8 §4.2: execute the v5 §7 pre-authorized matched-LR shift. V4_MATCHED_LR = 5e-6 (was 1e-5); add V4_OLD_MATCHED_LR = 1e-5 for evidence prose. V4_ARMS = (loraOP_lr5e6, cmftOP_lr5e6, cmftCN_lr5e6); the three 1e-5 slugs retained as role="evidence_only" (lr_name="old_matched"). V4_CONTRASTS drops delta_lr; delta_rank_matched + delta_data re-pointed to the 5e-6 arms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…CHED_LR 5e-6 Item 2 of plan v8 §4.2: help-text only (the --lr flag already exists from v5; the dispatcher passes --lr 5e-6 for loraOP_lr5e6). Align the docstring example slug + the matched-LR note to the v8 5e-6 value. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… evidence_only slugs Item 3 of plan v8 §4.2. (a) arm-parse guard raises a distinct refuse-to-train error when --arms names an evidence_only (falsified v5-round-1 1e-5) slug, so a stale launch command fails loud instead of silently training a refused arm (§8-bis resume-invalidation guard); (b) _train_v4_lora_arm + _train_v4_cmft_arm docstrings -> loraOP_lr5e6 / cmftOP_lr5e6,cmftCN_lr5e6 at 5e-6; (c) hardcoded HF lora_pole_upload path loraOP_lr1e5_ -> loraOP_lr5e6_; (d) --v4 + --arms help text -> 3 production slugs (loraOP_lr5e6,cmftOP_lr5e6,cmftCN_lr5e6; default all 3); contrasts metadata drops delta_lr. verify-imports OK (59 symbols). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…synthetic term Item 4 of plan v8 §4.2: update make_synthetic_v4's per-arm coeff map + no_bracket LoRA-arm checks to the v8 slugs (loraOP_lr5e6 = coeff(p); cmftOP_lr5e6 = +RANK_COEFF; cmftCN_lr5e6 = +RANK_COEFF+DATA_COEFF); remove the LR_COEFF synthetic term + known_delta_lr metadata (the Δ_LR contrast is gone, §3). Update the H_artifact verdict branch + legend/docstring prose to the 3-arm decomposition (Δ_rank_matched / Δ_data). Because V4_ARMS + V4_CONTRASTS are imported from i642_common, the contrast loop + figures follow automatically. Synthetic round-trip PASSes all 3 modes (bracket/no_bracket/opposite_direction); syn_check recovers both known deltas within tolerance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…test (§3 mechanizer) Item 6 of plan v8 §4.2 (Statistics-lens reconciler fix). Extract the v4 decision rule into _classify_outcome(delta_rank_ci, delta_data_ci, thresholds) -> (label, subreason|None), routing the 7 reachable (Δ_rank_matched, Δ_data) cells: H_survives / H_artifact / opposite_sign_rank / rank_in_band_data_quiet / rank_wide_data_separates / rank_wide_data_quiet / rank_positive_uncertain. The caller keeps the pre-lattice skipped-arm + determinacy guards and now persists the §3 subreason tag in headline.subreason (+ subreason_legend). tests/ test_i642_classify_outcome.py enumerates the 7 cells (exactly one label+subreason each), asserts 7 distinct outcomes, and sweeps a dense CI grid asserting the lattice is TOTAL + UNIQUE (no holes) — mechanizing the §3 totality claim so the v6-era non-exhaustiveness cannot re-recur. 11 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ollow-on) Stale '4 NEW arms' / '4-arm pools' wording in _phase0_data_v4 surfaced during the data-gen smoke. The v8 production set is 3 arms at matched LR 5e-6; the splice + byte-identical-negatives assert is unchanged. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…item 3 follow-on) Stale '(up to) 4 villain arms' wording surfaced during the training-phase carve- out smoke; the v8 production set is 3 arms. '4 GPUs' (hardware) is unchanged. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fter #642 r3 recurrence The repo-root-path trap recurred in round 3 despite the existing note (used repo-root absolute Edit paths + `cd repo-root && git`). Add a pre-first-commit `git rev-parse --abbrev-ref HEAD == issue-<N>` guard, the no-`cd`-to-repo-root git rule, and the cherry-pick-onto-branch recovery (non-destructive on shared pushed main). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…xempt) Plan v8 §4.3/§4.5/§4.6/§7/§9 exempt cmftOP_lr5e6 from the fresh install-pilot (it gate-PASSed the v5 pilot at the matched 5e-6 LR — gate-validation reuse). The inherited p0_5_pilot_gate looped ctx.arms (all 3 production arms) for both the pilot train and the gate, re-piloting the exempted arm: wasted GPU + a spurious step-4-collapse HALT risk on an arm the plan said to skip (concern issue642-v8-pilot-scope BLOCKER, reconciler epm:review-reconcile v3). Fix (Option A): narrow ctx.arms to V4_PILOT_ARMS (= V4_ARMS minus the new V4_PILOT_EXEMPT_ARMS constant) for the pilot train/stage-A/gate, adding ctx.arms to the save/restore tuple so production Phase 1 still trains all 3 arms. The verdict surfaces cmftOP_lr5e6 under exempted_arms (reused-via-v5-gate-validation), never silently dropped. Also: v4-aware final sentinel provenance note + the v4 header/reproducibility-card doc-drift fixes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion tests/test_i642_pilot_scope.py asserts (1) V4_PILOT_ARMS is exactly the 2 new arms and partitions V4_ARMS with the single exempt arm cmftOP_lr5e6, and (2) p0_5_pilot_gate narrows ctx.arms to the 2 new arms for the train/stage-A/gate callees (captured via monkeypatch), restores all 3 afterwards, and writes a pilot_gate.json carrying cmftOP_lr5e6 under exempted_arms (never piloted arms). Catches the issue642-v8-pilot-scope regression at test time, not GPU-launch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rift) Cosmetic doc-drift the v8 retag left behind (review round 3, non-blocking): - i642_analyze.py:1153 header v5/4-NEW-arms -> v8/3-production-arms-at-5e-6 - i642_lora_train_worker.py:9 module docstring "#642 does NOT train a LoRA arm in production" -> now false; the v8 --v4 path DOES train loraOP_lr5e6 (fresh on-policy LoRA pole, LR passed via --lr; default LORA_LR for the v3 path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… recurrence Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
[1/3] Add the v9 (second-behavior-rank-replication) constants to i642_common (V9_ARMS, V9_CONTRASTS, V9_MATCHED_LR/V9_FALLBACK_LR, V9 grids, the v9 HF namespace, the merged ARM_SPEC_ALL lookup so v9 slugs resolve through v4_arm_lr) and the substantively-new on-policy refusal elicitation worker. The worker elicits the villain refusal positives + freshly-elicited same-question helpful negatives via the #612 elicitation ladder (80% yield floor + equalize-down, fail-loud below floor — no canned backfill), hard-filters the elicitation questions disjoint from the refusal_50 eval probes, and records per-row tier + the villain base refusal rate. Content-hygiene: never prints completion/question text, only counts/tiers/hashes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
[2/3] Add --v9 to i642_dispatch as a one-variable amendment of --v4: forces v4=True so every v4-aware downstream phase (stage-A / select / stage-B / gen / upload) runs unchanged, then overrides the arm registry (2 arms), behavior (refusal), data path (_phase0_data_v9 runs the fresh elicitation + the v9 asserts: same-question-contrast, disjointness, train/eval-disjointness, eval-pool content-hash, across-arm byte-identity), and the install-pilot gate semantics (p0_5_pilot_gate_v9 FIRES the pre-registered single matched drop 5e-6->2e-6 on a dense step-4 collapse instead of HALTing; a 2e-6 collapse is kill criterion (a)). Adds the v9 reproducibility card (per-arm WandB run names + wandb_entity + the round-4 +0.063 comparison) and threads the chosen matched LR to both poles. CPU dry-run smoke runs all phases end-to-end exit 0 + [phase=done]. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
[3/3] Add --v9 to i642_analyze: _v9_analyze_behavior computes the single delta_rank_matched contrast (cmftRefOP - loraRefOP) at matched s*=0.50 over the #612 29-bystander panel with the crossed cluster bootstrap, classifies via _classify_outcome_v9 (REPLICATES/PARTIAL/FAILS lattice), attaches the round-4 sycophancy +0.063 comparison + the refusal parity anchor + per-arm install-failure (kill (a)) handling; make_synthetic_v9 fixture recovers the designed gap (REPLICATES verified). Add --v9 to i642_figures: the cross-behavior adapter-vs-dense hero + the exploratory dump (leakage-vs-strength, profile scatter, trajectories, heatmap, sweep, install-pilot panel) -> figures/issue_642/v9/. New tests mechanize the v9 lattice totality + the same-question-contrast / disjointness fail-loud guards. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nce + json circular-ref trap Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The on-policy elicitation ladder over-produces negatives; without a cap the realized positives:total-negatives ratio drifted to 1:3.75..1:6.75 vs the planned 1:2.5, breaking the contrastive-negatives dose contract and the recipe-match to the round-4 anchor. Add _cap_negatives (per-slot budget + global total cap, deterministic, slot-balanced); re-emit per-slot counts POST-cap. Regression tests assert len(neg) <= round(neg_ratio*n_pos)+n_slots. Closes round-1 reconcile B1 (i642_elicit_worker.py:453). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
phase3_select wrote a single no-suffix install_failure.json, but the v9 analyzer reads per-arm install_failure_<arm>.json — so a v9 production-gate kill was never detected and the analyzer fell through to full analysis on a killed behavior. Extract _write_install_failure: v9 writes one file per FAILING arm (mirrors the phase-0.5 pilot writer); v4 keeps the no-suffix file its reader expects. The resume-skip check is now v9-aware (any per-arm failure). Tests assert the v9 analyzer short-circuits to verdict=KILLED without _v9_analyze_behavior. Closes round-1 reconcile B2 (i642_dispatch.py:2836 vs i642_analyze.py:1564). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(B3, Minors) B3: v9 runs force ctx.v4=True but the v9 Reproducibility Card declares adapters under adapters/issue_642/v9/...; the uploader hardcoded v4/. Parameterize the dest prefix on ctx.v9 (v9 -> v9/, v4-only -> v4/) so the actual upload matches the card. Test asserts v9 loraRefOP lands at adapters/issue_642/v9/...; v4 keeps v4/. (The v9 card's lora_pole_upload prose + adapter_paths dict already say v9.) Minor 1: rename the v9 analyzer output analysis_v9.json -> analysis.json to match the plan v10 §6.5 primary_deliverable contract; update i642_figures.py reader + help to match. (v9 lives under its own refusal_v9/ eval_root — no collision with the v3 analysis.json.) Minor 2: rename test_disjointness_fails_when_villain_is_a_negative -> test_disjointness_fails_on_unexpected_negative_prompt (it trips the unexpected-negative-prompt branch, not a villain-as-negative check). Closes round-1 reconcile B3 (i642_dispatch.py:3063) + Claude Minor + Codex Minor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ariant-reader + cap-asymmetry classes Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…+ writer regression tests (M2) M1 (round-2 reconcile BLOCKER): the v9 analyzer wrote analysis.json at `eval_root/<behavior>/analysis.json`, so under the plan §10 command `--eval-root eval_results/issue_642/refusal_v9` it landed at `refusal_v9/refusal/analysis.json` — one subdir below the §6.5 primary_deliverable glob `refusal_v9/analysis.json`, which never resolved. The canonical bare `i642_figures.py --v9` (no --eval-root) also missed it (figures defaulted to the shared `issue_642/` parent). Fix (Option A): - i642_analyze.py: the v9 writer drops the `behavior/` subdir — the v9 eval_root already encodes the behavior. v3/v4 keep their behavior-subdir layout (their eval_root is the shared parent). Only the final WRITE moves up; fetched verdicts/trajectory still live under eval_root/<behavior>/. - i642_figures.py: `_figures_v9` reads the registered `<eval-root>/analysis.json`; the bare `--v9` command descends the default --eval-root into refusal_v9/ so the §10 canonical chain resolves with no override. Explicit --eval-root honored. M2 (round-2 standing rec): WRITER-side regression tests for the per-arm install-failure convention — v9 emits one install_failure_<arm>.json per failing arm and NO no-suffix file (the v9 reader would never see it); v4 keeps the no-suffix file. Plus M1 contract tests: analysis written at the §6.5 path (not nested), the bare `--v9` figures command reads the registered path, and an explicit --eval-root is honored as-is. Closes round-2 reconcile M1 (i642_analyze.py:1932 / i642_figures.py:548). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…on (findings-blind)
…igure + script
…(stage-A trajectory, stage-B verdicts, generations, refusal_v9 analysis)
…er-arm WandB IDs Two precision corrections to the second-behavior-rank-replication arm (round 5, refusal): - §3 training data: split the combined 332 negatives row into the precise even 3-way split (166 police_officer / 166 medical_doctor / 166 no-persona), verified against the actual train_pool.jsonl system-prompt counts. - §6 artifacts index: replaced the generic WandB project URL with the authoritative per-arm run IDs from epm:upload-verification v3 (loraRefOP prod swmowgwy + pilot bcqlug4u; cmftRefOP prod iq54kbwd + pilot uch881ph); added HF data-subtree row (812 files) and stage-B coverage row (210 refusal_eval_*.json across 7 cells). check_no_secret_shaped_strings.py: exit 0.
…step interpretation-critic.md only showed the epm:interp-critique marker BODY shape under "Output Format" with no task.py post-marker shell command, so the agent read "Post as <!-- ... -->" as "print the body in return text" (incident #642 round-5 round-2: verdict printed but never posted, orchestrator backfilled). Add an explicit "Posting your verdict (REQUIRED last step)" section with the copy-pasteable post-marker --file command + version-match note. code-reviewer.md and clean-result-critic.md already carried the command; add the "printing the body does NOT post it" reinforcement to both. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
Aligns §2 Hyperparameter table keys with the clean-result body Parameters table for verify_task_body.py check 21 (Body Parameters ⊆ methodology doc §2). Body uses: | LoRA (reused, NOT retrained) | ... | | Full-FT (reused, NOT retrained) | ... | Doc §2 now matches verbatim. The 'arm' token was dropped to also clear the audit_clean_results_body_discipline.py 'experimental_arm' pattern (incident: round 3 of 9a-bis on #642).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
superkaiba
added a commit
that referenced
this pull request
Jun 18, 2026
… + methodology doc Lands #642's round-5 same-issue follow-up artifacts on main without the full PR-rebase merge (which conflicted across .claude/agent-memory/ and the methodology doc add/add — the branch is 3833 commits behind main). Brings onto main: - eval_results/issue_642/refusal/ (stage-A trajectory, stage-B verdicts × 7 cells, on-policy generations) - eval_results/issue_642/refusal_v9/analysis.json (primary round-5 analysis) - figures/issue_642/refusal_r5_cross_behavior.{png,pdf,meta.json} (round-5 cross-behavior figure) - scripts/issue_642/i642_r5_figure.py + i642_r5_judge.py (round-5 figure + judge scripts) - docs/methodology/issue_642.md (round-5 EXTEND content + arm-drop §2 keys, commit be8d87c) Body's **Methodology:** + **Methodology reference:** links are SHA-pinned to be8d87c (reachable from origin/issue-642 via GitHub's blob URL resolution). PR #483 stays OPEN; the full rebase is deferred to a follow-up cleanup pass.
superkaiba
added a commit
that referenced
this pull request
Jun 23, 2026
Adds a 'read this first' specification section pinning the exact behaviors, context conditions, training hyperparameters, eval protocol, and extraction/storage config — each cell sourced to an issue/config/ rule. E1-E11 now reference §S instead of restating ingredients. Grounded from #480/#612/#521/#404, configs/{training,lora}/turner_em, marker-leakage/training-recipe rules, persona_pool #483, #617. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Round-5 same-issue follow-up of #642 (
second-behavior-rank-replication, source: user-chat).Headline: the adapter-vs-dense leakage gap replicates DIRECTIONALLY on a second behavior (refusal): Δ_rank_matched = +0.027, CI excludes zero, |point| < +0.04 separation band, profile Spearman ρ=0.94 — PARTIAL per the analysis verdict_legend; about half the magnitude of round-4 sycophancy +0.063.
What's in this PR:
eval_results/issue_642/refusal/(stage_a trajectory, verdicts × 7 cells, on-policy generations) +eval_results/issue_642/refusal_v9/analysis.jsonfigures/issue_642/refusal_r5_cross_behavior.{png,pdf,meta.json}### <finding>under## Findings+ Takeaways re-synthesis + tail-concentration caveat + bracket-asymmetry / s*-sweep / install-strength-contingent caveats; H1 rescoped: refusal is follow-up evidence, not goal-level conclusion; MODERATE confidence anchored by parent sycophancy decomposition).docs/methodology/issue_642.md§3-§6 round-5 sub-blocks + §2 hyperparameter column)..claude/agents/{interpretation-critic,code-reviewer,clean-result-critic}.md) — maketask.py post-markerthe REQUIRED last step.Adversarial review: interp-critic ensemble passed at round 3 (reconciler resolved round-2 tail-concentration disagreement); clean-result-critic ensemble passed at round 3 (reconciler resolved round-2 check-21 + Lens-10 disagreements). All round-5 reconciler verdicts logged on
events.jsonl.🤖 Generated with Claude Code via Happy
Co-Authored-By: Claude noreply@anthropic.com
Co-Authored-By: Happy yesreply@happy.engineering