issue-816: Persona Vectors Exp 2/4/5 + random-direction baseline#615
Draft
superkaiba wants to merge 13 commits into
Draft
issue-816: Persona Vectors Exp 2/4/5 + random-direction baseline#615superkaiba wants to merge 13 commits into
superkaiba wants to merge 13 commits into
Conversation
…core) - issue816/steering.py: ActivationSteerer forward hook (Exp-2 gen-time, raw vector, positions=response, layer_idx=19) + steered_generate batched HF gen. Ported from persona_vectors b8e0f04 activation_steer.py / eval_persona.py. - issue816/preventative.py: PreventativeSteeringCallback (Exp-4 training-time steer-toward hook via TrainerCallback on_train_begin/on_train_end + PEFT path-rewrite). RAW vector convention (faithful to training.py steer branch; plan §11 note describes the ablate branch, reconciled to code). - issue816/screening.py: Exp-5 projection-difference DeltaP + #778 null-battery reuse at frozen layer 20 + sample-level AUC separation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- issue816_lib.py: r_B fetch+sha-pin (plain-trait slug, verified on HF), norm-matched random dirs (deterministic per draw idx), eval-question load, #778 finetune-score reader + fail-loud preflight consumer-path assert. - issue816_steering.py: Exp-2 + Phase-0 probe; HF steered gen, per-cell JSON, per-draw diagnostics (len/refusal-preflag), deterministic seeds. - issue816_preventative.py: Exp-4 train (train_lora + PreventativeSteeringCallback, paper rsLoRA recipe, band-stop OFF) + vLLM post-ft eval-gen (chunked, use_tqdm=False). - issue816_screening.py: Exp-5 activation capture -> per-dataset mean-diff predictor tensor + per-sample layer-20 projections. - issue816_dispatch.py: unified smoke=sweep 8-GPU fan-out, CVD dual-pin (--gpu-id only on preventative), --cells threads to every phase. PASS_UNIFIED. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…dge/analysis - issue816_dispatch.sh: pod driver (clone pv@b8e0f04, unzip, REQUIRED §12 preflight assert of reused #778 finetune JSONs, fan-out dispatch, upload, sentinel + [phase=done]). Unified smoke via EPM_I816_SMOKE. - issue816_upload.py: Exp-4 adapters -> HF model repo; Exp-5 tensors + raw-generation JSONs -> HF data repo; builds reproducibility_card (adapter_paths verified, wandb_project/run_names/entity). dotenv-before-hf import order. - issue816_write_sentinel.py: epm:results sentinel (SENTINEL_REQUIRED_KEYS + card). - issue816_judge.py (Phase B, off-pod): graded Sonnet trait + coherence via judge_graded (N=6 @0.7, drop-never-coerce, Batch), per-cell _scored.json. - issue816_analysis.py (Phase C, off-pod CPU): Exp-5 null battery @ frozen layer 20 + BH; Exp-2 beat-count (coherence-gated); Exp-4 real-vs-random @ pre-frozen a*=1.25 + p-floor + winner's-curse caveat; hero figures via paper_plots. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…moke caught) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ric) - analysis reader: add "trait_score" as first _extract_trait_mean candidate key (the real #778 finetune JSON stores the mean there; was a guaranteed KeyError crashing Exp-4 coef-0 + Exp-5 y-axis after 68 GPU-h Phase A). - steering seed: derive per-(trait,coef,dir,rollout) generation seed via hashlib.sha256 instead of Python hash() (PYTHONHASHSEED unset -> hash() is per-process salted -> the recorded seeds were non-reproducible). - judge: port the coherence rubric VERBATIM from the persona-vectors clone eval/prompts.py Prompts["coherence_0_100"] (was paraphrased). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rainerCallback
The callback was standalone, so HF's CallbackHandler.call_event fired
on_init_end (the first lifecycle event, inside Trainer.__init__) on it and
crashed with AttributeError before any training step — the round-2 production
run died at SFTTrainer(**kwargs) in the preventative phase. Subclassing
TrainerCallback gives it the inherited no-op defaults for every lifecycle event
it does not override; the on_train_begin/on_train_end hook attach/remove logic
and the (self, args, state, control, **kwargs) signatures already matched the
contract (the model is read via kwargs.get("model")), so this is the only change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r slow per-file terminal upload
…2/4/5 null battery, hero figures)
…ull-rerun) - screening.py: rewrite run_null_battery_screening with 8-family honest null ladder (isotropic, neutral_cov, within_pos, within_neg, rb_out_iso, cross_trait, pca_top5, contaminated_pooled as labeled reference). Must-Fix A1: project out r_B BEFORE renorm in within-class families; Must-Fix S1: conservative empirical-p (r+1)/(n+1) + BH over stochastic families only. - issue816_lib.py: add fetch_neutral_cov() for loading v2 neutral covariance tensors from HF data repo; fix RUF002/SIM108 lint - issue816_analysis.py: separate --out-root (v3/) from --scored-root (parent for reading judge outputs); wire fetch_neutral_cov + new 8-key null battery into run_exp5(); update Exp-5 figure to use 8 honest null keys - issue816_wait_v2_artifacts.py: fix RUF059 (unpacked _found_set never used) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ache, bh key) Replace contaminated e2/e4_randnorm arm slugs with e2/e4_isotropic + e2/e4_neutral_cov families per plan §5 (isotropic: N(0,Iσ²) renormed, seeds base+0..4; neutral_cov: Cholesky sample from #778 v2 neutral_cov, seeds base+100..109). Add hard assert that no production cell carries the old contaminated slugs. Precompute 28 Cholesky factors before the draw loop (was ~5,600 per-draw factorizations). Emit bh_adjusted_stochastic top-level key from run_null_battery_screening. Fix Exp-4 empirical p to conservative (r+1)/(n+1). Set analysis --seed default to 42. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…s) + CONCERN 3/4 - dispatch.sh + dispatch.py: route all Phase-A output to eval_results/issue_816/v3/ (BLOCKER A); remove --n-random-dirs flag passed to preventative (BLOCKER B) - judge.py: default --out-root to eval_results/issue_816/v3 (BLOCKER A) - analysis.py: scored-root defaults to out-root itself (v3/), add belt-and-suspenders guard refusing non-v3 scored-root to prevent silent v2 reuse (BLOCKER A) - preventative.py: seed=42 (CONCERN 3); hoist fetch_neutral_cov per trait in main() to avoid per-cell re-download/re-factorize (CONCERN 4) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ssion test Three argparse --out-root defaults were still at eval_results/issue_816 (the v2 namespace) while dispatch.sh explicitly passes eval_results/issue_816/v3. - scripts/issue816_dispatch.py: change --out-root default to v3 - scripts/issue816_steering.py: extract build_parser() (importable for tests), change --out-root default to v3 - scripts/issue816_preventative.py: extract build_parser() (importable for tests), change --out-root default to v3 - tests/test_issue816_dispatch_argv.py: permanent regression test asserting (a) all three scripts default to v3 and (b) every argv flag _steering_cmds / _preventative_cmds / _probe_cmds constructs is accepted by the matching build_parser() — fails CI if a new flag is added to the dispatcher but not to the sub-script's parser Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…ntative.py - Flip --out-root default from eval_results/issue_816 to eval_results/issue_816/v3 in upload.py, upload_fast.py, screening.py, write_sentinel.py (all previously pointing at v2 root; concern issue816-upload-reads-v2-path) - Add explicit --out-root eval_results/issue_816/v3 to dispatch.sh upload call so the shell-level invocation matches the now-corrected default - Defer PreventativeSteeringCallback + TrainLoraConfig + train_lora imports into _train_cell() so build_parser() is importable without torch/transformers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #816. Plan: tasks/running/816/plans/plan.md (v2).