Phase 2 — fine-tune fleet + trained store + ground-truth leakage (#664)#487
Draft
superkaiba wants to merge 67 commits into
Draft
Phase 2 — fine-tune fleet + trained store + ground-truth leakage (#664)#487superkaiba wants to merge 67 commits into
superkaiba wants to merge 67 commits into
Conversation
…tract [task.py]
… `d1069a8bc3`. [task.py]
…ode-reviewer --> ## Reco [task.py]
… effort=high write=True pol [task.py]
## Consistency Check: #664 vs re [task.py]
… effort=high write=True pol [task.py]
…lg phase=done after 91s. [task.py]
…ci phase=done after 91s. [task.py]
…awn] ALIVE-BUT-ST [task.py]
… effort=high write=True pol [task.py]
…jh phase=done after 181s. [task.py]
…-extract [task.py]
…n 1/3). Current s [task.py]
[task.py]
…entation v2 --> ## Implementation [task.py]
…d=1 subagent=plan [task.py]
…gy coverage-contract [task.py]
… effort=high write=False po [task.py]
## Code-Reviewer Verdict — PASS [task.py]
….md` (gpu_hou [task.py]
[task.py]
…onomous plan-gate (tas [task.py]
…g panel) - 64-cell realized grid (16 gate marker + 40 content transfer + 4 nulls + 4 seed-1042) - contexts via on-main issue594_common 50-context battery - per-behavior recipes from #537 methodology doc §2 + marker rules - EM/bad-medical/insecure-code expressed in-process via train_lora (#545 opt-in overrides) -- the named §4.4 divergence (turner_em.yaml not on main) - contrastive negative panel with HARD panel∩sources=∅ assert Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…er/fact/EM carve-outs) Builds prompt-completion JSONL per (source, behavior, arm, dose) cell per plan v3 §4: marker programmatic carve-out, taught-fact carve-out, EM published-corpus replication, on-policy sycophancy/refusal with 80% yield floor; zero-truncation gate + panel-disjoint-from-sources + marker-token asserts. Also de-ambiguate unicode x/- in common.py comments (ruff RUF002/3). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…arker four-float) Per-cell P2.2 worker: answer-side mean v_plus(C') on all 50 contexts x 28 layers + v0(C') spine/own-battery + v0(C_neg) + last-input c_C (trained+base) + t_CB/r_plus at source; per-probe (R>=8) probe-split inputs; target_context_role per row (kill-3b exclusion); marker four-float slot stats (#530 contract) via compute_marker_slot_stats with the gauge assert. Merge-read-delete per cell (§9.2 disk budget); left-pad position_ids; vLLM spawn + reap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nues (no fleet crash) A source below the 80% on-policy yield floor crashed the whole fleet: the builder exited 1 and the dispatcher's subprocess.run(check=True) raised CalledProcessError. Plan v4 §11 mandates graceful degradation (drop + report, never crash). The builder now exits DROPPED_SOURCE_EXIT (3) + writes a per-cell drop sentinel; the dispatcher treats rc==3 as skip-this-cell-and-continue (rc!=0 stays fatal), writes a top-level dropped-cells manifest, and excludes dropped cells from train / extract / eval / manifest / upload / repro-card. Pinned by tests/test_issue664_pivot_invariants.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-probe AutoTokenizer.from_pretrained calls trigger an HF Hub model_info() HTTP probe inside transformers' _patch_mistral_regex, hitting the 2500/5min HF API rate-limit mid-fleet (crashed after ~16 cells at bm_librarian_posonly_d1_seed42 gen-phase, 04:24 UTC). Module-level cache fix: single load per process, no behavior change. Lines 181 + 315 retained — they're once-per-cell, not per-probe.
…s + wrapper) Add data-parallel sharding to the Phase-2 dispatcher so the independent (source x behavior x arm x dose) cells fan across 8 GPUs instead of running serially on one (the ~30h/7-idle-GPU recovery). Pure orchestration -- per-cell science is unchanged; cells partitioned by i % num_shards == shard_id after the post-P2.0 drop filter. - issue664_dispatch.py: --shard-id/--num-shards args (default 0/1 = no sharding, backward-compatible); _validate_shard + _shard_filter helpers; shard-0-only gates on phase0/manifest/readability-assert/live-judge/upload over the FULL post-drop fleet (all_cells), train/extract/eval loops over the shard subset. - Refactor the epm:results sentinel into _write_results_sentinel and fire it on --phase p3 shard 0 too: the 8-way wrapper runs phased (never --phase all), so the terminal contract would otherwise never be written (poll_pipeline #448). - issue664_launch_parallel.sh: pod-side wrapper sequencing p0 (1 GPU) -> p2 8-way (CVD-pinned per shard) -> p3 (1 GPU); emits the watched [phase=...]/[phase=done] contract + PID file; aborts before p3 on any shard failure (rc=1 -> poll reads NOT-done). - tests/test_issue664_sharding.py + tests/manual/print_issue664_shards.py: unit test + dry-run pinning disjoint-union cell coverage + validation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
r7 placed _write_manifest / _marker_readability_assert / _live_judge_smoke inside the --phase p2 block, so shard 0's p2 process fired the A7 readability HALT on its own ~2 of 20 marker cells the instant it finished -- racing the ~18 marker cells on shards 1-7 whose marker_slot_stats.json did not exist yet (silently continue'd; checked>0 masked the gap). That re-broke the #664 round-2 B3 readability gate. Move the three finalizers to the top of --phase p3 (ahead of upload_artifacts), which the wrapper invokes only after waiting for every p2 shard, so the full marker_slot_stats.json set exists. Rename the relocated phase_log names p2_* -> p3_*. Also fix the open raw-completions-upload-ignores-cells-arg concern: _upload_raw_completions now globs per selected cell.eval_key (>=1 file per cell, fail-loud on zero) instead of a blind reg_root.rglob. Adds two structural tests pinning WHERE each finalizer runs (fail pre-fix, pass post-fix). Closes BLOCKER post-p2-finalizers-race + CONCERN raw-completions-upload-ignores-cells-arg. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r8 EngineCore hang A single llm.generate(3000 prompts, ...) call in _elicit_secure_code deadlocks pod-664's CUDA worker (vLLM v1 EngineCore stuck on IPC, 0% GPU util forever). Chunk _greedy and _sample internally into batches of VLLM_GREEDY_CHUNK_SIZE (default 500, env EPM_VLLM_GREEDY_CHUNK_SIZE); order- and structure-preserving, behavior-identical for every caller. The per-chunk [vllm-chunk] INFO log keeps poll_pipeline's freshness check alive over the previously 60-min-silent stretch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eadlock pivot, both-pods futex_wait_queue) The #664 r9 chunked-greedy fix landed but the EngineCore still deadlocks on futex_wait_queue across all threads inside the first llm.generate() on BOTH the wedged previous pod and the fresh recovery pod (epm:progress v54). r10 strategy pivot: env override to flip enforce_eager=True at launch, bypassing CUDA-graph capture entirely. Default unchanged (off). To use: EPM_VLLM_ENFORCE_EAGER=1 bash scripts/issue664_launch_parallel.sh
…CURE_CODE_MAX_NEW (prefix-cache scheduler deadlock pivot) enforce_eager (r10) was not enough: the first _greedy(300_prompts, max_new=1024) from _elicit_secure_code still deadlocks the vLLM v0.11.0 EngineCore on futex_wait_queue even with CUDA graphs bypassed (epm:progress v55). The remaining engine-config suspect is the prefix-cache scheduler: the secure-code batch is the FIRST call where all 300 prompts share one long system-prompt prefix, unlike the earlier marker_R per-source calls. Add enable_prefix_caching passthrough (default on, "0" disables) and an env-overridable secure-code max_new (default 1024 unchanged) as the escalation knob. Both defaults preserve current behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…on to agent memory Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…via shared helper r11 wired EPM_VLLM_PREFIX_CACHING into the dispatcher's _vllm_engine only; the dispatcher shells out to issue664_eval (p2 eval-gen) and issue664_extract_store (p2 extract), each with its own LLM() defaulting enable_prefix_caching=True, so the vLLM v0.11.0 shared-prefix EngineCore deadlock would recur at p2 eval-gen on the AdvBench battery AFTER the train+extract GPU spend (concern p2-llm-constructors-prefix-cache, reconciler-binding FAIL). Centralize the two knob reads into issue664_common.vllm_env_kwargs() (honors EPM_VLLM_ENFORCE_EAGER / EPM_VLLM_PREFIX_CACHING, case-insensitive, raises on a typo instead of silently re-enabling caching) and **-splat it into all three production LLM() constructors. Regression test pins the env semantics + AST-asserts every production LLM() routes through the helper (fails pre-fix, passes post-fix). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…re_code hang The r11/r12 smokes used identical placeholder prompts (vLLM dedupes them, ran 9.5s) and never reproduced the production deadlock on 300 DIVERSE prompts from phase2_insecure_code.jsonl. This harness drives the real production prompt set through the same chunked _greedy path so a hang here reproduces production. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…locks production launcher) v16 'hang' root cause: _render() reloaded AutoTokenizer.from_pretrained on EVERY call, and the p0 elicitation loops render hundreds of prompts per ctx in tight list comprehensions (marker_R alone = 300 questions x every source + negative ctx). The repeated disk loads added MINUTES of dead silence before each _greedy call (v16 sat ~5.5 min between 'engine built' and the first [vllm-chunk]), making a still-progressing phase read as a 0%-GPU hang to the poller and to the diagnosing observer. Real-diverse-prompt reproducer (scripts/issue664_real_prompt_smoke.py) proves the batch itself is fine under the r12 prefix_caching=False fix: 4 configs all PASS on pod-664 -- fresh n=300 (15.4s), prior-calls n=300 (15.4s), production-faithful n=3000 chunk=500 (6 chunks clean), and the exact v16 marker_R shape n=300 max_new=2048 (24.9s). The r11/r12 smokes used IDENTICAL placeholder prompts (vLLM dedupes them) so never reproduced production load. Fix: cache the tokenizer (functools.lru_cache, one load/process) + add per-ctx [marker_R] progress logging so a long phase is never silent. Mirrors the r6 _prompt_text_for cache (cf73e52) applied to the dispatcher's own renderer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s; correct r11 false-positive memory Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…s p2 rehydrated-adapter crash) The tokenizer is invariant under a LoRA merge, so load it from base_model_path rather than the adapter dir. A rehydrated adapter dir (downloaded from HF with only the LoRA artifacts, or one that does not exist locally yet) lacks tokenizer_config.json, and AutoTokenizer.from_pretrained then treats the absolute local path as a Hub repo id and raises HFValidationError (the p2 fan-out crash on all 8 shards). Same fix independently arrived at in scripts/rerun_arms_ac.py::merge_lora_fixed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…riment-implementer memory Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…training; rehydrate path declined per plan §4) The launcher ran p0 -> p2 -> p3, SKIPPING p1 (training) on the assumption that an external step trained the fleet. On a fresh pod with no local-volume persistence that assumption breaks silently: p2's extract_and_eval_cell finds no local adapter to merge and crashes (concern p2-no-adapter-rehydrate-train-skipped). Plan §4 "Recipe-only reuse (HARD constraint)" requires training EVERY cell fresh -- no prior adapter loaded at the weight level. So the launcher now trains the whole fleet itself: a p1 (train) block with the IDENTICAL 8-way CVD-pinned fan-out as p2, inserted between p0 and p2. The dispatcher already exposes --phase p1 standalone (run_all trains the shard subset via train_cell), so this is a launcher-only change. Shard i trains then extract+evals the same cells it owns; train_cell writes the adapter dir p2 reads AND pushes it to HF, and is idempotent (skip-if-adapter-exists) so a restart resumes cleanly. The 48 r1-r6 adapters on HF are NOT reused (plan §4); fresh training overwrites them via train_lora's normal upload. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…experiment-implementer memory The smoke marker-mix built 0 rows during the r15 launcher smoke because the p0 skip-if-cache-exists guard on the SHARED smoke/production marker_R cache path left the 4 smoke questions absent from the production-keyed cache. Generalizable trap for any builder keyed by question text with a shared-path cache guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P1 (training) crashed all 8 shards onto physical GPU 0 with OOM on the 2026-06-28 H200 launch. Root cause: the launcher passed `--gpu-id 0` to every parallel shard while relying on `CUDA_VISIBLE_DEVICES=$shard` in the shell env. But `_warn_if_cvd_disagrees` in src/explore_persona_space/ train/sft.py:151 warns then UNCONDITIONALLY rewrites `os.environ["CUDA_VISIBLE_DEVICES"] = str(cfg.gpu_id)` (line 1221 for train_lora, 1583 for merge_lora). So the gpu_id=0 clobber wins, every shard pins to physical GPU 0, all 8 OOM on H200 (143GB single-GPU cap). The pre-r16 comment "in-process clobber is silently defeated by import-time cuInit" was wrong — the warning's own message says "the env value is NOT respected here. ... pass +gpu_id=N instead". Fix: pass `--gpu-id "$shard"` in BOTH the p1 and p2 loops so train/sft.py pins each shard to its own physical GPU. The leading `CUDA_VISIBLE_DEVICES=$shard` is kept as belt-and-suspenders (it agrees with cfg.gpu_id, so _warn_if_cvd_disagrees stays silent). Also add `START_PHASE` arg so we can resume at p1 after a crash without redoing the ~25-min p0 setup that already wrote baseline_propensity.json and all training mixes. Default p0 runs the full sequence; each phase is guarded by `should_run`. Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
P2 (extract+eval) crashed 7 of 8 shards with `torch.AcceleratorError:
CUDA error: invalid device ordinal` at `mem_get_info(index)` inside
`AutoModelForCausalLM.from_pretrained(... device_map={"": gpu_id} ...)`
(extract_store.py:153/_answer_side_means, then _marker_slots). Root
cause is the same family as r16: r16 made the launcher pass
`--gpu-id "$shard"` to subprocess.run, but extract_store.py THEN used
`gpu_id` (1-7) as the in-process device index. After the dispatcher's
env pins `CUDA_VISIBLE_DEVICES=$shard`, the in-process CUDA view has
exactly one device at index 0, so `cuda:1..7` is an invalid ordinal.
Only shard 0 survived (index 0 is valid in the restricted view).
Fix three sites in scripts/issue664_extract_store.py:
- `_extract_all`: `device = f"cuda:{gpu_id}"` → `device = "cuda:0"`
- `_marker_slots`: same change, preserving the CPU fallback branch
- `_marker_slots` AutoModelForCausalLM.from_pretrained:
`device_map={"": gpu_id}` → `device_map={"": 0}`
The `--gpu-id` CLI arg is retained: it is forwarded to `merge_lora`,
which handles its own CVD pin (sft.py:_warn_if_cvd_disagrees + clobber).
Inside extract_store after that pin, ALL device math is on index 0.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
…nly) Plan §11 Option A->B routing: the a7-assert HALTed the fleet at p3 because 7 band-stopped marker cells read non-clean at the faithful use_rslora=True gauge. Adopt Option B (staged classic alpha/r=2.0, use_rslora=False, eval-apply only) per #601's recovery procedure. - merge_lora: add force_use_rslora / force_classic_alpha_over_r override kwargs (default None = faithful Option A). Override patches a COPY of adapter_config in a temp dir; original adapter dir + HF artifacts never mutated. - extract_store: --read-gauge {optA,optB}; optB+marker -> staged classic gauge. - dispatch: --marker-read-gauge {optA,optB} threaded to extract subprocs; --behavior filter restricts per-cell loops (p3 finalizers stay full-fleet). - launch_parallel.sh: EPM_MARKER_READ_GAUGE env -> --marker-read-gauge; optB also adds --behavior marker at p2 (16 marker cells re-extract; 32 non-marker p2 results stay valid). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r18 dispatcher (Path 2) Recovery for the disk-cleaned pod-664: only the 20 mk_* cells remain local; the 48 non-rf/sy cells are fully on HF; the 16 rf/sy cells were never trained. The r18 p3 finalizers assert LOCAL existence for the WHOLE 64-cell fleet and re-upload unconditionally -> crashes on the 48 absent cells + blows the HF 256-commits/hr cap. Cherry-pick (adapted) from main's WaveDispatcher dispatcher (lines 1197-1328): _cell_extract_eval_done / _expected_eval_files / _expected_store_files / _classify_cell_hub_state / _cell_artifacts_on_hub / _cell_done_anywhere, plus a new _hub_data_listing (one retried full-repo listing for the whole loop, the data repo intermittently 504s) and HF_MARKER_SLOT_PREFIX in issue664_common. Wired into p2 (_run_p2_loop skips a cell done locally OR on HF) and the p3 finalizers (_upload_raw_completions / _upload_store_tensors skip HF-complete cells, upload only fresh ones, exact-set verify the fresh uploads). ADAPTATIONS vs main: marker-slot HF surface is OPTIONAL (r18 never uploads it; A7 assert reads it locally; the 20 mk_* have it local), and _cell_artifacts_on_hub takes a pre-fetched listing to avoid 64 full-repo round-trips. Preserves the issue-664 vLLM deadlock-escape knobs and does NOT introduce fleet.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eet p3) Sequences p0/p1/p2 for sycophancy then refusal (each 4-shard data-parallel, CUDA_VISIBLE_DEVICES per shard) then p3 full-fleet finalize. The cherry-picked HF-aware-skip primitives make p2 skip the 48 HF-complete + 20 local mk_* cells and p3 upload only the 16 fresh rf/sy cells; the dispatcher's own _write_results_sentinel fires at p3 shard-0 and the wrapper emits the single terminal [phase=done]. EPM_MARKER_READ_GAUGE=optB carried through for r22 consistency (no-op for rf/sy). Loads .env for HF_TOKEN/WANDB_API_KEY. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ells phase0 derived its elicitation behavior set + build-mixes loop from the FULL realized grid (--behavior is applied in run_all AFTER phase0), so a `--phase p0 --behavior sycophancy` recovery still ran marker_R elicitation + build_marker and crashed on the under-filled `default` marker_R cache (4/300). Fix: scope phase0's cells to --behavior when set (symmetric with the p1/p2 loops), so the 16-rf/sy recovery only builds rf/sy on-policy caches + mixes and never touches marker_R. Also wire _cell_done_anywhere into the build-mixes loop (one HF listing for the loop) so a partially-rehydrated pod skips re-building mixes for cells already complete locally or on HF. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…d (64->48) On-policy elicitation of the two alignment-conflicting content behaviors on Qwen-2.5-7B-Instruct yields 0-5.5% judge-positive (epm:experiment-implementation v22), below any trainable floor -> plan v7 Option C drops them (a finding, not a code workaround; the 80% floor is unchanged). transfer_behaviors loses sycophancy + refusal; realized_grid now returns 48 (= 64 - 16 rf/sy cells). The syco/refusal eval COLUMNS still fire on surviving trained models per applies_to() / ROBUSTNESS_COLUMNS (v7 A18); only the trained adapter cells are removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 20 marker_slot_stats.json (PRIMARY marker-gate DV, §6.5 row 2) are present locally on pod-664 (r22 marker leg, A7 PASS) but absent on HF -- the r18/r23 dispatcher does not upload the marker_slot surface by design. Standalone idempotent uploader (content-hash skip; fail-loud post-upload count==20 + hash-match) closes the gap BEFORE pod teardown so the off-pod analyzer can read the primary DV (#613 teardown-loss class). --smoke uploads+verifies one cell. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fo (not list_repo_files) The smoke caught it: list_repo_files(recursive=True) 504-Gateway-Timeouts on the ~67k-file data repo (same flakiness as r23). The upload itself worked (librarian cell landed) but the verify crashed. Replace the full-recursive listing with a targeted get_paths_info on the exactly-expected paths (per-path, fast, 504-immune, 1 retry on transient 5xx). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #664. Phase 2 fine-tune fleet + trained store + ground-truth leakage. Plan v3, 88 GPU-h auto-approved.