issue #641 — Phase 2 matched-dose install-resistance curves#478
Open
superkaiba wants to merge 28 commits into
Open
issue #641 — Phase 2 matched-dose install-resistance curves#478superkaiba wants to merge 28 commits into
superkaiba wants to merge 28 commits into
Conversation
… dose ladder Vendor branch-only modules from issue-537@35d5e737 (NOT on main; #537 is awaiting_promotion): - configs/condition/i537_em.yaml (EM Hydra condition) - src/.../experiments/i537_contexts.py (context registry, build_messages/prompt, NEGATIVE_CIDS, load_registry, registry_hash) - src/.../experiments/i537_judging.py (judge_request_for_row, parse_verdict_em, em_rates_from_verdicts with n_included/excluded_frac/p_mis_sweep, submit_judge_batch_raw) Trainer edits (train_phase SFTConfig, plan #641 §4.1 Must-Fix #1 + dose-ladder enabler): - save_steps=getattr(training,"save_steps",500): thread save_steps so +training.save_strategy=steps +training.save_steps=25 emits the dose ladder (HF defaults 500 -> only 500+final at max_steps=560). Default 500 = no behaviour change for any non-dose-ladder run. - EPM_KEEP_ADAPTER_DIR=1 fence in _finalize_phase: the ladder checkpoints live INSIDE adapter_dir; the default reap would destroy them before per-checkpoint eval. The fence lets the #641 driver own per-checkpoint eval + reap. No-op for every other run. Library-drift reconciliation (parent issue-537 -> main): no breaking signature changes; alignment.py/batch_judge.py drift is additive (judge_dispatch cache_control, batch routing), _chunk_requests/format_betley_judge_user_msg/parse_judge_json/ DEFAULT_JUDGE_MODEL all present + compatible on main. Edit targets verified against the CURRENT (newer) trainer.py, not the parent's. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…yrun smokes
issue_641 package (genuinely-new code, plan §4.1):
- data.py: build_em_mix (vendored published-corpus loader + the 5th `default`
contrastive negative + the §4.5/§4.7 disjointness invariant by
resolved-prompt-hash AND canonical-persona-id, run against the realized
panels). widened_neutral_candidates HARD-excludes the police_officer ⟷
neg_sp_police collision a raw-cid check would miss. load_em_pairs vendored
in-package (no `scripts.` import dependency).
- stats.py: hierarchical bootstrap (seeds -> probes -> completions, coherent
subset only) with per-replicate saturating dose-curve refit; per-seed
asymptotes REQUIRED; H5 asymptote-difference + classify_h5; Arm-B ΔL +
classify_h1_h2 (equivalence read: H2 = ΔL<=-0.10 AND CI excludes 0; H1 =
CI subset [-0.10,+0.10]; else AMBIGUOUS); Arm-A diagnostic regression.
scripts/issue641_dose_curves.py: unified dispatcher (PASS_UNIFIED §4.12).
- phases base-propensity / run (build+train-ladder+eval) / aggregate / upload-raw.
- Hydra train cmd: +training.max_steps/+training.save_steps (append, not in
turner_em schema) + ++training.save_strategy=steps/++save_total_limit
(force-override, ARE in turner_em.yaml) -- verified via `train.py --cfg job`.
- poll_pipeline contract: [phase=...] lines + terminal [phase=done] + sentinel
with _SENTINEL_REQUIRED_KEYS (verified via poll_pipeline._parse_sentinel).
- §6.5 per-completion JSONL (aligned/coherent/binary_misaligned) + per-cell
aggregate (n_included/excluded_frac/em_rate_at_threshold_{20,40}) + a
raw_completions.json mirror per cell for upload_raw_completions_to_data_repo.
- adapter-ladder HF persist (one checkpoint-<step> per ladder point) +
raw-completions upload before [phase=done].
- smoke phases: --smoke (full GPU, PASS_UNIFIED), --smoke-cpu (data-gen real +
disjointness collision + bootstrap fixture, no GPU), --dispatcher-dryrun
(phase seq + sentinel, no GPU), --stage-inputs, --verify-imports.
CPU smoke + dispatcher dry-run both exit 0; verify-imports + ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan §4.5/§4.9 — the matched neutral is picked AFTER P0. Add data.select_matched_neutral (narrow pool closest base-propensity match; widen to widened_neutral_candidates, which excludes the negative-panel collision personas, if no narrow match within ±0.10; below-floor is REPORTED not dropped — graceful degradation) + the --phase select-neutral driver phase (reads base_propensity.json -> writes matched_neutral.json). P0 default contexts now include the narrow neutral candidate pool so the selector has their base propensities. phase_base_propensity resolves non-registry PERSONAS-key candidates via neutral_source_ctx. This makes the post-P0 neutral pick mechanical instead of a hand-pick (closes the deferred-production-path step). CPU fixture smoke + ruff + verify-imports + workflow_lint clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e, H5 sign Fixes the 5 code-review blockers (round-1 verdict upheld by the reconciler): - B1/B2: promote _resolve_ctx to a module-level _resolve_source_ctx used by phase_run, _eval_checkpoint, and the realized-sources build. Arm-B matched-neutral PERSONAS keys (data_scientist, ...) no longer KeyError on bare registry[source], and the §4.7 disjointness assert is now TOTAL over the realized panel (never silently drops a persona-key source). - B4: add a dedicated harmful_advice YES/NO compliance judge to i537_judging (HARMFUL_ADVICE_JUDGE_PROMPT_V1 + harmful_advice_rate_from_verdicts); P0 dispatches the right judge+parser per tag (em -> Betley dual rubric, harmful_advice -> compliance) so the Arm-B H1 covariate measures harmful-advice compliance, not the broad EM rate (plan §4.6). - B5: fix classify_h5 sign direction — H5b ceiling requires the CI entirely NEGATIVE (resistant below non-resistant) with gap >= 0.15 + flat slope; a positive gap returns H5_REVERSED, never H5b. Smoke: CPU carve-out exercises the resolver via a PERSONAS key, the disjointness collision through build_em_mix's production path, the H5 sign on positive+negative fixtures, and the per-tag judge dispatch; new tests/test_issue641_stats_and_judge.py pins classify_h5 + the judge (8 tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-pool reachability, bootstrap seed coherence B1 (armb-matched-dose-not-resolved): phase_aggregate now resolves the Arm-B matched dose from the Arm-A median install crossing (plan §4.5) via stats.resolve_matched_dose — first ladder step whose pooled-median Arm-A install (across 6 sources × 2 seeds) crosses 0.5; fallback step 375 when Arm-A absent / no crossing; --matched-dose default None (cli-override distinguishable). The chosen dose + matched_dose_source written to dose_curve_results.json. B2 (armb-widened-pool-never-measured): the default base-propensity P0 contexts now include widened_neutral_candidates() (approach (a)), so the §4.5 widened-pool fallback in select_matched_neutral is reachable — widened keys are in candidate_propensity. Factored _default_base_propensity_contexts() as the single source of truth for main() + the reachability smoke. B3 (bootstrap-seed-trajectory-incoherent): bootstrap_dose_curve and bootstrap_class_asymptote_difference draw boot_seeds ONCE per replicate and thread the SAME draw across every dose step (new _resample_step_for_seeds); seeds are the replication unit (§6.3). Arm-B ΔL single-dose path (bootstrap_armB_delta / _resample_records) intentionally unchanged. Smokes (all exercise the production paths): B1 runs the REAL phase_aggregate against fixture cells whose median crosses at step 250 (asserts dose_curve_results.json matched_dose==250 / armA-median-crossing) + fallback + cli-override; B2 asserts the widened candidate is selected over far-narrow ones + default P0 contexts cover the widened pool; B3 uses a 2-seed opposing-curve fixture + a per-replicate trace to assert a pure-seed replicate traces that seed's own curve at every dose (the broken per-step draw gives late-gap -0.05 vs 0.85). 6 new unit tests in test_issue641_stats_and_judge.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…er_ho ⟷ kindergarten_teacher) The plan body / package docstring use the PERSONAS slug `kindergarten_teacher` while the records gate keys on `ARM_B_TEACHER_CID` (= the #537 registry cid `sp_teacher_ho`); both resolve to the same system prompt. If the operator typed `--sources kindergarten_teacher,<neutral>` (matching the plan body), per-cell records keyed under `kindergarten_teacher` while phase_aggregate's `ARM_B_TEACHER_CID in records` gate looked for `sp_teacher_ho` and found nothing, silently dropping the H1-vs-H2 headline (armB delta=None, no error). Fix (approach a — defensive in BOTH directions): add `canonicalize_source` + `TEACHER_ALIAS_MAP` in issue_641.data and apply it at every records-keying boundary (P0 per_context, P3 per-completion source) AND on the aggregate read side (_load_cell_records, _canonical_per_context), so the teacher's cells always collapse onto the single canonical key the gate fires on regardless of which slug is typed. Docstrings (package + dispatcher CLI) now name both aliases. Closes concern armb-teacher-source-alias-mismatch (BLOCKER, round 3 reconciler). Tests: + alias-roundtrip unit test, + phase_aggregate fires armB under BOTH slugs (parametrized), + explicit failure-mode test (non-canonical-slug records satisfy the gate after canonicalizing load). 20/20 unit tests pass (16 pre-existing + 4 new); the production-path CPU smoke gains a p4_teacher_alias phase asserting armB fires under both slugs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nified router Single --workload-cmd entrypoint driving the GPU pipeline phases in sequence (base-propensity -> Arm A run -> select-neutral -> Arm B run); the matched neutral is resolved at runtime from base_propensity/matched_neutral.json and threaded into the Arm-B --sources arg. P4 aggregate runs off-pod on the VM (plan §9/§10), not in this workload. TQDM_DISABLE=1 guards the #607 GCE metadata-runner SIGPIPE trap; set -euo pipefail per the wrapper-pipefail rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lity GCP startup script's EXIT trap powers off the VM on rc!=0, and the eps-router service account has no logging.read IAM, so workload failures were fully blind. Wrapping the dispatch with bash 'set -x' tracing + tee-to-file + on-EXIT HF upload to superkaiba1/explore-persona-space-data:issue641_debug/ so we can post-mortem the next attempt.
vLLM 0.11.0 _run_engine line 1610: pbar.format_dict["elapsed"] returns 0 when TQDM_DISABLE=1 makes tqdm a no-op, causing division by zero in the throughput calculation. Fix: pass use_tqdm=False to llm.generate() directly (bypasses the buggy code path at the API level) and remove the TQDM_DISABLE=1 env from the dispatch wrapper. Caught via HF debug-log upload on GCP attempt #4 (2026-06-14).
… 96 completions JSONLs (P4 aggregate input)
…ched_dose=100; H5=H5a; armB=AMBIGUOUS)
…sity regression + coherence-vs-dose
…, regression label declip, coherence non-monotone disclosure + layout fix
superkaiba
marked this pull request as ready for review
June 16, 2026 07:25
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
# Conflicts: # eval_results/INDEX.md
…low-up identity-conflict-more-seeds)
… (identity-conflict-more-seeds follow-up) Additive aggregate-phase change for the 8-seed Arm-B re-fold: - --extra-records-roots flag + _load_cell_records(extra_roots=…) unions the parent run's dose_curves/ with the follow-up subdir so the hierarchical bootstrap pools the parent's 2 seeds + the 6 new seeds. - assert_schedule_parity (round-1 BLOCK fix): fails loud if any pooled cell was not trained at max_steps=560 / linear, naming the offending cell. Runs in production aggregate only. - _write_run_metadata persists per-cell run_metadata.json (max_steps, lr_scheduler_type, lr, warmup_steps, seed) at eval time; backfilled for the 96 committed parent cells (provenance-grounded at 560/linear). - Two production-path smokes: schedule-parity fail-loud branches + the extra-records re-fold seed union. Training recipe, eval rig, contrastive negatives, EM mix, Betley judge, and the hierarchical bootstrap are unchanged from the parent run.
…root-silent-drop CONCERNs
Two binding reconciler concerns from round-1 code-review (autonomous mode
does not defer CONCERNs):
stale-schedule-metadata-attestation:
- Persist training-schedule provenance at ADAPTER creation time
(<adapter_dir>/adapter_run_metadata.json) in _train_dose_ladder, the
moment the trainer subprocess succeeds.
- _assert_adapter_schedule_parity validates max_steps==560 / linear BEFORE
the resume-skip on an existing adapter; raises ValueError (naming the dir
+ mismatched fields) on mismatch, missing sidecar, or malformed JSON.
A stale 100-step adapter now fails LOUD upstream and never reaches
_eval_checkpoint, where its cell sidecar would otherwise be relabeled 560
and silently pass the pool-side parity assert.
extra-root-silent-drop:
- _validate_extra_roots asserts every explicit --extra-records-roots entry
exists, has dose_curves/, and contributes >=1 completions__*.jsonl;
called from BOTH _load_cell_records and assert_schedule_parity.
- _assert_armB_seed_set guards the production step-100 headline: the pooled
Arm-B teacher seed set must include both parent seeds {42,1042} + >=5 new
seeds (gated to !smoke + matched_dose==100). A silently-dropped extra root
no longer yields a 6-seed aggregate that still bootstraps cleanly.
Tests + smokes:
- tests/test_issue641_concern_fixes.py (14 tests): adapter-provenance
PASS/mismatch/missing/malformed; extra-root 4 bad shapes + happy path +
both aggregate entry points; Arm-B seed-set positive/missing-parent/
too-few-new. Each fails pre-fix (symbols absent), passes post-fix.
- run_smoke_cpu adds smoke_adapter_schedule_provenance +
smoke_extra_root_validation, driving the REAL new functions CPU-side.
Training recipe, eval rig, contrastive negatives, EM mix, Betley judge, and
the hierarchical bootstrap are unchanged.
Arm-B-only re-run at parent-identical schedule (max_steps=560 + save_steps=25),
6 new seeds {1,7,123,2024,31337,98765}, eval ONLY step-100 via --ladder 100.
Routes writes to eval_results/issue_641/identity-conflict-more-seeds/.
EXIT-trap uploads debug log to HF data repo for post-mortem (matches parent
dispatch script's pattern; the GCE startup script powers off on rc!=0).
…loor) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…al JSONs Per-cell EM judge records (completions__*.jsonl) + install-rate summaries (em_rate__*.json) + run_metadata for the 2-source x 6-new-seed Arm-B step-100 dose-curve cells. Pulled from the GCP instance eps-issue-641 (root-owned /workspace) via sudo tar over SSH; raw_completions.json excluded (lives on the HF data repo). These are the inputs to the off-VM P4 8-seed re-fold aggregate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8-seed re-fold of the kindergarten-teacher vs matched-neutral identity-conflict pairing at step 100. teacher-neutral gap -0.07, 95% CI [-0.24, +0.11] (39% narrower than the parent 2-seed [-0.26, +0.31]); still AMBIGUOUS. All numbers re-extracted from committed eval JSONs via the issue_641 stats module. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-extracts the identity-conflict-more-seeds pooled teacher-minus-neutral gap (-0.067, CI [-0.24, +0.11], AMBIGUOUS), per-seed gaps + coherent included counts (teacher 192 / neutral 166), 4/8 teacher-below-neutral, and the 16-cell schedule-parity result — all off committed eval JSONs. Schema mirrors the parent analysis/dose_curve_results.json. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… + fix collapsed layout
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #641.