issue-1482: error analysis of the n1M context→answer map#1240
Merged
Conversation
…s, launcher) Inline follow-up round fitter-fair-comparison-n50k. CODE ONLY (no launch). - issue779_ffc_n50k_generate_capture.py: 40k NEW LMSYS contexts disjoint from all 11,500 already used (3-phase re-derivation, ctx0 assert); multi-GPU sharded (--num-shards/--shard-index, context-range slice of a shared manifest); per-chunk HF upload -> sha256-verify -> purge local (peak footprint ~one chunk); raw rollout text uploaded unconditionally. Reuses the n10k _capture_shard/_generate/load_models. - issue779_ffc_n50k_launch.sh: build manifest once, fan out K CVD-pinned detached shards (K=nvidia-smi count default; --dry-run), nohup + per-shard logs + pidfiles. - issue779_ffc_n50k_fits.py: n_train=50,000. Byte-identical val/test (fixed_split 5000/3600/400/1000/42) asserted == pinned n10k shas; new contexts train-only. Primal ridge (batched over LAMBDAS_N50K), EXACT RBF KRR (chunked kernel + single-GPU Cholesky, no Nystrom), MLP w=8192 (n10k recipe reused), ridge+MLP-residual. n50k capture stream-reduced by layer from HF (peak ~one chunk). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…split pin Scope update 2026-07-14 (pod-779 not resumed; fresh pod-77950): the n10k round's 6,500 captures are presumed unavailable, so the fits DEFAULT to plan-B and the generation target rises to 45,000. - gen driver: N_N50K 40000 -> 45000. Disjointness unchanged (3-phase re-derivation excludes the original 5000 AND the recomputable n10k 6500, best-effort). Docstring notes the n10k selection is seed-deterministic + recomputed + excluded. - fits driver: --include-n10k-captures flag. DEFAULT plan-B = pass_b(5000) + all 45000 new (n10k NOT loaded); plan-A swaps in n10k 6500 + drops the last 6500 new (fetches the n10k bundle from HF, fail-loud if absent). Combined corpus 50,000 both modes; realized train 48,600 (val/test carved from the original 5000). - split-sha PIN now sources the ORIGINAL round's fair_comparison.json split params (recomputed fixed_split(5000,3600,400,1000,42), cross-checked vs ORIG_VAL/TEST_SHA256 constants) — NOT the pod-side n10k aggregate. MLP recipe likewise reads the committed original fair_comparison.json (w8192/lr3e-4). corpus_mode resume-guard splits plan-a vs plan-b. - launcher: pod-77950 named in logs/breadcrumbs comment only (no SSH hardcoding). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… exactly Fits DEFAULT split (plan-B) = orig-train 3,600 + n50k new = train 50,000; the n10k 6,500 captures live only on pod-779's stopped volume (no HF prefix), so they are NOT in the default pool. Generation target 45,000 -> 46,600 so the plan-B pool (3,600 + 46,600 = 50,200) clears the 50,000 target and trims to it deterministically. plan-A (--include-n10k-captures) now pools orig-train 3,600 + n10k 6,500 + all 46,600 new -> trim to 50,000, and FAILS LOUD at startup if the n10k bundle is absent both locally and on HF (never a silent fallback). val/test stay byte-identical / sha-pinned to the ORIGINAL round. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…me index All 8 shards died at startup: _remote_index's resume-index build listed the final_token_capture / raw_completions prefixes, which do not exist on the first run (nothing uploaded yet). list_repo_tree is a LAZY generator, so the 404 (EntryNotFoundError) fired on ITERATION at `for f in tree:` — OUTSIDE the try that only wrapped the generator's creation — and crashed the shard. The prior `except Exception` was also too broad. Fix: materialize `list(list_repo_tree(...))` INSIDE the try and catch ONLY EntryNotFoundError -> return an empty index (first run proceeds). Repo-missing (RepositoryNotFoundError, a sibling class — not a subclass) and every other exception PROPAGATE (fail loud). Covers both prefixes (single _remote_index). Smoke gains a monkeypatched no-network check: a lazy generator raising EntryNotFoundError on iteration -> empty index; RepositoryNotFoundError -> propagates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…obs (generate() hang triad probes b+c, gotchas #1092 H100 mitigation) Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
…d capture, streaming/Nystrom fits, multi-pod launcher)
n_train ~= 1,000,000 extension of the n50k round along the single variable n.
- generate+capture: 3-phase disjoint re-derive (round1/n10k/n50k) + LMSYS-to-
exhaustion (phase A) + WildChat-1M top-up to a 960k NEW pool (phase B), per-context
provenance; near-dupe contamination gate (exact + char-5-gram Jaccard>=0.8) vs the
1,400 pinned val/test prompts; TRIMMED capture (cx_last + v_x at layers {14,19,26}
only, ~82 GB); HF-hosted sampling manifest (--build-sampling-manifest uploads,
--manifest-from-hf downloads); external-stream checkpoint+resume per corpus (#1092).
- fits: 4 provenance-aware points (lmsys_150k/500k pure, mixed_500k ratio-matched,
mixed_1m full pool); streaming primal ridge (fp64 X^TX, ==N50 numerically);
minibatched MLP w8192 + w32768; residual-skip; chunked Nystrom KRR validated vs
exact at n=50k (gap>tol fail-loud); byte-identical val/test (sha-asserted);
per-(point,predictor) checkpoint/resume.
- multi-pod launcher: --shard-offset (pod k of M -> global shards [k*8,k*8+8) of
--num-shards 32), CVD-pinned per local GPU, --manifest-from-hf.
CPU smokes green (near-dupe, selection, manifest determinism, shard tiling,
streaming-ridge==N50, Nystrom~exact); ruff clean; imports OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…derived round1, not the tensors-only pass_b bundle
Runtime bug (pod-77960 manifest build): the HF analysis_tensors/pass_b bundle
is tensors-only — its `prompts` field is stripped on the analysis-tensors
upload path (_ANALYSIS_TENSORS_TEXT_FIELDS in issue779_collect), and
source/metadata carry no per-row text — so _valtest_prompts fail-louded at
startup ("no 'prompts' field").
Fix (team-lead directed): recover the 1,400 pinned val/test prompt texts
DETERMINISTICALLY from round1 (returned by N50.sample_disjoint_n50k, which
phase A streams anyway before excluding). round1 IS the pass_b prompt list in
pass_b row order; applying the original round's fixed_split(5000,3600,400,1000,42)
val/test indices to it yields exactly the pinned targets — costs nothing extra.
- _valtest_prompts -> _valtest_prompts_from_round1(round1, *, check_ctx0=True);
real path guards stream drift via a round1[0] ctx0 assert (mirrors
N50.build_manifest), smoke passes check_ctx0=False on a synthetic round1.
- build_new_pool reordered: sample_disjoint_n50k FIRST (yields round1), then
derive valtest, then build the near-dupe gate.
- _load_pass_b_bundle kept (fits reads pass_b TENSORS via it, unaffected).
- Smoke sub-check (1b): 1,400 targets recovered from synthetic round1 + ctx0
guard rejects a drifted round1[0].
Gen smoke green; ruff clean; both n1m modules import.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(fix fleet-launch os.replace race) At fleet launch, 8 shards per pod concurrently hf_hub_download'd the same 88 manifest files into the same local_dir; the winner os.replace'd each source into place and the losers' os.replace hit FileNotFoundError -> 21/32 shards died. Recovery was operational (relaunch dead shards --skip --manifest-from-hf against the winner-materialized dir); this makes it not recur. _download_manifest is now fleet-safe (three guards): - (1) short-circuit if the manifest is already complete locally (_manifest_complete_locally: meta.json parses + part_*.jsonl count == n_parts) — no lock, no network. - (2) download+move serialized behind an exclusive flock on <dest>/.download.lock, with a post-acquire re-check of the short-circuit so a shard that blocked on the lock returns the winner's materialized dir instead of re-downloading (per-open-file-description flock serializes both same-process threads and separate shard processes on one pod's filesystem). - (3) os.replace tolerates a missing source when target already exists (another process moved it). Smoke (7): two concurrent threads against a monkeypatched hf_hub_download — asserts both return the complete dir, one full 3-file download ran (the other short-circuited under the lock), dir passes _manifest_complete_locally. Single patch context spans both threads (per-thread mock.patch would race the restore). Run is LIVE — matters only for future relaunches. Gen smoke green; ruff clean; module imports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-commits/hr 429 storm)
The fleet was upload-throttled: 32 shards x 2 commits/chunk (capture .pt + raw
json) per ~2.5-min chunk ~= 1,500 commits/hr vs HF's 256/hr account cap, so
every upload burned ~30 min of 429 retries and shards were ~90% blocked.
run_capture now ACCUMULATES chunk files locally and flushes the pending group as
ONE upload_folder commit per artifact kind (2 commits per BATCH instead of 2 per
chunk):
- UPLOAD_BATCH (default 10, env EPM_N1M_UPLOAD_BATCH) chunks per flush; flush
fires at the K-cadence in-loop AND for the final partial batch at shard end.
- _flush_upload_batch: hub._upload_folder_filtered per kind (ONE create_commit,
no per-file tree-listing; EXACT expected-set verify inside the helper). The
.pt kind additionally sha256-verifies each file vs Hub LFS metadata; raw json
(non-LFS text) relies on the helper's presence verify. Verify+purge happen
ONLY after the commit returns a non-empty URL, so a failed flush loses nothing.
- SIGTERM-safe: a SIGTERM handler raises SystemExit so a try/except best-effort-
flushes the pending batch before exit; the prior handler is restored in finally.
- Per-chunk filenames UNCHANGED (shard{SI:02d}_chunk{ci:04d}.{pt,json}), so the
resume-skip stays compatible with the ~120 already-uploaded chunks. A killed
shard regenerates at most its pending un-uploaded batch (<= K chunks).
- The 1,800s per-upload retry budget (hub retry wrappers) is unchanged; at
~19 commits/hr fleet-wide the 429s clear.
Disk exposure per shard ~= K*43 MB (.pt) + raw < 1 GB, trivially under quota.
Complexity kept under the ruff cap by extracting _build_capture_engine +
_capture_stage_chunk from run_capture and _smoke_upload_batching from _smoke.
Smoke (sub-check 8, gen --smoke): _flush_upload_batch does exactly 2 commits
(one per kind) carrying the exact batch filenames, verify+purge only after the
commit, and the cadence flushes [10, 10, 3] for 2*K+3 chunks. Also made the
concurrent-download smoke (7) idempotent across runs (wipe _smoke_dl first).
Gen smoke green x2; ruff clean; both n1m modules import.
Run is LIVE — matters for future relaunches / dead-shard re-runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…x WildChat-region shard crash) One WildChat prompt longer than max_model_len (shard 18: 9086 tok) hard-crashes the whole 500-prompt llm.generate call, killing the shard. Filter each chunk at generation time against a budget of max_model_len - gen_max_tokens - margin (8192 - 1024 - 64 = 7104 rendered tokens, matching N10._generate's apply_chat_template render). The manifest is frozen — never rebuilt. Skips are order-preserving so kept prompts/cis stay 1:1 for capture (no within-chunk index skew), recorded to a per-shard shardNN_skipped.json sidecar (complete across resume; ci + token count only, refusal-safe). The fits enumerate by capture-presence (issue779_ffc_n1m_fits.py:981 over the assembled X), so skipped rows are simply absent — no fits-side change needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…etry (fix ~3.5h re-stream on transient Hub failure)
The HF per-chunk stream accumulated the assembled per-layer arrays in memory with
no checkpoints; one chunk's hf_hub_download exhausting HF's internal retries
(LocalEntryNotFoundError after two absorbed ReadTimeouts) forfeited ~3.5h at
1825/1920 chunks. Two fixes (code-style checkpoint-per-phase law, external-stream
presumption):
1. Per-chunk download retry: _download_chunk_with_retry wraps hf_hub_download in a
bounded outer retry (4 attempts, backoff 10/30/90s; env EPM_N1M_DOWNLOAD_ATTEMPTS)
treating LocalEntryNotFoundError / ReadTimeout / ConnectionError / Timeout /
408·429·5xx as transient; every other error re-raises immediately (fail-loud).
2. Stream-slice checkpointing: every STREAM_CKPT_EVERY chunks (default 100, env
EPM_N1M_STREAM_CKPT_EVERY) persist the accumulated cx/vx/ci + cursor atomically
(layer{L}.npz + layer{L}.cursor.json, tmp+os.replace; npz-first, sidecar-last
with n_rows torn-write guard). Startup resumes from cursor keyed on (layer,
hf_prefix, sorted chunk-universe sha); a mismatched/torn checkpoint is refused
loudly and re-streamed from scratch, never silently reused. A COMPLETE
checkpoint short-circuits the whole re-stream. --fresh-stream forces a clean
re-stream. Parts collapse after each checkpoint so peak memory stays ~one array.
Also fixes the --hf-prefix conflation: --hf-prefix is the CAPTURE prefix
(<round-root>/final_token_capture), so the manifest resolver now takes a separate
--manifest-hf-prefix (default the round root) instead of appending sampling_manifest
to the capture prefix; the local-stage path is unchanged.
Fits enumeration stays capture-presence-driven (no fits-side change): skipped/absent
chunks never enter the assembled X.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…edict train at selected lambda only fit_residual_skip passed [val, tr, te] to _ridge_streaming_multi_lambda, which materialized an (n_train, 3584) fp64 pred for ALL 23 lambdas at once (~330 GB fp64 / ~150 GB lower) → cgroup rc=137 at n>=500k. Split the ridge solve into _ridge_factorize + _ridge_predict_one (blocked, one eval set at one lambda); residual now selects the base lambda on VAL preds only, then builds the train+test base at the SELECTED lambda only (peak ~one (n_train, D) array ≈ 13.8 GB). Self-contained selection (no checkpoint reuse) so --predictors residual_skip backfill works with no ridge cell. Guard: _ridge_streaming_multi_lambda now REFUSES the train pool as an eval set (fail-loud). fit_ridge/fit_mlp/fit_krr_nystrom audited clean (no train-pool multi-lambda materialization). Refactor is numerically identical (ridge==N50 <1e-4 smoke parity holds). Smoke (8): guard fires + residual never routes the train pool through the multi-lambda path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…+ gate-probe tests
…ocation; tiny-model carve-out; verify/upload arity fixes
…th hardening Binding concerns: p3-rank-stats-corpus-order (seeded permutation of the corpus-blocked holdout before split-half halving, recorded seed 1482; midrank average-tie Spearman batched, no per-feature loop) + p3-serial-single-gpu (phase_p3 sharded into 10 units — 3 arm-Grams + 6 MLP fits + aux — via the existing _run_children machinery, re-shard-safe unit-JSON skip, width-agnostic; CPU smoke widens to N un-pinned slots via --n-gpus). Minors: per-row NaN n_ans (counted, >5% fail-loud) + nanquantile contrasts; NaN/missing test-retest kappa demotes H1 (instrument-failure); atomic results sentinel (tmp+os.replace); per-chunk checkpoint+resume for the P5 text re-fetch; tautological _bh_fdr assert replaced with the real property; Gate A reconciliation.json now carries the sae-arm-passb-text scope_caveats entry; scoped Hub listings routed through hub.list_hf_files_under_path.
…tolerated, malformed middle fails loud)
…pool) Root cause of the P2-pilot FVE in [-7900,-3400] / L0 253-2708: encode/decode match the reference verbatim (andyrdt/dictionary_learning@andyrdt/qwen), but the pilot pooled ALL token positions while the suite's training AND published eval (FVE 0.806 / L0 60) run under remove_bos: first-8 positions dropped + >10x-median -norm rows filtered + var-based FVE. Massive-activation tokens explode L0 (~42k features/row at 30x norm, locally reproduced) and drive FVE to -10^3. - issue1482_sae.py: fve_l0 -> reference eval semantics (outlier filter, unbiased per-dim var FVE, fp64 accumulators, diag counts, min-rows gate); BOS_OFFSET / OUTLIER_NORM_FACTOR / token_inlier_mask; parametrized dims for tiny-SAE tests; local-check CLI (stored cx_last rows: FVE 0.7387 / L0 31.9 at L19 k64). - issue1482_error_analysis.py: _pilot strips BOS positions + records diag; per-row _row_features masks pooled reads (ans trio + psi_mean), psi_last stays position-pinned; ctx_n_out/ans_n_out/ans_all_out shard columns. - tests: exact encode/decode/threshold-placement pins, reference-parity FVE vs demonstrably-differing legacy recipe, scale-fold equivariance, masking + gates. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change 1 (att-20260718-055220 crash class): _download_chunk_with_retry now rides hub.retry_transient (Retry-After-aware, EPM_HF_RETRY_BUDGET_S wall budget 1800s, LocalEntryNotFoundError transient BY CLASS) — the legacy 4-attempt/~130s ladder died at attempt 4/4 under a >=8-min queue-full 429 storm; every caller (P0 stager, P2 workers, _collect_texts) inherits. Also wraps the remaining P0/P2-reachable bare fetches: SAE config+weights load, model/tokenizer from_pretrained, sampling-manifest listing+download, pass_b bundle fetch, n1m chunk-universe listing. Change 2: --seed-partial <attempt_id> stages a prior attempt's crash-persisted P1 percontext units + sae_fitness into out_eval so the existing resume predicates skip completed work; per-unit json<->npz validation (invalid => re-run, never silent), post-P0 split-identity reconcile drops stale seeds loudly, gate_a_check records seeded_from.
…ipped the pilot's staging phase_p2 main now stages both trainers' config.json+ae.pt into args.sae_dir idempotently BEFORE _run_children, regardless of pilot skip (the pilot was the only staging path; seeded runs sent N workers into concurrent hf_hub_download races against one empty shared local_dir — #1315 class; pod-1482 p2_w0 died at torch.load(ae.pt) FileNotFoundError). New BatchTopKSAE.ensure_downloaded rides hub.retry_transient like load(); regression test fails pre-fix.
… raw-chunk enumeration (p2 w3 KeyError 'rows') Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
…e (SAE-arm 49,152-col targets OOM'd at dense-sized blocks) + expandable_segments in fit children Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
…rfeature npz JSONs (reconciliation, sae_fitness, split_1482, percontext + sae_perfeature summaries, interp_digest, perdirection_pca, seed_partial) are the plan's git destination. sae_perfeature/*.npz (15 files, 7.3 MB) force-added past the .gitignore *.npz rule: they are NOT in the driver's P4 HF upload set (sae_pooled + percontext only) and had no permanent home — pod-only data loss risk at termination. percontext/*.npz stay out of git: verified byte-exact on HF issue1482_error_analysis/analysis_tensors/percontext/ (4 files are 144 MB, over GitHub's limit). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…atch_meta durable fix
Recovery for epm:failure v6 (P5 died loading pod-local scratch after pod
termination): scripts/issue1482_reconstruct_scratch.py rebuilds
scratch/{split_indices.npz,row_ci.npy,prov.npy} deterministically on the VM
(pinned fixed_split shas, raw-chunk ci order == capture order per
_capture_stage_chunk, frozen manifest corpus, exact P0 carve replay) and
fail-louds unless every sha/count matches split_1482.json + the pod's own
percontext holdout_rows. Durable fixes: P4 now uploads the three scratch
metadata files to analysis_tensors/scratch_meta/ (verified), and
issue1482_analysis.py fail-louds with the recovery recipe on missing scratch.
…4 refusal .901/.922 format .824; drops content=4 transport=29) Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
…r LOWER, delta -0.0175 CI [-0.0221,-0.0129]) Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
…LOAD_PREFIX_EXEMPT x2, HUB_VERIFY_RETRY_EXEMPT x3 — lineage ports, not exercised as uploaders by #1482)
…es (HUB_VERIFY_RETRY_EXEMPT — attribute refs, no network calls)
superkaiba
marked this pull request as ready for review
July 19, 2026 08:35
superkaiba
added a commit
that referenced
this pull request
Jul 19, 2026
…to-merge (merge_ [task.py]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #1482.