fwdllm: delete legacy JSON launch path - #73
Merged
Conversation
…ing) The flame.launch migration is merged (#68/#72) and proven in real n=100 runs, so the second, divergent JSON/MPI config path is redundant and risks a behaviour regression if launched by mistake. Remove it per DELETION_CANDIDATES.md. Deleted: - expts/run_tc_expts/ in full — 155 json_scripts (5 aggregator variants + trainer_0..149) + fedavg_main_tc.py, launch_single_run.py, run_three_*.sh, run_text_classification.sh, gpu_mapping.yaml, mpi_host_file. - aggregator/fl_main.py, trainer/fl_main.py — legacy argparse+MPI entrypoints. Coupled cleanups (so nothing dangles): - configs/trainer_base.yaml — drop the dead gpu_mapping_file/gpu_mapping_key keys (device pinning is the spawner's job; FedSgdTrainer hard-sets its device) and refresh the provenance comment. - expt_scripts/diagnose_partition_binning.py — repoint DEF_CACHE off the removed dir (it also pointed at a stale home path). - aggregator/main_fedfwd_agg.py — drop the fl_main.py reference in a comment. Kept (live): expts/initializer.py, configs/aggregator_base.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Removes the legacy JSON/MPI-based FwdLLM launch path (now superseded by flame.launch + YAML experiments) to eliminate redundant entrypoints/configs that could be launched accidentally and cause behavior drift.
Changes:
- Delete legacy MPI/JSON experiment harness under
expts/run_tc_expts/plus the oldaggregator/trainerfl_main.pyentrypoints. - Remove dead
gpu_mapping_*config keys fromconfigs/trainer_base.yamland refresh provenance/comments. - Repoint
diagnose_partition_binning.pycache default, and clean up remaining comments/docs referencing the removed path.
Reviewed changes
Copilot reviewed 93 out of 168 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| lib/python/examples/fwdllm/aggregator/fl_main.py | Deleted legacy argparse/JSON entrypoint (superseded by main_fedfwd_agg.py). |
| lib/python/examples/fwdllm/trainer/fl_main.py | Deleted legacy argparse/JSON entrypoint (superseded by trainer/main.py). |
| lib/python/examples/fwdllm/expts/run_tc_expts/launch_single_run.py | Deleted legacy JSON patch/spawn launcher script. |
| lib/python/examples/fwdllm/expts/run_tc_expts/run_three_parallel.sh | Deleted legacy multi-run orchestration script. |
| lib/python/examples/fwdllm/expts/run_tc_expts/run_three_experiments.sh | Deleted legacy sequential experiment sweep script. |
| lib/python/examples/fwdllm/expts/run_tc_expts/fedavg_main_tc.py | Deleted legacy experimental entrypoint (not used by current launcher path). |
| lib/python/examples/fwdllm/expts/run_tc_expts/gpu_mapping.yaml | Deleted legacy GPU topology mapping file. |
| lib/python/examples/fwdllm/expts/run_tc_expts/mpi_host_file | Deleted legacy MPI host file. |
| lib/python/examples/fwdllm/expts/run_tc_expts/json_scripts/aggregator*.json | Deleted legacy aggregator JSON configs (part of removed launch path). |
| lib/python/examples/fwdllm/expts/run_tc_expts/json_scripts/trainer_*.json | Deleted legacy per-trainer JSON configs (part of removed launch path). |
| lib/python/examples/fwdllm/configs/trainer_base.yaml | Removed dead gpu_mapping_file/gpu_mapping_key keys and updated provenance comments. |
| lib/python/examples/fwdllm/expt_scripts/diagnose_partition_binning.py | Updated DEF_CACHE default path away from the removed directory. |
| lib/python/examples/fwdllm/aggregator/main_fedfwd_agg.py | Removed stale comment reference to legacy fl_main.py. |
| lib/python/examples/fwdllm/DELETION_CANDIDATES.md | Updated deletion manifest to reflect launcher migration being complete and document coupled cleanups. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+19
to
+20
| Verified via repo-wide grep: nothing on the live path imports any file below; the only references are | ||
| inside the doomed files themselves (or commented out). Deletion is safe. |
dhruvsgarg
added a commit
that referenced
this pull request
Jul 9, 2026
The legacy JSON/MPI launch-path deletion it tracked is done and merged (#73), so the manifest's purpose is fulfilled (rationale preserved in that commit + git history). Repoint the two migration docs that linked it: mark the legacy-code cleanup as completed instead of pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dhruvsgarg
added a commit
that referenced
this pull request
Jul 21, 2026
The legacy JSON/MPI launch-path deletion it tracked is done and merged (#73), so the manifest's purpose is fulfilled (rationale preserved in that commit + git history). Repoint the two migration docs that linked it: mark the legacy-code cleanup as completed instead of pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dhruvsgarg
added a commit
that referenced
this pull request
Jul 23, 2026
* fwdllm docs: consolidate real/sim parity + baseline catalog/comparison-fit + full deletion manifest
Consolidate the scattered fwdllm markdown:
- Fold PARITY_LOGICAL_TASKS.md into simulate_fwdllm.md as §I (felix async
grounding, durable) + §J (#15 residual + resume plan — the one open sim
front); repoint all refs; delete the temp tracker.
- Delete async_cifar10/real-sim_parity_checker_plan.md (its scripts/parity/
proposal is fully implemented — spent).
Add _metadata/BASELINES.md — the cross-cutting baseline view:
- §1 config/knob catalog for all 12 baselines (both families).
- §2 5-baseline restructure to crisp names: FwdLLM/FedBuff (round),
FwdLLM-It/FedBuff-It (iteration), FluxTune (async-smart); fwdllm_plus
renamed to fwdllm_it.
- §3 comparison-fit tables (separate from config catalog): which of
feddance/felix/refl/oort legitimately measure against FluxTune vs Felix,
with contribution / doesn't-fit / does-fit / decision columns.
EXPERIMENTS.md: "don't create unnecessary comparison points" note + naming map.
DELETION_CANDIDATES.md: complete manifest now that the launcher is merged —
entire expts/run_tc_expts/ (JSON/MPI path) + fl_main.py entrypoints, plus the
coupled trainer_base.yaml gpu_mapping-key cleanup so nothing dangles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: remove spent DELETION_CANDIDATES.md manifest
The legacy JSON/MPI launch-path deletion it tracked is done and merged (#73),
so the manifest's purpose is fulfilled (rationale preserved in that commit +
git history). Repoint the two migration docs that linked it: mark the
legacy-code cleanup as completed instead of pending.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: per-stage wall-budget instrumentation + #N bin-7 checker fix
Adds one-sided sim<=real wall-clock rungs (drain/trainer-phase) and
distributional/equality rungs (step_timing, aggregate/eval compute) across
the selection->compute->commit->aggregation pipeline, so a future #15-style
regression (sim silently slower at one stage) is caught directly instead of
inferred from debug counters. Also splits cohort_sequence_parity's SET/
CADENCE/VAR/ORDER into independently-scoped targets so a full-run check no
longer demands exact cadence past the known bin-7 float-nondeterminism wall.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix sim selector blind to modeled delay D (async_oort speed signal)
RC1: fwdllm_aggregator stamped PROP_CLIENT_TASK_TRAIN_DURATION (async_oort's
speed-penalty input) from the real wall duration (WALL_SEND-WALL_RECV) in BOTH
modes. In sim the trainer never sleeps, so this collapsed to raw GPU time
(~3-6s) for every trainer, hiding the per-trainer modeled delay D (8-36s). Oort
could not prefer fast trainers -> selection mix flattened -> cohort SET diverged
-> sim virtual clock ~1.9x real. Sole aggregator missing the `if self.simulated`
branch (asyncfl/oort/syncfl all have it).
Fix: charge the modeled duration in sim (SIM_CLIENT_TASK_TRAIN_DURATION_S =
max(gpu,D), already sent by the trainer); real path unchanged. Not flag-gated --
a broken sim speed signal has no valid A/B. fwdllm_aggregator-only;
async_cifar10 untouched. tests/mode -k "fwdllm or parity" = 361 passed.
Also bundled (pre-existing working-tree work):
- delay_factor -> training_delay_divisor rename + crisped docs (K-D38; wire key kept)
- step_timing_breakdown real-only-func exemption (K-D37) + tests
- BASELINES.md L0/L1/L2 contribution framing; profile_eligibility_gap.py (#7)
- simulate_fwdllm.md: RC1 re-diagnosis supersedes #1d as the driver
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: RC3 same-tuple double-pick fix (both modes) + version_key plan
RC3 (fundamental fix; supersedes RC2's phantom-evict band-aid): a trainer that
already contributed to a (model_version, data_id, iteration) tuple must be
excluded from re-selection for that tuple. The selector guard existed but was
starved because the aggregator (a) stamped _trainer_state_dict only under
`simulated AND _sim_inflight_residence`, (b) from _curr_agg_version at return
rather than the message's answered tuple, and (c) pruned the stamp on COMMIT
(`e in outstanding`) -> a fast committer was re-picked for the tuple it just
answered -> abort_training (no grad) -> in sim it stranded in the in-flight
ledger forever (phantom starvation), collapsing the selection mix -> ~1.9x slow.
Fix (both modes):
- trainer sends MessageType.DATA_ID in the grad message so the aggregator can
key the exact answered tuple;
- _process_single_trainer_message stamps _trainer_state_dict[end] from the
message tuple for every grad contribution, in BOTH modes (gate removed);
- _sim_hold_busy_slots prunes the guard by `v == _curr_agg_version` (keep until
the tuple advances), not `e in outstanding` (drop on commit).
fwdllm_aggregator-only; async_cifar10 untouched. Two stale guard tests rewritten
to the new invariant; tests/mode + tests/selector = 682 passed.
Also: [VCLOCK_PROGRESS] log now flags " SLOWDOWN" only when sim_rate < 1 (both
the fwdllm and base syncfl aggregators), instead of the always-on hint.
Doc (simulate_fwdllm.md): RC1 validated-but-insufficient -> RC2 phantom
root-cause -> RC3 fundamental double-pick fix (landed); new SS-M plan for a
uniform `version_key = (model_version, iteration)` formulation across
trainers/aggregators/selectors in fwdllm + async_cifar10 (model_version bumps
per variance-pass, purge inc_model_version_per_data_id, staleness per data-bin).
Operator decision: land all fixes (RC3 + SS-M Steps 1-3) before one consolidated
re-baseline; SS-M implementation continues in a fresh context.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: land uniform version_key formulation (§M Steps 1-3)
Unifies version/staleness/no-repeat naming across trainer, aggregator, and
selector (previously three inconsistent shapes) behind one shared
version_key property, fixes a latent cross-round trainer dedup false-abort,
and makes model_version bump unconditionally per data-bin (purges the
inc_model_version_per_data_id flag). Folds RC3's re-pick guard into the new
vocabulary and routes async_cifar10's no-repeat plumbing through the same
property for symmetry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: land version_key notes + trim simulate_fwdllm.md
Records the version_key unification landing (K-D39) and collapses the
RC1/RC2/RC3 investigation narrative in §A to a 2-line status. Deletes §I
(FELIX grounding, redundant with §B.1/K-D17b/K-D34) and §J (fluxtune
resume plan, superseded; its one open TODO moved into §A's Next roots),
removes closed items from the open-issues table, and compresses §G/§K to
the one-liner format the doc's own preamble already specifies.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* nit fix after adding version_key across aggregators
* status after fluxtune 0.25 diagnostic run
* fwdllm #S1: refute commit-time version_key hypothesis, add staleness census telemetry
asyncfl/top_aggregator.py's tres.version staleness calc is unreachable through
fluxtune's TopAggregator (full _aggregate_weights override), and the path
fluxtune actually uses (_trainer_last_model_version) is already dispatch-time
in both real and sim -- refutes the "stamp compute-time version_key through
SimReorderBuffer" fix from the prior session. vclock commit-gate independently
verified clean (81/81 commits).
Add telemetry to find the real mechanism: per-contributor dispatch_version_key/
agg_version_key_at_commit on agg_round.contributor_intervals, a new
version_bump_census event snapshotting the trainer pool's in-flight staleness
at every model_version bump, and model_version on comm events -- all keyed on
the unified version_key 2-tuple (K-D39), not a bare model_version int.
Doc: mark the commit-time framing REFUTED, record the new telemetry and
re-run command, add §F principle 17 (version_key is the only version-identity
vocabulary, enforce everywhere, no legacy scalar shim).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm §R: fix real-mode busy-trainer residence violation (async_oort abandon + release-on-return)
fwdllm's real async path could dispatch fresh work to a trainer still
mid-round: _release_end_on_return freed the re-pick guard on RETURN instead
of COMMIT (fedbuff only carries a returned grad until agg_goal fills), and
async_oort's 90s abandon timeout had no liveness check and evicted trainers
still within their configured round budget. Both are the real mechanism
behind #S1's fluxtune staleness gap. Fix holds the guard to commit in both
modes and makes the abandon timeout an aggregator hyperparameter
(send_timeout_wait_s, threaded via channel_manager.py like _seed) instead of
a bare constant. async_cifar10 untouched -- its base aggregator never had
this bug. 1027 passed, 7 skipped, 0 failed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm/async_cifar10: unify inflight-residence across sync+async, real+sim
_release_end_on_return special-cased is_async, but sync's own release-on-
return is only harmless because agg_goal == c there (a barrier, so every
return already belongs to that cycle's commit) -- not because sync needs
different semantics. Collapsed to one check: hold to commit whenever
inflight_residence is set, no is_async branch, so the invariant reads the
same regardless of baseline.
Renamed sim_inflight_residence -> inflight_residence everywhere it's read
(config.py, asyncfl/oort/fwdllm aggregators, all yamls/tests/docs in both
examples) -- the "sim_" prefix was misleading once the same flag governs
sync, async, real, and sim. async_cifar10's oort/felix mechanisms are
otherwise untouched (rename only, no behavior change there).
All 6 fwdllm baseline yamls (fwdllm, fwdllm_plus, fluxtune x real/sim) now
set inflight_residence: true -- confirmed no-op for the two sync baselines
(agg_goal == c means there's no surplus to strand). Full suite: 1028 passed,
7 skipped, 0 failed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: reconcile #S1/§A/§H with §R's root-cause (no contradictions)
§A's "Next (RESUME HERE)" and the #S1 open-issues row still described the
superseded "sim staleness under-modeling" / population-level phase-alignment
hypothesis after §R root-caused it as a real-side busy-trainer dispatch bug.
Updated all #S1 references (§A, open-issues table, Next-roots, §H, D3) to
point to §R instead of restating stale causality, and recorded the actual
validation-run plan (single node, all 3 baselines, 0.25/900s smoke now;
~5400s scoreboard re-baseline separately later) so a fresh context can
resume cleanly once results land.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: fold §R validation into §G/K-D40, close #S1 + D3
Validation smoke (run_20260711_132820 onwards) confirms the busy-trainer
residence fix: staleness rung PASSES, iters-to-clear-bin-0 exact, cadence
24/24. Delete §R, fold into §G/K-D40, close #S1 open-issue row and D3.
Add new open issue: fluxtune total_commits/terminal_state async-clock
artifact. Add two-node 5400s re-baseline command.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm parity: fix convergence round-keying + step_timing exemption, add async_oort pref telemetry
checks.py: convergence_parity/convergence_loss_parity keyed on FL `round`,
which fwdllm holds static -> collapsed every eval to one entry, comparing
real's/sim's LAST checkpoints at mismatched data_id instead of matched
progress. Re-key by data_id (mirrors total_commits/terminal_state). Also
exempt train_with_data_id from step_timing_breakdown gating -- it's a
wrapper around the already-exempted _emulate_training_delay sleep.
async_oort.py: port the round_preferred_duration_s/round_threshold/
_system_util_summary telemetry that sync oort.py already had but async
never got, so the Oort speed-penalty (system_util stuck at 1.0 in sim
telemetry, unlike real which varies and binds ~75% of rounds) can be
observed directly instead of inferred.
simulate_fwdllm.md: record the full-length re-baseline scoreboard, the
two landed checker fixes with before/after numbers, and the still-open
fluxtune Oort-penalty investigation with the next smoke command.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: quick offline test for the async_oort pref/system_util telemetry
Unit test (no GPU/experiment) that drives AsyncOortSelector.select() with
a fabricated duration spread and asserts round_preferred_duration_s/
round_threshold/sys_util_mean/pref_binds land in the emitted selection
event, and that the live-path pref matches the pure
calculate_round_preferred_duration() computation -- confirms the telemetry
added in 2348001a is correctly wired before spending a real GPU run on it.
Building the test surfaced a real candidate mechanism: calculate_total_utility
early-returns on an empty utility_list (all candidates "unexplored", i.e.
PROP_STAT_UTILITY is None) before ever touching round_preferred_duration,
which would leave it stale. Noted in simulate_fwdllm.md as directly
checkable once the next fluxtune smoke is banked.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: refute the unexplored-trainer dilution hypothesis for async_oort pref
Operator asked what the round-duration default is for a never-used trainer
(calculate_round_preferred_duration's 60s HACK placeholder for None
PROP_CLIENT_TASK_TRAIN_DURATION). Worked the actual percentile arithmetic:
with round_threshold=10/N=10 the index only lands on a placeholder when
fewer than 2 of 10 candidates have ever returned a grad -- a cold-start-only
edge case, not something that could hold across all 2845 sim samples in a
full run. Records the refutation in §H so it isn't re-derived later; the
open issue still points at the landed round_preferred_duration_s telemetry
and the next smoke run as the way to actually see what's happening.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm/async_oort: fix Oort speed-penalty never binding in sim (singleton-population percentile)
Root-caused via the round_preferred_duration_s telemetry landed last commit,
read against a fresh 900s smoke: calculate_round_preferred_duration was
called with filtered_ends -- the per-select()-call feasible-to-dispatch
subset -- which is a SINGLETON in 236/244 sim SEND calls (async dispatches
one freed trainer at a time) vs ~3 in real (MQTT-polling batches multiple
arrivals). On N=1 the percentile trivially returns that one candidate's own
duration, so the <= comparison is always true and system_util is pinned at
1.0 regardless of how slow the trainer actually is.
Reference Oort (third_party/Oort/oort/oort.py getTopK:267-273) computes the
percentile from client_list = ALL tracked arms, independent of what's
feasible for this call. flame's sync oort.py matches this incidentally
(its `ends` param is naturally the full per-round batch); async_oort.py's
port narrowed it to filtered_ends -- an unfaithful port, same class as the
earlier pacer() bug (K-D40). Fix: widen calculate_total_utility's duration
population to the already-threaded connected_ends (the full registry,
existing "Challenge 13" plumbing) -- safe since every other use is an
ends[id] lookup on ids already in the wider set.
Regression-guarded: TestAsyncOortSystemUtilTelemetry::
test_singleton_filtered_ends_still_uses_full_pool_for_pref, confirmed to
fail pre-fix (pref=36.0, the singleton's own duration) and pass post-fix.
Full suite (1030 tests) green -- async_oort.py is shared with
async_cifar10/felix.
simulate_fwdllm.md: moved the closed issue to §G, reprioritized Next roots
to the _real_intrinsic_clock checker artifact.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* async_cifar10/parity: fix _real_intrinsic_clock racing ahead on overlapping async cycles
_real_intrinsic_clock cumulative-summed each agg_round's own intrinsic_span_s
(barrier + eval) as if cycles ran strictly sequentially -- correct for sync
(one round in flight at a time), but fluxtune's async cycles overlap in real
wall-time (multiple cohorts commit concurrently), so the coordinate raced
~3.8-4x ahead of raw wall. This truncated the matched-V window that
total_commits/terminal_state/throughput/overhead_residual/per_round_advance
all share, crediting real with only its first ~25% of actual progress
against sim's full run (reported 76-86% rel_diff, actually ~19% when
re-derived with raw wall by hand).
Fix: return None for is_async baselines, falling back to raw wall ts --
the same fallback path already used for async_cifar10 (which never emits
intrinsic_span_s), not a new invented overlap-dedup mechanism. Validated on
a live 900s fluxtune smoke: total_commits rel_diff 0.857->0.167, throughput
0.814->0.235, overhead_residual/per_round_advance ~0.81->~0.20 -- all from
this one fix, now consistent with the ~19% hand-derived at full length.
Regression-guarded: test_parity_checks.py::TestIntrinsicSpanAsyncOverlap,
confirmed to fail pre-fix (25 vs 100 commits in the synthetic scenario,
matching the live pathology). Full suite (1032 tests) green.
simulate_fwdllm.md: both fluxtune structural blockers now closed in §G;
Next roots points at a fresh smoke to validate live (neither fix has been
observed on a post-fix run yet).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: live-validate both landed fixes, log new real-mode MQTT hang blocker
Cross-run validation (new sim run_20260712_000706 vs last-good real
run_20260711_231105, mixed-timestamp but valid since real's own behavior is
unaffected by either fix): preferred_duration frac_diff 0.812->0.214 (sim
binding 0%->60.3%, just 0.014 over the 0.2 tol), convergence avg_accuracy_diff
0.0739->0.0298 (now passing, confirms the speed-selectivity hypothesis).
Both fixes confirmed working on live telemetry, not just synthetic tests.
Logs a new unresolved blocker: two consecutive real-mode fluxtune hangs on
the same node (different hang points -- join-never-resolves, then
dispatch-then-receive-never-resolves). Ruled out both landed fixes as the
cause (sim exercises the same selector code path 245x/run with no issue;
hang #1 died before select() is ever reached; hang #2's select() succeeded
and returned a valid result, hang was downstream in RECV_FIFO). Also ruled
out real+sim broker contention (separate nodes, per operator). Leading
hypothesis: node-level resource/session state, given the aggregator/trainer
MQTT client IDs are identical across every run (not randomized per run).
Reprioritized Next roots: the MQTT hang now blocks all further fluxtune
validation, ahead of fwdllm_plus's sim_rate regression.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fold aggregate() wall into vclock, fix reselect telemetry gap, wire G1 grad-norm
- #15/K-D41: sim_rate<1 was NOT a trainer-side compute floor (D>>gpu at the
official basis) but aggregate()'s real per-cycle wall never credited to the
vclock (only eval_s was). Fix: sim_model_agg_compute_time fold, config-gated
OFF by default, enabled in all 3 sim yamls.
- K-D42: the SEND-reselect call never threaded agg_version_key/data_id through
channel.ends(), and random.py's extraction sniffed a 3-tuple shape that's
been dead since K-D39 unified version_key to a 2-tuple. Fixed both; unblocks
localizing the fwdllm_plus sim_rate regression via data_id/iteration.
- K-D43: wired G1 grad-norm telemetry aggregator-side (no trainer-side change
needed, contrary to the initial read) so the rung stops permanently
SKIPping.
- logical_parity.py: added a cohort-size axis (num_eligible/num_chosen per
data_id/iteration) — the receive-SET/cadence checks alone missed the
fwdllm_plus regression at bin<=1.
- aggregation_compute_wall_parity: added vclock_fold_diagnostic (cumulative
uncredited-wall fraction, both modes) so the fold's coverage reads as one
number instead of per-cycle means.
All new mechanisms pytest-guarded; full suite green (1040 passed / 7 skipped).
Not yet run-validated against real telemetry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* expt_runner: add wall-clock backstop watchdog so a hung run can't block the sequence
expt_launch's `wait "$run_pid"` had no external timeout — it fully trusted the
aggregator's own max_runtime_s self-stop, which the documented real-mode MQTT
hangs (simulate_fwdllm.md §A) show can fail to fire at all (a hang stuck at
MQTT join never reaches the code path that checks it). A hung run would block
`wait` forever, blocking every subsequent baseline/mode in a
run_sequential.sh sequence.
Adds an unconditional watchdog subshell (parallel to the existing ticker and
convergence watcher) that fires budget_s + grace past launch: TERM the run's
process group, grace period, KILL -9, sweep EXPT_WORKER_PATS. Reports
TIMEOUT_KILLED through expt_assert_run / run_sequential.sh's summary, distinct
from CRASH/LAUNCH_EXIT. Verified with ad-hoc functional probes (a real hung
process gets killed on schedule with the sweep working; a normally-finishing
run leaves the watchdog a clean no-op).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm/§M: event-driven sim recv/barrier redesign, replace reactive-EMA grace
Replaces the reactive-EMA barrier-timeout mechanism (_sim_recv_grace_s and
three divergent per-subclass copies of a budget fallback) with one shared
per-trainer delay cache on syncfl.TopAggregator (_sim_known_delay_s), keyed
on the deterministic MODELED_DELAY_S field. Barrier calls now either know a
trainer's exact delay or block genuinely, instead of racing a locked EMA
ceiling that burned 829.6s/1237s (67%) of fwdllm_plus's sim wall on
zero-progress polls. Also retires the redundant TRAINING_BUDGET_S message
field and caches fwdllm_plus's version_key-scoped SEND-reselect to fix its
real/sim call-count mismatch.
723 tests pass (lib/python/tests/mode + tests/selector; 1053 full suite).
Live real+sim validation not yet run this session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix RandomSelector k<c livelock; add vclock/wall phase telemetry
RandomSelector._cleanup_recvd_ends capped free-up at a stale k=5 (n~150-scale
leftover) instead of c=10, starving the full-cohort pool every round; with
§M's reselect-gate cache this livelocked fwdllm_plus sim (zero progress,
external-watchdog killed) and silently starved real (1 commit/15min). Fix:
drop k entirely (unused by every other selector), drain-all cleanup matching
async_oort's existing pattern.
Add a vclock_now property (aggregator + trainer base classes, None in real
mode) wired through _phase()/timer_decorator/build_step_timing, both
examples, both modes -- replaces ~15 hand-written `x if self.simulated else
None` sites and closes the gap on 6 previously-hand-timed-only fields
(gpu_compute_s/pre_train_s/post_train_s/aggregate_fedavg_s/eval_s/
mqtt_fetch_s). New analyze_run.py::phase_vclock_plots + checks.py::
phase_vclock_bottlenecks flag any phase where sim costs real wall beyond
real's own cost and its own vclock doesn't credit the gap -- re-derives
fluxtune's mqtt_fetch_s bottleneck (real 4.6s/sim 15.9s, 0% credited) and a
smaller instance in fwdllm (3.9s/6.1s) automatically from existing telemetry.
849 tests green (723 selector+mode, 126 async_cifar10).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: instrument fluxtune's grad-loop past-dated commits; add sim_sct_ordered_drain A/B yaml; trim simulate_fwdllm.md
Ports felix's (_sim_recv_min) past-dated-commit gap/source tracking into
fluxtune's _sim_recv_min_grad, which never had it (pops _sim_buffer
directly instead of through _sim_pop_committable). Needed to make the
open question -- whether fluxtune's opt-in sim_sct_ordered_drain
(drain_ready) is actually preventing message-stranding, or is dead
weight felix never needed -- measurable rather than argued from
docstrings. Telemetry-only, no behavior change; 723 tests green.
Adds fluxtune_n10_smoke_sim_no_sct_drain.yaml, an A/B sibling with
sim_sct_ordered_drain: false (felix's default recv_fifo path), delays
matched to the 2026-07-13 baseline run, ready to run standalone via
run_experiment.py.
Rewrites simulate_fwdllm.md's §A with this session's verified findings
(fwdllm_plus livelock confirmed fixed; fluxtune's sim_rate<1 root
traced to one-grad-per-composer-tick throughput backlog, not the
drain-primitive poll fallback) and collapses the superseded §G/§H/§K
hypothesis chains into single final-answer entries (798 -> 348 lines).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fold both legs of the sct_drain A/B into one yaml, single command
run_experiment.py always goes through ExperimentBatch, which runs every
entry in a yaml's experiments: list sequentially (sweeping stragglers
between runs) -- no need for two separate invocations. Adds leg 1
(sim_sct_ordered_drain: true, matching the 2026-07-13 baseline config
but re-run with the new pastdated-commit instrumentation, since the
banked baseline log predates it) ahead of the existing leg 2
(sim_sct_ordered_drain: false). One command now runs both back-to-back.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix yaml training_delay_factor path; instrument syn_0 UN_AVL mystery
The A/B yaml set training_delay_factor directly under trainer:, which
TrainerConfig doesn't recognize (only trainer.hyperparameters is a
real field, per runner.py:571's fan-to-aggregator read) -- silently
dropped. Confirmed via run_20260713_103707's execution_config.yaml
showing training_delay_factor: null. Moved under
trainer.hyperparameters for both A/B legs; verified via
ExperimentBatch.from_yaml that it now resolves correctly.
That run also surfaced a separate, real bug independent of the A/B
lever: at ~5 real-wall-clock minutes in, every trainer's
notify_trainer_avail thread pushed a transition to UN_AVL/AVL_EVAL,
which -- via a PROP_AVL_STATE/PROP_END_AVL_STATE key collision (both
resolve to the string "avl_state") -- fed the selector's eligibility
gate even though sim_unavailability/track_trainer_avail/
availability_aware were all off. Result: 9/10 trainers ineligible,
zero dispatches, zero commits, frozen vclock (so the vclock-gated
max_runtime_s check never tripped), and the composer's
select()-with-0-candidates loop busy-spun with no backoff (~500
calls/sec, ~5M log lines in 12 minutes, 975MB log file).
Traced the trace data (synthetic_traces.yaml's syn_0 entry) and the
loader (flame.availability.trace.load_trace) and both are clean --
verified live, single [0, AVL_TRAIN] entry, no per_trainer overrides,
checksum matches what this run's snapshot.yaml recorded. That
contradicts a second UN_AVL pop ever occurring from
check_and_update_state_avl's state_avl_event_ts queue. Added two debug
log lines (FedSgdTrainer.py) to catch what the runtime state actually
looks like on the next run: the raw avl_events_syn_0 content at init,
and the full pending queue right before each pop. Temporary --
removeonce root-caused.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: add short (8min) diagnostic yaml for the syn_0 UN_AVL mystery
Standalone single-leg run, sim_sct_ordered_drain: true (matching the
config that showed the bug), max_runtime_s trimmed to 480s -- long
enough to clear the ~5-minute anomaly window and capture the new
[DEBUG_AVL_SYN0]/[DEBUG_AVL_POP] log lines without committing to a
full 20-25 min run that may hang again before the deadlock itself is
fixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: NPU-calibrated per-baseline training_delay_factor (§O); ignore third_party/ae
Derives baseline-specific training_delay_factor values (fwdllm/fwdllm_plus:
1.63, fluxtune: 0.48) from real FwdLLM-paper NPU per-sample JVP cost and
telemetry-confirmed fwd-pass-unit counts (fluxtune ~4x fwdllm, not the ~10x
perturbation_count alone suggests), replacing the single shared CNN-derived
divisor (0.5) with values grounded in the actual forward-gradient workload.
Also ignore the vendored FwdLLM AE (artifact-evaluation) data dump under
third_party/ae/ -- a data bundle, not source.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: pin sim_wall_ceiling_s on the avl_debug diagnostic so it self-terminates
max_runtime_s's primary check is vclock-gated in sim mode. The
deadlock this run exists to diagnose freezes the vclock, so if it
recurs that check never trips -- confirmed in run_20260713_103707,
which spun 12+ real minutes past its 1200s budget until manually
killed. The wall-clock failsafe backstop defaults to max_runtime_s x
20 (9600s here), useless as a bound for an 8-minute diagnostic.
Pinned sim_wall_ceiling_s: 600 (480s budget + 120s slack) so this run
reliably self-terminates instead of requiring a manual kill.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix client_notify.trace baseline-default leak; audit+fix telemetry/plots gaps
runner.py: exp.trainer.availability.mode only selected which avl_events_*
DATA a trainer loads -- it never synced hyperparameters.client_notify.trace,
so a baseline's hardcoded default (fluxtune: mobiperf_3st_50 in
baselines.yaml) silently won over an experiment's syn_0 intent. Root-caused
via run_20260713_115005: [DEBUG_AVL_SYN0] confirmed the loaded trace was
genuinely a clean single-entry syn_0, but check_and_update_state_avl's
queue matched the mobiperf_3st_50 branch instead. Fixed by fanning
availability.mode into client_notify.trace (trainer side) and
trackTrainerAvail.trace (aggregator side) as the final, highest-precedence
layer -- same pattern as the existing training-delay fan (#12). Refactored
the trainer-side merge into _build_trainer_baseline_overrides so it's
testable the same way _build_aggregator_config already is; 5 new regression
tests. run_sequential.sh already patched this correctly for its own runs
(not affected), but any yaml launched directly via
`python -m flame.launch.run_experiment` was exposed.
analyze_run.py: a 2026-07 sweep of every telemetry builder against the
plot functions found aggregation_plots' residence-time CDF reading
residence_rounds off EVENT_AGG_ROUND, a key that event never carries (only
EVENT_INFLIGHT_RESIDENCE has it) -- silently producing nothing every run.
Fixed to read the right event, added a by-commit-class CDF and a
carried-over-age CDF from the same event's other unread fields. Added
phase_wall_vclock_plots: trainer_round's per-phase wall breakdown
(mqtt_fetch_s/weights_to_{ram,gpu}_s/weights_from_gpu_s/post_cpu_s/
mqtt_send_s) and agg_round's per-cycle wall decomposition
(barrier_wait_s/drain_tail_s/aggregate_fedavg_s/eval_s/intrinsic_span_s/
wall_elapsed_s/sim_rate) -- both existed in telemetry with zero plots.
Parallelized the plot-group driver loop (ProcessPoolExecutor); resource_plots/
write_summary stay sequenced after the gather since they have real ordering
dependencies the other groups don't.
simulate_fwdllm.md: added two working-checklist rules -- always use conda
env dg_flame (the wrong env silently skips deps like sortedcontainers and
produces misleading errors, not a real signal), and new debugging telemetry
must ship with its plot in the same change.
fluxtune_n10_smoke_sim_no_sct_drain.yaml: trimmed max_runtime_s to 600s
(10 min) per leg and cut verbose comments, for a fast A/B re-run alongside
a fresh matched real run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: safe-fast-path sim gate, visibility-lag telemetry, background eval
Landed the full backlog/past-dating fix plan (§G in simulate_fwdllm.md):
- Bug A: _sim_gate_is_safe checks the reactive earlier_stuck gate BEFORE
the blocking recv_fifo/drain_ready call (felix + fluxtune), using a
near-zero fast-probe timeout instead of the full per-trainer bound when
already safe -- removes ~2s mean/12s max wasted blocking waits per commit
without changing eager-drain behavior.
- Bug B: carried-surplus commits at a data_id boundary were 100%
misclassified as "round1" (felix's round-axis classifier, never re-keyed
to fwdllm's data_id axis). Now bucketed "carried_surplus" and excluded
from the primary pastdated_* counters.
- update_visibility_lag_s wired for fluxtune's async path and fwdllm/
fwdllm_plus's sync path (sim mode); real-mode sync anchor deliberately
left open (§9 Q5 in simulate_fwdllm.md).
- fwdllm's eval_model() backgrounded on a daemon thread (mirrors
async_cifar10's evaluate()), after fixing a real race: it was reassigning
self.fmodel/self.params/self.buffers, the same attributes the training
path assigns before aggregate(). sim_model_eval_time's vclock fold
removed entirely (dead once eval is off the critical path in both modes).
1072 tests pass (was 1057 at session start), 0 failures.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: add short (8min) real/sim fluxtune yamls for quick perf comparison
Short-runtime siblings of fluxtune_n10_smoke.yaml/fluxtune_n10_smoke_sim.yaml
(480s instead of 1800s, distinct job ids) for a quick real-vs-sim check after
the safe-fast-path gate fix (simulate_fwdllm.md §G) without waiting 30 min.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix parity-CLI progress-axis/glob bugs; root-cause overhead_residual; open the vclock-overhead question
Two checker bugs fixed (scripts/parity/checks.py, cli.py): _progress_axis()
picked round-vs-data_id independently per side (incommensurate units once a
7200s sim leg completes a data-bin lap), raw cycle_data_id collided across
laps, and --batch's glob matched fwdllm_plus for the fwdllm tag. Regression
tests added (517 total, tests/mode + scripts/parity/).
Re-ran the 7200s real/sim scoreboard post-fix and root-caused overhead_residual
to two distinct mechanisms (not one shared root as first assumed): fwdllm/
fwdllm_plus's gap traces to a growing real<->sim retry-count divergence
(v1_iter_per_data_id), fluxtune's to the classic unmodeled-transport-overhead
direction just over tolerance. Further investigation found overhead_residual's
real-side formula is defined to mirror sim's vclock formula by construction,
so it structurally cannot see aggregator overhead -- quantified ~10s/cycle of
real-only aggregator overhead (a per-message collection-loop clamp with no sim
analog) currently invisible to every check. Flagged §F #1 ("never put overhead
on the vclock") as under review pending that investigation.
Recorded a 6-point next-session plan (aggregator step_timing checks, the
num_min_req=1 clamp, overhead_residual's construction, trimming per-message
overhead, wiring selector/perturbation seeds across all baselines, and a
fluxtune post-boundary accuracy-drop investigation). Trimmed §G to the doc's
own ≤20-word-per-entry convention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: resolve overhead_residual/num_min_req/logging investigation; add flag-gated server momentum (S1)
Session 7 parity work: agg-side step_timing check (previously invisible
overhead), overhead_residual root-caused to retry-count drift (not a clock
defect), num_min_req=1 clamp investigated and kept (compose-loop coupling),
recv_fifo's per-end INFO logging trimmed to DEBUG (425k lines/112MB per run),
seeded yaml pairs added for all 3 baselines.
Server-side momentum (fluxtune_contributions.md S1): shared, flag-gated
(hyperparameters.server_momentum, default 0.0 = byte-identical) heavy-ball
momentum in FedSgdAggregator.aggregate(), damping the undamped direct-SGD
update implicated in fluxtune's random-walk/collapse instability. Enabled
only in fluxtune's yaml for now -- scope across baselines is an open,
evidence-gated question (fwdllm shows the same early collapse signature).
Short (8min) real-only with/without A/B pairs for all 3 baselines, plus two
node-split launcher scripts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: seed all short/momentum A/B yamls (hyperparameters.seed=1234)
Same seed on both A/B legs (without/with server_momentum) controls
selector + model-init RNG, so momentum is the only thing that can differ
between the two runs -- no separate seeded run needed to get a clean A/B.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix perturbation tie-break RNG leak; add selector/perturbation RNG fingerprint logging
_train_one_batch's JVP-selection tie-break drew from process-global,
unseeded np.random.choice instead of the client's dedicated torch_rng --
live for fluxtune (select_perturbation_using_jvp=true via baselines.yaml),
silently breaking real<->real/real<->sim reproducibility. Fixed to draw
from torch_rng. Also add rng_fingerprint()/[RNG_FINGERPRINT] logging at
the selector's and each trainer's RNG call sites, for auditing determinism
directly instead of diffing set/dict print order (which is per-process
randomized via PYTHONHASHSEED and was the source of an apparent-but-false
selector RNG divergence investigated this session).
* fwdllm: seed 30min baseline yamls; add 30min + 4h/N100 momentum A/B pairs
Adds hyperparameters.seed=1234 to the 30min fluxtune/fwdllm/fwdllm_plus
base yamls (missing from the 07-14 seeding pass, which only covered the
short/sim variants) and their new server_momentum=0.9 siblings, plus two
node-split launcher scripts mirroring the existing 8min ones. Also adds a
4h, N=100/C=30/K=10 fluxtune momentum A/B pair (EXPTS_CHARTER.md's "Four
planned full runs" concurrency scale) to reach past the round/lap boundary
that even the 7200s n10 parity scoreboard run never completed for fluxtune.
* fwdllm: root-cause momentum divergence (S1 REFUTED), fix selector RNG-order bug
Server-side heavy-ball momentum (server_momentum=0.9) diverges to NaN loss
by data_id 73 in a 4h N=100 A/B (no-momentum leg stays healthy, 0.35->0.86
acc). Root cause: heavy-ball assumes correlated gradients across steps;
fluxtune's per-commit grad is a high-variance single-sample directional-
derivative estimate (F7: variance floor never denoises below threshold), so
the momentum recursion amplifies uncorrelated noise instead of damping it,
worst exactly at the known position-locked collapse zone (F5/F13). Paused,
written up in fluxtune_contributions.md S1 row; default server_momentum
stays 0.0 (byte-identical) everywhere except the dedicated *_momentum.yaml
A/B legs, so no other run is affected.
While diffing the A/B telemetry, found select_random (async_oort.py,
oort.py, async_random.py) fed a seeded RNG's deterministically-ordered
sample through a bare set() before returning it -- set iteration order
follows per-process string hashing (PYTHONHASHSEED), not the seed, so two
same-seed runs pick the identical trainer set but dispatch them in a
different order every launch. Fixed via dict.fromkeys() (preserves RNG
order, still dedupes) at all three sites. Added a cross-process regression
test (order must be invariant across PYTHONHASHSEED) since a same-process
test can't observe this -- hash seed is fixed for a process's lifetime.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: finalize baseline naming (P)/+IT/+O grammar, restructure BASELINES.md
Locks (P) as the ported-strategy marker (FedBuff(P), Felix(P) — not
Felix-selector(P), since felix_round/felix_it port selection, optimizer
rate, and availability tracking together, not just selection) plus the
+IT/+O compositional suffix grammar. Restructures the doc into
self-contained Felix and FluxTune sections, each with one identity +
comparison-fit table, replacing the old catalog/matrix/comparison-fit/
naming split that repeated the same baselines across up to 4 tables.
Trims the decision-log narrative and changelog; current status only.
Naming is display-name-only so far — propagating it into
baselines.yaml/tests/launcher/smoke-yamls is tracked in the doc's
"Remaining work" section, not done in this change.
* fwdllm: trim status doc, link overhead_residual/C1 to RNG-order fix, correct sct_drain default
Scoreboard in §A predates the selector RNG-order fix landed later the same
session; note that as an untested hypothesis for closing overhead_residual/C1,
not a confirmed fix. Add the round_threshold-vs-duration-population diagnostic
for fluxtune's open preferred_duration gap. Correct sim_sct_ordered_drain：
schema default is false (felix uses it); fluxtune's yamls override to true,
unvalidated by the still-unrun A/B — prior doc text called this a "config
default". Add 3h/10800s two-node parity re-run commands with matched
--delay-divisor/--num-gpus per §O. Condense session-8 narrative into §G
pointers per the doc's own trim rules.
* fwdllm: scale smoke yamls to n=100 trainers, c=30 (fluxtune) / c=10 (others)
Bumps num_trainers 10->100 across all 6 real/sim smoke yamls, using the
full agnews H5 partition population and the full §O trainer_registry.yaml
speed-class mix instead of a 10-trainer subset. Concurrency pool c goes to
30 for fluxtune (async, higher JVP cost/commit per §O) and stays 10 for
fwdllm/fwdllm_plus (now a real c<n selection each round, no longer a
full-cohort barrier); minInitialTrainers scaled to match c per baseline.
No `k` kwarg added -- RandomSelector/AsyncOortSelector don't read one
(confirmed in code), so it would be dead config.
Doc: updated the 3h two-node commands with --num-trainers/--c/--c-async,
and flagged a run_sequential.sh gotcha caught by --dry-run: --c together
with --num-trainers (without --min-initial-trainers) silently pins
minInitialTrainers to num_trainers instead of c, so Node B's command
pins it back to 10 explicitly.
* fwdllm: root-cause + fix r1_inflight_overlap (first-dispatch eviction, not a version_key bug)
The 19.4% fluxtune sim r1_inflight_overlap violation traced to
_sim_hold_busy_slots: `outstanding` only counted `_sim_inflight_expected |
buffered`, and `_sim_inflight_expected` deliberately withholds an entry until
a trainer's delay is learned from a prior message -- so a trainer's
first-ever dispatch in a run is invisible to it and gets wiped from
all_selected the instant any other trainer's commit triggers a reconcile,
long before its own grad can return. The version_key/triplet guard was never
reached; the eviction happens one layer below it.
Fix: fold `_sim_pending_commit` (added unconditionally at dispatch,
discarded only at actual commit) into `outstanding`. Also made the
SEND_TIMEOUT_WAIT_S abandon path in async_oort.py discard from
`_agg_pending_commit_ref`, else a genuinely-abandoned trainer would stay
stuck in that set forever once outstanding depends on it.
2 new regression tests (confirmed fail pre-fix, pass post-fix); updated one
existing test whose hand-rolled setup skipped the discard the real commit
path always performs. 662 fwdllm/parity/selector tests pass, 0 regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: land TIMING_OVERRUN root-cause+fix, n=100 smoke-yaml scale-up, split off FWDLLM_DESIGN.md
TIMING_OVERRUN (357x fluxtune / 169x fwdllm_plus overruns): traced to §O's
margin using the fast-class MEAN as its reference trainer instead of the
class FLOOR -- every overrun traces to the same handful of floor-adjacent
trainers. Fix: training_delay_floor_s (config.py, FedSgdTrainer.py) floors
the raw registry delay before dividing, so only those trainers get a wider
budget; wired through run_sequential.sh --delay-floor. 4 unit tests
(resolve_training_delay_s).
Smoke yamls scaled to n=100 trainers (c=30 fluxtune / c=10 others) to match
the delay-floor derivation's basis. FWDLLM_DESIGN.md split out of
simulate_fwdllm.md as the non-parity companion doc (build/roadmap/JVP
profile), keeping section letters stable for existing code-comment
cross-references.
Also gitignore the vendored third_party/{FedML,FwdLLM,Oort,REFL} reference
repos (own .git each, ~1.6GB) -- reference-only, matching the existing
third_party/ae/ precedent.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix --delay-floor NameError in run_sequential.sh pre-flight
DELAY_FLOOR was captured as a bash var (--delay-floor) but never exported
into the python heredoc's environment, and never read there either --
DELAY_FACTOR got both, DELAY_FLOOR got neither. Any run passing
--delay-floor crashed in patch() with "NameError: name 'DELAY_FLOOR' is not
defined" before launching anything (confirmed live: this validates the
launcher-wiring gap the doc already flagged as "not yet run through the
launcher end-to-end").
Also surfaced delay_floor in the pre-flight review table (① tier) and the
two-node condition fingerprint, matching how delay_factor is already shown,
so a launched run's actual floor value is visible before confirming.
Verified via --dry-run: pre-flight now passes 5/5 checks and the generated
sim yaml carries training_delay_floor_s: 7.0 alongside training_delay_factor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: promote Felix(P) to EVAL, C2 to pending, add asymmetry rationale in BASELINES.md
Felix's preprint is now public, so Felix(P) qualifies as a second published
anchor alongside FedBuff(P); Felix(P)+IT stays an ABLATION since grafting
FluxTune's own iteration-level cadence onto Felix's selector isn't Felix's
native behavior. Also marks C2 (dynamic_kc) as pending rather than shipped
in the headline fluxtune config (was inconsistent with EXPTS_CHARTER.md and
the off-by-default Opt-4 flag), and documents the mechanistic reason Felix's
paper doesn't need to run FluxTune as a baseline (C1/C3 have no backprop
analog; C2 is the one exception, flagged as a Felix-internal ablation
candidate).
* fwdllm: fix r1_inflight_overlap false positives on FedBuff staleness, restructure parity doc
r1_inflight_overlap flagged FedBuff's legitimate stale-accept-and-downweight
redispatch as a residence violation (checked dispatch-vs-any-resolve instead
of same-version_key duplication); verified 0/882 prior violations were
genuine duplicates before rescoping the checker per version_key. Validated
0.0%/0.0% on the 5400s fluxtune delay-floor run.
Also restructured simulate_fwdllm.md into a strict §A (scoreboard) / §B
(per-baseline open issues) / §G (one-line closed ledger) format to stop
issues drifting into multi-paragraph, multiply-tracked entries.
* fwdllm: seed the sim, fix dispatch-order leak, instrument the trainer batch
Parity fixes
- Sim ran UNSEEDED while real carried seed=1234: the sim yamls simply omitted
the key. `seed` drives both the selector's dedicated RNG and the process-global
torch/np/random (model init), so sim drew a different selection ORDER and a
different initial model -- cohort_sequence set_match_frac 0.0 at cycle 0, and
every variance/convergence rung downstream of it was measured against noise.
Added to all 3 sim yamls; config.py default None -> 1234, since parity is not
opt-in and a config that omits the key must still be deterministic.
- `_handle_recv_state` still built its cohort with a bare `set(_pyrng.sample(..))`
in async_oort/fedbuff/async_random -- the same PYTHONHASHSEED order leak
`select_random` already fixed. The RETURN is dispatch order, so same-seed runs
picked identical trainers and dispatched them in a different order every
launch. `dict.fromkeys` for the order, set for the callers that .remove()/.union().
Telemetry (the trainer batch was a black box)
- timer_decorator keys off args[0].fwd_llm_stage; every helper inside
_train_one_batch is a nested fn whose first arg is `device`, so the entire
batch interior emitted nothing. The n100 run reported 3952ms/batch with no way
to say where it went. Added `_stage_timer` + 10 `tb_*` phases (same
step_timing shape, so existing rungs/plots read them for free) and phase +
unaccounted CDFs in analyze_run.py.
Checker fixes
- agg_step_timing_breakdown flagged 4 funcs, 3 spuriously: KS on all-zero
distributions (_release_end_on_return: every sample 0.0 on both sides) and on
tight distributions with a small systematic shift (eval_model: ks=0.50, means
4.2% apart). Added a degenerate skip + a 5% mean escape, and exempted
`_distribute_weights_async`, whose gap IS a hardcoded real-only sleep(0.1).
- New rung `trainer_speed_identity`: P3 enforces the speed DISTRIBUTION and
cohort_sequence compares IDs, so nothing checked that trainer X is the same
speed in both modes. It fails today on `utility` (47/100 scrambled) -- a
prediction the seed fix should clear.
- aggregation_plots died on a NameError (pc_x/pc_y/pgm_y collection loop was
missing), silently killing the whole group; restored.
Overhead removal (no numerics touched; RNG stream verified byte-identical)
- Determinism-audit hashes gated behind FWDLLM_PERT_AUDIT=1/DEBUG. `params
hashes` was unfiltered over all 67.4M params and ran twice per batch (~489ms
each) -- re-hashing 253MB of frozen weights, ~59x the ~10ms of actual math,
for a log line DEBUG-off discards (f-string args evaluate before the level
check).
- Deleted _force_cuda_memory_cleanup and the post-send gc/empty_cache:
empty_cache() issues cudaFree (device-wide sync) and returns every cached
block for the next round to re-malloc. Measured +14% wall AND peak 817MB ->
1090MB -- it made pressure worse; without it a 300-cycle soak drifts 0.00MB.
- Dropped 127k INFO lines/run and 157 lines of dead code (two unreachable stale
forks of the perturbation path; _randn_like_wrapper, whose per-perturbation
cuda.synchronize() served only a debug hash and was never called).
Tests: 1088 passed / 118 parity passed. The 3 test_config_generator failures are
pre-existing at clean HEAD (yaml says num_trainers=100, test asserts 10).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: cut aggregator per-commit waste, add min-initial-frac lever, seed-log consistency
Aggregator perf (correctness-preserving; 267 fwdllm tests pass):
- FedSgdAggregator.aggregate: the same full-model deepcopy block was written 3x
identically per commit (copy-paste; only last used) -> collapsed to 1x.
- gate ~10 eager _calculate_hash GPU->CPU sha256 calls (f-strings built even at
INFO, where the run has 0 DEBUG lines) behind logger.isEnabledFor(DEBUG).
- gate diagnostic-only stats (calculate_real_var/snr/snr_gradients/cv) behind
DEBUG; keep self.var (drives the live var<=threshold commit gate). Verified no
live/cross-file consumer of the gated stats.
Parity:
- run_sequential.sh: --min-initial-frac computes minInitialTrainers=floor(frac*N)
into selector.kwargs (all selectors); startup-cohort determinism lever for the
dispatch-order root cause (simulate_fwdllm.md §B fluxtune #4). Inert when unset.
- add trainer-side seed:1234 to 6 base yamls so selection telemetry logs it
consistently with the aggregator (inert: trainer self-selects, n=1).
Doc: simulate_fwdllm.md §A/§B/§F/§G refreshed with the 07-16 post-seed run
findings (dispatch-order root, queue-bound aggregator, eval-not-residual, §F-18
logging-consistency rule).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: gate remaining compute-intensive logging on trainer hot path
Trainer-side eager hashes (GPU->CPU sha256 built even when the level drops the
line) now gated behind logger.isEnabledFor(DEBUG); correctness-inert (reads only):
- fwdllm_trainer.py: full state_dict hash per weight recv; partial_grad hash per
recv; full grad-dict + var-check hash per send (was INFO -> DEBUG).
- tc_transformer: dead compute_metrics_with_logging_train per-example hash gated.
Already-gated sites left as-is (_pert_audit flag; isEnabledFor; prior commit).
Doc: simulate_fwdllm.md §F-19 -- no compute on the critical path for a log the
run doesn't need; gate any log whose args are non-trivial (hash/.cpu/.item/tensor
ops/repr) behind DEBUG or a purpose flag; keep at INFO only what sanity/plot
scripts parse + telemetry emit(); applies to all baselines + both roles.
267 fwdllm tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: rename _pert_audit -> _perturb_audit (and FWDLLM_PERT_AUDIT env var)
"pert" read like a misspelled "perf"; spell out "perturb" for the perturbation
determinism-audit flag. Renames _pert_audit/_pert_audit_enabled and the
FWDLLM_PERT_AUDIT env var -> FWDLLM_PERTURB_AUDIT, plus doc references. No
behaviour change; 267 fwdllm tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: trim verbose comments in run_sequential.sh
Condense the header flag docs + variable-default comments and several verbose
body/check blocks to concise one-liners; also document --min-initial-frac in the
header (was missing). Comment-only: 214 -> 154 comment lines, no logic change,
bash -n clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fwdllm: fix selector dispatch-order leak + parity/telemetry follow-ups
Selector determinism (root of fluxtune cohort_sequence cycle-0 divergence):
- Canonicalize candidate order with sorted() before the seeded draw in
oort/async_oort (fetch_statistical_utility), fedbuff (shuffle), refl_oort,
feddance -- ends' dict order = trainer JOIN order, which differs real vs sim.
- Default selector RNG seed to 1234 when none threaded in (no unseeded path).
- min-initial-frac: fires at ends_count >= threshold, so <N races the pool size
(98 vs 99); default minInitialTrainers to N in run_sequential.sh for a
set-exact initial cohort (caveat: blocks if a trainer never joins).
Parity checks:
- Add v1b_iters_moving_avg rung: moving-average iters-per-data_id over the run
within a tight bound (catches drift v1's pooled KS+mean misses).
- Exempt eval_model from agg_step_timing_breakdown (daemon-backgrounded,
off-vclock; wall gap is GPU contention). Trainer-side phase_gpu_compute stays
enforced, so a real trainer regression still fails.
Telemetry / reproducibility:
- channel.add stamps PROP_AVL_STATE=AVL_TRAIN at registration (kills the startup
UNKNOWN transient); trace/tracker overwrites per selection.
- snapshot.yaml records the resolved seed.
- analyze_run: iters_per_data_bin plot (smoothed, P10-P90 band).
Docs: simulate_fwdllm.md scoreboard/§B/§G refresh; FWDLLM_DESIGN §O floor
re-derivation (7.0->4.0) + §M sim-sleep audit (fluxtune clean; sync TODOs).
Tests: candidate-order insulation, v1b rung, eval_model exemption,
AVL_TRAIN-at-registration (init/aware/unaware). 478 selector+parity+avail green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* latest run results
* fwdllm: cohort_sequence tie-window, agg_goal=10 for fluxtune, step_timing p99 fix
- cohort_sequence_parity grants a SET/ORDER divergence a TIE (not a fail)
when every differing trainer's registry-derived expected delay is within
tie_window_s of the others' -- an arrival race, not a bug. Falls back to
strict when the delay model is unavailable.
- fluxtune's agg_goal 3->10: K=3 was too tight for its c=30 near-degenerate
fast class, admitting only a coin-flip subset per commit cycle.
- step_timing_breakdown's degenerate-noise skip was max-gated, letting one
outlier sample defeat it; switched to p99-gated.
- Renamed fwdllm/fwdllm_plus yamls _n10_smoke* -> _n100_smoke* (filename
only; num_trainers was already 100).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix fluxtune round-1 cold-start cohort scramble in sim reorder gate
_sim_recv_min_grad's earlier_stuck gate only arms once a trainer's delay is
cached in _sim_known_delay_s (reactive, first-observation-only, no fallback),
so round 1 -- every trainer's first contact -- starts with an empty gate and
commits whichever grad physically lands first instead of the true
sct-minimum. Two objectively-faster trainers miss cycle 0's agg_goal window
and land in cycle 1, cascading cohort_sequence divergence for the rest of the
run (mean cohort overlap decays 8/10 -> 1.8/10 over 520 cycles).
Add unknown_stuck: hold the pop while a probed end is still delay-unknown and
within sim_gate_compute_cap_s of its dispatch time, instead of oracle-seeding
_sim_inflight_expected from the registry (would give sim knowledge the real
aggregator can't have). Adaptive, unconditional, costs wall time once per
trainer ever. Re-derive sim_gate_compute_cap_s 16.0->11.0 for fluxtune (the
old value predates the same-day aggregator/trainer overhead-removal fixes).
Fluxtune (async) only -- fwdllm/fwdllm_plus's sync _sync_sim_recv_first_k is
a full-cohort barrier, immune by construction; their own cohort_sequence
fails are untriaged. Flagged (not fixed) the same theoretical gap in felix's
_sim_recv_min in async_cifar10/PARITY.md.
4 new tests; 432 fwdllm+parity tests pass. Landed, unvalidated pending the
next live fluxtune pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: validate round-1 cold-start fix, flag new cycle-1 refill divergence
6-min fluxtune real/sim pair (run_20260716_233147/_233128, cap=11.0 active)
confirms the unknown_stuck gate fix: cycle 0 is now a clean 10/10 real/sim
cohort match (was 8/10 pre-fix). cohort_sequence still fails overall from a
new, different divergence starting at cycle 1 -- real dispatches and commits
3 fresh replacement candidates by cycle 1, sim doesn't reach them until
cycles 4-6. Dispatch-timing gap at the candidate refill/re-selection step,
not an in-buffer ordering gap like the round-1 bug -- root cause not yet
found, flagged as the next item to pick up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix minInitialTrainers join-race, all 3 baselines
c<N reopened a post-barrier join-order race (sim's faster
cycle cadence vs real's wall-clock); set to N=100 in all 5
n100 parity yamls. Adds CLAUDE.md commit-hygiene rule.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: fix minInitialTrainers on the right yaml; widen pre-flight table
fluxtune's smoke pair actually builds from fluxtune_n10_smoke[_sim].yaml,
not the _4h file the last commit fixed; corrected there. Also widens the
pre-flight header bars/note column to match the per-baseline table width.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector: fix recv-state deadlock on never-touched ends
_handle_recv_state compared a fresh end's Python None state against
the string VAL_END_STATE_NONE, mis-claiming it into all_selected
before send-state could dispatch -> permanent stall once
minInitialTrainers=N releases all trainers at once. Fixed in
async_oort/async_random; 2 new tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector: remove recv-state's redundant resample fallback
_handle_recv_state resampled new candidates when idle, racing
send-state's real dispatch and (via a None/"none" bug) claiming
never-touched ends before they got any payload -> sim deadlock.
Removed entirely; it's now read-only over selected_ends, matching
its own docstring. Full suite: 1120 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: validate join-race/deadlock fixes, flag var_ok gap
Fresh 6-min pair: sim completes normally (72 agg_rounds, 0 stalls),
cohort_sequence divergence pushed cycle1->cycle2 (0/1 now exact).
Surfaces a new, separate var_ok mismatch at cycle 0 despite an
identical cohort -- flagged in §B, not yet investigated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* parity status, cohort_sequence wip
* Fix cohort_sequence tie-window and S2 windowing for fwdllm
Tie check now compares observed boundary-adjacency instead of bare
registry delay, resolving false cohort divergences from mid-run slot
refills. SET rung capped like CADENCE/VAR. participation_parity now
windows by cycle for fwdllm instead of degenerate round-keying.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Root-cause agg_step_timing_breakdown's GPU contention
Aggregator shares its GPU with ~5 trainers in both legs; sim's
no-sleep trainers make it denser there. Existing runner.py logic
already dedicates an idle GPU when num_gpus < visible count --
just needs --num-gpus 7 at launch, no code change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refute GPU-sharing hypothesis for aggregator gap
Dedicated the aggregator's GPU (--num-gpus 7), verified working,
but agg_step_timing_breakdown showed no improvement. Not a device-
sharing issue -- likely arrival-density/queueing instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Narrow agg_step_timing_breakdown to a GC-pressure lead
Per-call timing shows the sim slowdown isn't a consistent penalty --
it's outlier calls inside dense commit bursts, coincident with
back-to-back full-model deepcopies. Not yet confirmed; needs live
GC profiling next session.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Sub-time aggregate()'s var-calc and deepcopy blocks
Refutes GPU/burst-density/CPU-contention for the agg_step_timing
gap; it's a fixed per-call cost on both branches. New timers
localize it between calculate_var and the retry-cache deepcopy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Time aggregate()'s FedAvg update loop separately
var-calc and deepcopy sub-timers tracked real closely (n=40
smoke); the sim/real gap lives in the untimed commit-path
FedAvg loop instead. Deduped its verbatim-duplicated code into
one timed method to measure it directly next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Time aggregate()'s remaining commit-branch code
FedAvg loop only explains half the commit-branch gap; the rest
is branch-independent (also present on rollback), so it lives
in the shared preamble. Extract + time it as _prepare_round_state,
dedupe the last_round_update clone/detach too.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Instrument every remaining aggregate() sub-block
Full coverage in one pass instead of iterating per run: time
the retry-cache accumulate/append loops and, notably,
get_global_model_params() -- not a free accessor, it does a
GPU->CPU .cpu().state_dict() unconditionally every call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Full aggregate() gap attribution; reopen contention angle
Sub-block sum now explains ~91% of the gap: deepcopy (37%) and
the FedAvg loop (25%) dominate. All top contributors are plain
CPU/memory-bound tensor ops with no mode branching of their own,
which reopens ambient contention (not workload-scaling, already
refuted) as the mechanism. Needs live profiling, not more logs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Rule out two contention mechanisms, queue n=15 A/B
Core-scheduling and a sim-only background thread both ruled
out by direct code inspection (runner.py CPU reservation is
symmetric across modes; virtual_clock.py has no threading).
Queue a low-trainer-count run as a decisive causal test of
ambient memory/cache contention vs. a structural code-path gap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* NUMA-isolate aggregator; drop dead retry-cache deepcopy
Confirmed via an n=15-vs-n=40 A/B that ambient trainer memory
contention drives the aggregate() sim/real gap. Fix: reserve
the aggregator's whole NUMA node from the trainer pool (was
core-ID-only), and remove get_global_model_params()'s deepcopy
in _snapshot_retry_cache -- its result was never read.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* NUMA isolation: prefer other node, spill don't exclude
Excluding the aggregator's whole node forced >1 trainer/core
once trainer count exceeded one node's size (e.g. 100 trainers,
64-core node). Trainers now prefer the other node(s) and only
spill onto the aggregator's node's spare cores as overflow --
still 1 core/trainer up to 120, same ceiling as before this
change existed. TrainerSpawner takes an optional ordered
core_order instead of just an exclusion set.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Widen agg_step_timing_breakdown's mean tolerance to 50%
Validated the NUMA/deepcopy fixes: real win on the retry-cache
deepcopy (+31%->+11.5%), partial on the FedAvg loop, muddied by
sample-size noise. Diminishing returns from further root-cause
work, so widen this rung's tolerance -- CPU-bound aggregator
bookkeeping co-located with sim's denser-than-real trainer pool
has some inherent gap, unlike GPU-bound trainer JVP compute.
Trainer-side rung's 5% tolerance is untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix fwdllm CUDA crash, canonical grad merge
CPU path in ForwardTextClassificationTrainer no longer crashes on
missing CUDA; CUDA_DEVICE_ORDER pinned to PCI bus order. Aggregator
now merges each cohort's gradients in canonical (D, trainer_id) order
instead of raw arrival order, fixing a real/sim divergence in
cohort_sequence's var_match_frac gate across all three fwdllm
baselines.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Default fwdllm delay knobs per-baseline
--delays/--delay-divisor/--delay-floor now default to each
baseline's settled value instead of a global off, so a forgotten
flag can't silently collapse sim throughput again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Land P0-2 sim collect fix + P2-6 telemetry
Sim sync barrier now collects incrementally (persistent per-instance
buffer) like real's num_min_req=1, instead of one-shot batch-and-discard.
Adds tb_prepare_perturbation branch telemetry…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flame.launch migration is merged (#68/#72) and proven in real n=100 runs, so the second, divergent JSON/MPI config path is redundant and risks a behaviour regression if launched by mistake. Remove it per DELETION_CANDIDATES.md.
Deleted:
Coupled cleanups (so nothing dangles):
Kept (live): expts/initializer.py, configs/aggregator_base.json.