fwdllm: async FL simulator (client unavailability + real↔sim parity) + fluxtune aggregation optimizations - #72
Merged
Merged
Conversation
Completes Phase-1 Batch 1 with its pytest-only gate (75 fwdllm tests green): - restore flag-off byte-identical: set simulated=False on the fakes in four pre-existing fwdllm agg/trainer tests that the landed sim guards broke - trainer: no-sleep-on-sim-path + additive _sim_completion_ts stamp - async grad loop: _sim_recv_min_grad sct-ordered commit, one-in-flight gate, and agg-goal-boundary rollback safety (_release_sim_slots_at_agg_goal) - sync barrier: _sync_sim_recv_first_k first-k-smallest-sct + _barrier_anchored_lags Refresh simulate_fwdllm.md: crisp out landed Batch-1 build detail (now history in the code + deviation log K-D1..D8) and replace it with a §I Batch-2 cold-start map (variance-cadence V/DK/G rungs + fluxtune oort validation), verified against the parity engine and the emitted agg_round telemetry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lemetry Adds the fwdllm variance-gated dynamic-K parity layer (PARITY.md §F.4) to the shared parity engine, plus the telemetry it reads. Checker (parity/checks.py, run_all_parity, CHECK_META): - Stage 6': v1_iter_per_data_id, v2_var_trajectory, v3_cached_v_pool (DIAG), v4_force_commit_rate, v5_variance_pass_ratio (force-commits excluded) - Stage 3': dk1_agg_goal_trajectory (SKIP on fixed-K), dk2_dynamic_c, dk3_eligible_ends_metric - Stage 7': g1_grad_norm, g2_grad_pool_size All SKIP cleanly on non-fwdllm runs (async_cifar10 ladder unaffected). Telemetry (fwdllm_aggregator.py): add cycle_data_id/cycle_iteration/ grad_pool_size/cached_v_size to the agg_round extra, snapshotted BEFORE the pass/fail branch mutates data_id/iteration. The existing post-mutation data_id/iteration_per_data_id are left unchanged (analyzer depends on them; real-mode byte-identical). This makes V1 = count(cycles) grouped by cycle_data_id exact for natural-pass and force-commit paths (§K-D9). DK3/G1 emit deferred with a logged SKIP -- inert for all three current baselines (§K-D10); the checker rungs read-if-present and are unit-tested both ways. Tests: per-rung known-PASS/known-broken cadence fixtures + run_all_parity fwdllm smoke + non-fwdllm-SKIP (test_parity_checks.py); driven pre-mutation snapshot assertions (test_fwdllm_agg_telemetry.py). Gate green: test_fwdllm_* 81, test_parity_checks 63, parity sub-package 115. Deviation log §K-D9/D10 + Batch-2 status updated in simulate_fwdllm.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…onsolidate example run scripts
Add examples/scripts/expt_runner.{sh,py} as the shared launch harness both
examples' drivers source: robust conda activation, PYTHONPATH pin, launch +
progress ticker, a health check that scans the RIGHT files (agg_round in
telemetry/*.jsonl, stopping-run/SIM_WALL_CEILING in *_aggregator.log), the
timeout + process-group-kill campaign primitive (expt_timed_run), the --after
hook dispatcher, and a tiered ①②③ hyperparameter display + feasibility gate
(expt_runner.py render_and_gate).
fwdllm run_sequential.sh: now drives the real<->sim launcher PAIRS (the _sim
yamls were previously orphaned) via --mode {sim|real|both}, tags run dirs
_real/_sim so scripts.parity.cli can glob the pair, sets enable_training_delays
identically on both sides via --delays {on|off} (K-D8), pre-flight gate blocks
infeasible configs (agg_goal>c, num_gpus>visible, num_trainers<minInitial), and
--after {parity,sanity,plot} runs post-launch hooks.
async_cifar10 debug_run.sh: migrated onto the harness (dedup conda + launch),
gained the pre-flight table + --after {parity,plot}; smoke_suite.sh sources the
harness (kill logic -> expt_timed_run, also fixes a latent near-deadline TIMEOUT
misreport); run_felix_streaming.sh conda block -> harness. Delete
compare_overnight.sh (absorbed into --after parity,plot on the maintained
parity/cli.py; it used the legacy checker).
simulate_fwdllm.md: add curated §B.1 real<->sim design-delta table (KEEP CURRENT)
+ §I.6 launch pointer. Update real-sim_parity_checker_plan.md refs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remediate the three parity defects + one tooling defect the 2026-07-03 syn_0 3-baseline smoke surfaced (simulate_fwdllm.md §L). fluxtune sim ran 2x the forward passes of real -- a residence violation on the async grad path. §L.4 mechanism (D-a/D-b, fluxtune): - _release_sim_slots_at_agg_goal now HOLDS busy trainers before clearing and CARRIES the surplus buffer across the agg-goal boundary (commit-then-carry), residence-gated on sim_inflight_residence; flag-off => byte-identical drop for the sync baselines. Enabled via fluxtune_n10_smoke_sim.yaml. §L.4 telemetry + rungs (R1/W1, K-D14): - trainer echoes SIM_SEND_TS; aggregator captures each contribution's exact [dispatch, commit] interval and emits contributor_intervals on agg_round. - new R1 (in-flight-overlap INV) + W1 (compute-conservation DIAG) parity rungs in the shared engine + CHECK_META (R1 wired upstream of V1); validate_real gains a real-side check_grad_residence gate. §L.4 staleness (D-c, K-D13/K-D15): - self.staleness_policy was never wired from config (silently ran as "none"); fixed in internal_init. fedbuff is now an explicit accept policy feeding the existing 1/sqrt(1+staleness) down-weighting. Set in both fluxtune yamls. §L.5 launcher fail-fast (D-d): - wait_until_ready detects a startup traceback / early death (returns fast, prints log tail); runner terminates trainers immediately on non-zero agg exit instead of burning ~86s. Tests: full pytest tests/ = 793 passed / 7 skipped. New test_fwdllm_sim_grad_residence.py + R1/W1 fixtures + staleness/telemetry tests; fixed test_fwdllm_oracular_avail for the base_dir rename. Doc §L/§H/§K + a new principle #9 (test cadence) updated. Step 6 (the syn_0 re-run) is the operator's next action. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ion bars Pre-flight dry-run display was jagged and hard to scan. Improve the shared renderer (expt_runner.py) + fwdllm tier-② spec: - Section headers (①②③) now render as solid full-width background bars (blue/magenta/grey) instead of plain bold, so they're clearly visible. - Tier ② per-baseline knobs become an ALIGNED TABLE (baselines = rows, knobs = columns) instead of one crammed horizontal string per baseline. - Cross-baseline diff highlighting: any column whose value is NOT identical across all baselines is bold-yellow (the knobs to eyeball); columns identical across all stay dim (expected). Legend lists the differing columns. - Baseline names are cyan; flag-overridden columns get a green '•'. - Tier ① values align to one column (widest label), and the "ok" row marker is 2 cells wide to match the emoji icons so labels don't drift. Verified via the fwdllm --dry-run: agg_goal (10/10/3) and k (5/5/–) highlight, identical knobs dim. Backward-compatible: rows-based tiers render unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ng tail The review-every-run rows trailed each note unbounded on the same line, which read as noisy. Render label | value | note as three aligned columns and WRAP the note to indented continuation lines (bounded to the bar width). Value column is sized to the note-bearing rows so all notes start at the same column; a note-less long value (the baselines list) overflows harmlessly. Also shortened the two "unset (…)" var-knob values (the parenthetical moved into the note). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ey collision Ran scripts.parity.cli on all 3 real/sim syn_0 pairs. Result: the §L residence fix (D-a/D-b) is validated — fluxtune sim forward passes 228→82 (2x recompute gone), R1 in-flight overlap sim=0.0%, V1 iters-per-data_id 80-vs-8 → 7.67-vs-8.0 (near-converged). fwdllm_plus no longer crashes. Full results + interpretation written to §H. Also found + fixed a config key collision that left D-c dead in the run: the fluxtune yamls set snake_case `staleness_policy: fedbuff`, but the base config carries camelCase `stalenessPolicy: none`; the dict-merge kept both and the camelCase base key won at pydantic resolution (aggregator logged `staleness_policy = none`). Switch the overrides to camelCase `stalenessPolicy` so they override the base. (Residence escaped this — no base residence key.) Remaining before a clean syn_0 sign-off (documented in §H): re-run to confirm the staleness fix; add the missing `vclock_now` agg_round emit (a pre-existing gap that blocks the clock/throughput/convergence family for ALL baselines); and look at the smaller residual W1 overlap gap (sim now slightly UNDER-computes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated the §H re-run rungs against telemetry rather than assuming: - V1 FAILs are short-run truncation — iters-per-data_id match on every full data_id (fluxtune data_id 0 exact 11=11) and diverge only on the truncated last one; 3 samples make one boundary difference KS 0.33. No code change; needs a longer run. - W1 gap is the async start-tail, not a residence bug: received==commits and R1==0 (aggregator drops nothing); trainer_round counts starts and a live async real system leaves a bigger in-flight start-tail than the clock-gated sim. W1 is now asymmetric — flags only sim OVER-computing (the recompute bug), treats sim under-computing as benign. fluxtune W1 now passes. - vclock_now was never stamped on fwdllm agg_round, gating off the whole clock/throughput/convergence family (K10). Emit it (mirror asyncfl). Tests updated for the asymmetric W1; §H rewritten with the data-backed causes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standing rule in simulate_fwdllm.md §F #10: comment the WHY crisply (conceptual/ architectural choices, real<->sim divergences, failure-mode learnings); skip trivial mechanical edits; keep it to a sentence or two, deep rationale in §K. Applied it: trimmed the verbose comments added on this branch (_release_sim_slots_at_agg_goal, staleness/residence/contributor-interval/ vclock_now blocks, the launcher fail-fast notes, validate_real, the fluxtune yamls) to crisp form. No code behavior change; full pytest tests/ = 794 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…cker For the after-every-fix loop. Fixes two footguns of calling scripts.parity.cli by hand: the `*fwdllm*` glob also matches fwdllm_plus (wrong pair), and stale dirs get picked. run_parity.py parses the baseline as an exact token from the run-dir name, takes the LATEST pair per baseline, reads agg_goal from each run's config, shows the pairs to confirm (unless --yes), then prints a compact cross-baseline rung summary. --validate also reports live staleness_policy + vclock_now presence. Verified on the banked runs (auto-picked the newest fwdllm pair; vclock_now confirmed emitting live). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…a_id re-key Resolves the fluxtune async concurrency-starvation root (D-e, §H open-root #1) and two checker-completeness roots, so the syn_0 re-run can move toward parity. D-e (K-D16, D6=Option A): the K-D12 carry parked a returned-but-uncommitted trainer in async_oort's selected_ends, which drives the dispatch top-up (extra = c - len(selected_ends)), collapsing sim concurrency to ~1.5 vs real ~5.8 and blowing sim wall to 2.4x. Option B was proven unsafe: async_oort's triplet filter only guards within a cycle (triplet advances each boundary) and _trainer_state_dict was never populated. async_oort already tracks the two lifetimes in separate ledgers; stop conflating them. Override fwdllm's _sim_hold_busy_slots so a returned/carried trainer frees its compute slot (selected_ends -> a different trainer refills C, mirroring real freeing on receipt) but stays un-re-pickable (all_selected guard) until COMMIT; carry + variance-FAIL rollback survive. Populate the (model_version, data_id, iteration) triplet at dispatch so the within-cycle filter has real state. Scope: fwdllm class override only; async_oort/oort/asyncfl and the sync path untouched. Clock-family re-key (#2): unified _progress_axis / _per_progress_last_event helper; throughput/total_commits/terminal_state now count on the axis the run advances (fwdllm data_id, normal FL round). async_cifar10 auto-detects round -> byte-identical. field_coverage aliases (#3): spec rows accept a tuple of names; gpu_compute_s / training_budget_s also match fwdllm's real_gpu_time_s / sim_round_duration_s. Tests: TestOptionASlotGuardSplit, TestProgressAxisRekey, TestFieldCoverageAlias + updated residence assertions. tests/mode + parity sub-package: 421 passed, 7 skipped. Doc: K-D16, F-D6 resolved, H, B.1, L updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t to commit); R1 regression open
Fixes the fluxtune async sim deadlock and the in_flight mis-measurement; leaves
one known open regression (R1) documented for the next session (§H NEXT SESSION).
K-D17 — unblock the K-D16 deadlock (fluxtune sim committed 2/10 grads, 0 agg_rounds):
- Bug A: _aggregate_grads_async now drains while _sim_buffer/_sim_inflight_expected
is non-empty even when the channel reports no RECV end. The sct reorder buffer,
not the transient channel RECV state, is the sim commit path's source of truth
(RECV gating is a real-transport artifact; principle #8).
- Bug B: the async_oort re-pick triplet is stamped on grad RETURN
(_process_single_trainer_message), not at dispatch — stamping the whole cohort at
_curr_agg_version froze the eligible pool before any commit could advance the
version (bootstrap deadlock the D6 note missed).
K-D17b — in_flight measurement (revert K-D16 Option-A slot split, realign with felix):
- _sim_hold_busy_slots now holds every dispatched-but-not-committed trainer
(computing ∪ carried) in BOTH selected_ends and all_selected until COMMIT, called
per-commit in _sim_recv_min_grad. len(selected_ends) now == virtual-time in-flight,
matching asyncfl::_sim_hold_busy_slots/_sim_recv_min. Validated: sim in_flight
3.38 -> 9.49 vs real 9.75; V2 var-trajectory now passes.
KNOWN OPEN REGRESSION (do not launch overnight fluxtune sim until fixed): K-D17b broke
R1 residence (0% -> 44.7% same-trainer in-flight overlap). Root lead: buffered ends
reset to KEY_END_STATE=NONE become re-selectable because async_oort doesn't exclude
all_selected. Full analysis + ordered fix options in §H "NEXT SESSION" + §K-D17b.
Scope: fwdllm class only; async_oort/oort/asyncfl + sync path untouched (fwdllm/
fwdllm_plus sim byte-identical). Full tests/mode green (315 passed / 7 skipped);
new test_fwdllm_sim_drain_and_repick.py + TestVirtualInflightSlotHold.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-green
Lands all instrumentation/checker/liveness dev so the Phase-1 sign-off run
measures a near-complete ladder. Full tests/ 854 passed / 7 skipped. Also
banks the prior uncommitted K-D19 (R1 guard-release-to-COMMIT) + piece A
(vclock-budget stop).
Stage C — fwdllm_plus real-mode sync-barrier liveness under scarcity:
_await_dispatchable_under_scarcity sleeps-to-next-avail (WAIT/accumulate,
cohort stays == agg_goal, parity-faithful) instead of hot-re-dispatching;
self-terminates at max_runtime_s; one warn/stall. No-op at full availability
(syn_0) and on the sim path. + test_fwdllm_sync_scarcity_wait.py
Stage A — instrumentation (un-skips the 12 rigor-gap rungs):
A1 trainer per-phase timing (_phase/_phase_times + pre/gpu/post/budget/phase)
in trainer_round; manifest note updated.
A2 agg per-round wall decomp (barrier_wait_s/drain_tail_s/aggregate_fedavg_s/
eval_s) in agg_round.
A3 re-key advance rungs K3/K4/K3a/K3b to the data_id progress axis (shared
parity engine; async_cifar10 byte-identical).
A4 new wall_disparity DIAG rung (|real_wall - sim_vclock| per matched unit).
Stage B — sct-model folds, config-gated OFF => byte-identical:
B1 simModelEvalTime (charge measured eval wall to the vclock),
B2 simStragglerSpreadS (stable per-trainer straggler offset, sim only),
B3 simWanTransferS (documented knob, left 0). Emergent calibration = the run.
Stage D — checker/telemetry gaps:
D1 fwdllm_trainer re-emits task_recv.sim_send_ts (un-SKIPs K6),
D2 failsafe_ok/K5 compares sim wall vs the RUN wall budget for a real-compute
sim (auto-detected), SKIPs without a run budget.
Stage E — gate the two _distribute_weights sleep(0.1) pads off the sim path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase-1 now defaults to syn_0 and the pre-flight table shows the RESOLVED config actually patched into each launched cfg (print == run), not a hardcoded default that masked the yaml's own mode. - run_sequential.sh: --avail-trace defaults to syn_0 (Phase 1), so patch() always sets trainer availability.mode + aggregator trackTrainerAvail.trace + client_notify.trace EXPLICITLY on every baseline. The trace row + a new per-baseline `avail` column are read back from the patched cfg. New feasibility gate BLOCKS a full-participation sync barrier under a non-syn_0 trace (sync agg_goal >= n_trainers + unavailability -> can't assemble -> stall; --force to override). - fwdllm_plus + fluxtune yamls (real+sim): mode flipped mobiperf_* -> syn_0; switch to a mobiperf trace via --avail-trace only for Phase-2. - simulate_fwdllm.md: K-D22. Verified: generated launch cfg carries syn_0 in all three availability fields; --avail-trace mobiperf_2st --only fwdllm_plus blocks at pre-flight (exit 2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Post-sign-off work (simulate_fwdllm.md §H). The 2026-07-04 sign-off run was unusable: every sim hit [SIM_WALL_CEILING] with vclock << wall (sim_rate~0.37x = a SLOWDOWN, root #13), real/sim reached mismatched data_id. Roots traced to (a) the sim still paying real transport waits it must skip (#13) and (b) the config flow (#12). This lands the two prerequisite phases; Phase 2 (the leak fix) and Phase 4 (stopping rule) remain. Phase 1 — speedup instrumentation (make the slowdown observable): - fwdllm agg_round now emits wall_elapsed_s (both modes) + sim_rate=vclock/wall (sim), and a live throttled [VCLOCK_PROGRESS] log (the inherited base log never fired on fwdllm's composer path). - new sim_speedup DIAG parity rung asserts the principle-#13 invariant (sim_rate >= 1) and reports wall_speedup = real_wall/sim_wall. K7 sim_rate only checks a sane range and silently passes a slowdown; this catches it. Validated on the banked 134801 sim: sim_rate=0.377, wall_speedup=0.986. - Phase 1d PDF plot deferred (metric already in telemetry+log+rung). Phase 3 — config-flow correctness (#12, framework fan chosen): - runner._build_aggregator_config fans exp.trainer.enable_training_delays (+training_delay_factor) into the aggregator hyperparameters as the final layer (single source, both roles agree), with a honored-100% tripwire. Safe for async_cifar10: training_delay_enabled is read only by the trainer, never the aggregator base, so the fanned field is truthful-but-inert. - effective delay flags banked in execution_config (were absent -> a --delays run "looked" off post-hoc). - run_sequential.sh --delay-factor knob (default trainer_base=10 shrinks the 4-18s registry delay /10; pass 1 for the full modeled delay). Principles #12 (consult async_cifar10 PARITY.md vclock rules) and #13 (vclock is virtual wall-time; sim must produce speedup, vclock >= physical-wall) locked in the doc, plus the phased plan and a Phase-2 cold-start leak inventory. Tests: launch 120, parity engine 208 (incl. async_cifar10), fwdllm/telemetry mode 228 — all green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ sct folds) Pre-next-run dev for the fwdllm real+sim, telemetry-grounded (principle #11a). Phase 2 (root #13, K-D23) — sim was a slowdown because it paid per-round wall with no fidelity value. Banked `134801` telemetry + async_cifar10 map REFUTED the doc's "trainer fetch is the dominant leak" hypothesis: the trainer recv is barrier-wait overlapping irreducible agg eval+GPU (must stay). The skippable per-round wall was two ungated sleeps, gated `and not self.simulated` (fwdllm-only, real byte-identical): - trainer `pause_execution` (time.sleep(1) at the tail of every loop iter) - `_check_availability` avail-spin (would freeze the vclock in sim) Phase 4 (K-D24): - 4a (root S1): decouple `sim_wall_ceiling_s` default from 1× max_runtime_s to `max_runtime_s × SIM_WALL_CEILING_FACTOR` (=20) so a real-compute sim (wall >> vclock by construction) isn't truncated before its data_id/vclock stop; explicit override honored. - 4c (#6): enable the two genuine unmodeled sct terms in the 3 sim yamls — B1 `sim_model_eval_time` (self-calibrating on measured eval_s) and B2 `sim_straggler_spread_s=0.9`. `wall_disparity` DIAG rung drives the residual to ~0; final B2 value calibrated at the sign-off run. Tests: test_fwdllm_sim_speedup_waits.py (6, new); early-stop ceiling cases updated (+2). `tests/mode -k fwdllm` 148 green. fwdllm-only blast radius. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…k-timeout (K-D26) #6 (K-D25) — root-caused the clock-rate "gap" to a CHECKER-ANCHOR artifact, not a sim under-charge: the rungs compared sim-Δvclock (genuine algorithmic time) vs real-Δts (full wall = genuine + a ~constant ~7.7s/round localhost transport artifact the sim correctly omits). Aggregator now emits intrinsic_span_s (barrier + eval, mirroring the sim vclock composition; fedavg excluded); the 5 clock-rate rungs (K2/K3/K3b/K8/U2) + wall_disparity anchor real on its cumulative intrinsic clock, ts fallback -> async_cifar10 byte-identical. Validated on banked fwdllm: throughput 0.47->0.048, overhead 21->1.1, wall_disparity 90->1.6. Root B (K-D25) — phase rungs compared a mode-dependent phase: post_train now stamped AFTER the delay (real stops bundling the slept delay); B2 straggler moved from _delay_s into the sct so training_budget_s is a mode-invariant input. Validated: post_train 0/0, training_budget 1.14/1.14. #1c (K-D26) — root-caused fluxtune R1 (60.4%) to a physical-wall vs vclock desync in the shared async_oort re-pick guard, exposed by the slow sim (wall >> vclock): the 90s SEND_TIMEOUT_WAIT_S abandon-timeout keyed on time.time() evicts a still-outstanding trainer from all_selected -> re-dispatch. Corrects K-D19 (fixed the wrong RETURN path; K-D17b's NONE hypothesis was right). Fix (part a): _abandon_clock_now runs the timeout on the vclock in sim; real -> time.time() -> async_cifar10 byte-identical. Sustained NONE-delete path (part b) is the same slow-sim root -> tracked as #13. Tests: intrinsic-span anchor + emit, straggler-separation, post_train, vclock abandon-timeout; full tests/ 879 passed / 7 skipped. Doc trimmed 1817->408 lines with a standing maintenance rule; K-D25/K-D26 logged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…K-D25 validated) backend/mqtt.py: join() now waits for the MQTT connection before subscribing and announcing presence. Previously, when the caller thread beat the async on_connect callback, both the topic subscriptions and the JOIN notify were silently dropped (notify's _is_connected guard) with no retry, so the end never registered with peers and the run hung below minInitialTrainers. This flakily lost trainer cisco-open#379 in the fluxtune sim -> 9/10 ends -> zero selections. _wait_for_connect() is a no-op once connected, so async_cifar10 and all other examples are unchanged. Full pytest tests/ 879-green. simulate_fwdllm.md: fresh 2026-07-04 PM pairs validate K-D25/K-D26 clock-rate + phase fixes on both sync baselines (fwdllm 44->49, fwdllm_plus 40->45 pass); surviving fails characterized (marginal-KS, mis-applied gpu_budget, #7 selection divergence, stop-mismatch boundary). #14 logged and fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fficient The #14 join-race fix let the fluxtune sim run to data_id=20 (10 ends). The banked pair confirms K-D26's prediction: part (a) (vclock-keyed abandon-timeout) alone recovered ~0 of R1 residence — sim still 62.9% (215 tripwire fires) at sim_rate 0.29. The sustained NONE-delete path (part b) / #13 slow-sim is required, not optional. All fluxtune clock/commit/staleness fails are downstream of this R1+#13 desync. Doc §A/scoreboard/#1c updated in place. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deep telemetry investigation of the post-#14 fluxtune sim (_2327). R1 part (a) recovered ~0 (still 62.9%, 215 SIM_R1_DISPATCH). Root cause: async_oort decides selection eligibility solely from all_selected (pruned on physical events - recv-fifo 2s re-select, RECVD/NONE cleanup) and never consults the aggregator's virtual in-flight set (_sim_inflight_expected u _sim_buffer). The intended bridge _agg_pending_commit_ref is read for telemetry but never assigned/filtered; _sim_pending_commit is accumulated but never bridged (felix's asyncfl maintains it). In the slow sim's large returned-but-uncommitted window, any physical prune frees a still-in-flight trainer -> re-dispatch -> R1. Real/async_cifar10 mask it via wall~=vclock. Evidence: num_eligible=10 while in_flight=10; in_pending_commit =0/879; CHANNEL_CLEANUP freed=0 x218; 0 fires of 90s/removed-ends prunes. Doc #1c + K-D27 updated; fix proposed (wire the bridge, exclude in filtered_ends). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause (K-D27): async_oort gated selection eligibility solely on all_selected (pruned on physical events - recv-fifo 2s re-select, RECVD/NONE cleanup) and never consulted the aggregator's virtual in-flight set, so in the slow sim a returned-but-uncommitted trainer got re-dispatched (R1 62.9%, 215 SIM_R1_DISPATCH). Fix (felix-parallel with asyncfl's _sim_pending_commit, plus the selector-side exclusion fwdllm needs because async_oort + the slow sim prune all_selected mid-window): - fwdllm_aggregator: maintain _sim_pending_commit as the live in-flight set - add on dispatch (2627), discard on commit (811), in-place reconcile to the still-outstanding set in _sim_hold_busy_slots (clear+update, never rebind; replaces the |= accumulate that would starve committed trainers), clear on the legacy-drop boundary. Bind sel._agg_pending_commit_ref to the live set. - async_oort: exclude _agg_pending_commit_ref from filtered_ends (both the eligible-set list and the candidate loop). Sim-only: the set is empty in real and felix leaves the ref unset, so real / async_cifar10 are byte-identical. Tests: TestPendingCommitExcludedFromSelection (selector) + TestPendingCommitBridge (aggregator). Full pytest tests/ 886-green. Doc: #1c + K-D27 + §G updated to Fix LANDED. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…outstanding (K-D27b) The K-D27 bind alone left R1 at 67.6% on the _005222 re-run (238 SIM_R1_DISPATCH). Telemetry proved the selector filter worked (in_pending_commit populated; chosen ∩ pending = 0), so the re-dispatched trainer was in _sim_inflight_expected but NOT in pending -- only possible via _sim_committed. Root: the reconcile computed outstanding = (inflight ∪ buffer) − _sim_committed, but _sim_committed is a stale cross-cycle marker (cleared only at the agg-goal boundary). A trainer that committed then was legitimately re-picked + re-dispatched (re-added to _sim_inflight_expected) got dropped from outstanding by that subtraction -> re-pickable while its new dispatch was still in flight -> R1. Fix: outstanding = inflight ∪ buffer (drop − _sim_committed). A this-cycle commit is already popped from both sets, so the subtraction was redundant for it and harmful for the re-dispatch case. Regression test test_recommitted_trainer_stays_pending_despite_stale_committed. Full tests/ 887-green. Also banks the _005222 parity (38/15/19, R1 67.6%) that surfaced this. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…handoff) Verified against async_cifar10 (felix asyncfl::_sim_recv_min): fwdllm's async drain _sim_recv_min_grad is an early port missing ALL of felix's straggler refinements (grep-confirmed felix=N/fwdllm=0): stuck-end eviction on deadline (inflight_expected.pop(_stuck_end)), probe-ceiling + _sim_end_has_ready_msg ready-gating, drain_ready/_sim_sct_ordered_drain direct ingest, _sim_free_slot_ts freed-slot refill. So on the 30s RECV_TIMEOUT_WAIT_S deadline fwdllm just breaks and leaves the stuck trainer in _sim_inflight_expected forever -> earlier_stuck re-fires the full 30s every commit cycle -> pipeline starvation (data_id=1 in ~9min, sim_rate 0.06, 205 recv 2s-timeouts). R1 is FIXED (K-D27b, 238->0); fixing it unmasked this drain stall. felix's _sim_recv_min IS the intended slot-based discrete-event model (dispatch c slots -> batch-collect cohort replies -> reorder by sct at the aggregator -> skip vclock to next sct -> free one slot -> re-select). Doc: new "#13 DESIGN" block with operator-confirmed design choices (invisible-until-sct residence, modeled-fastest-k set, variance rollbacks untouched, sim-only) + a 4-step incremental port plan + expected result. #13 promoted to top blocker; R1/#1c marked FIXED in the roots/scoreboard. Design locked for a fresh-context impl. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n (K-D28/b/c) Ports felix's straggler-handling refinements into fwdllm's async grad drain (_sim_recv_min_grad), fixing the #13 "sim physically slow / 30s drain stall". All sim-only, fwdllm-scoped; async_cifar10 byte-identical. - Step 1 (K-D28): stuck-end eviction on the RECV_TIMEOUT_WAIT_S deadline so a lost/idle straggler can't re-fire the full 30s every cycle; + fwdllm recv-grace floor 2->5s (SIM_RECV_GRACE_FLOOR_S, tunable — the forward-grad GPU can take ~4s). - Step 2 (K-D28b): probe-ceiling + ready-gating — probe a non-recv_end in-flight end only if physically ready or exp <= buffered_min+slack (stop burning grace on far-future stragglers). - Step 3 (K-D28c): direct drain_ready ingest (flag sim_sct_ordered_drain) — sweep each in-flight rxq directly instead of the recv_fifo streamer, which blocked the full grace per not-ready end (the 12-34s blocks) and could strand a delivered grad (past-dated commit). Validated on fluxtune smoke (run_..._024637 -> 032729): sim_rate 0.06->0.29, aggregate blocks >=5s 54->8, 30s stall gone, R1 stays 0. Step 4 (K-D28d, freed-slot staggered re-dispatch) is implemented + tested but INVESTIGATED-NEUTRAL and left DISABLED (see §H): the residual 11-12s holds are the drain correctly waiting (strict sct order) for the earliest-sct in-flight straggler — inherent to real-GPU + strict-order + trickle dispatch, not a dispatch-timing artifact. The residual sim_rate<1 is #12c-bound (no --delay-factor -> ~no vclock headroom); measure with --delay-factor before any further drain work. Tests: TestStuckEndEviction, TestProbeCeilingReadyGating, TestSctOrderedDrain, TestFreedSlotRefill; full tests/mode -k fwdllm green (168). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… model (K-D29) Checkpoint of the logical real<->sim parity work. Root diagnosis: all 3 baselines diverge from ONE cause — sim commit ORDER != real arrival ORDER (fwdllm var is a split-half stat over the commit-ordered grad list -> wrong order -> var flips -> per-trainer RNG desyncs -> grads diverge). Grads are deterministic given matched order, so exact parity is achievable. Checks (P1-1/2/3, P1-6 partial, P1-8): - New enforced `cohort_sequence` rung (EXACT, ungated): per-cycle cohort set + receive-order + cadence + var value must be identical. Keyed on the fwdllm cadence stream (not `round`); runs on all 3 fwdllm baselines. - V2 var_trajectory mean-guard (catches the ~1% systematic offset KS misses). - `--max-bin` window threaded through the engine + run_parity.py + cli.py. - Both rungs now correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus 36/17/21, fluxtune 35/19/19. - Tests: TestCohortSequence x9, TestVarTrajectoryMeanGuard x2. Timing model (K-D29 / P2-1/3/5/6, reverses K-D2's additive model): - Remainder-wait: real sleeps max(0, D-gpu); sct = max(gpu, D). Per-trainer registry delays supply the completion spread -> order deterministic + identical real<->sim. Overrun (gpu>D) flagged (training_overran, [TIMING_OVERRUN]). - crc32 straggler offset disabled in sim yamls; perturbation_count made a knob. - GPU profiled: fwdllm/plus ~1.0s, fluxtune 7.57s (JVP 20 passes) — the fluxtune overrun watch. Trainer duration tests rewritten to the new contract. Suite green: 606 mode/telemetry/selector + 115 async_cifar10 parity. Deferred to next session (recorded in PARITY_LOGICAL_TASKS.md SESSION CHECKPOINT): P1-4 (async_cifar10 cohort adapter), P1-5 (enforce determinism rungs) — both skipped en route to Phase 2; the databin1 validation run (--delay-factor 2 --max-data-id 1); P0-2 grad-determinism confirmation; fluxtune perturbation tuning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…GPU telemetry (K-D30) Pre-unavailability (Phase-1) checker rigor + GPU-optimization telemetry, ahead of the databin1 validation run. No workload behavior change; sim/checker-only, async_cifar10 byte-identical. Checker (P1-5 / K-D30): - Data-driven full-cohort determinism gate (_selection_is_deterministic / _full_cohort_selection): un-gates selection/aggregation_sequence/utility to GENUINE enforcement when num_chosen==num_candidates in both modes (fwdllm syn_0, K=all); fluxtune (agg_goal<K) + fwdllm_plus (#7) stay gated; count-less legacy telemetry falls back to the old selector-name rule. participation EXCLUDED (round-keyed -> mechanical KS on fwdllm's data_id axis). - New timing_overrun DIAG rung: surfaces the P2-6 training_overran fraction + first-overrun bin (the K-D29 order-determinism tell). Wired + re-exported. - P1-4 (async_cifar10 cohort adapter) assessed redundant, not built. GPU telemetry (P2-4): - EVENT_STEP_TIMING + build_step_timing; timer_decorator emits a structured per-function record when a trainer fwd_llm_stage is present (no-op when telemetry off; guarded so nested helpers never mis-attribute). - plot_step_timing.py: fine step_timing + coarse trainer_round phases -> table + PNGs (exact-baseline discovery via _RUN_RE). Banked-log finding: real wall is dominated by mqtt_fetch_s (fluxtune 18.3s/round) >> gpu_compute. - train_with_data_id sets fwd_llm_stage AFTER the abort check so an aborted round stays a true no-op. Trainer: fixed two stale K-D29 comments (still described the reversed additive delay model). Tests: 15 new (determinism gate x6, timing_overrun x4, step_timing builder + decorator emit x5). 735 mode/telemetry/selector/parity green. Banked scoreboard stable (fwdllm 41/13, fwdllm_plus 36/17, fluxtune 35/19; +1 timing_overrun SKIP). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
K-D31 (sync delay-tie): trainer stamps pure modeled delay D (MODELED_DELAY_S, both modes); aggregator canonicalizes each cohort's commit order by (D, str(end)) before the telemetry snapshot + aggregate(), so equal-D ties break by trainer_id identically in real and sim. Closes the sole databin1 residual — cohort_sequence order_match 0.5/0.0 -> 1.0 for the sync baselines, var/grads bit-identical. Validated: [COMMIT_CANON] fires in fluxtune (perms [2,0,1] etc); async_cifar10 byte-identical. K-D32 (fluxtune JVP perf-opt, config-gated `jvp_perf_opt`, all bit-identical): (a) trainable-only finite difference (calculate_jvp skips p-h*0 on the 98.5% frozen backbone, 1.26x + -251MB); (b) skip 3 diagnostic-only forward passes; (c) reuse the selected perturbation's cached JVP. fluxtune -37% GPU/batch to clear the delay_factor=1 timing overrun without changing fidelity. Default false (byte-identical); true in both fluxtune yamls (must match real<->sim). Profiled + validated by scripts/profile_jvp_opt.py (reuses real create_model + calc_jvp). Also: fluxtune_contributions.md (systems/ML contributions brief); simulate_fwdllm.md §L compute profile + K-D31/K-D32; 22 new pytests (commit_canon x9, jvp_perf_opt x4, + prior). 428 mode + 115 async_cifar10 parity green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Living experiment system for the fluxtune vs fwdllm/fwdllm_plus comparison. Runs are defined ONCE in experiments.yaml; all 5 experiments are pure-function reducers over the same run-set telemetry (no re-runs across baselines). Telemetry (shared flame lib + fwdllm): - WS2 convergence stop: runner-side watcher (converge_watch.py) polls agg_eval, terminates when the last W data bins all >= target acc; new CONVERGED / DID_NOT_CONVERGE health verdicts + per-run converge.json. Flags --target-acc / --converge-window on run_sequential.sh. - WS3-a network telemetry: new `comm` event (build_comm), emitted both directions from the aggregator (sync + async dispatch sites) and trainer upload. - WS3-b forward-pass/perturbation counters in fwdgrad_utils, surfaced per iteration + cumulative in trainer_round. Analysis: - compare_baselines.py: cross-baseline table/CSV + overlay plots for all 5 experiments, with a shared-axis mix-guard; degrades gracefully on old runs. Misconfig-proof pre-run gate: - condition_fp (hash of shared axes incl. delay_factor) for two-node match checks - tier (2) shows baseline internals (mode/selector/optimizer) - agg_goal-match-across-baselines check - --run-set NAME loads the shared condition from experiments.yaml (single source of truth; CLI flags override) Signed-off `main` condition: N=100, K=10, C=10(sync)/30(async), agg_goal=10 matched, partition alpha=0.1, syn_0, delays on at delay_factor=2, target 0.82. Validated end-to-end at N=10. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A convergence run (--target-acc set) is accuracy-governed, so the wall ceiling now defaults to 48h (172800s) instead of 1h -- a short default was killing runs that were still legitimately learning. Stall guard (early-out): terminate BEFORE the 48h ceiling if the run is clearly not learning -- best test accuracy hasn't gained >= stall_min_delta (default 0.01 = 1%) within stall_window_s (default 7200s = 2h). Any >=1% gain resets the clock; convergence is checked first so a just-converged run is never STALLED. New verdict STALLED (distinct from DID_NOT_CONVERGE), per-run stall.json. - converge_watch.py: --stall-window-s / --stall-min-delta / --stall-json; milestone-accuracy tracking; writes stall.json + kills the pgroup on stall. - expt_runner.sh: pass stall knobs to the watcher; detect stall.json -> EXPT_LAST_STALLED; STALLED health verdict; sweep orphans on either outcome. - run_sequential.sh: --stall-window-s/--stall-min-delta flags; default max_runtime_s to 48h when target-acc set; registry (--run-set) wiring; stall_guard gate row; stall params folded into condition_fp; STALLED summary. - experiments.yaml defaults: max_runtime_s 172800, stall_window_s 7200, stall_min_delta 0.01. Stall logic validated: flat accuracy -> STALLED (kills run); steady +1%/window -> clock resets, run continues. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ctrl+C/SIGTERM now cleanly tears down a run: the trap in expt_launch kills the run's process group (trainers+aggregator), the convergence watcher, and the progress ticker (moved to its own process group so its sleep child dies too), escalating SIGTERM->SIGKILL after a grace and sweeping stragglers so GPU/RAM free. Previously a terminal SIGINT never reached the run's own process group, orphaning workers and leaving the ticker looping forever. Added expt_assert_clean_slate: a pre-launch guard (called before every baseline) that refuses to start on top of a prior/crashed run's stray workers. Default aborts and prints the kill command; --clean / EXPT_AUTOCLEAN=1 auto-kills and re-verifies. Optional GPU-memory guard (EXPT_GPU_FREE_MB / EXPT_GPU_STRICT). main run-set partition changed niid_label_clients=100_alpha=0.1 -> alpha=1 (the base-config default; group present in agnews_partition.h5). condition_fp is now 04d64814. EXPERIMENTS.md updated: three-node launch (§7.3), teardown + clean-slate guard (§7.6), alpha=1 throughout, changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
plot_run.py: single-run twin of compare_baselines.py. Streams ONE run's telemetry in a single pass (the aggregator JSONL runs >1 GB — never loaded whole) and renders the full EXPERIMENTS.md §4 5-experiment plot set (E1 learning curves, E2 utilization, E3 productivity, E4 network, E5 sessions) + summary.json. Validated on the first N=100 fluxtune convergence run (run_20260706_185045…, STALLED at 84.08% > 0.82 target, 139 bins). expt_runner.sh: fix a harness hang exposed by that run's watcher-driven stop. The progress ticker was assumed to be its own process group (set -m), but job control did not place the backgrounded subshell in a fresh group, so kill -"$ticker_pid" (a process-group signal) missed it and the following wait "$ticker_pid" blocked forever — expt_launch hung after a watcher/Ctrl+C stop while the orphaned ticker kept printing "… Ns elapsed …" every 30s. Now the ticker self-terminates the instant the run process dies (kill -0 guard on run_pid) and is torn down by PID (+ its sleep child) via _expt_stop_ticker, never by process group; the interrupt path also kills run_pid by PID + name as insurance. Verified: ticker self-exits ~1s after run death with no orphan child, and _expt_stop_ticker returns in 0s instead of blocking 30s mid-sleep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ours alias)
The stall-guard idle window was already CLI-settable via --stall-window-s
(and CLI overrides the registry), but only in seconds. Add --stall-window-h H
as an ergonomic hours alias (6 => 21600s), validated numeric, converting to
STALL_WINDOW_S so it flows through the same registry-override + condition_fp
path. Motivated by a fwdllm N=100 run the default 2h guard killed at the very
start (accuracy flat but test-loss still falling slowly) — re-launch with a
wider idle window instead of editing experiments.yaml:
run_sequential.sh --run-set main --only fwdllm --mode real --clean --yes \
--stall-window-h 6
Dry-run confirms the gate's stall_guard row flips 2h -> 6h and the CLI value
overrides the registry. Documented in EXPERIMENTS.md (2 + changelog).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The stall guard reset its idle clock only on accuracy gains, so a run whose
accuracy plateaus while test-loss keeps falling (still learning) was killed
at the window. Make progress configurable via --stall-on (default: either):
acc best accuracy gained >= stall_min_delta (ABSOLUTE 1%; acc in [0,1])
loss best (running-min) test-loss dropped >= loss_min_rel_delta
(RELATIVE 1% vs running-best; loss is unbounded so the bar is
fractional, not absolute, and diminishing-returns aware)
either reset the clock if EITHER fired
Both signals track the running-best (max acc / min loss), so a single noisy
eval can neither reset the clock nor fake progress. Guards milestone_loss<=0
defensively (real CE loss is >0). stall.json now records stall_on, best_loss,
and milestone_loss alongside the accuracy fields.
Wired through: converge_watch.py (_scan returns loss_by_bin; --stall-on,
--loss-min-rel-delta; mode-aware STALLED message + payload); expt_runner.sh
(passes both to the watcher); run_sequential.sh (flags, registry emit +
CLI-override precedence, gate stall_guard row, condition_fp, env bridge);
experiments.yaml defaults (stall_on: either, loss_min_rel_delta: 0.01).
Validated end-to-end against the real converge_watch.py: flat acc + flat loss
-> STALLED [either]; flat acc + steadily-falling loss -> clock resets on the
loss signal and the run stays alive until loss also plateaus (stalled at
wall 17.2s, ~stall_window after appends stopped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add expt_scripts/plotlib/ as the single source for paper figures: SOCC-2026 style (PDF-only, pdf.fonttype=42, timestamped output + latest symlink), a baseline display registry (switchable system codename, colorblind + grayscale- safe families/tints/markers, legend emphasis), a streaming telemetry reducer, and single-panel figure builders. New make_paper_figs.py renders from a figs.yaml baseline->run-dir manifest (--run overrides); plot_run.py and compare_baselines.py migrate onto the shared reducer (removing duplicated metric code). Fix a latent reducer bug: accuracy/loss were keyed by data_id, which cycles per round, silently overwriting earlier rounds and corrupting Δloss. Now a time-ordered eval series with round-boundary markers. Per-run cutoff at the last significant test-loss improvement (loss, not the coarse accuracy readout, is the grounded learning signal); deterministic so no grace buffer. Visual-only EMA smoothing (--smooth) leaves scalar metrics on raw. Raise the main convergence target 0.82 -> 0.84 (fluxtune reaches it, the baselines don't) in experiments.yaml and the plot defaults. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- E1: add faint RAW envelope under the smoothed line so the true accuracy reached (e.g. fluxtune's 84% peak, at the target line) stays visible despite EMA smoothing pulling the smoothed curve down. - Add e1_loss_vs_time — test-loss vs wall time, the grounded companion to the accuracy curve (adjacent paper figure). - Split e3_productivity into e3_dloss_per_gpu_hour + e3_dloss_per_mfwd: the two denominators differ ~100x in scale, so a shared y-axis rendered one set of bars invisible (dataviz: two scales -> two charts). Each now has its own axis + value labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… speed) N=100 smoke data contradicts the E3/E4 takeaways: FwdLLM++ is more compute- efficient (Δloss/M-fwd 5.62 vs 1.27) and more communication-efficient (79 vs 146 GB) than Fluxtune. Add ⚠ observed-result notes under §4 E3 and E4 with the numbers, the mechanism (async staleness + speculative concurrency waste compute; continuous model re-pulls + more iterations inflate bytes; FwdLLM++'s sync rounds + oracular availability amortize both), and optimization directions. Fluxtune's win is E1 speed/accuracy, not per-unit efficiency. Changelog entry (f). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The E3/E4 (and E2) efficiency results are single-contribution: the current run-set exercises only Fluxtune's guided (JVP-magnitude) perturbation selection. Its efficiency mechanisms are off — dynamic K/C (static agg_goal=10/C=30) and intelligent aggregation (plain fedbuff down-weighting) — and those are exactly what target the observed inefficiencies (dynamic C throttles stale concurrency -> E3 compute + E4 bytes; intelligent aggregation weights by usefulness -> E3). So the E2/E3/E4 efficiency claims must be re-made with a full-system Fluxtune run (a separate condition, since dynamic K/C breaks the agg_goal-matched head-to-head). E1 (speed + accuracy) already holds on contribution 1 alone. Note under §4 + changelog. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Within a data-bin the model_version is constant and the full WEIGHTS+GRAD_POOL payload is byte-identical across iterations, yet the aggregator re-shipped it to the same trainers on every distribute call: measured 90% redundant / 22 GB (fwdllm), 71% / 68 GB (fluxtune); fwdllm_plus incidentally clean (3.5%) via per-iteration reselect. Root cause (verified from telemetry, staleness==0): the sync loop is distribute>>aggregate, so distribute runs ~10-12x per data-bin and each call takes the unconditional `var_good_enough -> WEIGHTS` branch. The existing is_stale guard sat after that short-circuit and never ran. Fix: shared `_should_send_full_weights` for both sync+async paths, backed by a send-time `_weights_sent_this_cycle` set (cleared on model_version advance) that is checked FIRST -> gates both branches. Invariant: <=1 full payload per trainer per model_version. VAR=bad prepared even at var_good_enough (commit-branch downgrade; trainer keeps training on cached current weights -> no deadlock). Flag `suppress_redundant_weights` (default off = byte-on-wire identical), enabled on all baselines (learning-neutral -> keeps E4 fair). Also fixed a sync telemetry mislabel (payload_kind keyed off var_good_enough, not the payload) and removed the iteration cap from the sync baselines (it is a fluxtune Opt-2 lever). Tests: tests/mode/test_fwdllm_suppress_redundant_weights.py (9 cases) + expt_scripts/audit_weight_redundancy.py (reusable per-databin regression check). Charter §5c/§5d + status updated. Validation run still pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…adlock) Fold run_20260708_001641 validation results into §5d/STATUS/§2f-1 in place; changelog (h). Opt-1 closed; next is Opt-2 (variance-plateau force-commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a variance-gate stopping policy so a data-bin stops grinding once its
JVP-variance curve flattens, instead of only committing on the fixed absolute
var<=0.3 gate. At the paper's alpha=1 operating point that gate is a fixed bar
below the achievable variance floor, so a bin catches it only on a noise dip and
grinds ~15->34 iters/bin (early->late), committing nothing 95.7% of the time.
Policy var_stopping_policy in {off, fixed_cap, plateau} (Extra.allow config):
* plateau (primary/early): commit when the relative var drop over the last N
iters < eps while var is still above threshold -- scale-invariant, so it
commits the DENOISED estimate at whatever iter the curve levels off.
* cap (late backstop, max_iterations_per_data_id): near convergence the plateau
onset drifts out (~33) but the model is stable, so the cap force-commits first.
Whichever fires first wins; commit_reason (natural/cap/plateau) + stopping_policy
are emitted on agg_round telemetry so the split is measurable.
Pure decision TopAggregator._should_force_commit_on_plateau (reads only instance
attrs, unit-tested in isolation), called from FedSGDAggregator.aggregate() right
after the cycle var is appended to var_prev_iter_list; it ORs into the existing
_force_commit_this_cycle cap path. Byte-identical when off.
Enabled as the fluxtune default in _metadata/baselines.yaml (plateau, N=3,
eps=0.15, cap=20 -- cap above the early plateau fire, below the late grind);
fluxtune-only (fwdllm/fwdllm_plus unset), disable/tune documented inline.
Also add expt_scripts/characterize_variance_curve.py: streams agg_round
telemetry, reconstructs per-databin var-decay curves (commit-count databin id,
robust to data_id cycling), and runs counterfactual fixed_cap/plateau sweeps --
the measure-first tool that picked these values from the alpha=1 run.
Tests: test_fwdllm_var_stopping_policy.py (8 cases) green; Opt-1's 9 still green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend characterize_variance_curve.py for the plateau-vs-baseline comparison: --max-databins N truncates analysis to the first N committed bins (like-for-like across runs of different length), and the per-bin commit_reason (natural/cap/ plateau; natural(legacy) on pre-Opt-2 runs) is captured and summarized so the stopping-trigger split is visible directly from telemetry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add agg_rate_conf.type=grad_aware to aggregate_grads_from_trainers, the
per-update weighting hook that the fedbuff old/new rate types already dispatch
on. The default `new` rate rescales an update's MAGNITUDE by staleness x utility
but cannot refuse a wrong DIRECTION, so anti-aligned JVP estimates are still
averaged (H1 -> the M-12 mid-run instability). grad_aware weights by:
* align gate (S2, primary): down-weight an update whose cosine vs the running
aggregate self.grad < align_floor, linearly -> 0 at cos=-1.
* inverse-variance (S1, opt-in): x min(1, var_ref/var_i) from the trainer's own
perturbation grads.
Both factors are in [0,1] so the result never exceeds the base rate -> the
effective server LR never inflates (no re-tune needed to stay stable).
Pure primitives TopAggregator._grad_aware_rate + _cosine_flat (13-case unit
test). Telemetry agg_rate_type + grad_aware_gated_total on agg_round. Baseline
stays type=new -> byte-identical; grad_aware enabled per-run via baselines.yaml
(documented). This is the "new aggregation strategy" for the operator's three
parallel target-acc runs (var-stop only / new-agg only / all features).
Charter: crisp STATUS + feature/run matrix at top; Opt-3 marked implemented
(§5c/§2f-3); changelog (l)+(m); consolidated the stale STATUS blocks.
Tests: 30 green across the three fwdllm opt suites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
type=new is the borrowed FeLiX scalar staleness x utility rate (charter N2), not a fluxtune contribution. Fluxtune now defaults to its own C3 aggregation (agg_rate_conf.type=grad_aware, align_gate on, base=new, inverse_var off); type=new is kept only for the FeLiX-baseline ablation arm. Default fluxtune is now the full stack (C1 + Opt-1 comm + Opt-2 plateau + Opt-3 grad_aware). Charter: crisp STATUS matrix updated; added the four planned N=100 target_acc=84% runs as a 2x2 over the two learning-affecting opts (Opt-2 var-stop, Opt-3 grad-aware) with C1+Opt-1 constant -- R1 FeLiX baseline, R2 +var-stop, R3 +grad-aware, R4 all(=default); noted remaining opts (Opt-4 dynamic C, Opt-5 staleness clock, Opt-1 cross-databin delta-cache). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trim the living doc to current state only: status + feature/run matrix, the measured bottleneck (§5a key rows) with the α=1 tuning that set the plateau values, the resolved decisions table, remaining work, and principles. Drop the persistent changelog (git has the history), the stale "Next steps START HERE", and the verbose per-opt design narratives (§5c/5d/5e) now superseded by the code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…un ablation run_sequential.sh gains two knobs (mirroring --var-threshold / --max-iter-per- data-id) to toggle Opt-2 and Opt-3 per run without editing baselines.yaml or creating per-run config files: --var-stopping-policy off|fixed_cap|plateau (Opt-2; writes hyperparameters) --agg-rate-type grad_aware|new|old (Opt-3; writes optimizer.agg_rate_conf) Both patch the per-run config_overrides, which win over the baselines.yaml catalog at launch. A full agg_rate_conf dict is written per type so the felix `new` branch keeps scale/a_exp/b_exp regardless of merge depth. Added to the pre-flight review table. Validated by --dry-run: R1 generated cfg shows var_stopping_policy=off + agg_rate_conf.type=new overriding the grad_aware/plateau defaults. Charter: run table now lists the exact CLI flags per arm + the common base command. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Register the 2026-07-08 N=100 α=1 opt-ablation 2×2 (R1 FluxTune-base / R2 +var-stop / R3 +grad-aware / R4 full) as its own paper-figure set and refresh the baseline comparison onto R4 (full-stack default): - figs_ablation.yaml: new manifest mapping the 4 runs (identified by aggregator_config var_stopping_policy × agg_rate type). - figs.yaml: fluxtune -> R4 run_20260708_025716 (was FeLiX-era 185045). - baselines.py: 4 ablation Style rows on a sequential-blue ramp (_ABLATION_BLUES, light->dark = more opts) + a color= override on _mk. R1 labeled FluxTune-base (C1 guided-JVP + Opt-1), NOT FeLiX. - reducers.py + make_paper_figs.py: new --cutoff-mode peak_acc (default) clips each run at its highest-accuracy eval; loss-scan retained as the learned-at-all gate so a never-learned run shows full. Excludes the round-2 divergence tail (Issue I-1). - figures.py: E1 acc plot marks each run's peak with a ★ + legend "peak X%" against the target line (shows gap-to-target per baseline). Docs: EXPERIMENTS.md §4 E1 observed results + §10 ledger rows; EXPTS_CHARTER.md landed-runs/peak-accuracy tables, Issue I-1 (round-2 catastrophic divergence: symptom, suspected epoch-boundary bug, next steps), and a Latest-figures pointer for paper embedding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open a training-stability track in fluxtune_contributions.md §8 (findings
ledger + resolution plan + strict in-place/flag-gate discipline). Root-cause
the α=1 N=100 non-convergence:
- Add H0 diagnostic (diagnose_partition_binning.py): per-client/per-data_id/
per-K-cohort class distribution across α∈{0.1,1,100}, overlaid on observed
collapses. Result: REFUTES data-class-bias (cohort near-balanced at α=1,
Spearman +0.002 vs collapses) and the suspected epoch-boundary bug (model
version monotonic). Driver = undamped, high-variance forward-grad optimizer;
the frozen data schedule only freezes the noise (position-lock).
- EXPERIMENTS.md: add motivation/ablation M1 (bin-size × heterogeneity, the
forward-mode bias/variance tradeoff) and M2 (aggregator optimizer vs random
walk — a fluxtune contribution).
- Reconcile the now-stale I-1 framing across all three docs; tighten for signal,
cut cross-doc redundancy. Next task = S1 (server-side optimizer, flag-gated).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Condense engineering-journal comments (multi-paragraph rationale, dated sign-offs, internal task IDs, root-cause narration) across the fwdllm sim work into tight what+why comments, and trim narration in the design docs. Comments/docstrings/prose only -- code is byte-identical (AST-verified), tests green. Net -415 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an async discrete-event simulation path and parity tooling for FwdLLM/FluxTune, along with new telemetry signals and several flag-gated aggregation/throughput optimizations (default behavior remains legacy/byte-identical).
Changes:
- Introduces simulation-time mechanics (virtual clock usage, sim ordering, client unavailability handling) plus parity harness/scripts.
- Adds new telemetry events (
step_timing,comm) and related trainer/aggregator emissions for timing + network accounting. - Adds three flag-gated aggregation optimizations and extensive new unit tests covering parity-critical invariants.
Reviewed changes
Copilot reviewed 92 out of 92 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| lib/python/tests/telemetry/test_event_schema.py | Tests for new telemetry events |
| lib/python/tests/selector/test_oort_selector.py | Selector eligibility tests |
| lib/python/tests/mode/test_fwdllm_var_stopping_policy.py | Var-plateau policy tests |
| lib/python/tests/mode/test_fwdllm_trainer_task_recv.py | task_recv telemetry tests |
| lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py | Sim duration model tests |
| lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py | Trainer phase timing tests |
| lib/python/tests/mode/test_fwdllm_sync_scarcity_wait.py | Scarcity wait tests |
| lib/python/tests/mode/test_fwdllm_suppress_redundant_weights.py | Weight resend suppression tests |
| lib/python/tests/mode/test_fwdllm_sim_sync_barrier.py | Sync barrier sim tests |
| lib/python/tests/mode/test_fwdllm_sim_speedup_waits.py | Sim wait gating tests |
| lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py | Async sim deadlock regressions |
| lib/python/tests/mode/test_fwdllm_sct_model.py | SCT model fold tests |
| lib/python/tests/mode/test_fwdllm_rounds_termination.py | Termination regression tweak |
| lib/python/tests/mode/test_fwdllm_recv_timeout.py | Recv timeout regression tweak |
| lib/python/tests/mode/test_fwdllm_oracular_avail.py | Oracular avail arg rename |
| lib/python/tests/mode/test_fwdllm_jvp_perf_opt.py | JVP perf-opt identity tests |
| lib/python/tests/mode/test_fwdllm_grad_aware_agg.py | Grad-aware aggregation tests |
| lib/python/tests/mode/test_fwdllm_early_stop_conditions.py | Sim early-stop budget tests |
| lib/python/tests/mode/test_fwdllm_commit_canon.py | Canonical commit ordering tests |
| lib/python/tests/mode/test_async_sim_ordering.py | Sim vclock timeout test |
| lib/python/tests/launch/test_runner_paths.py | Delay config fan-out tests |
| lib/python/tests/launch/test_config_generator.py | Execution-config banking tests |
| lib/python/flame/telemetry/events.py | New telemetry events/builders |
| lib/python/flame/selector/async_oort.py | Sim-clock timeout + pending filter |
| lib/python/flame/monitor/runtime.py | timer_decorator step telemetry |
| lib/python/flame/mode/message.py | New modeled delay message type |
| lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py | Phase timing + comm/task telemetry |
| lib/python/flame/launch/spawner.py | Post-spawn load-balance check |
| lib/python/flame/launch/runner.py | Aggregator GPU pin + delay fan-out |
| lib/python/flame/launch/execution_config_generator.py | Bank delay config in exec config |
| lib/python/flame/launch/aggregator_spawner.py | Startup crash detection + GPU pin |
| lib/python/flame/config.py | SCT fold hyperparameters |
| lib/python/flame/backend/mqtt.py | Join/connect race mitigation |
| lib/python/examples/fwdllm/trainer/main.py | time_mode threading + pin report |
| lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py | Perturbation count + JVP perf-opt |
| lib/python/examples/fwdllm/trainer/forward_training/fwdgrad_utils.py | Trainable-only JVP + pass counters |
| lib/python/examples/fwdllm/trainer/fl_main.py | Propagate new trainer knobs |
| lib/python/examples/fwdllm/telemetry_manifest.yaml | Manifest updates for timing fields |
| lib/python/examples/fwdllm/expt_scripts/run_parity.py | Parity runner script |
| lib/python/examples/fwdllm/expt_scripts/plotlib/style.py | Figure style utilities |
| lib/python/examples/fwdllm/expt_scripts/plotlib/baselines.py | Baseline plot registry |
| lib/python/examples/fwdllm/expt_scripts/plotlib/init.py | plotlib package init |
| lib/python/examples/fwdllm/expt_scripts/plot_step_timing.py | Step timing analysis script |
| lib/python/examples/fwdllm/expt_scripts/make_paper_figs.py | Paper figure pipeline |
| lib/python/examples/fwdllm/expt_scripts/logical_parity.py | Time-stripped parity diff |
| lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml | Smoke config tweaks |
| lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml | New sim smoke config |
| lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke.yaml | Smoke config tweaks |
| lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml | New sim smoke config |
| lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml | Smoke config tweaks |
| lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml | New sim smoke config |
| lib/python/examples/fwdllm/expt_scripts/figs.yaml | Paper figure manifest |
| lib/python/examples/fwdllm/expt_scripts/figs_ablation.yaml | Ablation figure manifest |
| lib/python/examples/fwdllm/expt_scripts/audit_weight_redundancy.py | Weight redundancy audit tool |
| lib/python/examples/fwdllm/expt_scripts/.gitignore | Ignore generated artifacts |
| lib/python/examples/fwdllm/experiments.yaml | Experiment registry |
| lib/python/examples/fwdllm/configs/trainer_base.yaml | Add perf-opt knob default |
| lib/python/examples/fwdllm/aggregator/FedSgdAggregator.py | Var-plateau commit logic |
| lib/python/examples/async_cifar10/scripts/smoke_suite.sh | Shared runner harness |
| lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh | Shared conda activation |
| lib/python/examples/async_cifar10/scripts/parity/validate_real.py | Add residence validation |
| lib/python/examples/async_cifar10/scripts/parity/cli.py | Add max-bin option |
| lib/python/examples/async_cifar10/scripts/parity_checks.py | Export additional checks |
| lib/python/examples/async_cifar10/scripts/compare_overnight.sh | Deleted (superseded) |
| lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md | Doc update re: deletion |
| lib/python/examples/async_cifar10/PARITY.md | Parity discipline update |
| lib/python/examples/_metadata/baselines.yaml | Baseline config additions/notes |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+140
to
+145
| try: | ||
| with open(p, "r", errors="replace") as f: | ||
| data = f.read() | ||
| return data[-max_bytes:] | ||
| except OSError: | ||
| return "" |
Comment on lines
+447
to
+463
| # visible physical GPUs on the node (independent of num_gpus we chose) | ||
| try: | ||
| import torch | ||
| visible_gpus = torch.cuda.device_count() | ||
| except Exception: | ||
| visible_gpus = 0 | ||
|
|
||
| issues = [] | ||
| # 1. Even GPU spread: round-robin guarantees max-min <= 1; flag otherwise. | ||
| if gpu_counts: | ||
| spread = max(gpu_counts.values()) - min(gpu_counts.values()) | ||
| if spread > 1: | ||
| issues.append(f"GPU imbalance: per-GPU trainer counts {dict(sorted(gpu_counts.items()))} (spread={spread}>1)") | ||
| # 2. Under-provisioning: physical GPUs left completely idle. | ||
| if visible_gpus and self.num_gpus < visible_gpus: | ||
| issues.append(f"under-provisioned: num_gpus={self.num_gpus} < visible={visible_gpus} " | ||
| f"→ {visible_gpus - self.num_gpus} GPU(s) idle; raise execution.num_gpus") |
| ``weights_to_ram_s``, ``mqtt_fetch_s``, ``post_train_s``). Always present; | ||
| a fallback when a run predates the step_timing telemetry. | ||
|
|
||
| Prints a breakdown table and (unless ``--no-plot``) writes PNGs next to the run. |
dhruvsgarg
added a commit
that referenced
this pull request
Jul 9, 2026
…ing) (#73) The flame.launch migration is merged (#68/#72) and proven in real n=100 runs, so the second, divergent JSON/MPI config path is redundant and risks a behaviour regression if launched by mistake. Remove it per DELETION_CANDIDATES.md. Deleted: - expts/run_tc_expts/ in full — 155 json_scripts (5 aggregator variants + trainer_0..149) + fedavg_main_tc.py, launch_single_run.py, run_three_*.sh, run_text_classification.sh, gpu_mapping.yaml, mpi_host_file. - aggregator/fl_main.py, trainer/fl_main.py — legacy argparse+MPI entrypoints. Coupled cleanups (so nothing dangles): - configs/trainer_base.yaml — drop the dead gpu_mapping_file/gpu_mapping_key keys (device pinning is the spawner's job; FedSgdTrainer hard-sets its device) and refresh the provenance comment. - expt_scripts/diagnose_partition_binning.py — repoint DEF_CACHE off the removed dir (it also pointed at a stale home path). - aggregator/main_fedfwd_agg.py — drop the fl_main.py reference in a comment. Kept (live): expts/initializer.py, configs/aggregator_base.json. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a discrete-event simulation path for FwdLLM/fluxtune async FL, validated for
real↔sim parity, plus telemetry, an experiment/paper-figure harness, and three
flag-gated aggregation optimizations.
real↔sim parity checkers.
Opt-1 suppress redundant intra-databin weight re-sends; Opt-2 variance-plateau
early commit; Opt-3 gradient-aware aggregation.
paper-figure pipeline.
Core library changes under flame/ (~12 files); remainder is examples, tests, docs.