From 0863255545b4ec2b3539f0581fe6df7c782465c3 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Thu, 2 Jul 2026 17:23:13 -0400 Subject: [PATCH 01/56] updated simulate_fwdllm.md plan --- lib/python/examples/fwdllm/simulate_fwdllm.md | 223 +++++++++++------- 1 file changed, 143 insertions(+), 80 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index adb27b247..d984139dd 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -1,10 +1,23 @@ # High-Fidelity Simulator for FwdLLM -- Design & Staged Build Plan -**Design-only.** This document is the plan for building a high-fidelity **simulated-clock** runner -for the `fwdllm` example (FedFwd / forward-gradient FL) that reaches **real<->sim parity** across the -**fluxtune / fwdllm / fwdllm++** baselines, at **100% availability (syn_0)** and under -**unavailability (syn_20, syn_50, mobiperf)**. The implementation is a **future branch/PR**; nothing -here is built yet. +**Active build (branch `dg/fwdllm_sim_unavail`).** This document is the plan for building a +high-fidelity **simulated-clock** runner for the `fwdllm` example (FedFwd / forward-gradient FL) that +reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baselines, at +**100% availability (syn_0)** first, then under **unavailability (syn_20, syn_50, mobiperf)**, then +**beyond syn_0 traces**. + +**Current state (landed on this branch, PRs #63-#69) -- the doc's original "nothing built yet" is stale.** +Stage-0 scaffolding is largely done and is now a *validation gate*, not a build: trainer telemetry +(`build_trainer_round`: `real_gpu_time_s`, `sim_round_duration_s`, `avail_state`, `stat_utility`, +`FedSgdTrainer.py:533`), aggregator telemetry (14 tests), the shared parity-check engine + fwdllm checks +(`test_parity_checks.py`, 40 tests), the oracular availability *read* (`read_trainer_unavailability`, +`fwdllm_aggregator.py:524`), per-iteration reselection, and the opt-in e2e parity harness +(`test_real_sim_e2e_parity.py`). **Still to build (the real work):** (1) the trainer sim path +(`_emulate_training_delay:481` still `time.sleep`; no `time_mode`/`simulated`, no `_sim_completion_ts`); +(2) the aggregator grad loop on the vclock (`_aggregate_grads_async:693` still raw `recv_fifo`; no +`_sim_recv_min`/`_vclock`/`_sim_hold_busy_slots` anywhere); (3) the variance-cadence rung layer; (4) the +availability *effect* path + `EVENT_AVAIL_CHANGE` emission. The build is organized as **4 pytest-gated +batches across 3 phases** (§E). **Prerequisites (read first):** - [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) -- the parity methodology (the ladder §1, roles/ @@ -138,86 +151,136 @@ forward-grad rungs §F.4). This section states only which rungs apply and how th --- -## §E Staged implementation + validation - -Each stage: config-gated (flag-off => byte-identical), test-guarded, exit criteria + min run length -(PARITY.md budget table). Land one mechanism per run round. **Smoke (5 min) before any multi-hour run.** - -### Stage 0 -- Telemetry coverage + real admissibility (gate) -Confirm fwdllm emits every field the ladder reads: `vclock_now`, `sim_completion_ts`/`sct`, -`model_version`, `data_id`, `iteration_per_data_id`, `var`, `var_threshold`, `_agg_goal`(K), -`dynamic_c`(C), `agg_goal_count`, per-commit `train_duration`, `WALL_SEND_TS`/`WALL_RECV_TS`, grad/JVP -norm, **and `avail_change`/`agg_belief_change`/`send_gate_wait`** (§A.3 item 4). Implement **TC1** for -fwdllm; **partition eval vs train commits** on the variance-pass eval (async_cifar10 dead-end: eval commits -polluting `agg_rounds` broke monotone/staleness). Run `validate_real` admissibility. **Min:** 5-10 min -smoke. **Exit:** TC1 all-present both modes; real admissible; report JSON no SKIPs. - -### Stage 1 -- Trainer sim path (modeled completion, no wall sleep) -Add a `simulated` mode to `FedSgdTrainer`: replace `_emulate_training_delay` (real `time.sleep`) with -stamping `_sim_completion_ts = sim_send_ts + max(real_gpu_s, modeled_budget_D) + sim_completion_leg_s`; -send `SIM_COMPLETION_TS` + `SIM_CLIENT_TASK_TRAIN_DURATION_S` + `WALL_SEND_TS`/`WALL_RECV_TS`. Stamp a -**per-eval** `sct` (dead-end: reusing the last train `sct` past-dates every eval); eval delay ~= train -(forward pass), **not** async_cifar10's 20x speedup -- **decision D4**. **Min:** 10 min. **Exit:** P3 matches; -K6 advancing; T2 matched; no real `time.sleep` on the sim path. - -### Stage 2 -- Drive the virtual clock from the grad loop (THE structural port) -Replace the raw `channel.recv_fifo(...,1)` in `_aggregate_grads_async` (`:693`) with the inherited -**`_sim_recv_min`** path (sct-ordered reorder buffer + §3.drain) so a fast trainer's grad buffers as a -future and grads are consumed in completion order; wire **`_sim_hold_busy_slots`** (§3.resid) for -one-in-flight-per-trainer. **fwdllm processes one message per `_aggregate_grads_async` call**, so key the -slot release on the **agg-goal boundary** (where `_per_agg_trainer_list` clears), NOT per-message -- the -hold/release points differ from asyncfl's batch `_aggregate_weights`. Keep `simCommitOverheadSeconds=0` -(overhead on the vclock is a hard dead-end). **Risk:** a data_id can span **many** agg-goal cycles -(rollback path) -- slot-hold + sct-buffer must survive rollbacks without leaking or double-committing a -grad; over-instrument `inflight_residence` + per-cycle buffer occupancy. **Min:** 45 min. **Exit:** K1 -monotone; K3a/K3b (per variance-pass) ~= 0 residual; U5 inter-arrival; one-in-flight overlap ~= real; -no past-dating (commit_gap ~= 0). - -### Stage Avail -- Wire unavailability into the grad loop (syn_20/50/mobiperf) -Wire the inherited `ClientAvailability` effect path into fwdllm: send-time gate (real) / -`delivery_ts = max(sct, next_avail)` buffering (sim); the two ledgers; per-baseline in-flight timing -(reactive-90s for all three fwdllm baselines; no proactive-evict baseline here); starvation vclock-advance -under scarcity; and per-baseline `avail_select_filter` (fwdllm off, fwdllm_plus/fluxtune on). Emit the -availability telemetry (Stage 0) so A6/A7/A8/K11 run. **Interaction risk (decision D3):** a withheld/late -grad meets the variance gate -- does it roll into `cached_v` on a rollback, and does a late grad against an -old `model_version` inflate the variance signal? This is genuinely new vs async_cifar10 (which commits -weights, not a variance-gated pool). **Min:** 45-90 min. **Exit:** A1-A5 + A6/A7/A8 PASS; withheld-then- -delivered (not dropped); self-stops (`"stopping run"`, no `SIM_WALL_CEILING`); syn_0 byte-identical gate -ON vs OFF. - -### Stage 3 -- Variance-cadence parity (the fwdllm prize) -Implement V1-V5, G1-G2, DK1-DK3 (PARITY.md §F.4). Likely roots (confirm via the lowest broken rung): -contributing-set/order divergence (U5/S2 upstream) -> V1 -> K2; grad-pool accumulation order -(`cached_v` carry-over, `grad_pool.append` order) -> V2 with matched inputs = a true sim bug; force-commit -rate (V4) = chronic variance divergence, not a separate bug. **DynamicKC coupling:** validate **DK3** -(policy *input*) before DK1/DK2 (fluxtune leaves dynamic_kc off, so DK is inert there; relevant only if a -baseline enables it). **Min:** 90 min - 2 h (variance-feedback-compounding). **Exit:** V1/V2/V5 PASS; K2 -(committed-data_ids/vsec) PASS; DK tracks if enabled. - -### Stage 4 -- Selection fidelity (fluxtune only, `async_oort`) -Validate A2c/Sx/Sd/S2 + the §S.pacer/§S.temporal/§S.dur stack (all landed in `flame/selector/oort.py`). -For the two `random`-selector baselines this stage is inert (S-rungs WARN/skip). **Min:** 45 min - 3 h (a -selection-mix residual can surface late). **Exit:** A2c PASS, Sd binding real~=sim, no pacer ratchet. - -### Stage 5 -- Sync-path parity (fwdllm / fwdllm_plus, `_aggregate_grads_sync`) -Give the sync grad barrier the barrier-anchored visibility-lag treatment (§6.u6). fluxtune streams (async), -commits each grad at its own sct -> leave per-message correct. **Min:** 45 min. **Exit:** U6 barrier- -anchored lag real~=sim; no per-message past-dating. - -### Stage 6 -- Convergence sign-off -C1 accuracy, C2 loss, K8/U2 terminal-state @ matched **data_id**. **Min:** full (3-4 h+). **Exit:** curves -within tolerance at matched data_id; K8/U2 rel within bar. +## §E Implementation plan -- 4 pytest-gated batches across 3 phases + +**Batching principle (why this shape).** Each batch is a *large dev push* that lands a complete +mechanism cluster **plus its pytest coverage together**, ending in ONE "pause to test" that is +**pytest-only** -- no experiment run in the inner loop. Experiment runs (smoke -> convergence) happen +**once per phase, gated behind green pytests**, never as the dev loop. This is affordable because the +parity **rungs are pure functions over telemetry** (`parity_checks.run_all_parity`): we test them by +constructing synthetic real/sim telemetry pairs with known-correct and known-broken deltas and asserting +PASS/FAIL; and the mechanisms (sct reorder buffer, one-in-flight hold, delivery buffering, sync barrier) +are driven with synthetic message sequences. Every batch is **config-gated: flag-off => byte-identical** +to today; the flag-off regression is itself a pytest assertion. + +**Phase order (locked at kickoff):** Phase 1 = **100% availability (syn_0) parity for all three +baselines**; Phase 2 = **unavailability (syn_20/50/mobiperf)**; Phase 3 = **beyond syn_0**. Do not start a +phase until the prior phase's sign-off run is banked in §H. + +**Two staging changes from the original stage list (below):** (1) the **sync-path port** +(`_aggregate_grads_sync`, old "Stage 5") moves **into Phase 1 Batch 1** -- fwdllm/fwdllm_plus are *sync*, +so their primary commit path is the barrier; they cannot reach syn_0 parity without it. (2) old "Stage 0" +is a *validation gate*, not a batch -- its telemetry is already landed (see Current-state callout). + +--- + +### PHASE 1 -- 100% availability (syn_0), all three baselines + +#### Batch 1 (large dev) -- both structural ports: trainer sim path + async grad loop + sync barrier +The single biggest structural batch. Lands three tightly-coupled ports (the trainer's stamped completion +is exactly what the aggregator's sct buffer consumes, so they are tested together). Reference port for the +trainer: async_cifar10 `trainer/pytorch/main.py` already has the full `time_mode`/`simulated`/ +`_sim_completion_ts`/`sim_completion_leg_s` plumbing (lines 106/173-174/840-841/1086-1093/1151-1155) -- +this is a port of a known-good shape, not a design problem. + +- **Trainer** (`FedSgdTrainer.py`): add `time_mode`/`simulated`; replace `_emulate_training_delay`'s real + `time.sleep` (`:481`) with stamping `_sim_completion_ts = sim_send_ts + max(real_gpu_s, modeled_budget_D) + + sim_completion_leg_s`; send `SIM_COMPLETION_TS` + `SIM_CLIENT_TASK_TRAIN_DURATION_S` + + `WALL_SEND_TS`/`WALL_RECV_TS`. Stamp a **per-eval** `sct` (dead-end: reusing the last train `sct` + past-dates every eval); eval delay ~= train (forward pass), **not** async_cifar10's 20x speedup -- D4. +- **Async grad loop** (`_aggregate_grads_async:693`, fluxtune): replace raw `channel.recv_fifo(...,1)` with + the inherited **`_sim_recv_min`** (sct-ordered reorder buffer + §3.drain) + **`_sim_hold_busy_slots`** + (§3.resid, one-in-flight) + `_advance_sim_clock`. **fwdllm processes one message per call**, so key the + slot release on the **agg-goal boundary** (where `_per_agg_trainer_list` clears), NOT per-message. Keep + `simCommitOverheadSeconds=0` (overhead on the vclock is a hard dead-end). **Rollback risk:** a data_id + spans **many** agg-goal cycles -- slot-hold + sct-buffer must survive rollbacks without leaking or + double-committing a grad; over-instrument `inflight_residence` + per-cycle buffer occupancy. +- **Sync barrier** (`_aggregate_grads_sync:1354`, fwdllm/fwdllm_plus): barrier-anchored visibility-lag + treatment (PARITY.md §6.u6); vclock advance at the barrier. *(Pulled forward from old Stage 5.)* +- **Launchers:** add `time_mode: simulated` variants of the three `expt_scripts/*_n10_smoke.yaml`. + +**Pause to test (pytest-only):** unit tests for (a) trainer stamping formula + per-eval `sct` + a +"no `time.sleep` on the sim path" assertion; (b) the sct reorder buffer & one-in-flight hold over a +synthetic out-of-order message sequence **including a rollback** (no leaked/double-committed grad); (c) +sync-barrier vclock advance + no per-message past-dating. Then one tiny in-process 2-3-trainer **seeded** +run per path -> feed telemetry to `parity_checks`. Full existing `test_fwdllm_*` suite green (flag-off +byte-identical regression). **Exit rungs:** P3 matches; K6 advancing; T2 matched; K1 monotone; K3a/K3b +(per variance-pass) ~= 0; U5 inter-arrival; one-in-flight overlap ~= real; U6 barrier lag real~=sim; +commit_gap ~= 0. + +#### Batch 2 (large dev) -- variance-cadence layer + fluxtune selection fidelity +- **Variance-cadence rungs** (PARITY.md §F.4): implement V1-V5 / DK1-DK3 / G1-G2 in the shared engine. + Most dev is checker + the telemetry to feed it: per-cycle `var` trajectory (at each agg-goal), + iterations-per-`data_id` (realized dynamic-K), force-commit (`max_iterations_per_data_id`) bypass rate, + and a `cached_v` carry-over diagnostic (V3). Likely roots, confirm via the *lowest broken rung*: + contributing-set/order divergence (U5/S2 upstream) -> V1 -> K2; grad-pool accumulation order + (`cached_v` carry-over, `grad_pool.append` order) -> V2 with matched inputs = a true sim bug; + force-commit rate (V4) = chronic variance divergence, not a separate bug. **DynamicKC: DK3 (policy + *input*) before DK1/DK2** (inert unless a baseline enables dynamic_kc; fluxtune leaves it off). +- **Selection fidelity (fluxtune only, `async_oort`):** validate A2c/Sx/Sd/S2 + the §S.pacer/§S.temporal/ + §S.dur stack (already landed in `flame/selector/oort.py`) -- validation + any fwdllm-specific wiring, not + a rebuild. Inert for the two `random`-selector baselines (S-rungs WARN/skip). + +**Pause to test (pytest-only):** unit-test each new rung function against **constructed real/sim telemetry +pairs** with known cadence deltas (known-PASS fixtures + known-broken force-commit-rate / var-trajectory +fixtures). Then a seeded mini-run per baseline -> `parity_checks` -> assert applicable rungs PASS. **Exit +rungs:** V1/V2/V5 PASS; K2 (committed-data_ids/vsec) PASS; A2c PASS + Sd binding real~=sim + no pacer +ratchet (fluxtune); DK tracks if enabled. + +#### Phase-1 sign-off run (the one time-consuming step of the phase) +Only after Batches 1-2 are green in pytest: one smoke (5 min) then one convergence run **per baseline** at +syn_0 -> `parity_checks` full battery + **C1/C2 at matched `data_id`**. Record in §H. Gate to Phase 2. + +--- + +### PHASE 2 -- unavailability (syn_20/50/mobiperf) + +#### Batch 3 (large dev) -- wire the ClientAvailability effect path into the grad loop +- **Effect path:** send-time gate (real) / `delivery_ts = max(sct, next_avail)` buffering (sim); the two + ledgers; reactive-90s in-flight for all three baselines (no proactive-evict baseline here); starvation + vclock-advance under scarcity; per-baseline `avail_select_filter` (fwdllm off; fwdllm_plus via the landed + oracular read; **fluxtune via `trace_read` for v1 -- D1 resolved (b), `client_notify` deferred to + Stage H**). +- **Availability telemetry (D2):** emit `EVENT_AVAIL_CHANGE` (trainer state machine) + + `agg_belief_change`/`send_gate_wait` (aggregator belief hooks). All three builders exist in + `flame/telemetry/events.py`; this unlocks A6/A7/A8/K11. +- **D3 interaction (over-instrument BEFORE trusting cadence):** a withheld/late grad meets the variance + gate -- does it roll into `cached_v` on a rollback? Does a late grad against a stale `model_version` + inflate the `var` signal (and thus dynamic-K)? Genuinely new vs async_cifar10 (which commits weights, + not a variance-gated pool). + +**Pause to test (pytest-only):** adapt async_cifar10's availability test patterns +(`scripts/parity/test_availability_rungs.py`, `test_delivery_ledger.py`, `test_starvation_termination.py`) +for fwdllm; unit-test delivery buffering, withheld-then-delivered (not dropped), starvation self-stop +(`"stopping run"`, no `SIM_WALL_CEILING`), and the **syn_0 gate-ON-vs-OFF byte-identical** invariant. +`test_eot_avail_catchup.py` already covers part of this. **Exit rungs:** A1-A5 + A6/A7/A8 PASS. + +#### Phase-2 sign-off run: one run per (baseline x trace) -> A1-A5 + A6/A7/A8 PASS; self-stops; withheld +grads delivered not dropped. Record in §H. Gate to Phase 3. + +--- + +### PHASE 3 -- beyond syn_0 (the added-complexity layer, last) + +#### Batch 4 -- full ladder under syn_20/50/mobiperf + convergence sign-off +Mostly runs + checker, minimal new dev. V/DK rungs under scarcity (force-commit rate shifts), K8/U2 +terminal-state @ matched `data_id`, C1/C2. **Bin V1/V2 by run-fraction** to separate a *constant* mix bias +from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **Min:** full (3-4 h+). +**Exit:** curves within tolerance at matched data_id; K8/U2 rel within bar; V1/V2 binned residual flat. --- ## §F Key design decisions (open -- resolved at implementation) -- **D1 -- tracking-mode strategy.** fluxtune's landed config uses `tracking_mode: client_notify`, which - UNAVAILABILITY_DESIGN.md treats as the **deferred Stage-H** message-transport model (async_cifar10 v1 is - all `trace_read`). Options: (a) implement `client_notify` first-class for fwdllm now (larger scope, but - it's baked into fluxtune's config and can't just be ignored), or (b) map fluxtune onto `trace_read` for a - v1 parity pass and treat `client_notify` as Stage H. **Lean (a)** -- fluxtune cannot run its intended - taxonomy without it. fwdllm_plus's `oracular` and fwdllm's unaware paths are already `trace_read`-shaped. +- **D1 -- tracking-mode strategy. RESOLVED: option (b) -- map fluxtune onto `trace_read` for the v1 parity + pass; `client_notify` is deferred to Stage H.** fluxtune's landed config uses `tracking_mode: + client_notify`, which UNAVAILABILITY_DESIGN.md treats as the **deferred Stage-H** message-transport model + (async_cifar10 v1 is all `trace_read`). We take the smaller Phase-2 scope: fluxtune runs an approximated + `trace_read`-shaped availability model in v1 (aware-at-selection via trace read, reactive-90s in-flight), + and first-class `client_notify` becomes a later stage once syn_0->unavailability parity is banked. + fwdllm_plus's `oracular` and fwdllm's unaware paths are already `trace_read`-shaped, so this makes all + three baselines share one substrate for v1. *(Original doc leaned (a); reversed at kickoff to keep + Phase 2 tractable.)* - **D2 -- availability telemetry port.** The fwdllm trainer emits no `EVENT_AVAIL_CHANGE`; the aggregator emits no `agg_belief_change`/`send_gate_wait`. All three builders exist in `flame/telemetry/events.py` -- port emission (trainer state machine + aggregator belief hooks) so A6/A7/A8/K11 light up. Prereq for From 32c2813be62ad8c7c04896d443e85068d305454c Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Thu, 2 Jul 2026 17:34:16 -0400 Subject: [PATCH 02/56] initial going through of code and documenting --- lib/python/examples/fwdllm/simulate_fwdllm.md | 171 +++++++++++++++++- 1 file changed, 168 insertions(+), 3 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index d984139dd..d9e56a2fe 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -336,6 +336,171 @@ those are baseline-defining knobs; a cadence gap is an upstream set/order/clock ## §H Status -*(empty -- first entry after Stage 0 smoke on the implementation branch. Record per-baseline run dirs, -lowest broken rung, root hypothesis, score X/N, JSON path; keep the run-length budget keyed. One section, -updated in place.)* +*(Run-log: record per-baseline run dirs, lowest broken rung, root hypothesis, score X/N, JSON path; keep +the run-length budget keyed. One section, updated in place. No parity run yet -- first entry lands after +Batch 1's pytest gate.)* + +--- + +## §I Batch 1 implementation map (code-level -- captured during study, before edits) + +**Purpose:** durable engineering map so Batch 1 can resume cold. All line numbers are as-of branch +`dg/fwdllm_sim_unavail` at capture time; re-grep before editing. Batch 1 = trainer sim path + async grad +loop vclock drive + sync barrier, all config-gated (flag-off => byte-identical), syn_0 only. + +### I.1 Class chain + where the sim substrate already lives (INHERITED, reuse -- don't reimplement) +- Trainer: `examples/fwdllm/trainer/forward_training/FedSgdTrainer.py::FedSGDTrainer(Trainer)` -> + `flame/mode/horizontal/syncfl/fwdllm_trainer.py::Trainer(Role)`. **The messaging (recv/send) lives in + the base `fwdllm_trainer.Trainer`, not FedSgdTrainer.** +- Aggregator: `FedSgdAggregator` -> `syncfl/fwdllm_aggregator.py::TopAggregator(AsyncTopAgg)` -> + `asyncfl/top_aggregator.py::TopAggregator` -> `syncfl/top_aggregator.py::TopAggregator`. +- **`self.simulated` / `self.time_mode` / `self._vclock` (VirtualClock) / `_advance_sim_clock(sct)` are + defined in `syncfl/top_aggregator.py:199-203, 331`** (`vclock=max(vclock,sct)`; overhead default 0 -- + keep 0). Read `time_mode` from `config.hyperparameters.time_mode` (default "simulated"). +- **`SimReorderBuffer` `self._sim_buffer` + `self._sim_committed`** inited in `syncfl/top_aggregator.py:228` + AND `asyncfl/top_aggregator.py:112`. Async `__init__` also inits `_sim_inflight_expected`, + `_sim_pending_commit`, `_sim_trainer_budget`, budget running-mean/min, withheld ledgers, cooldown, and + the flag knobs `sim_sct_ordered_drain` / `sim_inflight_residence` / `sim_clock_jump_clamp` / + `sim_staggered_redispatch` / `sim_redispatch_gap_s` (`asyncfl:112-195`). **TODO verify fwdllm agg's + `__init__`/`internal_init` actually chains through the async base so these exist** (else lazy-init guard + at `asyncfl:323` covers `_sim_inflight_expected` et al. but NOT `_sim_buffer`). +- **Sync-barrier machinery ALREADY EXISTS in the syncfl base and just needs calling:** + `_sync_sim_recv_first_k(channel, ends, first_k)` (`syncfl/top_aggregator.py:360`) -- drains the selected + set, commits the `first_k` smallest-sct, advances vclock to each, stamps + `PROP_CLIENT_TASK_TRAIN_DURATION` from `SIM_CLIENT_TASK_TRAIN_DURATION_S`; and + `_barrier_anchored_lags(durs)` (`:472`, the U6 helper: `lag_i = max_completion - completion_i`). +- **Async single-message commit machinery:** `_sim_recv_min(channel, recv_ends)` (`asyncfl:314`) returns + ONE sct-ordered committed `(msg, md)` -- maps onto fwdllm's one-message-per-call loop -- but is coupled + to WEIGHTS semantics (staggered-redispatch keys on WEIGHTS/WEIGHTS_BYTES; budget from + TRAINING_BUDGET_S; withheld/eval paths). With gate OFF + defaults OFF these are inert. `_sim_hold_busy_ + slots(channel)` (`asyncfl:1453`) holds busy trainers' concurrency slots until commit (one-in-flight). +- **Reference port for the trainer half:** async_cifar10 `trainer/pytorch/main.py` has the full + `time_mode`/`simulated`/`_sim_completion_ts`/`sim_completion_leg_s`/per-eval-sct plumbing + (`:106,173-174,818-845 (train sct),1086-1096 (eval sct),1151-1158`). Use it for the *plumbing shape*, + **NOT the timing formula** -- see the critical correction below. +- **!!! CRITICAL CORRECTION -- fwdllm uses an ADDITIVE delay model, NOT cifar10's `max(gpu, D)` !!!** + cifar10's real mode "sleeps to fill a budget": wall = `max(gpu, D)`. **fwdllm's real mode sleeps D ON TOP + of GPU time** (a flat additive emulated delay -- see `FedSgdTrainer._emulate_training_delay:481` docstring + "flat additive sleep on top of GPU time, not a budget-minus-actual model" and the EXISTING telemetry + `sim_round_duration_s = _real_gpu_time_s + _delay_s` at `FedSgdTrainer.py:545`). So for real<->sim parity + the sim sct MUST be additive too: + - `D = _emulate_training_delay()`'s modeled value = `training_delay_s / training_delay_factor` + (÷`speedup_factor`, which is 1.0). NOTE the method is named "training" but models the **eval/forward + cost** (`training_delay_factor` is the eval speedup); fwdllm's forward-grad "train" IS a forward pass. + - `sim_round_duration = _real_gpu_time_s + D` (ADDITIVE -- matches real's sleep-on-top) + - `_sim_completion_ts = (_sim_send_ts) + sim_round_duration + (leg if simulated)` + - **Port is minimal because the additive delay is already computed:** existing `train_with_data_id` + already does `_delay_s = self._emulate_training_delay()` (`:523`) and `sim_round_duration_s = + _real_gpu_time_s + _delay_s` (`:545`). The port just (a) makes `_emulate_training_delay` NOT sleep in + sim mode but still RETURN D, and (b) stamps `_sim_completion_ts` from that same `sim_round_duration_s`. + - **Latent bug to fix while here:** `_emulate_training_delay` gates on `training_delay_enabled == "True"` + (STRING compare, `:488`) but the config schema types it as `bool` (`config.py:197`, default False). So + with a real bool the delay NEVER fires. Normalize the check (`str(...)=="True" or ... is True`) or the + modeled D is silently 0 and real==sim only trivially. Verify what the launcher actually passes. + +### I.2 Trainer sim-path port -- exact edit sites +1. **`fwdllm_trainer.Trainer._fetch_weights` (`:168`, msg parse ~`:208-320`)** -- currently reads ROUND/ + WEIGHTS/DATA_ID but NOT sim stamps. ADD: `self._sim_send_ts = msg.get(MessageType.SIM_SEND_TS)` and + `self._wall_recv_ts = time.time()` on model receipt. (Base syncfl trainer already does the analogous + `_sim_send_ts = msg[SIM_SEND_TS]` at `syncfl/trainer.py:257` -- fwdllm's trainer doesn't extend it, so + port the read.) +2. **`FedSgdTrainer.__init__` (`:115`)** -- config knobs at `:176-182` (`training_delay_enabled`, + `training_delay_s`, `training_delay_factor`, `speedup_factor=1.0`). ADD: + `self.time_mode = config.hyperparameters.time_mode` (default "simulated"), + `self.simulated = self.time_mode == "simulated"`, `self.sim_completion_leg_s` (default 0.0), + init `self._sim_send_ts=None`, `self._sim_completion_ts=None`, `self._sim_round_duration_s=None`, + `self._wall_recv_ts=None`. +3. **`FedSgdTrainer._emulate_training_delay` (`:481`, called `:523`)** -- currently `time.sleep(_sleep_s)`. + GATE: when `self.simulated`, DO NOT sleep; still RETURN the modeled delay D (so `_delay_s` stays correct + for the additive `sim_round_duration_s`). Fix the `== "True"` string-gate bug (see CRITICAL CORRECTION). +4. **`FedSgdTrainer.train_with_data_id` (`:502`)** -- already computes `_delay_s` (`:523`) and + `sim_round_duration_s = _real_gpu_time_s + _delay_s` (`:545`, ADDITIVE). ADD: store + `self._sim_round_duration_s = _real_gpu_time_s + _delay_s` and + `self._sim_completion_ts = (self._sim_send_ts or _fallback) + self._sim_round_duration_s + (leg if + simulated)` for `_send_grads` to read. **Per-eval sct (D4):** fwdllm eval is a forward pass ~= train + cost (NOT cifar's /20; `training_delay_factor` already encodes the modest eval speedup). If the trainer + performs a distinct eval task, stamp eval's OWN sct = `_sim_send_ts + (real_eval_gpu + D_eval)` -- do NOT + reuse the last train sct (past-dates every eval, poisons the reorder-buffer key, cifar `:1086-1096`). + **VERIFY:** in fwdllm the *aggregator* runs `eval_model()` (`fwdllm_aggregator.py:1391`) on the global + model after a variance pass; the trainer's `task_to_perform=="eval"` only sends MODEL_VERSION+STAT_UTILITY + (`_send_grads:522-526`). So confirm whether a trainer-side eval sct is even needed for fwdllm, or whether + D4 collapses to "train sct only" (likely the latter -- eval lives on the aggregator, off the grad clock). +5. **`fwdllm_trainer.Trainer._send_grads` (`:438`, msg built `:506-521`)** -- ADD to the train msg: + `SIM_COMPLETION_TS=self._sim_completion_ts`, `SIM_CLIENT_TASK_TRAIN_DURATION_S=self._sim_round_duration_s`, + `TRAINING_BUDGET_S=`, `WALL_SEND_TS=time.time()`, `WALL_RECV_TS=self._wall_recv_ts`. The eval + msg (`:522-526`) also needs `SIM_COMPLETION_TS`/`WALL_*` for the per-eval sct. + +### I.3 Aggregator dispatch stamping (both distribute paths) -- exact edit sites +- **`_distribute_weights_sync` (`:1777`, per-end send loop `:1849-1900`, `channel.send` `:1899`)** and + **`_distribute_weights_async` (`:1904`, `channel.send` `:2008`)**. In each per-end loop, when + `self.simulated`: set `_sst = self._vclock.now`; inject `payload[MessageType.SIM_SEND_TS] = _sst` (or a + per-end copy -- payload is shared across ends, so stamp per-end or set the channel prop); + `channel.set_end_property(end, PROP_SIM_SEND_TS, _sst)`. **Async only:** populate the gate + `self._sim_inflight_expected[end] = _sst + ` (budget = running-mean/min floor, see + `asyncfl:1671` for the weights-path analog). `PROP_ROUND_START_TIME` already set at `:1866`. + `PROP_SIM_SEND_TS`/`PROP_SIM_COMPLETION_TS` imported from `flame.selector.properties`. + +### I.4 Aggregator grad-loop vclock drive -- exact edit sites + the fwdllm-specific wiring +- **Async: `_aggregate_grads_async` (`:693`)** -- currently `next(channel.recv_fifo(ends,1,timeout=...))` + at `:719`. REPLACE (sim branch only) with an sct-ordered buffered pop. Design decision (see §G risk + "one-message-per-call vs batch"): reuse the shared primitives (`SimReorderBuffer`, `_advance_sim_clock`, + `_sim_inflight_expected` gate) but wire them for the grad loop rather than calling weights-coupled + `_sim_recv_min` verbatim -- ingest all ready grad msgs into `_sim_buffer` keyed by + `msg[SIM_COMPLETION_TS]`, pop min-sct committable, `_advance_sim_clock(sct)`, return one `(msg, md)`. + Real branch unchanged. +- **Slot hold/release (one-in-flight):** call `_sim_hold_busy_slots(channel)` on dispatch; **release keys + on the AGG-GOAL BOUNDARY**, i.e. where `_process_aggregation_goal_met` clears + `self._per_agg_trainer_list = []` (`:1271`) and resets `_agg_goal_cnt=0` (`:1279`) -- NOT per message. + (The doc's §3.resid release-point warning: do not copy asyncfl's per-batch release.) +- **Sync: `_aggregate_grads_sync` (`:1354`) -> `sync_collect_and_accumulate_grads` (`:1313`)** -- the drain + loop `channel.recv_fifo(channel.ends(), num_min_req, timeout=...)` at `:1336`. REPLACE (sim branch) with + `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` then feed each committed + `(msg, md)` through `_process_single_trainer_message`. Apply `_barrier_anchored_lags` for U6 telemetry. +- **Commit path already computes intrinsic duration:** `_process_single_trainer_message` (`:745`) at + `:837` calls `real_client_task_train_duration(msg, sent_ts, timestamp)` (WALL_SEND-WALL_RECV) and sets + `PROP_CLIENT_TASK_TRAIN_DURATION` -- keep; this is the selector/telemetry duration (server-overhead-free). + +### I.5 ROLLBACK-SAFETY (the fwdllm-specific §3.resid risk -- D-note) +`_process_aggregation_goal_met` (`:1081`): `self.aggregate(self._round)` (`:1153`) runs the variance gate. +**PASS** (`var_good_enough`, `:1170`) -> advance `data_id`, `iteration_per_data_id=0`. **FAIL** (`:1237`) -> +retry SAME `data_id`, `iteration_per_data_id += 1` (the ROLLBACK path). Both branches clear +`_per_agg_trainer_list` (`:1271`) and reset `_agg_goal_cnt=0` (`:1279`). So a `data_id` can span MANY +agg-goal cycles. **Invariant to preserve:** the sct reorder buffer + slot-hold must not strand or +double-commit a grad across a rollback -- release slots and drain/clear the buffer's committed set at the +agg-goal boundary each cycle. `_sim_committed` is cleared by the async base `_reset_agg_goal_variables` +(`asyncfl:231-232`) but fwdllm's own reset (`:453`) does NOT -- **verify/port this clear into the fwdllm +agg-goal boundary** or the second cycle on a rolled-back data_id sees stale committed marks. Over-instrument +`inflight_residence` + per-cycle buffer occupancy. + +### I.6 MessageType / API facts -- VERIFIED (capture time) +- **CONFIRMED** MessageType keys exist (`flame/mode/message.py:82-87`): `SIM_SEND_TS=34`, + `SIM_COMPLETION_TS=35`, `SIM_CLIENT_TASK_TRAIN_DURATION_S=36`, `TRAINING_BUDGET_S=37`, `WALL_SEND_TS=38`, + `WALL_RECV_TS=39`. +- **CONFIRMED** `SimReorderBuffer` (`flame/sim/virtual_clock.py:52`): `.add(end, sim_completion_ts, + payload)`, `.has(end)`, `.pending_ends()->set`, `__len__`, `.peek_min_ts()->Optional[float]`, + `.pending_after(ts)->set`, `.pop_min()->Optional[(end, ts, payload)]`, `.discard(end)`, `.clear()`. + `VirtualClock`: `.now` (property), `.advance(ts)`, `.reset()`. +- **CONFIRMED** fwdllm agg chains the sim substrate: `fwdllm_aggregator.internal_init` (`:239`) calls + `super().internal_init()` (`:241`) -> asyncfl `internal_init` (`asyncfl:83`, inits `_sim_buffer` at + `:112`) -> syncfl base. So `_sim_buffer`/`_sim_inflight_expected`/`_vclock`/`simulated` all exist. (Role + pattern uses `internal_init`, NOT `__init__`.) +- **ROLLBACK HOLE CONFIRMED (act on I.5):** fwdllm's `_reset_agg_goal_variables` (`:450`) does NOT clear + `_sim_committed` (the async base's does, `asyncfl:231-232`). fwdllm's real agg-goal boundary is + `_process_aggregation_goal_met` (clears `_per_agg_trainer_list`/`_agg_goal_cnt` directly at `:1271,1279`). + **=> Port must explicitly `_sim_committed.clear()` + drain/clear any stranded `_sim_buffer` entries at that + boundary each cycle**, so a rolled-back data_id's next cycle starts clean. + +### I.7 Launcher configs +Add `time_mode: simulated` variants of `expt_scripts/{fluxtune,fwdllm,fwdllm_plus}_n10_smoke.yaml` +(current default is `time_mode: real`). Keep the real ones for the reference runs. + +### I.8 Batch 1 pytest gate (pytest-only, no experiment run) +1. **Trainer stamping** (extend `tests/mode/test_fwdllm_trainer_sim_duration.py`): `_sim_completion_ts == + _sim_send_ts + max(gpu,D) + leg`; per-eval sct distinct from last train sct; NO `time.sleep` on the sim + path (monkeypatch/assert). 2. **sct reorder buffer + one-in-flight** (new test): drive the grad loop + with a synthetic out-of-order message sequence incl. a variance-FAIL rollback cycle -> assert commit + order = sct order, vclock monotone, no leaked/double-committed grad across the rollback, slots released + at agg-goal boundary. 3. **Sync barrier** (new test): `_sync_sim_recv_first_k` commits the k smallest + sct; `_barrier_anchored_lags` correct; vclock advances to k-th; no per-message past-dating. + 4. **Flag-off byte-identical:** `time_mode: real` path unchanged. 5. Full existing `test_fwdllm_*` green. +**Exit rungs:** P3, K6, T2, K1 monotone, K3a/K3b~=0, U5, one-in-flight overlap~=real, U6 barrier lag, commit_gap~=0. From f2607f97a728f37baf46d00c6d85f95a45fc48c9 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Thu, 2 Jul 2026 18:11:52 -0400 Subject: [PATCH 03/56] in flight batch 1 changes --- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 101 +++++++ .../expt_scripts/fwdllm_n10_smoke_sim.yaml | 98 +++++++ .../fwdllm_plus_n10_smoke_sim.yaml | 94 ++++++ lib/python/examples/fwdllm/simulate_fwdllm.md | 277 ++++++++++++++++++ .../trainer/forward_training/FedSgdTrainer.py | 77 ++++- lib/python/examples/fwdllm/trainer/main.py | 19 +- .../horizontal/syncfl/fwdllm_aggregator.py | 239 +++++++++++++-- .../mode/horizontal/syncfl/fwdllm_trainer.py | 29 ++ 8 files changed, 892 insertions(+), 42 deletions(-) create mode 100644 lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml create mode 100644 lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml create mode 100644 lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml new file mode 100644 index 000000000..32770e305 --- /dev/null +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -0,0 +1,101 @@ +# FedFwd SIMULATED-clock smoke test: 10 trainers, agnews H5 partitions +# (client_idx 0-9 of the 100 available niid_label_clients=100_alpha=1 +# partitions). Sim sibling of fluxtune_n10_smoke.yaml -- IDENTICAL except +# time_mode: simulated and the job id (so run dirs don't collide). +# +# fluxtune is the ASYNC baseline (async_oort selector, fedbuff), so this +# variant exercises the Batch-1 async grad-loop sim drive: _sim_recv_min_grad +# (sct-ordered reorder buffer + in-flight gate + virtual-clock advance) with +# agg-goal-boundary slot release. Trainers still do real forward-grad compute +# (gradient VALUES mode-invariant); only the clock/ordering differ. +# +# enable_training_delays stays false (as in the real smoke) for direct +# comparability; modeled-delay-on (D>0) is for later parity runs, enabled in +# BOTH real reference and sim run together (see simulate_fwdllm.md §K-D8). +# +# Run: +# python -m flame.launch.run_experiment \ +# lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml + +experiments: + - name: fluxtune_n10_smoke_sim + description: "FedFwd SIM smoke (10 trainers, fluxtune baseline: async async_oort+fedbuff+JVP+3-tier, time_mode=simulated). Validates the Batch-1 async grad-loop sim drive." + + baseline: fluxtune + + example: + # fwdllm's trainer entrypoint has no pytorch/ subdirectory, unlike the + # ExampleConfig default (trainer/pytorch/main.py). + trainer_main: trainer/main.py + + trainer: + num_trainers: 10 + start_id: 1 + dataset: + name: agnews + # H5 path-style dataset -- no _metadata/dataset_splits/ index-list + # file exists (or ever will) for agnews; skips that lookup entirely. + path_style: true + availability: + # Cosmetic here -- fluxtune's 3-tier signal comes from + # trainer.client_notify (set in the baseline), not this mode. Kept + # consistent with the baseline's mobiperf_3st_50 trace. + mode: mobiperf_3st_50 + # Batch 1: drive the simulated virtual clock instead of real sleeps. + time_mode: simulated + # Skip the registry's 4-18s artificial delay for a fast smoke run. + enable_training_delays: false + # client_idx = (trainer_id - 1) % 100 -- wraps 10 trainers onto a + # distinct, non-colliding subset (0-9) of the 100 H5 partitions. + client_idx_modulo: 100 + config_overrides: + hyperparameters: + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + # Uniform/IID partition for smoke tests -- avoids the niid_label + # baseline default's data-skew confound while validating mechanics. + partition_method: uniform + + aggregator: + config_template: configs/aggregator_base.json + selector: async_oort + tracking_mode: client_notify + agg_goal: 3 + log_to_wandb: false + config_overrides: + job: + id: fluxtune_n10_smoke_sim + hyperparameters: + rounds: 50 # short smoke run, not the full 300 + # Smoke-test stop bar: whichever of these fires first ends the run + # (`rounds` alone is much too coarse, since one round is 150 + # data-id completions). + max_data_id_progress: 10 # stop once data_id reaches 10 + max_runtime_s: 1800 # ...or after 30 minutes wall time + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + partition_method: uniform + selector: + kwargs: + # Scaled down from the baseline's n=150-ish defaults + # (c=15, aggGoal=5, minInitialTrainers=15) to fit 10 trainers. + # minInitialTrainers=10 = wait for all 10 smoke trainers to join. + # aggGoal is set once via the top-level `agg_goal: 3` above -- + # the runner fans it into both this kwarg and + # hyperparameters.aggGoal so they can't drift apart; don't + # duplicate it here. + c: 8 + minInitialTrainers: 10 + + execution: + # 8, not 1, even though this YAML's own default is 10 trainers -- + # run_sequential.sh's --num-trainers commonly scales this up to 100+ + # without a matching GPU bump (it has no num_gpus override of its + # own), and cramming 100 trainer processes onto 1 GPU OOMs almost + # immediately (see fluxtune_n10_smoke run 20260629_221116). Spreading + # across 8 GPUs even at the 10-trainer default is harmless headroom. + num_gpus: 8 + sleep_between_spawns: 2.0 + aggregator_warmup_time: 20 + monitoring: + enabled: false diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml new file mode 100644 index 000000000..a8da9e792 --- /dev/null +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml @@ -0,0 +1,98 @@ +# FedFwd SIMULATED-clock smoke test: 10 trainers, agnews H5 partitions +# (client_idx 0-9 of the 100 available niid_label_clients=100_alpha=1 +# partitions). Sim sibling of fwdllm_n10_smoke.yaml -- IDENTICAL except +# time_mode: simulated and the job id (so run dirs don't collide). +# +# time_mode: simulated exercises the Batch-1 sim path: the trainer skips the +# real emulated-delay sleep and stamps a modeled sim_completion_ts; the +# aggregator orders commits by that sct on a virtual clock (sync barrier here, +# via _sync_sim_recv_first_k). Trainers still do real forward-grad compute, so +# gradient VALUES are mode-invariant -- only the clock/ordering differ. +# +# enable_training_delays stays false (as in the real smoke) so this variant is +# DIRECTLY comparable to fwdllm_n10_smoke.yaml: with D=0 the sct = dispatch + +# real GPU time, which still drives the reorder buffer + vclock advance and +# validates the mechanics. Modeled-delay-on (D>0, the full sim behavior) is for +# the later convergence/parity runs, enabled in BOTH the real reference and the +# sim run together (see simulate_fwdllm.md §K-D8). +# +# Run: +# python -m flame.launch.run_experiment \ +# lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml + +experiments: + - name: fwdllm_n10_smoke_sim + description: "FedFwd SIM smoke (10 trainers, fwdllm baseline: sync random+fedavg, time_mode=simulated). Validates the Batch-1 sim clock/ordering port." + + baseline: fwdllm + + example: + # fwdllm's trainer entrypoint has no pytorch/ subdirectory, unlike the + # ExampleConfig default (trainer/pytorch/main.py). + trainer_main: trainer/main.py + + trainer: + num_trainers: 10 + start_id: 1 + dataset: + name: agnews + # H5 path-style dataset -- no _metadata/dataset_splits/ index-list + # file exists (or ever will) for agnews; skips that lookup entirely. + path_style: true + availability: + mode: syn_0 + # Batch 1: drive the simulated virtual clock instead of real sleeps. + time_mode: simulated + # Skip the registry's 4-18s artificial delay for a fast smoke run. + enable_training_delays: false + # client_idx = (trainer_id - 1) % 100 -- wraps 10 trainers onto a + # distinct, non-colliding subset (0-9) of the 100 H5 partitions. + client_idx_modulo: 100 + config_overrides: + hyperparameters: + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + # Uniform/IID partition for smoke tests -- the niid_label baseline + # default adds a data-skew confound we don't want while validating + # the launcher mechanics. + partition_method: uniform + client_notify: + trace: syn_0 + + aggregator: + config_template: configs/aggregator_base.json + selector: random + tracking_mode: default + # Matches selector.kwargs.c below -- every selected trainer is required. + agg_goal: 10 + log_to_wandb: false + config_overrides: + job: + id: fwdllm_n10_smoke_sim + hyperparameters: + rounds: 50 # short smoke run, not the full 300 + # Smoke-test stop bar: whichever of these fires first ends the run + # (`rounds` alone is much too coarse, since one round is 150 + # data-id completions). + max_data_id_progress: 10 # stop once data_id reaches 10 + max_runtime_s: 1800 # ...or after 30 minutes wall time + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + partition_method: uniform + selector: + kwargs: + # Scaled down from the baseline's n=150-ish defaults (k=5, c=15) + # to fit 10 trainers. + k: 5 + c: 10 + minInitialTrainers: 10 + + execution: + # 8, not 1 -- see fluxtune_n10_smoke.yaml's execution.num_gpus comment; + # run_sequential.sh's --num-trainers scales trainer count without a + # matching GPU bump, and 100 trainers on 1 GPU OOMs almost immediately. + num_gpus: 8 + sleep_between_spawns: 2.0 + aggregator_warmup_time: 20 + monitoring: + enabled: false diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml new file mode 100644 index 000000000..67ab589bb --- /dev/null +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -0,0 +1,94 @@ +# FedFwd SIMULATED-clock smoke test: 10 trainers, agnews H5 partitions +# (client_idx 0-9 of the 100 available niid_label_clients=100_alpha=1 +# partitions). Sim sibling of fwdllm_plus_n10_smoke.yaml -- IDENTICAL except +# time_mode: simulated and the job id (so run dirs don't collide). +# +# fwdllm_plus is SYNC (random selector, fedavg, ORACULAR availability read, +# per-iteration reselection), so this variant exercises the Batch-1 sync +# barrier sim drive (_sync_sim_recv_first_k: commit k-smallest-sct, advance the +# virtual clock to the k-th). Trainers still do real forward-grad compute +# (gradient VALUES mode-invariant); only the clock/ordering differ. +# +# enable_training_delays stays false (as in the real smoke) for direct +# comparability; modeled-delay-on (D>0) is for later parity runs, enabled in +# BOTH real reference and sim run together (see simulate_fwdllm.md §K-D8). +# +# Run: +# python -m flame.launch.run_experiment \ +# lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml + +experiments: + - name: fwdllm_plus_n10_smoke_sim + description: "FedFwd SIM smoke (10 trainers, fwdllm_plus baseline: sync random+fedavg+ORACULAR, time_mode=simulated). Validates the Batch-1 sync barrier sim drive." + + baseline: fwdllm_plus + + example: + # fwdllm's trainer entrypoint has no pytorch/ subdirectory, unlike the + # ExampleConfig default (trainer/pytorch/main.py). + trainer_main: trainer/main.py + + trainer: + num_trainers: 10 + start_id: 1 + dataset: + name: agnews + # H5 path-style dataset -- no _metadata/dataset_splits/ index-list + # file exists (or ever will) for agnews; skips that lookup entirely. + path_style: true + availability: + # ORACULAR reads trainer 1-10's mobiperf_2st trace from _metadata -- + # mode here just keeps the trainer-side avl_events_* fields + # consistent with what the aggregator reads. + mode: mobiperf_2st + # Batch 1: drive the simulated virtual clock instead of real sleeps. + time_mode: simulated + # Skip the registry's 4-18s artificial delay for a fast smoke run. + enable_training_delays: false + # client_idx = (trainer_id - 1) % 100 -- wraps 10 trainers onto a + # distinct, non-colliding subset (0-9) of the 100 H5 partitions. + client_idx_modulo: 100 + config_overrides: + hyperparameters: + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + # Uniform/IID partition for smoke tests -- avoids the niid_label + # baseline default's data-skew confound while validating mechanics. + partition_method: uniform + + aggregator: + config_template: configs/aggregator_base.json + selector: random + tracking_mode: oracular + agg_goal: 2 + log_to_wandb: false + config_overrides: + job: + id: fwdllm_plus_n10_smoke_sim + hyperparameters: + rounds: 50 # short smoke run, not the full 300 + # Smoke-test stop bar: whichever of these fires first ends the run + # (`rounds` alone is much too coarse, since one round is 150 + # data-id completions). + max_data_id_progress: 10 # stop once data_id reaches 10 + max_runtime_s: 1800 # ...or after 30 minutes wall time + data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 + partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 + partition_method: uniform + selector: + kwargs: + # Scaled down from the baseline's n=150-ish defaults (k=5, c=15) + # to fit 10 trainers. + k: 5 + c: 10 + minInitialTrainers: 10 + + execution: + # 8, not 1 -- see fluxtune_n10_smoke.yaml's execution.num_gpus comment; + # run_sequential.sh's --num-trainers scales trainer count without a + # matching GPU bump, and 100 trainers on 1 GPU OOMs almost immediately. + num_gpus: 8 + sleep_between_spawns: 2.0 + aggregator_warmup_time: 20 + monitoring: + enabled: false diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index d9e56a2fe..8c92eb458 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -6,6 +6,19 @@ reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baseli **100% availability (syn_0)** first, then under **unavailability (syn_20, syn_50, mobiperf)**, then **beyond syn_0 traces**. +> **DESIGN PRINCIPLE -- concept-parity with async_cifar10, but deviate where the workload demands it.** +> We deliberately **reuse async_cifar10's concepts and machinery** (virtual clock, sct reorder buffer, +> in-flight gate, availability substrate, the parity ladder) wherever they transfer -- that shared +> substrate is the whole point and keeps the two examples auditable against each other. **But fwdllm is a +> fundamentally different workload** (aggregates GRADIENTS not weights, endogenous variance-gated +> dynamic-K commit cadence, `data_id` progress axis, one-message-per-call grad loop, rollback across +> agg-goal cycles), so a verbatim port is sometimes WRONG. When such a fork appears we **(a) evaluate the +> options carefully and make an explicit decision** (do not silently copy async_cifar10, and do not +> silently invent something new either), **and (b) APPEND that decision + its rationale to this document** +> -- so a later reader can tell an intentional fwdllm-specific divergence from an accidental discrepancy, +> and trace any real<->sim gap back to the choice that caused it. Deviation log lives in **§K** (running, +> newest-last); locked cross-cutting ones also surface in §F "Locked principles" / §J.4. + **Current state (landed on this branch, PRs #63-#69) -- the doc's original "nothing built yet" is stale.** Stage-0 scaffolding is largely done and is now a *validation gate*, not a build: trainer telemetry (`build_trainer_round`: `real_gpu_time_s`, `sim_round_duration_s`, `avail_state`, `stat_utility`, @@ -504,3 +517,267 @@ Add `time_mode: simulated` variants of `expt_scripts/{fluxtune,fwdllm,fwdllm_plu sct; `_barrier_anchored_lags` correct; vclock advances to k-th; no per-message past-dating. 4. **Flag-off byte-identical:** `time_mode: real` path unchanged. 5. Full existing `test_fwdllm_*` green. **Exit rungs:** P3, K6, T2, K1 monotone, K3a/K3b~=0, U5, one-in-flight overlap~=real, U6 barrier lag, commit_gap~=0. + +--- + +## §J Batch 1 EXECUTION PROGRESS (resume point -- uncommitted working tree) + +**Status:** Batch 1 in progress on branch `dg/fwdllm_sim_unavail`, **NOT committed**. Steps 1-2 of the +subtask list (trainer sim path, aggregator dispatch stamping) are **landed in the working tree and +`py_compile`-clean**. Step 3 (async grad loop drive) is **designed in full below but NOT yet written**. +Steps 4-6 (sync barrier, launchers, pytests) untouched. All line numbers below are as of the paused +working tree -- **re-grep before editing** (edits already shifted them from §I's capture). + +**RESUME CHECKLIST (start a new session here -- point at this §J):** +- [x] **1. Trainer sim path** -- landed (J.1(1)). +- [x] **2. Aggregator dispatch stamping** -- landed (J.1(2)). +- [x] **3. Async grad loop drive** -- LANDED (working tree, imports clean in `dg_flame`). `_sim_recv_min_grad` + + `_aggregate_grads_async` mode-branch + `_release_sim_slots_at_agg_goal` boundary cleanup. See §J.2 + (design) + J.1(3) below (as-landed sites). +- [x] **4. Sync barrier** -- LANDED (working tree, imports clean in `dg_flame`). `_sync_sim_recv_first_k` + sim-drive + `_barrier_anchored_lags` stash in `sync_collect_and_accumulate_grads`. See J.1(4). **U6 + full telemetry emission DEFERRED -- §K-D7.** +- [x] **5. Launchers** -- LANDED. `{fwdllm,fluxtune,fwdllm_plus}_n10_smoke_sim.yaml` created (yaml-parse + clean): identical to the real smokes except `time_mode: simulated` + `_sim` job id. `enable_training_delays` + kept false for direct comparability -- see §K-D8. +- [ ] **6. Pytest gate** -- trainer stamping + no-sleep, sct reorder buffer incl. rollback, sync barrier; + full `test_fwdllm_*` green (§I.8 + §J.3). Pytest-only (no broker in this env). + +### J.1 LANDED (in working tree, uncommitted, compiles) + +**(1) Trainer sim path -- DONE.** +- `examples/fwdllm/trainer/main.py:76` -- replaced the "no simulated-clock support" warning block with + `config.hyperparameters.time_mode = _cli_args.time_mode` (Hyperparameters is `extra=allow`, verified + attr-set works). So the trainer reads time_mode from config uniformly with the aggregator (runner injects + the agg's at `launch/runner.py:128`). **Deviation from §I.2:** default is `"real"` via + `getattr(..., "time_mode", "real")` (NOT "simulated") -- conservative for fwdllm's all-real existing + configs; the launcher/CLI default is also "real". Sim variants set `time_mode: simulated` explicitly. +- `FedSgdTrainer.__init__:184-208` -- added `self.time_mode`/`self.simulated`/`self.sim_completion_leg_s` + + inited `_sim_send_ts`/`_sim_completion_ts`/`_sim_round_duration_s`/`_wall_recv_ts = None`. +- `FedSgdTrainer._emulate_training_delay:500` (was :481) -- fixed the `== "True"` string-gate bug + (`_enabled = self.training_delay_enabled in (True, "True", "true")` at :516); when `self.simulated` DO + NOT `time.sleep` but STILL return modeled D. Real mode unchanged (still sleeps D). +- `FedSgdTrainer.train_with_data_id:560-572` -- ADDITIVE stamp (matches §I CRITICAL CORRECTION, NOT + cifar's max(gpu,D)): `self._sim_round_duration_s = _real_gpu_time_s + _delay_s`; when simulated, + `self._sim_completion_ts = (_sim_send_ts or time.time()) + _sim_round_duration_s + sim_completion_leg_s`. +- `fwdllm_trainer._fetch_weights:210-217` -- reads `self._sim_send_ts = msg.get(SIM_SEND_TS)` + + `self._wall_recv_ts = time.time()` on model receipt. +- `fwdllm_trainer._send_grads:530-556` -- train msg now sends `SIM_COMPLETION_TS`, + `SIM_CLIENT_TASK_TRAIN_DURATION_S`, `TRAINING_BUDGET_S` (all `= _sim_round_duration_s`; additive model + has no separate budget), `WALL_SEND_TS=time.time()`, `WALL_RECV_TS=_wall_recv_ts`. Eval msg sends + `SIM_COMPLETION_TS` + `WALL_*`. **D4 RESOLVED:** the trainer loop is + `task_get >> train_with_data_id >> put(_send_grads)` every iteration + (`fwdllm_trainer.py:817-828`), so `_sim_completion_ts` is ALWAYS fresh from the same iteration; fwdllm + eval lives on the AGGREGATOR (`eval_model`), the trainer eval msg is only a utility report -> **eval + reuses the same-iteration train sct, no distinct per-eval sct needed** (D4 collapses to "train sct only"). + +**(2) Aggregator dispatch stamping -- DONE.** +- `fwdllm_aggregator.py:64` -- `from flame.selector.properties import PROP_SIM_SEND_TS` (the oort import + block does NOT export it; `PROP_SIM_SEND_TS="sim_send_ts"`, `PROP_SIM_COMPLETION_TS="sim_completion_ts"`). +- `_distribute_weights_sync:1850` -- before the per-end loop: `_round_now = self._vclock.now` (sim); + inject `SIM_SEND_TS=_round_now` into both `payload_with_weights`/`payload_without_weights` (same instant + for the whole sync barrier); inside the loop `channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now)`. + No gate (`_sim_inflight_expected`) on the sync path. +- `_distribute_weights_async:1993, 2039-2041` -- same stamp into `payload_weights`/`payload_var_bad`; + inside the loop ALSO arms the gate: `_budget = self._sim_trainer_budget.get(end, self._sim_budget_min)` + then `self._sim_inflight_expected[end] = _round_now + _budget`. +- All sim attrs (`_vclock`, `simulated`, `_sim_buffer`, `_sim_committed`, `_sim_inflight_expected`, + `_sim_trainer_budget`, `_sim_budget_min=12.0`) confirmed inited via `internal_init` chain + (`fwdllm_aggregator.internal_init:239` -> `super().internal_init()` -> asyncfl `internal_init:83`). + +**(3) Async grad loop drive -- DONE** (working tree; imports clean in `dg_flame`, not yet pytest-run). +- `fwdllm_aggregator.py:50` -- import `_SIM_GATE_MAX_PASSES, _SIM_ORDER_SLACK_S` from the asyncfl module. +- `_sim_recv_min_grad:696` -- new method, body per §J.2 (ingest ready msgs into `_sim_buffer` keyed by + SIM_COMPLETION_TS -> in-flight gate -> pop min-sct -> clock-jump-clamp `_advance_sim_clock` -> learn + budget -> mark `_sim_committed`). Returns ONE `(msg, md)` or `(None, ("", now))`. Does NOT touch selector + slots (all slot/buffer/committed clearing is boundary-only). +- `_release_sim_slots_at_agg_goal:788` -- new boundary helper: clears `_sim_committed` + `_sim_buffer` + + `_sim_inflight_expected`; async-only `_sim_hold_busy_slots(channel)` (empty held set => releases slots). +- `_aggregate_grads_async:840-846` -- sim branch calls `_sim_recv_min_grad(channel, ends(VAL_CH_STATE_RECV))`; + real branch unchanged (`next(recv_fifo(...,1))`). +- `_process_aggregation_goal_met:1441-1442` -- after `channel.cleanup_recvd_ends():1434`, when simulated, + calls `_release_sim_slots_at_agg_goal(channel, is_async)` (reached by BOTH variance-PASS + FAIL branches). + +**(4) Sync barrier drive -- DONE** (working tree; imports clean in `dg_flame`, not yet pytest-run). +- `sync_collect_and_accumulate_grads:1448` -- sim branch (`:1479`) calls + `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` (base at + `syncfl/top_aggregator.py:360`: commits k-smallest-sct, advances vclock to k-th, stamps + PROP_CLIENT_TASK_TRAIN_DURATION), then feeds each `(msg, md)` through `_process_single_trainer_message`, + breaking at `_agg_goal_cnt >= _agg_goal`. Real branch moved into the `else:` -- byte-identical. +- `:1502` -- stashes `self._sync_barrier_lags_s = self._barrier_anchored_lags(_barrier_durs)` where + `_barrier_durs` = each committed msg's `SIM_CLIENT_TASK_TRAIN_DURATION_S` (dispatch-relative completion, + sim's analog of WALL_SEND - dispatch). **U6 emission deferred (§K-D7): fwdllm's sync path has no + `_round_update_values`/visibility-lag telemetry struct like the base -- the lag is computed + logged + (`[SYNC_SIM_BARRIER]`) and stashed on the instance, but not yet emitted as a telemetry event.** + +### J.2 Async grad loop drive -- design (LANDED, see J.1(3) for as-landed sites) + +Add a new method `_sim_recv_min_grad(self, channel, recv_ends)` on the fwdllm aggregator (place near +`_aggregate_grads_async:694`). It is a **purpose-built** grad analog of `asyncfl._sim_recv_min` (do NOT +call `_sim_recv_min` verbatim -- it does per-commit slot release via `_sim_pending_commit`/`sel.all_selected` +at asyncfl:615-627, but fwdllm must release on the AGG-GOAL boundary; and its withheld/staggered paths key +on WEIGHTS semantics). Needs module constants `_SIM_GATE_MAX_PASSES=64`, `_SIM_ORDER_SLACK_S=2.0` -- +**import them from `flame.mode.horizontal.asyncfl.top_aggregator`** (they are module-level there; not yet +imported into fwdllm_aggregator). `RECV_TIMEOUT_WAIT_S` already imported (:48), `datetime` (:23), `time` (:22). + +Method body (ingest -> gate -> pop -> advance -> budget -> mark committed; returns ONE `(msg, md)` matching +the shape of `next(channel.recv_fifo(...,1))`, or `(None, ("", datetime.now()))`): +```python +live = [e for e in (recv_ends or []) if channel.has(e)] +deadline = time.time() + RECV_TIMEOUT_WAIT_S +for _pass in range(_SIM_GATE_MAX_PASSES): + grace = self._sim_recv_grace_s() + to_probe = [e for e in set(live) | set(self._sim_inflight_expected) + if channel.has(e) and not self._sim_buffer.has(e) and e not in self._sim_committed] + if to_probe: + for m, md in channel.recv_fifo(to_probe, first_k=len(to_probe), timeout=grace): + if m is None: break + _e = md[0]; _s = m.get(MessageType.SIM_COMPLETION_TS) + self._sim_buffer.add(_e, float(_s) if _s is not None else self._vclock.now, (m, md)) + bmin = self._sim_buffer.peek_min_ts() + min_stuck = None + for e, exp in self._sim_inflight_expected.items(): + if self._sim_buffer.has(e) or e in self._sim_committed: continue + if min_stuck is None or exp < min_stuck: min_stuck = exp + earlier_stuck = (bmin is not None and min_stuck is not None and min_stuck + _SIM_ORDER_SLACK_S < bmin) + if bmin is None and not to_probe: break + if not earlier_stuck: break + if time.time() >= deadline: break +popped = self._sim_buffer.pop_min() +if popped is None: return None, ("", datetime.now()) +_end, sct, (m, md) = popped +# clock-jump clamp (don't lap a fresh in-flight cohort): +_now = self._vclock.now; _min_future = None +for e, exp in self._sim_inflight_expected.items(): + if e == _end or e in self._sim_committed: continue + if exp > _now and (_min_future is None or exp < _min_future): _min_future = exp +_advance_to = sct if _min_future is None else max(_now, min(sct, _min_future + _SIM_ORDER_SLACK_S)) +self._advance_sim_clock(_advance_to) +self._sim_committed.add(_end); self._sim_inflight_expected.pop(_end, None) +_b = m.get(MessageType.TRAINING_BUDGET_S) if isinstance(m, dict) else None +if _b is not None: + self._sim_trainer_budget[_end] = float(_b); self._sim_budget_min = min(self._sim_budget_min, float(_b)) +return m, md +``` +Then in `_aggregate_grads_async` REPLACE the unconditional `next(channel.recv_fifo(...,1,...))` at **:720** +with a mode branch: +```python +if self.simulated: + msg, metadata = self._sim_recv_min_grad(channel, channel.ends(VAL_CH_STATE_RECV)) +else: + msg, metadata = next(channel.recv_fifo(channel.ends(VAL_CH_STATE_RECV), 1, timeout=RECV_TIMEOUT_WAIT_S)) +end, timestamp = metadata +``` +(keep the existing `if not msg: return` guard right after). + +**Agg-goal-boundary cleanup (rollback-safety, §I.5).** Add a helper and call it from +`_process_aggregation_goal_met` right AFTER `channel.cleanup_recvd_ends()` (**:1309**) -- this point is +reached by BOTH the variance-PASS and variance-FAIL(rollback) branches (both fall through past the +`_per_agg_trainer_list = []` clear at :1272): +```python +def _release_sim_slots_at_agg_goal(self, channel, is_async): + if not self.simulated: return + self._sim_committed.clear() # else next cycle's gate skips a re-contributing end (rollback hole) + self._sim_buffer.clear() # drop stranded arrived-but-uncommitted grads so next cycle is clean + self._sim_inflight_expected.clear() + if is_async: + self._sim_hold_busy_slots(channel) # empty held set -> releases every committed end's slot +``` +call site: `if self.simulated: self._release_sim_slots_at_agg_goal(channel, is_async)` after :1309. +(This is why fwdllm's own `_reset_agg_goal_variables:450` NOT clearing `_sim_committed` -- the §I.6 ROLLBACK +HOLE -- is handled: we clear it explicitly at the real boundary instead.) `_sim_hold_busy_slots` +(asyncfl:1453) needs `channel._selector` with `.requester/.all_selected/.selected_ends` -- provide in the +fake-channel test. + +### J.3 REMAINING (untouched) + +- **Sync barrier** (subtask 4): in `sync_collect_and_accumulate_grads:1315`, replace the drain loop + `for msg, metadata in channel.recv_fifo(channel.ends(), num_min_req, timeout=...)` (**:1337**) with, in + the sim branch, `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` (base at + `syncfl/top_aggregator.py:360`, returns ascending-sct list of `(msg, md)`, already stamps + PROP_CLIENT_TASK_TRAIN_DURATION + advances vclock), then feed each through + `_process_single_trainer_message`, breaking at `_agg_goal_cnt >= _agg_goal`. Apply + `_barrier_anchored_lags(durs)` (`syncfl:472`) over the committed WALL_SEND completions for U6 telemetry. + Real branch unchanged. +- **Launchers** (subtask 5): add `time_mode: simulated` sibling yamls of + `expt_scripts/{fluxtune,fwdllm,fwdllm_plus}_n10_smoke.yaml` (current set `time_mode: real` at + fluxtune:40 / fwdllm:35 / fwdllm_plus:39). Keep the real ones for reference runs. +- **Pytests** (subtask 6): see §I.8. No broker in this env -> pytest-only, drive `_sim_recv_min_grad` / + `_sync_sim_recv_first_k` with a fake channel + synthetic out-of-order msgs incl. a variance-FAIL rollback + cycle. Extend `tests/mode/test_fwdllm_trainer_sim_duration.py` for the trainer stamping + no-sleep assert. + +### J.4 Open decisions taken during execution (record) +- time_mode default = `"real"` on the trainer getattr fallback (see J.1(1)); revisit if a sim run without an + explicit yaml value is ever needed. +- `_sim_recv_min_grad` deliberately does NOT touch selector slots per-commit; ALL slot release + buffer + + committed-mark clearing happens at the agg-goal boundary (`_release_sim_slots_at_agg_goal`). This is the + fwdllm-specific divergence from asyncfl's per-commit release and the crux of rollback-safety. +- `_sim_buffer.clear()` at the boundary DROPS any stranded arrived-but-uncommitted grad. Benign at syn_0 + (|selected| ~= agg_goal, all commit). If a future rung shows lost updates, revisit (commit-then-carry + instead of drop). Flagged as a fidelity watch-point. + +--- + +## §K DEVIATION LOG -- fwdllm-specific divergences from async_cifar10 (running, newest-last) + +Per the DESIGN PRINCIPLE at the top: every place we chose NOT to copy async_cifar10 verbatim (or chose to +copy it despite a workload difference) is logged here with the options weighed + why. One entry per +decision. Keep appending; do not rewrite history (supersede with a new dated entry instead). + +- **K-D1 time_mode default = `"real"` (trainer getattr fallback).** async_cifar10 defaults `time_mode` to + `"simulated"` (`trainer/pytorch/main.py:174`). **Options:** (a) mirror "simulated" default; (b) default + "real". **Chose (b):** fwdllm's entire existing config corpus is `time_mode: real` and FedFwd shipped + with no sim path at all, so a "simulated" default risks silently flipping a manual/legacy config into a + half-built sim path. Sim variants set `time_mode: simulated` explicitly. **Rationale/where:** J.1(1), + `FedSgdTrainer.__init__` `getattr(..., "time_mode", "real")`. Revisit if a sim run without an explicit + yaml value is ever wanted. +- **K-D2 Additive delay model `sim_round_duration = gpu + D`, NOT cifar's `max(gpu, D)`.** cifar "sleeps + to fill a budget" (wall = max(gpu, D)); fwdllm's real mode sleeps D *on top of* GPU time (flat additive, + `_emulate_training_delay` docstring + existing `sim_round_duration_s = real_gpu + delay`). **Chose + additive** so sim sct matches what real wall-clock actually does for THIS workload -- copying cifar's + max() would desync real<->sim. **Where:** §I CRITICAL CORRECTION, J.1(1) `train_with_data_id`. (This is a + workload fact, not a preference -- logged so the sct formula difference is traceable.) +- **K-D3 Per-eval sct collapses to the same-iteration train sct (D4).** cifar stamps a DISTINCT eval sct + = `send + max(gpu, D_eval)` with a ~20x eval speedup (`main.py:1086-1096`). **Options:** (a) port a + distinct per-eval sct; (b) reuse the train sct. **Chose (b):** in fwdllm eval lives on the AGGREGATOR + (`eval_model` on the global model); the trainer loop runs `train_with_data_id >> _send_grads` every + iteration so `_sim_completion_ts` is always fresh, and the trainer's eval message is only a utility + report, not a separately-clocked commit. No 20x factor (forward-grad "train" IS a forward pass). **Where:** + D4, J.1(1). Revisit only if a trainer-side distinct eval task is added. +- **K-D4 Grad loop uses a purpose-built `_sim_recv_min_grad`, NOT asyncfl `_sim_recv_min` verbatim.** + **Options:** (a) call `_sim_recv_min` directly (it already handles ingest/gate/advance); (b) write a + grad-specific drain reusing the same primitives (`SimReorderBuffer`, `_advance_sim_clock`, + `_sim_inflight_expected`). **Chose (b):** `_sim_recv_min` does per-commit slot release via + `_sim_pending_commit`/`sel.all_selected` (asyncfl:615-627) and its withheld/staggered paths key on WEIGHTS + semantics -- both wrong for fwdllm, which must release slots on the AGG-GOAL boundary and commits grads. + Reused the primitives, forked the orchestration. **Where:** §J.2, J.1(3). +- **K-D5 Slot release + committed/buffer clear on the AGG-GOAL boundary, not per-commit.** async_cifar10 + releases a slot the moment its update commits (inside `_sim_recv_min`). **Chose boundary release** + (`_release_sim_slots_at_agg_goal`) because fwdllm commits one grad per call and a `data_id` spans many + agg-goal cycles with variance-FAIL rollbacks; per-commit release would strand/re-skip a re-contributing + trainer across a rollback (the §I.6 rollback hole). **Where:** §I.5, §J.2, J.1(3). Locked principle #4. +- **K-D6 `_sim_buffer.clear()` at the boundary DROPS stranded arrived-but-uncommitted grads.** async path + in cifar carries buffered futures across the barrier. **Chose drop** for rollback-cleanliness (a stranded + grad was trained on a pre-rollback model_version -> stale next cycle anyway). Benign at syn_0 + (|selected| ~= agg_goal). **OPEN watch-point:** if a rung shows lost updates, switch to commit-then-carry. + **Where:** J.4, J.2. +- **K-D7 U6 sync-barrier visibility-lag telemetry DEFERRED (compute+stash now, emit later).** The base + syncfl `_aggregate_weights` finalizes `update_visibility_lag_s` from `_barrier_anchored_lags(_real_round_durs)` + into a `_round_update_values` dict it then emits (`syncfl/top_aggregator.py:742-786`). fwdllm's + `sync_collect_and_accumulate_grads` has NO such per-round telemetry struct. **Options:** (a) port the whole + `_round_update_values` visibility-lag telemetry plumbing into fwdllm's sync path now; (b) compute the + barrier lags now, stash on `self._sync_barrier_lags_s` + log, defer the telemetry-event emission. + **Chose (b)** to keep Batch 1 the clock-drive port, not a telemetry port -- the value is COMPUTED and + available (so the U6 exit-rung wiring in the pytest/parity step can read it), just not yet emitted as an + event. **Where:** J.1(4). **TODO before the Batch-1 U6 exit-rung check:** either emit `_sync_barrier_lags_s` + via the telemetry manifest or have `parity_checks` read the stashed value. +- **K-D8 sim smoke launchers keep `enable_training_delays: false` (D=0).** The sim variants + (`*_n10_smoke_sim.yaml`) mirror the real smokes exactly except `time_mode: simulated` + `_sim` job id. + **Options:** (a) enable modeled delays (D>0) in the sim smoke so the vclock advances by the modeled + budget (the "interesting" sim behavior); (b) keep D=0 as in the real smoke. **Chose (b)** for step 5: + with D=0 the sct = dispatch + real GPU time, which STILL drives the reorder buffer + vclock advance + + sct ordering -- enough to validate the Batch-1 MECHANICS (does the sim path run/advance/not crash), and + it keeps the sim smoke byte-for-byte comparable to its real sibling. Modeled-delay-on (D>0) belongs to + the later convergence/parity runs and MUST be enabled in BOTH the real reference and the sim run together + (else real sleeps D while sim charges D to the clock -> that's the point; mismatched enable would be a + false divergence). **Where:** the three `*_n10_smoke_sim.yaml` headers. diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index f64e9faf4..826bdfd70 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -181,6 +181,25 @@ def __init__( ) self.speedup_factor = 1.0 + # --- Simulated-clock support (Batch 1, config-gated) ------------------ + # time_mode is threaded into hyperparameters from the launcher's + # --time_mode CLI arg (see trainer/main.py). "simulated": _emulate_ + # training_delay() computes the modeled delay but does NOT sleep, and + # train_with_data_id() stamps a modeled completion timestamp + # (_sim_completion_ts) the aggregator orders updates by. "real" + # (default): unchanged wall-clock behavior (flag-off => byte-identical). + self.time_mode = getattr(self.config.hyperparameters, "time_mode", "real") + self.simulated = self.time_mode == "simulated" + _leg = getattr(self.config.hyperparameters, "sim_completion_leg_s", 0.0) + self.sim_completion_leg_s = float(_leg) if _leg is not None else 0.0 + # SIM_SEND_TS is stamped by the aggregator on each dispatch and read in + # the base trainer's _fetch_weights; the rest are stamped after training + # for _send_grads / telemetry / the intrinsic-duration selector signal. + self._sim_send_ts = None + self._sim_completion_ts = None + self._sim_round_duration_s = None + self._wall_recv_ts = None + self.trainer_start_ts = time.time() # TODO (ARM): Fix this to read traces better! # Storing synthetic avail traces @@ -479,23 +498,39 @@ def _perform_training(self): @timer_decorator def _emulate_training_delay(self): - """Returns the seconds actually slept (0.0 if delay emulation is - disabled) -- unlike cifar10's trainer, this is a flat additive sleep - on top of GPU time, not a budget-minus-actual "sleep to fill" model, - so there is no meaningful overrun/remaining_time_s/training_budget_s - concept here (see ../../../MIGRATING_TO_LAUNCHER.md §9). The - caller adds this to real_gpu_time_s to report sim_round_duration_s.""" - if self.training_delay_enabled == "True": + """Returns the modeled emulated-delay seconds D (0.0 if delay emulation + is disabled) -- unlike cifar10's trainer, this is a flat additive delay + on top of GPU time, not a budget-minus-actual "sleep to fill" model, so + there is no meaningful overrun/remaining_time_s/training_budget_s concept + here (see ../../../MIGRATING_TO_LAUNCHER.md §9). The caller adds this to + real_gpu_time_s to report sim_round_duration_s. + + In real mode the delay is realized by an actual time.sleep(D); in + simulated mode the sleep is SKIPPED (the aggregator advances a virtual + clock instead) but the same D is still RETURNED so the additive + sim_round_duration_s = real_gpu_time_s + D stays identical across modes. + """ + # config schema types training_delay_enabled as bool (default False) + # but historical launcher yamls pass the string "True"; accept both so + # the modeled delay is not silently dropped to 0. + _enabled = self.training_delay_enabled in (True, "True", "true") + if _enabled: # Eval is 3X faster than training on CPU # Eval on NPUs is 10-50X is faster than training on CPUs. We could take 20X if we wanted to consider an all-NPU client cohort for Eval (NPUs don't support training) eval_delay = self.training_delay_s / self.training_delay_factor - _sleep_s = eval_delay / self.speedup_factor - time.sleep(_sleep_s) - logger.info( - f"Delayed eval time for trainer " - f"{self.trainer_id} by {eval_delay}s. Sleeping for {_sleep_s}s." - ) - return _sleep_s + _delay_s = eval_delay / self.speedup_factor + if self.simulated: + logger.info( + f"time_mode=simulated: modeled eval delay for trainer " + f"{self.trainer_id} = {_delay_s}s (not slept; charged to vclock)." + ) + else: + time.sleep(_delay_s) + logger.info( + f"Delayed eval time for trainer " + f"{self.trainer_id} by {eval_delay}s. Sleeping for {_delay_s}s." + ) + return _delay_s return 0.0 @timer_decorator @@ -522,6 +557,20 @@ def train_with_data_id(self): # dataset size and/or network latency) _delay_s = self._emulate_training_delay() + # Sim-mode stamps (Batch 1): the modeled round duration is ADDITIVE + # (real_gpu + D), matching real mode's sleep-D-on-top-of-GPU semantics + # -- NOT cifar10's max(gpu, D) "sleep to fill a budget" model. The sct + # (when this update COMMITS on the virtual clock) is the aggregator's + # dispatch stamp (SIM_SEND_TS, read in _fetch_weights) + that duration + + # the optional pre-commit holding leg. _send_grads sends these so the + # aggregator can order updates by _sim_completion_ts. In real mode these + # stay None and the aggregator falls back to arrival order (unchanged). + self._sim_round_duration_s = _real_gpu_time_s + _delay_s + if self.simulated: + _leg = self.sim_completion_leg_s + _base = self._sim_send_ts if self._sim_send_ts is not None else time.time() + self._sim_completion_ts = _base + self._sim_round_duration_s + _leg + logger.info( f"completed training for trainer id: {self.trainer_id}, data_id = {self.data_id}" ) diff --git a/lib/python/examples/fwdllm/trainer/main.py b/lib/python/examples/fwdllm/trainer/main.py index bef631a79..b821fb4fb 100644 --- a/lib/python/examples/fwdllm/trainer/main.py +++ b/lib/python/examples/fwdllm/trainer/main.py @@ -53,12 +53,13 @@ def post_complete_message(tc_args): config = load_config_from_argv() # --time_mode is a launcher CLI-only arg (not in config JSON), unconditionally - # appended by TrainerSpawner.spawn_trainer() to every trainer's argv. FedFwd has - # no simulated-clock concept (FedSgdTrainer's _emulate_training_delay() always - # sleeps real wall-clock time) -- parsed here only so the launcher's argv - # injection doesn't crash with "unrecognized argument"; "simulated" is a - # documented no-op for now, deferred future work, not retrofitted in this - # migration. log_level is similarly launcher/manual-run CLI-only. + # appended by TrainerSpawner.spawn_trainer() to every trainer's argv. It is + # threaded into config.hyperparameters (extra-allowed) so FedSGDTrainer reads + # it uniformly with the aggregator (which the launcher injects into its own + # hyperparameters). "simulated": skip the real emulated-delay sleep and stamp + # a modeled completion timestamp the aggregator orders updates by; "real" + # (default): unchanged wall-clock behavior. log_level/battery_threshold are + # similarly launcher/manual-run CLI-only. _cli_parser = argparse.ArgumentParser(add_help=False) _cli_parser.add_argument("--time_mode", default="real") _cli_parser.add_argument("--log_level", default="INFO") @@ -72,11 +73,7 @@ def post_complete_message(tc_args): ) logging.debug(config) telemetry.configure(role="trainer", end_id=str(config.task_id)) - if _cli_args.time_mode != "real": - logging.warning( - f"--time_mode={_cli_args.time_mode!r} requested, but FedFwd has no " - "simulated-clock support yet; running in real wall-clock mode." - ) + config.hyperparameters.time_mode = _cli_args.time_mode set_seed(config.hyperparameters.manual_seed) # dataset attributes diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 8423004d2..e9c81cb7a 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -47,6 +47,8 @@ from flame.mode.horizontal.asyncfl.top_aggregator import ( RECV_TIMEOUT_WAIT_S, TopAggregator as AsyncTopAgg, + _SIM_GATE_MAX_PASSES, + _SIM_ORDER_SLACK_S, ) from flame.mode.message import MessageType from flame.mode.horizontal.client_duration import real_client_task_train_duration @@ -61,6 +63,7 @@ PROP_STAT_UTILITY, PROP_UPDATE_COUNT, ) +from flame.selector.properties import PROP_SIM_SEND_TS import functorch as fc import torch @@ -690,6 +693,118 @@ def aggregate_grad_pool(self, grad_list): self.print_trainable_params_stats(location="[end,aggregate_grad_pool()]") return grad + def _sim_recv_min_grad(self, channel, recv_ends): + """Grad-loop analog of asyncfl._sim_recv_min (simulate_fwdllm.md §J.2). + + Ingest every ready grad message into the sct-ordered reorder buffer + (keyed by SIM_COMPLETION_TS), HOLD the commit while an in-flight trainer + is EXPECTED (_sim_inflight_expected) to complete before the buffered + minimum, then pop and commit the single smallest-sct message, advancing + the virtual clock to it. Returns one (msg, metadata) -- matching the + shape of next(channel.recv_fifo(...,1)) -- or (None, ("", now)) when + nothing is committable. + + Purpose-built rather than calling _sim_recv_min verbatim because fwdllm + commits GRADIENTS one-per-call and releases concurrency slots on the + agg-goal boundary (_release_sim_slots_at_agg_goal), NOT per message, and + must survive variance-FAIL rollbacks without stranding or double- + committing a grad (§I.5). Slot/selector state is deliberately untouched + here -- all of it is cleared at the boundary. + """ + live = [e for e in (recv_ends or []) if channel.has(e)] + deadline = time.time() + RECV_TIMEOUT_WAIT_S + for _pass in range(_SIM_GATE_MAX_PASSES): + grace = self._sim_recv_grace_s() + # Probe live recv ends UNION the tracked in-flight set (an end that + # entered RECV after the snapshot is still drained), minus anything + # already buffered or committed this cycle. + to_probe = [ + e for e in set(live) | set(self._sim_inflight_expected) + if channel.has(e) + and not self._sim_buffer.has(e) + and e not in self._sim_committed + ] + if to_probe: + for m, md in channel.recv_fifo( + to_probe, first_k=len(to_probe), timeout=grace + ): + if m is None: # no more ready (grace expired or set drained) + break + _e = md[0] + _s = m.get(MessageType.SIM_COMPLETION_TS) + _s = float(_s) if _s is not None else self._vclock.now + self._sim_buffer.add(_e, _s, (m, md)) + # Gate: earliest expected completion among un-buffered in-flight ends. + bmin = self._sim_buffer.peek_min_ts() + min_stuck = None + for e, exp in self._sim_inflight_expected.items(): + if self._sim_buffer.has(e) or e in self._sim_committed: + continue + if min_stuck is None or exp < min_stuck: + min_stuck = exp + earlier_stuck = ( + bmin is not None and min_stuck is not None + and min_stuck + _SIM_ORDER_SLACK_S < bmin + ) + if bmin is None and not to_probe: + break # nothing to commit and nothing in flight + if not earlier_stuck: + break # the buffered minimum is the true next completion + if time.time() >= deadline: + break # failsafe: a stuck trainer never arrived; commit buffered + popped = self._sim_buffer.pop_min() + if popped is None: + return None, ("", datetime.now()) + _end, sct, (m, md) = popped + # Clock-jump clamp: a far-future straggler commit must not lap a fresh + # in-flight cohort still expected to complete earlier. + _now = self._vclock.now + _min_future = None + for e, exp in self._sim_inflight_expected.items(): + if e == _end or e in self._sim_committed: + continue + if exp > _now and (_min_future is None or exp < _min_future): + _min_future = exp + _advance_to = ( + sct if _min_future is None + else max(_now, min(sct, _min_future + _SIM_ORDER_SLACK_S)) + ) + self._advance_sim_clock(_advance_to) + self._sim_committed.add(_end) + self._sim_inflight_expected.pop(_end, None) + # Learn this end's MODELED budget (contention-free lower bound) so the + # gate fires on genuine stragglers for not-yet-observed trainers. + _b = m.get(MessageType.TRAINING_BUDGET_S) if isinstance(m, dict) else None + if _b is not None: + self._sim_trainer_budget[_end] = float(_b) + self._sim_budget_min = min(self._sim_budget_min, float(_b)) + logger.info( + f"[SIM_GRAD_RECV] round={getattr(self, '_round', -1)} " + f"end={str(_end)[-4:]} sct={sct:.1f} T_v={self._vclock.now:.1f} " + f"buf_depth={len(self._sim_buffer)}" + ) + return m, md + + def _release_sim_slots_at_agg_goal(self, channel, is_async): + """Sim rollback-safety at the agg-goal boundary (simulate_fwdllm.md §I.5). + + A data_id spans MANY agg-goal cycles (a variance-FAIL retries the same + data_id with iteration_per_data_id += 1). Clear the per-cycle committed + marks and drain any stranded reorder-buffer entries so the next cycle + starts clean -- otherwise the gate would skip a trainer re-contributing + on the rolled-back data_id (still flagged committed), and a stranded + buffered grad would commit past-dated into the next cycle. For the async + path, holding the now-empty busy set frees every committed trainer's + concurrency slot for re-selection (one-in-flight per cycle). + """ + if not self.simulated: + return + self._sim_committed.clear() + self._sim_buffer.clear() + self._sim_inflight_expected.clear() + if is_async: + self._sim_hold_busy_slots(channel) + def _aggregate_grads_async(self, tag: str) -> None: """ Aggregate local model GRADIENTS asynchronously for FwdLLM. @@ -716,10 +831,21 @@ def _aggregate_grads_async(self, tag: str) -> None: # to run -- the run hangs past its configured budget until manually # killed. Matches the same pattern asyncfl/top_aggregator.py's # _aggregate_weights already uses for this exact reason. - msg, metadata = next( - channel.recv_fifo(channel.ends(VAL_CH_STATE_RECV), 1, - timeout=RECV_TIMEOUT_WAIT_S) - ) + # + # Sim branch (Batch 1): instead of committing on wall arrival, order the + # commit by the modeled completion sct via _sim_recv_min_grad (sct-ordered + # reorder buffer + in-flight gate + virtual-clock advance). It returns ONE + # (msg, metadata) matching next(recv_fifo(...,1)) -- one grad per call, as + # this loop expects. Real branch byte-identical to before. + if self.simulated: + msg, metadata = self._sim_recv_min_grad( + channel, channel.ends(VAL_CH_STATE_RECV) + ) + else: + msg, metadata = next( + channel.recv_fifo(channel.ends(VAL_CH_STATE_RECV), 1, + timeout=RECV_TIMEOUT_WAIT_S) + ) end, timestamp = metadata if not msg: logger.debug(f"No data from {end}; skipping it") @@ -1307,6 +1433,14 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): ) channel.cleanup_recvd_ends() + # Sim rollback-safety: clear this cycle's committed marks + stranded + # reorder-buffer entries + in-flight gate, and (async) release held + # concurrency slots -- so the next agg-goal cycle of a rolled-back + # data_id starts clean (see §I.5/§J.2). Reached by BOTH the variance- + # PASS and variance-FAIL branches. Inert in real mode. + if self.simulated: + self._release_sim_slots_at_agg_goal(channel, is_async) + # Centralized cleanup # self._force_cuda_memory_cleanup() @@ -1333,20 +1467,58 @@ def sync_collect_and_accumulate_grads(self, tag, channel): # re-check _check_early_stop_conditions() (max_runtime_s/ # max_data_id_progress), and the run hangs past its configured # budget until manually killed. Same fix as _aggregate_grads_async. - for msg, metadata in channel.recv_fifo(channel.ends(), num_min_req, - timeout=RECV_TIMEOUT_WAIT_S): - end, timestamp = metadata - if not msg: - logger.info(f"No data from {end}; skipping it") - continue + # + # Sim barrier (Batch 1): instead of committing by physical arrival, + # _sync_sim_recv_first_k commits the num_min_req trainers with the + # SMALLEST modeled sim_completion_ts (the k that would physically finish + # first in real), advancing the vclock to the k-th smallest -- immune to + # arrival jitter. It returns an ascending-sct list of (msg, metadata) and + # stamps PROP_CLIENT_TASK_TRAIN_DURATION per commit; we then feed each + # through the SAME per-message path. Real branch byte-identical to before. + if self.simulated: + committed = self._sync_sim_recv_first_k( + channel, channel.ends(), num_min_req + ) + _barrier_durs = [] + for msg, metadata in committed: + end, timestamp = metadata + if not msg: + continue + # dispatch-relative completion for the U6 barrier anchor (sim's + # analog of WALL_SEND - dispatch) = the modeled round duration. + _barrier_durs.append( + msg.get(MessageType.SIM_CLIENT_TASK_TRAIN_DURATION_S) + ) + self._process_single_trainer_message(channel, msg, end, timestamp) + if self._agg_goal_cnt >= self._agg_goal: + logger.info( + f"Reached agg_goal of {self._agg_goal} (sim barrier); " + f"proceeding to aggregate." + ) + break + # U6 visibility lag on a strict sync barrier: lag_i = max_completion + # - completion_i over the committed cohort. Stashed for telemetry / + # the U6 parity rung; full emission is deferred (§K-D7). + self._sync_barrier_lags_s = self._barrier_anchored_lags(_barrier_durs) + logger.info( + f"[SYNC_SIM_BARRIER] round={self._round} committed={len(committed)} " + f"T_v={self._vclock.now:.1f} barrier_lags_s={self._sync_barrier_lags_s}" + ) + else: + for msg, metadata in channel.recv_fifo(channel.ends(), num_min_req, + timeout=RECV_TIMEOUT_WAIT_S): + end, timestamp = metadata + if not msg: + logger.info(f"No data from {end}; skipping it") + continue - self._process_single_trainer_message(channel, msg, end, timestamp) + self._process_single_trainer_message(channel, msg, end, timestamp) - if self._agg_goal_cnt >= self._agg_goal: - logger.info( - f"Reached agg_goal of {self._agg_goal} since agg_goal_count is {self._agg_goal_cnt}. Breaking from for loop, proceeding to aggregate." - ) - break + if self._agg_goal_cnt >= self._agg_goal: + logger.info( + f"Reached agg_goal of {self._agg_goal} since agg_goal_count is {self._agg_goal_cnt}. Breaking from for loop, proceeding to aggregate." + ) + break # Second loop @@ -1842,10 +2014,22 @@ def _distribute_weights_sync( payload_with_weights = self._prepare_distribution_payload(task_to_perform, force_weights=True) if not self.var_good_enough: payload_without_weights = self._prepare_distribution_payload(task_to_perform, force_weights=False) - + self._update_state_after_payload_prepared() + # Sim-clock dispatch stamp (Batch 1): in the sync barrier every selected + # end dispatches at the same virtual instant, so one _round_now stamp is + # injected into each payload variant (the trainer bases its modeled sct + # on SIM_SEND_TS) and recorded as a per-end property. Inert in real mode: + # SIM_SEND_TS is absent -> the trainer stamps nothing -> the aggregator + # falls back to arrival order (byte-identical to today). + _round_now = self._vclock.now if self.simulated else None + if self.simulated: + for _p in (payload_with_weights, payload_without_weights): + if _p is not None: + _p[MessageType.SIM_SEND_TS] = _round_now + for end in ends: trainer_version = self._trainer_last_model_version.get(end, -1) is_stale = (trainer_version != self._model_version) @@ -1866,6 +2050,8 @@ def _distribute_weights_sync( channel.set_end_property( end, PROP_ROUND_START_TIME, (self._round, datetime.now()) ) + if self.simulated: + channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now) if self.var_good_enough: logger.info( @@ -1975,6 +2161,18 @@ def _distribute_weights_async( self._update_state_after_payload_prepared() + # Sim-clock dispatch stamp (Batch 1): all ends in this dispatch share the + # same virtual instant, so one _round_now stamp is injected into each + # payload variant. Unlike the sync barrier the async path also arms the + # in-flight gate (_sim_inflight_expected[end] = dispatch vclock + a + # lower-bound budget) so the reorder-buffer drain can't lap a trainer + # whose modeled completion is still in the future. Inert in real mode. + _round_now = self._vclock.now if self.simulated else None + if self.simulated: + for _p in (payload_weights, payload_var_bad): + if _p is not None: + _p[MessageType.SIM_SEND_TS] = _round_now + _n_weights_sent = 0 _n_var_bad_sent = 0 for end in ends: @@ -2005,6 +2203,13 @@ def _distribute_weights_async( channel.set_end_property( end, PROP_ROUND_START_TIME, (self._round, datetime.now()) ) + if self.simulated: + channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now) + # Expected completion = dispatch vclock + a lower-bound budget + # (this end's own last-observed TRAINING_BUDGET_S, else the + # running min) so the gate never laps a not-yet-committed trainer. + _budget = self._sim_trainer_budget.get(end, self._sim_budget_min) + self._sim_inflight_expected[end] = _round_now + _budget channel.send(end, payload) logger.info( f"[Distribute] Done. Sent {_n_weights_sent} WEIGHTS + " diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index af5e2b537..5d33b08b8 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -207,6 +207,15 @@ def _fetch_weights(self, tag: str) -> None: logger.info(f"New message received for trainer_id {self.trainer_id}") + # Sim-clock stamps (Batch 1): the aggregator stamps SIM_SEND_TS (its + # virtual-clock "now" at dispatch) so the trainer can base its modeled + # completion sct on it; _wall_recv_ts is the real receipt wall time, sent + # back so the aggregator can derive the intrinsic (server-overhead-free) + # task duration (WALL_SEND - WALL_RECV). Both are inert in real mode: + # SIM_SEND_TS is absent (stays None) and the aggregator ignores WALL_*. + self._sim_send_ts = msg.get(MessageType.SIM_SEND_TS) + self._wall_recv_ts = time.time() + if MessageType.ROUND in msg: self._round = msg[MessageType.ROUND] @@ -518,11 +527,31 @@ def _send_grads(self, tag: str) -> None: MessageType.STAT_UTILITY: self._stat_utility, # - rn FedSgdTrainer has no utility MessageType.TOTAL_DATA_BINS: self.total_data_bins, + # Sim-clock stamps (Batch 1): the modeled completion sct the + # aggregator's reorder buffer keys on, the additive modeled round + # duration (= completion budget, used for the async in-flight + # gate), and the real wall send/recv pair the aggregator derives + # the intrinsic (server-overhead-free) task duration from. All + # None in real mode -> aggregator ignores them (arrival order). + MessageType.SIM_COMPLETION_TS: self._sim_completion_ts, + MessageType.SIM_CLIENT_TASK_TRAIN_DURATION_S: self._sim_round_duration_s, + MessageType.TRAINING_BUDGET_S: self._sim_round_duration_s, + MessageType.WALL_SEND_TS: time.time(), + MessageType.WALL_RECV_TS: self._wall_recv_ts, } else: + # D4: fwdllm eval lives on the AGGREGATOR (eval_model on the global + # model after a variance pass); this eval message is only a utility + # report, not a separately-clocked commit. train_with_data_id runs + # immediately before this in the same loop iteration, so its fresh + # _sim_completion_ts is a valid (not past-dated) sct to echo, and the + # WALL_* pair still lets the aggregator derive the intrinsic duration. msg = { MessageType.MODEL_VERSION: self._model_version, MessageType.STAT_UTILITY: self._stat_utility, + MessageType.SIM_COMPLETION_TS: self._sim_completion_ts, + MessageType.WALL_SEND_TS: time.time(), + MessageType.WALL_RECV_TS: self._wall_recv_ts, } channel.send(end, msg) From c13d830e1eea58df0ca2e16b93520432543cec0f Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 00:13:04 -0400 Subject: [PATCH 04/56] fwdllm sim Batch 1: pytest gate (subtask 6) + doc refresh to Batch 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes Phase-1 Batch 1 with its pytest-only gate (75 fwdllm tests green): - restore flag-off byte-identical: set simulated=False on the fakes in four pre-existing fwdllm agg/trainer tests that the landed sim guards broke - trainer: no-sleep-on-sim-path + additive _sim_completion_ts stamp - async grad loop: _sim_recv_min_grad sct-ordered commit, one-in-flight gate, and agg-goal-boundary rollback safety (_release_sim_slots_at_agg_goal) - sync barrier: _sync_sim_recv_first_k first-k-smallest-sct + _barrier_anchored_lags Refresh simulate_fwdllm.md: crisp out landed Batch-1 build detail (now history in the code + deviation log K-D1..D8) and replace it with a §I Batch-2 cold-start map (variance-cadence V/DK/G rungs + fluxtune oort validation), verified against the parity engine and the emitted agg_round telemetry. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 584 +++++------------- .../tests/mode/test_fwdllm_agg_telemetry.py | 3 + .../tests/mode/test_fwdllm_recv_timeout.py | 4 + .../mode/test_fwdllm_rounds_termination.py | 4 + .../tests/mode/test_fwdllm_sim_grad_loop.py | 249 ++++++++ .../mode/test_fwdllm_sim_sync_barrier.py | 150 +++++ .../mode/test_fwdllm_trainer_sim_duration.py | 120 +++- 7 files changed, 674 insertions(+), 440 deletions(-) create mode 100644 lib/python/tests/mode/test_fwdllm_sim_grad_loop.py create mode 100644 lib/python/tests/mode/test_fwdllm_sim_sync_barrier.py diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 8c92eb458..1a04b9805 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -19,18 +19,17 @@ reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baseli > and trace any real<->sim gap back to the choice that caused it. Deviation log lives in **§K** (running, > newest-last); locked cross-cutting ones also surface in §F "Locked principles" / §J.4. -**Current state (landed on this branch, PRs #63-#69) -- the doc's original "nothing built yet" is stale.** -Stage-0 scaffolding is largely done and is now a *validation gate*, not a build: trainer telemetry -(`build_trainer_round`: `real_gpu_time_s`, `sim_round_duration_s`, `avail_state`, `stat_utility`, -`FedSgdTrainer.py:533`), aggregator telemetry (14 tests), the shared parity-check engine + fwdllm checks -(`test_parity_checks.py`, 40 tests), the oracular availability *read* (`read_trainer_unavailability`, -`fwdllm_aggregator.py:524`), per-iteration reselection, and the opt-in e2e parity harness -(`test_real_sim_e2e_parity.py`). **Still to build (the real work):** (1) the trainer sim path -(`_emulate_training_delay:481` still `time.sleep`; no `time_mode`/`simulated`, no `_sim_completion_ts`); -(2) the aggregator grad loop on the vclock (`_aggregate_grads_async:693` still raw `recv_fifo`; no -`_sim_recv_min`/`_vclock`/`_sim_hold_busy_slots` anywhere); (3) the variance-cadence rung layer; (4) the -availability *effect* path + `EVENT_AVAIL_CHANGE` emission. The build is organized as **4 pytest-gated -batches across 3 phases** (§E). +**Current state -- Phase 1 Batch 1 COMPLETE; Batch 2 is NEXT.** The two structural clock ports are landed +and pytest-gated (75 green). **Done:** the trainer sim path (`time_mode`/`simulated`, additive +`_sim_completion_ts`, no-sleep on the sim path), the async grad loop on the vclock +(`_sim_recv_min_grad` sct reorder buffer + in-flight gate + agg-goal-boundary rollback cleanup), the +sync barrier (`_sync_sim_recv_first_k` first-k-smallest-sct + U6 lags), the sim launchers, and the +Batch-1 pytest gate incl. the flag-off byte-identical regression. Batch-1 build detail is now history +(landed in `ef4cb70e` + follow-ups; deviation rationale in **§K**); the code is the source of truth. +**Still to build:** **(Batch 2)** the variance-cadence rung layer (V/DK/G in the shared parity engine) ++ fluxtune oort selection-fidelity validation -> then the Phase-1 syn_0 sign-off run; **(Batch 3)** the +availability *effect* path + `EVENT_AVAIL_CHANGE` emission (Phase 2); **(Batch 4)** full ladder under +unavailability (Phase 3). See **§E** for the batch list and **§I** for the Batch-2 cold-start map. **Prerequisites (read first):** - [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) -- the parity methodology (the ladder §1, roles/ @@ -67,41 +66,33 @@ TopAggregator(AsyncTopAgg)` -> `asyncfl.TopAggregator` -> `syncfl.TopAggregator( So both the vclock machinery **and** the availability substrate are already **on the instance** -- the work is **wiring them into fwdllm's gradient loop + trainer**, not re-implementing them. -### A.1 Aggregator spine (inherited, baseline-agnostic) +### A.1 Aggregator spine -- clock machinery NOW DRIVEN (Batch 1) | Mechanism (from `asyncfl/top_aggregator.py`) | Status for fwdllm | |---|---| -| Virtual clock `_vclock`, `_advance_sim_clock` (`vclock = max(vclock, sct)`) | inherited; **NOT yet driven** by the grad loop | -| §3.drain sct-ordered drain `_sim_recv_min` | inherited; **bypassed** (fwdllm's `_aggregate_grads_async` uses raw `recv_fifo`) | -| §3.resid one-in-flight residence `_sim_hold_busy_slots` | inherited; **bypassed** | -| §4.5/§4.9 sct-gated pool exclusion / carry-over (oort overlay) | available if a baseline selects via oort (fluxtune) | -| `ClientAvailability` substrate (trace-read gate, two ledgers, proactive evict, starvation advance) | inherited; **NOT yet wired** into the fwdllm grad loop | - -### A.2 Selector + duration signal (already wired) -- §S.dur duration helper `client_duration.py::real_client_task_train_duration` (intrinsic - `WALL_SEND_TS - WALL_RECV_TS`) is already called in `fwdllm_aggregator._process_single_trainer_message` - (`:745`) to set `PROP_CLIENT_TASK_TRAIN_DURATION`. No port needed. -- Oort selector (faithful §S.pacer two-branch pacer, §S.temporal UCB, D1-D8 fixes) is in - `flame/selector/oort.py`, usable verbatim by fluxtune (`async_oort`). -- Dynamic-KC controller/policy (`flame/selector/dynamic_kc_{controller,policy}.py`) is already integrated - (`_build_dynamic_kc_metrics`, `_dynamic_kc_controller.step()`); fluxtune leaves it **disabled** (fixed - K/C) per its config. See [docs/dynamic_kc_design.md](docs/dynamic_kc_design.md). -- Analysis: `scripts/analysis/analyze_run.py` is already fwdllm-aware via `telemetry_manifest.yaml` - (data_id/iteration_per_data_id progress hierarchy; per-category populate/partial status documented). - -### A.3 NOT reusable as-is -- the actual work -1. **Trainer sim path.** `FedSgdTrainer._emulate_training_delay` (`:481`) uses real `time.sleep`; it does - NOT stamp a modeled completion (`SIM_COMPLETION_TS` / `SIM_CLIENT_TASK_TRAIN_DURATION_S`) or - `WALL_SEND_TS`/`WALL_RECV_TS` the way async_cifar10's `trainer/pytorch/main.py` does. **Port required.** -2. **Aggregator grad loop is off the virtual clock.** `_aggregate_grads_async` (`fwdllm_aggregator.py:693`) - calls `channel.recv_fifo(...,1)` directly and commits on wall arrival -- never consults `_sim_recv_min`, - never advances `_vclock`, never holds in-flight slots. **The single biggest structural port.** -3. **Endogenous commit cadence** -- variance-gated dynamic-K, not fixed-K (PARITY.md §F.1): a new emergent - layer the async_cifar10 ladder does not model. -4. **Availability telemetry.** The fwdllm trainer emits **no `EVENT_AVAIL_CHANGE`** - (`telemetry_manifest.yaml`: availability = *partial*), so A6/A7/A8 ground-truth rungs can't run. - The `avail_change` / `agg_belief_change` / `send_gate_wait` builders exist - (`flame/telemetry/events.py:225/521`, `task_send` fields) and just need emitting from the fwdllm - trainer + aggregator. +| Virtual clock `_vclock`, `_advance_sim_clock` (`vclock = max(vclock, sct)`) | **DRIVEN** by the grad loop (Batch 1) | +| sct-ordered drain + in-flight gate | **DRIVEN** via purpose-built `_sim_recv_min_grad` (K-D4) / sync `_sync_sim_recv_first_k` | +| one-in-flight residence `_sim_hold_busy_slots` | **DRIVEN** at the agg-goal boundary (`_release_sim_slots_at_agg_goal`, K-D5) | +| §4.5/§4.9 sct-gated pool exclusion / carry-over (oort overlay) | available if a baseline selects via oort (fluxtune); **Batch-2 validation** | +| `ClientAvailability` substrate (trace-read gate, two ledgers, proactive evict, starvation advance) | inherited; **NOT yet wired** into the grad loop -- **Batch 3 (Phase 2)** | + +### A.2 Selector + duration signal (already wired, baseline-agnostic) +- §S.dur duration helper `client_duration.py::real_client_task_train_duration` is already called in + `_process_single_trainer_message` to set `PROP_CLIENT_TASK_TRAIN_DURATION`. +- Oort selector (`flame/selector/oort.py`, faithful pacer/UCB/D1-D8) is usable verbatim by fluxtune + (`async_oort`) -- **Batch 2 validates its rungs fire**; inert for the two `random`-selector baselines. +- Dynamic-KC controller/policy is integrated (`_build_dynamic_kc_metrics`, `_dynamic_kc_controller.step()`); + fluxtune leaves it **disabled** (fixed K/C). See [docs/dynamic_kc_design.md](docs/dynamic_kc_design.md). +- Analysis `scripts/analysis/analyze_run.py` is fwdllm-aware via `telemetry_manifest.yaml`. + +### A.3 Remaining structural work (post-Batch-1) +1. **Endogenous commit cadence** -- variance-gated dynamic-K (PARITY.md §F.1): the emergent layer the + async_cifar10 ladder does not model. The clock is now driven; **Batch 2 adds the V/DK/G rung layer** that + verifies the *cadence* on top of it. Most of the per-cycle telemetry it needs already lands via + `build_agg_round` (§I). +2. **Availability telemetry + effect path.** The fwdllm trainer emits **no `EVENT_AVAIL_CHANGE`** + (`telemetry_manifest.yaml`: availability = *partial*), so A6/A7/A8 ground-truth rungs can't run yet, and + the send-gate/delivery-buffer effect path is not wired. Builders exist in `flame/telemetry/events.py`. + **Batch 3 (Phase 2).** --- @@ -189,39 +180,15 @@ is a *validation gate*, not a batch -- its telemetry is already landed (see Curr ### PHASE 1 -- 100% availability (syn_0), all three baselines -#### Batch 1 (large dev) -- both structural ports: trainer sim path + async grad loop + sync barrier -The single biggest structural batch. Lands three tightly-coupled ports (the trainer's stamped completion -is exactly what the aggregator's sct buffer consumes, so they are tested together). Reference port for the -trainer: async_cifar10 `trainer/pytorch/main.py` already has the full `time_mode`/`simulated`/ -`_sim_completion_ts`/`sim_completion_leg_s` plumbing (lines 106/173-174/840-841/1086-1093/1151-1155) -- -this is a port of a known-good shape, not a design problem. - -- **Trainer** (`FedSgdTrainer.py`): add `time_mode`/`simulated`; replace `_emulate_training_delay`'s real - `time.sleep` (`:481`) with stamping `_sim_completion_ts = sim_send_ts + max(real_gpu_s, modeled_budget_D) - + sim_completion_leg_s`; send `SIM_COMPLETION_TS` + `SIM_CLIENT_TASK_TRAIN_DURATION_S` + - `WALL_SEND_TS`/`WALL_RECV_TS`. Stamp a **per-eval** `sct` (dead-end: reusing the last train `sct` - past-dates every eval); eval delay ~= train (forward pass), **not** async_cifar10's 20x speedup -- D4. -- **Async grad loop** (`_aggregate_grads_async:693`, fluxtune): replace raw `channel.recv_fifo(...,1)` with - the inherited **`_sim_recv_min`** (sct-ordered reorder buffer + §3.drain) + **`_sim_hold_busy_slots`** - (§3.resid, one-in-flight) + `_advance_sim_clock`. **fwdllm processes one message per call**, so key the - slot release on the **agg-goal boundary** (where `_per_agg_trainer_list` clears), NOT per-message. Keep - `simCommitOverheadSeconds=0` (overhead on the vclock is a hard dead-end). **Rollback risk:** a data_id - spans **many** agg-goal cycles -- slot-hold + sct-buffer must survive rollbacks without leaking or - double-committing a grad; over-instrument `inflight_residence` + per-cycle buffer occupancy. -- **Sync barrier** (`_aggregate_grads_sync:1354`, fwdllm/fwdllm_plus): barrier-anchored visibility-lag - treatment (PARITY.md §6.u6); vclock advance at the barrier. *(Pulled forward from old Stage 5.)* -- **Launchers:** add `time_mode: simulated` variants of the three `expt_scripts/*_n10_smoke.yaml`. - -**Pause to test (pytest-only):** unit tests for (a) trainer stamping formula + per-eval `sct` + a -"no `time.sleep` on the sim path" assertion; (b) the sct reorder buffer & one-in-flight hold over a -synthetic out-of-order message sequence **including a rollback** (no leaked/double-committed grad); (c) -sync-barrier vclock advance + no per-message past-dating. Then one tiny in-process 2-3-trainer **seeded** -run per path -> feed telemetry to `parity_checks`. Full existing `test_fwdllm_*` suite green (flag-off -byte-identical regression). **Exit rungs:** P3 matches; K6 advancing; T2 matched; K1 monotone; K3a/K3b -(per variance-pass) ~= 0; U5 inter-arrival; one-in-flight overlap ~= real; U6 barrier lag real~=sim; -commit_gap ~= 0. - -#### Batch 2 (large dev) -- variance-cadence layer + fluxtune selection fidelity +#### Batch 1 -- both structural clock ports: trainer sim path + async grad loop + sync barrier -- ✅ DONE +Landed & pytest-gated (75 green). Trainer additive `_sim_completion_ts` + no-sleep; async +`_sim_recv_min_grad` sct buffer + in-flight gate + agg-goal-boundary rollback cleanup; sync +`_sync_sim_recv_first_k` first-k-smallest + U6 lags; sim launchers. **Build detail is history** -- see +the code (`ef4cb70e` +) and the deviation rationale **§K-D1..D8**. Cold-start map for the code lives in +**§I** (now repurposed as the Batch-2 map). U6 telemetry *emission* deferred (K-D7); the seeded mini-run +feeding `parity_checks` is folded into the Phase-1 sign-off run. + +#### Batch 2 (large dev, NEXT) -- variance-cadence layer + fluxtune selection fidelity - **Variance-cadence rungs** (PARITY.md §F.4): implement V1-V5 / DK1-DK3 / G1-G2 in the shared engine. Most dev is checker + the telemetry to feed it: per-cycle `var` trajectory (at each agg-goal), iterations-per-`data_id` (realized dynamic-K), force-commit (`max_iterations_per_data_id`) bypass rate, @@ -350,371 +317,110 @@ those are baseline-defining knobs; a cadence gap is an upstream set/order/clock ## §H Status *(Run-log: record per-baseline run dirs, lowest broken rung, root hypothesis, score X/N, JSON path; keep -the run-length budget keyed. One section, updated in place. No parity run yet -- first entry lands after -Batch 1's pytest gate.)* - ---- +the run-length budget keyed. One section, updated in place. Batch 1 pytest gate is green; **first run-log +entry lands after the Phase-1 syn_0 sign-off run** -- which is gated behind Batch 2.)* -## §I Batch 1 implementation map (code-level -- captured during study, before edits) - -**Purpose:** durable engineering map so Batch 1 can resume cold. All line numbers are as-of branch -`dg/fwdllm_sim_unavail` at capture time; re-grep before editing. Batch 1 = trainer sim path + async grad -loop vclock drive + sync barrier, all config-gated (flag-off => byte-identical), syn_0 only. - -### I.1 Class chain + where the sim substrate already lives (INHERITED, reuse -- don't reimplement) -- Trainer: `examples/fwdllm/trainer/forward_training/FedSgdTrainer.py::FedSGDTrainer(Trainer)` -> - `flame/mode/horizontal/syncfl/fwdllm_trainer.py::Trainer(Role)`. **The messaging (recv/send) lives in - the base `fwdllm_trainer.Trainer`, not FedSgdTrainer.** -- Aggregator: `FedSgdAggregator` -> `syncfl/fwdllm_aggregator.py::TopAggregator(AsyncTopAgg)` -> - `asyncfl/top_aggregator.py::TopAggregator` -> `syncfl/top_aggregator.py::TopAggregator`. -- **`self.simulated` / `self.time_mode` / `self._vclock` (VirtualClock) / `_advance_sim_clock(sct)` are - defined in `syncfl/top_aggregator.py:199-203, 331`** (`vclock=max(vclock,sct)`; overhead default 0 -- - keep 0). Read `time_mode` from `config.hyperparameters.time_mode` (default "simulated"). -- **`SimReorderBuffer` `self._sim_buffer` + `self._sim_committed`** inited in `syncfl/top_aggregator.py:228` - AND `asyncfl/top_aggregator.py:112`. Async `__init__` also inits `_sim_inflight_expected`, - `_sim_pending_commit`, `_sim_trainer_budget`, budget running-mean/min, withheld ledgers, cooldown, and - the flag knobs `sim_sct_ordered_drain` / `sim_inflight_residence` / `sim_clock_jump_clamp` / - `sim_staggered_redispatch` / `sim_redispatch_gap_s` (`asyncfl:112-195`). **TODO verify fwdllm agg's - `__init__`/`internal_init` actually chains through the async base so these exist** (else lazy-init guard - at `asyncfl:323` covers `_sim_inflight_expected` et al. but NOT `_sim_buffer`). -- **Sync-barrier machinery ALREADY EXISTS in the syncfl base and just needs calling:** - `_sync_sim_recv_first_k(channel, ends, first_k)` (`syncfl/top_aggregator.py:360`) -- drains the selected - set, commits the `first_k` smallest-sct, advances vclock to each, stamps - `PROP_CLIENT_TASK_TRAIN_DURATION` from `SIM_CLIENT_TASK_TRAIN_DURATION_S`; and - `_barrier_anchored_lags(durs)` (`:472`, the U6 helper: `lag_i = max_completion - completion_i`). -- **Async single-message commit machinery:** `_sim_recv_min(channel, recv_ends)` (`asyncfl:314`) returns - ONE sct-ordered committed `(msg, md)` -- maps onto fwdllm's one-message-per-call loop -- but is coupled - to WEIGHTS semantics (staggered-redispatch keys on WEIGHTS/WEIGHTS_BYTES; budget from - TRAINING_BUDGET_S; withheld/eval paths). With gate OFF + defaults OFF these are inert. `_sim_hold_busy_ - slots(channel)` (`asyncfl:1453`) holds busy trainers' concurrency slots until commit (one-in-flight). -- **Reference port for the trainer half:** async_cifar10 `trainer/pytorch/main.py` has the full - `time_mode`/`simulated`/`_sim_completion_ts`/`sim_completion_leg_s`/per-eval-sct plumbing - (`:106,173-174,818-845 (train sct),1086-1096 (eval sct),1151-1158`). Use it for the *plumbing shape*, - **NOT the timing formula** -- see the critical correction below. -- **!!! CRITICAL CORRECTION -- fwdllm uses an ADDITIVE delay model, NOT cifar10's `max(gpu, D)` !!!** - cifar10's real mode "sleeps to fill a budget": wall = `max(gpu, D)`. **fwdllm's real mode sleeps D ON TOP - of GPU time** (a flat additive emulated delay -- see `FedSgdTrainer._emulate_training_delay:481` docstring - "flat additive sleep on top of GPU time, not a budget-minus-actual model" and the EXISTING telemetry - `sim_round_duration_s = _real_gpu_time_s + _delay_s` at `FedSgdTrainer.py:545`). So for real<->sim parity - the sim sct MUST be additive too: - - `D = _emulate_training_delay()`'s modeled value = `training_delay_s / training_delay_factor` - (÷`speedup_factor`, which is 1.0). NOTE the method is named "training" but models the **eval/forward - cost** (`training_delay_factor` is the eval speedup); fwdllm's forward-grad "train" IS a forward pass. - - `sim_round_duration = _real_gpu_time_s + D` (ADDITIVE -- matches real's sleep-on-top) - - `_sim_completion_ts = (_sim_send_ts) + sim_round_duration + (leg if simulated)` - - **Port is minimal because the additive delay is already computed:** existing `train_with_data_id` - already does `_delay_s = self._emulate_training_delay()` (`:523`) and `sim_round_duration_s = - _real_gpu_time_s + _delay_s` (`:545`). The port just (a) makes `_emulate_training_delay` NOT sleep in - sim mode but still RETURN D, and (b) stamps `_sim_completion_ts` from that same `sim_round_duration_s`. - - **Latent bug to fix while here:** `_emulate_training_delay` gates on `training_delay_enabled == "True"` - (STRING compare, `:488`) but the config schema types it as `bool` (`config.py:197`, default False). So - with a real bool the delay NEVER fires. Normalize the check (`str(...)=="True" or ... is True`) or the - modeled D is silently 0 and real==sim only trivially. Verify what the launcher actually passes. - -### I.2 Trainer sim-path port -- exact edit sites -1. **`fwdllm_trainer.Trainer._fetch_weights` (`:168`, msg parse ~`:208-320`)** -- currently reads ROUND/ - WEIGHTS/DATA_ID but NOT sim stamps. ADD: `self._sim_send_ts = msg.get(MessageType.SIM_SEND_TS)` and - `self._wall_recv_ts = time.time()` on model receipt. (Base syncfl trainer already does the analogous - `_sim_send_ts = msg[SIM_SEND_TS]` at `syncfl/trainer.py:257` -- fwdllm's trainer doesn't extend it, so - port the read.) -2. **`FedSgdTrainer.__init__` (`:115`)** -- config knobs at `:176-182` (`training_delay_enabled`, - `training_delay_s`, `training_delay_factor`, `speedup_factor=1.0`). ADD: - `self.time_mode = config.hyperparameters.time_mode` (default "simulated"), - `self.simulated = self.time_mode == "simulated"`, `self.sim_completion_leg_s` (default 0.0), - init `self._sim_send_ts=None`, `self._sim_completion_ts=None`, `self._sim_round_duration_s=None`, - `self._wall_recv_ts=None`. -3. **`FedSgdTrainer._emulate_training_delay` (`:481`, called `:523`)** -- currently `time.sleep(_sleep_s)`. - GATE: when `self.simulated`, DO NOT sleep; still RETURN the modeled delay D (so `_delay_s` stays correct - for the additive `sim_round_duration_s`). Fix the `== "True"` string-gate bug (see CRITICAL CORRECTION). -4. **`FedSgdTrainer.train_with_data_id` (`:502`)** -- already computes `_delay_s` (`:523`) and - `sim_round_duration_s = _real_gpu_time_s + _delay_s` (`:545`, ADDITIVE). ADD: store - `self._sim_round_duration_s = _real_gpu_time_s + _delay_s` and - `self._sim_completion_ts = (self._sim_send_ts or _fallback) + self._sim_round_duration_s + (leg if - simulated)` for `_send_grads` to read. **Per-eval sct (D4):** fwdllm eval is a forward pass ~= train - cost (NOT cifar's /20; `training_delay_factor` already encodes the modest eval speedup). If the trainer - performs a distinct eval task, stamp eval's OWN sct = `_sim_send_ts + (real_eval_gpu + D_eval)` -- do NOT - reuse the last train sct (past-dates every eval, poisons the reorder-buffer key, cifar `:1086-1096`). - **VERIFY:** in fwdllm the *aggregator* runs `eval_model()` (`fwdllm_aggregator.py:1391`) on the global - model after a variance pass; the trainer's `task_to_perform=="eval"` only sends MODEL_VERSION+STAT_UTILITY - (`_send_grads:522-526`). So confirm whether a trainer-side eval sct is even needed for fwdllm, or whether - D4 collapses to "train sct only" (likely the latter -- eval lives on the aggregator, off the grad clock). -5. **`fwdllm_trainer.Trainer._send_grads` (`:438`, msg built `:506-521`)** -- ADD to the train msg: - `SIM_COMPLETION_TS=self._sim_completion_ts`, `SIM_CLIENT_TASK_TRAIN_DURATION_S=self._sim_round_duration_s`, - `TRAINING_BUDGET_S=`, `WALL_SEND_TS=time.time()`, `WALL_RECV_TS=self._wall_recv_ts`. The eval - msg (`:522-526`) also needs `SIM_COMPLETION_TS`/`WALL_*` for the per-eval sct. - -### I.3 Aggregator dispatch stamping (both distribute paths) -- exact edit sites -- **`_distribute_weights_sync` (`:1777`, per-end send loop `:1849-1900`, `channel.send` `:1899`)** and - **`_distribute_weights_async` (`:1904`, `channel.send` `:2008`)**. In each per-end loop, when - `self.simulated`: set `_sst = self._vclock.now`; inject `payload[MessageType.SIM_SEND_TS] = _sst` (or a - per-end copy -- payload is shared across ends, so stamp per-end or set the channel prop); - `channel.set_end_property(end, PROP_SIM_SEND_TS, _sst)`. **Async only:** populate the gate - `self._sim_inflight_expected[end] = _sst + ` (budget = running-mean/min floor, see - `asyncfl:1671` for the weights-path analog). `PROP_ROUND_START_TIME` already set at `:1866`. - `PROP_SIM_SEND_TS`/`PROP_SIM_COMPLETION_TS` imported from `flame.selector.properties`. - -### I.4 Aggregator grad-loop vclock drive -- exact edit sites + the fwdllm-specific wiring -- **Async: `_aggregate_grads_async` (`:693`)** -- currently `next(channel.recv_fifo(ends,1,timeout=...))` - at `:719`. REPLACE (sim branch only) with an sct-ordered buffered pop. Design decision (see §G risk - "one-message-per-call vs batch"): reuse the shared primitives (`SimReorderBuffer`, `_advance_sim_clock`, - `_sim_inflight_expected` gate) but wire them for the grad loop rather than calling weights-coupled - `_sim_recv_min` verbatim -- ingest all ready grad msgs into `_sim_buffer` keyed by - `msg[SIM_COMPLETION_TS]`, pop min-sct committable, `_advance_sim_clock(sct)`, return one `(msg, md)`. - Real branch unchanged. -- **Slot hold/release (one-in-flight):** call `_sim_hold_busy_slots(channel)` on dispatch; **release keys - on the AGG-GOAL BOUNDARY**, i.e. where `_process_aggregation_goal_met` clears - `self._per_agg_trainer_list = []` (`:1271`) and resets `_agg_goal_cnt=0` (`:1279`) -- NOT per message. - (The doc's §3.resid release-point warning: do not copy asyncfl's per-batch release.) -- **Sync: `_aggregate_grads_sync` (`:1354`) -> `sync_collect_and_accumulate_grads` (`:1313`)** -- the drain - loop `channel.recv_fifo(channel.ends(), num_min_req, timeout=...)` at `:1336`. REPLACE (sim branch) with - `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` then feed each committed - `(msg, md)` through `_process_single_trainer_message`. Apply `_barrier_anchored_lags` for U6 telemetry. -- **Commit path already computes intrinsic duration:** `_process_single_trainer_message` (`:745`) at - `:837` calls `real_client_task_train_duration(msg, sent_ts, timestamp)` (WALL_SEND-WALL_RECV) and sets - `PROP_CLIENT_TASK_TRAIN_DURATION` -- keep; this is the selector/telemetry duration (server-overhead-free). - -### I.5 ROLLBACK-SAFETY (the fwdllm-specific §3.resid risk -- D-note) -`_process_aggregation_goal_met` (`:1081`): `self.aggregate(self._round)` (`:1153`) runs the variance gate. -**PASS** (`var_good_enough`, `:1170`) -> advance `data_id`, `iteration_per_data_id=0`. **FAIL** (`:1237`) -> -retry SAME `data_id`, `iteration_per_data_id += 1` (the ROLLBACK path). Both branches clear -`_per_agg_trainer_list` (`:1271`) and reset `_agg_goal_cnt=0` (`:1279`). So a `data_id` can span MANY -agg-goal cycles. **Invariant to preserve:** the sct reorder buffer + slot-hold must not strand or -double-commit a grad across a rollback -- release slots and drain/clear the buffer's committed set at the -agg-goal boundary each cycle. `_sim_committed` is cleared by the async base `_reset_agg_goal_variables` -(`asyncfl:231-232`) but fwdllm's own reset (`:453`) does NOT -- **verify/port this clear into the fwdllm -agg-goal boundary** or the second cycle on a rolled-back data_id sees stale committed marks. Over-instrument -`inflight_residence` + per-cycle buffer occupancy. - -### I.6 MessageType / API facts -- VERIFIED (capture time) -- **CONFIRMED** MessageType keys exist (`flame/mode/message.py:82-87`): `SIM_SEND_TS=34`, - `SIM_COMPLETION_TS=35`, `SIM_CLIENT_TASK_TRAIN_DURATION_S=36`, `TRAINING_BUDGET_S=37`, `WALL_SEND_TS=38`, - `WALL_RECV_TS=39`. -- **CONFIRMED** `SimReorderBuffer` (`flame/sim/virtual_clock.py:52`): `.add(end, sim_completion_ts, - payload)`, `.has(end)`, `.pending_ends()->set`, `__len__`, `.peek_min_ts()->Optional[float]`, - `.pending_after(ts)->set`, `.pop_min()->Optional[(end, ts, payload)]`, `.discard(end)`, `.clear()`. - `VirtualClock`: `.now` (property), `.advance(ts)`, `.reset()`. -- **CONFIRMED** fwdllm agg chains the sim substrate: `fwdllm_aggregator.internal_init` (`:239`) calls - `super().internal_init()` (`:241`) -> asyncfl `internal_init` (`asyncfl:83`, inits `_sim_buffer` at - `:112`) -> syncfl base. So `_sim_buffer`/`_sim_inflight_expected`/`_vclock`/`simulated` all exist. (Role - pattern uses `internal_init`, NOT `__init__`.) -- **ROLLBACK HOLE CONFIRMED (act on I.5):** fwdllm's `_reset_agg_goal_variables` (`:450`) does NOT clear - `_sim_committed` (the async base's does, `asyncfl:231-232`). fwdllm's real agg-goal boundary is - `_process_aggregation_goal_met` (clears `_per_agg_trainer_list`/`_agg_goal_cnt` directly at `:1271,1279`). - **=> Port must explicitly `_sim_committed.clear()` + drain/clear any stranded `_sim_buffer` entries at that - boundary each cycle**, so a rolled-back data_id's next cycle starts clean. - -### I.7 Launcher configs -Add `time_mode: simulated` variants of `expt_scripts/{fluxtune,fwdllm,fwdllm_plus}_n10_smoke.yaml` -(current default is `time_mode: real`). Keep the real ones for the reference runs. - -### I.8 Batch 1 pytest gate (pytest-only, no experiment run) -1. **Trainer stamping** (extend `tests/mode/test_fwdllm_trainer_sim_duration.py`): `_sim_completion_ts == - _sim_send_ts + max(gpu,D) + leg`; per-eval sct distinct from last train sct; NO `time.sleep` on the sim - path (monkeypatch/assert). 2. **sct reorder buffer + one-in-flight** (new test): drive the grad loop - with a synthetic out-of-order message sequence incl. a variance-FAIL rollback cycle -> assert commit - order = sct order, vclock monotone, no leaked/double-committed grad across the rollback, slots released - at agg-goal boundary. 3. **Sync barrier** (new test): `_sync_sim_recv_first_k` commits the k smallest - sct; `_barrier_anchored_lags` correct; vclock advances to k-th; no per-message past-dating. - 4. **Flag-off byte-identical:** `time_mode: real` path unchanged. 5. Full existing `test_fwdllm_*` green. -**Exit rungs:** P3, K6, T2, K1 monotone, K3a/K3b~=0, U5, one-in-flight overlap~=real, U6 barrier lag, commit_gap~=0. +**Batch 1 landed (done):** all six subtasks complete, `test_fwdllm_*` = 75 green. Code is the source of +truth; the as-built decisions are logged in **§K-D1..D8**. Batch-1 pytest files: +`tests/mode/test_fwdllm_trainer_sim_duration.py` (trainer additive stamp + no-sleep), +`tests/mode/test_fwdllm_sim_grad_loop.py` (async sct buffer / in-flight gate / rollback), +`tests/mode/test_fwdllm_sim_sync_barrier.py` (first-k-smallest + U6 lags), plus the flag-off +`simulated=False` fixes across the four pre-existing fwdllm agg/trainer tests. --- -## §J Batch 1 EXECUTION PROGRESS (resume point -- uncommitted working tree) - -**Status:** Batch 1 in progress on branch `dg/fwdllm_sim_unavail`, **NOT committed**. Steps 1-2 of the -subtask list (trainer sim path, aggregator dispatch stamping) are **landed in the working tree and -`py_compile`-clean**. Step 3 (async grad loop drive) is **designed in full below but NOT yet written**. -Steps 4-6 (sync barrier, launchers, pytests) untouched. All line numbers below are as of the paused -working tree -- **re-grep before editing** (edits already shifted them from §I's capture). - -**RESUME CHECKLIST (start a new session here -- point at this §J):** -- [x] **1. Trainer sim path** -- landed (J.1(1)). -- [x] **2. Aggregator dispatch stamping** -- landed (J.1(2)). -- [x] **3. Async grad loop drive** -- LANDED (working tree, imports clean in `dg_flame`). `_sim_recv_min_grad` - + `_aggregate_grads_async` mode-branch + `_release_sim_slots_at_agg_goal` boundary cleanup. See §J.2 - (design) + J.1(3) below (as-landed sites). -- [x] **4. Sync barrier** -- LANDED (working tree, imports clean in `dg_flame`). `_sync_sim_recv_first_k` - sim-drive + `_barrier_anchored_lags` stash in `sync_collect_and_accumulate_grads`. See J.1(4). **U6 - full telemetry emission DEFERRED -- §K-D7.** -- [x] **5. Launchers** -- LANDED. `{fwdllm,fluxtune,fwdllm_plus}_n10_smoke_sim.yaml` created (yaml-parse - clean): identical to the real smokes except `time_mode: simulated` + `_sim` job id. `enable_training_delays` - kept false for direct comparability -- see §K-D8. -- [ ] **6. Pytest gate** -- trainer stamping + no-sleep, sct reorder buffer incl. rollback, sync barrier; - full `test_fwdllm_*` green (§I.8 + §J.3). Pytest-only (no broker in this env). - -### J.1 LANDED (in working tree, uncommitted, compiles) - -**(1) Trainer sim path -- DONE.** -- `examples/fwdllm/trainer/main.py:76` -- replaced the "no simulated-clock support" warning block with - `config.hyperparameters.time_mode = _cli_args.time_mode` (Hyperparameters is `extra=allow`, verified - attr-set works). So the trainer reads time_mode from config uniformly with the aggregator (runner injects - the agg's at `launch/runner.py:128`). **Deviation from §I.2:** default is `"real"` via - `getattr(..., "time_mode", "real")` (NOT "simulated") -- conservative for fwdllm's all-real existing - configs; the launcher/CLI default is also "real". Sim variants set `time_mode: simulated` explicitly. -- `FedSgdTrainer.__init__:184-208` -- added `self.time_mode`/`self.simulated`/`self.sim_completion_leg_s` - + inited `_sim_send_ts`/`_sim_completion_ts`/`_sim_round_duration_s`/`_wall_recv_ts = None`. -- `FedSgdTrainer._emulate_training_delay:500` (was :481) -- fixed the `== "True"` string-gate bug - (`_enabled = self.training_delay_enabled in (True, "True", "true")` at :516); when `self.simulated` DO - NOT `time.sleep` but STILL return modeled D. Real mode unchanged (still sleeps D). -- `FedSgdTrainer.train_with_data_id:560-572` -- ADDITIVE stamp (matches §I CRITICAL CORRECTION, NOT - cifar's max(gpu,D)): `self._sim_round_duration_s = _real_gpu_time_s + _delay_s`; when simulated, - `self._sim_completion_ts = (_sim_send_ts or time.time()) + _sim_round_duration_s + sim_completion_leg_s`. -- `fwdllm_trainer._fetch_weights:210-217` -- reads `self._sim_send_ts = msg.get(SIM_SEND_TS)` + - `self._wall_recv_ts = time.time()` on model receipt. -- `fwdllm_trainer._send_grads:530-556` -- train msg now sends `SIM_COMPLETION_TS`, - `SIM_CLIENT_TASK_TRAIN_DURATION_S`, `TRAINING_BUDGET_S` (all `= _sim_round_duration_s`; additive model - has no separate budget), `WALL_SEND_TS=time.time()`, `WALL_RECV_TS=_wall_recv_ts`. Eval msg sends - `SIM_COMPLETION_TS` + `WALL_*`. **D4 RESOLVED:** the trainer loop is - `task_get >> train_with_data_id >> put(_send_grads)` every iteration - (`fwdllm_trainer.py:817-828`), so `_sim_completion_ts` is ALWAYS fresh from the same iteration; fwdllm - eval lives on the AGGREGATOR (`eval_model`), the trainer eval msg is only a utility report -> **eval - reuses the same-iteration train sct, no distinct per-eval sct needed** (D4 collapses to "train sct only"). - -**(2) Aggregator dispatch stamping -- DONE.** -- `fwdllm_aggregator.py:64` -- `from flame.selector.properties import PROP_SIM_SEND_TS` (the oort import - block does NOT export it; `PROP_SIM_SEND_TS="sim_send_ts"`, `PROP_SIM_COMPLETION_TS="sim_completion_ts"`). -- `_distribute_weights_sync:1850` -- before the per-end loop: `_round_now = self._vclock.now` (sim); - inject `SIM_SEND_TS=_round_now` into both `payload_with_weights`/`payload_without_weights` (same instant - for the whole sync barrier); inside the loop `channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now)`. - No gate (`_sim_inflight_expected`) on the sync path. -- `_distribute_weights_async:1993, 2039-2041` -- same stamp into `payload_weights`/`payload_var_bad`; - inside the loop ALSO arms the gate: `_budget = self._sim_trainer_budget.get(end, self._sim_budget_min)` - then `self._sim_inflight_expected[end] = _round_now + _budget`. -- All sim attrs (`_vclock`, `simulated`, `_sim_buffer`, `_sim_committed`, `_sim_inflight_expected`, - `_sim_trainer_budget`, `_sim_budget_min=12.0`) confirmed inited via `internal_init` chain - (`fwdllm_aggregator.internal_init:239` -> `super().internal_init()` -> asyncfl `internal_init:83`). - -**(3) Async grad loop drive -- DONE** (working tree; imports clean in `dg_flame`, not yet pytest-run). -- `fwdllm_aggregator.py:50` -- import `_SIM_GATE_MAX_PASSES, _SIM_ORDER_SLACK_S` from the asyncfl module. -- `_sim_recv_min_grad:696` -- new method, body per §J.2 (ingest ready msgs into `_sim_buffer` keyed by - SIM_COMPLETION_TS -> in-flight gate -> pop min-sct -> clock-jump-clamp `_advance_sim_clock` -> learn - budget -> mark `_sim_committed`). Returns ONE `(msg, md)` or `(None, ("", now))`. Does NOT touch selector - slots (all slot/buffer/committed clearing is boundary-only). -- `_release_sim_slots_at_agg_goal:788` -- new boundary helper: clears `_sim_committed` + `_sim_buffer` + - `_sim_inflight_expected`; async-only `_sim_hold_busy_slots(channel)` (empty held set => releases slots). -- `_aggregate_grads_async:840-846` -- sim branch calls `_sim_recv_min_grad(channel, ends(VAL_CH_STATE_RECV))`; - real branch unchanged (`next(recv_fifo(...,1))`). -- `_process_aggregation_goal_met:1441-1442` -- after `channel.cleanup_recvd_ends():1434`, when simulated, - calls `_release_sim_slots_at_agg_goal(channel, is_async)` (reached by BOTH variance-PASS + FAIL branches). - -**(4) Sync barrier drive -- DONE** (working tree; imports clean in `dg_flame`, not yet pytest-run). -- `sync_collect_and_accumulate_grads:1448` -- sim branch (`:1479`) calls - `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` (base at - `syncfl/top_aggregator.py:360`: commits k-smallest-sct, advances vclock to k-th, stamps - PROP_CLIENT_TASK_TRAIN_DURATION), then feeds each `(msg, md)` through `_process_single_trainer_message`, - breaking at `_agg_goal_cnt >= _agg_goal`. Real branch moved into the `else:` -- byte-identical. -- `:1502` -- stashes `self._sync_barrier_lags_s = self._barrier_anchored_lags(_barrier_durs)` where - `_barrier_durs` = each committed msg's `SIM_CLIENT_TASK_TRAIN_DURATION_S` (dispatch-relative completion, - sim's analog of WALL_SEND - dispatch). **U6 emission deferred (§K-D7): fwdllm's sync path has no - `_round_update_values`/visibility-lag telemetry struct like the base -- the lag is computed + logged - (`[SYNC_SIM_BARRIER]`) and stashed on the instance, but not yet emitted as a telemetry event.** - -### J.2 Async grad loop drive -- design (LANDED, see J.1(3) for as-landed sites) - -Add a new method `_sim_recv_min_grad(self, channel, recv_ends)` on the fwdllm aggregator (place near -`_aggregate_grads_async:694`). It is a **purpose-built** grad analog of `asyncfl._sim_recv_min` (do NOT -call `_sim_recv_min` verbatim -- it does per-commit slot release via `_sim_pending_commit`/`sel.all_selected` -at asyncfl:615-627, but fwdllm must release on the AGG-GOAL boundary; and its withheld/staggered paths key -on WEIGHTS semantics). Needs module constants `_SIM_GATE_MAX_PASSES=64`, `_SIM_ORDER_SLACK_S=2.0` -- -**import them from `flame.mode.horizontal.asyncfl.top_aggregator`** (they are module-level there; not yet -imported into fwdllm_aggregator). `RECV_TIMEOUT_WAIT_S` already imported (:48), `datetime` (:23), `time` (:22). - -Method body (ingest -> gate -> pop -> advance -> budget -> mark committed; returns ONE `(msg, md)` matching -the shape of `next(channel.recv_fifo(...,1))`, or `(None, ("", datetime.now()))`): -```python -live = [e for e in (recv_ends or []) if channel.has(e)] -deadline = time.time() + RECV_TIMEOUT_WAIT_S -for _pass in range(_SIM_GATE_MAX_PASSES): - grace = self._sim_recv_grace_s() - to_probe = [e for e in set(live) | set(self._sim_inflight_expected) - if channel.has(e) and not self._sim_buffer.has(e) and e not in self._sim_committed] - if to_probe: - for m, md in channel.recv_fifo(to_probe, first_k=len(to_probe), timeout=grace): - if m is None: break - _e = md[0]; _s = m.get(MessageType.SIM_COMPLETION_TS) - self._sim_buffer.add(_e, float(_s) if _s is not None else self._vclock.now, (m, md)) - bmin = self._sim_buffer.peek_min_ts() - min_stuck = None - for e, exp in self._sim_inflight_expected.items(): - if self._sim_buffer.has(e) or e in self._sim_committed: continue - if min_stuck is None or exp < min_stuck: min_stuck = exp - earlier_stuck = (bmin is not None and min_stuck is not None and min_stuck + _SIM_ORDER_SLACK_S < bmin) - if bmin is None and not to_probe: break - if not earlier_stuck: break - if time.time() >= deadline: break -popped = self._sim_buffer.pop_min() -if popped is None: return None, ("", datetime.now()) -_end, sct, (m, md) = popped -# clock-jump clamp (don't lap a fresh in-flight cohort): -_now = self._vclock.now; _min_future = None -for e, exp in self._sim_inflight_expected.items(): - if e == _end or e in self._sim_committed: continue - if exp > _now and (_min_future is None or exp < _min_future): _min_future = exp -_advance_to = sct if _min_future is None else max(_now, min(sct, _min_future + _SIM_ORDER_SLACK_S)) -self._advance_sim_clock(_advance_to) -self._sim_committed.add(_end); self._sim_inflight_expected.pop(_end, None) -_b = m.get(MessageType.TRAINING_BUDGET_S) if isinstance(m, dict) else None -if _b is not None: - self._sim_trainer_budget[_end] = float(_b); self._sim_budget_min = min(self._sim_budget_min, float(_b)) -return m, md -``` -Then in `_aggregate_grads_async` REPLACE the unconditional `next(channel.recv_fifo(...,1,...))` at **:720** -with a mode branch: -```python -if self.simulated: - msg, metadata = self._sim_recv_min_grad(channel, channel.ends(VAL_CH_STATE_RECV)) -else: - msg, metadata = next(channel.recv_fifo(channel.ends(VAL_CH_STATE_RECV), 1, timeout=RECV_TIMEOUT_WAIT_S)) -end, timestamp = metadata -``` -(keep the existing `if not msg: return` guard right after). - -**Agg-goal-boundary cleanup (rollback-safety, §I.5).** Add a helper and call it from -`_process_aggregation_goal_met` right AFTER `channel.cleanup_recvd_ends()` (**:1309**) -- this point is -reached by BOTH the variance-PASS and variance-FAIL(rollback) branches (both fall through past the -`_per_agg_trainer_list = []` clear at :1272): -```python -def _release_sim_slots_at_agg_goal(self, channel, is_async): - if not self.simulated: return - self._sim_committed.clear() # else next cycle's gate skips a re-contributing end (rollback hole) - self._sim_buffer.clear() # drop stranded arrived-but-uncommitted grads so next cycle is clean - self._sim_inflight_expected.clear() - if is_async: - self._sim_hold_busy_slots(channel) # empty held set -> releases every committed end's slot -``` -call site: `if self.simulated: self._release_sim_slots_at_agg_goal(channel, is_async)` after :1309. -(This is why fwdllm's own `_reset_agg_goal_variables:450` NOT clearing `_sim_committed` -- the §I.6 ROLLBACK -HOLE -- is handled: we clear it explicitly at the real boundary instead.) `_sim_hold_busy_slots` -(asyncfl:1453) needs `channel._selector` with `.requester/.all_selected/.selected_ends` -- provide in the -fake-channel test. - -### J.3 REMAINING (untouched) - -- **Sync barrier** (subtask 4): in `sync_collect_and_accumulate_grads:1315`, replace the drain loop - `for msg, metadata in channel.recv_fifo(channel.ends(), num_min_req, timeout=...)` (**:1337**) with, in - the sim branch, `committed = self._sync_sim_recv_first_k(channel, channel.ends(), num_min_req)` (base at - `syncfl/top_aggregator.py:360`, returns ascending-sct list of `(msg, md)`, already stamps - PROP_CLIENT_TASK_TRAIN_DURATION + advances vclock), then feed each through - `_process_single_trainer_message`, breaking at `_agg_goal_cnt >= _agg_goal`. Apply - `_barrier_anchored_lags(durs)` (`syncfl:472`) over the committed WALL_SEND completions for U6 telemetry. - Real branch unchanged. -- **Launchers** (subtask 5): add `time_mode: simulated` sibling yamls of - `expt_scripts/{fluxtune,fwdllm,fwdllm_plus}_n10_smoke.yaml` (current set `time_mode: real` at - fluxtune:40 / fwdllm:35 / fwdllm_plus:39). Keep the real ones for reference runs. -- **Pytests** (subtask 6): see §I.8. No broker in this env -> pytest-only, drive `_sim_recv_min_grad` / - `_sync_sim_recv_first_k` with a fake channel + synthetic out-of-order msgs incl. a variance-FAIL rollback - cycle. Extend `tests/mode/test_fwdllm_trainer_sim_duration.py` for the trainer stamping + no-sleep assert. - -### J.4 Open decisions taken during execution (record) -- time_mode default = `"real"` on the trainer getattr fallback (see J.1(1)); revisit if a sim run without an - explicit yaml value is ever needed. -- `_sim_recv_min_grad` deliberately does NOT touch selector slots per-commit; ALL slot release + buffer + - committed-mark clearing happens at the agg-goal boundary (`_release_sim_slots_at_agg_goal`). This is the - fwdllm-specific divergence from asyncfl's per-commit release and the crux of rollback-safety. -- `_sim_buffer.clear()` at the boundary DROPS any stranded arrived-but-uncommitted grad. Benign at syn_0 - (|selected| ~= agg_goal, all commit). If a future rung shows lost updates, revisit (commit-then-carry - instead of drop). Flagged as a fidelity watch-point. +## §I Batch 2 implementation map (code-level -- cold-start; NEXT to build) + +**Purpose:** durable engineering map so Batch 2 can start cold in a new context. Batch 2 = the +**variance-cadence rung layer** (V1-V5 / DK1-DK3 / G1-G2 in the shared parity engine) + **fluxtune oort +selection-fidelity validation**. It is a **checker + telemetry** batch, NOT a clock port -- the vclock is +already driven (Batch 1). Config-gated nothing new here (the rungs are pure functions over telemetry; they +only run when a `parity_checks` battery is invoked). **Re-grep line numbers before editing.** + +### I.1 The prize + the localization discipline (read first) +fwdllm's headline risk (§G) is that the **variance-gated cadence is feedback-compounding**: a tiny +per-cycle grad-pool order difference -> a different `iterations_per_data_id` -> a different committed- +data_id throughput, which may only surface at 3 h. The V/DK/G rungs exist to **catch and localize** that. +**Never fix an EMERGENT rung directly** (PARITY.md §1 / §F.4): walk down to the lowest variance-cadence +rung whose *inputs* are matched. `var_threshold` / `max_iterations_per_data_id` are **baseline-defining +config knobs, not parity levers** -- a cadence gap is always an upstream set/order/clock divergence. Rung +definitions + deps live in **PARITY.md §F.3 (modified K3a/K3b/K2/U3/K8/U2) and §F.4 (new V/DK/G)**; this +section states only the fwdllm **build deltas** (where the checker + telemetry go). + +### I.2 Where rungs plug in -- the shared parity engine (VERIFIED) +- Engine: `examples/async_cifar10/scripts/parity/checks.py`. Each rung is a **check function** + `f(real_agg, sim_agg, ...) -> result_dict`, registered two places: + 1. **`run_all_parity(...)`** (`checks.py:~3431`): add `results["v1_iter_per_data_id"] = v1_...(real_agg, + sim_agg)` etc. in a new **"Stage 6' variance cadence"** block (after Stage 5 updates, before/near the + Stage 6 clock block -- deps are Stage-5 ordering + Stage-1 clock). + 2. **The STAGE map** (`checks.py:~3560`, the `{ "rung_name": {"stage":, "role":, "deps": ()} }` dict): + add each new rung with its role/tier + deps from PARITY.md §F.4 (e.g. `v2_var_trajectory` role + MECHANISM deps `("v1_iter_per_data_id",)`; `v5_pass_ratio` EMERGENT; `dk3_eligible_metric` CONTROL + before `dk1_k_trajectory`/`dk2_c_trajectory`; `g1_grad_norm`/`g2_grad_pool_size` EMERGENT). +- Parsed telemetry shape (VERIFIED, `cli.py:74`): the loader hands each check `real_agg["agg_rounds"]` / + `sim_agg["agg_rounds"]` = the list of **agg_round events**, plus `selection_train`, trainer dicts, etc. + So the V/DK rungs read the per-cycle agg_round series. +- Tests: `tests/mode/test_parity_checks.py` (the existing 40-test fwdllm/shared battery). Add per-rung + tests the **same way the batch was gated for Batch 1**: construct synthetic real/sim `agg_rounds` pairs + with a **known-PASS** fixture and a **known-broken** fixture (e.g. a force-commit-rate skew, a var- + trajectory divergence) and assert PASS/FAIL. **Pytest-only, no run.** + +### I.3 Telemetry sources -- what the emitted agg_round ALREADY carries (VERIFIED) +The per-cycle agg_round event is emitted at `fwdllm_aggregator.py:~1372` (`build_agg_round`, +`flame/telemetry/events.py:108`). **VERIFIED** its `extra` dict carries exactly: +`data_id`, `iteration_per_data_id`, `var`, `var_threshold`, `var_good_enough`, `force_commit_planned`, +`is_async` (plus the generic `agg_goal`, `agg_goal_count`, `staleness`, `trainer_speed_s`, +`contributing_trainers`, `agg_observed_s`). NOTE: `_build_dynamic_kc_metrics` (`:~1190`) computes MORE +(`var_pass_rate`, `max_iter_per_data_id`, `n_eligible_train/eval`, `target_iter_per_data_id`) but that +dict feeds the **dynamic_kc controller**, and is **NOT** in the emitted event -- add fields to the `extra` +dict if a rung needs them. Mapping to rungs: +- **V1 (iterations-per-data_id dist / realized dynamic-K):** derive from the emitted + `(data_id, iteration_per_data_id)` series -- max `iteration_per_data_id` reached before each `data_id` + advances. **Present, no new emit.** +- **V2 (per-cycle `var` trajectory):** the emitted `var` per cycle. **Present.** +- **V4 (force-commit / `max_iterations_per_data_id` bypass rate):** the emitted `force_commit_planned` + boolean (aggregator computes `_force_commit_this_cycle` at `:~1266`, `iter+1 >= max_iter`). **Present -- + V4 = mean(`force_commit_planned`) over cycles. No new emit** (correcting an earlier assumption). +- **V5 (variance-pass ratio per window):** derive from the emitted `var_good_enough` boolean series + (pass ratio = fraction True per window). **Present** (or add `var_pass_rate` to `extra` if you want it + precomputed). +- **V3 (`cached_v` pool size over time, DIAG) + G2 (grad_pool size at commit):** `grad_pool` (`:286`) and + `cached_shared_grad_pool_trainable` (fwdllm `cached_v`, `:287`) sizes are **NOT emitted** -> **NEW emit**: + add `grad_pool_size` + `cached_v_size` to the agg_round `extra` if V3/G2 are exercised. +- **DK1/DK2 (`_agg_goal` / `dynamic_c` trajectory):** inert for all three current baselines (dynamic_kc + disabled) -> **DK rungs WARN/skip**. **DK3** (CONTROL, eligible-ends metric fed to the policy) lives in + the controller metrics dict, not the event -> add `n_eligible_train`/`n_eligible_eval` to `extra` to + wire DK3. Do NOT fork the shared controller. +- **G1 (per-update grad/JVP norm or SNR):** NOT emitted per update. Grad values are mode-invariant given + identical input+perturbation seed, so G1 should be ~0 -- a FAIL means a perturbation seed/order leaked. + **NEW per-update emit** (trainer-side grad norm, into `trainer_round` or a new event) IF G1 is exercised; + else defer with a **logged skip** (§G "no silent caps"), do not silently drop. + +### I.4 fluxtune oort selection-fidelity -- VALIDATION, not a rebuild +The oort stack (`flame/selector/oort.py`, faithful pacer/UCB/D1-D8) is landed and the selection rungs +already exist in the engine STAGE map (`selection_detail`, `residence`, `selection_bias`, `selector_score`, +`preferred_duration`, `participation`, `decision_determinism`, `selection` -- stage 3). Batch-2 work: +**confirm they fire for fluxtune** (`async_oort` selector) and are correctly **inert (WARN/skip) for the +two `random`-selector baselines** (fwdllm/fwdllm_plus). Any gap is fwdllm-specific *wiring* (e.g. a +selection event field the fwdllm aggregator does not stamp), NOT an oort rebuild. Exit: A2c PASS + +`preferred_duration` (Sd) binding real~=sim + no pacer ratchet (fluxtune); S-rungs WARN for `random`. + +### I.5 Batch-2 pytest gate (pytest-only, no run) + exit rungs +1. Per new rung: known-PASS + known-broken synthetic `agg_rounds` fixtures in `test_parity_checks.py` + (force-commit-rate skew, var-trajectory divergence, iter-per-data_id divergence). +2. Any new emitted field (V4 `force_commit`, V3 `grad_pool_size`/`cached_v_size`, G1 grad norm): a small + emission test asserting the field lands in the agg_round/trainer_round event. +3. Full `test_fwdllm_*` + `test_parity_checks` green; flag-off byte-identical (adding fields must not + change real-mode behavior). +**Exit rungs (PARITY.md §F.4):** V1/V2/V5 PASS; K2 (committed-data_ids/vsec) PASS; V4 bypass-rate ~= real; +A2c PASS + Sd binding + no pacer ratchet (fluxtune); DK tracks-or-skips as configured. + +### I.6 After Batch 2 -- the Phase-1 sign-off run (first §H entry) +Only after Batches 1-2 are pytest-green: one smoke (~5 min) then one convergence run **per baseline** at +syn_0 (the `*_n10_smoke_sim.yaml` launchers + their real siblings) -> `parity_checks` full battery + +**C1/C2 at matched `data_id`**. **Enable modeled delays (D>0) in BOTH real and sim together** for the +convergence run (K-D8: the sim smokes keep D=0 for mechanics-only comparability; the parity run needs +D>0). Record in §H; gate to Phase 2 (Batch 3, availability). --- diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index c5300be37..dcbcafcc6 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -71,6 +71,9 @@ class _FakeAggregator: def __init__(self, contributors, var_good_enough, staleness_map=None, total_data_bins=150): + # Batch 1 flag-off regression: real path skips the sim boundary hook + # (_release_sim_slots_at_agg_goal), so telemetry is byte-identical. + self.simulated = False self._per_agg_trainer_list = list(contributors) self._agg_goal_cnt = len(contributors) self._agg_goal = len(contributors) or 1 diff --git a/lib/python/tests/mode/test_fwdllm_recv_timeout.py b/lib/python/tests/mode/test_fwdllm_recv_timeout.py index bcd487fb8..701ae934a 100644 --- a/lib/python/tests/mode/test_fwdllm_recv_timeout.py +++ b/lib/python/tests/mode/test_fwdllm_recv_timeout.py @@ -52,6 +52,8 @@ class _FakeAggregator: _aggregate_grads_async = TopAggregator._aggregate_grads_async def __init__(self, channel): + # Batch 1 flag-off: real path uses next(recv_fifo(...,1,timeout=...)). + self.simulated = False self.cm = _FakeChannelManager(channel) def test_recv_fifo_called_with_recv_timeout_wait_s(self): @@ -74,6 +76,8 @@ class _FakeAggregator: ) def __init__(self, channel, agg_goal=3): + # Batch 1 flag-off: real barrier path (unchanged recv_fifo drain). + self.simulated = False self._agg_goal = agg_goal self.ends_not_selected_yet = False self._round = 1 diff --git a/lib/python/tests/mode/test_fwdllm_rounds_termination.py b/lib/python/tests/mode/test_fwdllm_rounds_termination.py index 88cd9b975..1577a72d4 100644 --- a/lib/python/tests/mode/test_fwdllm_rounds_termination.py +++ b/lib/python/tests/mode/test_fwdllm_rounds_termination.py @@ -36,6 +36,10 @@ class _FakeAggregator: machinery stubbed out.""" def __init__(self, round_, rounds, total_data_bins=150, var_good_enough=True): + # Batch 1 flag-off regression: real path (time_mode: real). With + # simulated=False the new sim boundary hook (_release_sim_slots_at_agg_goal) + # is skipped, so _process_aggregation_goal_met behaves byte-identically. + self.simulated = False self._agg_goal = 2 self._agg_goal_cnt = 2 self._per_agg_trainer_list = [] diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py new file mode 100644 index 000000000..3740253cb --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py @@ -0,0 +1,249 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 1 (simulate_fwdllm.md §I.8/§J.3): the async grad-loop sim drive. + +fwdllm_aggregator._aggregate_grads_async committed a grad on WALL arrival; in +simulated mode it now commits by the modeled sim_completion_ts via +_sim_recv_min_grad -- an sct-ordered reorder buffer + a one-in-flight gate + +a virtual-clock advance -- so ordering is immune to physical arrival jitter. +These tests drive _sim_recv_min_grad directly with synthetic out-of-order grad +messages (a fake channel), and cover the fwdllm-specific rollback-safety of +_release_sim_slots_at_agg_goal (a data_id spans many agg-goal cycles; a +variance-FAIL rolls back to the same data_id and must NOT strand or double- +commit a grad). +""" + +from datetime import datetime + +from flame.mode.horizontal.asyncfl.top_aggregator import ( + TopAggregator as _AsyncBase, +) +from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator +from flame.mode.horizontal.syncfl.top_aggregator import TopAggregator as _SyncBase +from flame.mode.message import MessageType +from flame.sim.virtual_clock import SimReorderBuffer, VirtualClock + + +class _FakeGradChannel: + """Serves at most one grad message per end (as fwdllm's one-message-per-call + loop expects). A message with release_at=k is only yielded once recv_fifo + has been probed more than k times -- this models a straggler whose modeled + completion is earlier but whose physical arrival is later, which is exactly + what the in-flight gate must wait for.""" + + def __init__(self, ends): + self._ends_set = set(ends) + self._msgs = {} # end -> message dict + self._release_at = {} # end -> probe index gating physical arrival + self._delivered = set() + self._probe = 0 + + def add_msg(self, end, sct, budget=None, release_at=0): + m = {MessageType.SIM_COMPLETION_TS: sct} + if budget is not None: + m[MessageType.TRAINING_BUDGET_S] = budget + self._ends_set.add(end) + self._msgs[end] = m + self._release_at[end] = release_at + + def ends(self, state=None): + return list(self._ends_set) + + def has(self, end): + return end in self._ends_set + + def recv_fifo(self, end_ids, first_k=None, timeout=None): + self._probe += 1 + for e in end_ids: + if e in self._delivered or e not in self._msgs: + continue + if self._probe > self._release_at.get(e, 0): + self._delivered.add(e) + yield self._msgs[e], (e, datetime.now()) + + +class _FakeGradAgg: + """Binds the real sim grad-loop methods onto a minimal stand-in.""" + + _sim_recv_min_grad = TopAggregator._sim_recv_min_grad + _release_sim_slots_at_agg_goal = TopAggregator._release_sim_slots_at_agg_goal + _advance_sim_clock = _SyncBase._advance_sim_clock + _sim_recv_grace_s = _SyncBase._sim_recv_grace_s + _sim_hold_busy_slots = _AsyncBase._sim_hold_busy_slots + # grace-window class knobs the base method reads off self + SIM_RECV_GRACE_FLOOR_S = 0.0 + SIM_RECV_GRACE_FACTOR = 0.0 + + def __init__(self): + self.simulated = True + self._round = 1 + self._vclock = VirtualClock() + self._sim_buffer = SimReorderBuffer() + self._sim_committed = set() + self._sim_inflight_expected = {} + self._sim_trainer_budget = {} + self._sim_budget_min = 12.0 + self._sim_fill_ema = 0.0 + + def _drain(self, channel, recv_ends, n): + """Commit n grads, returning the ordered list of committed scts.""" + scts = [] + for _ in range(n): + msg, md = self._sim_recv_min_grad(channel, recv_ends) + if not msg: + break + scts.append(msg[MessageType.SIM_COMPLETION_TS]) + return scts + + +class TestSctOrderedCommit: + def test_commits_in_sct_order_not_arrival_order(self): + agg = _FakeGradAgg() + ch = _FakeGradChannel([]) + # Physical arrival order A,B,C but modeled completion order B nothing committable, no re-commit. + msg2, _ = agg._sim_recv_min_grad(ch, ["X"]) + assert msg2 is None + + def test_boundary_clear_lets_same_end_recontribute_after_rollback(self): + agg = _FakeGradAgg() + # --- cycle 1: X commits on data_id d, iteration 0 --- + agg._sim_inflight_expected = {"X": 10.0} + ch1 = _FakeGradChannel([]) + ch1.add_msg("X", sct=10.0) + m1, _ = agg._sim_recv_min_grad(ch1, ["X"]) + assert m1[MessageType.SIM_COMPLETION_TS] == 10.0 + assert agg._sim_committed == {"X"} + + # --- variance FAIL -> rollback: agg-goal boundary cleanup --- + agg._release_sim_slots_at_agg_goal(ch1, is_async=False) + assert agg._sim_committed == set() # not stranded as committed + assert len(agg._sim_buffer) == 0 # no stranded grad + assert agg._sim_inflight_expected == {} + + # --- cycle 2: SAME data_id, iteration 1, X re-contributes --- + agg._sim_inflight_expected = {"X": 25.0} # re-armed at dispatch + ch2 = _FakeGradChannel([]) + ch2.add_msg("X", sct=25.0) + m2, _ = agg._sim_recv_min_grad(ch2, ["X"]) + # X is committed again (NOT skipped as a stale committed mark). + assert m2[MessageType.SIM_COMPLETION_TS] == 25.0 + assert agg._sim_committed == {"X"} + assert agg._vclock.now == 25.0 + + +class _FakeEnd: + def set_property(self, *_args, **_kwargs): + pass + + +class _FakeSelector: + def __init__(self, selected): + self.requester = "agg" + self.all_selected = {e: 0.0 for e in selected} + self.selected_ends = {"agg": set(selected)} + + +class _FakeSelChannel(_FakeGradChannel): + def __init__(self, ends): + super().__init__(ends) + self._selector = _FakeSelector(ends) + self._ends = {e: _FakeEnd() for e in ends} + + +class TestAsyncBoundaryReleasesSlots: + def test_async_boundary_frees_every_committed_slot(self): + agg = _FakeGradAgg() + agg._sim_pending_commit = set() + agg._sim_inflight_residence = False + ch = _FakeSelChannel(["X", "Y"]) + agg._sim_committed = {"X", "Y"} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + # buffer empty -> nothing held -> both slots released for re-selection. + assert ch._selector.all_selected == {} + assert ch._selector.selected_ends["agg"] == set() + assert agg._sim_committed == set() + + +class TestFlagOffNoOp: + def test_release_is_noop_when_not_simulated(self): + agg = _FakeGradAgg() + agg.simulated = False + agg._sim_committed = {"X"} + agg._sim_inflight_expected = {"X": 1.0} + agg._sim_buffer.add("X", 1.0, None) + + # Real mode: the boundary hook must not touch any sim state. + agg._release_sim_slots_at_agg_goal(channel=None, is_async=False) + + assert agg._sim_committed == {"X"} + assert agg._sim_inflight_expected == {"X": 1.0} + assert len(agg._sim_buffer) == 1 diff --git a/lib/python/tests/mode/test_fwdllm_sim_sync_barrier.py b/lib/python/tests/mode/test_fwdllm_sim_sync_barrier.py new file mode 100644 index 000000000..39546f60b --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sim_sync_barrier.py @@ -0,0 +1,150 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 1 (simulate_fwdllm.md §I.8/§J.3): the SYNC-barrier sim drive. + +fwdllm/fwdllm_plus are sync -- their primary commit path is the barrier +(sync_collect_and_accumulate_grads). In simulated mode it commits the k +trainers with the SMALLEST modeled sim_completion_ts (the k that would +physically finish first in real), advancing the virtual clock to the k-th +smallest -- immune to physical arrival jitter -- via _sync_sim_recv_first_k. +The per-update U6 visibility lag on a strict barrier (lag_i = max_completion - +completion_i) is computed by _barrier_anchored_lags. These tests drive both +directly with the availability gate OFF (byte-identical to no-avail). +""" + +from datetime import datetime, timedelta + +from flame.mode.horizontal.syncfl.top_aggregator import TopAggregator as _SyncBase +from flame.mode.message import MessageType +from flame.sim.virtual_clock import SimReorderBuffer, VirtualClock +from flame.selector.properties import ( + PROP_CLIENT_TASK_TRAIN_DURATION, + PROP_SIM_SEND_TS, +) + + +class _FakeBarrierChannel: + """Drains every selected end in one recv_fifo pass (as the barrier does), + yielding each end's single grad message out of sct order to prove the + commit set is chosen by sct, not arrival.""" + + def __init__(self): + self._msgs = {} # end -> message dict + self._delivered = set() + self._end_props = {} # (end, prop) -> value + + def add_msg(self, end, sct, dur=None): + m = {MessageType.SIM_COMPLETION_TS: sct} + if dur is not None: + m[MessageType.SIM_CLIENT_TASK_TRAIN_DURATION_S] = dur + self._msgs[end] = m + + def ends(self, state=None): + return list(self._msgs) + + def has(self, end): + return end in self._msgs + + def recv_fifo(self, end_ids, first_k=None, timeout=None): + for e in end_ids: + if e in self._delivered or e not in self._msgs: + continue + self._delivered.add(e) + yield self._msgs[e], (e, datetime.now()) + + def get_end_property(self, end, prop): + return self._end_props.get((end, prop)) + + def set_end_property(self, end, prop, value): + self._end_props[(end, prop)] = value + + +class _FakeBarrierAgg: + """Binds the real sync-barrier sim methods onto a minimal stand-in with the + availability gate OFF (pending_withheld / trainer_event_dict unset).""" + + _sync_sim_recv_first_k = _SyncBase._sync_sim_recv_first_k + _advance_sim_clock = _SyncBase._advance_sim_clock + _sim_recv_grace_s = _SyncBase._sim_recv_grace_s + _note_sim_fill = _SyncBase._note_sim_fill + _sim_reinject_ready_withheld = _SyncBase._sim_reinject_ready_withheld + _sim_withhold_if_unavail = _SyncBase._sim_withhold_if_unavail + _barrier_anchored_lags = staticmethod(_SyncBase._barrier_anchored_lags) + SIM_RECV_GRACE_FLOOR_S = 0.0 + SIM_RECV_GRACE_FACTOR = 0.0 + + def __init__(self): + self._round = 1 + self._vclock = VirtualClock() + self._sim_buffer = SimReorderBuffer() + self._sim_fill_ema = 0.0 + + +class TestSyncFirstKSmallestSct: + def test_commits_the_k_smallest_sct_regardless_of_arrival(self): + agg = _FakeBarrierAgg() + ch = _FakeBarrierChannel() + # arrival order A,B,C,D but modeled completion 40,10,30,20. + ch.add_msg("A", sct=40.0, dur=40.0) + ch.add_msg("B", sct=10.0, dur=10.0) + ch.add_msg("C", sct=30.0, dur=30.0) + ch.add_msg("D", sct=20.0, dur=20.0) + + committed = agg._sync_sim_recv_first_k(ch, ch.ends(), first_k=2) + + scts = [m[MessageType.SIM_COMPLETION_TS] for m, _md in committed] + assert scts == [10.0, 20.0] # the two SMALLEST, ascending + + def test_vclock_advances_to_kth_smallest_no_past_dating(self): + agg = _FakeBarrierAgg() + ch = _FakeBarrierChannel() + for e, s in [("A", 40.0), ("B", 10.0), ("C", 30.0), ("D", 20.0)]: + ch.add_msg(e, sct=s, dur=s) + + agg._sync_sim_recv_first_k(ch, ch.ends(), first_k=2) + + # advanced to the k-th (2nd) smallest = 20.0, never past-dated below it. + assert agg._vclock.now == 20.0 + + def test_stamps_client_task_train_duration_for_committed(self): + agg = _FakeBarrierAgg() + ch = _FakeBarrierChannel() + ch.add_msg("B", sct=10.0, dur=10.0) + ch.add_msg("A", sct=40.0, dur=40.0) + + agg._sync_sim_recv_first_k(ch, ch.ends(), first_k=1) + + # B committed -> its intrinsic train duration is exposed to the selector. + assert ch.get_end_property("B", PROP_CLIENT_TASK_TRAIN_DURATION) == ( + timedelta(seconds=10.0) + ) + # A dropped (past the first_k quota) -> not stamped. + assert ch.get_end_property("A", PROP_CLIENT_TASK_TRAIN_DURATION) is None + + def test_falls_back_to_sct_minus_send_when_no_duration_field(self): + agg = _FakeBarrierAgg() + ch = _FakeBarrierChannel() + ch.add_msg("B", sct=10.0) # no SIM_CLIENT_TASK_TRAIN_DURATION_S + ch.set_end_property("B", PROP_SIM_SEND_TS, 3.0) + + agg._sync_sim_recv_first_k(ch, ch.ends(), first_k=1) + + assert ch.get_end_property("B", PROP_CLIENT_TASK_TRAIN_DURATION) == ( + timedelta(seconds=7.0) # sct(10) - sim_send(3) + ) + + +class TestBarrierAnchoredLags: + def test_lag_is_max_completion_minus_each(self): + lags = _SyncBase._barrier_anchored_lags([2.0, 5.0, 3.0]) + assert lags == [3.0, 0.0, 2.0] # barrier = 5.0 + + def test_none_safe_for_missing_completions(self): + lags = _SyncBase._barrier_anchored_lags([2.0, None, 5.0]) + assert lags == [3.0, None, 0.0] + + def test_all_none_yields_all_none(self): + assert _SyncBase._barrier_anchored_lags([None, None]) == [None, None] + + def test_empty_is_empty(self): + assert _SyncBase._barrier_anchored_lags([]) == [] diff --git a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py index 9e5863e4f..94baa1e9c 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py @@ -24,6 +24,7 @@ ), ) +import FedSgdTrainer as _fst_module # noqa: E402 from FedSgdTrainer import FedSGDTrainer # noqa: E402 @@ -34,11 +35,14 @@ class _FakeTrainer: _emulate_training_delay = FedSGDTrainer._emulate_training_delay def __init__(self, training_delay_enabled, training_delay_s=0.0, - training_delay_factor=1.0, speedup_factor=1.0): + training_delay_factor=1.0, speedup_factor=1.0, simulated=False): self.training_delay_enabled = training_delay_enabled self.training_delay_s = training_delay_s self.training_delay_factor = training_delay_factor self.speedup_factor = speedup_factor + # Batch 1: real mode (simulated=False) sleeps D; sim mode skips the + # sleep but still returns the same modeled D. + self.simulated = simulated self.trainer_id = "t1" @@ -65,3 +69,117 @@ def test_speedup_factor_scales_the_returned_delay(self): ) # eval_delay = 10.0 / 2.0 = 5.0; slept = 5.0 / 5.0 = 1.0 assert t._emulate_training_delay() == 1.0 + + +class TestNoSleepOnSimPath: + """Batch 1 core invariant: in simulated mode the trainer must NOT + time.sleep to emulate the delay -- the aggregator advances a virtual clock + instead -- yet must STILL return the same modeled D so the additive + sim_round_duration_s = real_gpu + D is identical across modes.""" + + def test_real_mode_sleeps_the_modeled_delay(self, monkeypatch): + slept = [] + monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) + t = _FakeTrainer( + training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0, simulated=False, + ) + d = t._emulate_training_delay() + # eval_delay = 4.0/2.0 = 2.0; slept = 2.0/1.0 = 2.0 + assert d == 2.0 + assert slept == [2.0] + + def test_sim_mode_does_not_sleep_but_still_returns_delay(self, monkeypatch): + slept = [] + monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) + t = _FakeTrainer( + training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0, simulated=True, + ) + d = t._emulate_training_delay() + assert d == 2.0 # same modeled D as real mode + assert slept == [] # but NOTHING was slept on the sim path + + def test_sim_mode_disabled_returns_zero_no_sleep(self, monkeypatch): + slept = [] + monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) + t = _FakeTrainer(training_delay_enabled="False", training_delay_s=9.0, + simulated=True) + assert t._emulate_training_delay() == 0.0 + assert slept == [] + + +class _FakeTime: + """Scripted time source so the additive-stamp arithmetic is deterministic. + Rebound only onto the FedSgdTrainer module's `time` name (not the shared + time module), so timer_decorator's own runtime.time is untouched.""" + + def __init__(self, ticks): + self._ticks = list(ticks) + self._i = 0 + + def time(self): + v = self._ticks[self._i] + self._i = min(self._i + 1, len(self._ticks) - 1) + return v + + def sleep(self, _s): # must never be called on the sim path + raise AssertionError("time.sleep called on the simulated path") + + +class _StampTrainer: + """Binds the real train_with_data_id onto a minimal stand-in, stubbing the + heavy compute so only the sim-stamp arithmetic is exercised.""" + + train_with_data_id = FedSGDTrainer.train_with_data_id + + def __init__(self, sim_send_ts, delay_d, leg_s=0.0): + self.simulated = True + self.abort_training = False + self.trainer_id = "t1" + self._round = 7 + self.data_id = 3 + self.iteration_per_data_id = 0 + self._sim_send_ts = sim_send_ts + self.sim_completion_leg_s = leg_s + self._sim_completion_ts = None + self._sim_round_duration_s = None + self._delay_d = delay_d + + def _check_availability(self): + return True + + def _perform_training(self): + pass # no GPU work; wall time is scripted via _FakeTime + + def _emulate_training_delay(self): + return self._delay_d # modeled D (no sleep in sim mode) + + +class TestSimCompletionStampIsAdditive: + """The sct the aggregator orders by must be ADDITIVE + (sim_round_duration = real_gpu + D), matching fwdllm real mode's + sleep-D-on-top-of-GPU semantics -- NOT cifar10's max(gpu, D). And + _sim_completion_ts = _sim_send_ts + sim_round_duration + leg (K-D2).""" + + def test_additive_round_duration_and_completion_ts(self, monkeypatch): + monkeypatch.setattr(_fst_module.telemetry, "is_enabled", lambda: False) + # round_start=100.0, gpu-end=100.5 -> real_gpu = 0.5s. + monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.5])) + t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=0.0) + + t.train_with_data_id() + + # ADDITIVE: 0.5 (gpu) + 2.0 (D) = 2.5 (max(gpu,D) would be 2.0) + assert t._sim_round_duration_s == 2.5 + # completion = send(10.0) + duration(2.5) + leg(0.0) + assert t._sim_completion_ts == 12.5 + + def test_completion_ts_includes_leg(self, monkeypatch): + monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.5])) + t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=1.5) + + t.train_with_data_id() + + assert t._sim_round_duration_s == 2.5 + assert t._sim_completion_ts == 14.0 # 10.0 + 2.5 + 1.5 From 8fa6598bdb050829e16ed70d100b93c3e89d5588 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 00:37:59 -0400 Subject: [PATCH 05/56] fwdllm sim Batch 2: variance-cadence rung layer (V/DK/G) + cadence telemetry MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the fwdllm variance-gated dynamic-K parity layer (PARITY.md §F.4) to the shared parity engine, plus the telemetry it reads. Checker (parity/checks.py, run_all_parity, CHECK_META): - Stage 6': v1_iter_per_data_id, v2_var_trajectory, v3_cached_v_pool (DIAG), v4_force_commit_rate, v5_variance_pass_ratio (force-commits excluded) - Stage 3': dk1_agg_goal_trajectory (SKIP on fixed-K), dk2_dynamic_c, dk3_eligible_ends_metric - Stage 7': g1_grad_norm, g2_grad_pool_size All SKIP cleanly on non-fwdllm runs (async_cifar10 ladder unaffected). Telemetry (fwdllm_aggregator.py): add cycle_data_id/cycle_iteration/ grad_pool_size/cached_v_size to the agg_round extra, snapshotted BEFORE the pass/fail branch mutates data_id/iteration. The existing post-mutation data_id/iteration_per_data_id are left unchanged (analyzer depends on them; real-mode byte-identical). This makes V1 = count(cycles) grouped by cycle_data_id exact for natural-pass and force-commit paths (§K-D9). DK3/G1 emit deferred with a logged SKIP -- inert for all three current baselines (§K-D10); the checker rungs read-if-present and are unit-tested both ways. Tests: per-rung known-PASS/known-broken cadence fixtures + run_all_parity fwdllm smoke + non-fwdllm-SKIP (test_parity_checks.py); driven pre-mutation snapshot assertions (test_fwdllm_agg_telemetry.py). Gate green: test_fwdllm_* 81, test_parity_checks 63, parity sub-package 115. Deviation log §K-D9/D10 + Batch-2 status updated in simulate_fwdllm.md. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 360 ++++++++++++++++++ .../async_cifar10/scripts/parity_checks.py | 12 + lib/python/examples/fwdllm/simulate_fwdllm.md | 81 +++- .../horizontal/syncfl/fwdllm_aggregator.py | 27 ++ .../tests/mode/test_fwdllm_agg_telemetry.py | 44 +++ lib/python/tests/mode/test_parity_checks.py | 239 ++++++++++++ 6 files changed, 751 insertions(+), 12 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 984939709..e04e24d04 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -3424,6 +3424,338 @@ def _curve(evs): "eval_rounds_compared": len(diffs), "loss_tol": loss_tol}, budget_s) +# ═══════════════════════════════════════════════════════════════════ +# §F FwdLLM variance-cadence layer (PARITY.md §F.4) — V/DK/G rungs +# ═══════════════════════════════════════════════════════════════════ +# +# fwdllm's commit cadence is ENDOGENOUS: at each agg-goal boundary aggregate() +# computes a gradient-pool `var`; var<=var_threshold commits (advance data_id, +# eval, clear cached_v) else rolls back and retries the same data_id. So +# updates-per-data_id is a random variable of the variance trajectory. These +# rungs verify that trajectory is mode-invariant given matched inputs — the +# fwdllm-specific layer the async_cifar10 ladder does not model. Never fix an +# EMERGENT rung directly: walk to the lowest rung whose *inputs* are matched. +# All read the per-cycle agg_round series (fwdllm_aggregator emits one event +# per agg-goal boundary carrying cycle_data_id / cycle_iteration / var / +# var_threshold / var_good_enough / force_commit_planned / grad_pool_size / +# cached_v_size). Inputs source: fwdllm_aggregator.py _process_aggregation_goal_met. + + +def _fwd_cadence_cycles(agg: dict) -> list: + """Ordered per-cycle agg_round events carrying fwdllm cadence fields. + + A cycle is one agg-goal boundary (a variance gate). Non-fwdllm runs (no + `var_good_enough`/`cycle_data_id`) yield [] -> the V/DK/G rungs SKIP. + """ + return [e for e in agg.get("agg_rounds", []) + if "cycle_data_id" in e or "var_good_enough" in e] + + +def _iters_per_data_id(cycles: list) -> dict: + """{cycle_data_id -> #cycles spent on it} = realized dynamic-K per data_id. + + Exact for both natural-pass and force-commit paths because each agg-goal + boundary emits exactly one cadence event tagged with the data_id it worked + on (cycle_data_id), pre-advance (§K-D9).""" + out: dict = {} + for e in cycles: + d = e.get("cycle_data_id") + if d is None: + continue + out[d] = out.get(d, 0) + 1 + return out + + +def iters_per_data_id_parity(real: dict, sim: dict, ks_tol: float = 0.2, + mean_tol_rel: float = 0.15) -> dict: + """V1 [DIST]: iterations-per-data_id distribution (realized dynamic K). + + The number of accumulation cycles a data_id needs to pass the variance gate. + A divergence means the contributing set/order (U5/U4 upstream) differs, so + the accumulated grad-pool composition — and thus the variance trajectory — + differs. KS on the per-data_id iteration counts + a mean guard. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + if not rc or not sc: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no fwdllm cadence events (non-fwdllm run or telemetry absent)"} + r_iters = list(_iters_per_data_id(rc).values()) + s_iters = list(_iters_per_data_id(sc).values()) + if not r_iters or not s_iters: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no cycle_data_id in cadence events — cannot bin by data_id"} + ks = ks_stat(s_iters, r_iters) + r_mean, _ = mean_std(r_iters) + s_mean, _ = mean_std(s_iters) + mean_rel = (abs(r_mean - s_mean) / max(r_mean, s_mean) + if max(r_mean, s_mean) > 0 else 0.0) + return { + "ok": ks <= ks_tol and mean_rel <= mean_tol_rel, + "tier": "DIST", + "real_mean_iters": round(r_mean, 3), + "sim_mean_iters": round(s_mean, 3), + "mean_rel_diff": round(mean_rel, 3), + "ks_stat": round(ks, 3), + "ks_tol": ks_tol, + "mean_tol_rel": mean_tol_rel, + "n_real_data_ids": len(r_iters), + "n_sim_data_ids": len(s_iters), + } + + +def var_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: + """V2 [DIST]: per-cycle `var` trajectory distribution. + + With V1's inputs matched, the variance *signal* itself must match; a + divergence with matched iterations points at a grad-pool accumulation-order + bug (a true sim bug, not an input divergence). KS over the per-cycle var + values (None dropped — a cycle before the first gate has no var). + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_var = [e["var"] for e in rc if e.get("var") is not None] + s_var = [e["var"] for e in sc if e.get("var") is not None] + if not r_var or not s_var: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no non-null `var` in cadence events"} + ks = ks_stat(s_var, r_var) + r_mean, _ = mean_std(r_var) + s_mean, _ = mean_std(s_var) + return { + "ok": ks <= ks_tol, + "tier": "DIST", + "real_mean_var": round(r_mean, 6), + "sim_mean_var": round(s_mean, 6), + "ks_stat": round(ks, 3), + "ks_tol": ks_tol, + "n_real_cycles": len(r_var), + "n_sim_cycles": len(s_var), + } + + +def cached_v_pool_parity(real: dict, sim: dict, ks_tol: float = 0.25) -> dict: + """V3 [DIAG]: `cached_v` (carried aggregated grad-pool) size over time. + + A rollback/cache bookkeeping divergence looks like a variance bug but is + stateful accounting — this DIAG localizes it. cached_v carries across + variance-FAIL rollbacks and clears on a commit; its size trajectory should + match once V1 matches. SKIP if the field was not emitted. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_sz = [e["cached_v_size"] for e in rc if e.get("cached_v_size") is not None] + s_sz = [e["cached_v_size"] for e in sc if e.get("cached_v_size") is not None] + if not r_sz or not s_sz: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no cached_v_size in cadence events (field not emitted)"} + ks = ks_stat(s_sz, r_sz) + return { + "ok": ks <= ks_tol, + "tier": "DIAG", + "real_mean_cached_v": round(sum(r_sz) / len(r_sz), 3), + "sim_mean_cached_v": round(sum(s_sz) / len(s_sz), 3), + "ks_stat": round(ks, 3), + "ks_tol": ks_tol, + } + + +def force_commit_rate_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: + """V4 [DIST]: force-commit frequency (max_iterations_per_data_id bypass rate). + + The fraction of cycles that hit the iteration cap and force-commit despite a + failed variance gate. A rate divergence = chronic variance divergence (the + cap fires at a different frequency), not a separate bug — walk to V1. + var_threshold / max_iterations_per_data_id are baseline-defining config + knobs, NOT parity levers. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + if not rc or not sc: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no fwdllm cadence events"} + r_rate = sum(1 for e in rc if e.get("force_commit_planned")) / len(rc) + s_rate = sum(1 for e in sc if e.get("force_commit_planned")) / len(sc) + return { + "ok": abs(r_rate - s_rate) <= tol, + "tier": "DIST", + "real_force_commit_rate": round(r_rate, 4), + "sim_force_commit_rate": round(s_rate, 4), + "abs_diff": round(abs(r_rate - s_rate), 4), + "tol": tol, + "n_real_cycles": len(rc), + "n_sim_cycles": len(sc), + } + + +def variance_pass_ratio_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: + """V5 [DIST]: genuine variance-pass ratio (the rollup feeding DynamicKC). + + A *genuine* pass is var<=var_threshold — a force-commit (var>threshold, + committed only because the iteration cap fired) is NOT a variance pass and is + excluded, so V5 tracks the true gate-pass rate the DynamicKC policy consumes. + EMERGENT rollup of V1/V2; localize down, do not tune it. + """ + def _ratio(cycles: list): + n, passes = 0, 0 + for e in cycles: + var, thr = e.get("var"), e.get("var_threshold") + if var is None or thr is None: + continue + n += 1 + if var <= thr: + passes += 1 + return (passes / n if n else None), n + + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_ratio, r_n = _ratio(rc) + s_ratio, s_n = _ratio(sc) + if r_ratio is None or s_ratio is None: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no var/var_threshold pairs in cadence events"} + return { + "ok": abs(r_ratio - s_ratio) <= tol, + "tier": "DIST", + "real_pass_ratio": round(r_ratio, 4), + "sim_pass_ratio": round(s_ratio, 4), + "abs_diff": round(abs(r_ratio - s_ratio), 4), + "tol": tol, + "n_real_gated": r_n, + "n_sim_gated": s_n, + } + + +def agg_goal_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: + """DK1 [DIST]: K (`_agg_goal`) trajectory. + + When DynamicKC is enabled it moves K from observed metrics; a divergence + feeds back into cadence. INERT for the fixed-K baselines: if K is constant + and equal across modes there is no dynamic behavior to check -> SKIP so a + fixed-K run does not spuriously PASS/FAIL a mechanism it never exercises. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_k = [e["agg_goal"] for e in rc if e.get("agg_goal") is not None] + s_k = [e["agg_goal"] for e in sc if e.get("agg_goal") is not None] + if not r_k or not s_k: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no agg_goal in cadence events"} + if len(set(r_k)) <= 1 and len(set(s_k)) <= 1: + ok = set(r_k) == set(s_k) + return {"ok": ok, "tier": "DIST", "status": "SKIP", + "note": f"DynamicKC disabled — constant K (real={r_k[0]}, sim={s_k[0]})", + "real_k": r_k[0], "sim_k": s_k[0]} + ks = ks_stat(s_k, r_k) + return { + "ok": ks <= ks_tol, "tier": "DIST", + "real_mean_k": round(sum(r_k) / len(r_k), 3), + "sim_mean_k": round(sum(s_k) / len(s_k), 3), + "ks_stat": round(ks, 3), "ks_tol": ks_tol, + } + + +def dynamic_c_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: + """DK2 [DIST]: C (`dynamic_c`) concurrency-target trajectory. + + SKIP unless a `dynamic_c` field is emitted (DynamicKC enabled). Do not fork + the shared controller per baseline (PARITY.md §F.4 locked principle 5). + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_c = [e["dynamic_c"] for e in rc if e.get("dynamic_c") is not None] + s_c = [e["dynamic_c"] for e in sc if e.get("dynamic_c") is not None] + if not r_c or not s_c: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no dynamic_c in cadence events (DynamicKC disabled)"} + ks = ks_stat(s_c, r_c) + return { + "ok": ks <= ks_tol, "tier": "DIST", + "real_mean_c": round(sum(r_c) / len(r_c), 3), + "sim_mean_c": round(sum(s_c) / len(s_c), 3), + "ks_stat": round(ks, 3), "ks_tol": ks_tol, + } + + +def eligible_ends_metric_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: + """DK3 [DIST/CONTROL]: eligible-ends-count metric fed to the DynamicKC policy. + + Validate the policy *input* before the policy (CONTROL before MECHANISM): a + diverging input means fix the metric, not the policy. SKIP unless the + n_eligible_train/n_eligible_eval counts are emitted — deferred while no + baseline enables DynamicKC (§K-D10), not silently dropped. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_e = [e["n_eligible_train"] for e in rc if e.get("n_eligible_train") is not None] + s_e = [e["n_eligible_train"] for e in sc if e.get("n_eligible_train") is not None] + if not r_e or not s_e: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no n_eligible_train in cadence events (DK3 emit deferred, §K-D10)"} + ks = ks_stat(s_e, r_e) + return { + "ok": ks <= ks_tol, "tier": "DIST", + "real_mean_eligible": round(sum(r_e) / len(r_e), 3), + "sim_mean_eligible": round(sum(s_e) / len(s_e), 3), + "ks_stat": round(ks, 3), "ks_tol": ks_tol, + } + + +def grad_norm_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: + """G1 [DIST]: per-update grad/JVP norm distribution. + + Gradient values are mode-invariant given identical input + perturbation seed, + so G1 should be ~0; a FAIL means a perturbation seed/order leaked across + modes. SKIP unless a per-update `grad_norm` field is emitted — trainer-side + per-update emit deferred (§K-D10), logged not silently dropped. + """ + def _norms(agg): + out = [] + for e in _fwd_cadence_cycles(agg): + gn = e.get("grad_norm") + if isinstance(gn, list): + out.extend(x for x in gn if x is not None) + elif gn is not None: + out.append(gn) + return out + + r_n, s_n = _norms(real), _norms(sim) + if not r_n or not s_n: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no grad_norm in cadence events (G1 emit deferred, §K-D10)"} + ks = ks_stat(s_n, r_n) + return { + "ok": ks <= ks_tol, "tier": "DIST", + "real_mean_grad_norm": round(sum(r_n) / len(r_n), 6), + "sim_mean_grad_norm": round(sum(s_n) / len(s_n), 6), + "ks_stat": round(ks, 3), "ks_tol": ks_tol, + } + + +def grad_pool_size_parity(real: dict, sim: dict, ks_tol: float = 0.2, + mean_tol_rel: float = 0.15) -> dict: + """G2 [DIST]: grad_pool size at commit (realized contributions per data_id). + + The number of gradients accumulated into the pool when a data_id commits — + an emergent rollup of V1 x K. Measured only on committed cycles + (var_good_enough True). SKIP if grad_pool_size was not emitted. + """ + def _sizes(agg): + return [e["grad_pool_size"] for e in _fwd_cadence_cycles(agg) + if e.get("var_good_enough") and e.get("grad_pool_size") is not None] + + r_sz, s_sz = _sizes(real), _sizes(sim) + if not r_sz or not s_sz: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no grad_pool_size on committed cycles"} + ks = ks_stat(s_sz, r_sz) + r_mean, _ = mean_std(r_sz) + s_mean, _ = mean_std(s_sz) + mean_rel = (abs(r_mean - s_mean) / max(r_mean, s_mean) + if max(r_mean, s_mean) > 0 else 0.0) + return { + "ok": ks <= ks_tol and mean_rel <= mean_tol_rel, + "tier": "DIST", + "real_mean_pool": round(r_mean, 3), + "sim_mean_pool": round(s_mean, 3), + "mean_rel_diff": round(mean_rel, 3), + "ks_stat": round(ks, 3), "ks_tol": ks_tol, "mean_tol_rel": mean_tol_rel, + "n_real_commits": len(r_sz), "n_sim_commits": len(s_sz), + } + + # ═══════════════════════════════════════════════════════════════════ # §4 Consolidated run_all_parity (extended) # ═══════════════════════════════════════════════════════════════════ @@ -3525,6 +3857,21 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["aggregation_sequence"] = aggregation_sequence_parity( real_agg, sim_agg, max_rounds) + # ── Stage 6'/3'/7' FwdLLM variance-cadence layer (PARITY.md §F.4) ── + # Pure functions over the per-cycle agg_round series; SKIP cleanly on + # non-fwdllm runs (no cadence fields emitted). V/G rungs feed off Stage-5 + # ordering + Stage-1 clock; DK rungs are inert unless DynamicKC is enabled. + results["v1_iter_per_data_id"] = iters_per_data_id_parity(real_agg, sim_agg) + results["v2_var_trajectory"] = var_trajectory_parity(real_agg, sim_agg) + results["v3_cached_v_pool"] = cached_v_pool_parity(real_agg, sim_agg) + results["v4_force_commit_rate"] = force_commit_rate_parity(real_agg, sim_agg) + results["v5_variance_pass_ratio"] = variance_pass_ratio_parity(real_agg, sim_agg) + results["dk1_agg_goal_trajectory"] = agg_goal_trajectory_parity(real_agg, sim_agg) + results["dk2_dynamic_c"] = dynamic_c_trajectory_parity(real_agg, sim_agg) + results["dk3_eligible_ends_metric"] = eligible_ends_metric_parity(real_agg, sim_agg) + results["g1_grad_norm"] = grad_norm_parity(real_agg, sim_agg) + results["g2_grad_pool_size"] = grad_pool_size_parity(real_agg, sim_agg) + # ── Stage 7 Statistical utility ── results["utility"] = utility_parity(real_agg, sim_agg) @@ -3618,6 +3965,19 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "commit_promptness": {"stage": 6, "role": "CONTROL", "deps": ("withheld_delivery",)}, "aggregation_sequence": {"stage": 6, "role": "EMERGENT", "deps": ("participation", "inter_arrival_order")}, "first_divergence_summary": {"stage": 6, "role": "DIAG", "deps": ()}, + # ── Stage 6' FwdLLM variance-gated aggregation cadence (PARITY.md §F.4) ── + "v1_iter_per_data_id": {"stage": 6, "role": "MECHANISM", "deps": ("inter_arrival_order",)}, + "v2_var_trajectory": {"stage": 6, "role": "MECHANISM", "deps": ("v1_iter_per_data_id",)}, + "v3_cached_v_pool": {"stage": 6, "role": "DIAG", "deps": ("v1_iter_per_data_id",)}, + "v4_force_commit_rate": {"stage": 6, "role": "MECHANISM", "deps": ("v1_iter_per_data_id",)}, + "v5_variance_pass_ratio": {"stage": 6, "role": "EMERGENT", "deps": ("v1_iter_per_data_id", "v2_var_trajectory")}, + # ── Stage 3' Dynamic K/C trajectory (inert unless DynamicKC enabled) ── + "dk1_agg_goal_trajectory": {"stage": 3, "role": "MECHANISM", "deps": ("v5_variance_pass_ratio",)}, + "dk2_dynamic_c": {"stage": 3, "role": "MECHANISM", "deps": ("v5_variance_pass_ratio",)}, + "dk3_eligible_ends_metric": {"stage": 3, "role": "CONTROL", "deps": ("avail_composition",)}, + # ── Stage 7' Forward-gradient quality ── + "g1_grad_norm": {"stage": 7, "role": "EMERGENT", "deps": ("selection_detail",)}, + "g2_grad_pool_size": {"stage": 7, "role": "EMERGENT", "deps": ("v1_iter_per_data_id", "dk1_agg_goal_trajectory")}, # ── Stage 7 Statistical utility ── "utility": {"stage": 7, "role": "EMERGENT", "deps": ("participation", "phase_gpu_compute", "staleness")}, # ── Stage 8 Emergent outcomes ── diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index a82adaa30..3d57e55d6 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -81,6 +81,18 @@ total_commits_parity, terminal_state_parity, budget_not_cap, + # §F fwdllm variance-cadence layer (V/DK/G rungs) + _iters_per_data_id, + iters_per_data_id_parity, + var_trajectory_parity, + cached_v_pool_parity, + force_commit_rate_parity, + variance_pass_ratio_parity, + agg_goal_trajectory_parity, + dynamic_c_trajectory_parity, + eligible_ends_metric_parity, + grad_norm_parity, + grad_pool_size_parity, # overall run_all_parity, overall_verdict, diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 1a04b9805..c2362f064 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -19,17 +19,22 @@ reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baseli > and trace any real<->sim gap back to the choice that caused it. Deviation log lives in **§K** (running, > newest-last); locked cross-cutting ones also surface in §F "Locked principles" / §J.4. -**Current state -- Phase 1 Batch 1 COMPLETE; Batch 2 is NEXT.** The two structural clock ports are landed -and pytest-gated (75 green). **Done:** the trainer sim path (`time_mode`/`simulated`, additive -`_sim_completion_ts`, no-sleep on the sim path), the async grad loop on the vclock -(`_sim_recv_min_grad` sct reorder buffer + in-flight gate + agg-goal-boundary rollback cleanup), the -sync barrier (`_sync_sim_recv_first_k` first-k-smallest-sct + U6 lags), the sim launchers, and the -Batch-1 pytest gate incl. the flag-off byte-identical regression. Batch-1 build detail is now history -(landed in `ef4cb70e` + follow-ups; deviation rationale in **§K**); the code is the source of truth. -**Still to build:** **(Batch 2)** the variance-cadence rung layer (V/DK/G in the shared parity engine) -+ fluxtune oort selection-fidelity validation -> then the Phase-1 syn_0 sign-off run; **(Batch 3)** the -availability *effect* path + `EVENT_AVAIL_CHANGE` emission (Phase 2); **(Batch 4)** full ladder under -unavailability (Phase 3). See **§E** for the batch list and **§I** for the Batch-2 cold-start map. +**Current state -- Phase 1 Batches 1-2 COMPLETE (checker + telemetry); Phase-1 syn_0 sign-off run is NEXT.** +Batch 1 landed the two structural clock ports; Batch 2 landed the variance-cadence rung layer + its +telemetry, pytest-gated. **Batch 1 done:** trainer sim path (`time_mode`/`simulated`, additive +`_sim_completion_ts`, no-sleep on the sim path), the async grad loop on the vclock (`_sim_recv_min_grad` +sct reorder buffer + in-flight gate + agg-goal-boundary rollback cleanup), the sync barrier +(`_sync_sim_recv_first_k` first-k-smallest-sct + U6 lags), the sim launchers, and the flag-off +byte-identical regression. **Batch 2 done:** the V1-V5 / DK1-DK3 / G1-G2 rungs in the shared parity engine +(`parity/checks.py` + `run_all_parity` + `CHECK_META`), the cycle-relative cadence telemetry they read +(`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size` added to the agg_round `extra`, +**§K-D9**), and their pytest gate (known-PASS + known-broken cadence fixtures + a driven emission test); +DK3/G1 emit deferred with a logged SKIP (**§K-D10**). fluxtune oort selection-fidelity is a **validation** +concern (the selection rungs already exist in the STAGE map, §I.4) folded into the sign-off run. Build +detail is history (code is the source of truth; deviation rationale in **§K**). **Still to build:** the +**Phase-1 syn_0 sign-off run** (smoke -> convergence per baseline -> `parity_checks` full battery, §I.6); +**(Batch 3)** the availability *effect* path + `EVENT_AVAIL_CHANGE` emission (Phase 2); **(Batch 4)** full +ladder under unavailability (Phase 3). See **§E** for the batch list and **§I** for the Batch-2 map. **Prerequisites (read first):** - [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) -- the parity methodology (the ladder §1, roles/ @@ -188,7 +193,14 @@ the code (`ef4cb70e` +) and the deviation rationale **§K-D1..D8**. Cold-start m **§I** (now repurposed as the Batch-2 map). U6 telemetry *emission* deferred (K-D7); the seeded mini-run feeding `parity_checks` is folded into the Phase-1 sign-off run. -#### Batch 2 (large dev, NEXT) -- variance-cadence layer + fluxtune selection fidelity +#### Batch 2 -- variance-cadence layer + fluxtune selection fidelity -- ✅ DONE (checker + telemetry, pytest-gated) +Landed: V1-V5 / DK1-DK3 / G1-G2 in `parity/checks.py` (registered in `run_all_parity` + `CHECK_META`), +the cycle-relative cadence telemetry they read (**§K-D9**), DK3/G1 emit deferred with a logged SKIP +(**§K-D10**). Pytest: `test_parity_checks.py` (per-rung known-PASS/known-broken + `run_all_parity` +fwdllm smoke + non-fwdllm-SKIP) and `test_fwdllm_agg_telemetry.py` (driven pre-mutation snapshot). The +fluxtune selection-fidelity rungs already exist in the STAGE map (§I.4) -> they fire in the sign-off run; +no new checker code. Original Batch-2 plan below kept for the rung-by-rung rationale. + - **Variance-cadence rungs** (PARITY.md §F.4): implement V1-V5 / DK1-DK3 / G1-G2 in the shared engine. Most dev is checker + the telemetry to feed it: per-cycle `var` trajectory (at each agg-goal), iterations-per-`data_id` (realized dynamic-K), force-commit (`max_iterations_per_data_id`) bypass rate, @@ -327,6 +339,13 @@ truth; the as-built decisions are logged in **§K-D1..D8**. Batch-1 pytest files `tests/mode/test_fwdllm_sim_sync_barrier.py` (first-k-smallest + U6 lags), plus the flag-off `simulated=False` fixes across the four pre-existing fwdllm agg/trainer tests. +**Batch 2 landed (done):** variance-cadence rung layer + telemetry, pytest-gated. `test_fwdllm_*` = 81 +green; `test_parity_checks.py` = 63 green (adds the V/DK/G suites); the parity sub-package = 115 green. +As-built decisions in **§K-D9/D10**. Files: `parity/checks.py` (V1-V5/DK1-DK3/G1-G2 + `run_all_parity` + +`CHECK_META`), `parity_checks.py` (re-export shim), `fwdllm_aggregator.py` (cadence `extra` snapshot), +`tests/mode/test_parity_checks.py` + `tests/mode/test_fwdllm_agg_telemetry.py`. **Next §H entry lands +after the Phase-1 syn_0 sign-off run** (§I.6). + --- ## §I Batch 2 implementation map (code-level -- cold-start; NEXT to build) @@ -487,3 +506,41 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent the later convergence/parity runs and MUST be enabled in BOTH the real reference and the sim run together (else real sleeps D while sim charges D to the clock -> that's the point; mismatched enable would be a false divergence). **Where:** the three `*_n10_smoke_sim.yaml` headers. +- **K-D9 Cycle-relative cadence telemetry (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/ + `cached_v_size`) added to the agg_round `extra`, NOT reconstructed from the post-mutation series + (Batch 2).** §I.3 assumed V1 needs "no new emit" -- derive iterations-per-data_id from the emitted + `(data_id, iteration_per_data_id)`. But those fields are emitted **post-mutation**: on a variance PASS the + branch advances `data_id += 1` and zeroes `iteration_per_data_id` BEFORE `build_agg_round`, so a commit + event carries the **next** data_id (and iter 0). Binning cycles by the emitted `data_id` therefore + attributes each commit to the following bin (off-by-one), and a first-try pass emits nothing under its own + data_id. **Options:** (a) keep "no new emit" and reconstruct with off-by-one accounting (commit-maps-to- + previous-data_id, force-commit vs natural-pass both counted) -- fragile, and fwdllm's headline risk (§G) is + exactly this cadence being feedback-compounding and hard to debug; (b) snapshot the cycle identity BEFORE + the pass/fail branch and emit it unambiguously. **Chose (b):** `cycle_data_id`/`cycle_iteration` name the + data_id this cycle worked on + its 0-based attempt index, so **V1 = count(cycles) grouped by + `cycle_data_id`** is exact for both natural-pass and force-commit paths. The existing post-mutation + `data_id`/`iteration_per_data_id` are LEFT UNCHANGED (the analyzer's `progress_key` depends on them -- + adding fields, not changing them, keeps real-mode behavior byte-identical). `grad_pool_size` (G2) and + `cached_v_size` (V3) are snapshotted at the same point (before `_update_state_after_payload_prepared` + clears `grad_pool` on a commit). All four are `getattr`-guarded like `var_threshold` so test doubles + without the pools still emit. **Where:** `fwdllm_aggregator.py::_process_aggregation_goal_met` (snapshot + block before the `if self.var_good_enough:` branch + the `build_agg_round` `extra`); + `parity/checks.py::_iters_per_data_id` / the V/G rung functions; `tests/mode/test_fwdllm_agg_telemetry.py` + (`test_cadence_fields_snapshot_pre_mutation`, `test_cycle_data_id_is_pre_advance_on_commit`). +- **K-D10 DK3 (`n_eligible_train`/`n_eligible_eval`) and G1 (per-update grad/JVP norm) emit DEFERRED -- + checker reads-if-present, logged SKIP otherwise (Batch 2).** §I.3 flagged both as candidate NEW emits. + DK3's eligible-ends metric feeds the DynamicKC policy, which is **disabled for all three current + baselines** (fluxtune fixed K/C; fwdllm/fwdllm_plus have no controller), so the metric is never consumed; + G1's per-update grad norm should be ~mode-invariant (grad values depend only on input+perturbation seed) + and needs a **trainer-side per-update** emit, a larger surface than this checker+agg-telemetry batch. + **Options:** (a) wire both emits now (add the eligible-count loop to every agg_round; add a per-update + trainer emit) despite no active consumer; (b) implement the **checker** rungs to read the fields when + present and **SKIP with a §K-D10-referencing note** when absent, deferring the emit. **Chose (b)** -- + matches the §G "no silent caps" discipline (a SKIP with a logged reason, not a dropped rung) and avoids + adding inert per-cycle overhead / a trainer-telemetry port to a checker batch. The DK3/G1 check functions + are still fully unit-tested (synthetic fixtures supply the fields for the PASS/FAIL paths + the absent- + field SKIP path). DK1 (K trajectory) similarly SKIPs when K is constant across both modes (fixed-K + baselines) rather than trivially passing. **Wire the emits when a DynamicKC baseline (DK3) or a G1 + exercise is added.** **Where:** `parity/checks.py::eligible_ends_metric_parity` / `grad_norm_parity` / + `agg_goal_trajectory_parity`; `tests/mode/test_parity_checks.py` (`TestDK3EligibleEndsMetric`, + `TestG1GradNorm`, `TestDK1AggGoalTrajectory`). diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index e9c81cb7a..c032b44f9 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -1293,6 +1293,27 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): f"force_commit_planned={_force_commit_planned}" ) + # Snapshot the cycle identity BEFORE the pass/fail branch mutates + # data_id/iteration_per_data_id below. The emitted `data_id`/ + # `iteration_per_data_id` fields are post-mutation (a commit advances + # data_id and zeroes iteration, so a commit event carries the NEXT + # data_id) -- fine for the analyzer's progress axis but ambiguous for + # the variance-cadence rungs (V1). `cycle_data_id`/`cycle_iteration` + # unambiguously identify the data_id this cycle worked on and its + # 0-based attempt index, so V1 = count(cycles) grouped by cycle_data_id + # is exact for both natural-pass and force-commit paths (§K-D9). + _cycle_data_id = self.data_id + _cycle_iteration = self.iteration_per_data_id + # Pool sizes at the variance gate (before _update_state_after_payload_ + # prepared clears grad_pool on a commit): grad_pool = realized + # contributions this data_id (G2); cached_v = carried aggregated pool + # across variance-FAIL rollbacks (V3). See §K-D9. getattr-guarded like + # var_threshold so test doubles without the pools still emit. + _grad_pool = getattr(self, "grad_pool", None) + _grad_pool_size = len(_grad_pool) if _grad_pool is not None else None + _cached_v = getattr(self, "cached_shared_grad_pool_trainable", None) + _cached_v_size = len(_cached_v) if _cached_v is not None else 0 + if self.var_good_enough: _pass_kind = ( "FORCE-COMMITTED (max_iter bypass)" @@ -1387,6 +1408,12 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): "var_good_enough": self.var_good_enough, "force_commit_planned": _force_commit_planned, "is_async": is_async, + # Variance-cadence rung inputs (Batch 2, §K-D9): + # cycle-relative identity for V1, pool sizes for V3/G2. + "cycle_data_id": _cycle_data_id, + "cycle_iteration": _cycle_iteration, + "grad_pool_size": _grad_pool_size, + "cached_v_size": _cached_v_size, }, ) telemetry.emit(ev, **fields) diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index dcbcafcc6..7466fc9c6 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -226,6 +226,50 @@ def test_staleness_computed_against_pre_cycle_model_version(self, tmp_path): finally: telemetry.shutdown() + def test_cadence_fields_snapshot_pre_mutation(self, tmp_path): + """Batch-2 variance-cadence inputs (§K-D9): cycle_data_id/cycle_iteration + identify the data_id this cycle WORKED on (pre-advance), and the pool + sizes are captured at the variance gate. On a variance FAIL data_id does + not advance, so cycle_data_id == the emitted (post) data_id == 3.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["cycle_data_id"] == 3 and r["cycle_iteration"] == 0 + # grad_pool got this cycle's grad appended before the snapshot; the + # fake sets no cached_v, so cached_v_size defaults to 0. + assert r["grad_pool_size"] == 1 and r["cached_v_size"] == 0 + finally: + telemetry.shutdown() + + def test_cycle_data_id_is_pre_advance_on_commit(self, tmp_path): + """On a variance PASS the emitted (post) data_id advances to 4, but + cycle_data_id stays 3 -- the data_id this cycle committed. This is the + off-by-one V1 relies on: bin cadence cycles by cycle_data_id, not the + post-mutation data_id (which would attribute a commit to the next bin).""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=True) + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["cycle_data_id"] == 3 # worked-on data_id + assert r["data_id"] == 4 # post-commit advance + finally: + telemetry.shutdown() + def test_contributor_list_captured_before_reset(self, tmp_path): """_per_agg_trainer_list is cleared at the end of this method -- agg_round's contributing_trainers must reflect this cycle's diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 0bfe4ef08..7331c49dd 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -467,3 +467,242 @@ def test_builder_paired_commit_class(self): round_num=1, time_mode="sim", in_flight_before=13, in_flight_after=13, residence_rounds=[1]) assert "residence_staleness" not in f2 + + +# ── §F FwdLLM variance-cadence layer (V/DK/G rungs, PARITY.md §F.4) ────────── + +def _cadence(cycle_data_id, cycle_iteration, var, var_threshold, + var_good_enough, force_commit_planned=False, grad_pool_size=None, + cached_v_size=None, agg_goal=None, round_=1, ts=0.0, **extra): + """One fwdllm agg-goal-boundary (variance gate) agg_round event.""" + e = {"event": "agg_round", "round": round_, "ts": ts, + "cycle_data_id": cycle_data_id, "cycle_iteration": cycle_iteration, + "var": var, "var_threshold": var_threshold, + "var_good_enough": var_good_enough, + "force_commit_planned": force_commit_planned} + for k, v in (("grad_pool_size", grad_pool_size), + ("cached_v_size", cached_v_size), ("agg_goal", agg_goal)): + if v is not None: + e[k] = v + e.update(extra) + return e + + +def _cadence_run(iters_per_data, var_threshold=1.0, pass_var=0.5, fail_var=2.0, + agg_goal=3): + """Series where data_id d takes iters_per_data[d] cycles: (n-1) variance + FAILs (var>thr) then one PASS (var<=thr). grad_pool grows each retry and is + at its max on the committing cycle; cached_v grows across the FAIL rollbacks. + """ + events = [] + for d, n in enumerate(iters_per_data): + for it in range(n): + committed = (it == n - 1) + events.append(_cadence( + cycle_data_id=d, cycle_iteration=it, + var=(pass_var if committed else fail_var), + var_threshold=var_threshold, var_good_enough=committed, + grad_pool_size=(it + 1) * agg_goal, cached_v_size=it, + agg_goal=agg_goal)) + return events + + +class TestV1IterPerDataId: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([1, 2, 1, 3, 1])) + sim = _agg(agg_rounds=_cadence_run([1, 2, 1, 3, 1])) + r = pc.iters_per_data_id_parity(real, sim) + assert r["ok"] and r["real_mean_iters"] == r["sim_mean_iters"], r + + def test_diverged_fails(self): + # sim compounds: every data_id needs 3 cycles vs real's 1 (a + # contributing-set/order divergence surfacing as more retries). + real = _agg(agg_rounds=_cadence_run([1, 1, 1, 1, 1])) + sim = _agg(agg_rounds=_cadence_run([3, 3, 3, 3, 3])) + r = pc.iters_per_data_id_parity(real, sim) + assert not r["ok"] and r["sim_mean_iters"] > r["real_mean_iters"], r + + def test_iters_binning_exact_for_force_commit(self): + # A data_id that force-commits after 2 fails still counts 3 cycles: the + # commit event carries cycle_data_id of the SAME data_id (pre-advance). + cycles = [ + _cadence(0, 0, 2.0, 1.0, False), + _cadence(0, 1, 2.0, 1.0, False), + _cadence(0, 2, 2.0, 1.0, True, force_commit_planned=True), # forced + ] + assert pc._iters_per_data_id(cycles) == {0: 3} + + def test_non_fwdllm_skips(self): + a = _agg(agg_rounds=[_round(1, ["a"], [0], vclock=1.0)]) # no cadence fields + r = pc.iters_per_data_id_parity(a, a) + assert r["ok"] and r.get("status") == "SKIP", r + + +class TestV2VarTrajectory: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([2, 2, 2])) + sim = _agg(agg_rounds=_cadence_run([2, 2, 2])) + assert pc.var_trajectory_parity(real, sim)["ok"] + + def test_diverged_fails(self): + # Same iteration counts but the variance *signal* differs (grad-pool + # accumulation-order bug): sim's var values sit far from real's. + real = _agg(agg_rounds=[_cadence(d, 0, 0.4, 1.0, True) for d in range(8)]) + sim = _agg(agg_rounds=[_cadence(d, 0, 5.0, 1.0, True) for d in range(8)]) + assert not pc.var_trajectory_parity(real, sim)["ok"] + + +class TestV4ForceCommitRate: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([1, 1, 1, 1])) + sim = _agg(agg_rounds=_cadence_run([1, 1, 1, 1])) + r = pc.force_commit_rate_parity(real, sim) + assert r["ok"] and r["sim_force_commit_rate"] == 0.0, r + + def test_diverged_fails(self): + # sim hits the iteration cap far more often (chronic variance divergence). + real = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True) for d in range(10)]) + sim = _agg(agg_rounds=[ + _cadence(d, 0, 2.0, 1.0, True, force_commit_planned=True) + for d in range(10)]) + r = pc.force_commit_rate_parity(sim, real) # order-agnostic + assert not r["ok"], r + + +class TestV5VariancePassRatio: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([1, 2, 1, 2])) + sim = _agg(agg_rounds=_cadence_run([1, 2, 1, 2])) + assert pc.variance_pass_ratio_parity(real, sim)["ok"] + + def test_force_commit_excluded_from_pass(self): + # A force-commit (var>thr, committed only because the cap fired) is NOT a + # genuine variance pass -> real (genuine) and sim (forced) diverge on V5 + # even though both "committed" every cycle. + real = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True) for d in range(10)]) + sim = _agg(agg_rounds=[ + _cadence(d, 0, 2.0, 1.0, True, force_commit_planned=True) + for d in range(10)]) + r = pc.variance_pass_ratio_parity(real, sim) + assert not r["ok"] + assert r["real_pass_ratio"] == 1.0 and r["sim_pass_ratio"] == 0.0, r + + +class TestV3CachedVPool: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([2, 3, 2])) + sim = _agg(agg_rounds=_cadence_run([2, 3, 2])) + assert pc.cached_v_pool_parity(real, sim)["ok"] + + def test_absent_skips(self): + real = _agg(agg_rounds=[_cadence(0, 0, 0.5, 1.0, True)]) + sim = _agg(agg_rounds=[_cadence(0, 0, 0.5, 1.0, True)]) + r = pc.cached_v_pool_parity(real, sim) + assert r["ok"] and r.get("status") == "SKIP", r + + +class TestDK1AggGoalTrajectory: + def test_constant_k_skips(self): + real = _agg(agg_rounds=_cadence_run([1, 1, 1], agg_goal=3)) + sim = _agg(agg_rounds=_cadence_run([1, 1, 1], agg_goal=3)) + r = pc.agg_goal_trajectory_parity(real, sim) + assert r["ok"] and r.get("status") == "SKIP" and "disabled" in r["note"], r + + def test_varying_k_matched_passes(self): + real = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, agg_goal=k) + for d, k in enumerate([3, 3, 4, 5, 4])]) + sim = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, agg_goal=k) + for d, k in enumerate([3, 3, 4, 5, 4])]) + assert pc.agg_goal_trajectory_parity(real, sim)["ok"] + + +class TestDK3EligibleEndsMetric: + def test_absent_skips_with_note(self): + real = _agg(agg_rounds=_cadence_run([1, 1])) + sim = _agg(agg_rounds=_cadence_run([1, 1])) + r = pc.eligible_ends_metric_parity(real, sim) + assert r["ok"] and r.get("status") == "SKIP" and "deferred" in r["note"], r + + def test_present_diverged_fails(self): + real = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, n_eligible_train=10) + for d in range(6)]) + sim = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, n_eligible_train=2) + for d in range(6)]) + assert not pc.eligible_ends_metric_parity(real, sim)["ok"] + + +class TestG1GradNorm: + def test_absent_skips_with_note(self): + real = _agg(agg_rounds=_cadence_run([1, 1])) + sim = _agg(agg_rounds=_cadence_run([1, 1])) + r = pc.grad_norm_parity(real, sim) + assert r["ok"] and r.get("status") == "SKIP" and "deferred" in r["note"], r + + +class TestG2GradPoolSize: + def test_matched_passes(self): + real = _agg(agg_rounds=_cadence_run([1, 2, 1, 2])) + sim = _agg(agg_rounds=_cadence_run([1, 2, 1, 2])) + assert pc.grad_pool_size_parity(real, sim)["ok"] + + def test_diverged_fails(self): + # sim accumulates far larger pools before committing (K x V1 rollup). + real = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, grad_pool_size=3) + for d in range(8)]) + sim = _agg(agg_rounds=[_cadence(d, 0, 0.5, 1.0, True, grad_pool_size=30) + for d in range(8)]) + assert not pc.grad_pool_size_parity(real, sim)["ok"] + + +class TestRunAllParityFwdllm: + _CADENCE_KEYS = [ + "v1_iter_per_data_id", "v2_var_trajectory", "v3_cached_v_pool", + "v4_force_commit_rate", "v5_variance_pass_ratio", + "dk1_agg_goal_trajectory", "dk2_dynamic_c", "dk3_eligible_ends_metric", + "g1_grad_norm", "g2_grad_pool_size", + ] + + def test_keys_present_and_identical_passes(self): + a = _agg(selection=[_sel(1, ["a", "b"])], + agg_rounds=_cadence_run([1, 2, 1, 3, 1])) + tr = {"aa": {"task_recv": [], "trainer_round": []}} + res = pc.run_all_parity(a, a, tr, tr, agg_goal=3) + for key in self._CADENCE_KEYS: + assert key in res, f"missing cadence key: {key}" + v = res[key] + # identical real==sim ⇒ every cadence rung PASSes or SKIPs cleanly + assert v["ok"], f"{key}: {v}" + + def test_non_fwdllm_run_skips_all_cadence(self): + a = _agg(agg_rounds=[_round(r, ["a"], [0], vclock=float(r)) + for r in range(1, 4)]) + tr: dict = {} + res = pc.run_all_parity(a, a, tr, tr) + for key in self._CADENCE_KEYS: + assert res[key].get("status") == "SKIP", f"{key} should SKIP: {res[key]}" + + def test_cadence_divergence_fails_verdict(self): + real = _agg(agg_rounds=_cadence_run([1, 1, 1, 1, 1, 1])) + sim = _agg(agg_rounds=_cadence_run([3, 3, 3, 3, 3, 3])) + tr: dict = {} + res = pc.run_all_parity(real, sim, tr, tr, agg_goal=3) + passed, roots, downstream, _w = pc.overall_verdict(res) + assert not passed + assert "v1_iter_per_data_id" in (set(roots) | set(downstream)) + + +class TestAggRoundCadenceEmission: + """The agg_round builder carries the Batch-2 cadence fields through `extra` + (the aggregator snapshots them pre-mutation, §K-D9).""" + + def test_cadence_fields_land_in_event(self): + from flame.telemetry.events import build_agg_round, EVENT_AGG_ROUND + ev, f = build_agg_round( + round_num=4, agg_goal=3, agg_goal_count=3, + extra={"cycle_data_id": 7, "cycle_iteration": 2, "var": 0.8, + "var_threshold": 1.0, "var_good_enough": True, + "force_commit_planned": False, "grad_pool_size": 9, + "cached_v_size": 2}) + assert ev == EVENT_AGG_ROUND + assert f["cycle_data_id"] == 7 and f["cycle_iteration"] == 2 + assert f["grad_pool_size"] == 9 and f["cached_v_size"] == 2 From d19b43801acf6a3abf86808c3e27fd771c675382 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 11:27:35 -0400 Subject: [PATCH 06/56] fwdllm sim: shared expt_runner harness + real<->sim launch tooling; consolidate example run scripts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add examples/scripts/expt_runner.{sh,py} as the shared launch harness both examples' drivers source: robust conda activation, PYTHONPATH pin, launch + progress ticker, a health check that scans the RIGHT files (agg_round in telemetry/*.jsonl, stopping-run/SIM_WALL_CEILING in *_aggregator.log), the timeout + process-group-kill campaign primitive (expt_timed_run), the --after hook dispatcher, and a tiered ①②③ hyperparameter display + feasibility gate (expt_runner.py render_and_gate). fwdllm run_sequential.sh: now drives the real<->sim launcher PAIRS (the _sim yamls were previously orphaned) via --mode {sim|real|both}, tags run dirs _real/_sim so scripts.parity.cli can glob the pair, sets enable_training_delays identically on both sides via --delays {on|off} (K-D8), pre-flight gate blocks infeasible configs (agg_goal>c, num_gpus>visible, num_trainers expt_timed_run, also fixes a latent near-deadline TIMEOUT misreport); run_felix_streaming.sh conda block -> harness. Delete compare_overnight.sh (absorbed into --after parity,plot on the maintained parity/cli.py; it used the legacy checker). simulate_fwdllm.md: add curated §B.1 real<->sim design-delta table (KEEP CURRENT) + §I.6 launch pointer. Update real-sim_parity_checker_plan.md refs. Co-Authored-By: Claude Opus 4.8 --- .../real-sim_parity_checker_plan.md | 9 +- .../scripts/compare_overnight.sh | 73 -- .../async_cifar10/scripts/debug_run.sh | 199 +++-- .../scripts/run_felix_streaming.sh | 24 +- .../async_cifar10/scripts/smoke_suite.sh | 72 +- .../fwdllm/expt_scripts/run_sequential.sh | 696 +++++++++++------- lib/python/examples/fwdllm/simulate_fwdllm.md | 37 +- lib/python/examples/scripts/expt_runner.py | 177 +++++ lib/python/examples/scripts/expt_runner.sh | 225 ++++++ 9 files changed, 1031 insertions(+), 481 deletions(-) delete mode 100755 lib/python/examples/async_cifar10/scripts/compare_overnight.sh create mode 100644 lib/python/examples/scripts/expt_runner.py create mode 100644 lib/python/examples/scripts/expt_runner.sh diff --git a/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md b/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md index 2290af0fa..4412ddf08 100644 --- a/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md +++ b/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md @@ -338,8 +338,10 @@ python -m scripts.parity.cli \ (real vs sim), overlap-factor bars, GPU-vs-budget, convergence-by-round. - Exit code per §3 verdict rule. -`compare_overnight.sh` is rewritten to call this one command per (baseline) pair -and to drop its separate `compare_parity.py` invocation. +Post-run parity is now invoked via `debug_run.sh --after parity` (the old +standalone `compare_overnight.sh` was deleted — its per-baseline sim-vs-real +parity + cross-baseline plots are absorbed into `--after parity,plot`, on the +maintained `scripts.parity.cli` engine). ### 4.2b Multi-baseline invocation (test parity per selector as we converge) @@ -433,7 +435,8 @@ test import path. The `analyze_*.py` scripts are **kept** but demoted to 4. **Build `report.py` + `cli.py`** single output; fix the convergence self-compare bug (C3). Add `--batch` multi-baseline mode (§4.2b). 5. **Add per-selector test layer** (`tests/mode/test_selector_invariants.py`, §4.2c). -6. **Rewrite `compare_overnight.sh`** to the single command; demote `analyze_*` +6. **`compare_overnight.sh` deleted** — folded into `debug_run.sh --after parity,plot` + (shared `examples/scripts/expt_runner.sh` dispatcher); demote `analyze_*` behind `--diagnostics`. 7. **Delete** the superseded files (§4.4); update any docs/HANDOFF references. 8. **Run** the new checker (`--batch`) on felix + refl pairs; paste the FAILing diff --git a/lib/python/examples/async_cifar10/scripts/compare_overnight.sh b/lib/python/examples/async_cifar10/scripts/compare_overnight.sh deleted file mode 100755 index 48a56dda3..000000000 --- a/lib/python/examples/async_cifar10/scripts/compare_overnight.sh +++ /dev/null @@ -1,73 +0,0 @@ -#!/bin/bash -# Post-run comparison for the overnight n300 runs. Two views: -# 1. PER-BASELINE sim-vs-real tracking (the only apples-to-apples pairing): -# parity_check.py for each baseline's sim run dir vs its real run dir. -# 2. CROSS-BASELINE, separately for sim and for real (one plot set each): -# analyze_run.py --compare-streaming over the 4 runs of a mode. -# -# compare_overnight.sh # auto-discovers latest run dir per (baseline,mode) -# Output: /tmp/overnight_compare/{parity_.txt, parity_.json, sim_cross/, real_cross/} -set -u - -# --- robust conda activation (handles non-default install locations) --- -ENVNAME="${FLAME_CONDA_ENV:-dg_flame}" -CB="" -if command -v conda >/dev/null 2>&1; then - CB="$(conda info --base 2>/dev/null)" -elif [ -n "${CONDA_EXE:-}" ]; then - CB="$(dirname "$(dirname "$CONDA_EXE")")" -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - for c in "$HOME/miniconda3" "/coc/scratch/${USER%??}/miniconda3" \ - "/coc/scratch/$USER/miniconda3" "$HOME/anaconda3" /opt/conda; do - [ -f "$c/etc/profile.d/conda.sh" ] && CB="$c" && break - done -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - echo "ERROR: conda not found. Activate '$ENVNAME' yourself or set CONDA_EXE." >&2; exit 1 -fi -source "$CB/etc/profile.d/conda.sh" -conda activate "$ENVNAME" || { echo "ERROR: 'conda activate $ENVNAME' failed" >&2; exit 1; } - -# paths derived from this script's location (portable across nodes) -EX="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -cd "$EX" || exit 1 -ROOT="$(cd "$EX/../../../.." && pwd)" -OUT=/tmp/overnight_compare; mkdir -p "$OUT" -BASELINES="felix oort refl feddance" - -# latest run dir for a (baseline, mode) — names end in _stream_ -rundir() { ls -dt experiments/run_*_${1}_n300_*_stream_${2}* 2>/dev/null | head -1; } - -echo "### 1. PER-BASELINE sim-vs-real parity ###" -for b in $BASELINES; do - dr=$(rundir "$b" real); ds=$(rundir "$b" sim) - if [ -z "$dr" ] || [ -z "$ds" ]; then - echo " $b: missing run dir (real='$dr' sim='$ds')"; continue - fi - echo " $b: real=$(basename "$dr") sim=$(basename "$ds")" - python scripts/parity_check.py \ - --real "$dr" --sim "$ds" \ - --json-out "$OUT/parity_${b}.json" \ - > "$OUT/parity_${b}.txt" 2>&1 - grep -E "\[OK\]|\[!!\]|\[XX\]|CHECKS" "$OUT/parity_${b}.txt" | sed 's/^/ /' -done - -echo "### 2. CROSS-BASELINE (sim plots, then real plots) ###" -for mode in sim real; do - dirs=(); labels=() - for b in $BASELINES; do - d=$(rundir "$b" "$mode"); [ -z "$d" ] && continue - dirs+=("$d/telemetry"); labels+=("$b") - done - if [ ${#dirs[@]} -ge 2 ]; then - echo " $mode: ${labels[*]}" - python "$ROOT"/scripts/analysis/analyze_run.py \ - --compare-streaming "${dirs[@]}" --labels "${labels[@]}" \ - --out "$OUT/${mode}_cross" > "$OUT/${mode}_cross.log" 2>&1 - echo " -> $OUT/${mode}_cross/ ($(find "$OUT/${mode}_cross" -name '*.pdf' 2>/dev/null | wc -l) plots)" - else - echo " $mode: <2 run dirs, skipping cross-baseline" - fi -done -echo "DONE -> $OUT" diff --git a/lib/python/examples/async_cifar10/scripts/debug_run.sh b/lib/python/examples/async_cifar10/scripts/debug_run.sh index fe24b6326..ce94009cd 100755 --- a/lib/python/examples/async_cifar10/scripts/debug_run.sh +++ b/lib/python/examples/async_cifar10/scripts/debug_run.sh @@ -24,29 +24,18 @@ # time for sync baselines like Refl). set -u -# --- robust conda activation --- -ENVNAME="${FLAME_CONDA_ENV:-dg_flame}" -CB="" -if command -v conda >/dev/null 2>&1; then - CB="$(conda info --base 2>/dev/null)" -elif [ -n "${CONDA_EXE:-}" ]; then - CB="$(dirname "$(dirname "$CONDA_EXE")")" -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - for c in "$HOME/miniconda3" "/coc/scratch/${USER%??}/miniconda3" \ - "/coc/scratch/$USER/miniconda3" "$HOME/anaconda3" /opt/conda; do - [ -f "$c/etc/profile.d/conda.sh" ] && CB="$c" && break - done -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - echo "ERROR: conda not found. Activate '$ENVNAME' yourself or set CONDA_EXE." >&2; exit 1 -fi -source "$CB/etc/profile.d/conda.sh" -conda activate "$ENVNAME" || { echo "ERROR: 'conda activate $ENVNAME' failed" >&2; exit 1; } -echo "conda: base=$CB env=$ENVNAME python=$(which python)" - # repo example dir (portable across nodes) -EX="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +EX="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # .../examples/async_cifar10 +REPO_ROOT="$(cd "$EX/../../../.." && pwd)" # flame/ + +# shared harness: conda activation, launch+ticker, log asserts, preflight bridge. +# The conda activation, PYTHONPATH pin, and launch/progress loop that used to be +# inlined here now live in examples/scripts/expt_runner.sh (shared with fwdllm). +# shellcheck source=../../scripts/expt_runner.sh +source "$REPO_ROOT/lib/python/examples/scripts/expt_runner.sh" + +expt_activate_conda dg_flame # default env dg_flame (FLAME_CONDA_ENV overrides) +expt_pin_pythonpath "$REPO_ROOT" cd "$EX" || exit 1 SCR=expt_scripts_2026 LOGDIR="${FLAME_LOGDIR:-/tmp/debug_run_logs}"; mkdir -p "$LOGDIR" @@ -59,6 +48,11 @@ SIM_WALL_CEILING_S="" # empty = max_experiment_runtime_s (1×, tight guard; sim MODE="both" # sim | real | both — which time_mode variant(s) of each baseline to run NUM_TRAINERS="" # empty = use whatever's in the parity config (300); non-smoke override only ALPHA="" # empty = use the parity config's dirichlet_alpha (0.1); e.g. 100 for homogeneous +DRY_RUN=0 # --dry-run: show the pre-flight table + checks, generate cfg, DON'T launch +SHOW_ALL=0 # --show-all: expand tier ③ + list passing checks +STRICT=0 # --strict: a BLOCKING pre-flight check aborts (default: warn + continue, so + # smoke_suite.sh's non-interactive timeout-wrapped runs never hang/abort) +AFTER="" # --after: comma list of post-launch hooks (parity,plot) -- see after_* below usage() { echo "usage: $0 [--baselines 'felix refl'] [--runtime-s 3600] [--mode sim|real|both] [--sim-wall-ceiling-s 2700] [--trace syn_20]" @@ -84,6 +78,14 @@ usage() { echo " --alpha Dirichlet alpha override (default: parity config's 0.1). Supported" echo " values have an n300 split: 0.1 / 1.0 / 10.0 / 100.0 (100=homogeneous)." echo " When set, the split lookup uses the n300 partition for that alpha." + echo " --dry-run show the pre-flight hyperparameter table + feasibility checks and the" + echo " generated cfg, then exit WITHOUT launching." + echo " --show-all expand tier ③ (config-baked rows) + list the passing checks too." + echo " --strict abort if a pre-flight check is BLOCKING (default: warn + continue, so" + echo " smoke_suite.sh's non-interactive runs never hang/abort)." + echo " --after HOOKS comma-separated post-launch hooks: parity (scripts.parity.cli --batch" + echo " sim-vs-real per baseline) and/or plot (analyze_run cross-baseline" + echo " streaming figs). Absorbs the old compare_overnight.sh. e.g. --after parity,plot" exit 2 } @@ -98,6 +100,10 @@ if [ "${1:-}" = "smoke" ]; then --mode) MODE="$2"; shift 2 ;; --trace) TRACE="$2"; shift 2 ;; --alpha) ALPHA="$2"; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + --show-all) SHOW_ALL=1; shift ;; + --strict) STRICT=1; shift ;; + --after) AFTER="$2"; shift 2 ;; *) shift ;; esac done @@ -113,6 +119,10 @@ else --trace) TRACE="$2"; shift 2 ;; --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; --alpha) ALPHA="$2"; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + --show-all) SHOW_ALL=1; shift ;; + --strict) STRICT=1; shift ;; + --after) AFTER="$2"; shift 2 ;; # --node is DEPRECATED (node1/node2 split removed): baselines are filtered # from a single node-agnostic parity config, so the node is irrelevant. # Accept+ignore so existing wrappers don't hard-error. @@ -309,46 +319,117 @@ print(len(d.get('experiments', []))) PY } +# Thin wrapper over the shared harness's expt_launch (identical mechanics: +# 30s progress ticker + run_* dir counting + the START/DONE log lines). Kept as +# a named function so the two call sites below are unchanged. run_node() { local label="$1" cfg="$2" budget_s="${3:-0}" n_exps="${4:-1}" - local start_ts; start_ts=$(date +%s) - # Baseline run-dir count — new dirs that appear are newly-started experiments. - local initial_runs; initial_runs=$(find experiments -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) + expt_launch "$label" "$cfg" "$EX" "$budget_s" "$n_exps" "$LOGDIR" +} - echo "[$(date '+%F %T')] START $label ($n_exps exp(s), ~${budget_s}s budget)" | tee -a "$LOGDIR/debug_run.log" +# cifar_preflight -- render the tiered hyperparameter table + feasibility +# checks (shared examples/scripts/expt_runner.py) for the just-generated combined +# cfg. Returns 2 if a check is BLOCKING. Callers decide what to do with that: +# by default a block only WARNs and continues (so smoke_suite.sh's non-interactive +# timeout-wrapped invocations never hang or abort); --strict makes it fatal. +GPUS_VISIBLE="$( (command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi --query-gpu=index --format=csv,noheader 2>/dev/null | wc -l) || echo 0)" +cifar_preflight() { + local cfg="$1" + EXPT_RUNNER_DIR="$EXPT_RUNNER_DIR" CFG="$cfg" \ + BASELINES="$BASELINES" MODE="$MODE" RUNTIME_S="$RUNTIME_S" TRACE="$TRACE" \ + ALPHA="$ALPHA" NUM_TRAINERS="$NUM_TRAINERS" SIM_WALL_CEILING_S="$SIM_WALL_CEILING_S" \ + GPUS_VISIBLE="$GPUS_VISIBLE" DRY_RUN="$DRY_RUN" SHOW_ALL="$SHOW_ALL" \ + EX="$EX" LOGDIR="$LOGDIR" \ + python - <<'PY' +import os, sys, yaml +sys.path.insert(0, os.environ["EXPT_RUNNER_DIR"]) +import expt_runner +env = os.environ.get +cfg = yaml.safe_load(open(env("CFG"), encoding="utf-8")) +exps = cfg.get("experiments", []) +gpus_vis = int(env("GPUS_VISIBLE") or "0") - # Background progress ticker: fires every 30s, prints elapsed/remaining/percent - # and how many experiments have started (each start creates a new run_* dir). - ( - while true; do - sleep 30 - local now; now=$(date +%s) - local elapsed=$(( now - start_ts )) - local pct=0 remaining=0 - if [ "$budget_s" -gt 0 ]; then - pct=$(( elapsed * 100 / budget_s )) - remaining=$(( budget_s - elapsed )) - [ "$pct" -gt 100 ] && pct=100 - [ "$remaining" -lt 0 ] && remaining=0 - fi - local curr; curr=$(find experiments -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) - local started=$(( curr - initial_runs )) - [ "$started" -lt 0 ] && started=0 - printf " [%s] %s | %ds elapsed / ~%ds (%d%%) | exp started: %d/%d\n" \ - "$(date '+%T')" "$label" "$elapsed" "$budget_s" "$pct" "$started" "$n_exps" - done - ) & - local ticker_pid=$! +rows2, checks = [], [] +for e in exps: + h = e["aggregator"]["config_overrides"]["hyperparameters"] + n = e.get("trainer", {}).get("num_trainers") + ng = e.get("execution", {}).get("num_gpus") + mts = h.get("min_trainers_to_start") + a = (e.get("trainer", {}).get("dataset", {}) or {}).get("dirichlet_alpha") + rows2.append({"label": e.get("name", "?")[:26], + "value": f"n_trainers={n} n_gpus={ng} min_start={mts} rounds={h.get('rounds')} alpha={a}"}) + if isinstance(n, int) and isinstance(mts, int) and n < mts: + checks.append({"name": f"num_trainers >= min_trainers_to_start ({e.get('name')})", + "level": "error", "detail": f"{n} < {mts} — join barrier never clears"}) + if isinstance(ng, int) and gpus_vis and ng > gpus_vis: + checks.append({"name": f"num_gpus <= gpus_visible ({e.get('name')})", + "level": "error", "detail": f"num_gpus={ng} > visible={gpus_vis}"}) - python -m flame.launch.run_experiment "$cfg" --example-dir "$EX" \ - < /dev/null >> "$LOGDIR/${label}.out" 2>&1 - local rc=$? +tiers = [ + {"name": "① REVIEW EVERY RUN", "rows": [ + {"label": "baselines", "value": env("BASELINES")}, + {"label": "mode", "value": env("MODE"), + **({"level": "warn", "note": "single-sided: parity needs both"} if env("MODE") != "both" else {})}, + {"label": "runtime_s", "value": env("RUNTIME_S")}, + {"label": "trace", "value": env("TRACE") or "", + **({"level": "warn", "note": "not syn_0"} if (env("TRACE") and env("TRACE") != "syn_0") else {})}, + {"label": "sim_wall_ceiling", "value": env("SIM_WALL_CEILING_S") or "= runtime_s (auto)"}, + {"label": "alpha", "value": env("ALPHA") or ""}, + ]}, + {"name": "② PER-EXPERIMENT (moderate)", "rows": rows2}, + {"name": "③ RARELY CHANGED", "collapsed": True, "rows": [ + {"label": "env", "value": os.environ.get("CONDA_DEFAULT_ENV", "?")}, + {"label": "gpus_visible", "value": str(gpus_vis)}, + {"label": "example_dir", "value": env("EX")}, + {"label": "logdir", "value": env("LOGDIR")}, + ]}, +] +checks.append({"name": "run names carry _real/_sim tags for parity glob", "level": "ok", + "detail": "make_debug_yaml keeps the parity config's _sim/_real suffixes"}) +spec = {"title": "CIFAR DEBUG RUN", "subtitle": f"{len(exps)} experiment(s)", + "dry_run": env("DRY_RUN") == "1", "tiers": tiers, "checks": checks} +sys.exit(expt_runner.render_and_gate(spec, show_all=(env("SHOW_ALL") == "1"))) +PY +} - kill "$ticker_pid" 2>/dev/null - wait "$ticker_pid" 2>/dev/null +# gate_or_continue -- shared post-preflight decision for both paths. +gate_or_continue() { + local rc="$1" + if [ "$DRY_RUN" = "1" ]; then + echo "--dry-run: generated cfg in $LOGDIR. Nothing launched."; exit 0 + fi + if [ "$rc" -eq 2 ]; then + if [ "$STRICT" = "1" ]; then + echo "Pre-flight BLOCKED (exit 2) and --strict set. Nothing launched." >&2; exit 2 + fi + echo "WARNING: pre-flight flagged a BLOCKING check (continuing; pass --strict to abort)." >&2 + fi +} - local elapsed=$(( $(date +%s) - start_ts )) - echo "[$(date '+%F %T')] DONE $label exit=$rc (took ${elapsed}s / ~${budget_s}s budget)" | tee -a "$LOGDIR/debug_run.log" +# ---- post-launch hooks (--after ...), dispatched by expt_dispatch_after ---- +# after_parity / after_plot absorb what compare_overnight.sh used to do (its +# per-baseline sim-vs-real parity + cross-baseline streaming plots), but off the +# maintained scripts.parity.cli engine (compare_overnight used the legacy +# scripts/parity_check.py). +after_parity() { + python -m scripts.parity.cli --batch --experiments-dir experiments \ + --baselines $BASELINES --json-out "$LOGDIR/parity_.json" +} +after_plot() { + local ar="$REPO_ROOT/scripts/analysis/analyze_run.py" mode b d + [ -f "$ar" ] || { echo " [after:plot] $ar not found — skipping" >&2; return 0; } + for mode in sim real; do + local dirs=() labels=() + for b in $BASELINES; do + d=$(ls -dt experiments/run_*dbg_*"${b}"*_"${mode}"* 2>/dev/null | head -1) + [ -n "$d" ] && [ -d "$d/telemetry" ] && { dirs+=("$d/telemetry"); labels+=("$b"); } + done + if [ "${#dirs[@]}" -ge 2 ]; then + echo " [after:plot] $mode cross-baseline: ${labels[*]}" + python "$ar" --compare-streaming "${dirs[@]}" --labels "${labels[@]}" \ + --out "$LOGDIR/${mode}_cross" || true + fi + done } # ---- smoke mode ---- @@ -361,7 +442,12 @@ if [ "$SMOKE" = "1" ]; then make_debug_yaml "$BASELINES" 240 "$cfg" 1 "$SIM_WALL_CEILING_S" "$MODE" "$TRACE" "" "$ALPHA" if [ -f "$cfg" ]; then _n=$(_count_exps "$cfg") + cifar_preflight "$cfg"; gate_or_continue $? run_node "dbg_smoke" "$cfg" $(( _n * 240 )) "$_n" + expt_assert_run "$EX" "$EXPT_LAST_MARKER" "dbg_smoke" + [ -n "$AFTER" ] && expt_dispatch_after "$AFTER" + elif [ "$DRY_RUN" = "1" ]; then + echo "--dry-run: no experiments matched baselines='$BASELINES' mode=$MODE. Nothing to show."; exit 0 fi echo "=== SMOKE RESULTS ===" for dd in experiments/run_*dbg_smoke_*; do @@ -389,6 +475,9 @@ fi _n_exps=$(_count_exps "$cfg") _budget=$(( _n_exps * RUNTIME_S )) echo " queued: $_n_exps exp(s), estimated budget ~${_budget}s (sim finishes faster than real)" +cifar_preflight "$cfg"; gate_or_continue $? run_node "debug_run" "$cfg" "$_budget" "$_n_exps" +expt_assert_run "$EX" "$EXPT_LAST_MARKER" "debug_run" +[ -n "$AFTER" ] && expt_dispatch_after "$AFTER" echo "Logs: $LOGDIR/debug_run.out" echo "Run dirs: experiments/run_*dbg_*" diff --git a/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh b/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh index d639ed5a8..a34dbfc12 100755 --- a/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh +++ b/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh @@ -14,26 +14,14 @@ # See docs/EXPERIMENT_felix_streaming.md (Distributed execution & pooling). set -u -# --- robust conda activation (mirrors debug_run.sh) --- -ENVNAME="${FLAME_CONDA_ENV:-dg_flame}" -CB="" -if command -v conda >/dev/null 2>&1; then - CB="$(conda info --base 2>/dev/null)" -elif [ -n "${CONDA_EXE:-}" ]; then - CB="$(dirname "$(dirname "$CONDA_EXE")")" -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - for c in "$HOME/miniconda3" "$HOME/anaconda3" /opt/conda; do - [ -f "$c/etc/profile.d/conda.sh" ] && CB="$c" && break - done -fi -[ -f "$CB/etc/profile.d/conda.sh" ] || { echo "ERROR: conda not found; activate '$ENVNAME' yourself." >&2; exit 1; } -source "$CB/etc/profile.d/conda.sh" -conda activate "$ENVNAME" || { echo "ERROR: conda activate $ENVNAME failed" >&2; exit 1; } -echo "conda: base=$CB env=$ENVNAME python=$(which python)" - EX="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # async_cifar10 example dir REPO="$(cd "$EX/../../../.." && pwd)" # repo root + +# shared harness: robust conda activation (same block debug_run.sh uses). +# shellcheck source=../../scripts/expt_runner.sh +source "$REPO/lib/python/examples/scripts/expt_runner.sh" +expt_activate_conda dg_flame + cd "$EX" || exit 1 SCR=expt_scripts_2026 CFG="$SCR/n50_alpha0.1_syn0_stream_unif_sim.yaml" diff --git a/lib/python/examples/async_cifar10/scripts/smoke_suite.sh b/lib/python/examples/async_cifar10/scripts/smoke_suite.sh index a1fe02753..fa423c68e 100755 --- a/lib/python/examples/async_cifar10/scripts/smoke_suite.sh +++ b/lib/python/examples/async_cifar10/scripts/smoke_suite.sh @@ -55,6 +55,13 @@ EX_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" # async_cifar10/ LIB_DIR="$(cd "$EX_DIR/../.." && pwd)" # lib/python/ DEBUG_RUN="$SCRIPT_DIR/debug_run.sh" +# Shared harness: the timeout + process-group-kill primitive (expt_timed_run) +# that used to be inlined in _run_baseline now lives here (also used by any +# future fwdllm campaign). No conda activation here — each debug_run.sh child +# activates its own env, and step 1 uses `conda run -n`. +# shellcheck source=../../scripts/expt_runner.sh +source "$LIB_DIR/examples/scripts/expt_runner.sh" + # ── Defaults ───────────────────────────────────────────────────────────────── RUNTIME_SYN0_S=900 RUNTIME_SYN20_S=1800 @@ -202,64 +209,15 @@ _run_baseline() { local ts_marker="$run_dir/.ts_start" touch "$ts_marker" - # set -m (job control) forces bash to assign PGID = runner_pid to the - # background job regardless of whether the suite is running interactively - # or not. All descendants inherit that PGID (flame's spawner.py uses plain - # subprocess.Popen with no start_new_session/os.setsid), so - # kill -TERM/-KILL on -$runner_pid reliably reaches the whole process tree. - # Rationale: setsid forks when the calling process is already a pg-leader - # (interactive terminals do this), making runner_pid point to a dead parent - # instead of the actual session leader → kill misses the tree entirely. - set -m - env FLAME_LOGDIR="$run_dir" bash "$DEBUG_RUN" "$@" \ - >"$run_dir/shell.log" 2>&1 & - local runner_pid=$! - set +m - - local deadline=$(( ts_start + wall_timeout )) - local timed_out=0 - local _ela _kill_in _prun _pdone # ticker temporaries - while kill -0 "$runner_pid" 2>/dev/null; do - sleep 5 - _ela=$(( $(date +%s) - ts_start )) - _kill_in=$(( deadline - $(date +%s) )); [[ "$_kill_in" -lt 0 ]] && _kill_in=0 - _prun=0; [[ "$runtime_s" -gt 0 ]] && _prun=$(( _ela * 100 / runtime_s )) - [[ "$_prun" -gt 100 ]] && _prun=100 - _pdone=0; [[ "$TOTAL_RUNS" -gt 0 ]] && _pdone=$(( COMPLETED_RUNS * 100 / TOTAL_RUNS )) - printf '\r %-52s %4ds/%-4ds(%3d%%) kill in %4ds | %d/%d done(%d%%) ' \ - "[$label]" "$_ela" "$runtime_s" "$_prun" "$_kill_in" \ - "$COMPLETED_RUNS" "$TOTAL_RUNS" "$_pdone" >&2 - if [[ "$(date +%s)" -ge "$deadline" ]]; then - printf '\n' >&2 - _log " [$label] TIMEOUT after ${wall_timeout}s — killing process group $runner_pid" - # SIGTERM first: lets ExperimentRunner._signal_handler call _cleanup() - # (terminate_all trainers + terminate aggregator). 20s grace lets Python - # flush open files and release MQTT connections before the hard kill. - kill -TERM -"$runner_pid" 2>/dev/null || true - sleep 20 - # SIGKILL for anything that survived (hung GPU op, stuck MQTT recv). - kill -KILL -"$runner_pid" 2>/dev/null || true - wait "$runner_pid" 2>/dev/null - timed_out=1 - break - fi - done - printf '\n' >&2 - [[ "$timed_out" == "0" ]] && wait "$runner_pid" 2>/dev/null + # Launch under the shared timeout+process-group-kill primitive: it runs + # `env FLAME_LOGDIR=... bash debug_run.sh ...` in its own PGID, SIGTERM→SIGKILL + # the whole tree on overrun, then sweeps stragglers and waits KILL_SETTLE_S for + # GPU memory to drain. Returns 124 on timeout. (Was ~55 inlined lines here.) + EXPT_KILL_SETTLE_S="$KILL_SETTLE_S" \ + expt_timed_run "$label" "$runtime_s" "$TIMEOUT_BUFFER_S" "$run_dir/shell.log" -- \ + env FLAME_LOGDIR="$run_dir" bash "$DEBUG_RUN" "$@" local run_rc=$? - - # ── Post-kill cleanup ───────────────────────────────────────────────────── - # When SIGKILL fires, run_experiment_batch's finally block (_sweep_stragglers) - # is in the killed group and may not complete. Replicate it here: hard-kill - # any surviving trainer/aggregator processes and wait for GPU memory to drain - # so the next run starts from a clean slate. - if [[ "$timed_out" == "1" ]]; then - _log " [$label] post-kill sweep: clearing straggler trainer/aggregator processes" - pkill -9 -f "trainer/pytorch/main.py" 2>/dev/null || true - pkill -9 -f "aggregator/pytorch/main_" 2>/dev/null || true - _log " [$label] waiting ${KILL_SETTLE_S}s for GPU memory to drain before next run" - sleep "$KILL_SETTLE_S" - fi + local timed_out=0; [[ "$run_rc" == "124" ]] && timed_out=1 local elapsed=$(( $(date +%s) - ts_start )) diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 98f5a0465..07daea347 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -1,121 +1,84 @@ #!/bin/bash -# Run multiple fwdllm YAMLs (fwdllm, fwdllm_plus, fluxtune) one after -# another, in a single conda env, logging each run separately. +# Drive the fwdllm real<->sim launcher pairs (fwdllm, fwdllm_plus, fluxtune) +# for the parity sign-off runs. Thin driver over the shared harness +# examples/scripts/expt_runner.sh -- the conda activation, launch+progress loop, +# and log-health assertions live there; this file owns only what's +# fwdllm-specific: the baseline->(real yaml, sim yaml) map, the knob patching, +# and the tier/check spec fed to the pre-flight gate. # -# Each YAML auto-terminates once data_id reaches a threshold or after a -# wall-time cap, whichever comes first. This script overrides those caps -# per invocation via --max-runtime-s/--max-data-id, generating a patched -# copy of each YAML rather than editing the originals. +# What changed vs the old real-only sequential runner: +# * --mode {sim|real|both} now drives the _sim sibling yamls too and pairs +# each baseline's real+sim runs (default: both -- that's the parity run). +# * run names carry a _real / _sim tag so scripts.parity.cli can glob the pair +# (*baseline*real* / *baseline*sim*). +# * --delays {on|off} sets enable_training_delays IDENTICALLY on both sides of +# a pair (K-D8: smokes keep D=0; the convergence/parity run needs D>0 in +# BOTH real and sim together -- mismatched D would be a false divergence). +# * A pre-flight gate prints the hyperparameters in three volatility tiers and +# refuses infeasible configs (see examples/scripts/expt_runner.py). --dry-run +# shows the table + checks and exits without launching; --yes skips the +# confirm for a real (GPU) run; --force overrides a blocking check. # # Usage (from anywhere): -# run_sequential.sh [--max-runtime-s 600] [--max-data-id 10] -# [--num-trainers N] [--num-gpus N] [--c C] [--k K] [--stop-on-fail] -# [--partition-method NAME] [--only name1,name2] +# run_sequential.sh [--mode sim|real|both] [--delays on|off] +# [--max-runtime-s 600] [--max-data-id 10] [--num-trainers N] [--num-gpus N] +# [--c C] [--c-async C] [--k K] [--agg-goal N] [--min-initial-trainers N] +# [--partition-method NAME] [--avail-trace NAME | --avail-traces N1,N2] +# [--only name1,name2] [--stop-on-fail] [--dry-run] [--yes] [--force] +# [--show-all] # -# --max-runtime-s wall-clock cap in seconds for each run (default: 600 = 10 min) -# --max-data-id stop a run once data_id reaches this value (default: 10) -# --num-trainers override trainer.num_trainers (default: each YAML's own, 10) -# --num-gpus override execution.num_gpus (default: each YAML's own, 1). -# Scale this with --num-trainers -- each YAML's default of -# 1 GPU is sized for its own default 10-trainer count. -# --c override selector.kwargs.c + minInitialTrainers + agg_goal -# (agg_goal matches c so no selected trainer goes stranded) -# -- superseded per-field by --agg-goal/--min-initial-trainers -# when those are also passed (see below). -# --c-async override selector.kwargs.c only for the async baseline -# (fluxtune) -- lets sync baselines run concurrency==agg_goal -# via --c/--agg-goal while fluxtune overcommits concurrency -# independent of agg_goal (matches async_oort's design: c -# ends in flight, agg_goal of them counted per round). -# --agg-goal override aggregator.agg_goal directly (fans into -# hyperparameters.aggGoal + selector.kwargs.aggGoal/aggr_num -# per runner.py) independent of --c/--c-async. When --c is -# also given without this, legacy behavior (agg_goal==c) -# still applies. -# --min-initial-trainers override selector.kwargs.minInitialTrainers -# directly, independent of --num-trainers/--c. -# --avail-trace override the availability trace used by ALL baselines: -# trainer.availability.mode (cosmetic/consistency), -# trainer hyperparameters.client_notify.trace (fluxtune's -# real signal), and aggregator -# hyperparameters.trackTrainerAvail.trace (fwdllm_plus's -# real ORACULAR signal). Use e.g. "syn_0" (always -# available) to isolate selection/aggregation bugs from -# trace-driven scarcity/churn. -# --avail-traces comma-separated list of traces, e.g. "syn_0,syn_20" -- -# runs the ENTIRE --only baseline sequence once per trace, -# back to back, in this one invocation/process (for an -# unattended overnight multi-trace comparison; no need to -# babysit and launch the next trace by hand). Takes -# precedence over --avail-trace if both are given. Each -# (baseline, trace) run's name/log/results are -# disambiguated by trace -- see the run-name note below. -# --k override selector.kwargs.k -# --stop-on-fail abort the remaining runs as soon as one exits non-zero -# (default: run all three regardless, report at the end) -# --partition-method override hyperparameters.partition_method on both the -# trainer and aggregator sides (default: each YAML's own, -# "uniform" -- IID, chosen for smoke tests to isolate -# launcher-mechanics validation from data-skew effects). -# Must be one of agnews_partition.h5's own group names, -# e.g. "niid_label_clients=100_alpha=0.1" for the most -# heterogeneous split available in the 100-client group -# (smaller alpha = more skewed/non-IID). -# --only comma-separated subset of baselines to execute, e.g. -# --only fwdllm_plus,fluxtune -# (default: all three -- fwdllm, fwdllm_plus, fluxtune) -# These are plain baseline names, independent of -# --num-trainers -- the "n10" in each source YAML's -# filename is just that file's own default trainer -# count, not part of the run's identity. +# --mode which time_mode variant(s) to run per baseline (default both). +# --delays enable_training_delays for BOTH sides of a pair (default off=D=0). +# --max-runtime-s wall/vclock cap per run (default 600 = 10 min). +# --max-data-id stop a run once data_id reaches this (default 10). +# --num-trainers override trainer.num_trainers (default: each YAML's own, 10). +# --num-gpus override execution.num_gpus (default: each YAML's own). +# --c / --c-async / --k / --agg-goal / --min-initial-trainers +# selector/aggregator knobs (see the per-flag notes below). +# --partition-method override hyperparameters.partition_method both sides. +# --avail-trace / --avail-traces availability trace(s); Phase 1 uses syn_0. +# --only comma-separated baseline subset (default all three). +# --after comma-separated post-launch hooks to run once all launches +# finish: parity (scripts.parity.cli --batch on the real/sim +# pairs), sanity (extract_sanity_checks.py per run dir), plot +# (analyze_run.py over the produced telemetry). e.g. --after parity,sanity +# --stop-on-fail abort remaining runs on first non-zero exit. +# --dry-run show the pre-flight table + checks, generate cfgs, DON'T launch. +# --yes don't prompt to confirm a real (GPU) run. +# --force launch even if a pre-flight check is BLOCKING (error). +# --show-all expand tier ③ (config-baked) + list passing checks. +# +# Per-flag knob notes (unchanged semantics): +# --c sets selector.kwargs.c (+ minInitialTrainers + agg_goal unless +# --agg-goal/--min-initial-trainers override those per-field). +# --c-async sets selector.kwargs.c only for the async baseline (fluxtune). +# --agg-goal sets aggregator.agg_goal directly (fans into hyperparameters.aggGoal +# + selector aggGoal/aggr_num per runner.py) independent of --c. +# --partition-method must be a group name in agnews_partition.h5 +# (e.g. niid_label_clients=100_alpha=0.1); default "uniform" (IID). set -u -# --- robust conda activation (same pattern as scripts/debug_run.sh) --- -# Env choice: FLAME_CONDA_ENV overrides; otherwise use whatever conda env is -# already active in the launching shell (CONDA_DEFAULT_ENV). No hardcoded -# fallback -- activate an env before calling this script, or set -# FLAME_CONDA_ENV explicitly. -ENVNAME="${FLAME_CONDA_ENV:-${CONDA_DEFAULT_ENV:-}}" -if [ -z "$ENVNAME" ]; then - echo "ERROR: no conda env active in this shell and FLAME_CONDA_ENV not set." >&2 - echo " Activate an env first (conda activate ) or pass FLAME_CONDA_ENV=." >&2 - exit 1 -fi -CB="" -if command -v conda >/dev/null 2>&1; then - CB="$(conda info --base 2>/dev/null)" -elif [ -n "${CONDA_EXE:-}" ]; then - CB="$(dirname "$(dirname "$CONDA_EXE")")" -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - for c in "$HOME/miniconda3" "/coc/scratch/${USER%??}/miniconda3" \ - "/coc/scratch/$USER/miniconda3" "$HOME/anaconda3" /opt/conda; do - [ -f "$c/etc/profile.d/conda.sh" ] && CB="$c" && break - done -fi -if [ -z "$CB" ] || [ ! -f "$CB/etc/profile.d/conda.sh" ]; then - echo "ERROR: conda not found. Activate '$ENVNAME' yourself or set CONDA_EXE." >&2; exit 1 -fi -source "$CB/etc/profile.d/conda.sh" -conda activate "$ENVNAME" || { echo "ERROR: 'conda activate $ENVNAME' failed" >&2; exit 1; } -echo "conda: base=$CB env=$ENVNAME python=$(which python)" - # repo paths (portable across nodes/checkouts) SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" EXAMPLE_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" # .../examples/fwdllm REPO_ROOT="$(cd "$EXAMPLE_DIR/../../../.." && pwd)" # flame/ +AC10_DIR="$REPO_ROOT/lib/python/examples/async_cifar10" # hosts scripts.parity.cli + +# shared harness: conda activation, launch+ticker, log asserts, preflight bridge +# shellcheck source=../../scripts/expt_runner.sh +source "$REPO_ROOT/lib/python/examples/scripts/expt_runner.sh" -# Force this checkout's flame package ahead of anything already on -# sys.path (e.g. a stale `pip install -e` editable pointing at a -# different clone) so the code that actually runs matches this repo. -export PYTHONPATH="$REPO_ROOT/lib/python${PYTHONPATH:+:$PYTHONPATH}" +expt_activate_conda # no default env: require an active env / FLAME_CONDA_ENV +expt_pin_pythonpath "$REPO_ROOT" # defaults -MAX_RUNTIME_S=600 # 10 minutes +MODE="both" +DELAYS="off" +MAX_RUNTIME_S=600 MAX_DATA_ID=10 STOP_ON_FAIL=0 -NUM_TRAINERS="" # empty = leave each YAML's own value -NUM_GPUS="" # empty = leave each YAML's own value +NUM_TRAINERS="" +NUM_GPUS="" SEL_C="" SEL_C_ASYNC="" SEL_K="" @@ -123,121 +86,58 @@ AGG_GOAL="" MIN_INIT_TRAINERS="" AVAIL_TRACE="" AVAIL_TRACES="" -PARTITION_METHOD="" # empty = leave each YAML's own value ("uniform") -ONLY="" # empty = run all three +PARTITION_METHOD="" +ONLY="" +AFTER="" # comma list of post-launch hooks: parity,sanity,plot (see after_* below) +DRY_RUN=0 +ASSUME_YES=0 +FORCE=0 +SHOW_ALL=0 + +usage() { + echo "usage: $0 [--mode sim|real|both] [--delays on|off] [--max-runtime-s S] [--max-data-id N]" >&2 + echo " [--num-trainers N] [--num-gpus N] [--c C] [--c-async C] [--k K] [--agg-goal N]" >&2 + echo " [--min-initial-trainers N] [--partition-method NAME]" >&2 + echo " [--avail-trace NAME | --avail-traces N1,N2] [--only n1,n2] [--stop-on-fail]" >&2 + echo " [--dry-run] [--yes] [--force] [--show-all]" >&2 + exit 2 +} while [[ $# -gt 0 ]]; do case "$1" in - --max-runtime-s) MAX_RUNTIME_S="$2"; shift 2 ;; - --max-data-id) MAX_DATA_ID="$2"; shift 2 ;; - --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; - --num-gpus) NUM_GPUS="$2"; shift 2 ;; - --c) SEL_C="$2"; shift 2 ;; - --c-async) SEL_C_ASYNC="$2"; shift 2 ;; - --k) SEL_K="$2"; shift 2 ;; - --agg-goal) AGG_GOAL="$2"; shift 2 ;; + --mode) MODE="$2"; shift 2 ;; + --delays) DELAYS="$2"; shift 2 ;; + --max-runtime-s) MAX_RUNTIME_S="$2"; shift 2 ;; + --max-data-id) MAX_DATA_ID="$2"; shift 2 ;; + --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; + --num-gpus) NUM_GPUS="$2"; shift 2 ;; + --c) SEL_C="$2"; shift 2 ;; + --c-async) SEL_C_ASYNC="$2"; shift 2 ;; + --k) SEL_K="$2"; shift 2 ;; + --agg-goal) AGG_GOAL="$2"; shift 2 ;; --min-initial-trainers) MIN_INIT_TRAINERS="$2"; shift 2 ;; - --avail-trace) AVAIL_TRACE="$2"; shift 2 ;; - --avail-traces) AVAIL_TRACES="$2"; shift 2 ;; - --stop-on-fail) STOP_ON_FAIL=1; shift ;; - --partition-method) PARTITION_METHOD="$2"; shift 2 ;; - --only) ONLY="$2"; shift 2 ;; - *) echo "usage: $0 [--max-runtime-s SECONDS] [--max-data-id N] [--num-trainers N] [--num-gpus N] [--c C] [--c-async C] [--k K] [--agg-goal N] [--min-initial-trainers N] [--avail-trace NAME | --avail-traces NAME1,NAME2,...] [--stop-on-fail] [--partition-method NAME] [--only name1,name2]" >&2; exit 2 ;; + --avail-trace) AVAIL_TRACE="$2"; shift 2 ;; + --avail-traces) AVAIL_TRACES="$2"; shift 2 ;; + --partition-method) PARTITION_METHOD="$2"; shift 2 ;; + --only) ONLY="$2"; shift 2 ;; + --after) AFTER="$2"; shift 2 ;; + --stop-on-fail) STOP_ON_FAIL=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --yes) ASSUME_YES=1; shift ;; + --force) FORCE=1; shift ;; + --show-all) SHOW_ALL=1; shift ;; + *) echo "ERROR: unknown arg '$1'" >&2; usage ;; esac done +case "$MODE" in sim|real|both) ;; *) echo "ERROR: --mode must be sim|real|both (got '$MODE')" >&2; exit 2 ;; esac +case "$DELAYS" in on|off) ;; *) echo "ERROR: --delays must be on|off (got '$DELAYS')" >&2; exit 2 ;; esac -# --avail-traces takes precedence; otherwise fall back to the single -# --avail-trace (may be empty, meaning "leave each YAML's own trace"). -if [ -n "$AVAIL_TRACES" ]; then - IFS=',' read -ra TRACE_LIST <<< "$AVAIL_TRACES" -else - TRACE_LIST=("$AVAIL_TRACE") -fi -MULTI_TRACE=0 -[ "${#TRACE_LIST[@]}" -gt 1 ] && MULTI_TRACE=1 - -LOGDIR="$SCRIPT_DIR/smoke_logs/$(date '+%Y%m%d_%H%M%S')" -mkdir -p "$LOGDIR" - -# Patch hyperparameters.max_runtime_s / max_data_id_progress, and optionally -# num_trainers / selector c+k+minInitialTrainers+agg_goal, in a copy of the -# YAML rather than the original -- keeps the checked-in smoke configs stable -# while letting this script's caller pick the scale per invocation. -patch_yaml() { - python - "$1" "$2" "$3" "$MAX_RUNTIME_S" "$MAX_DATA_ID" "$NUM_TRAINERS" "$NUM_GPUS" "$SEL_C" "$SEL_K" "$PARTITION_METHOD" "$SEL_C_ASYNC" "$AGG_GOAL" "$MIN_INIT_TRAINERS" "$AVAIL_TRACE" <<'PY' -import sys, yaml -(src, dst, run_key, max_runtime_s, max_data_id, num_trainers, num_gpus, sel_c, - sel_k, partition_method, sel_c_async, agg_goal, min_init_trainers, - avail_trace) = sys.argv[1:15] -# Only baseline in ALL_RUNS below that's async; --c-async targets it -# specifically so one invocation can decouple sync concurrency (==agg_goal) -# from async concurrency (overcommitted vs agg_goal) -- see async_oort.py. -IS_ASYNC_BASELINE = run_key == "fluxtune" -cfg = yaml.safe_load(open(src)) -for exp in cfg.get("experiments", []): - h = exp["aggregator"]["config_overrides"]["hyperparameters"] - h["max_runtime_s"] = int(max_runtime_s) - h["max_data_id_progress"] = int(max_data_id) - if partition_method: - h["partition_method"] = partition_method - exp["trainer"]["config_overrides"]["hyperparameters"]["partition_method"] = partition_method - if num_trainers: - exp["trainer"]["num_trainers"] = int(num_trainers) - # exp["name"] feeds the run directory name (run__); derive - # it from run_key + the actual trainer count rather than copying the - # source YAML's own checked-in name, which only reflects that file's - # default count. job.id must track exp["name"] (every checked-in - # YAML keeps them equal; it's the MQTT job/task id shared with - # trainers via runner.py). Include avail_trace when set so runs - # launched back-to-back under different traces (--avail-traces) - # don't produce identically-named run dirs/job ids. - new_name = ( - f"{run_key}_n{num_trainers}_{avail_trace}_smoke" - if avail_trace else f"{run_key}_n{num_trainers}_smoke" - ) - exp["name"] = new_name - exp["aggregator"]["config_overrides"]["job"]["id"] = new_name - if num_gpus: - exp["execution"]["num_gpus"] = int(num_gpus) - kwargs = exp["aggregator"]["config_overrides"]["selector"]["kwargs"] - if sel_c: - kwargs["c"] = int(sel_c) - if not min_init_trainers: - kwargs["minInitialTrainers"] = int(num_trainers) if num_trainers else int(sel_c) - if not agg_goal: - # legacy behavior: agg_goal matches c so no selected trainer goes - # uncounted/stranded. Superseded by --agg-goal below when given. - exp["aggregator"]["agg_goal"] = int(sel_c) - if sel_c_async and IS_ASYNC_BASELINE: - kwargs["c"] = int(sel_c_async) - if sel_k: - kwargs["k"] = int(sel_k) - if agg_goal: - exp["aggregator"]["agg_goal"] = int(agg_goal) - if min_init_trainers: - kwargs["minInitialTrainers"] = int(min_init_trainers) - if avail_trace: - # Cosmetic/consistency: trainer-side self-reported mode. - exp["trainer"].setdefault("availability", {})["mode"] = avail_trace - # Real signal for fluxtune (client_notify) and fwdllm/fwdllm_plus - # (dormant unless trackTrainerAvail below is ORACULAR). - t_hp = exp["trainer"].setdefault("config_overrides", {}).setdefault("hyperparameters", {}) - t_hp.setdefault("client_notify", {})["trace"] = avail_trace - # Real signal for fwdllm_plus (ORACULAR tracking reads this trace - # directly rather than waiting on trainer self-reports). - a_hp = exp["aggregator"]["config_overrides"]["hyperparameters"] - a_hp.setdefault("trackTrainerAvail", {})["trace"] = avail_trace -yaml.safe_dump(cfg, open(dst, "w"), sort_keys=False) -PY -} - -# Keys are plain baseline names -- independent of --num-trainers and of -# whatever scale is baked into each source YAML's own filename/checked-in -# default. The mapping to the actual YAML file lives only here. +# baseline -> (real yaml : sim yaml). Plain baseline names, independent of the +# "n10" baked into each source filename. ALL_RUNS=( - "fwdllm:$SCRIPT_DIR/fwdllm_n10_smoke.yaml" - "fwdllm_plus:$SCRIPT_DIR/fwdllm_plus_n10_smoke.yaml" - "fluxtune:$SCRIPT_DIR/fluxtune_n10_smoke.yaml" + "fwdllm:$SCRIPT_DIR/fwdllm_n10_smoke.yaml:$SCRIPT_DIR/fwdllm_n10_smoke_sim.yaml" + "fwdllm_plus:$SCRIPT_DIR/fwdllm_plus_n10_smoke.yaml:$SCRIPT_DIR/fwdllm_plus_n10_smoke_sim.yaml" + "fluxtune:$SCRIPT_DIR/fluxtune_n10_smoke.yaml:$SCRIPT_DIR/fluxtune_n10_smoke_sim.yaml" ) if [ -n "$ONLY" ]; then @@ -246,88 +146,336 @@ if [ -n "$ONLY" ]; then for want in "${ONLY_NAMES[@]}"; do found=0 for entry in "${ALL_RUNS[@]}"; do - if [ "${entry%%:*}" = "$want" ]; then - RUNS+=("$entry") - found=1 - break - fi + if [ "${entry%%:*}" = "$want" ]; then RUNS+=("$entry"); found=1; break; fi done if [ "$found" = "0" ]; then - echo "ERROR: --only name '$want' not recognized. Valid names: ${ALL_RUNS[*]%%:*}" >&2 - exit 2 + echo "ERROR: --only name '$want' not recognized. Valid: ${ALL_RUNS[*]%%:*}" >&2; exit 2 fi done else RUNS=("${ALL_RUNS[@]}") fi -declare -A RESULT -declare -A DURATION_S -ORDERED_KEYS=() # (name or name@trace) in the order actually run, for the summary +# trace list: --avail-traces wins; else single --avail-trace (may be empty). +if [ -n "$AVAIL_TRACES" ]; then TRACE_CSV="$AVAIL_TRACES"; else TRACE_CSV="$AVAIL_TRACE"; fi + +LOGDIR="$SCRIPT_DIR/smoke_logs/$(date '+%Y%m%d_%H%M%S')" +mkdir -p "$LOGDIR" +GPUS_VISIBLE="$( (command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi --query-gpu=index --format=csv,noheader 2>/dev/null | wc -l) || echo 0)" +MANIFEST="$LOGDIR/manifest.tsv" +SPEC_JSON="$LOGDIR/spec.json" + +# ---- PHASE A: generate all patched cfgs, build the tier/check spec, gate ---- +# One python step so the operator sees the WHOLE matrix (all baselines x traces x +# variants) once, then confirms once. Writes per-run cfgs + a launch manifest, +# renders the tiered table via the shared expt_runner.render_and_gate, and exits +# 2 if any check is blocking. +RUN_TSV="$LOGDIR/_runs.tsv"; : > "$RUN_TSV" +for entry in "${RUNS[@]}"; do + bl="${entry%%:*}"; rest="${entry#*:}"; real_y="${rest%%:*}"; sim_y="${rest#*:}" + printf '%s\t%s\t%s\n' "$bl" "$real_y" "$sim_y" >> "$RUN_TSV" +done + +EXPT_RUNNER_DIR="$EXPT_RUNNER_DIR" \ +MODE="$MODE" DELAYS="$DELAYS" MAX_RUNTIME_S="$MAX_RUNTIME_S" MAX_DATA_ID="$MAX_DATA_ID" \ +NUM_TRAINERS="$NUM_TRAINERS" NUM_GPUS="$NUM_GPUS" SEL_C="$SEL_C" SEL_C_ASYNC="$SEL_C_ASYNC" \ +SEL_K="$SEL_K" AGG_GOAL="$AGG_GOAL" MIN_INIT_TRAINERS="$MIN_INIT_TRAINERS" \ +PARTITION_METHOD="$PARTITION_METHOD" TRACE_CSV="$TRACE_CSV" GPUS_VISIBLE="$GPUS_VISIBLE" \ +LOGDIR="$LOGDIR" MANIFEST="$MANIFEST" RUN_TSV="$RUN_TSV" DRY_RUN="$DRY_RUN" SHOW_ALL="$SHOW_ALL" \ +EXAMPLE_DIR="$EXAMPLE_DIR" AC10_DIR="$AC10_DIR" \ +python - <<'PY' +import os, sys, copy, yaml, json +sys.path.insert(0, os.environ["EXPT_RUNNER_DIR"]) +import expt_runner + +env = os.environ.get +MODE = env("MODE"); DELAYS = env("DELAYS") +MAX_RUNTIME_S = int(env("MAX_RUNTIME_S")); MAX_DATA_ID = int(env("MAX_DATA_ID")) +NUM_TRAINERS = env("NUM_TRAINERS") or "" +NUM_GPUS = env("NUM_GPUS") or "" +SEL_C = env("SEL_C") or ""; SEL_C_ASYNC = env("SEL_C_ASYNC") or ""; SEL_K = env("SEL_K") or "" +AGG_GOAL = env("AGG_GOAL") or ""; MIN_INIT = env("MIN_INIT_TRAINERS") or "" +PART = env("PARTITION_METHOD") or "" +GPUS_VISIBLE = int(env("GPUS_VISIBLE") or "0") +LOGDIR = env("LOGDIR"); MANIFEST = env("MANIFEST") +DRY_RUN = env("DRY_RUN") == "1"; SHOW_ALL = env("SHOW_ALL") == "1" +delays_on = (DELAYS == "on") + +traces = [t for t in (env("TRACE_CSV") or "").replace(",", " ").split()] or [""] +multi_trace = len(traces) > 1 + +variants = {"real": 0, "sim": 1} if MODE == "both" else {MODE: (0 if MODE == "real" else 1)} + +runs = [] # (baseline, real_yaml, sim_yaml) +with open(env("RUN_TSV")) as fh: + for line in fh: + line = line.rstrip("\n") + if line: + runs.append(line.split("\t")) -CHILD_PID="" -cleanup() { - echo "" - echo "Interrupted. Killing child (PID=${CHILD_PID:-none})..." - [ -n "$CHILD_PID" ] && kill -- -"$CHILD_PID" 2>/dev/null - exit 130 +manifest = [] # (name, cfg_path, variant, budget_s) +per_baseline = {} # baseline -> resolved knobs (for the tier ② rows) +checks = [] + + +def patch(exp, run_key, variant, trace): + h = exp["aggregator"]["config_overrides"]["hyperparameters"] + h["max_runtime_s"] = MAX_RUNTIME_S + h["max_data_id_progress"] = MAX_DATA_ID + # enable_training_delays: SAME on both sides of a pair (K-D8). + exp["trainer"]["enable_training_delays"] = delays_on + if PART: + h["partition_method"] = PART + exp["trainer"]["config_overrides"]["hyperparameters"]["partition_method"] = PART + if NUM_TRAINERS: + exp["trainer"]["num_trainers"] = int(NUM_TRAINERS) + if NUM_GPUS: + exp["execution"]["num_gpus"] = int(NUM_GPUS) + kwargs = exp["aggregator"]["config_overrides"]["selector"]["kwargs"] + is_async = (run_key == "fluxtune") + if SEL_C: + kwargs["c"] = int(SEL_C) + if not MIN_INIT: + kwargs["minInitialTrainers"] = int(NUM_TRAINERS) if NUM_TRAINERS else int(SEL_C) + if not AGG_GOAL: + exp["aggregator"]["agg_goal"] = int(SEL_C) # legacy: agg_goal matches c + if SEL_C_ASYNC and is_async: + kwargs["c"] = int(SEL_C_ASYNC) + if SEL_K: + kwargs["k"] = int(SEL_K) + if AGG_GOAL: + exp["aggregator"]["agg_goal"] = int(AGG_GOAL) + if MIN_INIT: + kwargs["minInitialTrainers"] = int(MIN_INIT) + if trace: + exp["trainer"].setdefault("availability", {})["mode"] = trace + t_hp = exp["trainer"].setdefault("config_overrides", {}).setdefault("hyperparameters", {}) + t_hp.setdefault("client_notify", {})["trace"] = trace + h.setdefault("trackTrainerAvail", {})["trace"] = trace + # name / job id: carry a _real|_sim tag so scripts.parity.cli can glob the pair. + n = int(NUM_TRAINERS) if NUM_TRAINERS else exp["trainer"].get("num_trainers", 10) + parts = [run_key, f"n{n}", "smoke"] + if trace: + parts.append(trace) + parts.append(variant) + name = "_".join(parts) + exp["name"] = name + exp["aggregator"]["config_overrides"]["job"]["id"] = name + return name + + +for trace in traces: + for run_key, real_y, sim_y in runs: + for variant, _idx in variants.items(): + src = real_y if variant == "real" else sim_y + if not os.path.exists(src): + checks.append({"name": f"source yaml exists ({run_key} {variant})", + "level": "error", "detail": f"missing: {src}"}) + continue + cfg = yaml.safe_load(open(src, encoding="utf-8")) + exps = cfg.get("experiments", []) + for exp in exps: + name = patch(exp, run_key, variant, trace) + cfg["experiments"] = exps + out = os.path.join(LOGDIR, f"{name}.yaml") + yaml.safe_dump(cfg, open(out, "w", encoding="utf-8"), sort_keys=False) + manifest.append((name, out, variant, MAX_RUNTIME_S)) + + # record resolved knobs from the (first) patched experiment for display + e0 = exps[0] + h0 = e0["aggregator"]["config_overrides"]["hyperparameters"] + kw0 = e0["aggregator"]["config_overrides"]["selector"]["kwargs"] + per_baseline.setdefault(run_key, { + "c": kw0.get("c"), "k": kw0.get("k"), + "agg_goal": e0["aggregator"].get("agg_goal"), + "min_init": kw0.get("minInitialTrainers"), + "n_trainers": e0["trainer"].get("num_trainers"), + "n_gpus": e0.get("execution", {}).get("num_gpus"), + "partition": h0.get("partition_method"), + "delays": e0["trainer"].get("enable_training_delays"), + "async": (run_key == "fluxtune"), + }) + +with open(MANIFEST, "w") as fh: + for name, out, variant, budget in manifest: + fh.write(f"{name}\t{out}\t{variant}\t{budget}\n") + +# ---------------- build the tiered spec ---------------- +tiers = [] +# ① review every run +trace_disp = " ".join(traces) if any(traces) else "" +tier1 = {"name": "① REVIEW EVERY RUN", "rows": [ + {"label": "mode", "value": MODE, **({"level": "warn", "note": "single-sided: parity needs both"} if MODE != "both" else {})}, + {"label": "baselines", "value": " ".join(rk for rk, *_ in runs)}, + {"label": "stop", "value": f"max_runtime_s={MAX_RUNTIME_S} max_data_id_progress={MAX_DATA_ID}"}, + {"label": "trace", "value": trace_disp, + **({"level": "warn", "note": "Phase 1 is syn_0 (100% avail)"} if any(t and t != "syn_0" for t in traces) else {"note": "100% availability"})}, + {"label": "delays", "value": f"enable_training_delays={str(delays_on).lower()} (D={'>0' if delays_on else '0'})", + "level": "warn", "note": "matched on BOTH sides ✓ — K-D8" if MODE == "both" else "K-D8"}, +]} +tiers.append(tier1) + +# ② per-baseline +rows2 = [] +for rk in (r[0] for r in runs): + b = per_baseline.get(rk, {}) + val = (f"c={b.get('c')} agg_goal={b.get('agg_goal')} k={b.get('k')} " + f"minInit={b.get('min_init')} n_trainers={b.get('n_trainers')} " + f"n_gpus={b.get('n_gpus')} part={b.get('partition')}") + lvl = {} + if PART and PART != "uniform": + lvl = {"level": "warn", "note": "non-default partition"} + rows2.append({"label": rk, "value": val, **lvl}) +# fwdllm-defining knobs reminder (var_threshold / max_iterations_per_data_id are +# NOT exposed as flags here on purpose -- they are baseline-defining, not parity +# levers, §K). Flag loudly if someone ever wires them in. +rows2.append({"label": "var knobs", "value": "var_threshold / max_iterations_per_data_id = YAML default", + "note": "baseline-defining, NOT parity levers (§K)"}) +tiers.append({"name": "② PER-BASELINE (moderate)", "rows": rows2}) + +# ③ config-baked +tiers.append({"name": "③ RARELY CHANGED", "collapsed": True, "rows": [ + {"label": "env", "value": os.environ.get("CONDA_DEFAULT_ENV", "?")}, + {"label": "gpus_visible", "value": str(GPUS_VISIBLE)}, + {"label": "example_dir", "value": env("EXAMPLE_DIR")}, + {"label": "logdir", "value": LOGDIR}, +]}) + +# ---------------- feasibility checks ---------------- +# D matched across each pair (by construction, but assert it visibly). +if MODE == "both": + checks.append({"name": "enable_training_delays matched across every real/sim pair", + "level": "ok", "detail": f"D={'>0' if delays_on else '0'} both sides"}) +# agg_goal <= c (more required than concurrently selected -> stall). +for rk in (r[0] for r in runs): + b = per_baseline.get(rk, {}) + c, g = b.get("c"), b.get("agg_goal") + if isinstance(c, int) and isinstance(g, int) and g > c: + checks.append({"name": f"agg_goal <= c ({rk})", "level": "error", + "detail": f"agg_goal={g} > c={c} — selected trainers would be stranded"}) + else: + checks.append({"name": f"agg_goal <= c ({rk})", "level": "ok", "detail": f"agg_goal={g} c={c}"}) +# num_gpus <= visible. +for rk in (r[0] for r in runs): + b = per_baseline.get(rk, {}) + g = b.get("n_gpus") + if isinstance(g, int) and GPUS_VISIBLE and g > GPUS_VISIBLE: + checks.append({"name": f"num_gpus <= gpus_visible ({rk})", "level": "error", + "detail": f"num_gpus={g} > visible={GPUS_VISIBLE}"}) +# num_trainers >= minInitialTrainers. +for rk in (r[0] for r in runs): + b = per_baseline.get(rk, {}) + n, mi = b.get("n_trainers"), b.get("min_init") + if isinstance(n, int) and isinstance(mi, int) and n < mi: + checks.append({"name": f"num_trainers >= minInitialTrainers ({rk})", "level": "error", + "detail": f"num_trainers={n} < minInitialTrainers={mi} — join barrier never clears"}) +# non-uniform partition: can't verify the H5 group from here. +if PART and PART != "uniform": + checks.append({"name": "partition group exists in agnews_partition.h5", "level": "warn", + "detail": f"verify group '{PART}' exists"}) +# parity pairing naming (only meaningful for a both-mode matrix). +if MODE == "both": + ok_pair = all(any(n.endswith("_real") for n, *_ in manifest) and + any(n.endswith("_sim") for n, *_ in manifest) for _ in [0]) + checks.append({"name": "run names carry _real/_sim tags for parity glob", + "level": "ok" if ok_pair else "error", + "detail": "scripts.parity.cli globs *baseline*real* / *baseline*sim*"}) + +goal_for_parity = AGG_GOAL or (SEL_C or "10") +next_cmd = (f"(cd {env('AC10_DIR')} && python -m scripts.parity.cli --batch " + f"--experiments-dir {env('EXAMPLE_DIR')}/experiments " + f"--baselines {' '.join(rk for rk, *_ in runs)} --agg-goal {goal_for_parity})") + +spec = { + "title": "FWDLLM RUN", + "subtitle": f"mode={MODE} {len(manifest)} run(s)", + "dry_run": DRY_RUN, + "tiers": tiers, + "checks": checks, + "next": next_cmd, } -trap cleanup INT TERM +json.dump(spec, open(env("MANIFEST") + ".spec.json", "w"), indent=2) +rc = expt_runner.render_and_gate(spec, show_all=SHOW_ALL) +sys.exit(rc) +PY +GATE_RC=$? -cd "$REPO_ROOT" || exit 1 -echo "=== fwdllm sequential run: ${#RUNS[@]} runs (${RUNS[*]%%:*}) x ${#TRACE_LIST[@]} trace(s) (${TRACE_LIST[*]:-}), max_runtime_s=$MAX_RUNTIME_S max_data_id=$MAX_DATA_ID num_trainers=${NUM_TRAINERS:-} num_gpus=${NUM_GPUS:-} c=${SEL_C:-} k=${SEL_K:-} partition_method=${PARTITION_METHOD:-}, logs in $LOGDIR ===" +# ---- gate decision ---- +if [ "$GATE_RC" -eq 2 ] && [ "$FORCE" != "1" ]; then + echo "Pre-flight BLOCKED (exit 2). Fix the config or pass --force to override. Nothing launched." >&2 + exit 2 +fi +if [ "$DRY_RUN" = "1" ]; then + echo "--dry-run: generated cfgs in $LOGDIR (manifest: $MANIFEST). Nothing launched." + exit 0 +fi +# Real (GPU) run confirmation unless --yes. +if [ "$ASSUME_YES" != "1" ]; then + read -r -p "Launch the runs above? [y/N] " _ans < /dev/tty || _ans="" + case "$_ans" in y|Y|yes|YES) ;; *) echo "Aborted (no --yes / declined). Nothing launched."; exit 0 ;; esac +fi +# baseline names (for parity --batch and the summary), derived from RUNS. +RUNS_BASELINES="" +for _e in "${RUNS[@]}"; do RUNS_BASELINES="$RUNS_BASELINES ${_e%%:*}"; done +RUNS_BASELINES="${RUNS_BASELINES# }" + +# ---- post-launch hooks (--after ...), dispatched by expt_dispatch_after ---- +# Each is a shell function the shared harness calls by name; they own the +# fwdllm-specific command (parity CLI path / sanity extractor / plotter). +after_parity() { + # Real<->sim parity battery on the pairs just produced. The parity engine + # lives under async_cifar10/scripts (shared, fwdllm rungs registered in it). + ( cd "$AC10_DIR" && python -m scripts.parity.cli --batch \ + --experiments-dir "$EXAMPLE_DIR/experiments" \ + --baselines $RUNS_BASELINES --agg-goal "${AGG_GOAL:-${SEL_C:-10}}" \ + --json-out "$LOGDIR/parity_.json" ) +} +after_sanity() { + # Per-run sanity signals (selection / data_id / eval-per-data_id / partition). + local d + while IFS= read -r d; do + [ -d "$d" ] || continue + python "$SCRIPT_DIR/extract_sanity_checks.py" "$d" || true + done < <(find "$EXAMPLE_DIR/experiments" -maxdepth 1 -type d -name "run_*" -newer "$MANIFEST" 2>/dev/null) +} +after_plot() { + # Best-effort: analyze_run over the telemetry produced this session. + local ar="$REPO_ROOT/scripts/analysis/analyze_run.py" d + [ -f "$ar" ] || { echo " [after:plot] $ar not found — skipping" >&2; return 0; } + while IFS= read -r d; do + [ -d "$d/telemetry" ] || continue + python "$ar" "$d/telemetry" --out "$LOGDIR/plots_$(basename "$d")" || true + done < <(find "$EXAMPLE_DIR/experiments" -maxdepth 1 -type d -name "run_*" -newer "$MANIFEST" 2>/dev/null) +} + +# ---- PHASE B: launch each generated cfg sequentially ---- +declare -A RESULT DURATION_S +ORDERED_KEYS=() STOP_ALL=0 -for trace in "${TRACE_LIST[@]}"; do - AVAIL_TRACE="$trace" # read by patch_yaml() via the outer AVAIL_TRACE var - [ "$MULTI_TRACE" = "1" ] && echo "--- trace: ${trace:-} ---" - - for entry in "${RUNS[@]}"; do - name="${entry%%:*}" - src_cfg="${entry#*:}" - # Disambiguate by trace only when actually looping multiple traces, so a - # single-trace (or no-trace) invocation keeps today's exact file/key names. - if [ "$MULTI_TRACE" = "1" ]; then - key="${name}@${trace:-default}" - else - key="$name" - fi - cfg="$LOGDIR/${key}.yaml" - log="$LOGDIR/${key}.out" - patch_yaml "$src_cfg" "$cfg" "$name" - - start_ts=$(date +%s) - python -m flame.launch.run_experiment "$cfg" --example-dir "$EXAMPLE_DIR" \ - < /dev/null > "$log" 2>&1 & - CHILD_PID=$! - echo "[$(date '+%F %T')] START $key (PID=$CHILD_PID) -> $cfg (log: $log)" - echo " (to kill: kill -9 $CHILD_PID or Ctrl+C)" - wait "$CHILD_PID" - rc=$? - CHILD_PID="" - end_ts=$(date +%s) - DURATION_S[$key]=$((end_ts - start_ts)) - ORDERED_KEYS+=("$key") - if [ $rc -eq 0 ]; then - RESULT[$key]="PASS" - else - RESULT[$key]="FAIL(exit=$rc)" - fi - echo "[$(date '+%F %T')] DONE $key -> ${RESULT[$key]} (${DURATION_S[$key]}s)" +cd "$REPO_ROOT" || exit 1 +while IFS=$'\t' read -r name cfg variant budget; do + [ -n "$name" ] || continue + start_ts=$(date +%s) + expt_launch "$name" "$cfg" "$EXAMPLE_DIR" "$budget" 1 "$LOGDIR" + rc=$? + DURATION_S[$name]=$(( $(date +%s) - start_ts )) + ORDERED_KEYS+=("$name") + [ "$rc" -eq 0 ] && RESULT[$name]="PASS" || RESULT[$name]="FAIL(exit=$rc)" + expt_assert_run "$EXAMPLE_DIR" "$EXPT_LAST_MARKER" "$name" + if [ "$rc" -ne 0 ] && [ "$STOP_ON_FAIL" = "1" ]; then + echo "--stop-on-fail set; aborting remaining runs."; STOP_ALL=1; break + fi +done < "$MANIFEST" - if [ $rc -ne 0 ] && [ "$STOP_ON_FAIL" = "1" ]; then - echo "--stop-on-fail set; aborting remaining runs (including remaining traces)." - STOP_ALL=1 - break - fi - done - [ "$STOP_ALL" = "1" ] && break -done +# ---- post-launch hooks ---- +[ -n "$AFTER" ] && [ "$STOP_ALL" != "1" ] && expt_dispatch_after "$AFTER" echo "" echo "=== Summary ===" for key in "${ORDERED_KEYS[@]}"; do - printf " %-35s %-15s %ss\n" "$key" "${RESULT[$key]:-SKIPPED}" "${DURATION_S[$key]:-0}" + printf " %-40s %-15s %ss\n" "$key" "${RESULT[$key]:-SKIPPED}" "${DURATION_S[$key]:-0}" done -echo "Logs: $LOGDIR" +echo "Logs: $LOGDIR" echo "Run dirs: $EXAMPLE_DIR/experiments/run_*" +echo "Parity: $(cd "$AC10_DIR" && echo "(cd $AC10_DIR && python -m scripts.parity.cli --batch --experiments-dir $EXAMPLE_DIR/experiments --baselines ${RUNS[*]%%:*} --agg-goal ${AGG_GOAL:-${SEL_C:-10}})")" diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index c2362f064..ecfc76563 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -17,7 +17,8 @@ reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baseli > silently invent something new either), **and (b) APPEND that decision + its rationale to this document** > -- so a later reader can tell an intentional fwdllm-specific divergence from an accidental discrepancy, > and trace any real<->sim gap back to the choice that caused it. Deviation log lives in **§K** (running, -> newest-last); locked cross-cutting ones also surface in §F "Locked principles" / §J.4. +> newest-last); the **curated at-a-glance delta table is §B.1** (kept current, unlike §K's append-only log); +> locked cross-cutting ones also surface in §F "Locked principles" / §J.4. **Current state -- Phase 1 Batches 1-2 COMPLETE (checker + telemetry); Phase-1 syn_0 sign-off run is NEXT.** Batch 1 landed the two structural clock ports; Batch 2 landed the variance-cadence rung layer + its @@ -114,6 +115,30 @@ rollback/`cached_v` at each `_agg_goal` boundary; `total_data_bins=150` (`:271`) `_max_iter_per_data_id` (`:276`, config key `max_iterations_per_data_id`); `_reselect_each_iteration` (`:295`, per-iteration reselection for fwdllm++); sync path `_aggregate_grads_sync` (`:1354`). +### §B.1 Curated real<->sim design deltas vs async_cifar10 -- **KEEP CURRENT** + +Every place fwdllm's real+sim design intentionally diverges from async_cifar10, in one at-a-glance table +so a reviewer can separate an **intentional fwdllm-specific choice** from an accidental discrepancy without +reading the whole §K log. **This table is CURATED (rewritten in place to reflect the current design); +§K is the append-only rationale log.** *Maintenance rule (per the DESIGN PRINCIPLE at the top): when a +deviation lands or changes, append its rationale to §K **and** update the matching row here so this stays +the true current picture.* The `§K` column points at the full rationale. + +| # | Axis | async_cifar10 | fwdllm | Why fwdllm differs | §K | +|---|---|---|---|---|---| +| 1 | Aggregated object | model **weights** | **gradients** (forward-grad JVPs) | grad values are mode-invariant given identical input+perturbation seed, so parity reduces to clock+order+selection parity **plus** the variance-cadence layer | §F.1 | +| 2 | Progress axis | update / round count | committed **`data_id`** (variance passes) | cadence (updates-per-data_id) is an **output to match**, not an input to assume | principle #2 | +| 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer the cifar ladder doesn't model; V/DK/G rungs verify it | §F.1 | +| 4 | sct delay model | `sct = send + max(gpu, D)` (sleep-to-fill-budget) | `sct = send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU time; copying cifar's `max()` would desync real<->sim | **K-D2** | +| 5 | Per-eval sct | distinct eval sct, ~20x eval speedup | **collapses to the train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20x factor); trainer eval msg is a utility report, not a clocked commit | **K-D3** | +| 6 | Slot release | per-commit (inside `_sim_recv_min`) | at the **agg-goal boundary** (`_release_sim_slots_at_agg_goal`) | one grad per call and a `data_id` spans many agg-goal cycles with variance-FAIL rollbacks; per-commit release would strand a re-contributing trainer across a rollback | **K-D5**, principle #4 | +| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **dropped** (`_sim_buffer.clear()` at boundary) | a stranded grad was trained on a pre-rollback `model_version` -> stale next cycle anyway; benign at syn_0. **OPEN watch-point** -- switch to commit-then-carry if a rung shows lost updates | **K-D6** | +| 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` (reuse the primitives, fork the orchestration) | `_sim_recv_min`'s per-commit slot release + withheld/staggered paths key on WEIGHTS semantics -- wrong for a grad pool released on the agg-goal boundary | **K-D4** | +| 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's entire config corpus is `time_mode: real` and shipped with no sim path; a "simulated" default risks silently half-activating an unbuilt path | **K-D1** | +| 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | the post-mutation `data_id`/`iteration_per_data_id` advance BEFORE the event emits, so binning by them is off-by-one; snapshot before the pass/fail branch makes V1 exact | **K-D9** | +| 11 | Availability tracking (v1) | all `trace_read` | **mixed**: fwdllm unaware, fwdllm_plus `oracular`, fluxtune `client_notify` mapped onto approx `trace_read` | baselines carry different tracking models; first-class `client_notify` deferred to Stage H to keep Phase 2 tractable | **D1** | +| 12 | Launch tooling | single parity template (sim+real pairs, `baseline:` field) driven by `debug_run.sh` | per-baseline yamls + **separate `_sim` files** driven by `run_sequential.sh` | different config models; **both drivers now source the shared harness `examples/scripts/expt_runner.{sh,py}`** (conda activation, launch+ticker, log asserts, pre-flight + tiered hyperparam display) so only the config-discovery/patch adapter differs per example | this work | + --- ## §C Baseline matrix -- RESOLVED (from the landed launcher configs) @@ -441,6 +466,16 @@ syn_0 (the `*_n10_smoke_sim.yaml` launchers + their real siblings) -> `parity_ch convergence run (K-D8: the sim smokes keep D=0 for mechanics-only comparability; the parity run needs D>0). Record in §H; gate to Phase 2 (Batch 3, availability). +**How to launch (tooling landed post-Batch-2).** `expt_scripts/run_sequential.sh` drives the real<->sim +pairs via the shared harness `examples/scripts/expt_runner.{sh,py}` (§B.1 row 12): `--mode {sim|real|both}` +pairs each baseline and tags run dirs `_real`/`_sim` so `scripts.parity.cli` globs the pair; `--delays +{on|off}` sets `enable_training_delays` identically on both sides (K-D8); a pre-flight gate prints the +hyperparameters in three volatility tiers (① review-every-run, ② per-baseline, ③ config-baked) and blocks +infeasible configs (`agg_goal>c`, `num_gpus>visible`, `num_trainers --max-data-id ` (D>0 both sides). +Parity: the `NEXT:` line the runner prints (`scripts.parity.cli --batch` pointed at `fwdllm/experiments`). + --- ## §K DEVIATION LOG -- fwdllm-specific divergences from async_cifar10 (running, newest-last) diff --git a/lib/python/examples/scripts/expt_runner.py b/lib/python/examples/scripts/expt_runner.py new file mode 100644 index 000000000..09c7447f2 --- /dev/null +++ b/lib/python/examples/scripts/expt_runner.py @@ -0,0 +1,177 @@ +"""Shared experiment-launch display + pre-flight gate for the flame examples. + +Generic and example-agnostic: it knows how to *render* a tiered hyperparameter +table and *gate* on a list of feasibility checks, but nothing about any specific +example's knobs. Each example's driver (fwdllm `run_sequential.sh`, +async_cifar10 `debug_run.sh`) builds a `spec` dict describing its own tiers and +checks and calls `render_and_gate(spec)`. + +Design goal (why this exists): a live GPU/MQTT run is expensive, so before we +fire one we (a) show the operator the hyperparameters organized by how often +they change -- ① review-every-run, ② per-baseline, ③ config-baked -- and +(b) run correctness checks that block infeasible configs. Colour/callout markers +draw the eye to anything overridden or dangerous. + +Spec schema +----------- + spec = { + "title": str, # e.g. "FWDLLM RUN" + "subtitle": str, # e.g. "mode=both baselines=fwdllm ..." + "dry_run": bool, # cosmetic tag in the header + "tiers": [ + {"name": "① REVIEW EVERY RUN", + "collapsed": False, # tier ③ sets True -> hidden unless EXPT_SHOW_ALL=1 + "rows": [ + {"label": "stop", "value": "max_runtime_s=600 max_data_id=10"}, + {"label": "delays", "value": "D=0", "level": "warn", + "note": "both sides matched"}, + ]}, + ], + "checks": [ + {"name": "D matched across real/sim pair", "level": "ok", + "detail": "..."}, + {"name": "agg_goal <= c (sync)", "level": "error", + "detail": "fwdllm_plus agg_goal=5 > c=2"}, + ], + "next": "python -m scripts.parity.cli --batch ...", # optional hand-off + } + +`level` is one of "ok" (default), "warn", "error". `render_and_gate` returns 2 if +any check (or any *row*) is level "error", else 0 -- the driver treats 2 as +"blocked unless --force". +""" + +from __future__ import annotations + +import json +import os +import sys + +# ---- colour / icon helpers ------------------------------------------------- + +def _use_colour(stream) -> bool: + if os.environ.get("NO_COLOR") is not None: + return False + if os.environ.get("EXPT_FORCE_COLOR") is not None: + return True + return hasattr(stream, "isatty") and stream.isatty() + + +class _Style: + def __init__(self, on: bool): + self.on = on + + def _w(self, code: str, s: str) -> str: + return f"\033[{code}m{s}\033[0m" if self.on else s + + def red(self, s): return self._w("1;31", s) + def yellow(self, s): return self._w("1;33", s) + def green(self, s): return self._w("32", s) + def dim(self, s): return self._w("2", s) + def bold(self, s): return self._w("1", s) + + +# level -> (icon, colouriser name). "ok" rows stay quiet; warn/error shout. +_ICON = {"ok": " ", "warn": "\U0001f7e1", "error": "\U0001f534"} # · 🟡 🔴 +_CHECK_ICON = {"ok": "✓", "warn": "⚠", "error": "✗"} # ✓ ⚠ ✗ + + +def _colour_for(st: _Style, level: str): + return {"ok": st.dim, "warn": st.yellow, "error": st.red}.get(level, st.dim) + + +# ---- renderer -------------------------------------------------------------- + +_RULE = "─" * 79 # ───── + + +def render_and_gate(spec: dict, show_all: bool | None = None, stream=None) -> int: + """Render the tiered table + checks; return 2 if any error, else 0. + + Rows/checks at level "error" mark the config infeasible; the driver should + refuse to launch (unless the operator passes --force). + """ + stream = stream or sys.stdout + st = _Style(_use_colour(stream)) + if show_all is None: + show_all = os.environ.get("EXPT_SHOW_ALL") not in (None, "", "0") + + def out(s=""): + print(s, file=stream) + + title = spec.get("title", "EXPERIMENT RUN") + subtitle = spec.get("subtitle", "") + tag = st.yellow("[DRY-RUN]") if spec.get("dry_run") else "" + out() + out(f" {st.bold(title)} {subtitle} {tag}".rstrip()) + out(" " + _RULE) + + n_err_rows = 0 + for tier in spec.get("tiers", []): + name = tier.get("name", "") + rows = tier.get("rows", []) + collapsed = tier.get("collapsed", False) + out(" " + st.bold(name)) + if collapsed and not show_all: + out(st.dim(f" [{len(rows)} row(s) hidden — set EXPT_SHOW_ALL=1]")) + continue + for r in rows: + level = r.get("level", "ok") + if level == "error": + n_err_rows += 1 + icon = _ICON.get(level, " ") + col = _colour_for(st, level) + label = f"{r.get('label', ''):<13}" + value = r.get("value", "") + note = r.get("note", "") + note_s = f" {st.dim('· ' + note)}" if note else "" + # ok rows: label dim, value plain. warn/error: value coloured. + if level == "ok": + out(f" {icon} {st.dim(label)}{value}{note_s}") + else: + out(f" {icon} {st.bold(label)}{col(value)}{note_s}") + + out(" " + _RULE) + + # ---- checks / gate ---- + checks = spec.get("checks", []) + n_ok = sum(1 for c in checks if c.get("level", "ok") == "ok") + n_warn = sum(1 for c in checks if c.get("level") == "warn") + n_err = sum(1 for c in checks if c.get("level") == "error") + summary = (f" PRE-FLIGHT: {len(checks)} check(s) … " + f"{st.green(str(n_ok) + ' ✓')} " + f"{st.yellow(str(n_warn) + ' ⚠')} " + f"{st.red(str(n_err) + ' ✗')}") + out(summary) + for c in checks: + level = c.get("level", "ok") + if level == "ok" and not show_all: + continue # keep the passing checks quiet unless asked + icon = _CHECK_ICON.get(level, "?") + col = _colour_for(st, level) + detail = c.get("detail", "") + detail_s = f" {st.dim('— ' + detail)}" if detail else "" + out(f" {col(icon)} {c.get('name', '')}{detail_s}") + + blocked = (n_err + n_err_rows) > 0 + if blocked: + out(" " + st.red("BLOCKED: infeasible config — fix, or re-run with --force to override.")) + if spec.get("next"): + out(" " + st.bold("NEXT: ") + spec["next"]) + out() + return 2 if blocked else 0 + + +def main(argv=None) -> int: + """CLI form: `expt_runner.py ` -> render + gate, exit 0/2.""" + argv = argv if argv is not None else sys.argv[1:] + if not argv: + print("usage: expt_runner.py ", file=sys.stderr) + return 2 + with open(argv[0], encoding="utf-8") as fh: + spec = json.load(fh) + return render_and_gate(spec) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/lib/python/examples/scripts/expt_runner.sh b/lib/python/examples/scripts/expt_runner.sh new file mode 100644 index 000000000..d8fa83c5f --- /dev/null +++ b/lib/python/examples/scripts/expt_runner.sh @@ -0,0 +1,225 @@ +#!/bin/bash +# Shared experiment-launch harness for the flame examples. +# +# Sourced by each example's driver (async_cifar10 scripts/debug_run.sh, +# fwdllm expt_scripts/run_sequential.sh) so the genuinely-identical mechanics +# live in ONE place: robust conda activation, PYTHONPATH pinning, the +# launch+progress-ticker loop, and post-run log-health assertions. The +# example-specific parts (config discovery, YAML patching, the per-baseline +# knob set, and the tier/check spec fed to expt_runner.py) stay in each driver. +# +# Usage (from a driver): +# source "/lib/python/examples/scripts/expt_runner.sh" +# expt_activate_conda [default_env] # activate + report python +# expt_pin_pythonpath "$REPO_ROOT" # this checkout's flame ahead of any editable install +# expt_launch label cfg example_dir budget_s n_exps logdir [experiments_dir] +# expt_assert_log "$logdir/label.out" label +# +# Pre-flight display + gate is done by the Python side (expt_runner.py): a driver +# builds a `spec` dict of tiers+checks and calls render_and_gate() -- see that +# file. $EXPT_RUNNER_DIR / $EXPT_RUNNER_PY are exported here so a driver's own +# python heredoc can `sys.path.insert(0, EXPT_RUNNER_DIR); import expt_runner`. + +# Absolute dir of THIS harness (works when sourced), so drivers/python can find +# the Python renderer next to it regardless of the caller's cwd. +EXPT_RUNNER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EXPT_RUNNER_PY="$EXPT_RUNNER_DIR/expt_runner.py" +export EXPT_RUNNER_DIR EXPT_RUNNER_PY + +# --- robust conda activation ------------------------------------------------ +# Env choice: FLAME_CONDA_ENV overrides; else the shell's already-active env +# (CONDA_DEFAULT_ENV); else the caller-supplied default (arg $1). No hardcoded +# fallback beyond what the caller passes -- cifar passes dg_flame, fwdllm passes +# nothing (so it requires an active env, matching its original behavior). +expt_activate_conda() { + local default_env="${1:-}" + local envname="${FLAME_CONDA_ENV:-${CONDA_DEFAULT_ENV:-$default_env}}" + if [ -z "$envname" ]; then + echo "ERROR: no conda env active and FLAME_CONDA_ENV not set." >&2 + echo " Activate an env first (conda activate ), set FLAME_CONDA_ENV=," >&2 + echo " or have the driver pass a default to expt_activate_conda." >&2 + return 1 + fi + local cb="" + if command -v conda >/dev/null 2>&1; then + cb="$(conda info --base 2>/dev/null)" + elif [ -n "${CONDA_EXE:-}" ]; then + cb="$(dirname "$(dirname "$CONDA_EXE")")" + fi + if [ -z "$cb" ] || [ ! -f "$cb/etc/profile.d/conda.sh" ]; then + local c + for c in "$HOME/miniconda3" "/coc/scratch/${USER%??}/miniconda3" \ + "/coc/scratch/$USER/miniconda3" "$HOME/anaconda3" /opt/conda; do + [ -f "$c/etc/profile.d/conda.sh" ] && cb="$c" && break + done + fi + if [ -z "$cb" ] || [ ! -f "$cb/etc/profile.d/conda.sh" ]; then + echo "ERROR: conda not found. Activate '$envname' yourself or set CONDA_EXE." >&2 + return 1 + fi + # shellcheck disable=SC1091 + source "$cb/etc/profile.d/conda.sh" + conda activate "$envname" || { echo "ERROR: 'conda activate $envname' failed" >&2; return 1; } + echo "conda: base=$cb env=$envname python=$(which python)" +} + +# Force this checkout's flame package ahead of anything already on sys.path +# (e.g. a stale `pip install -e` editable pointing at a different clone). +expt_pin_pythonpath() { + local repo_root="$1" + export PYTHONPATH="$repo_root/lib/python${PYTHONPATH:+:$PYTHONPATH}" +} + +# expt_launch label cfg example_dir budget_s n_exps logdir [experiments_dir] +# Runs one flame experiment-config file in the foreground with a 30s progress +# ticker (elapsed/remaining/percent + how many run_* dirs have appeared). Sets +# EXPT_LAST_RC and returns the launcher's exit code. +expt_launch() { + local label="$1" cfg="$2" example_dir="$3" budget_s="${4:-0}" n_exps="${5:-1}" logdir="$6" + local exp_dir="${7:-$example_dir/experiments}" + mkdir -p "$logdir" + # Marker touched BEFORE launch so expt_assert_run can find exactly the run + # dir(s)/logs this launch produced (find -newer "$marker"). Exported for the + # caller's health check. + EXPT_LAST_MARKER="$logdir/.marker_${label}"; : > "$EXPT_LAST_MARKER"; export EXPT_LAST_MARKER + local start_ts; start_ts=$(date +%s) + local initial_runs; initial_runs=$(find "$exp_dir" -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) + + echo "[$(date '+%F %T')] START $label ($n_exps exp(s), ~${budget_s}s budget)" | tee -a "$logdir/expt_runner.log" + + # Background progress ticker. + ( + while true; do + sleep 30 + local now elapsed pct=0 remaining=0 + now=$(date +%s); elapsed=$(( now - start_ts )) + if [ "$budget_s" -gt 0 ]; then + pct=$(( elapsed * 100 / budget_s )); remaining=$(( budget_s - elapsed )) + [ "$pct" -gt 100 ] && pct=100; [ "$remaining" -lt 0 ] && remaining=0 + fi + local curr started + curr=$(find "$exp_dir" -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) + started=$(( curr - initial_runs )); [ "$started" -lt 0 ] && started=0 + printf " [%s] %s | %ds elapsed / ~%ds (%d%%) | exp started: %d/%d\n" \ + "$(date '+%T')" "$label" "$elapsed" "$budget_s" "$pct" "$started" "$n_exps" + done + ) & + local ticker_pid=$! + + python -m flame.launch.run_experiment "$cfg" --example-dir "$example_dir" \ + < /dev/null >> "$logdir/${label}.out" 2>&1 + local rc=$? + + kill "$ticker_pid" 2>/dev/null + wait "$ticker_pid" 2>/dev/null + + local elapsed=$(( $(date +%s) - start_ts )) + echo "[$(date '+%F %T')] DONE $label exit=$rc (took ${elapsed}s / ~${budget_s}s budget)" | tee -a "$logdir/expt_runner.log" + EXPT_LAST_RC=$rc + return $rc +} + +# expt_assert_run example_dir marker [label] -- post-run health check that scans +# the RIGHT files (the FL signals do NOT land in the runner .out): agg_round +# events live in experiments/run_*/telemetry/aggregator_*.jsonl; "stopping run" / +# SIM_WALL_CEILING / SIM_STARVATION / tracebacks live in the dedicated +# experiments/run_*/*_aggregator.log. Only files newer than `marker` (touched by +# expt_launch pre-launch) are scanned, so it attributes signals to THIS run. +# Prints a one-line verdict; returns 0 only if there were agg_rounds, no +# [SIM_WALL_CEILING], and no crash marker. +expt_assert_run() { + local example_dir="$1" marker="$2" label="${3:-run}" + local expdir="$example_dir/experiments" + if [ ! -e "$marker" ]; then + echo " [$label] NO_MARKER (cannot scope health check)"; return 1 + fi + local agg=0 stop=0 wall=0 starv=0 crash=0 nlogs=0 f n + while IFS= read -r f; do + n=$(grep -c "agg_round" "$f" 2>/dev/null); agg=$(( agg + ${n:-0} )) + done < <(find "$expdir" -name "aggregator_*.jsonl" -newer "$marker" 2>/dev/null) + while IFS= read -r f; do + nlogs=$(( nlogs + 1 )) + n=$(grep -ic "stopping run" "$f" 2>/dev/null); stop=$(( stop + ${n:-0} )) + n=$(grep -c "SIM_WALL_CEILING" "$f" 2>/dev/null); wall=$(( wall + ${n:-0} )) + n=$(grep -c "\[SIM_STARVATION\]" "$f" 2>/dev/null); starv=$(( starv + ${n:-0} )) + n=$(grep -cE "Traceback \(most recent call last\)|CUDA error|Segmentation fault" "$f" 2>/dev/null); crash=$(( crash + ${n:-0} )) + done < <(find "$expdir" -name "*_aggregator.log" -newer "$marker" 2>/dev/null) + local status="PASS" + [ "$agg" -eq 0 ] && status="NO_AGG_ROUNDS" + [ "$wall" -gt 0 ] && status="WALL_CEILING" + [ "$crash" -gt 0 ] && status="CRASH" + printf " [%s] %-14s agg_round=%s stopping_run=%s wall_ceiling=%s starvation=%s crash=%s (%s agg log(s))\n" \ + "$label" "$status" "$agg" "$stop" "$wall" "$starv" "$crash" "$nlogs" + [ "$agg" -gt 0 ] && [ "$wall" -eq 0 ] && [ "$crash" -eq 0 ] +} + +# expt_timed_run label runtime_s buffer_s shell_log -- cmd... -- run a command in +# its OWN process group under a wall-clock timeout, killing the whole tree +# (SIGTERM grace -> SIGKILL) if it overruns, then sweeping stragglers so no +# orphan trainers/aggregator survive and GPU memory drains. This is the campaign +# primitive shared by smoke_suite.sh (and any future fwdllm suite). Emits a 5s +# elapsed / kill-in ticker to stderr. Returns the cmd's rc, or 124 on timeout. +expt_timed_run() { + local label="$1" runtime_s="$2" buffer_s="$3" shell_log="$4"; shift 4 + [ "${1:-}" = "--" ] && shift + local settle_s="${EXPT_KILL_SETTLE_S:-20}" grace_s="${EXPT_KILL_GRACE_S:-20}" + local wall_timeout=$(( runtime_s + buffer_s )) + local ts_start; ts_start=$(date +%s) + local deadline=$(( ts_start + wall_timeout )) timed_out=0 + # set -m: assign the background job its own PGID so kill - hits the tree + # (flame's spawner uses plain Popen with no setsid, so descendants inherit it). + set -m + "$@" > "$shell_log" 2>&1 & + local pid=$! + set +m + local ela kin + while kill -0 "$pid" 2>/dev/null; do + sleep 5 + # Re-test liveness: if the child finished DURING the sleep, exit normally + # rather than force-killing a dead pid and misreporting TIMEOUT (this bites + # whenever a run completes near the deadline). + kill -0 "$pid" 2>/dev/null || break + ela=$(( $(date +%s) - ts_start )); kin=$(( deadline - $(date +%s) )); [ "$kin" -lt 0 ] && kin=0 + printf '\r [%s] %ds/%ds kill in %ds ' "$label" "$ela" "$runtime_s" "$kin" >&2 + if [ "$(date +%s)" -ge "$deadline" ]; then + printf '\n' >&2 + echo " [$label] TIMEOUT after ${wall_timeout}s — killing process group $pid" >&2 + kill -TERM -"$pid" 2>/dev/null || true + sleep "$grace_s" + kill -KILL -"$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null + timed_out=1; break + fi + done + printf '\n' >&2 + [ "$timed_out" = "0" ] && wait "$pid" 2>/dev/null + local rc=$? + if [ "$timed_out" = "1" ]; then + pkill -9 -f "trainer/pytorch/main.py" 2>/dev/null || true + pkill -9 -f "trainer/forward_training" 2>/dev/null || true + pkill -9 -f "aggregator/pytorch/main_" 2>/dev/null || true + sleep "$settle_s" + EXPT_LAST_RC=124; return 124 + fi + EXPT_LAST_RC=$rc; return "$rc" +} + +# expt_dispatch_after "hook1,hook2" -- run the requested post-launch hooks. Each +# hook `foo` is delegated to a shell function `after_foo` the DRIVER defines +# (e.g. after_parity, after_plot, after_sanity) -- so the mechanism is shared but +# each example supplies its own parity CLI / plotter / analysis command. Unknown +# or unsupported hooks warn and are skipped (not fatal). +expt_dispatch_after() { + local csv="${1:-}"; [ -n "$csv" ] || return 0 + local hook + IFS=',' read -ra _hooks <<< "$csv" + for hook in "${_hooks[@]}"; do + hook="${hook// /}"; [ -n "$hook" ] || continue + if declare -F "after_${hook}" >/dev/null 2>&1; then + echo "=== after: ${hook} ===" + "after_${hook}" || echo " [after:${hook}] returned non-zero (continuing)" + else + echo " [after:${hook}] not supported for this example — skipping" >&2 + fi + done +} From 4c75db4513380e29f505a56ed46520a8e46fa91c Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 15:33:08 -0400 Subject: [PATCH 07/56] =?UTF-8?q?fwdllm=20sim=20Batch=202.5:=20async=20gra?= =?UTF-8?q?d-path=20residence=20+=20commit-then-carry=20(=C2=A7L)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remediate the three parity defects + one tooling defect the 2026-07-03 syn_0 3-baseline smoke surfaced (simulate_fwdllm.md §L). fluxtune sim ran 2x the forward passes of real -- a residence violation on the async grad path. §L.4 mechanism (D-a/D-b, fluxtune): - _release_sim_slots_at_agg_goal now HOLDS busy trainers before clearing and CARRIES the surplus buffer across the agg-goal boundary (commit-then-carry), residence-gated on sim_inflight_residence; flag-off => byte-identical drop for the sync baselines. Enabled via fluxtune_n10_smoke_sim.yaml. §L.4 telemetry + rungs (R1/W1, K-D14): - trainer echoes SIM_SEND_TS; aggregator captures each contribution's exact [dispatch, commit] interval and emits contributor_intervals on agg_round. - new R1 (in-flight-overlap INV) + W1 (compute-conservation DIAG) parity rungs in the shared engine + CHECK_META (R1 wired upstream of V1); validate_real gains a real-side check_grad_residence gate. §L.4 staleness (D-c, K-D13/K-D15): - self.staleness_policy was never wired from config (silently ran as "none"); fixed in internal_init. fedbuff is now an explicit accept policy feeding the existing 1/sqrt(1+staleness) down-weighting. Set in both fluxtune yamls. §L.5 launcher fail-fast (D-d): - wait_until_ready detects a startup traceback / early death (returns fast, prints log tail); runner terminates trainers immediately on non-zero agg exit instead of burning ~86s. Tests: full pytest tests/ = 793 passed / 7 skipped. New test_fwdllm_sim_grad_residence.py + R1/W1 fixtures + staleness/telemetry tests; fixed test_fwdllm_oracular_avail for the base_dir rename. Doc §L/§H/§K + a new principle #9 (test cadence) updated. Step 6 (the syn_0 re-run) is the operator's next action. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 153 ++++++++- .../scripts/parity/validate_real.py | 50 ++- .../async_cifar10/scripts/parity_checks.py | 6 + .../expt_scripts/fluxtune_n10_smoke.yaml | 7 +- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 14 +- .../expt_scripts/fwdllm_plus_n10_smoke.yaml | 2 +- .../fwdllm_plus_n10_smoke_sim.yaml | 2 +- .../fwdllm/expt_scripts/run_sequential.sh | 134 ++++++-- lib/python/examples/fwdllm/simulate_fwdllm.md | 291 +++++++++++++++++- lib/python/examples/scripts/expt_runner.py | 18 +- lib/python/examples/scripts/expt_runner.sh | 13 +- lib/python/flame/launch/aggregator_spawner.py | 53 +++- lib/python/flame/launch/runner.py | 14 +- .../horizontal/syncfl/fwdllm_aggregator.py | 132 +++++++- .../mode/horizontal/syncfl/fwdllm_trainer.py | 5 + .../tests/mode/test_fwdllm_agg_telemetry.py | 88 ++++++ .../tests/mode/test_fwdllm_oracular_avail.py | 4 +- .../mode/test_fwdllm_sim_grad_residence.py | 154 +++++++++ lib/python/tests/mode/test_parity_checks.py | 99 ++++++ 19 files changed, 1181 insertions(+), 58 deletions(-) create mode 100644 lib/python/tests/mode/test_fwdllm_sim_grad_residence.py diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index e04e24d04..1d4d59dfa 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -3756,6 +3756,142 @@ def _sizes(agg): } +# ═══════════════════════════════════════════════════════════════════ +# §3.5 FwdLLM async residence rungs (R1 / W1) — simulate_fwdllm.md §L.3 +# ═══════════════════════════════════════════════════════════════════ +# +# The 2026-07-03 fluxtune smoke showed sim running 2x the forward passes of +# real: a residence violation on the async grad path (surplus grads dropped + +# re-dispatched every agg-goal cycle). These two rungs would have localized it +# instantly. R1 is the finest check (per-trainer interval overlap); W1 is the +# coarse compute-conservation tell that first flags the wasted recompute. + +_R1_EPS = 1e-6 + + +def _overlap_fraction(cycles: list) -> tuple: + """(overlap_frac, n_pairs, n_intervals) over per-trainer dispatch->commit + intervals reconstructed from the agg_round `contributor_intervals` field. + + A trainer's contribution i "overlaps" if its dispatch_ts precedes the + latest commit_ts among that trainer's earlier contributions -- i.e. it was + re-dispatched while a prior update was still outstanding, the one-in-flight + residence violation (PARITY.md §3.resid, "measure overlap from intervals + not counters"). 0.0 = strict residence; both modes must be ~0. + """ + by_end: dict = {} + for e in cycles: + for iv in (e.get("contributor_intervals") or []): + d, c = iv.get("dispatch_ts"), iv.get("commit_ts") + if d is None or c is None: + continue + by_end.setdefault(iv.get("end"), []).append((float(d), float(c))) + n_pairs = 0 + n_overlap = 0 + n_intervals = 0 + for ivs in by_end.values(): + ivs.sort() + n_intervals += len(ivs) + running_max_commit = float("-inf") + for i, (d, c) in enumerate(ivs): + if i > 0: + n_pairs += 1 + if d < running_max_commit - _R1_EPS: + n_overlap += 1 + running_max_commit = max(running_max_commit, c) + frac = (n_overlap / n_pairs) if n_pairs else 0.0 + return frac, n_pairs, n_intervals + + +def inflight_overlap_parity(real: dict, sim: dict, tol_frac: float = 0.02) -> dict: + """R1 [INV]: one-in-flight-per-trainer residence on the grad path. + + Per-trainer dispatch->commit intervals must NOT overlap (a trainer is + re-pickable only after its update commits). Real satisfies this by channel + construction (~0%); sim must model it (commit-then-carry + slot hold, §L.4 + step 4). A non-zero sim fraction with real ~0 is the residence bug (2x + forward passes). Checked per mode -- both must sit under tol_frac. + """ + rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + r_frac, r_pairs, r_n = _overlap_fraction(rc) + s_frac, s_pairs, s_n = _overlap_fraction(sc) + if r_pairs == 0 and s_pairs == 0: + return {"ok": True, "tier": "INV", "status": "SKIP", + "note": "no contributor_intervals with >=2 contributions per " + "trainer (field absent, non-fwdllm run, or no re-selection)"} + ok = r_frac <= tol_frac and s_frac <= tol_frac + return { + "ok": ok, + "tier": "INV", + "real_overlap_frac": round(r_frac, 4), + "sim_overlap_frac": round(s_frac, 4), + "tol_frac": tol_frac, + "n_real_pairs": r_pairs, + "n_sim_pairs": s_pairs, + "n_real_intervals": r_n, + "n_sim_intervals": s_n, + "interpretation": ( + f"real {r_frac:.1%} / sim {s_frac:.1%} of same-trainer intervals " + f"overlap a prior one; >0 = re-dispatched while still in flight " + f"(residence violation)." + ), + } + + +def _forward_passes(trainers: dict) -> int: + """Total forward passes = trainer_round events across all trainers.""" + return sum(len(t.get("trainer_round", []) or []) for t in trainers.values()) + + +def _committed_grads(agg: dict) -> int: + """Total committed grads = contributors summed over committed cycles.""" + total = 0 + for e in _fwd_cadence_cycles(agg): + ivs = e.get("contributor_intervals") + if ivs is not None: + total += len(ivs) + else: + total += len(e.get("contributing_trainers") or []) + return total + + +def compute_conservation_parity(real: dict, sim: dict, + real_trainers: dict, sim_trainers: dict, + ratio_tol: float = 0.25) -> dict: + """W1 [DIAG]: compute-conservation — forward passes vs committed grads. + + forward_passes ~= committed + in_flight_at_stop + stale_rejected, so the + forward/commit ratio is ~1 plus a small tail. The tell is the real<->sim + RATIO of that ratio: sim doing far more forward passes per commit than real + = wasted recompute (the residence violation, §L.1). Localizes to R1. + """ + r_fwd, s_fwd = _forward_passes(real_trainers), _forward_passes(sim_trainers) + r_com, s_com = _committed_grads(real), _committed_grads(sim) + if r_com == 0 or s_com == 0 or r_fwd == 0 or s_fwd == 0: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no forward passes or committed grads (non-fwdllm run " + "or telemetry absent)"} + r_ratio = r_fwd / r_com + s_ratio = s_fwd / s_com + rel = abs(s_ratio - r_ratio) / max(r_ratio, s_ratio) + return { + "ok": rel <= ratio_tol, + "tier": "DIAG", + "real_forward_passes": r_fwd, + "sim_forward_passes": s_fwd, + "real_committed": r_com, + "sim_committed": s_com, + "real_fwd_per_commit": round(r_ratio, 3), + "sim_fwd_per_commit": round(s_ratio, 3), + "ratio_rel_diff": round(rel, 3), + "ratio_tol": ratio_tol, + "interpretation": ( + f"real {r_ratio:.2f} vs sim {s_ratio:.2f} forward passes per commit; " + f"a large sim excess = recompute wasted on dropped/re-dispatched grads." + ), + } + + # ═══════════════════════════════════════════════════════════════════ # §4 Consolidated run_all_parity (extended) # ═══════════════════════════════════════════════════════════════════ @@ -3872,6 +4008,14 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["g1_grad_norm"] = grad_norm_parity(real_agg, sim_agg) results["g2_grad_pool_size"] = grad_pool_size_parity(real_agg, sim_agg) + # ── Stage 3' FwdLLM async residence (R1/W1, §L.3) ── + # R1 is the finest residence check (per-trainer interval overlap); W1 is the + # coarse compute-conservation tell that feeds V1/K2. Both SKIP cleanly when + # contributor_intervals is absent (sync baselines / non-fwdllm runs). + results["r1_inflight_overlap"] = inflight_overlap_parity(real_agg, sim_agg) + results["w1_compute_conservation"] = compute_conservation_parity( + real_agg, sim_agg, real_trainers, sim_trainers) + # ── Stage 7 Statistical utility ── results["utility"] = utility_parity(real_agg, sim_agg) @@ -3965,8 +4109,15 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "commit_promptness": {"stage": 6, "role": "CONTROL", "deps": ("withheld_delivery",)}, "aggregation_sequence": {"stage": 6, "role": "EMERGENT", "deps": ("participation", "inter_arrival_order")}, "first_divergence_summary": {"stage": 6, "role": "DIAG", "deps": ()}, + # ── Stage 3' FwdLLM async residence (R1/W1, simulate_fwdllm.md §L.3) ── + # R1 is the residence INV the fluxtune 2x-recompute bug violated; W1 is the + # compute-conservation DIAG that first flags it and localizes to R1. V1's + # cadence divergence is DOWNSTREAM of R1 (a residence violation changes the + # contributing set/order), so V1 deps on it -- the dep chain proves it. + "r1_inflight_overlap": {"stage": 3, "role": "MECHANISM", "deps": ("participation",)}, + "w1_compute_conservation": {"stage": 3, "role": "DIAG", "deps": ("r1_inflight_overlap",)}, # ── Stage 6' FwdLLM variance-gated aggregation cadence (PARITY.md §F.4) ── - "v1_iter_per_data_id": {"stage": 6, "role": "MECHANISM", "deps": ("inter_arrival_order",)}, + "v1_iter_per_data_id": {"stage": 6, "role": "MECHANISM", "deps": ("inter_arrival_order", "r1_inflight_overlap")}, "v2_var_trajectory": {"stage": 6, "role": "MECHANISM", "deps": ("v1_iter_per_data_id",)}, "v3_cached_v_pool": {"stage": 6, "role": "DIAG", "deps": ("v1_iter_per_data_id",)}, "v4_force_commit_rate": {"stage": 6, "role": "MECHANISM", "deps": ("v1_iter_per_data_id",)}, diff --git a/lib/python/examples/async_cifar10/scripts/parity/validate_real.py b/lib/python/examples/async_cifar10/scripts/parity/validate_real.py index 48d636ab3..997e8b899 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/validate_real.py +++ b/lib/python/examples/async_cifar10/scripts/parity/validate_real.py @@ -45,7 +45,14 @@ if str(_HERE.parent) not in sys.path: sys.path.insert(0, str(_HERE.parent)) -from parity.checks import load_run_dir, agg_goal_cycles_ok # noqa: E402 +from parity.checks import ( # noqa: E402 + load_run_dir, + agg_goal_cycles_ok, + _fwd_cadence_cycles, + _overlap_fraction, + _forward_passes, + _committed_grads, +) # Allow a tiny fraction of telemetry edge cases (interleaving/late events) before @@ -250,6 +257,45 @@ def check_aggregation(agg: dict) -> dict: } +def check_grad_residence(agg: dict, trainer: dict) -> dict: + """FwdLLM async grad-path residence (R1) + compute-conservation (W1) on the + REAL side — the hard gate that blocks the §L.4-step-4 sim mechanism change. + + R1: real per-trainer dispatch->commit intervals (contributor_intervals) must + not overlap (~0%). If real itself overlaps, real is inadmissible and D-b is a + two-sided fix — fix real FIRST, never tune sim toward a broken real + (simulate_fwdllm.md §L.3 sanity gate / PARITY.md principle #6). + W1: report forward_passes vs committed grads so the operator can confirm the + real fwd-commit gap is explained by end-of-run in-flight + stale-rejects, not + by real ALSO dropping carried grads. Informational (loose bound) since real + stale-reject / in-flight-at-stop counts aren't separately emitted. + + SKIPs cleanly (ok=True) when the real run carries no contributor_intervals + (a non-fwdllm real run) so async_cifar10 validate_real is unaffected. + """ + cycles = _fwd_cadence_cycles(agg) + have_intervals = any(e.get("contributor_intervals") for e in cycles) + if not have_intervals: + return {"ok": True, "status": "SKIP", + "note": "no contributor_intervals (non-fwdllm real run)"} + frac, n_pairs, n_intervals = _overlap_fraction(cycles) + fwd = _forward_passes(trainer) + com = _committed_grads(agg) + ratio = (fwd / com) if com else None + # R1 is the hard invariant; W1 ratio is reported (a huge real ratio would + # itself be suspicious, so bound it generously — real overcommit is small). + ok = frac <= _VIOLATION_FRAC_TOL + return { + "ok": ok, + "r1_overlap_frac": round(frac, 4), + "r1_overlap_pairs": n_pairs, + "n_intervals": n_intervals, + "w1_forward_passes": fwd, + "w1_committed": com, + "w1_fwd_per_commit": round(ratio, 3) if ratio is not None else None, + } + + def validate_real(run_dir: str) -> bool: agg, trainer = load_run_dir(run_dir) tiv = _train_intervals(trainer) @@ -264,9 +310,11 @@ def validate_real(run_dir: str) -> bool: "concurrency": check_concurrency(agg, tiv), "selection": check_selection(agg), "aggregation": check_aggregation(agg), + "grad_residence": check_grad_residence(agg, trainer), } all_ok = True for name, r in results.items(): + r.pop("status", None) # SKIP marker — informational, not printed as a field ok = r.pop("ok") all_ok = all_ok and ok tag = "[OK] " if ok else "[FAIL]" diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index 3d57e55d6..53297d216 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -93,6 +93,12 @@ eligible_ends_metric_parity, grad_norm_parity, grad_pool_size_parity, + # fwdllm async residence (R1/W1, simulate_fwdllm.md §L.3) + _overlap_fraction, + _forward_passes, + _committed_grads, + inflight_overlap_parity, + compute_conservation_parity, # overall run_all_parity, overall_verdict, diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index 22bc34477..7dfb888b3 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -62,6 +62,11 @@ experiments: id: fluxtune_n10_smoke hyperparameters: rounds: 50 # short smoke run, not the full 300 + # Async fedbuff baseline: accept a stale grad (trained on model_version + # V that lands at V'>V) and down-weight by (V'-V) rather than reject it + # (K-D13/§L D5). Baseline DEFINITION -- set identically real+sim, never + # tuned to close a gap. Feeds the U3 staleness rung. + staleness_policy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). @@ -79,7 +84,7 @@ experiments: # the runner fans it into both this kwarg and # hyperparameters.aggGoal so they can't drift apart; don't # duplicate it here. - c: 8 + c: 10 minInitialTrainers: 10 execution: diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 32770e305..62e7f1b65 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -67,6 +67,18 @@ experiments: id: fluxtune_n10_smoke_sim hyperparameters: rounds: 50 # short smoke run, not the full 300 + # Async residence (SIM-only, §L step 4): with c=10 >> agg_goal=3 the + # sim path must COMMIT-THEN-CARRY the surplus arrived grads and hold + # still-busy trainers in their slots across the agg-goal boundary, + # instead of dropping ~7 grads/cycle and re-dispatching them (the 2x + # forward-pass residence violation, K-D12). Real satisfies this by + # channel construction, so this knob is inert in the real sibling. + sim_inflight_residence: true + # Async fedbuff baseline: carried grads (residence above) are stale by + # construction, so accept + down-weight by (V'-V) rather than + # reject/ignore (K-D13/§L D5). Baseline DEFINITION, set identically in + # the real sibling -- never a parity lever. + staleness_policy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). @@ -84,7 +96,7 @@ experiments: # the runner fans it into both this kwarg and # hyperparameters.aggGoal so they can't drift apart; don't # duplicate it here. - c: 8 + c: 10 minInitialTrainers: 10 execution: diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml index 119238eff..7767a406f 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml @@ -54,7 +54,7 @@ experiments: config_template: configs/aggregator_base.json selector: random tracking_mode: oracular - agg_goal: 2 + agg_goal: 10 log_to_wandb: false config_overrides: job: diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml index 67ab589bb..9077b85f5 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -60,7 +60,7 @@ experiments: config_template: configs/aggregator_base.json selector: random tracking_mode: oracular - agg_goal: 2 + agg_goal: 10 log_to_wandb: false config_overrides: job: diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 07daea347..8974dc0fa 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -36,6 +36,11 @@ # --c / --c-async / --k / --agg-goal / --min-initial-trainers # selector/aggregator knobs (see the per-flag notes below). # --partition-method override hyperparameters.partition_method both sides. +# --var-threshold set the variance-pass gate threshold (hyperparameters.var_threshold) +# on both sides. It VARIES with data heterogeneity, so it's a +# review-every-run knob (shown in tier ①), not a fixed default. +# --max-iter-per-data-id set the force-commit cap (max_iterations_per_data_id) +# on both sides (review-every-run, tier ①). # --avail-trace / --avail-traces availability trace(s); Phase 1 uses syn_0. # --only comma-separated baseline subset (default all three). # --after comma-separated post-launch hooks to run once all launches @@ -76,6 +81,12 @@ MODE="both" DELAYS="off" MAX_RUNTIME_S=600 MAX_DATA_ID=10 +# Companion "was this passed on the command line?" flags. Needed because MODE/ +# DELAYS/MAX_RUNTIME_S/MAX_DATA_ID have non-empty defaults, so their value alone +# can't tell "operator passed it (override -> green)" from "defaulted". (The +# empty-default knobs like SEL_C/AGG_GOAL/VAR_THRESHOLD don't need this: non-empty +# already means "passed".) +MODE_SET=0; DELAYS_SET=0; MAX_RUNTIME_S_SET=0; MAX_DATA_ID_SET=0 STOP_ON_FAIL=0 NUM_TRAINERS="" NUM_GPUS="" @@ -87,6 +98,8 @@ MIN_INIT_TRAINERS="" AVAIL_TRACE="" AVAIL_TRACES="" PARTITION_METHOD="" +VAR_THRESHOLD="" # variance-pass gate threshold; varies with data heterogeneity -> review every run +MAX_ITER_PER_DATA_ID="" # force-commit cap (max_iterations_per_data_id); review every run ONLY="" AFTER="" # comma list of post-launch hooks: parity,sanity,plot (see after_* below) DRY_RUN=0 @@ -98,6 +111,7 @@ usage() { echo "usage: $0 [--mode sim|real|both] [--delays on|off] [--max-runtime-s S] [--max-data-id N]" >&2 echo " [--num-trainers N] [--num-gpus N] [--c C] [--c-async C] [--k K] [--agg-goal N]" >&2 echo " [--min-initial-trainers N] [--partition-method NAME]" >&2 + echo " [--var-threshold F] [--max-iter-per-data-id N]" >&2 echo " [--avail-trace NAME | --avail-traces N1,N2] [--only n1,n2] [--stop-on-fail]" >&2 echo " [--dry-run] [--yes] [--force] [--show-all]" >&2 exit 2 @@ -105,10 +119,10 @@ usage() { while [[ $# -gt 0 ]]; do case "$1" in - --mode) MODE="$2"; shift 2 ;; - --delays) DELAYS="$2"; shift 2 ;; - --max-runtime-s) MAX_RUNTIME_S="$2"; shift 2 ;; - --max-data-id) MAX_DATA_ID="$2"; shift 2 ;; + --mode) MODE="$2"; MODE_SET=1; shift 2 ;; + --delays) DELAYS="$2"; DELAYS_SET=1; shift 2 ;; + --max-runtime-s) MAX_RUNTIME_S="$2"; MAX_RUNTIME_S_SET=1; shift 2 ;; + --max-data-id) MAX_DATA_ID="$2"; MAX_DATA_ID_SET=1; shift 2 ;; --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; --num-gpus) NUM_GPUS="$2"; shift 2 ;; --c) SEL_C="$2"; shift 2 ;; @@ -119,6 +133,8 @@ while [[ $# -gt 0 ]]; do --avail-trace) AVAIL_TRACE="$2"; shift 2 ;; --avail-traces) AVAIL_TRACES="$2"; shift 2 ;; --partition-method) PARTITION_METHOD="$2"; shift 2 ;; + --var-threshold) VAR_THRESHOLD="$2"; shift 2 ;; + --max-iter-per-data-id) MAX_ITER_PER_DATA_ID="$2"; shift 2 ;; --only) ONLY="$2"; shift 2 ;; --after) AFTER="$2"; shift 2 ;; --stop-on-fail) STOP_ON_FAIL=1; shift ;; @@ -181,6 +197,8 @@ MODE="$MODE" DELAYS="$DELAYS" MAX_RUNTIME_S="$MAX_RUNTIME_S" MAX_DATA_ID="$MAX_D NUM_TRAINERS="$NUM_TRAINERS" NUM_GPUS="$NUM_GPUS" SEL_C="$SEL_C" SEL_C_ASYNC="$SEL_C_ASYNC" \ SEL_K="$SEL_K" AGG_GOAL="$AGG_GOAL" MIN_INIT_TRAINERS="$MIN_INIT_TRAINERS" \ PARTITION_METHOD="$PARTITION_METHOD" TRACE_CSV="$TRACE_CSV" GPUS_VISIBLE="$GPUS_VISIBLE" \ +VAR_THRESHOLD="$VAR_THRESHOLD" MAX_ITER_PER_DATA_ID="$MAX_ITER_PER_DATA_ID" \ +MODE_SET="$MODE_SET" DELAYS_SET="$DELAYS_SET" MAX_RUNTIME_S_SET="$MAX_RUNTIME_S_SET" MAX_DATA_ID_SET="$MAX_DATA_ID_SET" \ LOGDIR="$LOGDIR" MANIFEST="$MANIFEST" RUN_TSV="$RUN_TSV" DRY_RUN="$DRY_RUN" SHOW_ALL="$SHOW_ALL" \ EXAMPLE_DIR="$EXAMPLE_DIR" AC10_DIR="$AC10_DIR" \ python - <<'PY' @@ -196,6 +214,10 @@ NUM_GPUS = env("NUM_GPUS") or "" SEL_C = env("SEL_C") or ""; SEL_C_ASYNC = env("SEL_C_ASYNC") or ""; SEL_K = env("SEL_K") or "" AGG_GOAL = env("AGG_GOAL") or ""; MIN_INIT = env("MIN_INIT_TRAINERS") or "" PART = env("PARTITION_METHOD") or "" +VAR_THRESHOLD = env("VAR_THRESHOLD") or ""; MAX_ITER = env("MAX_ITER_PER_DATA_ID") or "" +# "was it passed on the command line?" (override -> green) for the defaulted flags +MODE_SET = env("MODE_SET") == "1"; DELAYS_SET = env("DELAYS_SET") == "1" +MAX_RUNTIME_S_SET = env("MAX_RUNTIME_S_SET") == "1"; MAX_DATA_ID_SET = env("MAX_DATA_ID_SET") == "1" GPUS_VISIBLE = int(env("GPUS_VISIBLE") or "0") LOGDIR = env("LOGDIR"); MANIFEST = env("MANIFEST") DRY_RUN = env("DRY_RUN") == "1"; SHOW_ALL = env("SHOW_ALL") == "1" @@ -227,6 +249,12 @@ def patch(exp, run_key, variant, trace): if PART: h["partition_method"] = PART exp["trainer"]["config_overrides"]["hyperparameters"]["partition_method"] = PART + # Variance-cadence knobs (review-every-run). Only patched when explicitly set, + # so an unset run keeps the code/trainer default (surfaced as "(D)" below). + if VAR_THRESHOLD: + h["var_threshold"] = float(VAR_THRESHOLD) + if MAX_ITER: + h["max_iterations_per_data_id"] = int(MAX_ITER) if NUM_TRAINERS: exp["trainer"]["num_trainers"] = int(NUM_TRAINERS) if NUM_GPUS: @@ -301,17 +329,55 @@ with open(MANIFEST, "w") as fh: fh.write(f"{name}\t{out}\t{variant}\t{budget}\n") # ---------------- build the tiered spec ---------------- +# Row colour convention (legend in the subtitle): +# 🟢 set = value came from a command-line flag -> OVERRIDES the yaml (even if +# the yaml happens to agree). Shown WITHOUT "(D)". +# 🟡 warn = review/attention (a review-every-run knob still on its default). +# (D) + dim = plain yaml/code default, not overridden this run. +def dflt(val, overridden): + # inline "(D)" marker for the bundled per-baseline knob strings (tier ②) + return f"{val}" if overridden else f"{val} (D)" + +def scalar_row(label, val, overridden, note=None, review=False): + if overridden: # operator passed the flag -> override + d = {"label": label, "value": f"{val}", "level": "set"} + elif review: # defaulted but must be eyeballed each run + d = {"label": label, "value": f"{val} (D)", "level": "warn"} + else: # plain default + d = {"label": label, "value": f"{val} (D)", "level": "ok"} + if note: + d["note"] = note + return d + tiers = [] # ① review every run -trace_disp = " ".join(traces) if any(traces) else "" +trace_overridden = any(traces) +trace_val = " ".join(traces) if trace_overridden else "syn_0" +# mode: single-sided always warns (parity needs both), regardless of override. +if MODE != "both": + mode_row = {"label": "mode", "value": MODE, "level": "warn", "note": "single-sided: parity needs both"} +else: + mode_row = scalar_row("mode", MODE, MODE_SET, note="real+sim pair (--mode)") tier1 = {"name": "① REVIEW EVERY RUN", "rows": [ - {"label": "mode", "value": MODE, **({"level": "warn", "note": "single-sided: parity needs both"} if MODE != "both" else {})}, + mode_row, {"label": "baselines", "value": " ".join(rk for rk, *_ in runs)}, - {"label": "stop", "value": f"max_runtime_s={MAX_RUNTIME_S} max_data_id_progress={MAX_DATA_ID}"}, - {"label": "trace", "value": trace_disp, - **({"level": "warn", "note": "Phase 1 is syn_0 (100% avail)"} if any(t and t != "syn_0" for t in traces) else {"note": "100% availability"})}, - {"label": "delays", "value": f"enable_training_delays={str(delays_on).lower()} (D={'>0' if delays_on else '0'})", - "level": "warn", "note": "matched on BOTH sides ✓ — K-D8" if MODE == "both" else "K-D8"}, + # The two similarly-named-but-DIFFERENT knobs, disambiguated + on their own rows: + scalar_row("max_runtime_s", MAX_RUNTIME_S, MAX_RUNTIME_S_SET, note="wall/vclock cap (--max-runtime-s)"), + scalar_row("max_data_id_progress", MAX_DATA_ID, MAX_DATA_ID_SET, + note="STOP condition: stop when data_id reaches this (--max-data-id)"), + scalar_row("trace", trace_val, trace_overridden, + note=("Phase 1 is syn_0 (100% avail)" if any(t and t != "syn_0" for t in traces) else "100% availability")), + scalar_row("enable_training_delays", str(delays_on).lower(), DELAYS_SET, + note=f"modeled training delay {'ON (D>0)' if delays_on else 'OFF (D=0)'}; matched on BOTH sides — K-D8"), + # var_threshold / max_iterations_per_data_id vary with data heterogeneity -> + # review-every-run (warn when defaulted). NOTE: max_iters_per_data_id is the + # FORCE-COMMIT cap and is NOT the same as max_data_id_progress (the stop) above. + scalar_row("var_threshold", VAR_THRESHOLD if VAR_THRESHOLD else "unset (trainer/code default)", + bool(VAR_THRESHOLD), review=True, + note="variance-pass gate; varies w/ data heterogeneity (--var-threshold)"), + scalar_row("max_iters_per_data_id", MAX_ITER if MAX_ITER else "unset (code default)", + bool(MAX_ITER), review=True, + note="FORCE-COMMIT cap (--max-iter-per-data-id) — NOT the max_data_id_progress stop above"), ]} tiers.append(tier1) @@ -319,18 +385,18 @@ tiers.append(tier1) rows2 = [] for rk in (r[0] for r in runs): b = per_baseline.get(rk, {}) - val = (f"c={b.get('c')} agg_goal={b.get('agg_goal')} k={b.get('k')} " - f"minInit={b.get('min_init')} n_trainers={b.get('n_trainers')} " - f"n_gpus={b.get('n_gpus')} part={b.get('partition')}") + # each knob shows "(D)" unless its dedicated flag overrode it this run + val = (f"c={dflt(b.get('c'), bool(SEL_C) or (rk=='fluxtune' and bool(SEL_C_ASYNC)))} " + f"agg_goal={dflt(b.get('agg_goal'), bool(AGG_GOAL) or bool(SEL_C))} " + f"k={dflt(b.get('k'), bool(SEL_K))} " + f"minInit={dflt(b.get('min_init'), bool(MIN_INIT))} " + f"n_trainers={dflt(b.get('n_trainers'), bool(NUM_TRAINERS))} " + f"n_gpus={dflt(b.get('n_gpus'), bool(NUM_GPUS))} " + f"part={dflt(b.get('partition'), bool(PART))}") lvl = {} if PART and PART != "uniform": lvl = {"level": "warn", "note": "non-default partition"} rows2.append({"label": rk, "value": val, **lvl}) -# fwdllm-defining knobs reminder (var_threshold / max_iterations_per_data_id are -# NOT exposed as flags here on purpose -- they are baseline-defining, not parity -# levers, §K). Flag loudly if someone ever wires them in. -rows2.append({"label": "var knobs", "value": "var_threshold / max_iterations_per_data_id = YAML default", - "note": "baseline-defining, NOT parity levers (§K)"}) tiers.append({"name": "② PER-BASELINE (moderate)", "rows": rows2}) # ③ config-baked @@ -355,6 +421,11 @@ for rk in (r[0] for r in runs): "detail": f"agg_goal={g} > c={c} — selected trainers would be stranded"}) else: checks.append({"name": f"agg_goal <= c ({rk})", "level": "ok", "detail": f"agg_goal={g} c={c}"}) +# (No k-vs-agg_goal check: in the random selector, send-side selection/concurrency +# is driven by `c` (required_trainers = min(len(ends), c - in_use)); `k` is the +# RECV-side batch size (num_ends_to_remove = min(..., self.k)), NOT a selection +# cap -- so k < agg_goal is fine, the barrier still collects agg_goal grads across +# RECV passes. c <= num_trainers and agg_goal <= c are the binding invariants.) # num_gpus <= visible. for rk in (r[0] for r in runs): b = per_baseline.get(rk, {}) @@ -388,7 +459,7 @@ next_cmd = (f"(cd {env('AC10_DIR')} && python -m scripts.parity.cli --batch " spec = { "title": "FWDLLM RUN", - "subtitle": f"mode={MODE} {len(manifest)} run(s)", + "subtitle": f"mode={MODE} {len(manifest)} run(s) · 🟢 set=flag override · (D)=yaml/code default", "dry_run": DRY_RUN, "tiers": tiers, "checks": checks, @@ -401,6 +472,13 @@ PY GATE_RC=$? # ---- gate decision ---- +# render_and_gate returns 0 (ok) or 2 (blocking check). Anything else means the +# pre-flight step itself failed (e.g. a bad YAML or a spec-builder bug) -- abort +# rather than silently launch on an unvalidated config. +if [ "$GATE_RC" -ne 0 ] && [ "$GATE_RC" -ne 2 ]; then + echo "ERROR: pre-flight step failed (exit $GATE_RC) -- see traceback above. Nothing launched." >&2 + exit "$GATE_RC" +fi if [ "$GATE_RC" -eq 2 ] && [ "$FORCE" != "1" ]; then echo "Pre-flight BLOCKED (exit 2). Fix the config or pass --force to override. Nothing launched." >&2 exit 2 @@ -461,8 +539,22 @@ while IFS=$'\t' read -r name cfg variant budget; do rc=$? DURATION_S[$name]=$(( $(date +%s) - start_ts )) ORDERED_KEYS+=("$name") - [ "$rc" -eq 0 ] && RESULT[$name]="PASS" || RESULT[$name]="FAIL(exit=$rc)" + # Health verdict (COMPLETED / CRASH / NO_AGG_ROUNDS / WALL_CEILING) is the + # source of truth for the summary -- NOT the launcher exit code, which is 0 + # even when the aggregator subprocess crashed (run_experiment swallows the + # child's non-zero exit). This keeps the per-run line and the summary in + # agreement, and never calls a mere completion "PASS" (PASS is for checks). expt_assert_run "$EXAMPLE_DIR" "$EXPT_LAST_MARKER" "$name" + if [ "$rc" -ne 0 ]; then + # Launcher itself failed: surface that, but keep the health word if the + # scan caught a more specific cause (e.g. CRASH) than a bare exit code. + case "${EXPT_LAST_HEALTH:-}" in + COMPLETED|NO_MARKER|"") RESULT[$name]="LAUNCH_EXIT=$rc" ;; + *) RESULT[$name]="${EXPT_LAST_HEALTH}(exit=$rc)" ;; + esac + else + RESULT[$name]="${EXPT_LAST_HEALTH:-COMPLETED}" + fi if [ "$rc" -ne 0 ] && [ "$STOP_ON_FAIL" = "1" ]; then echo "--stop-on-fail set; aborting remaining runs."; STOP_ALL=1; break fi diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ecfc76563..fa68021df 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -132,7 +132,7 @@ the true current picture.* The `§K` column points at the full rationale. | 4 | sct delay model | `sct = send + max(gpu, D)` (sleep-to-fill-budget) | `sct = send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU time; copying cifar's `max()` would desync real<->sim | **K-D2** | | 5 | Per-eval sct | distinct eval sct, ~20x eval speedup | **collapses to the train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20x factor); trainer eval msg is a utility report, not a clocked commit | **K-D3** | | 6 | Slot release | per-commit (inside `_sim_recv_min`) | at the **agg-goal boundary** (`_release_sim_slots_at_agg_goal`) | one grad per call and a `data_id` spans many agg-goal cycles with variance-FAIL rollbacks; per-commit release would strand a re-contributing trainer across a rollback | **K-D5**, principle #4 | -| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **dropped** (`_sim_buffer.clear()` at boundary) | a stranded grad was trained on a pre-rollback `model_version` -> stale next cycle anyway; benign at syn_0. **OPEN watch-point** -- switch to commit-then-carry if a rung shows lost updates | **K-D6** | +| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **dropped** (`_sim_buffer.clear()` at boundary) — **⚠ TRIGGERED for fluxtune (2026-07-03), see §L** | K-D6 assumed drop is benign *because* `\|selected\| ≈ agg_goal`; that holds for the two `random`/sync baselines (c≈agg_goal) but **NOT fluxtune** (async, c=10 ≫ agg_goal=3), where it drops ~7 arrived grads/cycle → re-dispatch → 2× forward passes. **DECISION REVERSED → commit-then-carry (§L).** | **K-D6**, **K-D12**, §L | | 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` (reuse the primitives, fork the orchestration) | `_sim_recv_min`'s per-commit slot release + withheld/staggered paths key on WEIGHTS semantics -- wrong for a grad pool released on the agg-goal boundary | **K-D4** | | 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's entire config corpus is `time_mode: real` and shipped with no sim path; a "simulated" default risks silently half-activating an unbuilt path | **K-D1** | | 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | the post-mutation `data_id`/`iteration_per_data_id` advance BEFORE the event emits, so binning by them is off-by-one; snapshot before the pass/fail branch makes V1 exact | **K-D9** | @@ -324,6 +324,39 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M selection-mix residual appears, check whether the **real** input is the divergent side. 7. **syn_0 byte-identical gate OFF; unavailability config-gated.** Reuse the async_cifar10 flag axes and the `simUnavailability` gate; do not delete eligibility plumbing. +8. **Fix the concept, not the symptom -- and never regress a working real<->sim example.** Before patching + a sim divergence, ask *what the mechanism is for*, not just how to get past the failure: + - **Classify the mechanism: real-transport artifact vs. algorithmic property.** Devices that exist only + to manage the MQTT/channel message queue (incremental "commit 1 per pass" draining, `recv_fifo` + timeouts, "waiting to clear up queue", persistence of uncommitted messages across calls) are + **real-transport artifacts with no sim analog** -- the sim commit path is the *barrier* (`sct`-ordered, + single-pass, slot-release at the agg-goal boundary), not a trickling queue. Applying a real-transport + discipline to the sim barrier is a category error. Gate such logic `and not self.simulated`; do **not** + re-derive it in the sim path. + - **Prefer the byte-identical-for-the-working-side fix.** When one side already works (usually real), + the correct fix leaves that side provably unchanged. `and not self.simulated` on a real-only clamp is + safe; globally reverting a flag that a prior stall-hotfix overloaded (e.g. `ends_not_selected_yet`'s + `len(ends) >= agg_goal` clause from #42) is *not* -- it risks re-opening the bug that hotfix closed. + A greedy fix in one direction is still greedy. + - **Scope-check before editing shared code.** `grep` the symbol: if it lives only in + `fwdllm_aggregator.py` the blast radius is fwdllm; if it's in `top_aggregator.py` / the shared + `parity` engine / `_sim_recv_min`, a change can silently break **async_cifar10**, the previously-green + baseline. Never trade a fwdllm fix for an async_cifar10 regression. (See principle #6: a real<->sim + gap has two fix directions -- pick the one that doesn't move the already-admissible side.) +9. **Test cadence -- match the pytest scope to the blast radius; full-suite after a cross-stack change, + not before every run.** The parity rungs + sim mechanisms are pure/deterministic over synthetic + telemetry, so pytest is the fast gate -- but scope it to what you touched: + - **fwdllm-only edit** (`fwdllm_aggregator.py`/`fwdllm_trainer.py`): `pytest tests/mode -k fwdllm`. + - **shared parity engine / `parity_checks` shim** (also feeds async_cifar10): add + `pytest examples/async_cifar10/scripts/parity` + `tests/mode -k parity`. + - **shared stack** (`flame/launch/*`, `flame/telemetry/*`, `flame/config.py`, optimizer/selector, any + `top_aggregator.py`): run the **full `pytest tests/`** -- these fan out across BOTH examples and every + aggregator/trainer, so a subset can miss the regression they'd cause. + You do **NOT** need full `pytest tests/` before every real<->sim run -- the per-scope subset is the + inner-loop gate. Run the full suite **once after a major set of changes that spans the code stack / + could affect other aggregators/trainers/examples** (this Batch 2.5 is exactly that: it touched + `flame/launch/*` + the shared `parity` engine), and bank that green before launching. A cluster run is + far more expensive than the full unit suite (minutes, no GPU/MQTT), so the insurance is cheap. --- @@ -371,6 +404,30 @@ As-built decisions in **§K-D9/D10**. Files: `parity/checks.py` (V1-V5/DK1-DK3/G `tests/mode/test_parity_checks.py` + `tests/mode/test_fwdllm_agg_telemetry.py`. **Next §H entry lands after the Phase-1 syn_0 sign-off run** (§I.6). +**Phase-1 sign-off — IN PROGRESS.** First `fwdllm` syn_0 smoke (`--mode both --delays off --max-runtime-s +300 --max-data-id 3`, 2026-07-03): real ✅ aggregated + hit `max_data_id`; sim ❌ deadlocked to +`max_runtime`, 0 aggregations. Root-caused + fixed (**K-D11**: real-only clamp on `ends_not_selected_yet`). + +**Full 3-baseline syn_0 smoke (2026-07-03, `--mode both --delays off --max-runtime-s 300 --max-data-id 3`, +batch runner) — three defects surfaced; remediation plan in §L:** +| baseline | real | sim | finding | +|---|---|---|---| +| **fwdllm** (sync) | ✅ 6 aggs, 211s | ✅ 6 aggs, 181s | mechanics OK post-K-D11; parity not yet scored | +| **fwdllm_plus** (sync) | ❌ **CRASH** | ❌ **CRASH** | `read_trainer_unavailability(base_dir=…)` kwarg mismatch vs the `metadata_dir` override → `TypeError` at `internal_init` (both modes; config-level, ORACULAR-only path). Fixed (param rename). Launcher then wasted 86s (no fail-fast) → §L defect **D-d**. | +| **fluxtune** (async) | ✅ 21 aggs, 206s | ✅ 22 aggs, **356s (1.7×)** | **residence violation on the grad path.** Per-forward-pass compute identical (5.1s), but sim ran **228 forward passes vs real 109** (even 22–23/trainer vs uneven 7–14) — sim recomputes the whole c=10 cohort every agg-goal cycle. Root: `_release_sim_slots_at_agg_goal` clears `_sim_buffer`/`_sim_inflight_expected` **before** `_sim_hold_busy_slots` reads them (holds nothing) + `sim_inflight_residence=False` + K-D6 drop. Cadence diverged downstream (data_id 2: 80 vs 8 iters → V1). §L defects **D-a/D-b/D-c**. | + +**§L Batch-2.5 LANDED (2026-07-03), pytest-gated — re-run (step 6) is now the next action.** All the +remediation code shipped and is green: the R1/W1 residence telemetry (`contributor_intervals` on +agg_round + trainer `SIM_SEND_TS` echo), the R1 in-flight-overlap + W1 compute-conservation rungs (+ +`validate_real` real-side gate), the D-a/D-b commit-then-carry fix in `_release_sim_slots_at_agg_goal` +(residence-gated: `sim_inflight_residence: true` in the fluxtune sim yaml; flag-off ⇒ byte-identical drop +for the sync baselines), the D-c fedbuff staleness-accept path (+ the `self.staleness_policy` config +wiring that was silently missing — K-D15), and the §L.5 launcher fail-fast (agg traceback detection + +immediate trainer terminate on non-zero agg exit). Pytest: `tests/mode` 297 passed / 7 skipped; parity +sub-package 115. **Next: the 3-baseline syn_0 re-run** — expect fluxtune sim forward passes ≈ real (W1 +ratio→1), R1==0% both modes, fwdllm_plus no longer crashing. Fluxtune / fwdllm_plus first-time results +above. + --- ## §I Batch 2 implementation map (code-level -- cold-start; NEXT to build) @@ -520,7 +577,9 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent in cifar carries buffered futures across the barrier. **Chose drop** for rollback-cleanliness (a stranded grad was trained on a pre-rollback model_version -> stale next cycle anyway). Benign at syn_0 (|selected| ~= agg_goal). **OPEN watch-point:** if a rung shows lost updates, switch to commit-then-carry. - **Where:** J.4, J.2. + **Where:** J.4, J.2. **⚠ SUPERSEDED by K-D12 (2026-07-03):** the watch-point TRIGGERED for fluxtune + (async, c=10 ≫ agg_goal=3) — the "|selected| ≈ agg_goal" premise holds only for the sync baselines; + async drops ~7 grads/cycle → residence violation → switch to commit-then-carry (see K-D12, §L). - **K-D7 U6 sync-barrier visibility-lag telemetry DEFERRED (compute+stash now, emit later).** The base syncfl `_aggregate_weights` finalizes `update_visibility_lag_s` from `_barrier_anchored_lags(_real_round_durs)` into a `_round_update_values` dict it then emits (`syncfl/top_aggregator.py:742-786`). fwdllm's @@ -579,3 +638,231 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent exercise is added.** **Where:** `parity/checks.py::eligible_ends_metric_parity` / `grad_norm_parity` / `agg_goal_trajectory_parity`; `tests/mode/test_parity_checks.py` (`TestDK3EligibleEndsMetric`, `TestG1GradNorm`, `TestDK1AggGoalTrajectory`). +- **K-D11 `ends_not_selected_yet` "commit 1 per pass" clamp gated real-only (`and not self.simulated`).** + Surfaced by the first Phase-1 syn_0 sign-off smoke (`--mode both`, 2026-07-03): the **real** run aggregated + and hit `max_data_id`; the **sim** run deadlocked to `max_runtime` with 0 aggregations. Root cause: in + `sync_collect_and_accumulate_grads` (`:~1486`) the flag clamps `num_min_req` (the barrier `first_k`) to 1. + That clamp is a **real-transport draining discipline** — it only works because uncommitted messages persist + in the channel queue and are re-collected next pass (real accumulates 1→agg_goal over ~10 passes). The sim + barrier `_sync_sim_recv_first_k` is single-pass: it drains the whole cohort, commits `first_k`, and **drops + the rest** (slots release at the agg-goal boundary, K-D5). With `first_k=1` it committed 1 and discarded 9; + trainers had already sent once and block in `_fetch_weights`, so the queue never refilled → goal never met. + The `len(ends) >= agg_goal` clause that trips the flag on a *full* fresh cohort is itself an overload from a + real-mode stall hotfix (#42); the flag's original meaning was just "no cohort selected yet". **Options:** + (a) revert the #42 overload globally — risks re-opening the real stall; (b) gate the clamp real-only. + **Chose (b):** real path byte-identical (`self.simulated=False`), async_cifar10 untouched (flag is + fwdllm-only), sim keeps `num_min_req=min(agg_goal, len(recv_ends))` and commits the full dynamic-K cohort + in one barrier pass. Instance of locked principle **#8** (real-transport artifact, no sim analog). **Where:** + `fwdllm_aggregator.py::sync_collect_and_accumulate_grads`. **Verify on re-run:** sim `[SIM_BARRIER]` shows + `first_k=10` + non-zero `buf_depth`, `committed` reaches agg_goal, `data_id` advances to `max_data_id` + (stops on the cap, not `max_runtime`), eval metrics emit; then parity-compare vs the real sibling. +- **K-D12 fluxtune async grad path: commit-then-CARRY the surplus buffer + hold residence (REVERSES + K-D6 for the async baseline).** Surfaced by the 2026-07-03 3-baseline smoke (§H, §L): fluxtune sim ran + 2× the forward passes of real. **Options:** (a) keep K-D6 drop (rollback-clean, but for c≫agg_goal it + discards ~7 arrived grads/cycle → residence violation → 2× compute → cadence divergence); (b) carry the + arrived-but-uncommitted grads across the agg-goal boundary and hold their trainers busy until commit — + the fwdllm analog of felix `simInflightResidence` (§3.resid) + oort `simInflightCarryover` (§4.9). + **Chose (b) for async (fluxtune); (a) stays correct for the sync baselines** (fwdllm/fwdllm_plus have + c≈agg_goal, no surplus). K-D6's "benign at syn_0" premise held only for `\|selected\|≈agg_goal`; async + breaks it. Full remediation design + staging is **§L**. **Where:** `_release_sim_slots_at_agg_goal` + (clear-before-hold ordering bug) + `_sim_hold_busy_slots` + the `sim_inflight_residence` gate. Instance + of locked principle #4 (slot residence must survive the boundary) and #8 (don't apply a sync-barrier + discipline to the async path). +- **K-D14 R1/W1 residence telemetry sourced from an ECHOED per-contribution interval, not the aggregator's + per-end dispatch stamp (Batch 2.5).** R1 (in-flight overlap) needs each grad's exact `[dispatch, + completion]` interval. **Options:** (a) read dispatch from the aggregator's `PROP_SIM_SEND_TS` end + property at commit time; (b) have the trainer ECHO `SIM_SEND_TS` back in the grad message (it already + stores the received stamp) and capture `[SIM_SEND_TS, SIM_COMPLETION_TS]` in + `_process_single_trainer_message`. **Chose (b):** the property (a) is overwritten on a re-dispatch, so if + residence is broken (exactly the case R1 must catch) it would read the NEW dispatch and hide the overlap; + the echoed pair is immutable per contribution. Emitted per cycle as `contributor_intervals` on the + agg_round event; real mode falls back to the wall interval (`PROP_ROUND_START_TIME`..receipt). W1 + (compute-conservation) reads forward passes from `trainer_round` counts vs committed contributors — both + already emitted. The rungs SKIP cleanly on the sync baselines (no re-selection ⇒ no ≥2 intervals/trainer) + and non-fwdllm runs (field absent). R1 is wired UPSTREAM of V1 in `CHECK_META` so the dep chain proves a + cadence divergence is downstream of a residence violation. **Where:** `fwdllm_trainer.py` (SIM_SEND_TS + echo), `fwdllm_aggregator.py` (`_sim_contrib_intervals` capture + `contributor_intervals` emit), + `parity/checks.py::inflight_overlap_parity`/`compute_conservation_parity`, `validate_real.py:: + check_grad_residence`, `tests/mode/test_fwdllm_sim_grad_residence.py` + R1/W1 fixtures. +- **K-D15 `self.staleness_policy` was never wired from config — the gate silently ran as `none` for every + run (as-built fix, Batch 2.5).** Implementing D-c (fedbuff staleness-accept) surfaced that + `_process_single_trainer_message` read `getattr(self, "staleness_policy", "none")` but NOTHING ever + assigned `self.staleness_policy` — so `round_data_id`/`exact`/`fedbuff` configs were all inert (always + "none"). **Fix:** set `self.staleness_policy` in `internal_init` from + `config.hyperparameters.staleness_policy`, and recognize `fedbuff` as an explicit ACCEPT policy (no reject, + no "unrecognized" warning) so its carried stale grads flow into the EXISTING fedbuff down-weighting + (`aggregate_grads_from_trainers`, `rate = 1/sqrt(1+staleness)` when `_weighted_aggregation_enabled`). A + guard warns if `staleness_policy=fedbuff` is set without a grad-aggregation-weighting optimizer (rate + stays 1.0). The U3 staleness telemetry (`_cycle_staleness` on agg_round) already emitted, so no new emit. + **Where:** `fwdllm_aggregator.py::internal_init` + `_process_single_trainer_message`; fluxtune yamls + (`staleness_policy: fedbuff`, real+sim identical); `tests/mode/test_fwdllm_agg_telemetry.py:: + TestStalenessPolicy`. +- **K-D13 fluxtune staleness_policy = fedbuff staleness-weighted accept (resolves §L D5; provisional).** + fluxtune's yaml leaves `staleness_policy` unset → defaults `none` (no staleness handling), wrong for an + async fedbuff baseline whose carried grads (K-D12) are stale by construction. **Options:** (a) fedbuff + staleness-weighted accept (down-weight by `V′−V`); (b) `round_data_id` reject+record (fwdllm_plus's sync + policy); (c) keep `none`. **Chose (a)** — fluxtune IS async fedbuff, so accept-and-down-weight is the + faithful treatment and feeds a non-trivial U3 staleness signal; **provisional** (may revisit the policy + later). Set **identically in real and sim** (a baseline definition, not a parity lever — never tuned to + close a gap; PARITY.md §F.4 / principle #3). **Obligation:** confirm the fedbuff optimizer weighting path + is active and `staleness = V′−V` emits on the commit event (U3). **Where:** `fluxtune_n10_smoke*.yaml` + `staleness_policy`; fedbuff optimizer weighting; commit-event `staleness` emit. Implement in §L step 5. + +--- + +## §L Phase-1 sign-off remediation — async grad-path residence & carry-over (Batch 2.5, staged) + +**Status: IMPLEMENTED & pytest-gated (2026-07-03); the one re-run (step 6) is the only remaining step.** +All of §L.4 steps 1–5 + §L.5 landed and are green (`tests/mode` = 297 passed / 7 skipped; parity +sub-package = 115; the new suites: `test_fwdllm_sim_grad_residence.py`, R1/W1 fixtures in +`test_parity_checks.py`, contributor-interval + staleness-accept tests in `test_fwdllm_agg_telemetry.py`). +As-built decisions are logged in **§K-D14/D15**; the code is the source of truth. **Step 6 (the 3-baseline +syn_0 re-run → `parity_checks`) is the operator's next action** — it needs the cluster and cannot run in +pytest. The first real↔sim 3-baseline +syn_0 smoke (§H, 2026-07-03) surfaced three parity defects + one tooling defect. This section states the +**principled correctness model** (what BOTH real and sim must do), root-causes each defect against the +PARITY.md ladder, and gives the **staged, pytest-gated build plan** with exact code sites, new +telemetry/checks/tests, and exit criteria. This is a Batch inserted between Batch 2 and the Phase-1 +convergence sign-off (the mechanics must be right before convergence/parity numbers mean anything). + +### §L.1 The correctness model — what real AND sim must do (async fedbuff forward-grad, fluxtune) + +The execution contract, mode-invariant (this is the "right thing" both sides must obey): + +1. **Concurrency budget C.** The aggregator keeps up to C=`c` trainers in flight. A dispatched trainer is + **BUSY** — held in the selector's `selected_ends` (a *slot*, not the unavailable list) — from dispatch + until its update **commits**. +2. **One-in-flight-per-trainer (residence invariant, PARITY.md §3.resid).** A trainer is **re-pickable + ONLY after its update returns AND is committed.** It is never re-dispatched while a prior update is + outstanding, and never re-picked for the same `(round, data_id, iteration)` (fwdllm's tightening of the + "not twice per round" FL rule). Real satisfies this by construction (the channel holds a dispatched end + out of `VAL_CH_STATE_SEND` until aggregated → real overlap 0%). Sim must model it explicitly. +3. **Buffered accumulation (fedbuff).** Arriving grads accumulate in a buffer. When `agg_goal`=K grads are + available, the aggregator runs one fedbuff step (staleness-weighted server-LR SGD) + the variance gate. + The **surplus** in-flight trainers keep computing; a completed-but-not-yet-committed grad is **retained + in the buffer and applied at a later step (carry-over)** — it is **never silently dropped**. +4. **Staleness is handled, not ignored.** A grad trained on `model_version` V that lands after the model + advanced to V′>V is **stale by V′−V** and must be either (a) accepted with fedbuff staleness weighting, + or (b) explicitly rejected AND recorded (speed/utility props still logged — PARITY.md "stale-property + recording" lesson). "Silently drop" is neither and is wrong. +5. **Progress is `data_id`** (committed variance passes), not raw update count (§B, PARITY.md §F.1). + +**Consequence to hold as an invariant/telemetry check:** over a run, **total forward passes ≈ total +committed grads + (grads still in flight at stop) + (grads explicitly stale-rejected)**. A large +unexplained gap = wasted recompute = a residence/carry violation. Real 109 fwd / 63 commit and sim 228 / +66 both show a gap; **sim's is far larger AND its participation is lock-step even** — the tell. + +### §L.2 Root-caused defects (lowest-rung first) + +- **D-a — Residence hold is nullified on the grad path (the root).** `_release_sim_slots_at_agg_goal` + (`fwdllm_aggregator.py`) runs `self._sim_buffer.clear(); self._sim_inflight_expected.clear()` **before** + `self._sim_hold_busy_slots(channel)`, which holds exactly `pending_in_buffer (∪ _sim_inflight_expected)` + — both now empty → `held={}` → it **releases every trainer**. Plus `sim_inflight_residence=False` for + fluxtune, so even the buffered set wouldn't be augmented by the in-flight set. Net: all C trainers freed + every agg-goal cycle → re-dispatched → recompute. **Evidence:** even 22–23 rounds/trainer (sim) vs uneven + 7–14 (real); 228 vs 109 forward passes; identical per-pass GPU time. **Ladder:** this is the fwdllm async + analog of felix's overlapping-re-dispatch root (PARITY.md §3.resid, "measure overlap from intervals not + counters"). +- **D-b — K-D6 buffer drop discards arrived grads (compounds D-a).** `_sim_buffer.clear()` at the boundary + drops the ~7 arrived-but-uncommitted grads (c=10, K=3). Reverses to **commit-then-carry** (K-D12). For + the sync baselines c≈K so there's no surplus — the drop stays correct there. +- **D-c — fluxtune `staleness_policy` unset → defaults `none`.** The yaml never sets it (§H config dump). + For an async fedbuff baseline the surplus grads that carry over are stale-by-construction; `none` means + no principled handling. **RESOLVED (D5/K-D13): fedbuff staleness-weighted accept**, set identically + real+sim. NOT a parity lever to tune — a *definition* fixed once (may revisit the policy later). +- **D-d — launcher has no fail-fast (tooling).** fwdllm_plus crashed in `internal_init` at ~5s but the run + burned 86s: `aggregator_spawner.wait_until_ready` is a fixed 5s "process alive" heuristic (misses a + crash landing at ~5s), so trainers spawned anyway; then `runner.py` `wait_all(timeout_per_trainer=30.0)` + waits 30s for trainers that will never get an EOT. Waste the operator could spend fixing the error. + +### §L.3 What "correct" telemetry/checks look like — mirror the async_cifar10 lessons + +The PARITY.md discipline (§1 ladder, "growth rule: every root leaves behind the finest check that would +have localized it"): D-a would have been caught instantly by a **residence/overlap invariant** rung. Add +these, reusing the shared engine (`async_cifar10/scripts/parity/checks.py` + STAGE map, §I.2): + +| New/validate | Rung | Role/Tier | What it asserts | Telemetry it needs | +|---|---|---|---|---| +| **NEW** | **R1 in-flight overlap** | MECHANISM/**INV** | per-trainer dispatch→commit intervals do NOT overlap (one-in-flight); ≈0% both modes (felix metric, PARITY.md L295-298) | per-(trainer,cycle) `sim_send_ts`/dispatch ts + commit `vclock`/wall ts | +| **NEW** | **W1 compute conservation** | CONTROL/DIAG | `forward_passes ≈ commits + in_flight_at_stop + stale_rejected`, and the **real↔sim ratio ≈1**; localizes wasted recompute | count `trainer_round` events vs committed grads per mode (already emitted) | +| validate | **S2 participation** | EMERGENT/DIST | per-trainer chosen-count matches; the even-vs-uneven split is exactly this — must FAIL pre-fix, PASS post-fix (fluxtune only) | `selection_train` (exists) | +| validate | **V1 iters-per-data_id** | MECHANISM/DIST | data_id-2 80-vs-8 must flag; confirms V1 (Batch 2) fires and is **downstream** of R1 (dep chain proves it) | `cycle_data_id` (K-D9, exists) | +| validate | **U3 staleness** | MECHANISM/DIST | fedbuff staleness dist matched + non-trivial once D-c fixed | `staleness` in agg_round (exists) | + +**Sanity gate (validate_real FIRST — HARD GATE, PARITY.md principle #6 / "verify real is correct before +tuning sim").** Real also shows fwd(109) > commit(63). Before making sim match real, **confirm real is +admissible:** (1) real R1 overlap is truly ~0% (channel residence), (2) the real fwd−commit gap is +explained by end-of-run in-flight + any stale-rejections, NOT by real *also* dropping carried grads. **If +real is itself wrong (overlap > 0, or real drops carried grads), FIX REAL FIRST** — make real correct, +re-bank it as the reference, and only then bring sim to parity (D-b becomes a two-sided fix, real side +first). Never tune sim toward a real that has not passed its own admissibility. Add a `validate_real` +assertion for R1==0% and the W1 decomposition; it **blocks** the §L.4 step-4 sim mechanism change. + +### §L.4 Staged build (pytest-gated; flag-off ⇒ byte-identical; async-only, sync untouched) + +Ordering follows the ladder: instrument → establish real-correctness → fix the mechanism → prove with +tests → re-run. Each step gated by pytest before the one run. **Steps 1–5 DONE & green (2026-07-03); +step 6 is the operator re-run.** As-built notes in K-D14/D15. + +1. **✅ Telemetry (enables R1/W1).** Emit per-(trainer,cycle) dispatch ts + commit ts so overlap intervals + are reconstructable (extend the agg_round `extra` and/or a `SIM_GRAD_RECV`-adjacent field; + `sim_send_ts` + commit `vclock` already exist internally — surface them). Add nothing to the real hot + path beyond a timestamp. *Exit:* a driven emission test asserts the fields land (mirror + `test_fwdllm_agg_telemetry.py`). +2. **✅ Checks R1 + W1 in the shared engine** (`checks.py` `run_all_parity` + STAGE map; deps: R1 under + Stage 3 residence, W1 CONTROL feeding V1/K2). *Exit:* known-PASS + known-broken synthetic fixtures + (an overlapping-interval fixture FAILS R1; a 2×-recompute fixture FAILS W1) in `test_parity_checks.py`. +3. **✅ validate_real sanity** (R1==0% + W1 decomposition on the real dir) — run against the **already-banked + 2026-07-03 real run dirs** (checker-side, no cluster run; PARITY.md "checker fixes validate instantly"). + Decides whether D-b is one-sided or two-sided. +4. **✅ Mechanism fix (D-a + D-b), async grad path only.** In `_release_sim_slots_at_agg_goal`: (i) **hold + before clear** — compute the held set from the pre-clear buffer/in-flight, release only the *committed* + subset, keep the surplus buffered (commit-then-carry); (ii) turn on residence for fluxtune + (`sim_inflight_residence=True` in `fluxtune_n10_smoke_sim.yaml`, and confirm `_sim_hold_busy_slots` holds + `pending_in_buffer ∪ _sim_inflight_expected`). Reference implementations: felix `_sim_hold_busy_slots` + (`asyncfl/top_aggregator.py:1453`, §3.resid) and oort `simInflightCarryover` (§4.9). **Scope-guard + (principle #8):** the change is in `fwdllm_aggregator.py` (fwdllm blast radius) + a fluxtune yaml; do + NOT touch the shared `_sim_recv_min`/`asyncfl` weight path (async_cifar10 is green). Sync path + (`_sync_sim_recv_first_k`, K-D5/K-D11) unchanged — K-D6 drop stays correct there. *Exit:* a + `test_fwdllm_sim_grad_residence.py` (mirror `test_async_inflight_residence.py`): drive c>K synthetic + grads, assert surplus carried (not dropped), busy trainers not re-selected, R1==0%; + flag-off + byte-identical. +5. **✅ Staleness definition (D-c) — D5 LOCKED: fedbuff-accept (K-D13/K-D15).** Wire fluxtune to fedbuff + staleness-weighted accept: set `staleness_policy` in all fluxtune yamls (real+sim identical) so a + carried stale grad is **accepted and down-weighted by `V′−V`**, and confirm the fedbuff optimizer's + weighting path is active + emits `staleness` on the commit event (for U3). *Exit:* a unit test that a + carried stale grad is accepted+weighted (never silently dropped) and that `staleness=V′−V` is emitted. +6. **⏳ One re-run** (operator step; only after 1–5 green — now the case): the 3-baseline syn_0 smoke → `parity_checks`. *Exit:* fluxtune + sim forward passes ≈ real (W1 ratio →1), R1==0% both modes, S2 PASS, V1 data_id-iters converge, sim + wall ≈ real; fwdllm_plus no longer crashes; record in §H. Then proceed to the Phase-1 convergence + sign-off (§I.6). + +### §L.5 ✅ Launcher fail-fast (D-d) — parallel tooling task (no parity coupling) — DONE + +Independent of the aggregation work; do in the harness/launcher. Two changes in `flame/launch/`: +- **Detect early aggregator death before committing to trainers.** `aggregator_spawner.wait_until_ready` + (`:135`) should treat a process exit during the window as `False` immediately (it already checks + `is_running()` each second — return `False` on death instead of the 5s "alive ⇒ ready" heuristic; better, + watch the agg log for a readiness marker / a traceback). If not ready → abort the run, do NOT + `spawn_all` trainers. +- **On aggregator non-zero exit, terminate trainers immediately.** `runner.py:324-327`: when + `agg_rc not in (0, None)`, skip `wait_all(timeout_per_trainer=30.0)` and call the trainer spawner's + terminate path now (the run has failed; the 30s EOT grace is only meaningful on a clean finish). +*Exit:* a crashed-aggregator run returns in ~seconds, not ~90s; the harness health line already reads +CRASH (done this session). Optional: surface the aggregator's traceback tail in the runner output so the +operator sees the cause without opening the log. + +### §L.6 Locked decisions + +- **D5 — fluxtune staleness treatment: RESOLVED → (a) fedbuff staleness-weighted accept (K-D13).** A + carried grad trained on `model_version` V that commits at V′ is **accepted and down-weighted by `V′−V`** + (fedbuff default, faithful to an async buffered aggregator). Rejected: (b) `round_data_id` reject+record + (fwdllm_plus's sync policy — discards work an async baseline is designed to consume); (c) unset→`none` + (no staleness signal for U3). **Provisional lock** — keep fedbuff-accept for the Phase-1 async parity + pass; MAY revisit later. It is a baseline *definition*, not a parity lever (PARITY.md §F.4 / principle + #3): **set identically real+sim, never tuned to close a gap.** Obligation: confirm the fedbuff optimizer + weighting path is wired and `staleness = V′−V` emits on the commit event so U3 can score it. +- **Verify-real-first is a HARD GATE.** Real correctness is established BEFORE any sim mechanism change + (§L.3 sanity gate blocks §L.4 step 4). If real is itself wrong (R1 overlap > 0, or real drops carried + grads), we **fix real first**, re-bank it as the reference, then bring sim to parity — never tune sim to + an inadmissible real (PARITY.md "verify real is correct first — a gap has two fix directions"). diff --git a/lib/python/examples/scripts/expt_runner.py b/lib/python/examples/scripts/expt_runner.py index 09c7447f2..9c14a1ab5 100644 --- a/lib/python/examples/scripts/expt_runner.py +++ b/lib/python/examples/scripts/expt_runner.py @@ -71,13 +71,16 @@ def dim(self, s): return self._w("2", s) def bold(self, s): return self._w("1", s) -# level -> (icon, colouriser name). "ok" rows stay quiet; warn/error shout. -_ICON = {"ok": " ", "warn": "\U0001f7e1", "error": "\U0001f534"} # · 🟡 🔴 +# level -> (icon, colouriser name). "ok" rows stay quiet; set/warn/error shout. +# set = value explicitly overridden by a command-line flag (overrides yaml) -> green +# warn = review / attention -> yellow +# error = infeasible -> red +_ICON = {"ok": " ", "set": "\U0001f7e2", "warn": "\U0001f7e1", "error": "\U0001f534"} # 🟢 🟡 🔴 _CHECK_ICON = {"ok": "✓", "warn": "⚠", "error": "✗"} # ✓ ⚠ ✗ def _colour_for(st: _Style, level: str): - return {"ok": st.dim, "warn": st.yellow, "error": st.red}.get(level, st.dim) + return {"ok": st.dim, "set": st.green, "warn": st.yellow, "error": st.red}.get(level, st.dim) # ---- renderer -------------------------------------------------------------- @@ -121,15 +124,18 @@ def out(s=""): n_err_rows += 1 icon = _ICON.get(level, " ") col = _colour_for(st, level) - label = f"{r.get('label', ''):<13}" + # Pad short labels to a column; always keep >=1 space before the value + # so a label at/over the column width doesn't run into it. + raw = r.get("label", "") + label = raw.ljust(13) if len(raw) < 13 else raw + " " value = r.get("value", "") note = r.get("note", "") note_s = f" {st.dim('· ' + note)}" if note else "" # ok rows: label dim, value plain. warn/error: value coloured. if level == "ok": - out(f" {icon} {st.dim(label)}{value}{note_s}") + out(f" {icon} {st.dim(label)} {value}{note_s}") else: - out(f" {icon} {st.bold(label)}{col(value)}{note_s}") + out(f" {icon} {st.bold(label)} {col(value)}{note_s}") out(" " + _RULE) diff --git a/lib/python/examples/scripts/expt_runner.sh b/lib/python/examples/scripts/expt_runner.sh index d8fa83c5f..04b94e0ad 100644 --- a/lib/python/examples/scripts/expt_runner.sh +++ b/lib/python/examples/scripts/expt_runner.sh @@ -125,12 +125,19 @@ expt_launch() { # SIM_WALL_CEILING / SIM_STARVATION / tracebacks live in the dedicated # experiments/run_*/*_aggregator.log. Only files newer than `marker` (touched by # expt_launch pre-launch) are scanned, so it attributes signals to THIS run. -# Prints a one-line verdict; returns 0 only if there were agg_rounds, no +# Prints a one-line verdict and exports EXPT_LAST_HEALTH with the verdict word +# (COMPLETED / CRASH / NO_AGG_ROUNDS / WALL_CEILING / NO_MARKER) so a driver can +# fold it into its own summary; returns 0 only if there were agg_rounds, no # [SIM_WALL_CEILING], and no crash marker. +# +# Vocabulary note: a healthy run is "COMPLETED", NOT "PASS". Completing is a +# process outcome, not a check result -- PASS/FAIL is reserved for actual +# checks (e.g. the real<->sim parity battery in scripts.parity.cli). expt_assert_run() { local example_dir="$1" marker="$2" label="${3:-run}" local expdir="$example_dir/experiments" if [ ! -e "$marker" ]; then + EXPT_LAST_HEALTH="NO_MARKER"; export EXPT_LAST_HEALTH echo " [$label] NO_MARKER (cannot scope health check)"; return 1 fi local agg=0 stop=0 wall=0 starv=0 crash=0 nlogs=0 f n @@ -144,10 +151,12 @@ expt_assert_run() { n=$(grep -c "\[SIM_STARVATION\]" "$f" 2>/dev/null); starv=$(( starv + ${n:-0} )) n=$(grep -cE "Traceback \(most recent call last\)|CUDA error|Segmentation fault" "$f" 2>/dev/null); crash=$(( crash + ${n:-0} )) done < <(find "$expdir" -name "*_aggregator.log" -newer "$marker" 2>/dev/null) - local status="PASS" + # A healthy, fully-run experiment is COMPLETED (a process outcome), not PASS. + local status="COMPLETED" [ "$agg" -eq 0 ] && status="NO_AGG_ROUNDS" [ "$wall" -gt 0 ] && status="WALL_CEILING" [ "$crash" -gt 0 ] && status="CRASH" + EXPT_LAST_HEALTH="$status"; export EXPT_LAST_HEALTH printf " [%s] %-14s agg_round=%s stopping_run=%s wall_ceiling=%s starvation=%s crash=%s (%s agg log(s))\n" \ "$label" "$status" "$agg" "$stop" "$wall" "$starv" "$crash" "$nlogs" [ "$agg" -gt 0 ] && [ "$wall" -eq 0 ] && [ "$crash" -eq 0 ] diff --git a/lib/python/flame/launch/aggregator_spawner.py b/lib/python/flame/launch/aggregator_spawner.py index e109062c8..6a4e43ff4 100644 --- a/lib/python/flame/launch/aggregator_spawner.py +++ b/lib/python/flame/launch/aggregator_spawner.py @@ -119,12 +119,46 @@ def is_running(self) -> bool: return False return self.process.poll() is None + # A startup crash (e.g. a config/init TypeError) writes a Python traceback to + # the log just before the process exits. Detecting it lets wait_until_ready + # fail FAST instead of the old 5s "alive ⇒ ready" heuristic waving through a + # crash that lands at ~5s -- which then wasted ~86s spawning + waiting on + # trainers that never get an EOT (simulate_fwdllm.md §L.5 defect D-d). + _CRASH_MARKER = "Traceback (most recent call last)" + + def _read_log_tail(self, max_bytes: int = 65536) -> str: + if not self.log_file: + return "" + p = Path(self.log_file) + if not p.exists(): + return "" + try: + with open(p, "r", errors="replace") as f: + data = f.read() + return data[-max_bytes:] + except OSError: + return "" + + def _print_crash_tail(self, n_lines: int = 25) -> None: + """Surface the aggregator log tail so the operator sees the cause without + opening the log (§L.5).""" + tail = self._read_log_tail() + if not tail: + return + lines = tail.rstrip().splitlines()[-n_lines:] + print(" ── aggregator log tail ──") + for ln in lines: + print(f" {ln}") + print(" ─────────────────────────") + def wait_until_ready(self, timeout: int = 30) -> bool: """ - Wait for aggregator to be ready. + Wait for aggregator to be ready, failing fast on a startup crash. - Simple implementation: just wait fixed time and check process is alive. - Could be enhanced with log monitoring for "ready" message. + Positive signal: process alive for >=5s with no traceback in the log. + Negative signals (return False immediately): the process exits during the + window, or a traceback appears in the log -- either way the run has + failed, so the caller must NOT spawn trainers. Args: timeout: Maximum time to wait in seconds @@ -139,14 +173,23 @@ def wait_until_ready(self, timeout: int = 30) -> bool: while time.time() - start_time < timeout: if not self.is_running(): - print(f" ✗ Aggregator process died") + print(f" ✗ Aggregator process died during startup") + self._print_crash_tail() + return False + if self._CRASH_MARKER in self._read_log_tail(): + print(f" ✗ Aggregator logged a traceback during startup") + self._print_crash_tail() return False time.sleep(check_interval) elapsed = time.time() - start_time - # Simple heuristic: if process is alive for 5 seconds, assume ready + # Alive for 5s AND no startup traceback => assume ready. if elapsed >= 5: + if self._CRASH_MARKER in self._read_log_tail(): + print(f" ✗ Aggregator logged a traceback during startup") + self._print_crash_tail() + return False print(f" ✓ Aggregator ready (process alive for {elapsed:.1f}s)") return True diff --git a/lib/python/flame/launch/runner.py b/lib/python/flame/launch/runner.py index 86eba2de3..84775aa7b 100644 --- a/lib/python/flame/launch/runner.py +++ b/lib/python/flame/launch/runner.py @@ -323,8 +323,18 @@ def run_experiment(self, exp: ExperimentConfig) -> None: self.aggregator_spawner.terminate() agg_rc = getattr(self.aggregator_spawner.process, "returncode", None) rc_msg = f"exit={agg_rc}" if agg_rc == 0 else f"exit={agg_rc} ⚠" - print(f" aggregator done ({rc_msg}), waiting for trainers to exit...") - self.trainer_spawner.wait_all(timeout_per_trainer=30.0) + if agg_rc not in (0, None): + # The aggregator failed (non-zero exit). The 30s-per-trainer EOT + # grace below is only meaningful on a CLEAN finish -- on a crash + # the trainers will never get an EOT, so terminate them now + # instead of burning ~30s each waiting (simulate_fwdllm.md §L.5 + # defect D-d). + print(f" aggregator FAILED ({rc_msg}); terminating trainers now " + f"(skipping EOT grace).") + self.trainer_spawner.terminate_all() + else: + print(f" aggregator done ({rc_msg}), waiting for trainers to exit...") + self.trainer_spawner.wait_all(timeout_per_trainer=30.0) print("\nexperiment completed.") # Auto post-run analysis: parse the telemetry JSONL and emit plots. diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index c032b44f9..202e06a61 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -289,6 +289,15 @@ def internal_init(self) -> None: self.ends_not_selected_yet = False self.iteration_per_data_id = 0 + # end_id -> {"dispatch_ts", "commit_ts"} for the most recent accepted + # contribution of that end, captured in _process_single_trainer_message + # and emitted per-cycle as `contributor_intervals` on the agg_round + # event. Feeds the R1 in-flight-overlap + W1 compute-conservation rungs + # (simulate_fwdllm.md §L.3). Sim uses the modeled [SIM_SEND_TS, + # SIM_COMPLETION_TS] vclock interval; real falls back to the wall + # interval (PROP_ROUND_START_TIME .. receipt). + self._sim_contrib_intervals = {} + # Selection granularity for the sync path (fwdllm/fwdllm_plus): # True (default, preserves pre-existing behavior) = re-select # trainers on every SEND-state call, i.e. every iteration of every @@ -308,11 +317,31 @@ def internal_init(self) -> None: # but not formally departed (see ROUND_CACHE_STUCK_TIMEOUT_S). self._round_cache_activity_ts: dict = {} + # Staleness handling (flame/config.py Hyperparameters.staleness_policy). + # Read from config into an instance attr so the message handler's + # getattr(self, "staleness_policy", "none") resolves the configured + # value -- without this it silently stayed "none" for every run. Values: + # none (accept, no gate) / round_data_id / exact (reject stale) / + # fedbuff (accept + down-weight by V'-V, the async baseline default, + # K-D13). See _process_single_trainer_message. + self.staleness_policy = getattr( + self.config.hyperparameters, "staleness_policy", None + ) or "none" + logger.info(f"staleness_policy = {self.staleness_policy}") + self._optimizer_sort_value = self.config.optimizer.sort OPTIMIZERS_SUPPORTING_GRAD_AGGREGATION = (OptimizerType.FEDBUFF,) self._weighted_aggregation_enabled = ( self._optimizer_sort_value in OPTIMIZERS_SUPPORTING_GRAD_AGGREGATION ) + if self.staleness_policy == "fedbuff" and not self._weighted_aggregation_enabled: + logger.warning( + "staleness_policy=fedbuff accepts stale grads but optimizer.sort=" + f"{self._optimizer_sort_value} does not support staleness " + "down-weighting (rate stays 1.0); staleness will be recorded but " + "not weighted. Set optimizer.sort=fedbuff for the intended " + "down-weight-by-(V'-V) treatment (K-D13)." + ) if not self._weighted_aggregation_enabled: logger.info( f"Setting rate=1.0 for all updates because optimizer.sort is " @@ -525,7 +554,7 @@ def _process_trainer_heartbeat(self, msg, end) -> None: logger.warning(f"Got invalid {msg} while processing heartbeat") def read_trainer_unavailability( - self, trace=None, metadata_dir: Optional[Union[str, Path]] = None + self, trace=None, base_dir: Optional[Union[str, Path]] = None ) -> dict: """Build task_id -> SortedDict(timestamp -> state) for `trace`. @@ -533,10 +562,20 @@ def read_trainer_unavailability( traces), mirroring async_cifar10/aggregator/pytorch/main_oort_sync_agg.py's pattern -- not from the legacy per-trainer json_scripts/trainer_*.json files. + + NOTE: the parameter MUST be named `base_dir` to match the caller in + ClientAvailability._init_availability (client_availability.py), which + invokes self.read_trainer_unavailability(trace=..., base_dir=...). This + method shadows the mixin's wrapper of the same name; a mismatched + parameter name here raises TypeError at aggregator init (the fwdllm_plus + ORACULAR-availability crash). The canonical implementation now lives in + flame.availability.trace.read_trainer_unavailability -- this override is + a candidate for deletion once mobiperf/syn parity with load_trace is + confirmed. """ logger.info(f"Reading trainer unavailability for trace: {trace}") - metadata_dir = Path(metadata_dir) if metadata_dir is not None else _METADATA_DIR + metadata_dir = Path(base_dir) if base_dir is not None else _METADATA_DIR registry_path = metadata_dir / "trainer_registry.yaml" with open(registry_path) as f: @@ -786,19 +825,47 @@ def _sim_recv_min_grad(self, channel, recv_ends): return m, md def _release_sim_slots_at_agg_goal(self, channel, is_async): - """Sim rollback-safety at the agg-goal boundary (simulate_fwdllm.md §I.5). + """Sim residence + rollback-safety at the agg-goal boundary. A data_id spans MANY agg-goal cycles (a variance-FAIL retries the same - data_id with iteration_per_data_id += 1). Clear the per-cycle committed - marks and drain any stranded reorder-buffer entries so the next cycle - starts clean -- otherwise the gate would skip a trainer re-contributing - on the rolled-back data_id (still flagged committed), and a stranded - buffered grad would commit past-dated into the next cycle. For the async - path, holding the now-empty busy set frees every committed trainer's - concurrency slot for re-selection (one-in-flight per cycle). + data_id with iteration_per_data_id += 1). The per-cycle committed marks + (_sim_committed) MUST clear each cycle so the gate does not skip a + trainer re-contributing on the rolled-back data_id. + + Two boundary policies, keyed on the async grad path + the residence flag + (simulate_fwdllm.md §L / K-D12): + + * async + sim_inflight_residence (fluxtune, c >> agg_goal) -- + COMMIT-THEN-CARRY. Hold the still-busy trainers (surplus buffered + arrivals ∪ not-yet-arrived in-flight) in their concurrency slots + BEFORE clearing anything, so residence survives the boundary + (_sim_hold_busy_slots reads _sim_buffer/_sim_inflight_expected -- both + must still be populated). Only the committed subset (already popped + from the buffer in _sim_recv_min_grad) is released for re-selection. + The surplus arrived-but-uncommitted grads are RETAINED in the buffer + and applied at a later fedbuff step (never dropped); their version gap + is handled by staleness_policy (K-D13), not by discarding work. This is + the fwdllm analog of felix simInflightResidence + oort + simInflightCarryover. + + * everything else (the two sync baselines c ≈ agg_goal, or async with the + flag off) -- LEGACY DROP (K-D5/K-D6). A single-pass barrier has no + surplus, so clearing the buffer + in-flight gate is correct and keeps + the flag-off path byte-identical to Batch 1. """ if not self.simulated: return + _residence = getattr(self, "_sim_inflight_residence", False) + if is_async and _residence: + # Hold BEFORE clear: the held set = pending_in_buffer ∪ + # _sim_inflight_expected (residence on), released set = committed. + self._sim_hold_busy_slots(channel) + self._sim_committed.clear() + # Deliberately DO NOT clear _sim_buffer / _sim_inflight_expected: + # the surplus carries to the next agg-goal cycle (buffered fedbuff + # accumulation) and the not-yet-arrived trainers stay in flight. + return + # Legacy drop path (sync barrier, or async without residence). self._sim_committed.clear() self._sim_buffer.clear() self._sim_inflight_expected.clear() @@ -898,6 +965,12 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): # (round, data_id) when inc_model_version_per_data_id is set (it # only advances on a data_id transition), so "round_data_id" needs # no extra fields; "exact" additionally checks iteration_per_data_id. + # REJECT policies: round_data_id, exact -- drop a stale grad. + # ACCEPT policies: none (no gate), fedbuff (accept + down-weight by + # V'-V in aggregate_grads_from_trainers, K-D13/§L D5). fedbuff is + # the async baseline default: its carried surplus grads (§L + # commit-then-carry) are stale by construction, so they must be + # consumed and weighted, never dropped. policy = getattr(self, "staleness_policy", "none") stale, stale_reason = False, None if policy == "round_data_id": @@ -916,7 +989,7 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): f"iteration_per_data_id={msg_iter} != " f"agg iteration_per_data_id={self.iteration_per_data_id}" ) - elif policy != "none": + elif policy not in ("none", "fedbuff"): logger.warning( f"Unrecognized staleness_policy={policy!r}; treating as 'none' " f"(no staleness gate)." @@ -967,6 +1040,25 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): logger.info( f"Set PROP_CLIENT_TASK_TRAIN_DURATION for {end}: {round_duration.total_seconds():.3f}s" ) + + # Record this contribution's [dispatch, commit] interval for the R1 + # in-flight-overlap + W1 compute-conservation rungs (§L.3). Sim: the + # modeled vclock interval the trainer echoes back (exact per + # contribution, so a re-dispatch overwrite can't corrupt a prior + # interval). Real: the wall interval (dispatch prop .. receipt). The + # per-cycle list is assembled + emitted in _process_aggregation_goal_met. + if getattr(self, "_sim_contrib_intervals", None) is not None: + if self.simulated: + _disp = msg.get(MessageType.SIM_SEND_TS) + _comm = msg.get(MessageType.SIM_COMPLETION_TS) + else: + _disp = (round_start_time_tup[1].timestamp() + if round_start_time_tup is not None else None) + _comm = timestamp.timestamp() if hasattr(timestamp, "timestamp") else None + self._sim_contrib_intervals[end] = { + "dispatch_ts": float(_disp) if _disp is not None else None, + "commit_ts": float(_comm) if _comm is not None else None, + } else: logger.error( f"Invalid message received from {end} in aggregate_weights: {msg}" @@ -1313,6 +1405,16 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): _grad_pool_size = len(_grad_pool) if _grad_pool is not None else None _cached_v = getattr(self, "cached_shared_grad_pool_trainable", None) _cached_v_size = len(_cached_v) if _cached_v is not None else 0 + # Per-contributor [dispatch, commit] intervals for R1/W1 (§L.3): one + # entry per end that committed into THIS cycle. History is preserved + # because each cycle emits its own list (the per-end dict is overwritten + # on a later contribution, but the emitted event already captured it). + _contrib_map = getattr(self, "_sim_contrib_intervals", None) or {} + _contributor_intervals = [ + {"end": str(_e), **_contrib_map.get(_e, {"dispatch_ts": None, + "commit_ts": None})} + for _e in _cycle_contributors + ] if self.var_good_enough: _pass_kind = ( @@ -1414,6 +1516,9 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): "cycle_iteration": _cycle_iteration, "grad_pool_size": _grad_pool_size, "cached_v_size": _cached_v_size, + # R1/W1 residence rungs (§L.3): per-contributor + # [dispatch_ts, commit_ts] intervals for this cycle. + "contributor_intervals": _contributor_intervals, }, ) telemetry.emit(ev, **fields) @@ -1483,7 +1588,10 @@ def sync_collect_and_accumulate_grads(self, tag, channel): num_min_req = self._agg_goal # change hardcoding, set it to aggGoal logger.info(f"Total ends: {len(recv_ends)}, required : {num_min_req}") num_min_req = min(num_min_req, len(recv_ends)) - if self.ends_not_selected_yet: + # Real-only: "commit 1 per pass" relies on uncommitted msgs persisting in + # the queue. The sim barrier drains + drops past first_k, so clamping to 1 + # strands the cohort -> deadlock. See simulate_fwdllm.md §F #8. + if self.ends_not_selected_yet and not self.simulated: logger.info(f"We are waiting to clear up queue") num_min_req = min(num_min_req, 1) diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index 5d33b08b8..e168833bb 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -534,6 +534,11 @@ def _send_grads(self, tag: str) -> None: # the intrinsic (server-overhead-free) task duration from. All # None in real mode -> aggregator ignores them (arrival order). MessageType.SIM_COMPLETION_TS: self._sim_completion_ts, + # Echo the dispatch stamp back so the aggregator can reconstruct + # this contribution's exact [dispatch, completion] interval for + # the R1 in-flight-overlap rung (simulate_fwdllm.md §L.3). None + # in real mode -> aggregator falls back to the wall interval. + MessageType.SIM_SEND_TS: self._sim_send_ts, MessageType.SIM_CLIENT_TASK_TRAIN_DURATION_S: self._sim_round_duration_s, MessageType.TRAINING_BUDGET_S: self._sim_round_duration_s, MessageType.WALL_SEND_TS: time.time(), diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index 7466fc9c6..19f88d163 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -322,6 +322,58 @@ def test_agg_observed_s_keyed_by_end_id(self, tmp_path): telemetry.shutdown() +class TestContributorIntervalsEmission: + """R1/W1 residence rungs (§L.3) read a per-contributor [dispatch, commit] + interval list off each agg_round event. It must land once per contributor, + carrying the ts captured in _process_single_trainer_message.""" + + def test_intervals_emitted_per_contributor(self, tmp_path): + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1", "t2"], var_good_enough=False) + # Simulate the per-contribution capture done in the message handler. + agg._sim_contrib_intervals = { + "t1": {"dispatch_ts": 1.0, "commit_ts": 6.0}, + "t2": {"dispatch_ts": 2.0, "commit_ts": 9.0}, + } + channel = _FakeChannel( + durations={"t1": timedelta(seconds=5), "t2": timedelta(seconds=7)}, + ) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + ci = {d["end"]: d for d in r["contributor_intervals"]} + assert set(ci) == {"t1", "t2"} + assert ci["t1"]["dispatch_ts"] == 1.0 and ci["t1"]["commit_ts"] == 6.0 + assert ci["t2"]["dispatch_ts"] == 2.0 and ci["t2"]["commit_ts"] == 9.0 + finally: + telemetry.shutdown() + + def test_intervals_present_even_without_captured_ts(self, tmp_path): + """When no interval was captured (e.g. a test double / real run with the + dict unpopulated) the field is still emitted with null ts, so the rung + SKIPs cleanly rather than the field being absent.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + channel = _FakeChannel(durations={"t1": timedelta(seconds=5)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["contributor_intervals"] == [ + {"end": "t1", "dispatch_ts": None, "commit_ts": None}] + finally: + telemetry.shutdown() + + class _UtilityFakeChannel: """Generic fake for _process_single_trainer_message's channel calls -- stores per-end properties in a dict, doesn't care about specific PROP_* @@ -488,6 +540,42 @@ def test_noop_when_telemetry_disabled(self, tmp_path): assert not (tmp_path / "aggregator.jsonl").exists() +class TestStalenessPolicy: + """staleness_policy gate in _process_single_trainer_message (§L D-c/K-D13). + + REJECT policies (round_data_id/exact) drop a stale grad; ACCEPT policies + (none/fedbuff) consume it -- fedbuff is the async baseline default so its + carried surplus grads (commit-then-carry) are accepted + down-weighted by + (V'-V) in aggregate_grads_from_trainers, never silently dropped.""" + + def _run(self, policy, msg_version, agg_version=5): + agg = _UtilityFakeAggregator(model_version=agg_version, is_async=True) + agg.staleness_policy = policy + channel = _UtilityFakeChannel() + agg.process(channel, _msg(model_version=msg_version, stat_utility=0.5), + "t1", timestamp=0) + return agg + + def test_fedbuff_accepts_stale_grad(self): + # msg trained on v3, agg now at v5 -> stale by 2, but fedbuff ACCEPTS it. + agg = self._run("fedbuff", msg_version=3) + assert agg._agg_goal_cnt == 1 # grad consumed + assert agg._per_agg_trainer_list == ["t1"] + + def test_none_accepts_stale_grad(self): + agg = self._run("none", msg_version=3) + assert agg._agg_goal_cnt == 1 + + def test_round_data_id_rejects_stale_grad(self): + agg = self._run("round_data_id", msg_version=3) + assert agg._agg_goal_cnt == 0 # dropped + assert agg._per_agg_trainer_list == [] + + def test_fedbuff_accepts_fresh_grad(self): + agg = self._run("fedbuff", msg_version=5) # not stale + assert agg._agg_goal_cnt == 1 + + class TestRoundCacheActivityResetOnContribution: """A real accepted contribution must restart the round-cache stuck-timeout clock (see TestStuckCachePruning in diff --git a/lib/python/tests/mode/test_fwdllm_oracular_avail.py b/lib/python/tests/mode/test_fwdllm_oracular_avail.py index aeb0875a1..f7e16e53f 100644 --- a/lib/python/tests/mode/test_fwdllm_oracular_avail.py +++ b/lib/python/tests/mode/test_fwdllm_oracular_avail.py @@ -45,7 +45,7 @@ def test_reads_from_metadata_bundle(tmp_path): metadata_dir = _write_metadata(tmp_path) result = TopAggregator.read_trainer_unavailability( - None, trace="mobiperf_2st", metadata_dir=metadata_dir + None, trace="mobiperf_2st", base_dir=metadata_dir ) assert set(result.keys()) == {"task-aaa", "task-bbb"} @@ -62,6 +62,6 @@ def fail_glob(*args, **kwargs): monkeypatch.setattr(glob_module, "glob", fail_glob) result = TopAggregator.read_trainer_unavailability( - None, trace="mobiperf_2st", metadata_dir=metadata_dir + None, trace="mobiperf_2st", base_dir=metadata_dir ) assert len(result) == 2 diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py new file mode 100644 index 000000000..ba5f6d450 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -0,0 +1,154 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 2.5 (simulate_fwdllm.md §L.4 step 4): async grad-path residence + +commit-then-carry. + +The 2026-07-03 fluxtune smoke ran 2x the forward passes of real: a residence +violation. `_release_sim_slots_at_agg_goal` cleared the reorder buffer + the +in-flight gate BEFORE `_sim_hold_busy_slots` read them, so it held nothing and +freed every trainer every agg-goal cycle -> re-dispatch -> recompute; and the +K-D6 buffer drop discarded the ~7 arrived-but-uncommitted grads (c=10 >> K=3). + +The fix (K-D12): for the async path with `sim_inflight_residence` on, HOLD the +still-busy trainers (surplus buffered ∪ not-yet-arrived in-flight) in their +slots BEFORE clearing anything, release only the committed subset, and CARRY the +surplus buffer to the next cycle (never dropped). These tests drive the boundary +directly and assert (a) surplus carried, (b) busy trainers not re-selected, +(c) R1 one-in-flight residence holds, and (d) flag-off => byte-identical to the +Batch-1 drop behavior (sync baselines + async-without-residence unchanged). +""" + +from flame.mode.horizontal.asyncfl.top_aggregator import ( + TopAggregator as _AsyncBase, +) +from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator +from flame.mode.horizontal.syncfl.top_aggregator import TopAggregator as _SyncBase +from flame.mode.message import MessageType +from flame.sim.virtual_clock import SimReorderBuffer, VirtualClock + +from tests.mode.test_fwdllm_sim_grad_loop import ( + _FakeGradAgg, + _FakeSelChannel, +) + + +def _residence_agg(residence: bool) -> _FakeGradAgg: + agg = _FakeGradAgg() + agg._sim_pending_commit = set() + agg._sim_inflight_residence = residence + return agg + + +class TestCommitThenCarryResidenceOn: + """async + sim_inflight_residence=True: hold-before-clear + carry surplus.""" + + def test_surplus_carried_and_busy_held(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B", "C", "D", "E"]) + # Committed this cycle: A, B (already popped from the buffer). Surplus + # arrived: C, D (still buffered). Still in flight (not arrived): E. + agg._sim_committed = {"A", "B"} + agg._sim_buffer.add("C", 30.0, None) + agg._sim_buffer.add("D", 40.0, None) + agg._sim_inflight_expected = {"E": 50.0} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + # Surplus NOT dropped -- carried to the next cycle. + assert set(agg._sim_buffer.pending_ends()) == {"C", "D"} + # Not-yet-arrived trainer still in flight. + assert agg._sim_inflight_expected == {"E": 50.0} + # Per-cycle committed marks cleared so a re-contributor isn't skipped. + assert agg._sim_committed == set() + # Busy trainers (surplus ∪ in-flight) held in their slots; the two + # committed ones (A, B) released for re-selection. + assert set(ch._selector.all_selected) == {"C", "D", "E"} + assert ch._selector.selected_ends["agg"] == {"C", "D", "E"} + + def test_end_to_end_surplus_commits_next_cycle_no_refetch(self): + agg = _residence_agg(residence=True) + ends = ["A", "B", "C", "D", "E"] + ch = _FakeSelChannel([]) + for e, sct in zip(ends, (10.0, 20.0, 30.0, 40.0, 50.0)): + ch.add_msg(e, sct) + agg._sim_inflight_expected = {e: sct + 0.1 for e, sct + in zip(ends, (10, 20, 30, 40, 50))} + + # agg_goal = 2 commits this cycle (A, B by sct order); C, D, E arrive + # into the buffer as surplus. + scts = agg._drain(ch, ends, 2) + assert scts == [10.0, 20.0] + assert set(agg._sim_buffer.pending_ends()) == {"C", "D", "E"} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + # Carried, not dropped. + assert set(agg._sim_buffer.pending_ends()) == {"C", "D", "E"} + + # Next cycle: the carried surplus commits from the buffer WITHOUT a new + # dispatch/forward pass (no re-fetch), in sct order. + scts2 = agg._drain(ch, ends, 3) + assert scts2 == [30.0, 40.0, 50.0] + + def test_no_same_trainer_reselected_while_in_flight(self): + """R1 residence: a held (still-busy) trainer must not reappear as a fresh + selection slot -- that is exactly the re-dispatch-while-in-flight bug.""" + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B", "C"]) + agg._sim_committed = {"A"} + agg._sim_buffer.add("B", 20.0, None) # surplus, arrived + agg._sim_inflight_expected = {"C": 30.0} # still computing + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + held = set(ch._selector.all_selected) + # The busy trainers stay held (not freed for a duplicate dispatch); + # only the committed A is released. + assert "B" in held and "C" in held and "A" not in held + assert "B" in agg._sim_pending_commit and "C" in agg._sim_pending_commit + + +class TestFlagOffByteIdentical: + """Residence OFF (default) => the Batch-1 legacy drop, unchanged. Sync + baselines never take the carry path, so their boundary is untouched.""" + + def test_async_residence_off_drops_and_releases_all(self): + agg = _residence_agg(residence=False) + ch = _FakeSelChannel(["X", "Y"]) + agg._sim_committed = {"X", "Y"} + agg._sim_buffer.add("Z", 5.0, None) + agg._sim_inflight_expected = {"W": 9.0} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + # Legacy drop: buffer + in-flight cleared, every slot released. + assert len(agg._sim_buffer) == 0 + assert agg._sim_inflight_expected == {} + assert agg._sim_committed == set() + assert ch._selector.all_selected == {} + + def test_sync_barrier_always_drops_even_with_residence_flag(self): + # The sync path (fwdllm/fwdllm_plus, c ~= agg_goal) has no surplus, so it + # keeps the drop regardless of the residence flag. + agg = _residence_agg(residence=True) + agg._sim_committed = {"X"} + agg._sim_buffer.add("X", 1.0, None) + agg._sim_inflight_expected = {"X": 1.0} + + agg._release_sim_slots_at_agg_goal(channel=None, is_async=False) + + assert agg._sim_committed == set() + assert len(agg._sim_buffer) == 0 + assert agg._sim_inflight_expected == {} + + def test_not_simulated_is_noop(self): + agg = _residence_agg(residence=True) + agg.simulated = False + agg._sim_committed = {"X"} + agg._sim_buffer.add("X", 1.0, None) + agg._sim_inflight_expected = {"X": 1.0} + + agg._release_sim_slots_at_agg_goal(channel=None, is_async=True) + + assert agg._sim_committed == {"X"} + assert len(agg._sim_buffer) == 1 + assert agg._sim_inflight_expected == {"X": 1.0} diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 7331c49dd..e98df4129 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -706,3 +706,102 @@ def test_cadence_fields_land_in_event(self): assert ev == EVENT_AGG_ROUND assert f["cycle_data_id"] == 7 and f["cycle_iteration"] == 2 assert f["grad_pool_size"] == 9 and f["cached_v_size"] == 2 + + +# ── FwdLLM async residence rungs R1 / W1 (simulate_fwdllm.md §L.3) ── + +def _cyc(cycle_data_id, intervals, contributing=None): + """A committed cadence cycle carrying per-contributor [dispatch, commit] + intervals. `intervals` = list of (end, dispatch_ts, commit_ts).""" + ci = [{"end": e, "dispatch_ts": d, "commit_ts": c} for (e, d, c) in intervals] + return {"event": "agg_round", "round": 1, "ts": 0.0, + "cycle_data_id": cycle_data_id, "var_good_enough": True, + "agg_goal_count": len(ci), + "contributing_trainers": contributing or [e for (e, _, _) in intervals], + "contributor_intervals": ci} + + +def _trainers_with_rounds(counts): + """{short_id: {"trainer_round": [ ...n events ]}} for W1 forward-pass counts.""" + return {sid: {"trainer_round": [{"event": "trainer_round"} for _ in range(n)]} + for sid, n in counts.items()} + + +class TestR1InflightOverlap: + """R1 [INV]: per-trainer dispatch->commit intervals must not overlap + (one-in-flight residence). The fluxtune 2x-recompute bug violated this.""" + + def test_non_overlapping_intervals_pass(self): + # Each trainer's two contributions are strictly sequential (commit before + # the next dispatch) in BOTH modes. + agg = _agg(agg_rounds=[ + _cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)]), + _cyc(1, [("A", 6.0, 11.0), ("B", 6.0, 11.0)]), + ]) + r = pc.inflight_overlap_parity(agg, agg) + assert r["ok"] + assert r["real_overlap_frac"] == 0.0 and r["sim_overlap_frac"] == 0.0 + + def test_sim_overlap_fails_with_clean_real(self): + # Real: A's 2nd dispatch (6.0) is after its 1st commit (5.0) -> clean. + real = _agg(agg_rounds=[ + _cyc(0, [("A", 0.0, 5.0)]), + _cyc(1, [("A", 6.0, 11.0)]), + ]) + # Sim: A re-dispatched at 2.0 while its 1st contribution (commit 5.0) was + # still in flight -> overlap = the residence violation. + sim = _agg(agg_rounds=[ + _cyc(0, [("A", 0.0, 5.0)]), + _cyc(1, [("A", 2.0, 7.0)]), + ]) + r = pc.inflight_overlap_parity(real, sim) + assert not r["ok"] + assert r["real_overlap_frac"] == 0.0 + assert r["sim_overlap_frac"] > 0.0 + + def test_skips_without_contributor_intervals(self): + # Sync baselines / non-fwdllm runs don't emit contributor_intervals. + agg = _agg(agg_rounds=[_round(1, ["a"], [0])]) + r = pc.inflight_overlap_parity(agg, agg) + assert r.get("status") == "SKIP" and r["ok"] + + +class TestW1ComputeConservation: + """W1 [DIAG]: forward passes per committed grad; a large sim excess over + real = wasted recompute (the residence violation), localizes to R1.""" + + def test_matched_ratio_passes(self): + real_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + sim_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + # 2 committed each; ~1.5 forward passes per commit in both modes. + real_tr = _trainers_with_rounds({"A": 2, "B": 1}) + sim_tr = _trainers_with_rounds({"A": 2, "B": 1}) + r = pc.compute_conservation_parity(real_agg, sim_agg, real_tr, sim_tr) + assert r["ok"] + assert r["real_fwd_per_commit"] == r["sim_fwd_per_commit"] + + def test_sim_recompute_excess_fails(self): + real_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + sim_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + # Same 2 commits both modes, but sim ran ~2x the forward passes. + real_tr = _trainers_with_rounds({"A": 1, "B": 1}) # 2 fwd / 2 commit = 1.0 + sim_tr = _trainers_with_rounds({"A": 3, "B": 3}) # 6 fwd / 2 commit = 3.0 + r = pc.compute_conservation_parity(real_agg, sim_agg, real_tr, sim_tr) + assert not r["ok"] + assert r["sim_fwd_per_commit"] > r["real_fwd_per_commit"] + + def test_skips_without_data(self): + empty = _agg(agg_rounds=[]) + r = pc.compute_conservation_parity(empty, empty, {}, {}) + assert r.get("status") == "SKIP" and r["ok"] + + +class TestR1W1Registered: + """Both rungs run in run_all_parity and are wired into the causal registry + (R1 upstream of V1 -- the dep chain proving cadence is downstream).""" + + def test_present_in_run_all_and_meta(self): + assert "r1_inflight_overlap" in pc.CHECK_META + assert "w1_compute_conservation" in pc.CHECK_META + assert "r1_inflight_overlap" in pc.CHECK_META["v1_iter_per_data_id"]["deps"] + assert pc.CHECK_META["w1_compute_conservation"]["deps"] == ("r1_inflight_overlap",) From 51541c8d0dbd32b1a599437ff124b4cbe5d6a41f Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:00:31 -0400 Subject: [PATCH 08/56] expt_runner: aligned per-baseline table + diff highlight + solid section bars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-flight dry-run display was jagged and hard to scan. Improve the shared renderer (expt_runner.py) + fwdllm tier-② spec: - Section headers (①②③) now render as solid full-width background bars (blue/magenta/grey) instead of plain bold, so they're clearly visible. - Tier ② per-baseline knobs become an ALIGNED TABLE (baselines = rows, knobs = columns) instead of one crammed horizontal string per baseline. - Cross-baseline diff highlighting: any column whose value is NOT identical across all baselines is bold-yellow (the knobs to eyeball); columns identical across all stay dim (expected). Legend lists the differing columns. - Baseline names are cyan; flag-overridden columns get a green '•'. - Tier ① values align to one column (widest label), and the "ok" row marker is 2 cells wide to match the emoji icons so labels don't drift. Verified via the fwdllm --dry-run: agg_goal (10/10/3) and k (5/5/–) highlight, identical knobs dim. Backward-compatible: rows-based tiers render unchanged. Co-Authored-By: Claude Opus 4.8 --- .../fwdllm/expt_scripts/run_sequential.sh | 38 ++++--- lib/python/examples/scripts/expt_runner.py | 106 ++++++++++++++++-- 2 files changed, 122 insertions(+), 22 deletions(-) diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 8974dc0fa..5b36bec9a 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -381,23 +381,33 @@ tier1 = {"name": "① REVIEW EVERY RUN", "rows": [ ]} tiers.append(tier1) -# ② per-baseline +# ② per-baseline -- an ALIGNED TABLE (baselines = rows, knobs = columns). The +# renderer highlights any column whose value differs across the baselines (those +# are the ones to eyeball); columns identical across all 3 stay dim (expected). +tier2_cols = [ + ("c", "c"), ("agg_goal", "agg_goal"), ("k", "k"), + ("min_init", "minInit"), ("n_trainers", "n_trainers"), + ("n_gpus", "n_gpus"), ("partition", "part"), +] +overridden2 = [] +if bool(SEL_C) or bool(SEL_C_ASYNC): overridden2.append("c") +if bool(AGG_GOAL) or bool(SEL_C): overridden2.append("agg_goal") +if bool(SEL_K): overridden2.append("k") +if bool(MIN_INIT): overridden2.append("min_init") +if bool(NUM_TRAINERS): overridden2.append("n_trainers") +if bool(NUM_GPUS): overridden2.append("n_gpus") +if bool(PART): overridden2.append("partition") rows2 = [] for rk in (r[0] for r in runs): b = per_baseline.get(rk, {}) - # each knob shows "(D)" unless its dedicated flag overrode it this run - val = (f"c={dflt(b.get('c'), bool(SEL_C) or (rk=='fluxtune' and bool(SEL_C_ASYNC)))} " - f"agg_goal={dflt(b.get('agg_goal'), bool(AGG_GOAL) or bool(SEL_C))} " - f"k={dflt(b.get('k'), bool(SEL_K))} " - f"minInit={dflt(b.get('min_init'), bool(MIN_INIT))} " - f"n_trainers={dflt(b.get('n_trainers'), bool(NUM_TRAINERS))} " - f"n_gpus={dflt(b.get('n_gpus'), bool(NUM_GPUS))} " - f"part={dflt(b.get('partition'), bool(PART))}") - lvl = {} - if PART and PART != "uniform": - lvl = {"level": "warn", "note": "non-default partition"} - rows2.append({"label": rk, "value": val, **lvl}) -tiers.append({"name": "② PER-BASELINE (moderate)", "rows": rows2}) + rows2.append({"name": rk, "cells": { + "c": b.get("c"), "agg_goal": b.get("agg_goal"), "k": b.get("k"), + "min_init": b.get("min_init"), "n_trainers": b.get("n_trainers"), + "n_gpus": b.get("n_gpus"), "partition": b.get("partition"), + }}) +tiers.append({"name": "② PER-BASELINE (moderate)", + "table": {"columns": tier2_cols, "rows": rows2, + "overridden": overridden2}}) # ③ config-baked tiers.append({"name": "③ RARELY CHANGED", "collapsed": True, "rows": [ diff --git a/lib/python/examples/scripts/expt_runner.py b/lib/python/examples/scripts/expt_runner.py index 9c14a1ab5..776eebfb8 100644 --- a/lib/python/examples/scripts/expt_runner.py +++ b/lib/python/examples/scripts/expt_runner.py @@ -21,11 +21,18 @@ "tiers": [ {"name": "① REVIEW EVERY RUN", "collapsed": False, # tier ③ sets True -> hidden unless EXPT_SHOW_ALL=1 - "rows": [ + "bar": "44", # optional header-bar bg colour code (default cycles) + "rows": [ # label/value rows … {"label": "stop", "value": "max_runtime_s=600 max_data_id=10"}, {"label": "delays", "value": "D=0", "level": "warn", "note": "both sides matched"}, ]}, + {"name": "② PER-BASELINE", # … OR an aligned table (mutually exclusive with rows) + "table": { + "columns": ["c", "agg_goal", ("min_init", "minInit")], # key or (key, header) + "rows": [{"name": "fluxtune", "cells": {"c": 10, "agg_goal": 3}}], + "overridden": ["c"], # optional: columns set by a CLI flag (green •) + }}, ], "checks": [ {"name": "D matched across real/sim pair", "level": "ok", @@ -67,15 +74,24 @@ def _w(self, code: str, s: str) -> str: def red(self, s): return self._w("1;31", s) def yellow(self, s): return self._w("1;33", s) def green(self, s): return self._w("32", s) + def cyan(self, s): return self._w("1;36", s) def dim(self, s): return self._w("2", s) def bold(self, s): return self._w("1", s) + def bar(self, s: str, width: int, code: str = "44") -> str: + """A solid-background full-width header bar (bold bright-white on `code`), + so the ①②③ section markers read as distinct blocks, not plain text.""" + txt = (" " + s).ljust(width) + return self._w(f"1;97;{code}", txt) + # level -> (icon, colouriser name). "ok" rows stay quiet; set/warn/error shout. # set = value explicitly overridden by a command-line flag (overrides yaml) -> green # warn = review / attention -> yellow # error = infeasible -> red -_ICON = {"ok": " ", "set": "\U0001f7e2", "warn": "\U0001f7e1", "error": "\U0001f534"} # 🟢 🟡 🔴 +# NB: the emoji markers render 2 cells wide, so "ok" uses TWO spaces to keep the +# label column aligned with the set/warn/error rows. +_ICON = {"ok": " ", "set": "\U0001f7e2", "warn": "\U0001f7e1", "error": "\U0001f534"} # 🟢 🟡 🔴 _CHECK_ICON = {"ok": "✓", "warn": "⚠", "error": "✗"} # ✓ ⚠ ✗ @@ -86,6 +102,69 @@ def _colour_for(st: _Style, level: str): # ---- renderer -------------------------------------------------------------- _RULE = "─" * 79 # ───── +_BAR_W = 79 # header-bar width (matches the rule) + +# Per-tier header-bar background colours so ①②③ are visually distinct blocks. +_TIER_BAR = ["44", "45", "100"] # blue, magenta, bright-black(grey) + + +def _cell(v) -> str: + """Table cell text: None -> '–', everything else str().""" + return "–" if v is None else str(v) + + +def _render_table(out, st, table: dict) -> None: + """Render an aligned per-entity table (tier ②'s per-baseline knobs). + + table = { + "columns": [key | (key, header), ...], # column order + "rows": [{"name": str, "cells": {key: value}}, ...], + "overridden": [key, ...], # optional: flag-overridden cols + } + + Columns whose value is NOT identical across every row are HIGHLIGHTED (bold + yellow header + cells) -- those are the knobs that differ between baselines + and must be eyeballed. Columns identical across all rows stay dim (expected). + Row (baseline) names are cyan. A flag-overridden column gets a green '•'. + """ + cols = [(c, c) if isinstance(c, str) else (c[0], c[1]) for c in table["columns"]] + rows = table["rows"] + overridden = set(table.get("overridden", [])) + + # Which columns differ across baselines? + differs = {} + for key, _h in cols: + vals = {_cell(r["cells"].get(key)) for r in rows} + differs[key] = len(vals) > 1 + + name_w = max([len("baseline")] + [len(_cell(r["name"])) for r in rows]) + col_w = {} + for key, hdr in cols: + col_w[key] = max(len(hdr), max((len(_cell(r["cells"].get(key))) for r in rows), + default=0)) + + # header row + hcells = [st.dim("baseline".ljust(name_w))] + for key, hdr in cols: + mark = st.green("•") if key in overridden else " " + htxt = hdr.ljust(col_w[key]) + hcells.append((st.yellow(htxt) if differs[key] else st.dim(htxt)) + mark) + out(" " + " ".join(hcells)) + + # data rows + for r in rows: + line = [st.cyan(_cell(r["name"]).ljust(name_w))] + for key, hdr in cols: + v = _cell(r["cells"].get(key)).ljust(col_w[key]) + line.append((st.yellow(v) if differs[key] else st.dim(v)) + " ") + out(" " + " ".join(line)) + # legend for the highlighting + diff_names = [h for k, h in cols if differs[k]] + legend = st.yellow("yellow") + st.dim(" = differs across baselines (review)") + if overridden: + legend += st.dim(" ") + st.green("•") + st.dim(" = flag override") + out(" " + st.dim("· ") + legend + + (st.dim(f" [{', '.join(diff_names)}]") if diff_names else "")) def render_and_gate(spec: dict, show_all: bool | None = None, stream=None) -> int: @@ -110,24 +189,35 @@ def out(s=""): out(" " + _RULE) n_err_rows = 0 - for tier in spec.get("tiers", []): + for ti, tier in enumerate(spec.get("tiers", [])): name = tier.get("name", "") rows = tier.get("rows", []) + table = tier.get("table") collapsed = tier.get("collapsed", False) - out(" " + st.bold(name)) + bar_code = tier.get("bar", _TIER_BAR[ti % len(_TIER_BAR)]) + out(" " + st.bar(name, _BAR_W, bar_code)) + n_hidden = len(table["rows"]) if table else len(rows) if collapsed and not show_all: - out(st.dim(f" [{len(rows)} row(s) hidden — set EXPT_SHOW_ALL=1]")) + out(st.dim(f" [{n_hidden} row(s) hidden — set EXPT_SHOW_ALL=1]")) + continue + if table: + _render_table(out, st, table) continue + # Align every value in this tier to one column: pad labels to the widest + # label present (so a long `max_data_id_progress` doesn't stagger the + # shorter rows' values). Cap so a pathological label doesn't push values + # off-screen. + label_w = min(max((len(r.get("label", "")) for r in rows), default=13), 24) for r in rows: level = r.get("level", "ok") if level == "error": n_err_rows += 1 icon = _ICON.get(level, " ") col = _colour_for(st, level) - # Pad short labels to a column; always keep >=1 space before the value - # so a label at/over the column width doesn't run into it. + # Pad every label to the same width + one trailing space, so all + # values in the tier start at the same column (no stagger). raw = r.get("label", "") - label = raw.ljust(13) if len(raw) < 13 else raw + " " + label = raw.ljust(label_w) + " " value = r.get("value", "") note = r.get("note", "") note_s = f" {st.dim('· ' + note)}" if note else "" From be6e94d72c2fa21802a9a8cc78e8ef80b4167a4d Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:05:58 -0400 Subject: [PATCH 09/56] =?UTF-8?q?expt=5Frunner:=20tier-=E2=91=A0=20notes?= =?UTF-8?q?=20get=20their=20own=20wrapping=20column,=20not=20a=20trailing?= =?UTF-8?q?=20tail?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The review-every-run rows trailed each note unbounded on the same line, which read as noisy. Render label | value | note as three aligned columns and WRAP the note to indented continuation lines (bounded to the bar width). Value column is sized to the note-bearing rows so all notes start at the same column; a note-less long value (the baselines list) overflows harmlessly. Also shortened the two "unset (…)" var-knob values (the parenthetical moved into the note). Co-Authored-By: Claude Opus 4.8 --- .../fwdllm/expt_scripts/run_sequential.sh | 8 ++-- lib/python/examples/scripts/expt_runner.py | 40 ++++++++++++------- 2 files changed, 30 insertions(+), 18 deletions(-) diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 5b36bec9a..9a7c9d77e 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -372,12 +372,12 @@ tier1 = {"name": "① REVIEW EVERY RUN", "rows": [ # var_threshold / max_iterations_per_data_id vary with data heterogeneity -> # review-every-run (warn when defaulted). NOTE: max_iters_per_data_id is the # FORCE-COMMIT cap and is NOT the same as max_data_id_progress (the stop) above. - scalar_row("var_threshold", VAR_THRESHOLD if VAR_THRESHOLD else "unset (trainer/code default)", + scalar_row("var_threshold", VAR_THRESHOLD if VAR_THRESHOLD else "unset", bool(VAR_THRESHOLD), review=True, - note="variance-pass gate; varies w/ data heterogeneity (--var-threshold)"), - scalar_row("max_iters_per_data_id", MAX_ITER if MAX_ITER else "unset (code default)", + note="variance-pass gate; varies w/ data heterogeneity (--var-threshold). unset ⇒ trainer/code default"), + scalar_row("max_iters_per_data_id", MAX_ITER if MAX_ITER else "unset", bool(MAX_ITER), review=True, - note="FORCE-COMMIT cap (--max-iter-per-data-id) — NOT the max_data_id_progress stop above"), + note="FORCE-COMMIT cap (--max-iter-per-data-id) — NOT the max_data_id_progress stop above. unset ⇒ code default"), ]} tiers.append(tier1) diff --git a/lib/python/examples/scripts/expt_runner.py b/lib/python/examples/scripts/expt_runner.py index 776eebfb8..f7dbbbdb5 100644 --- a/lib/python/examples/scripts/expt_runner.py +++ b/lib/python/examples/scripts/expt_runner.py @@ -51,6 +51,7 @@ from __future__ import annotations import json +import textwrap import os import sys @@ -203,29 +204,40 @@ def out(s=""): if table: _render_table(out, st, table) continue - # Align every value in this tier to one column: pad labels to the widest - # label present (so a long `max_data_id_progress` doesn't stagger the - # shorter rows' values). Cap so a pathological label doesn't push values - # off-screen. + # Three aligned columns: label | value | note. Labels pad to the widest + # label; the note lives in its OWN column and WRAPS to indented + # continuation lines (bounded), instead of trailing unbounded on one + # line. Value width is sized to the note-bearing rows so their notes all + # start at the same column (a long note-less value, e.g. the baselines + # list, may overflow harmlessly since nothing follows it). label_w = min(max((len(r.get("label", "")) for r in rows), default=13), 24) + _noted = [r for r in rows if r.get("note")] + value_w = min(max((len(str(r.get("value", ""))) for r in _noted), default=0), 20) + # cell column at which the note starts: 2 lead + 2 icon + 1 + label + 1 + value + 1 + note_col = 2 + 2 + 1 + label_w + 1 + value_w + 1 + note_w = max(24, _BAR_W - note_col) + cont_pad = " " * (note_col + 2) # continuation lines align under the note text for r in rows: level = r.get("level", "ok") if level == "error": n_err_rows += 1 icon = _ICON.get(level, " ") col = _colour_for(st, level) - # Pad every label to the same width + one trailing space, so all - # values in the tier start at the same column (no stagger). raw = r.get("label", "") - label = raw.ljust(label_w) + " " - value = r.get("value", "") + label = raw.ljust(label_w) + value = str(r.get("value", "")) + valpad = value.ljust(value_w) + # ok rows: label dim, value plain. warn/error: label bold, value coloured. + lbl_s = st.dim(label) if level == "ok" else st.bold(label) + val_s = valpad if level == "ok" else col(valpad) note = r.get("note", "") - note_s = f" {st.dim('· ' + note)}" if note else "" - # ok rows: label dim, value plain. warn/error: value coloured. - if level == "ok": - out(f" {icon} {st.dim(label)} {value}{note_s}") - else: - out(f" {icon} {st.bold(label)} {col(value)}{note_s}") + if not note: + out(f" {icon} {lbl_s} {val_s}".rstrip()) + continue + wrapped = textwrap.wrap(note, note_w) or [""] + out(f" {icon} {lbl_s} {val_s} {st.dim('· ' + wrapped[0])}") + for cont in wrapped[1:]: + out(cont_pad + st.dim(cont)) out(" " + _RULE) From 54644b05c61408785e6fbefcdbad137af5650849 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:19:22 -0400 Subject: [PATCH 10/56] =?UTF-8?q?fwdllm=20sim:=20=C2=A7L=20re-run=20parity?= =?UTF-8?q?=20results=20in=20=C2=A7H=20+=20fix=20fluxtune=20staleness=20ke?= =?UTF-8?q?y=20collision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran scripts.parity.cli on all 3 real/sim syn_0 pairs. Result: the §L residence fix (D-a/D-b) is validated — fluxtune sim forward passes 228→82 (2x recompute gone), R1 in-flight overlap sim=0.0%, V1 iters-per-data_id 80-vs-8 → 7.67-vs-8.0 (near-converged). fwdllm_plus no longer crashes. Full results + interpretation written to §H. Also found + fixed a config key collision that left D-c dead in the run: the fluxtune yamls set snake_case `staleness_policy: fedbuff`, but the base config carries camelCase `stalenessPolicy: none`; the dict-merge kept both and the camelCase base key won at pydantic resolution (aggregator logged `staleness_policy = none`). Switch the overrides to camelCase `stalenessPolicy` so they override the base. (Residence escaped this — no base residence key.) Remaining before a clean syn_0 sign-off (documented in §H): re-run to confirm the staleness fix; add the missing `vclock_now` agg_round emit (a pre-existing gap that blocks the clock/throughput/convergence family for ALL baselines); and look at the smaller residual W1 overlap gap (sim now slightly UNDER-computes). Co-Authored-By: Claude Opus 4.8 --- .../expt_scripts/fluxtune_n10_smoke.yaml | 7 ++- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 7 ++- lib/python/examples/fwdllm/simulate_fwdllm.md | 59 +++++++++++++++++++ 3 files changed, 69 insertions(+), 4 deletions(-) diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index 7dfb888b3..4d02bd4da 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -65,8 +65,11 @@ experiments: # Async fedbuff baseline: accept a stale grad (trained on model_version # V that lands at V'>V) and down-weight by (V'-V) rather than reject it # (K-D13/§L D5). Baseline DEFINITION -- set identically real+sim, never - # tuned to close a gap. Feeds the U3 staleness rung. - staleness_policy: fedbuff + # tuned to close a gap. Feeds the U3 staleness rung. MUST be camelCase + # `stalenessPolicy` to OVERRIDE the base config's key (the snake_case + # form is a distinct dead key that loses at pydantic resolution — the + # fluxtune run ran on `none`, §H re-run finding #2 / K-D15). + stalenessPolicy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 62e7f1b65..71abd6fa8 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -77,8 +77,11 @@ experiments: # Async fedbuff baseline: carried grads (residence above) are stale by # construction, so accept + down-weight by (V'-V) rather than # reject/ignore (K-D13/§L D5). Baseline DEFINITION, set identically in - # the real sibling -- never a parity lever. - staleness_policy: fedbuff + # the real sibling -- never a parity lever. MUST be camelCase + # `stalenessPolicy` to OVERRIDE the base config's key; the snake_case + # form is a distinct dead key that loses at pydantic resolution and + # left the fluxtune run on `none` (§H re-run finding #2 / K-D15). + stalenessPolicy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index fa68021df..ba731499e 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -428,6 +428,65 @@ sub-package 115. **Next: the 3-baseline syn_0 re-run** — expect fluxtune sim f ratio→1), R1==0% both modes, fwdllm_plus no longer crashing. Fluxtune / fwdllm_plus first-time results above. +**§L re-run RESULTS (2026-07-03, `--mode both --delays off --max-runtime-s 300 --max-data-id 3`; parity +`scripts.parity.cli` per pair, agg_goal fwdllm/plus=10, fluxtune=3).** All 6 runs completed (fwdllm_plus no +longer crashes — D-d/param-rename confirmed). The runs carry the Batch-2.5 telemetry (`contributor_intervals` +present) and `simInflightResidence=True` is active on the sim side. + +**HEADLINE — the residence fix (D-a/D-b) is VALIDATED:** +| metric (fluxtune async) | pre-fix (first smoke) | post-fix (this run) | +|---|---|---| +| sim forward passes | **228** (vs real 109) | **82** (vs real 120) | +| sim wall vs real | 356s (**1.7×**) | comparable | +| R1 in-flight overlap (sim) | — (violated) | **0.0%** (real 1.7%, within tol) → **PASS** | +| V1 iters-per-data_id | **80 vs 8** (data_id 2) | mean **real 7.67 / sim 8.0** (near-converged) | + +The 2× recompute is gone: sim went from over-computing (228) to slightly UNDER-computing (82 < real 120). +R1 residence is exact (sim 0.0% overlap). V1 cadence collapsed from an 80-vs-8 blowup to ~8-vs-8. + +**Per-baseline enforced score:** fwdllm **33/35**, fwdllm_plus **27/34**, fluxtune **28/36** (many of the +misses are ONE shared pre-existing telemetry gap, below — not regressions): + +| rung | fwdllm | fwdllm_plus | fluxtune | note | +|---|---|---|---|---| +| **R1** in-flight overlap | PASS (0/0%) | PASS (0/0%) | **PASS** (real 1.7% / sim 0.0%) | residence exact | +| **W1** compute-conservation | PASS (1.13/1.17) | PASS (1.14/1.17) | **WARN** (real 1.74 / sim 1.14, +35%) | see #1 below | +| **V1** iters-per-data_id | PASS (2.0/2.0) | FAIL (1.67/2.0) | FAIL (7.67/8.0, KS .33) | means close; KS just over tol | +| V2 var-trajectory | PASS | PASS | PASS | grad values mode-invariant ✓ | +| **U3** staleness | PASS | PASS | **FAIL** (KS .27) | D-c NOT active — see #2 | +| participation / S2 | PASS | PASS | PASS | | + +**Three residual issues (ranked; the syn_0 pass is NOT yet clean):** +1. **W1 flipped direction for fluxtune (WARN, not the old 2× blowup).** Sim now does 1.14 forward passes + per commit vs real 1.74 (+35% gap) — the fix removed the wasteful recompute but sim now UNDER-models + real's legitimate async in-flight overlap tail (real leaves more arrived-but-uncommitted grads in flight + at the matched window; sim's commit-then-carry commits more of them, 72 vs real 69). This is the next + thing to localize — a *residual* overlap-modeling gap, opposite sign to the original bug, and far + smaller. It reads as a DIAG WARN, not a hard fail. +2. **D-c staleness did NOT take effect — config key collision (found + FIXED this session).** fluxtune + logged `staleness_policy = none` (my K-D15 log line), so U3 failed. Root (verified from the generated + config, which carried BOTH keys `{stalenessPolicy: none, staleness_policy: fedbuff}`): the yaml override + used snake_case `staleness_policy: fedbuff` but the fluxtune BASE config already carries camelCase + `stalenessPolicy: none`; the raw dict-merge keeps both as distinct keys and the camelCase base key wins + at pydantic resolution. (Residence escaped this because the base has no `simInflightResidence` key, so + the snake override was unopposed.) **FIXED:** the fluxtune real+sim yamls now set `stalenessPolicy` + (camelCase) so it overwrites the base key. **Not yet re-run** — the banked run above still shows U3 FAIL; + confirm on the next launch that the aggregator logs `staleness_policy = fedbuff` and U3 recovers. +3. **Pre-existing `vclock_now` telemetry gap blocks the whole clock/throughput family (ALL 3 baselines).** + The fwdllm aggregator never stamps `vclock_now` on the agg_round event, so K10/`vclock_telemetry`, + K2/`throughput`, `total_commits`, `terminal_state`, `field_coverage` all FAIL/SKIP for every baseline — + which is why the raw scores look lower than the mechanics warrant. This is NOT a §L regression; it is a + separate telemetry port (mirror asyncfl's `vclock_now` emit) and is a **prerequisite for the convergence + sign-off** (§I.6 C1/C2 at matched data_id need a virtual budget V). Also: fwdllm_plus/fluxtune miss + several availability/selection rungs (`eligibility`, `selection_detail`, `avail_composition`, + `selector_score`, `preferred_duration`) — oracular/oort telemetry-coverage gaps, Batch-2 validation + territory, not residence. + +**Net:** the residence remediation did exactly what §L predicted (recompute + cadence fixed, R1 exact); +what remains before a clean syn_0 sign-off is (2) the one-line staleness-config fix, then (3) the +`vclock_now` emit to unblock the clock/throughput/convergence family, then re-score — plus a look at the +smaller residual W1 overlap gap (#1). JSON: `parity_{fwdllm,fwdllm_plus,fluxtune}.json`. + --- ## §I Batch 2 implementation map (code-level -- cold-start; NEXT to build) From 9570e89c018490991c88e1e8b3ae5dc6d9f3080d Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:37:24 -0400 Subject: [PATCH 11/56] =?UTF-8?q?fwdllm=20parity:=20emit=20vclock=5Fnow,?= =?UTF-8?q?=20make=20W1=20asymmetric;=20=C2=A7H=20deep=20root-cause?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Investigated the §H re-run rungs against telemetry rather than assuming: - V1 FAILs are short-run truncation — iters-per-data_id match on every full data_id (fluxtune data_id 0 exact 11=11) and diverge only on the truncated last one; 3 samples make one boundary difference KS 0.33. No code change; needs a longer run. - W1 gap is the async start-tail, not a residence bug: received==commits and R1==0 (aggregator drops nothing); trainer_round counts starts and a live async real system leaves a bigger in-flight start-tail than the clock-gated sim. W1 is now asymmetric — flags only sim OVER-computing (the recompute bug), treats sim under-computing as benign. fluxtune W1 now passes. - vclock_now was never stamped on fwdllm agg_round, gating off the whole clock/throughput/convergence family (K10). Emit it (mirror asyncfl). Tests updated for the asymmetric W1; §H rewritten with the data-backed causes. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 29 ++++++--- lib/python/examples/fwdllm/simulate_fwdllm.md | 63 ++++++++++--------- .../horizontal/syncfl/fwdllm_aggregator.py | 7 +++ lib/python/tests/mode/test_parity_checks.py | 15 ++++- 4 files changed, 72 insertions(+), 42 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 1d4d59dfa..8ff9ce24d 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -3858,12 +3858,17 @@ def _committed_grads(agg: dict) -> int: def compute_conservation_parity(real: dict, sim: dict, real_trainers: dict, sim_trainers: dict, ratio_tol: float = 0.25) -> dict: - """W1 [DIAG]: compute-conservation — forward passes vs committed grads. - - forward_passes ~= committed + in_flight_at_stop + stale_rejected, so the - forward/commit ratio is ~1 plus a small tail. The tell is the real<->sim - RATIO of that ratio: sim doing far more forward passes per commit than real - = wasted recompute (the residence violation, §L.1). Localizes to R1. + """W1 [DIAG]: compute-conservation — sim must not WASTE forward passes. + + `trainer_round` counts forward-pass STARTS, so forward/commit ~= 1 plus an + in-flight-at-stop + stale-reject tail. W1 exists to catch the residence bug + where sim RE-DISPATCHES dropped grads and thus does far MORE forward passes + per commit than real (the 2x-recompute, §L.1). So the check is ASYMMETRIC: + only a sim EXCESS over real is a violation. Sim doing FEWER passes than real + is not wasted recompute — a live async real system accrues a larger in-flight + START tail over wall-time than the clock-gated sim, so real > sim is expected + (and amortizes with run length / D>0). Localizes to R1; R1==0 + received== + committed is the precise residence signal, W1 is the coarse compute tell. """ r_fwd, s_fwd = _forward_passes(real_trainers), _forward_passes(sim_trainers) r_com, s_com = _committed_grads(real), _committed_grads(sim) @@ -3873,9 +3878,13 @@ def compute_conservation_parity(real: dict, sim: dict, "or telemetry absent)"} r_ratio = r_fwd / r_com s_ratio = s_fwd / s_com - rel = abs(s_ratio - r_ratio) / max(r_ratio, s_ratio) + # signed excess of sim over real (positive = sim wastes more compute) + excess = (s_ratio - r_ratio) / max(r_ratio, s_ratio) + direction = ("sim OVER-computes (recompute waste — check R1)" if excess > ratio_tol + else "sim under-computes (async start-tail; benign for compute waste)" + if excess < -ratio_tol else "matched") return { - "ok": rel <= ratio_tol, + "ok": excess <= ratio_tol, "tier": "DIAG", "real_forward_passes": r_fwd, "sim_forward_passes": s_fwd, @@ -3883,11 +3892,11 @@ def compute_conservation_parity(real: dict, sim: dict, "sim_committed": s_com, "real_fwd_per_commit": round(r_ratio, 3), "sim_fwd_per_commit": round(s_ratio, 3), - "ratio_rel_diff": round(rel, 3), + "sim_excess_rel": round(excess, 3), "ratio_tol": ratio_tol, "interpretation": ( f"real {r_ratio:.2f} vs sim {s_ratio:.2f} forward passes per commit; " - f"a large sim excess = recompute wasted on dropped/re-dispatched grads." + f"{direction}." ), } diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ba731499e..4ed57e4a2 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -456,36 +456,39 @@ misses are ONE shared pre-existing telemetry gap, below — not regressions): | **U3** staleness | PASS | PASS | **FAIL** (KS .27) | D-c NOT active — see #2 | | participation / S2 | PASS | PASS | PASS | | -**Three residual issues (ranked; the syn_0 pass is NOT yet clean):** -1. **W1 flipped direction for fluxtune (WARN, not the old 2× blowup).** Sim now does 1.14 forward passes - per commit vs real 1.74 (+35% gap) — the fix removed the wasteful recompute but sim now UNDER-models - real's legitimate async in-flight overlap tail (real leaves more arrived-but-uncommitted grads in flight - at the matched window; sim's commit-then-carry commits more of them, 72 vs real 69). This is the next - thing to localize — a *residual* overlap-modeling gap, opposite sign to the original bug, and far - smaller. It reads as a DIAG WARN, not a hard fail. -2. **D-c staleness did NOT take effect — config key collision (found + FIXED this session).** fluxtune - logged `staleness_policy = none` (my K-D15 log line), so U3 failed. Root (verified from the generated - config, which carried BOTH keys `{stalenessPolicy: none, staleness_policy: fedbuff}`): the yaml override - used snake_case `staleness_policy: fedbuff` but the fluxtune BASE config already carries camelCase - `stalenessPolicy: none`; the raw dict-merge keeps both as distinct keys and the camelCase base key wins - at pydantic resolution. (Residence escaped this because the base has no `simInflightResidence` key, so - the snake override was unopposed.) **FIXED:** the fluxtune real+sim yamls now set `stalenessPolicy` - (camelCase) so it overwrites the base key. **Not yet re-run** — the banked run above still shows U3 FAIL; - confirm on the next launch that the aggregator logs `staleness_policy = fedbuff` and U3 recovers. -3. **Pre-existing `vclock_now` telemetry gap blocks the whole clock/throughput family (ALL 3 baselines).** - The fwdllm aggregator never stamps `vclock_now` on the agg_round event, so K10/`vclock_telemetry`, - K2/`throughput`, `total_commits`, `terminal_state`, `field_coverage` all FAIL/SKIP for every baseline — - which is why the raw scores look lower than the mechanics warrant. This is NOT a §L regression; it is a - separate telemetry port (mirror asyncfl's `vclock_now` emit) and is a **prerequisite for the convergence - sign-off** (§I.6 C1/C2 at matched data_id need a virtual budget V). Also: fwdllm_plus/fluxtune miss - several availability/selection rungs (`eligibility`, `selection_detail`, `avail_composition`, - `selector_score`, `preferred_duration`) — oracular/oort telemetry-coverage gaps, Batch-2 validation - territory, not residence. - -**Net:** the residence remediation did exactly what §L predicted (recompute + cadence fixed, R1 exact); -what remains before a clean syn_0 sign-off is (2) the one-line staleness-config fix, then (3) the -`vclock_now` emit to unblock the clock/throughput/convergence family, then re-score — plus a look at the -smaller residual W1 overlap gap (#1). JSON: `parity_{fwdllm,fwdllm_plus,fluxtune}.json`. +**Deeper root-cause of the four non-clean rungs (all diagnosed to artifact or config, none to a residence +regression) + the fixes made this session:** + +1. **W1 (fluxtune WARN) — start-tail artifact, not a modeling bug. Rung refined.** `trainer_round` counts + forward-pass STARTS. Both modes drop NOTHING at the aggregator (received grads == commits: real 69==69, + sim 73≈72) and R1 sim-overlap==0, so residence is exact. The 120-vs-82 gap is that a live async real + system dispatches continuously over wall-time and leaves a large in-flight START tail (51), while the + clock-gated sim leaves a small one (10) — real > sim is EXPECTED and amortizes with run length / D>0. + W1 was symmetric and fired on this benign direction. **FIX: W1 is now asymmetric** — it flags only a sim + EXCESS over real (the 2× recompute it was built for); sim under-computing is reported as benign. fluxtune + W1 now PASSES (`sim_excess_rel −0.35`). +2. **V1 (fwdllm_plus + fluxtune FAIL) — short-run truncation, not a cadence bug. No code change.** The + per-data_id iteration series match on every FULL data_id and diverge only on the last, truncated one: + fluxtune real `{0:11,1:8,2:4}` vs sim `{0:11,1:7,2:6}` (data_id 0 EXACT), fwdllm_plus `{0:2,1:2,2:1}` vs + `{0:2,1:2,2:2}`. With 3 data_ids a single boundary difference → KS 0.33. **Fix is operational: a longer + run (more data_ids) for a scoreable distribution**, per the §I.6 convergence config (D>0 + real budget). +3. **U3 staleness (fluxtune FAIL) — config key collision. FIXED.** The generated config carried BOTH + `{stalenessPolicy: none (base), staleness_policy: fedbuff (override)}`; the camelCase base key wins at + pydantic resolution, so the run logged `staleness_policy = none`. (Residence escaped this — no base + `simInflightResidence` key.) **FIX: the fluxtune yamls now use camelCase `stalenessPolicy`.** Confirm on + re-run that the aggregator logs `staleness_policy = fedbuff` and U3 recovers. +4. **Clock/throughput family (ALL baselines FAIL/SKIP) — missing `vclock_now`. FIXED.** The fwdllm + aggregator never stamped `vclock_now` on agg_round, so K10 gated off the whole clock family + (K2/throughput/total_commits/terminal_state/field_coverage). **FIX: emit `vclock_now` (mirror asyncfl).** + This unblocks the virtual-budget V the convergence sign-off (C1/C2) needs. Separately, fwdllm_plus/ + fluxtune still miss oracular/oort availability+selection rungs — Batch-2 telemetry-coverage validation, + not residence. + +**Net:** residence + cadence are correct (R1==0, received==commits, V1 exact on full data_ids). Every +non-clean rung traced to an artifact (W1 start-tail, V1 truncation) or a config/telemetry gap (U3 key, +vclock_now) — all fixed in code except the V1 short-run, which needs the longer D>0 run. **Before re-launch: +the fixes above are in; re-run with the §I.6 convergence config (D>0, longer budget) and re-score.** JSON: +`parity_{fwdllm,fwdllm_plus,fluxtune}.json`. --- diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 202e06a61..7ed889b3a 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -1503,6 +1503,13 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): contributing_trainers=_cycle_contributors, agg_observed_s=_cycle_agg_observed_s, extra={ + # Virtual clock at commit (sim only) -- the parity engine + # gates its whole clock/throughput/convergence family on + # this (K10). asyncfl emits the same; fwdllm never did, so + # K2/throughput/total_commits/terminal_state/convergence + # were unscoreable for every fwdllm baseline. + "vclock_now": (self._vclock.now if self.simulated + and getattr(self, "_vclock", None) else None), "data_id": self.data_id, "iteration_per_data_id": self.iteration_per_data_id, "var": self.var, diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index e98df4129..0b4331b23 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -783,12 +783,23 @@ def test_matched_ratio_passes(self): def test_sim_recompute_excess_fails(self): real_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) sim_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) - # Same 2 commits both modes, but sim ran ~2x the forward passes. + # Same 2 commits both modes, but sim ran ~2x the forward passes (the + # residence-bug direction: sim OVER-computes -> violation). real_tr = _trainers_with_rounds({"A": 1, "B": 1}) # 2 fwd / 2 commit = 1.0 sim_tr = _trainers_with_rounds({"A": 3, "B": 3}) # 6 fwd / 2 commit = 3.0 r = pc.compute_conservation_parity(real_agg, sim_agg, real_tr, sim_tr) assert not r["ok"] - assert r["sim_fwd_per_commit"] > r["real_fwd_per_commit"] + assert r["sim_excess_rel"] > 0 + + def test_sim_under_computing_is_ok(self): + # W1 is ASYMMETRIC: sim doing FEWER forward passes than real (the async + # start-tail, not wasted recompute) must NOT fail. + real_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + sim_agg = _agg(agg_rounds=[_cyc(0, [("A", 0.0, 5.0), ("B", 0.0, 5.0)])]) + real_tr = _trainers_with_rounds({"A": 3, "B": 3}) # 3.0 fwd/commit + sim_tr = _trainers_with_rounds({"A": 1, "B": 1}) # 1.0 fwd/commit + r = pc.compute_conservation_parity(real_agg, sim_agg, real_tr, sim_tr) + assert r["ok"] and r["sim_excess_rel"] < 0 def test_skips_without_data(self): empty = _agg(agg_rounds=[]) From 5548384df542b4e998f13000fd342fed64dc213c Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:43:54 -0400 Subject: [PATCH 12/56] docs: add comment-discipline principle (#10) + trim verbose comments MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Standing rule in simulate_fwdllm.md §F #10: comment the WHY crisply (conceptual/ architectural choices, real<->sim divergences, failure-mode learnings); skip trivial mechanical edits; keep it to a sentence or two, deep rationale in §K. Applied it: trimmed the verbose comments added on this branch (_release_sim_slots_at_agg_goal, staleness/residence/contributor-interval/ vclock_now blocks, the launcher fail-fast notes, validate_real, the fluxtune yamls) to crisp form. No code behavior change; full pytest tests/ = 794 passed. Co-Authored-By: Claude Opus 4.8 --- .../scripts/parity/validate_real.py | 18 +-- .../expt_scripts/fluxtune_n10_smoke.yaml | 11 +- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 21 ++-- lib/python/examples/fwdllm/simulate_fwdllm.md | 8 ++ lib/python/flame/launch/aggregator_spawner.py | 8 +- lib/python/flame/launch/runner.py | 7 +- .../horizontal/syncfl/fwdllm_aggregator.py | 113 ++++++------------ .../mode/horizontal/syncfl/fwdllm_trainer.py | 6 +- 8 files changed, 67 insertions(+), 125 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/validate_real.py b/lib/python/examples/async_cifar10/scripts/parity/validate_real.py index 997e8b899..ce50860cd 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/validate_real.py +++ b/lib/python/examples/async_cifar10/scripts/parity/validate_real.py @@ -258,20 +258,10 @@ def check_aggregation(agg: dict) -> dict: def check_grad_residence(agg: dict, trainer: dict) -> dict: - """FwdLLM async grad-path residence (R1) + compute-conservation (W1) on the - REAL side — the hard gate that blocks the §L.4-step-4 sim mechanism change. - - R1: real per-trainer dispatch->commit intervals (contributor_intervals) must - not overlap (~0%). If real itself overlaps, real is inadmissible and D-b is a - two-sided fix — fix real FIRST, never tune sim toward a broken real - (simulate_fwdllm.md §L.3 sanity gate / PARITY.md principle #6). - W1: report forward_passes vs committed grads so the operator can confirm the - real fwd-commit gap is explained by end-of-run in-flight + stale-rejects, not - by real ALSO dropping carried grads. Informational (loose bound) since real - stale-reject / in-flight-at-stop counts aren't separately emitted. - - SKIPs cleanly (ok=True) when the real run carries no contributor_intervals - (a non-fwdllm real run) so async_cifar10 validate_real is unaffected. + """FwdLLM async residence (R1) + compute-conservation (W1) on the REAL side — + the gate blocking the sim mechanism change (§L.3 / PARITY.md principle #6): + if real itself overlaps (R1 > ~0), fix real FIRST, never tune sim to it. + W1 is reported for context. SKIPs on a non-fwdllm real run (no intervals). """ cycles = _fwd_cadence_cycles(agg) have_intervals = any(e.get("contributor_intervals") for e in cycles) diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index 4d02bd4da..e5c45cdd1 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -62,13 +62,10 @@ experiments: id: fluxtune_n10_smoke hyperparameters: rounds: 50 # short smoke run, not the full 300 - # Async fedbuff baseline: accept a stale grad (trained on model_version - # V that lands at V'>V) and down-weight by (V'-V) rather than reject it - # (K-D13/§L D5). Baseline DEFINITION -- set identically real+sim, never - # tuned to close a gap. Feeds the U3 staleness rung. MUST be camelCase - # `stalenessPolicy` to OVERRIDE the base config's key (the snake_case - # form is a distinct dead key that loses at pydantic resolution — the - # fluxtune run ran on `none`, §H re-run finding #2 / K-D15). + # Async fedbuff: accept stale grads + down-weight by (V'-V), not reject + # (K-D13). Set identically real+sim; never a parity lever. camelCase + # MUST match the base key to override it — snake_case is a dead key + # that loses at pydantic resolution (K-D15). stalenessPolicy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 71abd6fa8..9fc5831b0 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -67,20 +67,15 @@ experiments: id: fluxtune_n10_smoke_sim hyperparameters: rounds: 50 # short smoke run, not the full 300 - # Async residence (SIM-only, §L step 4): with c=10 >> agg_goal=3 the - # sim path must COMMIT-THEN-CARRY the surplus arrived grads and hold - # still-busy trainers in their slots across the agg-goal boundary, - # instead of dropping ~7 grads/cycle and re-dispatching them (the 2x - # forward-pass residence violation, K-D12). Real satisfies this by - # channel construction, so this knob is inert in the real sibling. + # SIM-only: with c=10 >> agg_goal=3, commit-then-carry the surplus + # grads + hold busy trainers across the boundary, instead of dropping + # + re-dispatching them (the 2x-forward-pass residence bug, K-D12). + # Real satisfies this by channel construction, so it's inert there. sim_inflight_residence: true - # Async fedbuff baseline: carried grads (residence above) are stale by - # construction, so accept + down-weight by (V'-V) rather than - # reject/ignore (K-D13/§L D5). Baseline DEFINITION, set identically in - # the real sibling -- never a parity lever. MUST be camelCase - # `stalenessPolicy` to OVERRIDE the base config's key; the snake_case - # form is a distinct dead key that loses at pydantic resolution and - # left the fluxtune run on `none` (§H re-run finding #2 / K-D15). + # Async fedbuff: accept stale carried grads + down-weight by (V'-V), + # not reject (K-D13). Set identically in the real sibling; never a + # parity lever. camelCase MUST match the base key to override it — + # snake_case is a dead key that loses at pydantic resolution (K-D15). stalenessPolicy: fedbuff # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 4ed57e4a2..5764f1850 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -357,6 +357,14 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M could affect other aggregators/trainers/examples** (this Batch 2.5 is exactly that: it touched `flame/launch/*` + the shared `parity` engine), and bank that green before launching. A cluster run is far more expensive than the full unit suite (minutes, no GPU/MQTT), so the insurance is cheap. +10. **Comment discipline -- comment the WHY, crisply; skip the trivial.** Write a comment when it carries + knowledge a later reader (or a later you) would otherwise have to re-derive or re-learn the hard way: + a conceptual/algorithmic choice, a baseline/architectural decision, a real<->sim divergence and its + rationale, or a failure mode + why the fix takes the shape it does (so the mistake isn't repeated). + Do **not** comment trivial mechanical edits (snake_case→camelCase, a rename, an obvious guard). When you + do comment, be **crisp** -- one or two tight sentences, not a paragraph restating the code. The deeper + rationale belongs in §K (append-only) with a one-line pointer from the code; the code comment states the + decision, not its whole history. --- diff --git a/lib/python/flame/launch/aggregator_spawner.py b/lib/python/flame/launch/aggregator_spawner.py index 6a4e43ff4..badf23356 100644 --- a/lib/python/flame/launch/aggregator_spawner.py +++ b/lib/python/flame/launch/aggregator_spawner.py @@ -119,11 +119,9 @@ def is_running(self) -> bool: return False return self.process.poll() is None - # A startup crash (e.g. a config/init TypeError) writes a Python traceback to - # the log just before the process exits. Detecting it lets wait_until_ready - # fail FAST instead of the old 5s "alive ⇒ ready" heuristic waving through a - # crash that lands at ~5s -- which then wasted ~86s spawning + waiting on - # trainers that never get an EOT (simulate_fwdllm.md §L.5 defect D-d). + # Detect a startup crash from its traceback so wait_until_ready fails FAST, + # instead of the old 5s "alive ⇒ ready" heuristic waving through a crash that + # lands at ~5s and then wasting ~86s on trainers that never get an EOT (§L.5). _CRASH_MARKER = "Traceback (most recent call last)" def _read_log_tail(self, max_bytes: int = 65536) -> str: diff --git a/lib/python/flame/launch/runner.py b/lib/python/flame/launch/runner.py index 84775aa7b..6bb0d2059 100644 --- a/lib/python/flame/launch/runner.py +++ b/lib/python/flame/launch/runner.py @@ -324,11 +324,8 @@ def run_experiment(self, exp: ExperimentConfig) -> None: agg_rc = getattr(self.aggregator_spawner.process, "returncode", None) rc_msg = f"exit={agg_rc}" if agg_rc == 0 else f"exit={agg_rc} ⚠" if agg_rc not in (0, None): - # The aggregator failed (non-zero exit). The 30s-per-trainer EOT - # grace below is only meaningful on a CLEAN finish -- on a crash - # the trainers will never get an EOT, so terminate them now - # instead of burning ~30s each waiting (simulate_fwdllm.md §L.5 - # defect D-d). + # On a crash the trainers never get an EOT, so the 30s-per-trainer + # grace below is wasted -- terminate them now instead (§L.5 D-d). print(f" aggregator FAILED ({rc_msg}); terminating trainers now " f"(skipping EOT grace).") self.trainer_spawner.terminate_all() diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 7ed889b3a..1b36dbabc 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -289,13 +289,8 @@ def internal_init(self) -> None: self.ends_not_selected_yet = False self.iteration_per_data_id = 0 - # end_id -> {"dispatch_ts", "commit_ts"} for the most recent accepted - # contribution of that end, captured in _process_single_trainer_message - # and emitted per-cycle as `contributor_intervals` on the agg_round - # event. Feeds the R1 in-flight-overlap + W1 compute-conservation rungs - # (simulate_fwdllm.md §L.3). Sim uses the modeled [SIM_SEND_TS, - # SIM_COMPLETION_TS] vclock interval; real falls back to the wall - # interval (PROP_ROUND_START_TIME .. receipt). + # end_id -> {dispatch_ts, commit_ts} of its last accepted contribution, + # emitted per-cycle as contributor_intervals for the R1/W1 rungs (§L.3). self._sim_contrib_intervals = {} # Selection granularity for the sync path (fwdllm/fwdllm_plus): @@ -317,13 +312,8 @@ def internal_init(self) -> None: # but not formally departed (see ROUND_CACHE_STUCK_TIMEOUT_S). self._round_cache_activity_ts: dict = {} - # Staleness handling (flame/config.py Hyperparameters.staleness_policy). - # Read from config into an instance attr so the message handler's - # getattr(self, "staleness_policy", "none") resolves the configured - # value -- without this it silently stayed "none" for every run. Values: - # none (accept, no gate) / round_data_id / exact (reject stale) / - # fedbuff (accept + down-weight by V'-V, the async baseline default, - # K-D13). See _process_single_trainer_message. + # Wire staleness_policy from config into an instance attr; without this + # the message handler's getattr fell back to "none" for every run (K-D15). self.staleness_policy = getattr( self.config.hyperparameters, "staleness_policy", None ) or "none" @@ -335,12 +325,11 @@ def internal_init(self) -> None: self._optimizer_sort_value in OPTIMIZERS_SUPPORTING_GRAD_AGGREGATION ) if self.staleness_policy == "fedbuff" and not self._weighted_aggregation_enabled: + # fedbuff accepts stale grads expecting the optimizer to down-weight + # them by (V'-V); without a weighting optimizer the rate stays 1.0. logger.warning( - "staleness_policy=fedbuff accepts stale grads but optimizer.sort=" - f"{self._optimizer_sort_value} does not support staleness " - "down-weighting (rate stays 1.0); staleness will be recorded but " - "not weighted. Set optimizer.sort=fedbuff for the intended " - "down-weight-by-(V'-V) treatment (K-D13)." + f"staleness_policy=fedbuff but optimizer.sort={self._optimizer_sort_value} " + "does not down-weight staleness (rate stays 1.0)." ) if not self._weighted_aggregation_enabled: logger.info( @@ -825,47 +814,26 @@ def _sim_recv_min_grad(self, channel, recv_ends): return m, md def _release_sim_slots_at_agg_goal(self, channel, is_async): - """Sim residence + rollback-safety at the agg-goal boundary. - - A data_id spans MANY agg-goal cycles (a variance-FAIL retries the same - data_id with iteration_per_data_id += 1). The per-cycle committed marks - (_sim_committed) MUST clear each cycle so the gate does not skip a - trainer re-contributing on the rolled-back data_id. - - Two boundary policies, keyed on the async grad path + the residence flag - (simulate_fwdllm.md §L / K-D12): - - * async + sim_inflight_residence (fluxtune, c >> agg_goal) -- - COMMIT-THEN-CARRY. Hold the still-busy trainers (surplus buffered - arrivals ∪ not-yet-arrived in-flight) in their concurrency slots - BEFORE clearing anything, so residence survives the boundary - (_sim_hold_busy_slots reads _sim_buffer/_sim_inflight_expected -- both - must still be populated). Only the committed subset (already popped - from the buffer in _sim_recv_min_grad) is released for re-selection. - The surplus arrived-but-uncommitted grads are RETAINED in the buffer - and applied at a later fedbuff step (never dropped); their version gap - is handled by staleness_policy (K-D13), not by discarding work. This is - the fwdllm analog of felix simInflightResidence + oort - simInflightCarryover. - - * everything else (the two sync baselines c ≈ agg_goal, or async with the - flag off) -- LEGACY DROP (K-D5/K-D6). A single-pass barrier has no - surplus, so clearing the buffer + in-flight gate is correct and keeps - the flag-off path byte-identical to Batch 1. + """Sim slot release at the agg-goal boundary. Two policies (§L / K-D12): + + - async + sim_inflight_residence (fluxtune, c >> agg_goal): COMMIT-THEN- + CARRY. Hold still-busy trainers (surplus buffer ∪ in-flight) in their + slots BEFORE clearing, release only the committed, and CARRY the surplus + buffer to the next fedbuff step (never dropped; version gap handled by + staleness_policy). fwdllm analog of felix/oort residence+carryover. + - else (sync barriers c ≈ agg_goal, or residence off): LEGACY DROP + (K-D5/K-D6) -- no surplus, so clearing is correct and flag-off is + byte-identical to Batch 1. + + `_sim_committed` always clears so a variance-FAIL re-contributor on the + rolled-back data_id isn't skipped. """ if not self.simulated: return - _residence = getattr(self, "_sim_inflight_residence", False) - if is_async and _residence: - # Hold BEFORE clear: the held set = pending_in_buffer ∪ - # _sim_inflight_expected (residence on), released set = committed. - self._sim_hold_busy_slots(channel) + if is_async and getattr(self, "_sim_inflight_residence", False): + self._sim_hold_busy_slots(channel) # reads buffer/in-flight -> hold before clear self._sim_committed.clear() - # Deliberately DO NOT clear _sim_buffer / _sim_inflight_expected: - # the surplus carries to the next agg-goal cycle (buffered fedbuff - # accumulation) and the not-yet-arrived trainers stay in flight. - return - # Legacy drop path (sync barrier, or async without residence). + return # keep _sim_buffer / _sim_inflight_expected -> carry surplus self._sim_committed.clear() self._sim_buffer.clear() self._sim_inflight_expected.clear() @@ -960,17 +928,13 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): logger.info( f"Received grad with staleness={self._model_version-version}." ) - # Mode semantics: Hyperparameters.staleness_policy (flame/config.py). - # Implementation note: model_version alone already identifies - # (round, data_id) when inc_model_version_per_data_id is set (it - # only advances on a data_id transition), so "round_data_id" needs - # no extra fields; "exact" additionally checks iteration_per_data_id. - # REJECT policies: round_data_id, exact -- drop a stale grad. - # ACCEPT policies: none (no gate), fedbuff (accept + down-weight by - # V'-V in aggregate_grads_from_trainers, K-D13/§L D5). fedbuff is - # the async baseline default: its carried surplus grads (§L - # commit-then-carry) are stale by construction, so they must be - # consumed and weighted, never dropped. + # staleness_policy (flame/config.py). REJECT: round_data_id, exact. + # ACCEPT: none (no gate); fedbuff (consume + down-weight by V'-V in + # aggregate_grads_from_trainers) -- the async default, since its + # carried surplus grads (§L) are stale by construction (K-D13). + # model_version alone identifies (round, data_id) when + # inc_model_version_per_data_id is set, so round_data_id needs no + # extra field; exact also checks iteration_per_data_id. policy = getattr(self, "staleness_policy", "none") stale, stale_reason = False, None if policy == "round_data_id": @@ -1041,12 +1005,9 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): f"Set PROP_CLIENT_TASK_TRAIN_DURATION for {end}: {round_duration.total_seconds():.3f}s" ) - # Record this contribution's [dispatch, commit] interval for the R1 - # in-flight-overlap + W1 compute-conservation rungs (§L.3). Sim: the - # modeled vclock interval the trainer echoes back (exact per - # contribution, so a re-dispatch overwrite can't corrupt a prior - # interval). Real: the wall interval (dispatch prop .. receipt). The - # per-cycle list is assembled + emitted in _process_aggregation_goal_met. + # Record this contribution's [dispatch, commit] interval for R1/W1 + # (§L.3): sim uses the vclock interval the trainer echoes back (exact + # per contribution); real uses the wall interval (dispatch .. receipt). if getattr(self, "_sim_contrib_intervals", None) is not None: if self.simulated: _disp = msg.get(MessageType.SIM_SEND_TS) @@ -1503,11 +1464,9 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): contributing_trainers=_cycle_contributors, agg_observed_s=_cycle_agg_observed_s, extra={ - # Virtual clock at commit (sim only) -- the parity engine + # Virtual clock at commit (sim only). The parity engine # gates its whole clock/throughput/convergence family on - # this (K10). asyncfl emits the same; fwdllm never did, so - # K2/throughput/total_commits/terminal_state/convergence - # were unscoreable for every fwdllm baseline. + # this (K10); fwdllm never emitted it (§H finding #4). "vclock_now": (self._vclock.now if self.simulated and getattr(self, "_vclock", None) else None), "data_id": self.data_id, diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index e168833bb..6a69afd36 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -534,10 +534,8 @@ def _send_grads(self, tag: str) -> None: # the intrinsic (server-overhead-free) task duration from. All # None in real mode -> aggregator ignores them (arrival order). MessageType.SIM_COMPLETION_TS: self._sim_completion_ts, - # Echo the dispatch stamp back so the aggregator can reconstruct - # this contribution's exact [dispatch, completion] interval for - # the R1 in-flight-overlap rung (simulate_fwdllm.md §L.3). None - # in real mode -> aggregator falls back to the wall interval. + # Echo the dispatch stamp so the aggregator can reconstruct this + # contribution's [dispatch, completion] interval for R1 (§L.3). MessageType.SIM_SEND_TS: self._sim_send_ts, MessageType.SIM_CLIENT_TASK_TRAIN_DURATION_S: self._sim_round_duration_s, MessageType.TRAINING_BUDGET_S: self._sim_round_duration_s, From f19dd8934d79431d22a526ff414948ee16b68d4d Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 16:57:12 -0400 Subject: [PATCH 13/56] =?UTF-8?q?fwdllm:=20add=20run=5Fparity.py=20?= =?UTF-8?q?=E2=80=94=20repeatable,=20collision-safe=20real<->sim=20checker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For the after-every-fix loop. Fixes two footguns of calling scripts.parity.cli by hand: the `*fwdllm*` glob also matches fwdllm_plus (wrong pair), and stale dirs get picked. run_parity.py parses the baseline as an exact token from the run-dir name, takes the LATEST pair per baseline, reads agg_goal from each run's config, shows the pairs to confirm (unless --yes), then prints a compact cross-baseline rung summary. --validate also reports live staleness_policy + vclock_now presence. Verified on the banked runs (auto-picked the newest fwdllm pair; vclock_now confirmed emitting live). Co-Authored-By: Claude Opus 4.8 --- .../fwdllm/expt_scripts/run_parity.py | 226 ++++++++++++++++++ lib/python/examples/fwdllm/simulate_fwdllm.md | 6 +- 2 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 lib/python/examples/fwdllm/expt_scripts/run_parity.py diff --git a/lib/python/examples/fwdllm/expt_scripts/run_parity.py b/lib/python/examples/fwdllm/expt_scripts/run_parity.py new file mode 100644 index 000000000..14ed8e836 --- /dev/null +++ b/lib/python/examples/fwdllm/expt_scripts/run_parity.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +"""Repeatable real<->sim parity checker for the fwdllm baselines. + +Discovers the correct, MOST-RECENT real/sim run pair per baseline, shows them for +confirmation, runs the shared parity battery, and prints one compact cross- +baseline summary. Built for the after-every-fix loop (simulate_fwdllm.md §H). + +Two footguns it removes vs calling scripts.parity.cli by hand: + - the `*fwdllm*` glob also matches `fwdllm_plus` (and sorted()[-1] can pick the + wrong one). Here the baseline is parsed as an exact token from the run-dir + name (`run___n_smoke[_]_`), so `fwdllm` + never captures `fwdllm_plus`. + - it always takes the LATEST pair (by the timestamp in the dir name) and reads + agg_goal from each run's own config, so no hand-passed --agg-goal drifts. + +Usage: + python run_parity.py # all 3 baselines, latest pairs, confirm + python run_parity.py --baselines fluxtune # one baseline + python run_parity.py --yes # skip the confirm prompt + python run_parity.py --validate # + live-run checks (staleness/vclock_now) +""" +from __future__ import annotations + +import argparse +import glob +import json +import os +import re +import sys +from pathlib import Path + +_HERE = Path(__file__).resolve().parent +_LIB_PYTHON = _HERE.parents[2] # lib/python +_PARITY_SCRIPTS = _LIB_PYTHON / "examples" / "async_cifar10" / "scripts" +_DEFAULT_EXPERIMENTS = _HERE.parent / "experiments" # examples/fwdllm/experiments + +for _p in (str(_LIB_PYTHON), str(_PARITY_SCRIPTS)): + if _p not in sys.path: + sys.path.insert(0, _p) + +# `run____n_smoke[_]_` +_RUN_RE = re.compile( + r"^run_(?P\d{8}_\d{6})_(?P.+)_n(?P\d+)_smoke" + r"(?:_(?P.+))?_(?Preal|sim)$" +) + + +def _discover(experiments_dir: str) -> dict: + """{(baseline, trace): {'real': (ts, path), 'sim': (ts, path)}} keeping the + latest ts per (baseline, trace, variant).""" + out: dict = {} + for path in glob.glob(os.path.join(experiments_dir, "run_*")): + m = _RUN_RE.match(os.path.basename(path)) + if not m: + continue + key = (m["baseline"], m["trace"] or "") + slot = out.setdefault(key, {}) + prev = slot.get(m["variant"]) + if prev is None or m["ts"] > prev[0]: # ts is fixed-width -> lexical == chronological + slot[m["variant"]] = (m["ts"], path) + return out + + +def _agg_goal(run_dir: str) -> int | None: + cfg = os.path.join(run_dir, "aggregator_config.json") + if not os.path.exists(cfg): + return None + try: + h = json.load(open(cfg)).get("hyperparameters", {}) + g = h.get("aggGoal", h.get("agg_goal")) + return int(g) if g is not None else None + except (ValueError, OSError, TypeError): + return None + + +# rungs shown in the summary (name -> short label), in ladder-ish order +_HEADLINE = [ + ("vclock_telemetry", "vclock"), ("throughput", "thru"), + ("total_commits", "commits"), ("terminal_state", "terminal"), + ("r1_inflight_overlap", "R1"), ("w1_compute_conservation", "W1"), + ("v1_iter_per_data_id", "V1"), ("v2_var_trajectory", "V2"), + ("staleness", "U3"), ("participation", "S2"), + ("convergence", "conv"), ("convergence_loss", "conv_loss"), +] + + +def _status(res: dict) -> str: + if not isinstance(res, dict): + return "–" + if res.get("status") == "SKIP": + return "–" + if res.get("ok"): + return "✓" + return "~" if res.get("tier") == "DIAG" else "✗" + + +def _live_checks(baseline: str, real_dir: str, sim_dir: str) -> list[str]: + """--validate extras: confirm the config/telemetry a fresh run must carry.""" + notes = [] + for tag, d in (("real", real_dir), ("sim", sim_dir)): + logs = glob.glob(os.path.join(d, "*aggregator*.log")) + pol = "?" + if logs: + for line in open(logs[0], errors="replace"): + if "staleness_policy =" in line: + pol = line.split("staleness_policy =")[1].strip() + notes.append(f"{tag} staleness_policy={pol}") + # vclock_now present in sim agg_round telemetry? + tel = glob.glob(os.path.join(sim_dir, "telemetry", "aggregator_*.jsonl")) + n_vclock = 0 + if tel: + for line in open(tel[0], errors="replace"): + if '"event": "agg_round"' in line or '"event":"agg_round"' in line: + e = json.loads(line) + if e.get("vclock_now") is not None: + n_vclock += 1 + notes.append(f"sim vclock_now agg_rounds={n_vclock}") + return notes + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--experiments-dir", default=str(_DEFAULT_EXPERIMENTS)) + ap.add_argument("--baselines", nargs="+", + default=["fwdllm", "fwdllm_plus", "fluxtune"]) + ap.add_argument("--json-dir", default=None, + help="where to write per-pair JSON (default /_parity_reports)") + ap.add_argument("--yes", action="store_true", help="skip the confirm prompt") + ap.add_argument("--validate", action="store_true", + help="also report staleness_policy + vclock_now presence") + args = ap.parse_args(argv) + + from parity.checks import load_run_dir, run_all_parity # noqa: E402 + try: + from parity.checks import _WARN_ONLY_CHECKS # noqa: E402 + except ImportError: + _WARN_ONLY_CHECKS = set() + + found = _discover(args.experiments_dir) + # Build the ordered work list: one pair per requested (baseline, trace). + pairs = [] + for base in args.baselines: + matches = sorted(k for k in found if k[0] == base) + if not matches: + print(f" [SKIP] {base}: no run dirs found in {args.experiments_dir}") + continue + for key in matches: + slot = found[key] + if "real" not in slot or "sim" not in slot: + have = ",".join(slot) or "none" + print(f" [SKIP] {'/'.join(k for k in key if k)}: " + f"missing a side (have: {have})") + continue + pairs.append((key, slot["real"], slot["sim"])) + + if not pairs: + print("No complete real/sim pairs to check.") + return 1 + + # ── confirmation: show exactly which dirs will be compared ── + label_w = max(len("/".join(k for k in key if k)) for key, _, _ in pairs) + print("\n Real<->sim pairs to check (latest per baseline):") + for key, (rts, rdir), (sts, sdir) in pairs: + goal_r, goal_s = _agg_goal(rdir), _agg_goal(sdir) + goal = f"agg_goal={goal_r}" + (f"!={goal_s}⚠" if goal_s != goal_r else "") + label = "/".join(k for k in key if k) + print(f" {label.ljust(label_w)} {goal}") + print(f" real {rts} {os.path.basename(rdir)}") + print(f" sim {sts} {os.path.basename(sdir)}") + if not args.yes and sys.stdin.isatty(): + if input("\n Proceed with these pairs? [y/N] ").strip().lower() not in ("y", "yes"): + print(" Aborted.") + return 0 + + json_dir = args.json_dir or os.path.join(args.experiments_dir, "_parity_reports") + os.makedirs(json_dir, exist_ok=True) + + # ── run + collect ── + summary = [] # (label, {name: result}, json_path, tally, live_notes) + for key, (rts, rdir), (sts, sdir) in pairs: + label = "/".join(k for k in key if k) + goal = _agg_goal(rdir) or 0 + real_agg, real_tr = load_run_dir(rdir) + sim_agg, sim_tr = load_run_dir(sdir) + res = run_all_parity(real_agg, sim_agg, real_tr, sim_tr, agg_goal=goal) + jpath = os.path.join(json_dir, f"parity_{label.replace('/', '_')}_{sts}.json") + json.dump(res, open(jpath, "w"), indent=2, default=str) + n_pass = sum(1 for v in res.values() + if isinstance(v, dict) and v.get("ok") and not v.get("status")) + # a real FAIL = not-ok, not a DIAG warn, not in the warn-only allowlist + fails = [k for k, v in res.items() + if isinstance(v, dict) and v.get("ok") is False + and v.get("tier") != "DIAG" and k not in _WARN_ONLY_CHECKS] + n_skip = sum(1 for v in res.values() + if isinstance(v, dict) and v.get("status") == "SKIP") + tally = (n_pass, len(fails), n_skip) + live = _live_checks(label, rdir, sdir) if args.validate else [] + summary.append((label, res, jpath, tally, fails, live)) + + # ── compact cross-baseline table ── + print("\n" + "=" * 78) + lab_w = max(len(s[0]) for s in summary) + hdr = " " + "baseline".ljust(lab_w) + " " + " ".join(lbl for _, lbl in _HEADLINE) + print(hdr) + for label, res, _jp, _tally, _fails, _live in summary: + cells = [] + for name, lbl in _HEADLINE: + cells.append(_status(res.get(name, {})).center(max(len(lbl), 1))) + print(" " + label.ljust(lab_w) + " " + " ".join(cells)) + print(" legend: ✓ pass ✗ fail ~ warn(diag) – skip") + print("=" * 78) + for label, _res, jpath, tally, fails, live in summary: + p, f, s = tally + print(f" {label}: {p} pass / {f} fail / {s} skip" + + (f" FAILS={fails}" if fails else "")) + for note in live: + print(f" · {note}") + print(f" json: {jpath}") + + any_fail = any(t[1] for _, _, _, t, _, _ in summary) + return 1 if any_fail else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 5764f1850..9cff0ce8d 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -601,7 +601,11 @@ hyperparameters in three volatility tiers (① review-every-run, ② per-baselin infeasible configs (`agg_goal>c`, `num_gpus>visible`, `num_trainers --max-data-id ` (D>0 both sides). -Parity: the `NEXT:` line the runner prints (`scripts.parity.cli --batch` pointed at `fwdllm/experiments`). +Parity: `python expt_scripts/run_parity.py` (the repeatable post-fix checker) — it picks the LATEST +real/sim pair per baseline (parsing the baseline as an exact dir-name token, so `fwdllm` never captures +`fwdllm_plus`; the raw `scripts.parity.cli --batch` glob does), reads agg_goal from each run's config, +prints the pairs to confirm, then a compact cross-baseline rung summary. `--yes` skips the prompt; +`--validate` also reports live `staleness_policy` + `vclock_now` presence. --- From 71852cd55e12eeb4a0af923af4cb4413076aa6ac Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Fri, 3 Jul 2026 22:41:21 -0400 Subject: [PATCH 14/56] fwdllm D-e: Option-A two-lifetime slot/guard split + clock-family data_id re-key MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the fluxtune async concurrency-starvation root (D-e, §H open-root #1) and two checker-completeness roots, so the syn_0 re-run can move toward parity. D-e (K-D16, D6=Option A): the K-D12 carry parked a returned-but-uncommitted trainer in async_oort's selected_ends, which drives the dispatch top-up (extra = c - len(selected_ends)), collapsing sim concurrency to ~1.5 vs real ~5.8 and blowing sim wall to 2.4x. Option B was proven unsafe: async_oort's triplet filter only guards within a cycle (triplet advances each boundary) and _trainer_state_dict was never populated. async_oort already tracks the two lifetimes in separate ledgers; stop conflating them. Override fwdllm's _sim_hold_busy_slots so a returned/carried trainer frees its compute slot (selected_ends -> a different trainer refills C, mirroring real freeing on receipt) but stays un-re-pickable (all_selected guard) until COMMIT; carry + variance-FAIL rollback survive. Populate the (model_version, data_id, iteration) triplet at dispatch so the within-cycle filter has real state. Scope: fwdllm class override only; async_oort/oort/asyncfl and the sync path untouched. Clock-family re-key (#2): unified _progress_axis / _per_progress_last_event helper; throughput/total_commits/terminal_state now count on the axis the run advances (fwdllm data_id, normal FL round). async_cifar10 auto-detects round -> byte-identical. field_coverage aliases (#3): spec rows accept a tuple of names; gpu_compute_s / training_budget_s also match fwdllm's real_gpu_time_s / sim_round_duration_s. Tests: TestOptionASlotGuardSplit, TestProgressAxisRekey, TestFieldCoverageAlias + updated residence assertions. tests/mode + parity sub-package: 421 passed, 7 skipped. Doc: K-D16, F-D6 resolved, H, B.1, L updated. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 94 +++++- lib/python/examples/fwdllm/simulate_fwdllm.md | 296 +++++++++++------- .../horizontal/syncfl/fwdllm_aggregator.py | 88 +++++- .../tests/mode/test_fwdllm_sim_grad_loop.py | 9 +- .../mode/test_fwdllm_sim_grad_residence.py | 52 ++- lib/python/tests/mode/test_parity_checks.py | 102 ++++++ 6 files changed, 507 insertions(+), 134 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 8ff9ce24d..0f0e54816 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -351,6 +351,43 @@ def _per_round_last_event(agg_rounds: list) -> dict: return by_round +def _progress_axis(agg_rounds: list) -> str: + """The run's true progress axis -- the shared "progress key" abstraction + (simulate_fwdllm.md §H open-root #2 / §F.3 re-key). + + Normal FL advances the FL `round` (model_version). fwdllm holds `round` + static (one model, gradients aggregated in place) and advances committed + `data_id` (variance passes), so a clock-family rung keyed on `round` divides + by a counter stuck at 1 and reads the wrong axis. Auto-detect so ONE helper + serves both stacks: use `round` when the run advances it (>1 distinct), else + `data_id` when the fwdllm cadence field `cycle_data_id` (§K-D9) is present, + else fall back to `round`. Async_cifar10 (round-advancing) -> "round" -> + byte-identical; only fwdllm re-keys.""" + rounds = {e.get("round") for e in agg_rounds if e.get("round") is not None} + if len(rounds) > 1: + return "round" + if any(e.get("cycle_data_id") is not None for e in agg_rounds): + return "data_id" + return "round" + + +def _per_progress_last_event(agg_rounds: list, axis: str) -> dict: + """{progress_unit -> last event on that unit (by ts)} on the given axis. + Mirrors _per_round_last_event but keyed on the run's true progress axis + (`round` or fwdllm's `cycle_data_id`), so the clock family measures + progress-per-time on the axis the run actually advances.""" + if axis == "round": + return _per_round_last_event(agg_rounds) + out: dict = {} + for e in agg_rounds: + k = e.get("cycle_data_id") + if k is None: + continue + if k not in out or e.get("ts", 0) > out[k].get("ts", 0): + out[k] = e + return out + + def _per_round_max_speed(agg_rounds: list) -> dict: """Per FL round: max trainer_speed_s across all commits in that round.""" out: dict = {} @@ -1818,7 +1855,11 @@ def throughput_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: return {"ok": False, "tier": "EXACT", "note": "K10: no vclock_now in sim agg_round events — cannot compute throughput"} final_vclock = max(sim_vclock_vals) - sim_by_round = _per_round_last_event(sim["agg_rounds"]) + # Progress-axis re-key (§H #2): count units on the axis the run advances -- + # `round` for normal FL, committed `data_id` for fwdllm (else n_rounds==1). + axis = "data_id" if "data_id" in (_progress_axis(sim["agg_rounds"]), + _progress_axis(real["agg_rounds"])) else "round" + sim_by_round = _per_progress_last_event(sim["agg_rounds"], axis) n_sim_rounds = len(sim_by_round) sim_throughput = n_sim_rounds / final_vclock if final_vclock > 0 else 0.0 @@ -1827,7 +1868,7 @@ def throughput_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: return {"ok": True, "tier": "EXACT", "status": "SKIP", "note": "insufficient real ts data (< 2 agg_round events)"} wall_elapsed = max(real_ts) - min(real_ts) - real_by_round = _per_round_last_event(real["agg_rounds"]) + real_by_round = _per_progress_last_event(real["agg_rounds"], axis) n_real_rounds = len(real_by_round) real_throughput = n_real_rounds / wall_elapsed if wall_elapsed > 0 else 0.0 @@ -1983,9 +2024,22 @@ def total_commits_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: if V <= 0: return {"ok": True, "tier": "EXACT", "status": "SKIP", "note": "matched virtual budget V ≤ 0 — run too short to measure"} - n_sim = sum(1 for e in sim["agg_rounds"] if (e.get("vclock_now") or 0) <= V + 1e-9) - n_real = sum(1 for e in real["agg_rounds"] - if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9) + # Progress-axis re-key (§H #2): count commits on the axis the run advances. + # Normal FL commits once per agg_round event; fwdllm's committed unit is the + # `data_id` (a variance-FAIL cycle rolls back, so raw cycle events overcount), + # so count DISTINCT committed data_ids within V -- keeping U2 == K8's rollup. + axis = "data_id" if "data_id" in (_progress_axis(sim["agg_rounds"]), + _progress_axis(real["agg_rounds"])) else "round" + if axis == "round": + n_sim = sum(1 for e in sim["agg_rounds"] if (e.get("vclock_now") or 0) <= V + 1e-9) + n_real = sum(1 for e in real["agg_rounds"] + if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9) + else: + sim_units = _per_progress_last_event(sim["agg_rounds"], axis) + real_units = _per_progress_last_event(real["agg_rounds"], axis) + n_sim = sum(1 for e in sim_units.values() if (e.get("vclock_now") or 0) <= V + 1e-9) + n_real = sum(1 for e in real_units.values() + if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9) if max(n_sim, n_real, 1) == 0: return {"ok": True, "tier": "EXACT", "note": "no commits in V window"} rel_diff = abs(n_sim - n_real) / max(n_sim, n_real) @@ -2024,18 +2078,23 @@ def terminal_state_parity(real: dict, sim: dict, return {"ok": True, "tier": "EXACT", "status": "SKIP", "note": "matched virtual budget V ≤ 0 — run too short to measure"} - sim_by_round = _per_round_last_event(sim["agg_rounds"]) - real_by_round = _per_round_last_event(real["agg_rounds"]) + # Progress-axis re-key (§H #2): "rounds at V" is really "progress units at V" + # -- FL rounds for normal FL, committed data_ids for fwdllm (round static). + axis = "data_id" if "data_id" in (_progress_axis(sim["agg_rounds"]), + _progress_axis(real["agg_rounds"])) else "round" + unit_key = "round" if axis == "round" else "cycle_data_id" + sim_by_round = _per_progress_last_event(sim["agg_rounds"], axis) + real_by_round = _per_progress_last_event(real["agg_rounds"], axis) sim_rounds_at_V = {r for r, e in sim_by_round.items() if (e.get("vclock_now") or 0) <= V + 1e-9} real_rounds_at_V = {r for r, e in real_by_round.items() if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9} - def _trainers(agg_rounds, round_set): + def _trainers(agg_rounds, unit_set): ts = set() for e in agg_rounds: - if e.get("round") in round_set: + if e.get(unit_key) in unit_set: ts.update(e.get("contributing_trainers", [])) return ts @@ -2215,8 +2274,12 @@ def collect_phases(trainers: dict) -> dict: ("selection.num_eligible", "sel", "num_eligible", "both"), ("selection.avail_composition", "sel", "avail_composition", "both"), ("selection.num_chosen", "sel", "num_chosen", "both"), - ("trainer_round.gpu_compute_s", "trainer_round", "gpu_compute_s", "both"), - ("trainer_round.training_budget_s", "trainer_round", "training_budget_s", "both"), + # fwdllm's trainer emits the same data under different names (§H #3): its + # forward-grad "compute" is real_gpu_time_s and its budget is + # sim_round_duration_s (gpu + modeled delay). Accept either spelling so the + # coverage matrix agrees on both examples instead of false-FAILing fwdllm. + ("trainer_round.gpu_compute_s", "trainer_round", ("gpu_compute_s", "real_gpu_time_s"), "both"), + ("trainer_round.training_budget_s", "trainer_round", ("training_budget_s", "sim_round_duration_s"), "both"), ("task_recv.sim_send_ts", "task_recv", "sim_send_ts", "sim"), ] @@ -2230,10 +2293,15 @@ def field_coverage(real_agg: dict, sim_agg: dict, into "these fields are absent in sim". FAIL-LOUD when an expected field has zero density in a mode that requires it. """ - def _density(events: list, field: str) -> Optional[float]: + def _density(events: list, field) -> Optional[float]: + # `field` may be a single name or a tuple of accepted aliases (an event + # counts as covered if ANY alias is present) -- lets one canonical spec + # row match a different-but-equivalent field name per example (§H #3). if not events: return None - n = sum(1 for e in events if e.get(field) not in (None, [], {})) + fields = field if isinstance(field, tuple) else (field,) + n = sum(1 for e in events + if any(e.get(f) not in (None, [], {}) for f in fields)) return n / len(events) def _agg_evs(agg, src): diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 9cff0ce8d..2cd5547bd 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -131,8 +131,8 @@ the true current picture.* The `§K` column points at the full rationale. | 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer the cifar ladder doesn't model; V/DK/G rungs verify it | §F.1 | | 4 | sct delay model | `sct = send + max(gpu, D)` (sleep-to-fill-budget) | `sct = send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU time; copying cifar's `max()` would desync real<->sim | **K-D2** | | 5 | Per-eval sct | distinct eval sct, ~20x eval speedup | **collapses to the train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20x factor); trainer eval msg is a utility report, not a clocked commit | **K-D3** | -| 6 | Slot release | per-commit (inside `_sim_recv_min`) | at the **agg-goal boundary** (`_release_sim_slots_at_agg_goal`) | one grad per call and a `data_id` spans many agg-goal cycles with variance-FAIL rollbacks; per-commit release would strand a re-contributing trainer across a rollback | **K-D5**, principle #4 | -| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **dropped** (`_sim_buffer.clear()` at boundary) — **⚠ TRIGGERED for fluxtune (2026-07-03), see §L** | K-D6 assumed drop is benign *because* `\|selected\| ≈ agg_goal`; that holds for the two `random`/sync baselines (c≈agg_goal) but **NOT fluxtune** (async, c=10 ≫ agg_goal=3), where it drops ~7 arrived grads/cycle → re-dispatch → 2× forward passes. **DECISION REVERSED → commit-then-carry (§L).** | **K-D6**, **K-D12**, §L | +| 6 | Slot release | per-commit (inside `_sim_recv_min`), one ledger | **two-lifetime split at the boundary** — compute slot (`selected_ends`→`extra`) frees on grad RETURN; re-pick guard (`all_selected`→`filtered_ends`) frees on COMMIT | async_oort already tracks both ledgers; conflating them starved concurrency (D-e). Split: a returned trainer frees its slot (real frees on receipt) but stays un-re-pickable until commit, surviving carry + rollback | **K-D5**/**K-D16**, principle #4 | +| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **carried** (async/fluxtune, K-D12) — commit-then-carry + Option-A slot/guard split (K-D16); **dropped** stays correct for sync (c≈agg_goal, no surplus) | drop was benign only for `\|selected\|≈agg_goal` (sync); fluxtune (c=10≫agg_goal=3) dropped ~7 grads/cycle → 2× passes → reversed to carry; the carry then needed the slot-release timing fix (K-D16) | **K-D6**, **K-D12**, **K-D16**, §L | | 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` (reuse the primitives, fork the orchestration) | `_sim_recv_min`'s per-commit slot release + withheld/staggered paths key on WEIGHTS semantics -- wrong for a grad pool released on the agg-goal boundary | **K-D4** | | 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's entire config corpus is `time_mode: real` and shipped with no sim path; a "simulated" default risks silently half-activating an unbuilt path | **K-D1** | | 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | the post-mutation `data_id`/`iteration_per_data_id` advance BEFORE the event emits, so binning by them is off-by-one; snapshot before the pass/fail branch makes V1 exact | **K-D9** | @@ -307,6 +307,46 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M fwdllm; over-instrument in Stage Avail before Stage 3. - **D4 -- eval-delay factor.** FedFwd eval is a forward pass (~= train cost), unlike async_cifar10's ~20x-faster eval. Confirm the factor per baseline before Stage 1's per-eval `sct` stamp. +- **D6 -- D-e fix: how to decouple compute-slot release from the re-pick guard. RESOLVED → Option A + (K-D16).** *Decision gate answered:* async_oort's triplet filter (`async_oort.py:1660`) skips only when + `trainer_version_states[end] == agg_version_state`, i.e. WITHIN the current cycle; since the triplet + advances every agg-goal boundary it does NOT block re-pick of a *carried* trainer across the boundary, and + `_trainer_state_dict` was never even populated — so **Option B is unsafe** and A is required. Landed: free + `selected_ends` (compute slot) on RETURN, hold `all_selected` (re-pick guard) to COMMIT, carry the grad, + survive the rollback boundary; triplet guard now populated at dispatch. See §H open-root #1 for the + root-cause and K-D16 for the as-built. *Original options analysis kept below for the record.* The bug: the + K-D12 commit-then-carry fix leaves a + returned-but-uncommitted trainer in the selector's `all_selected`, which the dispatch top-up reads as an + occupied compute slot (`extra = c − |all_selected|`, `count_avl_train` excludes `all_selected`, + `async_oort.py:1580`) → sim refills the pipeline by ~1/cycle → concurrency collapses to ~1.5 vs real ~5.8. + `all_selected` conflates two lifetimes that must split: **compute-slot occupancy** (must end at grad + **RETURN** — a returned trainer is idle, its slot should refill to keep C computing, exactly what real's + channel does on receipt) vs the **re-pick guard** (must hold until **COMMIT** — don't re-select the same + trainer for the same `(model_version, data_id, iteration)` triplet while its grad is still carried). + Two implementation strategies: + - **Option A — two explicit sets (faithful, more code).** Add a distinct in-flight/computing set: a trainer + enters at dispatch, leaves at grad RETURN; the top-up uses `extra = c − |in_flight_computing|` so the + pipeline refills to C on return. A separate guard set holds the carried-but-uncommitted trainers + un-re-pickable until commit, then clears. *Pros:* mirrors async_cifar10 exactly (§3.resid frees on + receipt + §4.9 `simInflightCarryover` carries the grad); explicit, auditable, unit-testable in isolation + (assert concurrency ≈ C and R1 overlap == 0%). *Cons:* new state to keep consistent with the carry buffer + **across variance-FAIL rollbacks** at the agg-goal boundary (K-D5) — the main correctness surface to test. + - **Option B — release the slot at return, reuse the existing triplet-version filter as the guard + (minimal change).** Only fix WHEN the slot frees: remove a returned trainer from `all_selected` at grad + receipt (so `count_avl_train`/`extra` see it free), and rely on the EXISTING `trainer_version_states` + triplet filter (`async_oort.py:1660`, skips a trainer already holding the current triplet) to block + re-pick until commit advances the version. *Pros:* smallest diff, no new bookkeeping, leans on selector + machinery already present. *Cons:* the residence guarantee now depends on selector-internal filtering + rather than an explicit set (harder to reason about / test alone); correctness hinges on the triplet + filter covering the ENTIRE return→commit window **including across a variance rollback** (where + model_version/data_id/iter can shift and the guard could leak → re-pick a still-carried trainer → R1 + violation). **Must prove R1 stays 0% under rollback before choosing B.** + - **Leaning A** (explicit + testable + faithful) unless a scoping pass shows the triplet filter provably + covers the return→commit window across rollbacks — then B is the cheaper equivalent. **Decision gate:** + does `trainer_version_states` block re-pick for a returned-uncommitted trainer through a variance-FAIL + rollback? Answer that first; it picks A vs B. Either way: `fwdllm_aggregator.py` + fluxtune yaml only, + sync path untouched (K-D5/K-D11), shared weight path untouched (principle #8); log the chosen option in + §K as the D-e resolution. ### Locked principles (from async_cifar10, carried over) 1. **Sim does real forward-grad compute, charges modeled time.** GPU runs the real JVP; the agg stamps @@ -394,109 +434,114 @@ those are baseline-defining knobs; a cadence gap is an upstream set/order/clock ## §H Status -*(Run-log: record per-baseline run dirs, lowest broken rung, root hypothesis, score X/N, JSON path; keep -the run-length budget keyed. One section, updated in place. Batch 1 pytest gate is green; **first run-log -entry lands after the Phase-1 syn_0 sign-off run** -- which is gated behind Batch 2.)* - -**Batch 1 landed (done):** all six subtasks complete, `test_fwdllm_*` = 75 green. Code is the source of -truth; the as-built decisions are logged in **§K-D1..D8**. Batch-1 pytest files: -`tests/mode/test_fwdllm_trainer_sim_duration.py` (trainer additive stamp + no-sleep), -`tests/mode/test_fwdllm_sim_grad_loop.py` (async sct buffer / in-flight gate / rollback), -`tests/mode/test_fwdllm_sim_sync_barrier.py` (first-k-smallest + U6 lags), plus the flag-off -`simulated=False` fixes across the four pre-existing fwdllm agg/trainer tests. - -**Batch 2 landed (done):** variance-cadence rung layer + telemetry, pytest-gated. `test_fwdllm_*` = 81 -green; `test_parity_checks.py` = 63 green (adds the V/DK/G suites); the parity sub-package = 115 green. -As-built decisions in **§K-D9/D10**. Files: `parity/checks.py` (V1-V5/DK1-DK3/G1-G2 + `run_all_parity` + -`CHECK_META`), `parity_checks.py` (re-export shim), `fwdllm_aggregator.py` (cadence `extra` snapshot), -`tests/mode/test_parity_checks.py` + `tests/mode/test_fwdllm_agg_telemetry.py`. **Next §H entry lands -after the Phase-1 syn_0 sign-off run** (§I.6). - -**Phase-1 sign-off — IN PROGRESS.** First `fwdllm` syn_0 smoke (`--mode both --delays off --max-runtime-s -300 --max-data-id 3`, 2026-07-03): real ✅ aggregated + hit `max_data_id`; sim ❌ deadlocked to -`max_runtime`, 0 aggregations. Root-caused + fixed (**K-D11**: real-only clamp on `ends_not_selected_yet`). - -**Full 3-baseline syn_0 smoke (2026-07-03, `--mode both --delays off --max-runtime-s 300 --max-data-id 3`, -batch runner) — three defects surfaced; remediation plan in §L:** -| baseline | real | sim | finding | -|---|---|---|---| -| **fwdllm** (sync) | ✅ 6 aggs, 211s | ✅ 6 aggs, 181s | mechanics OK post-K-D11; parity not yet scored | -| **fwdllm_plus** (sync) | ❌ **CRASH** | ❌ **CRASH** | `read_trainer_unavailability(base_dir=…)` kwarg mismatch vs the `metadata_dir` override → `TypeError` at `internal_init` (both modes; config-level, ORACULAR-only path). Fixed (param rename). Launcher then wasted 86s (no fail-fast) → §L defect **D-d**. | -| **fluxtune** (async) | ✅ 21 aggs, 206s | ✅ 22 aggs, **356s (1.7×)** | **residence violation on the grad path.** Per-forward-pass compute identical (5.1s), but sim ran **228 forward passes vs real 109** (even 22–23/trainer vs uneven 7–14) — sim recomputes the whole c=10 cohort every agg-goal cycle. Root: `_release_sim_slots_at_agg_goal` clears `_sim_buffer`/`_sim_inflight_expected` **before** `_sim_hold_busy_slots` reads them (holds nothing) + `sim_inflight_residence=False` + K-D6 drop. Cadence diverged downstream (data_id 2: 80 vs 8 iters → V1). §L defects **D-a/D-b/D-c**. | - -**§L Batch-2.5 LANDED (2026-07-03), pytest-gated — re-run (step 6) is now the next action.** All the -remediation code shipped and is green: the R1/W1 residence telemetry (`contributor_intervals` on -agg_round + trainer `SIM_SEND_TS` echo), the R1 in-flight-overlap + W1 compute-conservation rungs (+ -`validate_real` real-side gate), the D-a/D-b commit-then-carry fix in `_release_sim_slots_at_agg_goal` -(residence-gated: `sim_inflight_residence: true` in the fluxtune sim yaml; flag-off ⇒ byte-identical drop -for the sync baselines), the D-c fedbuff staleness-accept path (+ the `self.staleness_policy` config -wiring that was silently missing — K-D15), and the §L.5 launcher fail-fast (agg traceback detection + -immediate trainer terminate on non-zero agg exit). Pytest: `tests/mode` 297 passed / 7 skipped; parity -sub-package 115. **Next: the 3-baseline syn_0 re-run** — expect fluxtune sim forward passes ≈ real (W1 -ratio→1), R1==0% both modes, fwdllm_plus no longer crashing. Fluxtune / fwdllm_plus first-time results -above. - -**§L re-run RESULTS (2026-07-03, `--mode both --delays off --max-runtime-s 300 --max-data-id 3`; parity -`scripts.parity.cli` per pair, agg_goal fwdllm/plus=10, fluxtune=3).** All 6 runs completed (fwdllm_plus no -longer crashes — D-d/param-rename confirmed). The runs carry the Batch-2.5 telemetry (`contributor_intervals` -present) and `simInflightResidence=True` is active on the sim side. - -**HEADLINE — the residence fix (D-a/D-b) is VALIDATED:** -| metric (fluxtune async) | pre-fix (first smoke) | post-fix (this run) | -|---|---|---| -| sim forward passes | **228** (vs real 109) | **82** (vs real 120) | -| sim wall vs real | 356s (**1.7×**) | comparable | -| R1 in-flight overlap (sim) | — (violated) | **0.0%** (real 1.7%, within tol) → **PASS** | -| V1 iters-per-data_id | **80 vs 8** (data_id 2) | mean **real 7.67 / sim 8.0** (near-converged) | - -The 2× recompute is gone: sim went from over-computing (228) to slightly UNDER-computing (82 < real 120). -R1 residence is exact (sim 0.0% overlap). V1 cadence collapsed from an 80-vs-8 blowup to ~8-vs-8. - -**Per-baseline enforced score:** fwdllm **33/35**, fwdllm_plus **27/34**, fluxtune **28/36** (many of the -misses are ONE shared pre-existing telemetry gap, below — not regressions): - -| rung | fwdllm | fwdllm_plus | fluxtune | note | -|---|---|---|---|---| -| **R1** in-flight overlap | PASS (0/0%) | PASS (0/0%) | **PASS** (real 1.7% / sim 0.0%) | residence exact | -| **W1** compute-conservation | PASS (1.13/1.17) | PASS (1.14/1.17) | **WARN** (real 1.74 / sim 1.14, +35%) | see #1 below | -| **V1** iters-per-data_id | PASS (2.0/2.0) | FAIL (1.67/2.0) | FAIL (7.67/8.0, KS .33) | means close; KS just over tol | -| V2 var-trajectory | PASS | PASS | PASS | grad values mode-invariant ✓ | -| **U3** staleness | PASS | PASS | **FAIL** (KS .27) | D-c NOT active — see #2 | -| participation / S2 | PASS | PASS | PASS | | - -**Deeper root-cause of the four non-clean rungs (all diagnosed to artifact or config, none to a residence -regression) + the fixes made this session:** - -1. **W1 (fluxtune WARN) — start-tail artifact, not a modeling bug. Rung refined.** `trainer_round` counts - forward-pass STARTS. Both modes drop NOTHING at the aggregator (received grads == commits: real 69==69, - sim 73≈72) and R1 sim-overlap==0, so residence is exact. The 120-vs-82 gap is that a live async real - system dispatches continuously over wall-time and leaves a large in-flight START tail (51), while the - clock-gated sim leaves a small one (10) — real > sim is EXPECTED and amortizes with run length / D>0. - W1 was symmetric and fired on this benign direction. **FIX: W1 is now asymmetric** — it flags only a sim - EXCESS over real (the 2× recompute it was built for); sim under-computing is reported as benign. fluxtune - W1 now PASSES (`sim_excess_rel −0.35`). -2. **V1 (fwdllm_plus + fluxtune FAIL) — short-run truncation, not a cadence bug. No code change.** The - per-data_id iteration series match on every FULL data_id and diverge only on the last, truncated one: - fluxtune real `{0:11,1:8,2:4}` vs sim `{0:11,1:7,2:6}` (data_id 0 EXACT), fwdllm_plus `{0:2,1:2,2:1}` vs - `{0:2,1:2,2:2}`. With 3 data_ids a single boundary difference → KS 0.33. **Fix is operational: a longer - run (more data_ids) for a scoreable distribution**, per the §I.6 convergence config (D>0 + real budget). -3. **U3 staleness (fluxtune FAIL) — config key collision. FIXED.** The generated config carried BOTH - `{stalenessPolicy: none (base), staleness_policy: fedbuff (override)}`; the camelCase base key wins at - pydantic resolution, so the run logged `staleness_policy = none`. (Residence escaped this — no base - `simInflightResidence` key.) **FIX: the fluxtune yamls now use camelCase `stalenessPolicy`.** Confirm on - re-run that the aggregator logs `staleness_policy = fedbuff` and U3 recovers. -4. **Clock/throughput family (ALL baselines FAIL/SKIP) — missing `vclock_now`. FIXED.** The fwdllm - aggregator never stamped `vclock_now` on agg_round, so K10 gated off the whole clock family - (K2/throughput/total_commits/terminal_state/field_coverage). **FIX: emit `vclock_now` (mirror asyncfl).** - This unblocks the virtual-budget V the convergence sign-off (C1/C2) needs. Separately, fwdllm_plus/ - fluxtune still miss oracular/oort availability+selection rungs — Batch-2 telemetry-coverage validation, - not residence. - -**Net:** residence + cadence are correct (R1==0, received==commits, V1 exact on full data_ids). Every -non-clean rung traced to an artifact (W1 start-tail, V1 truncation) or a config/telemetry gap (U3 key, -vclock_now) — all fixed in code except the V1 short-run, which needs the longer D>0 run. **Before re-launch: -the fixes above are in; re-run with the §I.6 convergence config (D>0, longer budget) and re-score.** JSON: -`parity_{fwdllm,fwdllm_plus,fluxtune}.json`. +*Discipline (from PARITY.md §Status / §Settled-roots / §Dead-ends): this section is the **consolidated +ground state**, rewritten in place — NOT an append-per-run log. The end goal is end-to-end real↔sim parity, +so what we keep is (1) the current per-baseline state table, (2) the **open roots** ranked lowest-rung-first, +(3) a **fixes-landed** ledger (what worked, so we don't redo it), and (4) a **dead-ends & corrections** +ledger (what we tried or believed that was wrong, so we don't retry it). Per-run detail lives in the JSON + +git history; the as-built rationale lives in §K.* + +**Status (2026-07-03 → Option-A landed; the numbers below are the PRE-fix run, `max_data_id=3` — a SHORT +convergence run, agg_goal fwdllm/plus=10, fluxtune=3).** Batches 1–2 + §L Batch-2.5 + the **Option-A D-e fix +(K-D16)** + the **clock-family re-key (#2)** + the **field-coverage alias (#3)** are landed & pytest-green +(`tests/mode` 421 passed / 7 skipped incl. parity sub-package). Residence (R1) is exact on all three; grad +values are mode-invariant. **No open sim-mechanism root remains in code** — the three checker/mechanism roots +below (#1 D-e, #2 clock-family axis, #3 field-name) are FIXED and moved to *Fixes landed*; the only remaining +action is the **§L step-6 re-run** to bank the post-fix numbers (esp. fluxtune concurrency vs real ~5.8 and +wall) and a **longer run (max_data_id=10, the yaml default)** to clear the #4 short-run truncation. The table +below is the pre-fix baseline the re-run is measured against. + +**Per-baseline ground state (PRE-fix; re-run pending):** +| baseline | wall real→sim | R1 resid | U3 staleness | V1/V2 cadence | status | +|---|---|---|---|---|---| +| **fwdllm** (sync) | 107→**96s** ✓ | PASS (0%) | PASS | PASS / PASS | clock-family axis re-keyed (#2 fixed); re-run to confirm | +| **fwdllm_plus** (sync) | 335→**184s** ✓ | PASS (0%) | PASS | FAIL(trunc) / FAIL | #2 fixed; V1 trunc needs the longer run (#4); selection_detail after re-run | +| **fluxtune** (async) | 125→**304s ✗ (2.4×)** | PASS (0%) | **FAIL** (sim 0.230** (real trainers sleep ≈1.13s/pass; sim charges ≈1.12s to the vclock). The clock-family fails are + real, and trace to the rung being keyed on `round` not `data_id` (open root #2) — not to D being off. +- **Tuning `var_threshold` / `max_iterations_per_data_id` to close a cadence gap** — pre-emptively rejected + (§G): these are baseline-defining knobs, not parity levers. A cadence gap is an upstream set/order/clock + divergence (here, D-e). --- @@ -781,6 +826,32 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent close a gap; PARITY.md §F.4 / principle #3). **Obligation:** confirm the fedbuff optimizer weighting path is active and `staleness = V′−V` emits on the commit event (U3). **Where:** `fluxtune_n10_smoke*.yaml` `staleness_policy`; fedbuff optimizer weighting; commit-event `staleness` emit. Implement in §L step 5. +- **K-D16 D-e resolution — Option A two-lifetime split (free compute slot on RETURN, hold re-pick guard to + COMMIT).** The K-D12 commit-then-carry fix parked a returned-but-uncommitted trainer in async_oort's + `selected_ends`, which drives the dispatch top-up (`extra = c − len(selected_ends)`), so the pipeline + refilled ~1/cycle and sim wall blew up 2.4× (§H open-root D-e). **Decision gate (D6) answered by reading + the code:** async_oort's triplet filter (`async_oort.py:1660`, skip if `trainer_version_states[end] == + agg_version_state`) is a WITHIN-cycle guard only — the triplet advances every agg-goal cycle, so it does + NOT block re-pick of a *carried* trainer across the boundary (its recorded triplet ≠ the new current one). + So **Option B is unsafe** (its fallback guard doesn't cover the carry window), and `_trainer_state_dict` + was in fact **never populated** (filter inert). **Chose Option A** (§F-D6): async_oort already tracks the + two lifetimes in SEPARATE ledgers — `selected_ends` = compute-slot occupancy (drives `extra`), + `all_selected` = re-pick guard (drives `filtered_ends`) — they were merely conflated. Override fwdllm's + `_sim_hold_busy_slots` to hold a RETURNED/carried trainer in `all_selected` ONLY (frees its slot so a + DIFFERENT trainer refills concurrency to C, mirroring real freeing on channel receipt) while + still-computing trainers hold BOTH; the guard clears on COMMIT so one-in-flight-per-trainer survives the + carry boundary + variance-FAIL rollback (principle #4). ALSO populate `_trainer_state_dict` at dispatch = + `(model_version, data_id, iteration)` and prune to the busy set at the boundary, so the triplet filter has + real state (the within-cycle guard the user asked to re-key) and the R1 rung can prove no per-triplet + re-pick. **Options weighed:** (a) explicit two-set split [CHOSEN — the ledgers already exist; smallest + faithful change]; (b) reuse the triplet filter as the guard [REJECTED — proven not to cover the carry + window across rollbacks]. **Scope:** `fwdllm_aggregator.py` override + dispatch (fwdllm blast radius); + async_oort/oort and the shared asyncfl `_sim_hold_busy_slots` UNTOUCHED (principle #8); sync path untouched + (K-D5/K-D11). **Where:** `fwdllm_aggregator.py::_sim_hold_busy_slots` (override) + + `_release_sim_slots_at_agg_goal` + `_distribute_weights_async` (triplet emit); + `tests/mode/test_fwdllm_sim_grad_residence.py::TestOptionASlotGuardSplit`. **Concurrency is emergent** — + the split restores dispatch top-up; the actual in-flight number (vs real ~5.8) is measured by the §L step-6 + re-run, not asserted analytically. --- @@ -906,9 +977,18 @@ step 6 is the operator re-run.** As-built notes in K-D14/D15. carried stale grad is **accepted and down-weighted by `V′−V`**, and confirm the fedbuff optimizer's weighting path is active + emits `staleness` on the commit event (for U3). *Exit:* a unit test that a carried stale grad is accepted+weighted (never silently dropped) and that `staleness=V′−V` is emitted. -6. **⏳ One re-run** (operator step; only after 1–5 green — now the case): the 3-baseline syn_0 smoke → `parity_checks`. *Exit:* fluxtune - sim forward passes ≈ real (W1 ratio →1), R1==0% both modes, S2 PASS, V1 data_id-iters converge, sim - wall ≈ real; fwdllm_plus no longer crashes; record in §H. Then proceed to the Phase-1 convergence +6. **✅ First re-run (2026-07-03).** Confirmed W1→PASS (sim under-computes, not over), R1==0% both modes, + fwdllm_plus no longer crashes. **Surfaced D-e (concurrency starvation): the commit-then-carry fix freed + the compute slot at COMMIT instead of at RETURN, collapsing sim in-flight concurrency to ~1.5 vs real + ~5.8 → fluxtune sim wall 2.4× real, U3 low.** +7. **✅ D-e fixed — Option A two-lifetime split (K-D16).** Free `selected_ends` (compute slot) on RETURN, + hold `all_selected` (re-pick guard) to COMMIT, carry + rollback survive; triplet populated at dispatch. + Landed with the clock-family `data_id` re-key (#2) and field-coverage aliases (#3); pytest-green + (`tests/mode` 421 passed / 7 skipped incl. parity). Current ground state in **§H**. +8. **▶ IN FLIGHT (operator launched the post-fix re-run, 2026-07-03).** One `--mode both` 3-baseline run at syn_0 with the yaml + default `max_data_id_progress=10` (do NOT cap at 3 — clears the #4 truncation) → `run_parity.py`. Confirm: + fluxtune concurrency recovers toward real ~5.8, wall gap closes, U3 non-trivial, clock-family rungs now + read on the `data_id` axis, V1/V2 within tolerance on the longer run. Then the Phase-1 convergence sign-off (§I.6). ### §L.5 ✅ Launcher fail-fast (D-d) — parallel tooling task (no parity coupling) — DONE diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 1b36dbabc..5c12de7f2 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -32,7 +32,7 @@ from flame.common.constants import DeviceType from flame.common.util import weights_to_device, weights_to_model_device from flame.config import OptimizerType, TrainerAvailState -from flame.end import PROP_END_AVL_STATE +from flame.end import KEY_END_STATE, PROP_END_AVL_STATE, VAL_END_STATE_NONE from flame.mode.composer import CloneComposer import pickle from flame.mode.horizontal.syncfl.top_aggregator import ( @@ -814,13 +814,17 @@ def _sim_recv_min_grad(self, channel, recv_ends): return m, md def _release_sim_slots_at_agg_goal(self, channel, is_async): - """Sim slot release at the agg-goal boundary. Two policies (§L / K-D12): + """Sim slot release at the agg-goal boundary. Two policies (§L / K-D12/D16): - async + sim_inflight_residence (fluxtune, c >> agg_goal): COMMIT-THEN- - CARRY. Hold still-busy trainers (surplus buffer ∪ in-flight) in their - slots BEFORE clearing, release only the committed, and CARRY the surplus - buffer to the next fedbuff step (never dropped; version gap handled by - staleness_policy). fwdllm analog of felix/oort residence+carryover. + CARRY with the Option-A two-lifetime split (§H D-e / D6, K-D16). Hold + still-busy trainers BEFORE clearing and CARRY the surplus buffer to the + next fedbuff step (never dropped). The overridden `_sim_hold_busy_slots` + then splits the two lifetimes async_oort already tracks in separate + ledgers: a RETURNED-but-uncommitted (carried) trainer frees its compute + slot (selected_ends -> `extra`) so the next distribute refills + concurrency to C, while staying un-re-pickable (all_selected -> + filtered_ends) until its grad commits. - else (sync barriers c ≈ agg_goal, or residence off): LEGACY DROP (K-D5/K-D6) -- no surplus, so clearing is correct and flag-off is byte-identical to Batch 1. @@ -840,6 +844,70 @@ def _release_sim_slots_at_agg_goal(self, channel, is_async): if is_async: self._sim_hold_busy_slots(channel) + def _sim_hold_busy_slots(self, channel) -> None: + """Option-A two-lifetime split (§H open-root D-e / D6 resolution, K-D16). + + Overrides felix's `_sim_hold_busy_slots` (which held every busy trainer in + BOTH selector ledgers) to fix the fluxtune concurrency-starvation root: the + K-D12 carry parked a RETURNED-but-uncommitted trainer in `selected_ends`, + which drives the dispatch top-up (`extra = c − len(selected_ends)` in + async_oort), so the pipeline refilled ~1 trainer/cycle and sim wall blew up + 2.4× (§H). async_oort already tracks the two lifetimes in SEPARATE ledgers; + we just stop conflating them: + - `selected_ends` -> COMPUTE-SLOT occupancy (drives `extra`). A returned + trainer's grad is done computing, so its slot frees on RETURN — exactly + what real does when the channel leaves SEND on receipt — letting the + next distribute refill concurrency to C with a DIFFERENT trainer. + - `all_selected` -> RE-PICK guard (drives `filtered_ends`). Held until + the grad COMMITS, so one-in-flight-per-trainer survives the carry + boundary (K-D12) and variance-FAIL rollbacks (principle #4): the same + trainer is never re-dispatched while its carried grad is outstanding. + + Runs at the agg-goal boundary. Robust whether or not `cleanup_recvd_ends` + already stripped the RECVD (returned) ends from the ledgers: it explicitly + releases every no-longer-busy trainer and re-establishes the split for the + busy sets. fwdllm-class override only -> async_cifar10 (asyncfl) untouched + (principle #8).""" + sel = getattr(channel, "_selector", None) + if sel is None: + return + requester = getattr(sel, "requester", None) + all_selected = getattr(sel, "all_selected", None) + selected_ends = getattr(sel, "selected_ends", None) + + buffered = set(self._sim_buffer.pending_ends()) # returned, carried grad + computing = set(self._sim_inflight_expected) - buffered # dispatched, not returned + busy = buffered | computing + self._sim_pending_commit |= busy + # Prune the (model_version, data_id, iteration) triplet guard to the + # still-outstanding set (a trainer whose grad committed is re-pickable). + self._trainer_state_dict = { + e: v for e, v in getattr(self, "_trainer_state_dict", {}).items() + if e in busy + } + + if isinstance(all_selected, dict): + # Release slot+guard for every trainer no longer busy (its grad + # committed this cycle) -> re-pickable next cycle. + for eid in [e for e in list(all_selected.keys()) if e not in busy]: + del all_selected[eid] + if channel is not None and channel.has(eid): + channel._ends[eid].set_property(KEY_END_STATE, VAL_END_STATE_NONE) + if isinstance(selected_ends, dict) and requester in selected_ends: + selected_ends[requester].discard(eid) + # Guard: block re-pick of every still-busy trainer (carried ∪ computing). + for eid in busy: + if eid not in all_selected: + all_selected[eid] = time.time() + + # Compute slot: held ONLY by still-computing trainers. Carried (returned) + # trainers are removed from selected_ends so `extra` reopens their slot. + if isinstance(selected_ends, dict) and requester in selected_ends: + for eid in computing: + selected_ends[requester].add(eid) + for eid in buffered: + selected_ends[requester].discard(eid) + def _aggregate_grads_async(self, tag: str) -> None: """ Aggregate local model GRADIENTS asynchronously for FwdLLM. @@ -2311,6 +2379,14 @@ def _distribute_weights_async( # running min) so the gate never laps a not-yet-committed trainer. _budget = self._sim_trainer_budget.get(end, self._sim_budget_min) self._sim_inflight_expected[end] = _round_now + _budget + # Record the (model_version, data_id, iteration) triplet this + # trainer is now outstanding at, so async_oort's triplet filter + # (the within-cycle re-pick guard) has real state to read and the + # R1 residence rung can prove no re-pick per triplet (§H D6). The + # boundary split prunes this to the still-busy set. Residence-only + # so the sync baselines stay byte-identical (empty map -> inert). + if getattr(self, "_sim_inflight_residence", False): + self._trainer_state_dict[end] = self._curr_agg_version channel.send(end, payload) logger.info( f"[Distribute] Done. Sent {_n_weights_sent} WEIGHTS + " diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py index 3740253cb..6b6d544e3 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py @@ -15,9 +15,6 @@ from datetime import datetime -from flame.mode.horizontal.asyncfl.top_aggregator import ( - TopAggregator as _AsyncBase, -) from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator from flame.mode.horizontal.syncfl.top_aggregator import TopAggregator as _SyncBase from flame.mode.message import MessageType @@ -69,7 +66,8 @@ class _FakeGradAgg: _release_sim_slots_at_agg_goal = TopAggregator._release_sim_slots_at_agg_goal _advance_sim_clock = _SyncBase._advance_sim_clock _sim_recv_grace_s = _SyncBase._sim_recv_grace_s - _sim_hold_busy_slots = _AsyncBase._sim_hold_busy_slots + # fwdllm overrides felix's hold with the Option-A two-lifetime split (K-D16). + _sim_hold_busy_slots = TopAggregator._sim_hold_busy_slots # grace-window class knobs the base method reads off self SIM_RECV_GRACE_FLOOR_S = 0.0 SIM_RECV_GRACE_FACTOR = 0.0 @@ -84,6 +82,9 @@ def __init__(self): self._sim_trainer_budget = {} self._sim_budget_min = 12.0 self._sim_fill_ema = 0.0 + self._sim_pending_commit = set() + self._sim_inflight_residence = False + self._trainer_state_dict = {} def _drain(self, channel, recv_ends, n): """Commit n grads, returning the ordered list of committed scts.""" diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py index ba5f6d450..12b753589 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -60,10 +60,14 @@ def test_surplus_carried_and_busy_held(self): assert agg._sim_inflight_expected == {"E": 50.0} # Per-cycle committed marks cleared so a re-contributor isn't skipped. assert agg._sim_committed == set() - # Busy trainers (surplus ∪ in-flight) held in their slots; the two - # committed ones (A, B) released for re-selection. + # Option-A two-lifetime split (K-D16): every busy trainer (carried + # surplus C,D ∪ in-flight E) stays un-re-pickable in the all_selected + # GUARD; the two committed ones (A, B) are released. But only the + # still-COMPUTING trainer (E) holds a compute SLOT (selected_ends, which + # drives `extra`) -- the returned/carried C,D freed their slots so the + # next distribute refills concurrency with a different trainer. assert set(ch._selector.all_selected) == {"C", "D", "E"} - assert ch._selector.selected_ends["agg"] == {"C", "D", "E"} + assert ch._selector.selected_ends["agg"] == {"E"} def test_end_to_end_surplus_commits_next_cycle_no_refetch(self): agg = _residence_agg(residence=True) @@ -105,6 +109,48 @@ def test_no_same_trainer_reselected_while_in_flight(self): # only the committed A is released. assert "B" in held and "C" in held and "A" not in held assert "B" in agg._sim_pending_commit and "C" in agg._sim_pending_commit + # Two-lifetime split: returned/carried B freed its compute slot (frees + # `extra` -> a DIFFERENT trainer refills the pipeline), still-computing C + # keeps its slot. Neither can be re-picked (both in the all_selected guard). + assert ch._selector.selected_ends["agg"] == {"C"} + + +class TestOptionASlotGuardSplit: + """Option-A (§H D-e / D6 / K-D16): compute-slot occupancy (selected_ends, + drives `extra`) frees on RETURN; re-pick guard (all_selected) frees on + COMMIT. The two ledgers async_oort already tracks, no longer conflated.""" + + def test_returned_trainer_frees_slot_but_stays_guarded(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B", "C", "D"]) + # A,B still computing; C,D returned (buffered surplus, not yet consumed). + agg._sim_inflight_expected = {"A": 10.0, "B": 20.0, "C": 30.0, "D": 40.0} + agg._sim_buffer.add("C", 30.0, None) + agg._sim_buffer.add("D", 40.0, None) + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + # Slot ledger (extra): only the two still-computing trainers occupy it. + assert ch._selector.selected_ends["agg"] == {"A", "B"} + # Guard ledger: all four busy trainers un-re-pickable. + assert set(ch._selector.all_selected) == {"A", "B", "C", "D"} + + def test_triplet_guard_pruned_to_busy_on_commit(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B", "C"]) + # A committed (popped) this cycle; B carried; C computing. + agg._trainer_state_dict = { + "A": (1, 0, 0), "B": (1, 0, 0), "C": (1, 0, 0), + } + agg._sim_committed = {"A"} + agg._sim_buffer.add("B", 20.0, None) + agg._sim_inflight_expected = {"C": 30.0} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + # The committed A is dropped from the triplet guard (re-pickable); the + # still-outstanding B, C remain so async_oort's filter keeps skipping them. + assert set(agg._trainer_state_dict) == {"B", "C"} class TestFlagOffByteIdentical: diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 0b4331b23..01a808f6e 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -312,6 +312,108 @@ def test_diverged_fails(self): assert not r["ok"], r +def _fwd_round(data_id, contributing, vclock=None, ts=0.0): + """fwdllm-style agg_round: `round` (model_version) static, progress on the + committed `data_id` axis (cycle_data_id, §K-D9).""" + e = {"event": "agg_round", "round": 1, "ts": ts, + "cycle_data_id": data_id, "var_good_enough": True, + "contributing_trainers": contributing, "staleness": [0], + "agg_goal_count": 1} + if vclock is not None: + e["vclock_now"] = vclock + return e + + +class TestProgressAxisRekey: + """§H open-root #2 / §F.3: the clock family must measure progress on the axis + the run advances. fwdllm keeps `round` at 1 and advances committed `data_id`, + so a rung keyed on `round` divides by a counter stuck at 1. The re-key auto- + detects the axis; async_cifar10 (round-advancing) stays byte-identical.""" + + def test_throughput_counts_data_ids_not_static_round(self): + # 10 committed data_ids, round pinned at 1, matched 10 units / 100s. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float((d + 1) * 10)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float((d + 1) * 10), + ts=float(d + 1)) + for d in range(10)]) + r = pc.throughput_parity(real, sim, tol_rel=0.10) + # Re-keyed to data_id: 10 units, NOT collapsed to 1 stuck round. + assert r["sim_rounds"] == 10 and r["real_rounds"] == 10, r + assert r["ok"], r + + def test_throughput_divergence_caught_on_data_id_axis(self): + # sim crawls (10 data_ids in 200s vclock) vs real (10 in 100s wall) -> 2x. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float((d + 1) * 10)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float((d + 1) * 20), + ts=float(d + 1)) + for d in range(10)]) + r = pc.throughput_parity(real, sim, tol_rel=0.10) + assert not r["ok"], r + + def test_terminal_state_data_ids_at_V_nonzero(self): + real = _agg(agg_rounds=[_fwd_round(d, ["a", "b"], ts=float((d + 1) * 10)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a", "b"], vclock=float(d * 10), + ts=float(d + 1)) + for d in range(10)]) + r = pc.terminal_state_parity(real, sim) + # Previously real_rounds_at_V==0 (round static); now counts data_ids. + assert r["real_rounds_at_V"] > 0 and r["sim_rounds_at_V"] > 0, r + assert r["ok"], r + + def test_total_commits_counts_distinct_data_ids(self): + # A variance-FAIL retry emits 2 cycles on the SAME data_id; the commit + # count must be distinct data_ids (2), not raw cycles (3). + real = _agg(agg_rounds=[ + _fwd_round(0, ["a"], ts=0.0), + _fwd_round(0, ["a"], ts=5.0), # retry, same data_id + _fwd_round(1, ["a"], ts=10.0), + ]) + sim = _agg(agg_rounds=[ + _fwd_round(0, ["a"], vclock=0.0, ts=1.0), + _fwd_round(0, ["a"], vclock=5.0, ts=2.0), + _fwd_round(1, ["a"], vclock=10.0, ts=3.0), + ]) + r = pc.total_commits_parity(real, sim, tol_rel=0.05) + assert r["n_sim_commits"] == 2 and r["n_real_commits"] == 2, r + assert r["ok"], r + + def test_normal_fl_still_keyed_on_round(self): + # >1 distinct round -> axis stays "round"; identical to pre-re-key. + real = _agg(agg_rounds=[_round(r, ["a"], [0], ts=float(r * 10)) + for r in range(1, 11)]) + sim = _agg(agg_rounds=[_round(r, ["a"], [0], vclock=float(r * 10), + ts=float(r)) for r in range(1, 11)]) + r = pc.throughput_parity(real, sim, tol_rel=0.10) + assert r["sim_rounds"] == 10, r + assert r["ok"], r + + +class TestFieldCoverageAlias: + """§H open-root #3: fwdllm's trainer emits gpu/budget under different names; + the coverage matrix must accept either spelling instead of false-FAILing.""" + + def test_fwdllm_field_aliases_count_as_covered(self): + agg = _agg(agg_rounds=[{"event": "agg_round", "round": 1, "ts": 0.0, + "vclock_now": 1.0, "trainer_speed_s": [1.0], + "staleness": [0], "stat_utility": [1.0], + "contributing_trainers": ["t1"]}]) + sel = {"num_eligible": 5, "avail_composition": {"a": 1}, + "num_chosen": 3} + agg["selection_train"] = [sel] + # trainer emits the fwdllm spellings only. + tr = {"t1": {"trainer_round": [ + {"real_gpu_time_s": 1.2, "sim_round_duration_s": 2.3}]}} + r = pc.field_coverage(agg, agg, tr, tr) + gpu = r["matrix"]["trainer_round.gpu_compute_s"] + bud = r["matrix"]["trainer_round.training_budget_s"] + assert gpu["real"] == 1.0 and gpu["sim"] == 1.0, r + assert bud["real"] == 1.0 and bud["sim"] == 1.0, r + assert "trainer_round.gpu_compute_s(real)" not in r["violations"], r + + class TestTrainerSpeedParity: def test_identical_passes(self): rounds_with_speed = [ From 50819d04f5d45fbdc32c7d39f8dc2f4097a48a7d Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 02:19:03 -0400 Subject: [PATCH 15/56] fwdllm sim: K-D17 (drain-gate + triplet-on-return) + K-D17b (hold slot to commit); R1 regression open MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes the fluxtune async sim deadlock and the in_flight mis-measurement; leaves one known open regression (R1) documented for the next session (§H NEXT SESSION). K-D17 — unblock the K-D16 deadlock (fluxtune sim committed 2/10 grads, 0 agg_rounds): - Bug A: _aggregate_grads_async now drains while _sim_buffer/_sim_inflight_expected is non-empty even when the channel reports no RECV end. The sct reorder buffer, not the transient channel RECV state, is the sim commit path's source of truth (RECV gating is a real-transport artifact; principle #8). - Bug B: the async_oort re-pick triplet is stamped on grad RETURN (_process_single_trainer_message), not at dispatch — stamping the whole cohort at _curr_agg_version froze the eligible pool before any commit could advance the version (bootstrap deadlock the D6 note missed). K-D17b — in_flight measurement (revert K-D16 Option-A slot split, realign with felix): - _sim_hold_busy_slots now holds every dispatched-but-not-committed trainer (computing ∪ carried) in BOTH selected_ends and all_selected until COMMIT, called per-commit in _sim_recv_min_grad. len(selected_ends) now == virtual-time in-flight, matching asyncfl::_sim_hold_busy_slots/_sim_recv_min. Validated: sim in_flight 3.38 -> 9.49 vs real 9.75; V2 var-trajectory now passes. KNOWN OPEN REGRESSION (do not launch overnight fluxtune sim until fixed): K-D17b broke R1 residence (0% -> 44.7% same-trainer in-flight overlap). Root lead: buffered ends reset to KEY_END_STATE=NONE become re-selectable because async_oort doesn't exclude all_selected. Full analysis + ordered fix options in §H "NEXT SESSION" + §K-D17b. Scope: fwdllm class only; async_oort/oort/asyncfl + sync path untouched (fwdllm/ fwdllm_plus sim byte-identical). Full tests/mode green (315 passed / 7 skipped); new test_fwdllm_sim_drain_and_repick.py + TestVirtualInflightSlotHold. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 206 +++++++++++++++--- .../horizontal/syncfl/fwdllm_aggregator.py | 165 +++++++++----- .../mode/test_fwdllm_sim_drain_and_repick.py | 205 +++++++++++++++++ .../tests/mode/test_fwdllm_sim_grad_loop.py | 6 + .../mode/test_fwdllm_sim_grad_residence.py | 58 +++-- 5 files changed, 535 insertions(+), 105 deletions(-) create mode 100644 lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 2cd5547bd..ca57d93ee 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -131,7 +131,7 @@ the true current picture.* The `§K` column points at the full rationale. | 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer the cifar ladder doesn't model; V/DK/G rungs verify it | §F.1 | | 4 | sct delay model | `sct = send + max(gpu, D)` (sleep-to-fill-budget) | `sct = send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU time; copying cifar's `max()` would desync real<->sim | **K-D2** | | 5 | Per-eval sct | distinct eval sct, ~20x eval speedup | **collapses to the train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20x factor); trainer eval msg is a utility report, not a clocked commit | **K-D3** | -| 6 | Slot release | per-commit (inside `_sim_recv_min`), one ledger | **two-lifetime split at the boundary** — compute slot (`selected_ends`→`extra`) frees on grad RETURN; re-pick guard (`all_selected`→`filtered_ends`) frees on COMMIT | async_oort already tracks both ledgers; conflating them starved concurrency (D-e). Split: a returned trainer frees its slot (real frees on receipt) but stays un-re-pickable until commit, surviving carry + rollback | **K-D5**/**K-D16**, principle #4 | +| 6 | Slot release | per-commit (inside `_sim_recv_min`), hold slot to COMMIT | **same as felix — hold slot to COMMIT** (`selected_ends` AND `all_selected` held for every dispatched-but-not-committed trainer, released on commit) | a returned-but-uncommitted trainer is still in flight in VIRTUAL time (grad commits when vclock reaches sct), so its slot is occupied; K-D16 briefly freed it on physical RETURN → undercounted `in_flight` 3× (K-D17b reverted, realigned with felix) | **K-D5**/**K-D16**/**K-D17b**, principle #4 | | 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **carried** (async/fluxtune, K-D12) — commit-then-carry + Option-A slot/guard split (K-D16); **dropped** stays correct for sync (c≈agg_goal, no surplus) | drop was benign only for `\|selected\|≈agg_goal` (sync); fluxtune (c=10≫agg_goal=3) dropped ~7 grads/cycle → 2× passes → reversed to carry; the carry then needed the slot-release timing fix (K-D16) | **K-D6**, **K-D12**, **K-D16**, §L | | 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` (reuse the primitives, fork the orchestration) | `_sim_recv_min`'s per-commit slot release + withheld/staggered paths key on WEIGHTS semantics -- wrong for a grad pool released on the agg-goal boundary | **K-D4** | | 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's entire config corpus is `time_mode: real` and shipped with no sim path; a "simulated" default risks silently half-activating an unbuilt path | **K-D1** | @@ -313,8 +313,10 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M advances every agg-goal boundary it does NOT block re-pick of a *carried* trainer across the boundary, and `_trainer_state_dict` was never even populated — so **Option B is unsafe** and A is required. Landed: free `selected_ends` (compute slot) on RETURN, hold `all_selected` (re-pick guard) to COMMIT, carry the grad, - survive the rollback boundary; triplet guard now populated at dispatch. See §H open-root #1 for the - root-cause and K-D16 for the as-built. *Original options analysis kept below for the record.* The bug: the + survive the rollback boundary; triplet guard populated on grad RETURN (**corrected in K-D17** — the K-D16 + as-built stamped it at DISPATCH, which froze the pool before the first commit and deadlocked the re-run). + See §H open-root #1/#1b for the root-cause and K-D16/K-D17 for the as-built. *Original options analysis + kept below for the record.* The bug: the K-D12 commit-then-carry fix leaves a returned-but-uncommitted trainer in the selector's `all_selected`, which the dispatch top-up reads as an occupied compute slot (`extra = c − |all_selected|`, `count_avl_train` excludes `all_selected`, @@ -441,27 +443,68 @@ so what we keep is (1) the current per-baseline state table, (2) the **open root ledger (what we tried or believed that was wrong, so we don't retry it). Per-run detail lives in the JSON + git history; the as-built rationale lives in §K.* -**Status (2026-07-03 → Option-A landed; the numbers below are the PRE-fix run, `max_data_id=3` — a SHORT -convergence run, agg_goal fwdllm/plus=10, fluxtune=3).** Batches 1–2 + §L Batch-2.5 + the **Option-A D-e fix -(K-D16)** + the **clock-family re-key (#2)** + the **field-coverage alias (#3)** are landed & pytest-green -(`tests/mode` 421 passed / 7 skipped incl. parity sub-package). Residence (R1) is exact on all three; grad -values are mode-invariant. **No open sim-mechanism root remains in code** — the three checker/mechanism roots -below (#1 D-e, #2 clock-family axis, #3 field-name) are FIXED and moved to *Fixes landed*; the only remaining -action is the **§L step-6 re-run** to bank the post-fix numbers (esp. fluxtune concurrency vs real ~5.8 and -wall) and a **longer run (max_data_id=10, the yaml default)** to clear the #4 short-run truncation. The table -below is the pre-fix baseline the re-run is measured against. +**Status (2026-07-04 EOD → K-D17 + K-D17b landed. in_flight measurement FIXED (sim 9.49 vs real 9.75) but +K-D17b introduced an R1 residence regression (0%→44.7%). STOPPED before the overnight run; resume from the +"NEXT SESSION" block below.)** +> **NEXT SESSION (2026-07-05) pickup — start here.** +> **Where we are:** fluxtune sim is healthy (no deadlock, K-D17). K-D17b made `in_flight` correct (virtual +> in-flight ~9.5, matches real ~9.75) and fixed V2 — BUT it broke R1 residence (44.7% same-trainer in-flight +> overlap; run `020254`, report `parity_fluxtune_20260704_020254.json`). Sync baselines (fwdllm/fwdllm_plus) +> are untouched by all this and are safe to ground-run anytime. +> **The R1 bug (prime hypothesis, ~1 fix + 1 smoke to confirm):** `_sim_hold_busy_slots` +> (`fwdllm_aggregator.py`) resets buffered-but-uncommitted ends to `KEY_END_STATE=NONE` to preserve their +> `selected_ends` slot. But async_oort selects by avail/end-state and does NOT exclude `all_selected`, so a +> NONE-state buffered trainer is re-selectable → re-dispatched while in flight → R1 overlap. +> **Fix options to try (in order):** (1) **Drop the `KEY_END_STATE=NONE` reset** for buffered ends and rely +> only on the explicit `selected_ends[requester].add(eid)` to hold the slot — then re-run a fluxtune sim +> smoke and check BOTH `sel_ends`≈8 (measurement still good) AND R1≈0% (`grep SIM_GRAD_RECV`; run +> `run_parity.py --baselines fluxtune`). If the channel re-strips the slot without the reset (sel_ends drops +> back to ~3), then (2) make async_oort's selection **exclude `all_selected` members** (or keep buffered ends +> in RECV state, not NONE) so the guard actually blocks re-pick while the slot stays counted. (3) Failing +> that, keep buffered ends un-selectable via the K-D17 triplet (verify the triplet covers them — it may have +> advanced). **Validate:** R1 back to 0%, in_flight still ~9.5, `tests/mode` green, add a pytest that a +> buffered-uncommitted trainer is NOT re-selected. +> **Then:** re-check `selection_detail.chosen` (sim 0.83 vs real 1.76 — likely follows R1) and the +> throughput/commits/terminal fails; run the **longer run** (`--max-data-id` dropped → 10) to clear the #4 +> short-run truncation before judging V1/g2/throughput. Only after R1 is clean, launch the overnight +> grounding run: `bash run_sequential.sh --mode both --delays on --max-runtime-s 3600 --yes`. +> **Do NOT redo:** the in_flight measurement fix (K-D17b) is correct and validated — keep the hold-to-commit; +> only the state-reset that re-opens selection is at fault. The temporary `[SIM_GRAD_RECV] ... inflight_exp=/ +> sel_ends=` diagnostic in `_sim_recv_min_grad` is useful — keep it until R1 is closed, then optionally trim. + +**Status (2026-07-04 → K-D17 landed; the K-D16 re-run DEADLOCKED, root-caused to two bugs, now fixed — +awaiting a fresh fluxtune re-run).** Batches 1–2 + §L Batch-2.5 + Option-A (K-D16) + clock-family re-key (#2) ++ field alias (#3) + the **K-D17 drain-gate + triplet-at-return fixes** are landed & pytest-green (full +`tests/mode` 314 passed / 7 skipped). The **post-K-D16 fluxtune smoke (`run_20260703_230407`) did NOT +recover — it deadlocked** (1 cohort dispatched, only 2 grads committed, 0 `agg_round` events, in-flight +0.28, 1663 empty select-rounds). Root cause = **two independent bugs (K-D17)**: (A) the sim drain loop +gated on channel RECV state instead of the `_sim_buffer` — the greedy first drain emptied RECV and stranded +the other ~8 already-received grads (`"no ends yet"` ×1681), so agg_goal=3 was never met from a full cohort; +(B) K-D16 stamped the async_oort re-pick triplet at DISPATCH, freezing the whole eligible pool before any +commit could advance the version (bootstrap case the D6 note missed). Both fixed: drain now keys on the +buffer/in-flight set; the triplet is stamped on grad RETURN. Residence (R1) exact on all three; grad values +mode-invariant. **Remaining action: a fresh fluxtune re-run** to bank concurrency (vs real ~5.8) / wall / U3 +/ V1 / selection, then the **longer run (`max_data_id_progress=10`)** to clear #4. The table below is the +PRE-K-D16 baseline the re-run is measured against (the K-D16 run is void — it deadlocked). **Per-baseline ground state (PRE-fix; re-run pending):** | baseline | wall real→sim | R1 resid | U3 staleness | V1/V2 cadence | status | |---|---|---|---|---|---| | **fwdllm** (sync) | 107→**96s** ✓ | PASS (0%) | PASS | PASS / PASS | clock-family axis re-keyed (#2 fixed); re-run to confirm | | **fwdllm_plus** (sync) | 335→**184s** ✓ | PASS (0%) | PASS | FAIL(trunc) / FAIL | #2 fixed; V1 trunc needs the longer run (#4); selection_detail after re-run | -| **fluxtune** (async) | 125→**304s ✗ (2.4×)** | PASS (0%) | **FAIL** (sim 0.23 0 or _sim_inflight_expected` even with RECV empty; real + path unchanged (guarded on `self.simulated`). + - **Bug B — the K-D16 triplet was stamped at DISPATCH.** Stamping the whole cohort at the current + `_curr_agg_version` made every dispatched trainer match `agg_version_state`, so async_oort's filter + (`async_oort.py:1660`, skip if equal) excluded the ENTIRE pool → `filtered_ends=0` → no re-dispatch. The + version only advances at a commit boundary, which (via Bug A) is never reached → permanent freeze. **This + is the bootstrap case the D6 analysis missed:** D6 reasoned the triplet is safe because "it advances + every agg-goal boundary" — true only once commits flow; BEFORE the first commit (or whenever the + in-flight set alone can't reach agg_goal), it freezes the pool. Conceptually the triplet means "already + CONTRIBUTED this triplet," a property of a RETURNED trainer; an in-flight-but-not-returned trainer is + already guarded by its compute slot (`selected_ends`). *Fix:* stamp `_trainer_state_dict[end] = + _curr_agg_version` on grad RETURN (in `_process_single_trainer_message`, residence-gated), NOT at + dispatch. Preserves every K-D16 benefit (slot frees on return, guard holds return→commit, R1 intact) and + unfreezes the top-up. + Either fix alone breaks the deadlock (A lets the initial cohort drain to a commit which advances the + version and unfreezes the triplet; B restores the stream of RECV transitions that masked A pre-K-D16) — + both landed for correctness. **Where:** `fwdllm_aggregator.py::_aggregate_grads_async` (drain gate) + + `_process_single_trainer_message` (triplet on return) + `_distribute_weights_async` (dispatch stamp + removed); `tests/mode/test_fwdllm_sim_drain_and_repick.py` (`TestDrainGateNotBlockedByEmptyRecv` + + `TestRepickTripletStampedOnReturn`). **Scope:** fwdllm class only; async_oort/oort/asyncfl + sync path + untouched (empty triplet map ⇒ inert). Full `tests/mode` green (314 passed / 7 skipped). Concurrency/wall + recovery is emergent — measured by the fluxtune re-run, not asserted. + +- **K-D17b The K-D17 fluxtune concurrency "gap" was mostly a MEASUREMENT artifact — revert the K-D16 + Option-A slot split, realign with felix (hold the slot to COMMIT).** The post-K-D17 fluxtune run was + healthy (no deadlock) but `selection_detail` still FAILed: reported `in_flight` sim **3.38** vs real 9.75. + A diagnostic (`inflight_exp` = dispatched-not-committed vs `sel_ends` = `len(selected_ends)`) settled it: + the sim keeps **~7.8** trainers in flight in VIRTUAL time (close to real's 9.75), but the `in_flight` + telemetry reads `len(selected_ends)` ≈ **2.73** — trainers physically COMPUTING right now. *Root:* the + `in_flight` metric is `len(selected_ends)` (`selector/__init__.py:161`), and K-D16 Option A freed + `selected_ends` on **physical RETURN** — a wall event with no virtual-time meaning. In virtual time a + returned-but-uncommitted trainer is still in flight (its grad commits only when the vclock reaches its + sct), so its slot is genuinely occupied. Freeing it undercounts `in_flight` 3× AND makes + `extra = c − len(selected_ends)` read false free capacity (the selector churns `desired_extra` 7–10, + blocked by the guard). *Confirmed against felix:* `asyncfl/top_aggregator.py::_sim_hold_busy_slots` + (:1453) holds the FULL dispatched-but-not-committed set in `selected_ends` and `_sim_recv_min` (:606-627) + releases only on COMMIT; its docstring names freeing-the-slot-on-return as the "over-selection bug." So + fwdllm's Option-A split was a deviation from the proven reference; freeing the slot manifested as an + UNDER-count only because fwdllm's extra `all_selected` guard blocked the over-selection felix warns of. + *Fix (K-D17b):* rewrite `_sim_hold_busy_slots` to hold EVERY outstanding trainer (`_sim_inflight_expected` + ∪ buffered surplus, minus committed) in BOTH `selected_ends` and `all_selected` until commit, and call it + per-commit inside `_sim_recv_min_grad` (mirroring felix's per-commit reset, since recv_fifo marks + freshly-buffered ends RECVD and the channel strips their slots). Now `len(selected_ends)` = virtual-time + in-flight, so both `in_flight` and `extra` are correct. Safe to revert Option A *now* only because K-D17 + fixed the drain (the pre-K-D16 "concurrency 1.5" was the drain stall, not holding-to-commit). **Where:** + `_sim_hold_busy_slots` + `_sim_recv_min_grad` (per-commit hold) + `_release_sim_slots_at_agg_goal` + (docstring); `tests/mode/test_fwdllm_sim_grad_residence.py::TestVirtualInflightSlotHold` (+ updated + `TestCommitThenCarryResidenceOn`). Full `tests/mode` green (315/7). + + **VALIDATION RESULT (run `020254`, 2026-07-04) — PARTIAL WIN + A NEW R1 REGRESSION (do not ship as-is):** + - ✅ **in_flight measurement FIXED.** `sel_ends` 2.73 → **7.78**, matching `inflight_exp` (8.10). In the + parity battery `selection_detail.in_flight` = sim **9.49** vs real 9.75 (**rel_diff 0.027**, was 0.65). + The core goal is met — the telemetry now reports virtual-time in-flight. + - ✅ **V2 var-trajectory now PASSES** (sim 0.894 vs real 1.008, KS 0.177<0.20); U3/S2/conv/W1 pass. + - ❌ **R1 residence REGRESSED: 0.0% → 44.7% overlap** (real 1.8%, tol 2%). K-D17b re-dispatches a trainer + while its prior grad is still in flight. **Root lead:** `_sim_hold_busy_slots` resets buffered-but- + uncommitted ends to `KEY_END_STATE=NONE` (to stop the RECVD-strip of their slot, mirroring felix), but + **async_oort's selection filter keys on avail-state/end-state, NOT on `all_selected`** (`async_oort.py` + ~1643-1695 filters by `count_avl_train`+triplet; `all_selected` at :405 is only *logged*). So a NONE-state + buffered trainer looks idle/available -> re-selected -> overlapping in-flight interval -> residence + violation. fwdllm's async_oort does not exclude `all_selected` the way felix's path effectively does. + - ❌ still failing: `selection_detail.chosen` (sim 0.83 vs real 1.76), throughput/total_commits/terminal + (sim commits faster — 3 vs 1 at matched virtual budget 27.2s; partly short-run #4), field_coverage, + avail_composition, V1/g2 (#4 truncation). + + **STOPPED before the overnight run** — R1 is a correctness regression; banking 6h of runs on it would be + wrong. Resume steps are in **§H -> "NEXT SESSION (2026-07-05) pickup"**. + --- ## §L Phase-1 sign-off remediation — async grad-path residence & carry-over (Batch 2.5, staged) diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 5c12de7f2..9a2ef8977 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -806,10 +806,28 @@ def _sim_recv_min_grad(self, channel, recv_ends): if _b is not None: self._sim_trainer_budget[_end] = float(_b) self._sim_budget_min = min(self._sim_budget_min, float(_b)) + # Reassert selected_ends == the virtual-time in-flight set after this + # commit (K-D17b): recv_fifo just marked the freshly-buffered ends RECVD, + # which strips their slots, but they are still in flight until THEY commit. + # felix does this reset per-commit in _sim_recv_min; without it the + # in_flight telemetry (sampled at the next distribute) undercounts ~3x. + if getattr(self, "_sim_inflight_residence", False): + self._sim_hold_busy_slots(channel) + # in_flight (virtual, dispatched-not-committed) vs the physically-computing + # selected_ends that feeds the in_flight telemetry — kept for concurrency + # parity debugging. + _sel = getattr(getattr(channel, "_selector", None), "selected_ends", None) + if isinstance(_sel, dict): + _sel_n = sum(len(v) for v in _sel.values()) + elif isinstance(_sel, (set, list)): + _sel_n = len(_sel) + else: + _sel_n = -1 logger.info( f"[SIM_GRAD_RECV] round={getattr(self, '_round', -1)} " f"end={str(_end)[-4:]} sct={sct:.1f} T_v={self._vclock.now:.1f} " - f"buf_depth={len(self._sim_buffer)}" + f"buf_depth={len(self._sim_buffer)} " + f"inflight_exp={len(self._sim_inflight_expected)} sel_ends={_sel_n}" ) return m, md @@ -817,14 +835,13 @@ def _release_sim_slots_at_agg_goal(self, channel, is_async): """Sim slot release at the agg-goal boundary. Two policies (§L / K-D12/D16): - async + sim_inflight_residence (fluxtune, c >> agg_goal): COMMIT-THEN- - CARRY with the Option-A two-lifetime split (§H D-e / D6, K-D16). Hold - still-busy trainers BEFORE clearing and CARRY the surplus buffer to the - next fedbuff step (never dropped). The overridden `_sim_hold_busy_slots` - then splits the two lifetimes async_oort already tracks in separate - ledgers: a RETURNED-but-uncommitted (carried) trainer frees its compute - slot (selected_ends -> `extra`) so the next distribute refills - concurrency to C, while staying un-re-pickable (all_selected -> - filtered_ends) until its grad commits. + CARRY (K-D12). Hold still-busy trainers BEFORE clearing and CARRY the + surplus buffer to the next fedbuff step (never dropped). The overridden + `_sim_hold_busy_slots` holds EVERY dispatched-but-not-committed trainer + (computing ∪ carried) in BOTH its compute slot (selected_ends) and the + re-pick guard (all_selected) until its grad commits — the virtual-time + in-flight set, matching felix (K-D17b; supersedes the K-D16 Option-A + split that wrongly freed the slot on physical RETURN). - else (sync barriers c ≈ agg_goal, or residence off): LEGACY DROP (K-D5/K-D6) -- no surplus, so clearing is correct and flag-off is byte-identical to Batch 1. @@ -845,29 +862,29 @@ def _release_sim_slots_at_agg_goal(self, channel, is_async): self._sim_hold_busy_slots(channel) def _sim_hold_busy_slots(self, channel) -> None: - """Option-A two-lifetime split (§H open-root D-e / D6 resolution, K-D16). - - Overrides felix's `_sim_hold_busy_slots` (which held every busy trainer in - BOTH selector ledgers) to fix the fluxtune concurrency-starvation root: the - K-D12 carry parked a RETURNED-but-uncommitted trainer in `selected_ends`, - which drives the dispatch top-up (`extra = c − len(selected_ends)` in - async_oort), so the pipeline refilled ~1 trainer/cycle and sim wall blew up - 2.4× (§H). async_oort already tracks the two lifetimes in SEPARATE ledgers; - we just stop conflating them: - - `selected_ends` -> COMPUTE-SLOT occupancy (drives `extra`). A returned - trainer's grad is done computing, so its slot frees on RETURN — exactly - what real does when the channel leaves SEND on receipt — letting the - next distribute refill concurrency to C with a DIFFERENT trainer. - - `all_selected` -> RE-PICK guard (drives `filtered_ends`). Held until - the grad COMMITS, so one-in-flight-per-trainer survives the carry - boundary (K-D12) and variance-FAIL rollbacks (principle #4): the same - trainer is never re-dispatched while its carried grad is outstanding. - - Runs at the agg-goal boundary. Robust whether or not `cleanup_recvd_ends` - already stripped the RECVD (returned) ends from the ledgers: it explicitly - releases every no-longer-busy trainer and re-establishes the split for the - busy sets. fwdllm-class override only -> async_cifar10 (asyncfl) untouched - (principle #8).""" + """Assert `selected_ends` == the VIRTUAL-time in-flight set, i.e. every + dispatched-but-not-committed trainer (`_sim_inflight_expected` ∪ buffered + surplus), holding BOTH its compute slot (`selected_ends`, drives + `extra = c − len(selected_ends)`) AND its re-pick guard (`all_selected`) + until its grad COMMITS. This realigns fwdllm with the felix reference + (`asyncfl/top_aggregator.py::_sim_hold_busy_slots` / `_sim_recv_min` + :606-627,1453) after the K-D16 Option-A deviation. + + WHY (K-D17b): a returned-but-uncommitted trainer is still in flight in + VIRTUAL time — its grad commits only when the vclock reaches its sct — so + its slot is genuinely occupied. Option A freed that slot on physical + RETURN (a wall event with no virtual-time meaning): `selected_ends` + collapsed to the ~3 physically-computing while ~8 were truly in flight, + so the `in_flight` telemetry (`len(selected_ends)`) undercounted 3× and + `extra` read false free capacity (measured `_sim_inflight_expected`≈7.8 + vs `selected_ends`≈2.7 vs real 9.75). Freeing on physical return is the + exact "over-selection" hazard felix documents; fwdllm's added guard + turned it into an under-count instead. Holding to COMMIT also keeps + one-in-flight-per-trainer across the carry boundary + variance-FAIL + rollbacks (principle #4). Idempotent — safe to call per-commit + (`_sim_recv_min_grad`) and at the boundary. fwdllm-class override only + (adds the triplet prune); async_cifar10 (asyncfl) untouched (principle #8). + """ sel = getattr(channel, "_selector", None) if sel is None: return @@ -875,38 +892,42 @@ def _sim_hold_busy_slots(self, channel) -> None: all_selected = getattr(sel, "all_selected", None) selected_ends = getattr(sel, "selected_ends", None) - buffered = set(self._sim_buffer.pending_ends()) # returned, carried grad - computing = set(self._sim_inflight_expected) - buffered # dispatched, not returned - busy = buffered | computing - self._sim_pending_commit |= busy + buffered = set(self._sim_buffer.pending_ends()) # returned, grad carried + # Outstanding = still in flight in virtual time (not yet committed). + outstanding = (set(self._sim_inflight_expected) | buffered) - self._sim_committed + self._sim_pending_commit |= outstanding # Prune the (model_version, data_id, iteration) triplet guard to the # still-outstanding set (a trainer whose grad committed is re-pickable). self._trainer_state_dict = { e: v for e, v in getattr(self, "_trainer_state_dict", {}).items() - if e in busy + if e in outstanding } + # recv_fifo marked every delivered end RECVD, which the channel would + # strip from selected_ends; reset the still-buffered (returned-but- + # uncommitted) ends to NONE so they KEEP their slot until they commit. + for eid in buffered: + if channel is not None and channel.has(eid): + channel._ends[eid].set_property(KEY_END_STATE, VAL_END_STATE_NONE) if isinstance(all_selected, dict): - # Release slot+guard for every trainer no longer busy (its grad - # committed this cycle) -> re-pickable next cycle. - for eid in [e for e in list(all_selected.keys()) if e not in busy]: + # Release slot + guard for every trainer no longer outstanding + # (its grad committed this cycle) -> re-pickable next cycle. + for eid in [e for e in list(all_selected.keys()) if e not in outstanding]: del all_selected[eid] if channel is not None and channel.has(eid): channel._ends[eid].set_property(KEY_END_STATE, VAL_END_STATE_NONE) if isinstance(selected_ends, dict) and requester in selected_ends: selected_ends[requester].discard(eid) - # Guard: block re-pick of every still-busy trainer (carried ∪ computing). - for eid in busy: + # Hold every still-outstanding trainer in the re-pick guard. + for eid in outstanding: if eid not in all_selected: all_selected[eid] = time.time() - # Compute slot: held ONLY by still-computing trainers. Carried (returned) - # trainers are removed from selected_ends so `extra` reopens their slot. + # Compute slot: held by EVERY outstanding trainer (computing OR returned- + # but-uncommitted) — both are in flight in virtual time until commit. if isinstance(selected_ends, dict) and requester in selected_ends: - for eid in computing: + for eid in outstanding: selected_ends[requester].add(eid) - for eid in buffered: - selected_ends[requester].discard(eid) def _aggregate_grads_async(self, tag: str) -> None: """ @@ -920,7 +941,20 @@ def _aggregate_grads_async(self, tag: str) -> None: logger.info("No channel found") return - if channel.ends(VAL_CH_STATE_RECV) is None: + recv_ends = channel.ends(VAL_CH_STATE_RECV) + # Sim: the sct reorder buffer -- NOT the transient channel RECV state -- + # is the source of truth for what is committable. _sim_recv_min_grad + # greedily drains ALL ready channel messages into _sim_buffer on its + # first call (which empties RECV), so gating loop re-entry on RECV strands + # the already-received grads in the buffer: 10 grads received, 1 popped, + # then "no ends yet" forever -> agg_goal never met from a full cohort + # (fluxtune deadlock, §K-D17). Keep draining while the buffer holds grads + # or in-flight trainers are still expected. Real path unchanged + # (principle #8: RECV gating is a real-transport artifact, no sim analog). + sim_has_pending = self.simulated and ( + len(self._sim_buffer) > 0 or bool(self._sim_inflight_expected) + ) + if recv_ends is None and not sim_has_pending: logger.info("no ends yet") return # time.sleep(0.1) # Slight delay to allow messages to arrive @@ -941,9 +975,7 @@ def _aggregate_grads_async(self, tag: str) -> None: # (msg, metadata) matching next(recv_fifo(...,1)) -- one grad per call, as # this loop expects. Real branch byte-identical to before. if self.simulated: - msg, metadata = self._sim_recv_min_grad( - channel, channel.ends(VAL_CH_STATE_RECV) - ) + msg, metadata = self._sim_recv_min_grad(channel, recv_ends or []) else: msg, metadata = next( channel.recv_fifo(channel.ends(VAL_CH_STATE_RECV), 1, @@ -1107,6 +1139,23 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): self._updates_in_queue += 1 self._per_agg_trainer_list.append(end) + # Re-pick guard (async_oort triplet filter): record the (model_version, + # data_id, iteration) this trainer just CONTRIBUTED at, so it is not + # re-selected until the agg version advances (the commit boundary prunes + # committed ends in _sim_hold_busy_slots). Stamped on RETURN, not + # dispatch (§K-D17): at dispatch the whole cohort matches _curr_agg_version + # and, since the version only advances at a commit, that froze the entire + # eligible pool before any commit could happen -> re-dispatch deadlock. + # The compute slot (selected_ends) already guards an in-flight trainer; + # the triplet only needs to cover the return->commit carry window. + # Residence-only so the sync baselines keep an empty map (inert). + if getattr(self, "simulated", False) and getattr( + self, "_sim_inflight_residence", False + ): + _agg_ver = getattr(self, "_curr_agg_version", None) + if _agg_ver is not None: + self._trainer_state_dict[end] = _agg_ver + if end not in self._updates_received.keys(): self._updates_received[end] = 1 else: @@ -2379,14 +2428,12 @@ def _distribute_weights_async( # running min) so the gate never laps a not-yet-committed trainer. _budget = self._sim_trainer_budget.get(end, self._sim_budget_min) self._sim_inflight_expected[end] = _round_now + _budget - # Record the (model_version, data_id, iteration) triplet this - # trainer is now outstanding at, so async_oort's triplet filter - # (the within-cycle re-pick guard) has real state to read and the - # R1 residence rung can prove no re-pick per triplet (§H D6). The - # boundary split prunes this to the still-busy set. Residence-only - # so the sync baselines stay byte-identical (empty map -> inert). - if getattr(self, "_sim_inflight_residence", False): - self._trainer_state_dict[end] = self._curr_agg_version + # NOTE: the async_oort re-pick triplet is stamped on grad RETURN + # (in _process_single_trainer_message), NOT here at dispatch -- + # stamping the whole cohort at the current _curr_agg_version froze + # the eligible pool before any commit could advance the version + # (re-dispatch deadlock, §K-D17). An in-flight-but-not-returned + # trainer is already guarded by its compute slot (selected_ends). channel.send(end, payload) logger.info( f"[Distribute] Done. Sent {_n_weights_sent} WEIGHTS + " diff --git a/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py b/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py new file mode 100644 index 000000000..8c36cff03 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py @@ -0,0 +1,205 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Regression for the 2026-07-04 fluxtune sim deadlock (simulate_fwdllm.md §K-D17). + +Two independent bugs conspired to freeze the fluxtune async sim after one cohort: + + Bug A -- drain gated on channel RECV state, not on the sct reorder buffer. + _aggregate_grads_async early-returned ("no ends yet") whenever the channel + reported no end in RECV. But _sim_recv_min_grad greedily drains ALL ready + channel messages into _sim_buffer on its first call (emptying RECV), so the + remaining already-received grads were stranded in the buffer and the loop + never popped them: 10 grads received, agg_goal=3 never met. The sim commit + path's readiness must key on _sim_buffer / _sim_inflight_expected, not on the + real transport's RECV bookkeeping (principle #8: real-transport artifact). + + Bug B -- the async_oort re-pick triplet was stamped at DISPATCH. + Stamping the whole cohort at the current _curr_agg_version made every + dispatched trainer match the aggregator's version; since the version only + advances at a commit boundary (never reached, see Bug A), async_oort's + filter excluded the entire pool (filtered_ends=0) -> no re-dispatch, ever. + The triplet is now stamped on grad RETURN, so an in-flight-but-not-returned + trainer stays eligible (its compute slot already guards it), and the pool is + never frozen before the first commit. +""" + +import torch + +from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator +from flame.mode.message import MessageType +from flame.sim.virtual_clock import SimReorderBuffer + + +# --------------------------------------------------------------------------- # +# Bug A -- the sim drain loop must not stall on empty RECV while the buffer is +# non-empty. +# --------------------------------------------------------------------------- # +class _GateChannel: + def __init__(self, recv_ends): + self._recv_ends = recv_ends + + def ends(self, state=None): + return self._recv_ends + + +class _CM: + def __init__(self, channel): + self._channel = channel + + def get_by_tag(self, tag): + return self._channel + + +class _DrainGateAgg: + """Binds the real _aggregate_grads_async and spies _sim_recv_min_grad so we + observe ONLY the entry-guard decision (drain vs. early-return).""" + + _aggregate_grads_async = TopAggregator._aggregate_grads_async + + def __init__(self, recv_ends, buffer_scts=(), inflight=(), simulated=True): + self.simulated = simulated + self._sim_buffer = SimReorderBuffer() + for i, s in enumerate(buffer_scts): + self._sim_buffer.add(f"b{i}", float(s), None) + self._sim_inflight_expected = {e: 1.0 for e in inflight} + self.cm = _CM(_GateChannel(recv_ends)) + self.drain_calls = [] + + def _sim_recv_min_grad(self, channel, recv_ends): + # Spy: record the call and return "nothing committable" so + # _aggregate_grads_async returns at `if not msg` without needing the + # heavy _process_single_trainer_message path. + self.drain_calls.append(list(recv_ends)) + return None, ("", 0) + + +class TestDrainGateNotBlockedByEmptyRecv: + def test_buffered_grads_drain_with_no_recv_ends(self): + """The core deadlock: RECV empty but the reorder buffer still holds grads + -> the loop MUST proceed to drain (call _sim_recv_min_grad), not stall.""" + agg = _DrainGateAgg(recv_ends=None, buffer_scts=(10.0, 20.0, 30.0)) + agg._aggregate_grads_async("param-channel") + assert agg.drain_calls == [[]], ( + "buffered grads must drain even when the channel reports no RECV ends" + ) + + def test_inflight_expected_keeps_loop_alive_with_no_recv_ends(self): + """A trainer still computing (in the in-flight gate) must also keep the + loop draining so the buffered minimum is not stranded behind it.""" + agg = _DrainGateAgg(recv_ends=None, inflight=("A", "B")) + agg._aggregate_grads_async("param-channel") + assert agg.drain_calls == [[]] + + def test_early_return_when_truly_idle(self): + """RECV empty AND buffer empty AND nothing in flight -> genuinely nothing + to do, so the loop still early-returns (no busy-spin on the drain).""" + agg = _DrainGateAgg(recv_ends=None) + agg._aggregate_grads_async("param-channel") + assert agg.drain_calls == [] + + def test_recv_ends_present_always_drains(self): + agg = _DrainGateAgg(recv_ends=["A"]) + agg._aggregate_grads_async("param-channel") + assert agg.drain_calls == [["A"]] + + def test_real_mode_ignores_sim_buffer(self): + """Real path must stay byte-identical: a populated _sim_buffer must NOT + make the real loop proceed when RECV is empty.""" + agg = _DrainGateAgg( + recv_ends=None, buffer_scts=(10.0,), simulated=False + ) + agg._aggregate_grads_async("param-channel") + assert agg.drain_calls == [] + + +# --------------------------------------------------------------------------- # +# Bug B -- the re-pick triplet is stamped on grad RETURN, not at dispatch. +# --------------------------------------------------------------------------- # +class _RepickChannel: + def __init__(self): + self._props = {} + self.provided_cleaned_up = [] + + class _Sel: + ordered_updates_recv_ends = [] + + self._selector = _Sel() + + def set_end_property(self, end, key, value): + self._props[(end, key)] = value + + def get_end_property(self, end, key): + return self._props.get((end, key)) + + def cleanup_provided_ends(self, end): + self.provided_cleaned_up.append(end) + + def cleanup_recvd_end(self, end): + pass + + +class _RepickAgg: + """Drives the real _process_single_trainer_message far enough to observe the + triplet-stamp side effect. aggregate_grads_from_trainers is stubbed (the + stamp lands before it) and telemetry stays disabled by default.""" + + process = TopAggregator._process_single_trainer_message + + def __init__(self, residence=True, simulated=True, curr_ver=(5, 2, 1)): + self.simulated = simulated + self._sim_inflight_residence = residence + self._curr_agg_version = curr_ver + self._trainer_state_dict = {} + self._per_agg_trainer_list = [] + self._agg_goal_cnt = 0 + self._round = 1 + self.data_id = 2 + self.iteration_per_data_id = 1 + self.is_async = True + self._model_version = 5 + self._sim_contrib_intervals = None + self._round_cache_activity_ts = {} + self._updates_in_queue = 0 + self._updates_received = {} + self._trainer_last_model_version = {} + self.grad_pool = [] + + def aggregate_grads_from_trainers(self, *a, **k): + pass # stub: the stamp executes before this call + + +def _valid_grad_msg(): + return { + MessageType.MODEL_VERSION: 5, + MessageType.GRADIENTS: torch.zeros(1), + MessageType.GRADIENTS_FOR_VAR_CHECK: torch.zeros(1), + MessageType.STAT_UTILITY: 1.0, + MessageType.DATASET_SIZE: 8, + } + + +class TestRepickTripletStampedOnReturn: + def test_returning_trainer_is_stamped_at_current_agg_version(self): + agg = _RepickAgg(residence=True, curr_ver=(5, 2, 1)) + ch = _RepickChannel() + + assert agg.process(ch, _valid_grad_msg(), "t1", timestamp=0) is True + # A RETURNED trainer records the triplet it just contributed at, so + # async_oort skips re-picking it until the agg version advances. + assert agg._trainer_state_dict == {"t1": (5, 2, 1)} + + def test_residence_off_leaves_triplet_map_empty(self): + """Sync baselines (residence off): empty map -> async_oort filter inert + -> byte-identical to Batch 1.""" + agg = _RepickAgg(residence=False) + ch = _RepickChannel() + + assert agg.process(ch, _valid_grad_msg(), "t1", timestamp=0) is True + assert agg._trainer_state_dict == {} + + def test_not_simulated_leaves_triplet_map_empty(self): + agg = _RepickAgg(residence=True, simulated=False) + ch = _RepickChannel() + + assert agg.process(ch, _valid_grad_msg(), "t1", timestamp=0) is True + assert agg._trainer_state_dict == {} diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py index 6b6d544e3..01f2b2af9 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py @@ -217,6 +217,12 @@ def __init__(self, ends): self._selector = _FakeSelector(ends) self._ends = {e: _FakeEnd() for e in ends} + def add_msg(self, end, sct, budget=None, release_at=0): + # Keep the real channel invariant has(e) <=> e in _ends (the slot-hold + # path does channel._ends[e] guarded only by channel.has(e)). + super().add_msg(end, sct, budget, release_at) + self._ends.setdefault(end, _FakeEnd()) + class TestAsyncBoundaryReleasesSlots: def test_async_boundary_frees_every_committed_slot(self): diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py index 12b753589..b2593ac3d 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -60,14 +60,13 @@ def test_surplus_carried_and_busy_held(self): assert agg._sim_inflight_expected == {"E": 50.0} # Per-cycle committed marks cleared so a re-contributor isn't skipped. assert agg._sim_committed == set() - # Option-A two-lifetime split (K-D16): every busy trainer (carried - # surplus C,D ∪ in-flight E) stays un-re-pickable in the all_selected - # GUARD; the two committed ones (A, B) are released. But only the - # still-COMPUTING trainer (E) holds a compute SLOT (selected_ends, which - # drives `extra`) -- the returned/carried C,D freed their slots so the - # next distribute refills concurrency with a different trainer. + # Felix-aligned virtual-time in-flight (K-D17b): every OUTSTANDING trainer + # (carried surplus C,D ∪ still-computing E) holds BOTH its re-pick guard + # (all_selected) AND its compute slot (selected_ends) until it commits -- + # a returned-but-uncommitted trainer is still in flight in virtual time. + # The two committed ones (A, B) are released. assert set(ch._selector.all_selected) == {"C", "D", "E"} - assert ch._selector.selected_ends["agg"] == {"E"} + assert ch._selector.selected_ends["agg"] == {"C", "D", "E"} def test_end_to_end_surplus_commits_next_cycle_no_refetch(self): agg = _residence_agg(residence=True) @@ -109,18 +108,20 @@ def test_no_same_trainer_reselected_while_in_flight(self): # only the committed A is released. assert "B" in held and "C" in held and "A" not in held assert "B" in agg._sim_pending_commit and "C" in agg._sim_pending_commit - # Two-lifetime split: returned/carried B freed its compute slot (frees - # `extra` -> a DIFFERENT trainer refills the pipeline), still-computing C - # keeps its slot. Neither can be re-picked (both in the all_selected guard). - assert ch._selector.selected_ends["agg"] == {"C"} + # Felix-aligned (K-D17b): both the carried B and still-computing C keep + # their compute slot (they're in flight in virtual time until commit); + # neither can be re-picked (both in the all_selected guard). + assert ch._selector.selected_ends["agg"] == {"B", "C"} -class TestOptionASlotGuardSplit: - """Option-A (§H D-e / D6 / K-D16): compute-slot occupancy (selected_ends, - drives `extra`) frees on RETURN; re-pick guard (all_selected) frees on - COMMIT. The two ledgers async_oort already tracks, no longer conflated.""" +class TestVirtualInflightSlotHold: + """K-D17b (felix-aligned, supersedes K-D16 Option-A): a returned-but- + uncommitted trainer is still in flight in VIRTUAL time (its grad commits + when the vclock reaches its sct), so it KEEPS its compute slot + (selected_ends, drives `extra`) until COMMIT, not on physical return. Both + ledgers track the same virtual-time in-flight set until commit.""" - def test_returned_trainer_frees_slot_but_stays_guarded(self): + def test_returned_trainer_keeps_slot_until_commit(self): agg = _residence_agg(residence=True) ch = _FakeSelChannel(["A", "B", "C", "D"]) # A,B still computing; C,D returned (buffered surplus, not yet consumed). @@ -130,11 +131,30 @@ def test_returned_trainer_frees_slot_but_stays_guarded(self): agg._release_sim_slots_at_agg_goal(ch, is_async=True) - # Slot ledger (extra): only the two still-computing trainers occupy it. - assert ch._selector.selected_ends["agg"] == {"A", "B"} - # Guard ledger: all four busy trainers un-re-pickable. + # Slot ledger (extra = c - len(selected_ends)): ALL four outstanding + # trainers occupy a slot -- carried C,D are still in flight in virtual + # time, so `in_flight` telemetry (= len(selected_ends)) counts them. + assert ch._selector.selected_ends["agg"] == {"A", "B", "C", "D"} + # Guard ledger: all four un-re-pickable until they commit. assert set(ch._selector.all_selected) == {"A", "B", "C", "D"} + def test_selected_ends_tracks_inflight_across_a_commit(self): + """The invariant that fixes the concurrency mis-measurement: after each + commit, selected_ends == the still-outstanding (dispatched-not-committed) + set. Committing one trainer releases exactly its slot; the rest stay.""" + agg = _residence_agg(residence=True) + ch = _FakeSelChannel([]) + for e, sct in zip(["A", "B", "C"], (10.0, 20.0, 30.0)): + ch.add_msg(e, sct) + agg._sim_inflight_expected = {"A": 10.0, "B": 20.0, "C": 30.0} + + # Commit the smallest-sct grad (A). _sim_recv_min_grad reasserts the hold. + scts = agg._drain(ch, ["A", "B", "C"], 1) + assert scts == [10.0] + # A committed -> slot released; B, C still in flight -> keep their slots. + assert ch._selector.selected_ends["agg"] == {"B", "C"} + assert "A" not in ch._selector.all_selected + def test_triplet_guard_pruned_to_busy_on_commit(self): agg = _residence_agg(residence=True) ch = _FakeSelChannel(["A", "B", "C"]) From 70c2fa180ac1685bd8962cebc8bd7f71699d55d1 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 13:00:50 -0400 Subject: [PATCH 16/56] =?UTF-8?q?fwdllm=20sim:=20land=20pre-next-run=20pla?= =?UTF-8?q?n=20(K-D21)=20=E2=80=94=20Stages=20A+B+C+D+E,=20pytest-green?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lands all instrumentation/checker/liveness dev so the Phase-1 sign-off run measures a near-complete ladder. Full tests/ 854 passed / 7 skipped. Also banks the prior uncommitted K-D19 (R1 guard-release-to-COMMIT) + piece A (vclock-budget stop). Stage C — fwdllm_plus real-mode sync-barrier liveness under scarcity: _await_dispatchable_under_scarcity sleeps-to-next-avail (WAIT/accumulate, cohort stays == agg_goal, parity-faithful) instead of hot-re-dispatching; self-terminates at max_runtime_s; one warn/stall. No-op at full availability (syn_0) and on the sim path. + test_fwdllm_sync_scarcity_wait.py Stage A — instrumentation (un-skips the 12 rigor-gap rungs): A1 trainer per-phase timing (_phase/_phase_times + pre/gpu/post/budget/phase) in trainer_round; manifest note updated. A2 agg per-round wall decomp (barrier_wait_s/drain_tail_s/aggregate_fedavg_s/ eval_s) in agg_round. A3 re-key advance rungs K3/K4/K3a/K3b to the data_id progress axis (shared parity engine; async_cifar10 byte-identical). A4 new wall_disparity DIAG rung (|real_wall - sim_vclock| per matched unit). Stage B — sct-model folds, config-gated OFF => byte-identical: B1 simModelEvalTime (charge measured eval wall to the vclock), B2 simStragglerSpreadS (stable per-trainer straggler offset, sim only), B3 simWanTransferS (documented knob, left 0). Emergent calibration = the run. Stage D — checker/telemetry gaps: D1 fwdllm_trainer re-emits task_recv.sim_send_ts (un-SKIPs K6), D2 failsafe_ok/K5 compares sim wall vs the RUN wall budget for a real-compute sim (auto-detected), SKIPs without a run budget. Stage E — gate the two _distribute_weights sleep(0.1) pads off the sim path. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/async_cifar10/PARITY.md | 7 + .../async_cifar10/scripts/parity/checks.py | 80 +++- .../async_cifar10/scripts/parity_checks.py | 1 + .../expt_scripts/fluxtune_n10_smoke.yaml | 2 +- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 2 +- .../fwdllm/expt_scripts/fwdllm_n10_smoke.yaml | 2 +- .../expt_scripts/fwdllm_n10_smoke_sim.yaml | 2 +- .../expt_scripts/fwdllm_plus_n10_smoke.yaml | 2 +- .../fwdllm_plus_n10_smoke_sim.yaml | 2 +- .../fwdllm/expt_scripts/run_sequential.sh | 11 +- lib/python/examples/fwdllm/simulate_fwdllm.md | 417 ++++++++++++++++-- .../examples/fwdllm/telemetry_manifest.yaml | 19 +- .../trainer/forward_training/FedSgdTrainer.py | 58 ++- lib/python/flame/config.py | 16 + .../horizontal/syncfl/fwdllm_aggregator.py | 185 +++++++- .../mode/horizontal/syncfl/fwdllm_trainer.py | 60 ++- .../tests/mode/test_fwdllm_agg_telemetry.py | 89 ++++ .../mode/test_fwdllm_early_stop_conditions.py | 45 ++ .../tests/mode/test_fwdllm_sct_model.py | 104 +++++ .../mode/test_fwdllm_sim_drain_and_repick.py | 1 + .../tests/mode/test_fwdllm_sim_grad_loop.py | 11 + .../mode/test_fwdllm_sim_grad_residence.py | 41 ++ .../mode/test_fwdllm_sync_scarcity_wait.py | 182 ++++++++ .../mode/test_fwdllm_trainer_phase_timing.py | 150 +++++++ .../mode/test_fwdllm_trainer_sim_duration.py | 8 +- .../mode/test_fwdllm_trainer_task_recv.py | 104 +++++ lib/python/tests/mode/test_parity_checks.py | 135 ++++++ 27 files changed, 1633 insertions(+), 103 deletions(-) create mode 100644 lib/python/tests/mode/test_fwdllm_sct_model.py create mode 100644 lib/python/tests/mode/test_fwdllm_sync_scarcity_wait.py create mode 100644 lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py create mode 100644 lib/python/tests/mode/test_fwdllm_trainer_task_recv.py diff --git a/lib/python/examples/async_cifar10/PARITY.md b/lib/python/examples/async_cifar10/PARITY.md index bb07ce2a7..cb86f51e9 100644 --- a/lib/python/examples/async_cifar10/PARITY.md +++ b/lib/python/examples/async_cifar10/PARITY.md @@ -44,6 +44,13 @@ Last green (Jun 24): selector+mode+sim+parity **351 pass / 7 skip** (incl. `Test A fix that could perturb another baseline → serialize (one baseline per run round). 4. **Shortest run that exhibits the issue** (table below). Reserve long runs for C1/C2. 5. **Crisp comments (≤1 sentence); context-free names** (Naming discipline below). +6. **Telemetry-FIRST; a cluster run is the LAST resort, never the debugger.** Before launching anything to + validate/refute a hypothesis, name the exact stored field/log-line that would confirm it and go read the + banked telemetry — most roots are already visible there. Ship every new mechanism WITH its telemetry + + plot + pytest in the same change (over-instrument: cheap to log, expensive to re-run for), so the next + root is catchable from disk. Launch ONLY to observe an emergent quantity no stored telemetry can yield + (fresh convergence / concurrency-after-a-change / a longer trajectory), then the shortest length that + shows it. (fwdllm restates this as its principle #11.) ### Run-length budget (state min duration up front, keyed here; never default to 3–4h) | validating | min run | why | diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 0f0e54816..22982e757 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -402,13 +402,20 @@ def _per_round_max_speed(agg_rounds: list) -> dict: def _per_round_advances(agg_rounds: list, use_vclock: bool) -> list: - """Compute per-FL-round time advances. + """Compute per-progress-unit time advances (Stage A3 re-key). - use_vclock=True: Δvclock_now between consecutive rounds (sim mode). - use_vclock=False: Δts (wall) between consecutive rounds (real mode). + use_vclock=True: Δvclock_now between consecutive units (sim mode). + use_vclock=False: Δts (wall) between consecutive units (real mode). Returns list of positive advances. + + Keyed on the run's TRUE progress axis (_progress_axis), not raw `round`: + fwdllm holds `round` static and advances committed `data_id`, so keying on + `round` yields <2 units and an empty list ("<2 sim rounds"). Auto-detect -> + async_cifar10 advances `round` -> _per_progress_last_event falls through to + _per_round_last_event -> byte-identical; only fwdllm re-keys to data_id. """ - by_round = _per_round_last_event(agg_rounds) + axis = _progress_axis(agg_rounds) + by_round = _per_progress_last_event(agg_rounds, axis) rounds_sorted = sorted(by_round.keys()) if len(rounds_sorted) < 2: return [] @@ -1803,13 +1810,25 @@ def sim_rate_ok(sim: dict, min_rate: float = 0.01, max_rate: float = 100.0) -> d def failsafe_ok(sim: dict, budget_s: Optional[float] = None, - max_overshoot: float = 0.20) -> dict: - """K5 [INV]: sim wall must not overshoot sim_wall_ceiling_s by > 20%.""" + max_overshoot: float = 0.20, + real_compute_sim: Optional[bool] = None) -> dict: + """K5 [INV]: sim wall must not overshoot the budget by > 20%. + + D2 (§H #9): for a REAL-COMPUTE sim (fwdllm runs the real forward-grad GPU + pass in sim mode), sim wall ≫ vclock BY CONSTRUCTION -- so the vclock is the + wrong budget to compare wall against (K5 would false-fail every fwdllm run, + 184 s wall vs 78 s vclock). Compare sim wall against the RUN WALL budget + (`max_runtime_s`, passed as budget_s) instead; if no run wall budget is + available, SKIP rather than falling back to the vclock. Auto-detected via the + progress axis when not passed explicitly. A cheap-compute sim (async_cifar10, + wall≈vclock) keeps the vclock fallback -> byte-identical.""" rounds = [e for e in sim["agg_rounds"] if e.get("event") == "agg_round"] all_evs = sim.get("_all_events", sim["agg_rounds"]) # agg_rounds used as proxy if len(rounds) < 2: return {"ok": True, "tier": "INV", "status": "SKIP", "note": "fewer than 2 agg_round events"} + if real_compute_sim is None: + real_compute_sim = _progress_axis(sim["agg_rounds"]) == "data_id" wall_elapsed = rounds[-1]["ts"] - rounds[0]["ts"] failsafe_fired = any( "SIM_WALL_CEILING" in str(e.get("stop_reason", "")) or @@ -1817,6 +1836,11 @@ def failsafe_ok(sim: dict, budget_s: Optional[float] = None, for e in all_evs ) if budget_s is None: + if real_compute_sim: + return {"ok": True, "tier": "INV", "status": "SKIP", + "note": "real-compute sim (sim wall ≫ vclock by construction); " + "no run wall budget to compare against — SKIP not " + "wall-vs-vclock (§H #9)"} vclock_final = rounds[-1].get("vclock_now") if not vclock_final: return {"ok": True, "tier": "INV", "status": "SKIP", @@ -1832,6 +1856,7 @@ def failsafe_ok(sim: dict, budget_s: Optional[float] = None, "overshoot_frac": round(overshoot, 3), "failsafe_fired": failsafe_fired, "max_overshoot": max_overshoot, + "real_compute_sim": real_compute_sim, } @@ -1943,6 +1968,47 @@ def per_round_advance_parity(real: dict, sim: dict, } +def wall_disparity(real: dict, sim: dict) -> dict: + """wall_disparity [DIAG]: |real_wall − sim_vclock| per matched progress unit + (Stage A4 / K-D20 #6). The recurring sanity metric to drive to ~0 -- surfaces + the gap between real's per-unit WALL and the sim's per-unit VCLOCK every run, + without gating (a persistent residual is root #6, the sct-model work, not a + ladder failure). Both clocks are measured cumulatively from the FIRST matched + unit (real: ts − ts0; sim: vclock_now − v0) so a run that does not start at + unit 0 still aligns. Keyed on the run's progress axis (data_id for fwdllm, + round otherwise). Never fails.""" + axis = "data_id" if "data_id" in (_progress_axis(sim["agg_rounds"]), + _progress_axis(real["agg_rounds"])) else "round" + real_by = _per_progress_last_event(real["agg_rounds"], axis) + sim_by = _per_progress_last_event(sim["agg_rounds"], axis) + matched = [k for k in sorted(set(real_by) & set(sim_by)) + if real_by[k].get("ts") is not None + and sim_by[k].get("vclock_now") is not None] + if len(matched) < 2: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "fewer than 2 matched units with both real ts and " + "sim vclock_now"} + ts0 = real_by[matched[0]]["ts"] + v0 = sim_by[matched[0]]["vclock_now"] + residuals, per_unit = [], {} + for k in matched: + real_wall = real_by[k]["ts"] - ts0 + sim_vclock = sim_by[k]["vclock_now"] - v0 + resid = abs(real_wall - sim_vclock) + residuals.append(resid) + per_unit[k] = round(resid, 2) + mean_resid = sum(residuals) / len(residuals) + return { + "ok": True, # DIAG: informational, never gates the ladder + "tier": "DIAG", + "axis": axis, + "mean_abs_disparity_s": round(mean_resid, 2), + "max_abs_disparity_s": round(max(residuals), 2), + "n_matched_units": len(residuals), + "per_unit_abs_disparity_s": per_unit, + } + + def overlap_factor(real: dict, sim: dict, tol: float = 0.3) -> dict: """K4 [DIAG]: async overlap factor diagnostic. @@ -4011,6 +4077,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["overlap_factor"] = overlap_factor(real_agg, sim_agg) results["per_round_advance"] = per_round_advance_parity(real_agg, sim_agg) results["throughput"] = throughput_parity(real_agg, sim_agg) + results["wall_disparity"] = wall_disparity(real_agg, sim_agg) # ── Stage 2 Availability ── results["avail_composition"] = avail_composition_parity(real_agg, sim_agg) @@ -4135,6 +4202,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "overlap_factor": {"stage": 1, "role": "DIAG", "deps": ("trainer_speed", "sim_commit_monotone")}, "per_round_advance": {"stage": 1, "role": "EMERGENT", "deps": ("overhead_residual",)}, "throughput": {"stage": 1, "role": "EMERGENT", "deps": ("per_round_advance",)}, + "wall_disparity": {"stage": 1, "role": "DIAG", "deps": ("throughput",)}, # ── Stage 2 Availability ── "avail_composition": {"stage": 2, "role": "MECHANISM", "deps": ()}, "eligibility": {"stage": 2, "role": "MECHANISM", "deps": ("avail_composition",)}, diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index 53297d216..b3015ca7f 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -78,6 +78,7 @@ throughput_parity, per_round_advance_parity, overlap_factor, + wall_disparity, total_commits_parity, terminal_state_parity, budget_not_cap, diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index e5c45cdd1..0f1039530 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -70,7 +70,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 9fc5831b0..828e9abc2 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -80,7 +80,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke.yaml index 348eba5de..07871f491 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke.yaml @@ -64,7 +64,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml index a8da9e792..4c93ad40f 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml @@ -74,7 +74,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml index 7767a406f..fcfa7143d 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml @@ -64,7 +64,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml index 9077b85f5..25372126b 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -70,7 +70,7 @@ experiments: # Smoke-test stop bar: whichever of these fires first ends the run # (`rounds` alone is much too coarse, since one round is 150 # data-id completions). - max_data_id_progress: 10 # stop once data_id reaches 10 + max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 9a7c9d77e..c1b337574 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -30,7 +30,11 @@ # --mode which time_mode variant(s) to run per baseline (default both). # --delays enable_training_delays for BOTH sides of a pair (default off=D=0). # --max-runtime-s wall/vclock cap per run (default 600 = 10 min). -# --max-data-id stop a run once data_id reaches this (default 10). +# --max-data-id stop a run once data_id reaches this (default 9999, i.e. +# effectively unbounded -> the run is governed by --max-runtime-s. +# Pass a small value (e.g. 10, 3) ONLY when you deliberately want +# a short data-id-capped run. Defaulting high avoids the trap of a +# "1h" run silently stopping early at a low data-id cap. # --num-trainers override trainer.num_trainers (default: each YAML's own, 10). # --num-gpus override execution.num_gpus (default: each YAML's own). # --c / --c-async / --k / --agg-goal / --min-initial-trainers @@ -80,7 +84,10 @@ expt_pin_pythonpath "$REPO_ROOT" MODE="both" DELAYS="off" MAX_RUNTIME_S=600 -MAX_DATA_ID=10 +# Default high so a run is governed by --max-runtime-s, NOT a silent low data-id cap. +# (An overnight "1h" run once stopped at data_id=10 because the default was 10; the +# real cost of a too-high default is zero since --max-runtime-s still bounds the run.) +MAX_DATA_ID=9999 # Companion "was this passed on the command line?" flags. Needed because MODE/ # DELAYS/MAX_RUNTIME_S/MAX_DATA_ID have non-empty defaults, so their value alone # can't tell "operator passed it (override -> green)" from "defaulted". (The diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ca57d93ee..ddb7e1f25 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -407,6 +407,29 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M do comment, be **crisp** -- one or two tight sentences, not a paragraph restating the code. The deeper rationale belongs in §K (append-only) with a one-line pointer from the code; the code comment states the decision, not its whole history. +11. **Telemetry-FIRST, then instrument, then (rarely) run — do not launch runs to guess-and-check a + hypothesis.** A cluster run is the single most expensive and slowest step in this loop; treat it as the + LAST resort, never the debugger. (Codifies async_cifar10/PARITY.md "Workflow policy" #1–#2 for fwdllm.) + - **(a) Validate/refute a hypothesis from the telemetry ALREADY ON DISK before considering a run.** Every + banked run dir carries per-commit/per-round/per-phase telemetry + the aggregator log; most roots + (the R1 overlap in `[SIM_GRAD_RECV]`, the #6 rate gap in the per-phase timings, the #7 real-speed surplus + in the `runtime.py` decorator lines) are already visible there. If you catch yourself saying "let's run it + and see," stop — first name the exact stored field/line that would confirm or kill the hypothesis, and go + read it. A run that only re-confirms what the logs already show is wasted. + - **(b) When you implement anything new, ship its telemetry + plot + pytest IN THE SAME CHANGE.** Over- + instrument pre-emptively (cheap to log, expensive to re-run for): add the per-event field that makes the + new mechanism's correctness DIRECTLY observable in the checker (not inferred), a plot if it has a + trajectory, and a pytest that drives it on synthetic telemetry and asserts the invariant. This is what + lets (a) work next time: the R1 regression was catchable only because the diagnostic line existed. A fix + with no new instrument/test is not done. + - **(c) A run is justified only to observe an EMERGENT quantity that no stored telemetry can yield** (fresh + convergence, a concurrency/wall number after a mechanism change, a cadence trajectory over more + `data_id`s than any banked run reached) — and then run the SHORTEST length that exhibits it + (PARITY.md run-length budget), smoke first, one mechanism per round. + - **(d) Cross-pollinate fluxtune ↔ fwdllm/fwdllm_plus.** fluxtune is furthest from parity, so it surfaces + roots first; when a fluxtune fix or learning could apply to the sync baselines (e.g. the #6/#1 shared + time-base question, a selector-ignores-hold-set bug, a duration-input anchor), note it in §B.1/§K and + check the others from THEIR banked telemetry before assuming it's async-only. --- @@ -443,31 +466,52 @@ so what we keep is (1) the current per-baseline state table, (2) the **open root ledger (what we tried or believed that was wrong, so we don't retry it). Per-run detail lives in the JSON + git history; the as-built rationale lives in §K.* -**Status (2026-07-04 EOD → K-D17 + K-D17b landed. in_flight measurement FIXED (sim 9.49 vs real 9.75) but -K-D17b introduced an R1 residence regression (0%→44.7%). STOPPED before the overnight run; resume from the -"NEXT SESSION" block below.)** +**Status (2026-07-05 → overnight 10-`data_id` grounding runs BANKED; parity read below (K-D18). The fluxtune +R1 residence regression (K-D17b) is STILL the top blocker — do it first (fluxtune ran real-only overnight, so +its sim is unchanged from run `020254`, R1=44.7%).** New findings from the 10-`data_id` runs (bigger than the +prior 3-`data_id` runs — the operator intended 1 h but the `--max-data-id` default was 10, now raised to 9999, +K-D18): +> - **(a) fwdllm sync cadence is HEALTHY at 10 `data_id`s.** V1/V2/g2/R1/W1/U3/S2/conv/conv_loss all PASS — +> the #4 short-run truncation cleared exactly as predicted (37 pass / 5 fail). +> - **(b) fwdllm's surviving fails are the clock-RATE family** (throughput/total_commits/terminal_state), NOT +> the #2 axis re-key: at a matched virtual budget sim charges **~7.8 vclock-s/round** while real spends +> **~34.9 wall-s/round** (both completed all 10 `data_id`s). New **open root #6** — must classify real's +> ~27 s/round surplus as real-transport overhead (correct to exclude, principle #1/#8) vs a sim D under-charge, +> FROM the existing per-phase timing telemetry, before touching the clock. **Shared-shaped with the fluxtune +> 2.4× wall gap** — the cross-baseline learning to watch. +> - **(c) fwdllm_plus real is pathologically slow.** It ran the FULL 1 h (`max_runtime_s=3600` cap) and reached +> only **`data_id`≈3** (86 wall-s/round; sim did all 10 in 81 vclock-s / 274 wall-s). So its V1/V2/g2/ +> eligibility/selection fails are REAL-side **truncation** (comparing sim's 10 full `data_id`s to real's 3), +> plus a genuine **real-speed root #7** (why is fwdllm_plus real ~2.5× slower/round than fwdllm real? — the +> per-iteration reselection is the suspect; localize from telemetry, not a re-run). +> - **(d) field_coverage now fails on a real telemetry GAP** — `task_recv.sim_send_ts` is absent in both modes +> (this is NOT the #3 gpu/budget alias, which passes). New **open root #8** (INV). `failsafe`/K5 also fires +> falsely for fwdllm (sim WALL ≫ vclock because forward-grad compute is REAL) — a checker exemption, root #9. +> +> _Original 2026-07-04 EOD state: K-D17 + K-D17b landed; in_flight measurement FIXED (sim 9.49 vs real 9.75) +> but K-D17b introduced the R1 residence regression (0%→44.7%). Resume from the "NEXT SESSION" block below._ > **NEXT SESSION (2026-07-05) pickup — start here.** -> **Where we are:** fluxtune sim is healthy (no deadlock, K-D17). K-D17b made `in_flight` correct (virtual -> in-flight ~9.5, matches real ~9.75) and fixed V2 — BUT it broke R1 residence (44.7% same-trainer in-flight -> overlap; run `020254`, report `parity_fluxtune_20260704_020254.json`). Sync baselines (fwdllm/fwdllm_plus) -> are untouched by all this and are safe to ground-run anytime. -> **The R1 bug (prime hypothesis, ~1 fix + 1 smoke to confirm):** `_sim_hold_busy_slots` -> (`fwdllm_aggregator.py`) resets buffered-but-uncommitted ends to `KEY_END_STATE=NONE` to preserve their -> `selected_ends` slot. But async_oort selects by avail/end-state and does NOT exclude `all_selected`, so a -> NONE-state buffered trainer is re-selectable → re-dispatched while in flight → R1 overlap. -> **Fix options to try (in order):** (1) **Drop the `KEY_END_STATE=NONE` reset** for buffered ends and rely -> only on the explicit `selected_ends[requester].add(eid)` to hold the slot — then re-run a fluxtune sim -> smoke and check BOTH `sel_ends`≈8 (measurement still good) AND R1≈0% (`grep SIM_GRAD_RECV`; run -> `run_parity.py --baselines fluxtune`). If the channel re-strips the slot without the reset (sel_ends drops -> back to ~3), then (2) make async_oort's selection **exclude `all_selected` members** (or keep buffered ends -> in RECV state, not NONE) so the guard actually blocks re-pick while the slot stays counted. (3) Failing -> that, keep buffered ends un-selectable via the K-D17 triplet (verify the triplet covers them — it may have -> advanced). **Validate:** R1 back to 0%, in_flight still ~9.5, `tests/mode` green, add a pytest that a -> buffered-uncommitted trainer is NOT re-selected. -> **Then:** re-check `selection_detail.chosen` (sim 0.83 vs real 1.76 — likely follows R1) and the -> throughput/commits/terminal fails; run the **longer run** (`--max-data-id` dropped → 10) to clear the #4 -> short-run truncation before judging V1/g2/throughput. Only after R1 is clean, launch the overnight -> grounding run: `bash run_sequential.sh --mode both --delays on --max-runtime-s 3600 --yes`. +> **UPDATE: #1c (R1) is now FIXED in code — K-D19.** The prime hypothesis below was **wrong on the mechanism** +> (async_oort *does* exclude `all_selected`; the NONE reset is a red herring). The real root was the guard +> being released on physical grad RETURN before the carried grad committed; fixed fwdllm-only via +> `_release_end_on_return` (defer release to COMMIT), + a `[SIM_R1_DISPATCH]` tripwire + a RETURN-path pytest, +> `tests/mode -k fwdllm` 110 green. **The only step left on #1c is the emergent smoke** (`bash +> run_sequential.sh --mode both --delays on --max-runtime-s 3600 --only fluxtune --yes`, or a shorter +> `--max-data-id 3` smoke first) → `run_parity.py --baselines fluxtune`, confirm R1≤2% + `in_flight`~9.5. +> _The original hypothesis + fix-options are kept below only as the record of what was refuted (K-D19)._ +> **[REFUTED] R1 prime hypothesis:** `_sim_hold_busy_slots` resets buffered ends to `KEY_END_STATE=NONE`… but +> async_oort selects by avail/end-state and does NOT exclude `all_selected`, so a NONE-state buffered trainer +> is re-selectable → R1 overlap. **Refuted:** async_oort excludes `all_selected` (`async_oort.py:1586-87`) and +> the channel passes the full pool regardless of `KEY_END_STATE`; exclusion is by `all_selected` membership +> alone, and the trainer left the guard on RETURN. See K-D19. +> **Then:** re-check `selection_detail.chosen` (sim 0.83 vs real 1.76 — likely follows R1). The **longer run +> is already banked** (the overnight 10-`data_id` runs, K-D18) and cleared #4 for fwdllm; for fluxtune the +> fresh sim re-run is still owed (overnight was real-only). When re-running, the `--max-data-id` default is now +> **9999** (K-D18) so `--max-runtime-s` governs; pass `--max-data-id 10` only for a deliberately short run. +> Launch (after R1 is clean): `bash run_sequential.sh --mode both --delays on --max-runtime-s 3600 --only +> fluxtune --yes`. **First-principle before any launch: exhaust the existing telemetry** (principle #11) — the +> R1 root is already visible in run `020254`'s `[SIM_GRAD_RECV]` log; validate the fix there + by pytest, not +> by a launch. > **Do NOT redo:** the in_flight measurement fix (K-D17b) is correct and validated — keep the hold-to-commit; > only the state-reset that re-opens selection is at fault. The temporary `[SIM_GRAD_RECV] ... inflight_exp=/ > sel_ends=` diagnostic in `_sim_recv_min_grad` is useful — keep it until R1 is closed, then optionally trim. @@ -487,24 +531,135 @@ mode-invariant. **Remaining action: a fresh fluxtune re-run** to bank concurrenc / V1 / selection, then the **longer run (`max_data_id_progress=10`)** to clear #4. The table below is the PRE-K-D16 baseline the re-run is measured against (the K-D16 run is void — it deadlocked). -**Per-baseline ground state (PRE-fix; re-run pending):** -| baseline | wall real→sim | R1 resid | U3 staleness | V1/V2 cadence | status | -|---|---|---|---|---|---| -| **fwdllm** (sync) | 107→**96s** ✓ | PASS (0%) | PASS | PASS / PASS | clock-family axis re-keyed (#2 fixed); re-run to confirm | -| **fwdllm_plus** (sync) | 335→**184s** ✓ | PASS (0%) | PASS | FAIL(trunc) / FAIL | #2 fixed; V1 trunc needs the longer run (#4); selection_detail after re-run | -| **fluxtune** (async) | 125→304s (pre) | **K-D17b REGRESSED: 44.7%** (was 0%) | PASS | V2 now PASS / — | K-D17→K-D17b: **in_flight FIXED (sim 9.49 vs real 9.75)** + V2 PASS, but **R1 residence broke (0%→44.7%)**; STOPPED, see NEXT SESSION block | +**Per-baseline ground state (2026-07-05, overnight 10-`data_id` runs; K-D18):** +| baseline | run (real→sim) | R1 | V1/V2/g2 cadence | S2/U3/conv | surviving fails | status | +|---|---|---|---|---|---|---| +| **fwdllm** (sync) | 10 `data_id`s both; wall 349→184s | PASS (0%) | **PASS / PASS / PASS** | PASS / PASS / PASS | throughput·commits·terminal (#6 clock-RATE), field_coverage (#8), failsafe (#9 false-pos) | **37 pass / 5 fail.** Cadence CLEAN at 10 `data_id`s (#4 cleared). Remaining fails all trace to the RATE gap #6 + two checker/telemetry gaps — no cadence bug | +| **fwdllm_plus** (sync) | sim 10 `data_id`s (274s wall); **real hit the 1 h cap at `data_id`≈3** | PASS (0%) | FAIL / FAIL / FAIL | PASS / PASS / PASS | +eligibility, selection_detail (real_chosen 4.88 vs sim 4.39) | **31 pass / 10 fail.** Cadence fails are **REAL-side truncation** (3 vs 10 `data_id`s) driven by real being ~2.5× slower/round (#7); NOT confirmed a sim bug until real completes 10 | +| **fluxtune** (async) | overnight **real-only** (10 `data_id`s); sim unchanged from `020254` | **FIXED in code (K-D19)**; was 44.7% | V2 PASS / — | — | (re-run owed) | **#1c FIXED + pytest-green.** Root = guard released on grad RETURN, not the doc's `all_selected` hypothesis; deferred to COMMIT via `_release_end_on_return`. in_flight measurement (9.49 vs 9.75) preserved. **Emergent-only step left:** one sim smoke to bank R1≤2% | -### Open roots (fix lowest-rung-first) +### Parity scoreboard — LATEST vs PENULTIMATE only (rewrite in place each run; do NOT append history) -**TOP OPEN ROOT: #1c — the R1 residence regression K-D17b introduced (see NEXT SESSION block above). Fix this -before anything else; do not launch the overnight run until R1 is back to ~0%.** Roots #1 (D-e), #1b (K-D17 -deadlock), #2 (clock-family axis), #3 (field_coverage) are FIXED in code (see *Fixes landed*). +*Discipline: keep exactly two columns per baseline — the latest parity JSON and the one before it — so drift is +visible without a growing log. `run_parity.py` regenerates both from the banked dirs.* -1c. **[OPEN — REGRESSION from K-D17b] R1 in-flight overlap 0% → 44.7% (fluxtune).** K-D17b's hold-to-commit - made `in_flight` correct (9.49 vs real 9.75) but re-dispatches a trainer while its prior grad is still in - flight. Root lead: buffered ends reset to `KEY_END_STATE=NONE` become re-selectable because async_oort - doesn't exclude `all_selected`. Full analysis + fix options in the NEXT SESSION block (top of §H) and - K-D17b (§K). Run `020254` / `parity_fluxtune_20260704_020254.json`. +| baseline | penultimate (run · pass/fail/skip) | LATEST (run · pass/fail/skip) | Δ | failing rungs (latest) | +|---|---|---|---|---| +| **fwdllm** | `0703_224641` (3 `data_id`) · 37/5/32 | `0704_022610` (10 `data_id`) · **37/5/32** | =0 | field_coverage, throughput, total_commits, terminal_state, failsafe | +| **fwdllm_plus** | `0703_225614` (3 `data_id`) · 30/11/32 | `0704_033206` (10 `data_id`, real stalled) · **31/10/32** | +1 | above + eligibility, selection_detail, v1, v2, g2_grad_pool_size | +| **fluxtune** | `0704_020254` (sim, R1=44.7%) · — | overnight **real-only → no sim pair** · — | — | (R1 fixed in code K-D19; re-run owed) | + +### SKIP audit — 32 skips, only ~16 are legitimate (rest are rigor gaps to close) + +The user rule: skips are OK only for a true **example/config disparity** (the mechanism genuinely isn't in this +baseline); otherwise add the equivalent fwdllm test. Breakdown of the 32: +- **~16 LEGIT (Phase-1 syn_0 + `random` selector; correct to skip):** 7 availability ground-truth rungs + (`*_trace_fidelity`, `agg_belief_fidelity_*`, `send_gate_wait_*`) + 4 delivery/withheld + (`duty_cycle*`, `abandon_timeout`, `withheld_delivery`, `commit_promptness`, `state_timeline_agreement`) — + the **availability effect path is Phase 2 (Batch 3)**, not built; + 3 DynamicKC (`dk1/dk2/dk3`) disabled by + design; + 2 oort-only (`selector_score`, `preferred_duration`) — fwdllm uses `random`; + `residence` (async + telemetry). No equivalent test exists because the mechanism isn't present. *(These un-skip as Phase 2 lands.)* +- **4 clock-advance rungs skip on the WRONG AXIS → RIGOR GAP (root #10):** `modeled_compute_advance` (K3a), + `overhead_residual` (K3b), `per_round_advance` (K3), `overlap_factor` (K4) report "fewer than 2 sim rounds" + even though the sim did 26 — they're still keyed on `round` (fwdllm stays at round=1, advancing `data_id`), + the exact bug #2 fixed for throughput but never applied to these. **Re-key to the `data_id` axis** (extend the + `_progress_axis` helper) → they run and directly validate root #6. *(shared parity engine → full-suite gate.)* +- **8 per-phase timing rungs skip on MISSING telemetry → RIGOR GAP = the root-#6 instrument (step 2):** + `training_budget_s`, `gpu_compute_s`, `mqtt_fetch_s`, `weights_to_gpu_s`, `weights_to_ram_s`, `pre_train_s`, + `post_train_s`, `trainer_phase`. Adding the per-phase wall telemetry un-skips them AND enables the #6 + decomposition. *(root #10 / step 2.)* + +### Open issues — cross-baseline master index (OPEN top, RESOLVED moved down; 1-sentence issue + fix) + +*Rewrite in place: when an issue resolves, move its row into the RESOLVED block with a crisp principled-fix +summary. Detailed rationale lives in the "Open roots" prose below + §K.* + +| # | issue (1 sentence) | baseline(s) | principled fix / next step | +|---|---|---|---| +| **#6** ⭐ | sim vclock under-models real wall — 3.0 vclock-s vs 13.4 wall-s/round. **Decomp (K-D20): only 8.7 s/round is GENUINE** (gap ~2.9×, not 4.5×); 4.7 s/round is harness artifact | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **PRIORITY.** Fold the two genuine unmodeled terms into `sct`: **`eval_s`** (+3.34 s/round, server eval) + **widen the barrier straggler spread** to real `trainer_speed_s` (+2.3 s/round). Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). Recurring sanity = `\|real_wall − sim_vclock\|/data_id`→~0. Piece A (vclock-budget stop) **landed**. | +| **#7** | fwdllm_plus real STALLED — 5 agg-rounds then dead-spun to the 1 h cap (log ballooned to 292 MB) | fwdllm_plus | **ROOT-CAUSED (K-D20):** oracular `mobiperf_2st` drops avail to 1/10 < `agg_goal=10`; real sync barrier can't assemble the cohort and spins (`wait_until_next_avl=False`). Genuine unavailability exposing a **real-mode liveness bug** (sim jumps its vclock past it → completes). Fix real-mode: relax goal to available-count OR honor `wait_until_next_avl`. Not a sim bug. | +| **#8** | `field_coverage` INV fail: `task_recv.sim_send_ts` absent both modes | fwdllm, fwdllm_plus | Emit `task_recv.sim_send_ts` (over-instrument) or add it to the coverage alias tuple; unblocks K6. | +| **#9** | `failsafe`/K5 false-positive: sim WALL ≫ vclock (real forward-grad compute) | fwdllm, fwdllm_plus | K5 should compare sim wall vs the RUN wall budget, not the vclock, for a real-compute sim (checker fix). Partly reframed by piece A's `sim_wall_ceiling_s`. | +| **#10** | 12 rungs SKIP as rigor gaps (4 advance rungs wrong-axis, 8 phase rungs missing telemetry) | fwdllm, fwdllm_plus | Re-key K3a/K3b/K3/K4 to `data_id` (shared engine → full suite); add per-phase timing telemetry (= step 2, feeds #6). | +| **#11** | real-mode critical-path waste surfaced by profiling: fixed `time.sleep(0.1)` pads (≥28.7 s/run, `fwdllm_aggregator.py:2209/2338`) + one-grad-per-poll-tick drain tail (2.48 s/round) | fwdllm, fwdllm_plus (real) | Correctness-preserving optimization (profile-driven): verify each sleep's purpose, gate/remove off the critical path; drain-tail is the real 1-per-pass clamp (leave the sim all-k barrier). Watch GPU-shared trainer compute. Do NOT change grad values / cadence. | +| — RESOLVED ↓ — | | | | +| ~~#1c~~ | R1 in-flight overlap 0%→44.7% (fluxtune) | fluxtune | **FIXED K-D19** — root was guard released on grad RETURN, not `all_selected`; deferred release to COMMIT (`_release_end_on_return`), fwdllm-only. Emergent smoke owed. | +| ~~#1~~ | async concurrency starved to ~1.5 vs real ~5.8 (D-e) | fluxtune | **FIXED K-D16** — Option-A two-lifetime split (free slot on return, hold guard to commit). | +| ~~#1b~~ | fluxtune K-D16 re-run deadlocked | fluxtune | **FIXED K-D17** — drain keys on `_sim_buffer` not RECV; triplet stamped at RETURN not dispatch. | +| ~~#2~~ | clock family collapsed to `sim_rounds=1` | all | **FIXED** — re-key throughput/commits/terminal to the `data_id` axis (`_progress_axis`). Exposed #6. | +| ~~#3~~ | field_coverage false-fail on gpu/budget field names | all | **FIXED** — coverage spec accepts fwdllm aliases (tuple of names). | +| ~~#4~~ | V1/V2/g2 short-run truncation | fwdllm (cleared), fwdllm_plus | **CLEARED for fwdllm** at 10 `data_id`s; for fwdllm_plus it's real-side truncation → folds into #7. | +| ~~#5~~ | fluxtune `selection_detail` in_flight (2.7 vs 9.75) | fluxtune | **MOSTLY FIXED K-D17b** — hold the compute slot to COMMIT so `len(selected_ends)` = virtual in-flight. | + +### Pre-next-run implementation plan (staged — the "definition of done" before spending another run) + +*Goal: land ALL dev that lets the NEXT parity run extract maximum information — un-skip the 12 rigor-gap rungs, +make #6/#7 first-class in telemetry, and close the sct model — so the run measures a nearly-complete ladder, +not a half-instrumented one. Rung math: 74 total; today **42 active / 32 skip**; after Stage A → **~54 active / +20 skip** (the remaining 20 skips are the legit Phase-2/config ones). Each task ships telemetry + pytest +(principle #11). Anchors in **K-D20**.* + +**PROGRESS (update markers in place as stages land):** +`✅ piece A` vclock-budget stop (landed, 17 tests) · `✅ K-D20` decomposition + anchors · +`✅ A1` trainer phase timing · `✅ A2` agg wall-decomp · `✅ A3` re-key advance rungs · `✅ A4` wall_disparity rung · +`✅ B1/B2` sct model (eval_s + straggler spread, config-gated) · `✅ B3` WAN knob (doc-only) · `✅ C` fwdllm_plus liveness · +`✅ D1/D2` task_recv.sim_send_ts + K5 real-compute exemption · `✅ E` sleep(0.1)-pad gating (#11). +**ALL landed & pytest-green (K-D21). NEXT: the Phase-1 sign-off RUN** — smoke → convergence per baseline; +it (a) exercises the 12 un-skipped rungs, (b) drives `wall_disparity`→~0 and calibrates B1/B2's `simModelEvalTime`/ +`simStragglerSpreadS`, (c) confirms fwdllm_plus completes 10 `data_id`s, (d) banks the fluxtune R1 emergent smoke. + +- **Stage A — instrumentation (un-skip 12 rungs; make #6/#7 measurable).** Depends on nothing. + - **A1** Trainer per-phase timing → un-skips the 8 phase rungs. Populate `_phase_times` in fwdllm's + `_fetch_weights` override (`fwdllm_trainer.py:168`) + emit `mqtt_fetch_s`/`weights_to_ram_s`/ + `weights_to_gpu_s`/`pre_train_s`/`gpu_compute_s`/`post_train_s`/`training_budget_s`/`trainer_phase` in + `FedSgdTrainer.train_with_data_id` extra (`:598-602`). Mirror `async_cifar10 main.py:784-908`. + manifest + pytest. + - **A2** Aggregator per-round wall decomposition → `barrier_wait_s` (dispatch→last grad), `drain_tail_s` + (last grad→commit, the artifact), `aggregate_fedavg_s`, `eval_s` (from the `eval_model` decorator) in + `agg_round`. Makes phases (b)–(e) telemetry, not log-grep. + pytest. + - **A3** Re-key the 4 advance rungs (K3a/K3b/K3/K4) to the `data_id` axis (extend `_progress_axis`) so they + stop reporting "<2 sim rounds." **Shared parity engine → FULL `tests/` + `examples/async_cifar10/scripts/ + parity` gate** (principle #9); assert async_cifar10 byte-identical (auto-detects `round`). + - **A4** New DIAG rung `wall_disparity` = `|real_wall − sim_vclock|` per matched `data_id` — the recurring + sanity metric to drive → ~0 (surfaced every run). + pytest. +- **Stage B — #6 `sct` model (piece B), reads A2's telemetry.** Fold the two GENUINE unmodeled terms into the + vclock; leave artifacts off (principle #1). + - **B1** `eval_s` per `data_id` onto the vclock (+3.34 s/round). + - **B2** Widen the sim barrier straggler completion-spread to match real `trainer_speed_s` dispersion + (2.66–4.05 s), so the k-th-smallest `sct` reflects real (+~2.3 s/round). + pytest. + - **B3** WAN payload-transfer term (1.96 MB up / 3.45 MB down) — NOT measurable on localhost; add a documented + config knob, **do not enable this run** (no ground truth). **Exit: `wall_disparity/data_id` → residual ≈ the + 4.7 s/round artifact, and throughput/K2/K3b within tolerance.** +- **Stage C — #7 fwdllm_plus real liveness** (so its 10-`data_id` run actually completes). Real-mode sync barrier + under scarcity: relax the agg-goal to available-count OR honor `wait_until_next_avl` (sleep-to-next-avail) + instead of spin-to-cap; bound the log. + pytest. Independent of A/B. +- **Stage D — cheap checker/telemetry gaps.** **D1** #8 emit `task_recv.sim_send_ts` (or alias). **D2** #9 + `failsafe`/K5 compares sim wall vs the RUN wall budget (not vclock) for a real-compute sim. +- **Stage E — optimization pass (#11; correctness-preserving, opportunistic).** Audit the `time.sleep(0.1)` pads + (`fwdllm_aggregator.py:2209/2338`, ≥28.7 s/run) — verify each one's purpose, gate/remove off the critical + path; profile GPU-shared trainer compute (A1's `gpu_compute_s` per trainer) for waste. **Re-run pytest to + prove grad values + cadence unchanged** (never trade correctness for wall). + +**Definition of done:** A + B + C + D pytest-green; **full `tests/` + shared parity suite green** (A3 spans the +stack); E opportunistic. THEN one run per baseline banks: 12 un-skipped rungs live, `wall_disparity`→~0, +fwdllm_plus completing 10 `data_id`s, and the fluxtune R1 emergent smoke. + +### Open roots (fix lowest-rung-first) + +**TOP: #1c (R1 residence regression) is now FIXED in code (K-D19) + pytest-green; the only thing owed on it is +ONE fluxtune sim smoke to bank the emergent R1/in_flight numbers — the LAST step, not the debugger (principle +#11c).** Roots #1 (D-e), #1b (K-D17 deadlock), #3 (field_coverage alias) FIXED; **#2 (clock-family AXIS) fixed +but exposed the RATE gap #6.** **Ranking of what's OPEN:** bank the #1c re-run → then the sync-baseline work +#6 (clock-RATE, shared with fluxtune #1) → #7 (fwdllm_plus real speed, gates #4) → #8/#9 (telemetry/checker +gaps). Attack #6 and #1's wall gap together — they are the same time-base question on the async vs sync path. + +1c. **[FIXED in code — K-D19; emergent re-run pending] R1 in-flight overlap 0% → 44.7% (fluxtune).** Root was + NOT the doc's hypothesis (async_oort *does* exclude `all_selected`) but the guard being released on physical + grad **RETURN** (`_process_single_trainer_message` → `cleanup_provided_ends`) while the carried grad hadn't + committed in virtual time. Fixed fwdllm-only via `_release_end_on_return` (defers the async+sim+residence + release to COMMIT, felix-aligned); + a `[SIM_R1_DISPATCH]` tripwire + `TestReturnPathGuardHeldToCommit` + (the RETURN-path blind spot). `tests/mode -k fwdllm` 110 green. Full rationale **K-D19**. **Remaining:** one + fluxtune sim smoke to confirm R1≤2% + `in_flight`~9.5 (emergent — measured, not assertable from banked + telemetry; the `020254` raw dir was cleaned). 1. **[FIXED — Option A / K-D16] D-e async concurrency starvation (fluxtune).** *Symptom (pre-fix):* sim kept ~1.5 trainers computing vs real's ~5.8, did fewer forward passes (89 vs 109) yet took **2.4× the wall** @@ -526,20 +681,24 @@ deadlock), #2 (clock-family axis), #3 (field_coverage) are FIXED in code (see *F RETURN (not dispatch). Full `tests/mode` green (314/7). Details **K-D17**. **Remaining: the fresh fluxtune re-run** confirms recovery (concurrency, wall, U3, V1, selection) — emergent, measured not asserted. -2. **[FIXED — #2] Clock family re-keyed to the `data_id` progress axis.** `throughput_parity` / - `total_commits_parity` / `terminal_state_parity` now count units on the axis the run advances (a unified - `_progress_axis` / `_per_progress_last_event` helper): `round` for normal FL, committed `data_id` for - fwdllm (was collapsing to `sim_rounds=1`). async_cifar10 auto-detects `round` → byte-identical. Checker - fix, validates instantly on the banked dirs. +2. **[AXIS FIXED — #2; but exposed the RATE gap #6] Clock family re-keyed to the `data_id` progress axis.** + `throughput_parity` / `total_commits_parity` / `terminal_state_parity` now count units on the axis the run + advances (a unified `_progress_axis` / `_per_progress_last_event` helper): `round` for normal FL, committed + `data_id` for fwdllm (was collapsing to `sim_rounds=1`). async_cifar10 auto-detects `round` → byte-identical. + The re-key was correct, but with the axis fixed the 10-`data_id` run shows these rungs **still FAIL on a + genuine per-round TIME-BASE gap** (sim vclock-s/round ≪ real wall-s/round) — that residual is **new root #6**, + not this checker fix. 3. **[FIXED — #3] field_coverage accepts fwdllm field aliases.** The coverage spec rows for `gpu_compute_s` / `training_budget_s` now accept the fwdllm spellings `real_gpu_time_s` / `sim_round_duration_s` (a spec field may be a tuple of accepted names). No more false-FAIL / benign SKIP. -4. **[OPEN — operational] V1/V2/V5 short-run truncation (fwdllm_plus + fluxtune).** Per-data_id series match - on every FULL data_id and diverge only on the last, truncated one; with 3 data_ids one boundary diff → KS - 0.33. **Fix = the longer run** (`max_data_id_progress=10`, already the yaml default — do NOT pass - `--max-data-id 3`), not code. For fluxtune, partly downstream of the now-fixed D-e. +4. **[CLEARED for fwdllm; now REAL-side for fwdllm_plus] V1/V2/V5/g2 short-run truncation.** Per-`data_id` + series match on every FULL `data_id` and diverge only on the last, truncated one. **fwdllm: CLEARED** — at + 10 `data_id`s V1/V2/g2 all PASS (K-D18). **fwdllm_plus: now the truncation is on the REAL side** — sim did + 10 `data_id`s but real reached only ≈3 (1 h cap, root #7), so KS blows up comparing 10 vs 3. Fix is to make + real reach 10 (i.e. resolve #7), NOT code and NOT a knob. The `--max-data-id` default is now 9999 (K-D18) so + `--max-runtime-s` governs — but for fwdllm_plus even 1 h wasn't enough, so #7 must be understood first. 5. **[MOSTLY FIXED — K-D17b] fluxtune selection_detail (in_flight).** Was NOT an oort-fidelity gap: the `in_flight` metric = `len(selected_ends)`, and K-D16 freed `selected_ends` on physical RETURN, so it @@ -547,8 +706,48 @@ deadlock), #2 (clock-family axis), #3 (field_coverage) are FIXED in code (see *F K-D17b holds the slot to COMMIT (felix-aligned) so `len(selected_ends)` = virtual in-flight. Residual ~7.8 vs 9.75 (~20%) + preferred_duration to re-validate on the fresh re-run. +6. **[OPEN — sync clock-RATE gap; new] throughput/commits/terminal: sim charges far less time/round than real.** + At 10 `data_id`s (both sides complete), fwdllm sim advances **~7.8 vclock-s/round** vs real **~34.9 + wall-s/round** (rel 0.78); fwdllm_plus **~8.1** vs **~86** (but real truncated, #7 confounds). Do NOT read + this as a cadence bug (V1/V2/g2 PASS — the WORK per `data_id` matches). It is a **time-base** question: + real's per-round WALL bundles MQTT/dispatch/aggregation/eval overhead that the sim (correctly, principle #1) + does NOT put on the vclock (`vclock = max(vclock, sct)`), and real emits **no `vclock_now`** so the checker + uses real WALL as real's axis. **First move (telemetry, not a run):** decompose real's per-round wall into + D + per-phase transport overhead using the existing `trainer_phase` / `mqtt_fetch_s` / per-phase splits, and + compare against the sim `sct` model. Two outcomes: (i) the surplus is real-transport overhead → the rung is + comparing WALL-vs-VCLOCK and should anchor real on an intrinsic per-round span (like async_cifar10's + `WALL_SEND−WALL_RECV` duration lesson), a checker fix; (ii) part is modeled D the sim under-charges → a sim + `sct` fix. **Shared-shaped with the fluxtune 2.4× wall gap (#1)** — resolve the classification once, apply to + both. Dead-end guard: do NOT add a scalar overhead to the vclock (principle #1, async_cifar10 dead-ends). + +7. **[FIXED in code — Stage C / K-D21; emergent re-run pending] fwdllm_plus real spun to the cap under + `mobiperf_2st` scarcity (agg_goal=10 > avail).** ROOT was the real sync barrier hot-re-dispatching when the + eligible pool < `agg_goal` (292 MB log, reached `data_id`≈3). Fixed: `_await_dispatchable_under_scarcity` + sleep-to-next-avail (WAIT/accumulate — cohort stays == agg_goal, parity-faithful vs the sim vclock-jump; + operator-chosen over relax-to-available). Self-terminates at `max_runtime_s`; one warn per stall. **Remaining: + the sign-off run confirms fwdllm_plus now completes 10 `data_id`s** (emergent). The separate question of WHY + fwdllm_plus real is slower/round (per-iteration reselection + oracular read) is a real-config profiling item, + localizable from the banked per-phase log — now that the run can complete, it is judgeable. + +8. **[FIXED — Stage D1 / K-D21] field_coverage INV `task_recv.sim_send_ts` absent.** `fwdllm_trainer._fetch_ + weights` re-emits `task_recv` carrying `sim_send_ts` (the override had dropped the base trainer's emission); + null in real, non-null in sim. Un-SKIPs K6. `test_fwdllm_trainer_task_recv.py`. + +9. **[FIXED — Stage D2 / K-D21] `failsafe`/K5 false-positive for fwdllm.** K5 flags sim WALL overshooting + the virtual budget by >20% (184s wall vs 78s vclock). For fwdllm this is EXPECTED and benign: the sim does + **real forward-grad GPU compute** (many passes), so sim wall ≫ vclock by construction — unlike async_cifar10 + where compute is cheap. FIXED: `failsafe_ok` now compares sim wall against the RUN wall budget (`max_runtime_s`) + for a real-compute sim (auto-detected via the progress axis) and SKIPs when no run budget is available, instead + of the wall-vs-vclock fallback. `TestFailsafeRealComputeSim`. + ### Fixes landed (what worked — do not redo) +- **Pre-next-run plan A+B+C+D+E (K-D21)** — Stage C scarcity-wait liveness; A1 trainer phase timing; A2 agg + wall-decomp; A3 advance-rung re-key (shared engine, async byte-identical); A4 `wall_disparity` DIAG rung; B1/B2/ + B3 sct-model folds (config-gated OFF ⇒ byte-identical); D1 `task_recv.sim_send_ts`; D2 K5 real-compute exemption; + E `sleep(0.1)`-pad gating. Full `tests/` 854 passed / 7 skipped. Emergent calibration (wall_disparity→~0, B1/B2 + values, fwdllm_plus 10-`data_id` completion, fluxtune R1) is the sign-off run's job. + - **Trainer sim clock** — additive `sct = send + gpu + D`, no-sleep on the sim path (Batch 1, K-D2/K-D3). - **Async grad loop on the vclock** — purpose-built `_sim_recv_min_grad` sct reorder buffer + in-flight gate + agg-goal-boundary rollback cleanup (Batch 1, K-D4/K-D5). @@ -575,6 +774,20 @@ deadlock), #2 (clock-family axis), #3 (field_coverage) are FIXED in code (see *F (`asyncfl::_sim_hold_busy_slots`/`_sim_recv_min`). Reverts K-D16's slot-on-return; safe now that K-D17 fixed the drain. Full `tests/mode` green (315/7); `TestVirtualInflightSlotHold`. Concurrency/downstream recovery measured by the fresh fluxtune re-run. +- **K-D19: R1 regression fixed — guard release deferred from RETURN to COMMIT (`_release_end_on_return`).** + Root was NOT the recorded `all_selected` hypothesis (async_oort *does* exclude it) but + `_process_single_trainer_message` freeing the re-pick guard on physical grad RETURN while the carried grad + was still uncommitted in virtual time. Fixed fwdllm-only (no shared `async_oort`/`asyncfl` edit — principle + #8); the async+sim+residence release now happens at COMMIT via `_sim_hold_busy_slots`, felix-aligned. Shipped + WITH a `[SIM_R1_DISPATCH]` dispatch-time tripwire + `TestReturnPathGuardHeldToCommit` (closes the RETURN-path + test blind spot). `tests/mode -k fwdllm` 110 green. Emergent R1/in_flight numbers pending one fluxtune smoke. +- **Piece A: vclock-budget stop (matched-budget convergence axis; step 1 toward #6).** `_check_early_stop_ + conditions` now interprets `max_runtime_s` as **wall in real / `vclock.now` in sim** (mirrors base + `max_experiment_runtime_s`, `top_aggregator.py:1140`), so one budget = "3600 wall-s of real work" AND "3600 + virtual-s of modeled work". A `sim_wall_ceiling_s` failsafe (default = budget) stops a runaway when the vclock + is under-modeled (root #6) — that ceiling firing IS the #6 signal until piece B lands. `tests/mode/ + test_fwdllm_early_stop_conditions.py` 17 green (4 new sim/vclock cases). **Does NOT alone fix #6** — the + vclock must first model real wall (step 2) or a vclock-bounded sim over-runs to the wall ceiling. - **Clock-family re-key to `data_id` (#2)** — unified `_progress_axis`/`_per_progress_last_event` helper; `throughput`/`total_commits`/`terminal_state` now count on the axis the run advances (fwdllm `data_id`, normal FL `round`). async_cifar10 byte-identical (auto-detects `round`). `TestProgressAxisRekey`. @@ -1005,6 +1218,114 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent **STOPPED before the overnight run** — R1 is a correctness regression; banking 6h of runs on it would be wrong. Resume steps are in **§H -> "NEXT SESSION (2026-07-05) pickup"**. +- **K-D18 Overnight 10-`data_id` grounding runs + the `--max-data-id` default raised to 9999.** The operator + launched `run_sequential.sh --mode both --max-runtime-s 3600 --only fwdllm,fwdllm_plus` (then fluxtune + real-only), intending 1 h runs, but `run_sequential.sh`'s `--max-data-id` **defaulted to 10** and its + `patch()` always overrides the yaml, so every run stopped at `data_id=10` instead of at 1 h. This was a + useful accident — 10 `data_id`s is >3× the prior banked runs and cleared the #4 truncation for fwdllm — but + the trap is real, so the default is now **9999** (effectively unbounded; `--max-runtime-s` governs) in both + `run_sequential.sh` and the six `expt_scripts/*_n10_smoke*.yaml`; pass `--max-data-id 10` (or 3) for a + deliberately short run. **Parity read** (`run_parity.py --baselines fwdllm fwdllm_plus`, JSONs + `parity_fwdllm_20260704_022610` / `parity_fwdllm_plus_20260704_033206`): *fwdllm* 37 pass / 5 fail — cadence + (V1/V2/g2/R1/W1/U3/S2/conv) CLEAN, surviving fails = the clock-RATE family (open root #6), field_coverage + (#8), failsafe (#9); *fwdllm_plus* 31 pass / 10 fail — but its **real run hit the 1 h cap at `data_id`≈3** + (86 wall-s/round; sim did 10 in 81 vclock-s), so the cadence/eligibility/selection fails are REAL-side + truncation (#4) driven by real-speed root #7, NOT a confirmed sim bug. *fluxtune* ran real-only (no sim), so + no parity — its sim is unchanged from `020254` (R1=44.7%, #1c). **Discipline applied (principle #11):** all of + the above was read from the ALREADY-banked telemetry + logs (stop-reason lines, per-round `runtime.py` + timings, the parity JSONs); no new run was launched to reach these conclusions. + +- **K-D19 The R1 residence regression (#1c) — root-caused to guard-release-on-RETURN, fixed fwdllm-only + + pytest + tripwire. (Corrects the K-D17b/§H "NEXT SESSION" hypothesis.)** The hypothesis on record was + "`_sim_hold_busy_slots` resets buffered ends to `KEY_END_STATE=NONE` and async_oort doesn't exclude + `all_selected`, so they're re-selectable." **That was wrong on the mechanism:** async_oort *does* exclude + `all_selected` (`async_oort.py:1586-1587`) and the channel passes the full pool regardless of `KEY_END_STATE` + (`channel.py:221-240`) — so the NONE reset is a red herring for re-selection; exclusion depends solely on + `all_selected` membership. **Real root:** on the async accept path, `_process_single_trainer_message` called + `channel.cleanup_provided_ends(end)` on **physical grad RETURN, per-message, mid-cycle** + (`fwdllm_aggregator.py:1259-60` pre-fix) → `async_oort._cleanup_provided_ends` (:889-916) deletes the trainer + from `all_selected`. But in sim residence the grad is **carried** (its `sct` is in the future), so the trainer + is torn out of the re-pick guard while still in flight in VIRTUAL time → any `_distribute_weights_async` + before its commit re-dispatches it → R1 overlap 0%→44.7%. K-D17b's per-**commit** `_sim_hold_busy_slots` + re-asserts the guard, but too late — it can't cover the RETURN→COMMIT window. felix (`asyncfl:1316`) releases + the guard only at the agg-goal boundary, never per-return — the divergence. *This is the D6 "carry was right, + release TIMING was wrong" pattern applied to the guard (`all_selected`), not just the compute slot.* **Fix + (fwdllm-only, principle #8 clean — no shared `async_oort`/`asyncfl` edit):** extracted the release into + `_release_end_on_return`, which in async+sim+residence **defers the release to COMMIT** (owned by + `_sim_hold_busy_slots`); real mode / non-residence / sync unchanged (return≈commit → byte-identical). + **Over-instrument (principle #11):** a dispatch-time `[SIM_R1_DISPATCH]` tripwire warns the instant a still- + outstanding trainer is re-dispatched (violation visible live, not reconstructed offline). **Pytest:** + `TestReturnPathGuardHeldToCommit` in `test_fwdllm_sim_grad_residence.py` drives the RETURN path directly — + the blind spot that let R1=44.7% ship with `tests/mode` green (the prior residence tests only poked + `_sim_hold_busy_slots`/`_release_sim_slots_at_agg_goal`, never `_release_end_on_return`). `tests/mode -k + fwdllm` 110 green. **Remaining = emergent-only validation** (principle #11c): one fluxtune sim smoke to + confirm R1≤2% with `in_flight` still ~9.5 — measured, not assertable from banked telemetry (the `020254` raw + dir was cleaned). + +- **K-D20 Real per-round wall decomposition — grounds #6 (`sct` model) and root-causes #7. (telemetry-only, + no run.)** For fwdllm sync the real aggregator spends **13.4 s/agg-round**, the sim vclock **3.0 s** — but the + gap is NOT all a modeling deficit. Barrier-anchored decomposition (dispatch → 1st grad → 10th grad → commit → + eval, from `runtime.py` decorator lines + `RECV_FIFO` markers + `agg_round`/`agg_eval` telemetry): **(a)** + trainer compute + round-trip 3.96 s (gpu 1.05 s; `sim_round_duration_s` 2.19 s already modeled), **(b)** + straggler/sync-barrier wait 3.06 s (GENUINE), **(c)** aggregator drain tail 2.48 s (**ARTIFACT** — real clamps + `num_min_req=min(agg_goal,1)` and acks one grad per poll-tick; the sim's all-k barrier is the correct model), + **(d)** `eval_model` 8.68 s/`data_id` = 3.34 s/round amortized (GENUINE server eval, the largest unmodeled + term), **(e)** FedAvg + redispatch ~0.6 s (mostly the `time.sleep(0.1)` pads, ARTIFACT). **Net: ~8.7 s/round + genuine vs ~4.7 s/round artifact → the true sct gap is ~2.9×, not 4.5×.** So piece B folds ONLY `eval_s` + (+3.34) + a widened straggler spread (+~2.3) into the vclock; the drain tail, the ≥28.7 s of `sleep(0.1)` + pads, and localhost-MQTT/GIL stay OFF the clock (principle #1) — and become optimization targets (#11). **#7 + root cause:** fwdllm_plus real spun to the 1 h cap (log → 292 MB) because oracular `mobiperf_2st` drops avail + to 1/10 < `agg_goal=10` and the real sync barrier can't assemble the cohort (`wait_until_next_avl=False`); the + SIM completes the same trace (vclock jumps past the unavailability window), proving it's a **real-mode liveness + bug**, not a sim/model bug. Full anchors (trainer `_phase_times` already exist at `syncfl/trainer.py:143-150` + but fwdllm's `_fetch_weights` override drops them; per-field wrap points; the `async_cifar10 main.py:784-908` + reference) are in the Stage-A/A1 plan above. Feeds the entire "Pre-next-run implementation plan." + +- **K-D21 Pre-next-run implementation plan LANDED — Stages A + B + C + D + E (pytest-gated, config-gated; + emergent calibration deferred to the sign-off run).** All dev that lets the next run measure a nearly-complete + ladder is in code + pytest-green; nothing here needed a run (principle #11a/b — telemetry-first). + - **Stage C (fwdllm_plus liveness, decision: WAIT/accumulate — parity-faithful, operator-chosen over relax):** + `_await_dispatchable_under_scarcity` (real mode) sleep-to-next-avail instead of hot-re-dispatching when the + eligible pool < `agg_goal` — KEEPS cohort == `agg_goal` (matches the sim's vclock-jump, so cohort size stays + identical real↔sim; relaxing real's cohort would desync cadence). Self-terminates via + `_check_early_stop_conditions` at `max_runtime_s`; one warn per stall (bounds the 292 MB log). Byte-identical + when availability tracking is off / on the sim path. `test_fwdllm_sync_scarcity_wait.py` (8). *Note:* §C's + fwdllm_plus `agg_goal` row read 2; the landed yaml is **10** (all clients) — the yaml is source of truth. + - **Stage A1 (trainer per-phase timing):** `_phase`/`_phase_times` added to `fwdllm_trainer` (+ `mqtt_fetch_s`/ + `weights_to_{ram,gpu}_s` wrapped in `_fetch_weights`); `FedSGDTrainer.train_with_data_id` drains those + + `pre_train_s`/`gpu_compute_s`/`post_train_s`/`training_budget_s`/`trainer_phase` into `trainer_round`. Un-skips + the 8 phase rungs. Manifest note updated (the phase fields ARE emitted for the #6 decomposition; the DERIVED + budget-slack/overrun plots stay not_populated — fwdllm's delay is a flat additive sleep). Reversed the Part-6 + "don't emit training_budget_s" note for the decomposition (not for budget-slack). + - **Stage A2 (agg per-round wall decomposition):** `barrier_wait_s` (dispatch→last grad), `drain_tail_s` (last + grad→commit, the K-D20(c) ARTIFACT), `aggregate_fedavg_s`, `eval_s` in `agg_round` — via a dispatch wall stamp + (`_round_dispatch_wall_ts`) + a last-grad stamp (`_last_grad_wall_ts`) + timers around `aggregate()`/ + `eval_model()`. Null-safe (rung SKIPs, no crash). `TestPerRoundWallDecomposition` (4). + - **Stage A3 (advance-rung re-key — SHARED engine):** `_per_round_advances` (feeds K3/K4/K3a/K3b) now keys on + `_progress_axis` via `_per_progress_last_event` — fwdllm advances on `data_id` (was <2 `round`s → SKIP); + async_cifar10 (round-advancing) falls through to `_per_round_last_event` → byte-identical. Full `tests/` + + `examples/async_cifar10/scripts/parity` re-run green. + - **Stage A4 (`wall_disparity` DIAG rung):** new `|real_wall − sim_vclock|` per matched progress unit + (cumulative-from-first-matched so a non-zero-start run still aligns); registered in `run_all_parity`/ + `CHECK_META`/shim; never gates. The recurring #6 sanity metric → drive to ~0. `TestWallDisparity` (5). + - **Stage B (sct-model folds — CONFIG-GATED, flag-off ⇒ byte-identical; emergent calibration = the run):** + **B1** `simModelEvalTime` charges the MEASURED server-eval wall to the vclock after each committed data_id + (consistent with the existing real-gpu-on-vclock model; the largest genuine unmodeled term). **B2** + `simStragglerSpreadS` adds a STABLE per-trainer (crc32) offset in [0, spread) to the modeled delay in SIM + only, widening the barrier's k-th sct to real dispersion. **B3** `simWanTransferS` — documented knob, left 0 + (no localhost ground truth; do NOT enable). All default OFF. `test_fwdllm_sct_model.py` (7) covers mechanism + + flag-off byte-identical. **Whether the folds drive `wall_disparity`→~0 is the sign-off run's job**, not + asserted here (principle #11c). + - **Stage D (checker/telemetry gaps):** **D1** `fwdllm_trainer._fetch_weights` re-emits `task_recv` carrying + `sim_send_ts` (the override had dropped the base trainer's emission → field_coverage INV + K6 had null/null); + `test_fwdllm_trainer_task_recv.py` (3). **D2** `failsafe_ok`/K5 now compares sim wall against the RUN wall + budget for a real-compute sim (auto-detected via the progress axis) and SKIPs when no run budget is available, + instead of the wall-vs-vclock fallback that false-failed every fwdllm run (sim wall ≫ vclock BY construction — + real forward-grad compute); `TestFailsafeRealComputeSim` (3). + - **Stage E (optimization):** the two `_distribute_weights_{sync,async}` `time.sleep(0.1)` "busy wait" pads + (real-transport artifacts, ≥28.7 s/run) gated `if not self.simulated` — real byte-identical, sim skips pure + wall overhead. Full pytest suite re-run proves grad values + cadence unchanged. + --- ## §L Phase-1 sign-off remediation — async grad-path residence & carry-over (Batch 2.5, staged) diff --git a/lib/python/examples/fwdllm/telemetry_manifest.yaml b/lib/python/examples/fwdllm/telemetry_manifest.yaml index 4a0bfabac..05e74c9ad 100644 --- a/lib/python/examples/fwdllm/telemetry_manifest.yaml +++ b/lib/python/examples/fwdllm/telemetry_manifest.yaml @@ -58,14 +58,17 @@ event_categories: # path) unlocks selected_utility_believed_vs_actual*/belief_gap*. # agg_observed_s (Part 6, reusing the same PROP_ROUND_DURATION already # read for agg_round's trainer_speed_s) unlocks runtime_agg_vs_trainer/ - # runtime_overhead_*. Deliberately NOT added (Part 6 design decision): - # training_budget_s/overran/remaining_time_s/wait_time_s/pre_train_s/ - # post_train_s -- fwdllm's delay is a flat additive sleep, not cifar10's - # budget-minus-actual sleep-to-fill model, so there is no faithful - # "overrun"/budget-slack concept to report; budget_slack_cdf/ - # trainer_late_fraction_hist/trainer_response_lateness_cdf/ - # trainer_runtime_expected_vs_actual stay not_populated rather than - # reporting a fabricated budget signal. + # runtime_overhead_*. Per-phase wall breakdown ADDED in Stage A1 for the + # #6 wall-decomposition (real vs sim s/round): mqtt_fetch_s/weights_to_ram_s/ + # weights_to_gpu_s/pre_train_s/gpu_compute_s/post_train_s/training_budget_s/ + # trainer_phase -- these un-skip the 8 phase rungs. NOTE the earlier Part-6 + # decision still holds for the DERIVED budget-slack plots: fwdllm's delay is + # a flat additive sleep (training_budget_s = _delay_s, the modeled D), not + # cifar10's budget-minus-actual sleep-to-fill model, so there is no faithful + # "overrun"/slack concept -- overran/remaining_time_s/wait_time_s stay + # unemitted and budget_slack_cdf/trainer_late_fraction_hist/ + # trainer_response_lateness_cdf/trainer_runtime_expected_vs_actual stay + # not_populated rather than reporting a fabricated budget signal. selection: populated # selection, emitted from flame/selector/random.py's select() (P5.5 fix # -- RandomSelector never called emit_selection before this session, so diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index 826bdfd70..8c181b50d 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -6,6 +6,7 @@ import json import hashlib import os +import zlib import numpy as np from datetime import datetime import ast @@ -530,9 +531,27 @@ def _emulate_training_delay(self): f"Delayed eval time for trainer " f"{self.trainer_id} by {eval_delay}s. Sleeping for {_delay_s}s." ) + _delay_s += self._sim_straggler_offset_s() return _delay_s return 0.0 + def _sim_straggler_offset_s(self) -> float: + """B2 (K-D20 #6): the modeled delay D is flat across trainers, so the + sync barrier's k-th-smallest sct under-spreads vs real's trainer_speed_s + dispersion (~2.3 s/round). In SIM only (real gets its spread from GPU + contention) add a STABLE per-trainer offset in [0, simStragglerSpreadS) + so the cohort completion spread matches real. Deterministic in + trainer_id (crc32, not salted like hash()) ⇒ reproducible; spread 0 ⇒ + byte-identical.""" + if not self.simulated: + return 0.0 + _hp = getattr(getattr(self, "config", None), "hyperparameters", None) + spread = float(getattr(_hp, "sim_straggler_spread_s", 0.0) or 0.0) + if spread <= 0.0: + return 0.0 + frac = (zlib.crc32(str(self.trainer_id).encode()) % 1000) / 1000.0 + return spread * frac + @timer_decorator def train_with_data_id(self): # Create FwdLLMStage for timing/metrics logging @@ -546,12 +565,17 @@ def train_with_data_id(self): ) return + # Phase-timing entry (Stage A1): everything up to the compute loop is + # pre_train (avail check, FwdLLMStage setup, loader state). + _phase_entry = time.time() if not self._check_availability(): return _round_start_ts = time.time() + _pre_train_s = _round_start_ts - _phase_entry self._perform_training() _real_gpu_time_s = time.time() - _round_start_ts + _phase_post_start = time.time() # emulate delays in training (due to compute resource and/or # dataset size and/or network latency) @@ -565,7 +589,15 @@ def train_with_data_id(self): # the optional pre-commit holding leg. _send_grads sends these so the # aggregator can order updates by _sim_completion_ts. In real mode these # stay None and the aggregator falls back to arrival order (unchanged). - self._sim_round_duration_s = _real_gpu_time_s + _delay_s + # B3 (K-D20 #6): WAN payload-transfer term (up+down). NOT measurable on + # localhost (no ground truth), so this is a DOCUMENTED knob left at 0 -- + # do NOT enable without a real WAN measurement. Byte-identical at 0. + _hp = getattr(getattr(self, "config", None), "hyperparameters", None) + _wan_s = ( + float(getattr(_hp, "sim_wan_transfer_s", 0.0) or 0.0) + if (self.simulated and _hp is not None) else 0.0 + ) + self._sim_round_duration_s = _real_gpu_time_s + _delay_s + _wan_s if self.simulated: _leg = self.sim_completion_leg_s _base = self._sim_send_ts if self._sim_send_ts is not None else time.time() @@ -584,14 +616,17 @@ def train_with_data_id(self): _stat_utility = float(self._stat_utility) except (TypeError, ValueError): _stat_utility = None + # Post-compute overhead (delay emulation + telemetry build); the last + # trainer-side phase term for the #6 wall decomposition (Stage A1). + _post_train_s = time.time() - _phase_post_start ev, fields = build_trainer_round( round_num=int(self._round), real_gpu_time_s=_real_gpu_time_s, - # real GPU compute + the emulated delay (0 if disabled) -- - # NOT a budget-vs-actual quantity (fwdllm has no sleep-to- - # fill-budget model, unlike async_cifar10's trainer); this is - # simply the total wall time this round actually took. - sim_round_duration_s=_real_gpu_time_s + _delay_s, + # real GPU compute + the emulated delay (0 if disabled) + any + # sim sct-model folds (B2 straggler spread in _delay_s, B3 WAN) -- + # NOT a budget-vs-actual quantity (fwdllm has no sleep-to-fill- + # budget model); the total modeled wall this round took. + sim_round_duration_s=self._sim_round_duration_s, avail_state=self.avl_state.value, dataset_size=self.dataset_size, stat_utility=_stat_utility, @@ -599,6 +634,17 @@ def train_with_data_id(self): "data_id": self.data_id, "iteration_per_data_id": self.iteration_per_data_id, "model_version": self._model_version, + # Per-phase wall breakdown (Stage A1): pre/gpu/post are stamped + # here; mqtt_fetch_s + weights_to_{ram,gpu}_s ride in via + # _phase_times (populated in fwdllm_trainer._fetch_weights). + "pre_train_s": _pre_train_s, + "gpu_compute_s": _real_gpu_time_s, + "post_train_s": _post_train_s, + "training_budget_s": _delay_s, + "trainer_phase": ( + f"{self._round}/{self.data_id}/{self.iteration_per_data_id}" + ), + **getattr(self, "_phase_times", {}), }, ) telemetry.emit(ev, **fields) diff --git a/lib/python/flame/config.py b/lib/python/flame/config.py index dc8f28fb1..2578ffd16 100644 --- a/lib/python/flame/config.py +++ b/lib/python/flame/config.py @@ -273,6 +273,22 @@ class Hyperparameters(FlameSchema, extra=Extra.allow): sim_unavailability: t.Optional[bool] = Field( alias="simUnavailability", default=False ) + # sct-model folds (fwdllm Stage B / K-D20 #6). Default OFF ⇒ byte-identical. + # simModelEvalTime (B1): charge the measured server-eval wall to the vclock + # after each committed data_id -- the largest GENUINE unmodeled term (~3.34 + # s/round). simStragglerSpreadS (B2): per-trainer straggler dispersion (s) + # added to the modeled delay so the sync barrier's k-th sct reflects real + # trainer_speed_s spread (~2.3 s/round). simWanTransferS (B3): WAN payload- + # transfer term -- NOT measurable on localhost, documented knob, leave 0. + sim_model_eval_time: t.Optional[bool] = Field( + alias="simModelEvalTime", default=False + ) + sim_straggler_spread_s: t.Optional[float] = Field( + alias="simStragglerSpreadS", default=0.0 + ) + sim_wan_transfer_s: t.Optional[float] = Field( + alias="simWanTransferS", default=0.0 + ) # Per-baseline: aware baselines free stalled slots proactively at the next # selection boundary (Stage D); unaware wait for the 90s vclock abandon. # Kept for backward compat — new code reads proactive_inflight_evict first. diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 9a2ef8977..4975315b5 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -1003,6 +1003,26 @@ def _aggregate_grads_async(self, tag: str) -> None: self._process_aggregation_goal_met(tag, channel, is_async=True) @timer_decorator + def _release_end_on_return(self, channel, end) -> None: + """Release a returned trainer's compute slot + re-pick guard on grad + RETURN -- EXCEPT on the async sim residence path, where the grad is + CARRIED and commits later in virtual time. There the guard/slot + lifetime is owned solely by `_sim_hold_busy_slots` (held to COMMIT, + mirroring felix's agg-goal-boundary release, `asyncfl:1316`); releasing + `all_selected` here on the physical RETURN would re-eligible a trainer + whose carried grad has NOT yet committed -> re-dispatch-while-in-flight + -> R1 residence violation (K-D19: 0% -> 44.7%). Real mode / non-residence: + return ~= commit, so release immediately as before (byte-identical). + """ + if self.is_async: + if getattr(self, "simulated", False) and getattr( + self, "_sim_inflight_residence", False + ): + return # guard/slot held to COMMIT by _sim_hold_busy_slots + channel.cleanup_provided_ends(end) + else: + channel.cleanup_recvd_end(end) + def _process_single_trainer_message(self, channel, msg, end, timestamp): # An end may only contribute once per (round, data_id, # iteration_per_data_id) collection cycle -- _per_agg_trainer_list is @@ -1076,6 +1096,9 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): f"with model version {msg[MessageType.MODEL_VERSION]}" ) self._agg_goal_cnt += 1 + # Wall of the most-recent accepted grad -> barrier_wait_s / drain_ + # tail_s in the per-round wall decomposition (Stage A2). + self._last_grad_wall_ts = time.time() channel.set_end_property( end, PROP_LAST_SELECTED_ROUND, msg[MessageType.MODEL_VERSION] @@ -1251,15 +1274,11 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): logger.info( f"Received grads from {end}. It was trained on model version {version}, with {count} samples" ) - # Not remove_from_selected_ends(): it never clears the selector's - # all_selected set, permanently blocking this trainer from future - # reselection. Async selectors only implement the batch - # _cleanup_recvd_ends/_cleanup_provided_ends path; cleanup_recvd_end() - # is sync-only (random selector). - if self.is_async: - channel.cleanup_provided_ends(end) - else: - channel.cleanup_recvd_end(end) + # Release the slot/guard on return -- but the async sim residence path + # defers that release to COMMIT (K-D19); see _release_end_on_return. + # (cleanup_recvd_end() is sync-only, random selector; the async batch + # _cleanup_provided_ends path is the only one async selectors implement.) + self._release_end_on_return(channel, end) return True def _log_and_reset_model_version_stats(self): @@ -1446,7 +1465,19 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): f"if variance check fails in aggregate()." ) + # Per-round wall decomposition (Stage A2, grounds #6 / K-D20): the FedAvg + # merge, the dispatch→last-grad barrier wait, and the last-grad→commit + # drain tail (a real-transport ARTIFACT the sim's all-k barrier does not + # model). eval_s is measured on the pass path below. All wall (real); + # the sim advances the vclock instead, so these read ~0 there. + _agg_start_wall = time.time() self.aggregate(self._round) + _aggregate_fedavg_s = time.time() - _agg_start_wall + _disp = getattr(self, "_round_dispatch_wall_ts", None) + _lastg = getattr(self, "_last_grad_wall_ts", None) + _barrier_wait_s = (_lastg - _disp) if (_disp and _lastg) else None + _drain_tail_s = (_agg_start_wall - _lastg) if _lastg else None + _eval_s = None # set below only when the variance gate passes (eval runs) _var_thr = getattr(self, "var_threshold", None) _ratio = ( @@ -1504,7 +1535,20 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): f"Variance check {_pass_kind}. Evaluating model and advancing data_id." ) self.iteration_per_data_id += 1 + _eval_start_wall = time.time() result, _, _ = self.eval_model() + _eval_s = time.time() - _eval_start_wall # GENUINE server-eval term (B1) + # B1 (K-D20 #6): the server eval is genuine algorithmic time between + # committed data_ids that the sim's sct model omits, so the vclock + # under-counts by ~eval_s/commit. When enabled, charge the MEASURED + # eval wall (sim runs real eval compute, so it ≈ real's) to the + # vclock -- consistent with the existing model already putting + # real_gpu on the vclock via sct (principle #1: genuine compute, not + # transport overhead). Config-gated OFF ⇒ byte-identical. + if self.simulated and getattr( + self.config.hyperparameters, "sim_model_eval_time", False + ): + self._vclock.advance(self._vclock.now + _eval_s) logger.info( f"Round {self._round}, Data ID {self.data_id} Eval Loss: {result['eval_loss']}" ) @@ -1602,6 +1646,12 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): # R1/W1 residence rungs (§L.3): per-contributor # [dispatch_ts, commit_ts] intervals for this cycle. "contributor_intervals": _contributor_intervals, + # Per-round wall decomposition (Stage A2, #6/K-D20): + # barrier wait + drain tail (artifact) + fedavg + eval. + "barrier_wait_s": _barrier_wait_s, + "drain_tail_s": _drain_tail_s, + "aggregate_fedavg_s": _aggregate_fedavg_s, + "eval_s": _eval_s, }, ) telemetry.emit(ev, **fields) @@ -2163,6 +2213,52 @@ def _select_ends_respecting_reselect_gate(self, channel, task_to_perform: str): return merged return new_ends + def _await_dispatchable_under_scarcity(self, task_to_perform: str) -> None: + """Real-mode sync-barrier liveness under availability scarcity (Stage C). + + With `agg_goal` clients required but a trace (e.g. mobiperf_2st) keeping + the eligible pool below `agg_goal`, the plain loop hot-re-dispatches every + pass -- burning the wall budget with no progress and ballooning the log + (payload-size lines ×∞, the observed 292 MB). Parity-faithful fix: KEEP + the cohort == `agg_goal` and WAIT for availability to recover + (sleep-to-next-avail, mirroring the trainer's `wait_until_next_avl` loop) + rather than spin -- the sim assembles the same full cohort by jumping its + vclock past the unavailable window, so cohort size stays identical + real↔sim. Self-terminates via `_check_early_stop_conditions` at + `max_runtime_s`. Byte-identical when availability tracking is off + (`trainer_event_dict is None` ⇒ nobody unavailable ⇒ never waits) and on + the sim path (the vclock, not a wall sleep, models the wait). + """ + if self.simulated or getattr(self, "trainer_event_dict", None) is None: + return + # Only a genuine post-join scarcity, not startup join-lag: wait only once + # at least `agg_goal` trainers have joined. + if len(getattr(self, "all_trainers", ())) < self._agg_goal: + return + poll_s = float(getattr(self.config.hyperparameters, "scarcity_poll_s", 2.0)) + warned = False + while not self._work_done: + unavail = set(self.get_curr_unavail_trainers()) + contributed = set(self._per_agg_trainer_list or []) + # A trainer can still advance this cycle's barrier iff it is available + # AND has not already contributed to it. + dispatchable = [ + e for e in self.all_trainers + if e not in unavail and e not in contributed + ] + if dispatchable or self._agg_goal_cnt >= self._agg_goal: + return # progress possible this pass (or barrier already met) + if not warned: # one line per stall, not one per spin + logger.warning( + f"[SYNC_SCARCITY_WAIT] round={self._round} data_id={self.data_id} " + f"committed={self._agg_goal_cnt}/{self._agg_goal} " + f"unavail={len(unavail)}/{len(self.all_trainers)}; waiting for " + f"availability (poll={poll_s}s) instead of spin-dispatching." + ) + warned = True + time.sleep(poll_s) + self._check_early_stop_conditions() # self-terminate at max_runtime_s + @timer_decorator def _distribute_weights_sync( self, tag: str, task_to_perform: str = "train" @@ -2182,6 +2278,9 @@ def _distribute_weights_sync( return channel.await_join() + # Sleep-to-next-avail under real-mode scarcity instead of hot-spinning + # (Stage C); no-op on the sim path / when availability tracking is off. + self._await_dispatchable_under_scarcity(task_to_perform) global_model_params = self.get_global_model_params() format_hash = lambda d: {k: _calculate_hash(v)[:8] for k, v in d.items()} logging.info( @@ -2189,9 +2288,14 @@ def _distribute_weights_sync( ) self.weights = global_model_params - logger.debug(f"Starting busy wait at time {time.time()}") - time.sleep(0.1) - logger.debug(f"Ended busy wait at time {time.time()}") + # Real-transport pad to let just-distributed messages settle before the + # selection read (real-mode MQTT artifact, principle #8). No sim analog: + # the sim orders by sct, not physical arrival, so this is pure wall + # overhead there (≥28.7 s/run, Stage E) -- skip it. Real unchanged. + if not self.simulated: + logger.debug(f"Starting busy wait at time {time.time()}") + time.sleep(0.1) + logger.debug(f"Ended busy wait at time {time.time()}") if self.trainer_event_dict is not None: curr_unavail_trainer_list = self.get_curr_unavail_trainers() @@ -2304,6 +2408,11 @@ def _distribute_weights_sync( logger.info(f"Sent weights to {end}") # self.invoke_gc() + # Cohort-dispatch wall -> barrier_wait_s anchor (Stage A2). Sync barrier: + # all `ends` dispatch in this one pass, so this marks the start of the + # dispatch→last-grad window. + self._round_dispatch_wall_ts = time.time() + @timer_decorator def _distribute_weights_async( self, tag: str, task_to_perform: str = "train" @@ -2318,9 +2427,14 @@ def _distribute_weights_async( channel.await_join() global_model_params = self.get_global_model_params() self.weights = global_model_params - logger.debug(f"Starting busy wait at time {time.time()}") - time.sleep(0.1) - logger.debug(f"Ended busy wait at time {time.time()}") + # Real-transport pad to let just-distributed messages settle before the + # selection read (real-mode MQTT artifact, principle #8). No sim analog: + # the sim orders by sct, not physical arrival, so this is pure wall + # overhead there (≥28.7 s/run, Stage E) -- skip it. Real unchanged. + if not self.simulated: + logger.debug(f"Starting busy wait at time {time.time()}") + time.sleep(0.1) + logger.debug(f"Ended busy wait at time {time.time()}") if self.trainer_event_dict is not None: curr_unavail_trainer_list = self.get_curr_unavail_trainers() channel.set_curr_unavailable_trainers( @@ -2422,6 +2536,15 @@ def _distribute_weights_async( end, PROP_ROUND_START_TIME, (self._round, datetime.now()) ) if self.simulated: + # R1 tripwire (K-D19): dispatching a trainer already outstanding + # (in flight OR grad carried-but-uncommitted) IS the residence + # violation -- surface it at the dispatch instant instead of only + # reconstructing it offline from contributor_intervals. + if end in self._sim_inflight_expected or self._sim_buffer.has(end): + logger.warning( + f"[SIM_R1_DISPATCH] end={end} re-dispatched while still " + f"outstanding (vclock={_round_now}) -- R1 residence violation" + ) channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now) # Expected completion = dispatch vclock + a lower-bound budget # (this end's own last-observed TRAINING_BUDGET_S, else the @@ -2480,13 +2603,41 @@ def _check_early_stop_conditions(self) -> None: max_runtime_s = getattr(self.config.hyperparameters, "max_runtime_s", None) if max_runtime_s is not None: - elapsed = time.time() - self.agg_start_time_ts + # Mode-dependent clock (mirrors base syncfl `max_experiment_runtime_s`, + # top_aggregator.py:1140): real -> WALL seconds; sim -> VIRTUAL seconds + # (`vclock.now`). One budget then means "3600 wall-s of real work" AND + # "3600 virtual-s of MODELED work" -- the matched-budget axis + # convergence parity needs (real runs 3600s wall; sim runs until its + # vclock reaches the SAME 3600s of modeled real-time). Until root #6 + # (vclock models real wall) lands, the sim vclock under-counts, so the + # wall failsafe below will fire first -- which is itself the #6 signal. + if self.simulated and hasattr(self, "_vclock"): + elapsed = float(self._vclock.now) + clock_label = "sim/vclock" + else: + elapsed = time.time() - self.agg_start_time_ts + clock_label = "real/wall" if elapsed >= float(max_runtime_s): logger.info( f"max_runtime_s={max_runtime_s}s reached " - f"(elapsed={elapsed:.0f}s); stopping run." + f"({clock_label}_elapsed={elapsed:.0f}s); stopping run." ) self._work_done = True + return + # Sim wall failsafe: a well-behaved (correctly-clocked) sim finishes + # in <= real wall, so cap sim WALL at `sim_wall_ceiling_s` (default = + # the budget) so an under-modeled vclock (root #6) can't run away. + if self.simulated: + _wall = time.time() - self.agg_start_time_ts + _ceil = getattr(self.config.hyperparameters, "sim_wall_ceiling_s", None) + _ceil = float(_ceil) if _ceil is not None else float(max_runtime_s) + if _wall > _ceil: + logger.warning( + f"[SIM_WALL_CEILING] wall={_wall:.0f}s > ceiling={_ceil:.0f}s " + f"(vclock={self._vclock.now:.0f}s) -- sim slower than budget " + f"OR vclock under-modeled (root #6); stopping run." + ) + self._work_done = True def _async_inner_loop_done(self) -> bool: """Exit condition for the async/hybrid compose path's inner diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index 6a69afd36..509a2c31a 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -19,6 +19,7 @@ import logging import math import time +from contextlib import contextmanager import torch from flame.channel import VAL_CH_STATE_HTBT_SEND, VAL_CH_STATE_RECV, VAL_CH_STATE_SEND @@ -39,6 +40,8 @@ from flame.mode.composer import Composer from flame.mode.message import MessageType from flame.mode.role import Role +from flame import telemetry +from flame.telemetry.events import build_task_recv from flame.mode.tasklet import Loop, Tasklet from flame.optimizers import optimizer_provider from flame.privacies import privacy_provider @@ -159,6 +162,23 @@ def internal_init(self) -> None: self.abort_training = False self._stat_utility = 0 + # Per-round phase-timing accumulator (Stage A1) -- the fwdllm-side of the + # #6 wall decomposition. Reset each fetch; drained into the trainer_round + # telemetry `extra` so the 8 phase rungs become measurable instead of + # SKIP. Mirrors the base syncfl trainer's _phase/_phase_times. + self._phase_times: dict = {} + + @contextmanager + def _phase(self, name: str): + """Time a named phase and accumulate into self._phase_times.""" + t0 = time.time() + try: + yield + finally: + self._phase_times[name] = self._phase_times.get(name, 0.0) + ( + time.time() - t0 + ) + def get(self, tag: str) -> None: """Get data from remote role(s).""" if tag == TAG_FETCH: @@ -172,6 +192,8 @@ def _fetch_weights(self, tag: str) -> None: ) self.fetch_success = False + # Reset per-round phase accumulator at the round boundary (Stage A1). + self._phase_times = {} channel = self.cm.get_by_tag(tag) if not channel: logger.info( @@ -192,7 +214,10 @@ def _fetch_weights(self, tag: str) -> None: # one aggregator is sufficient end = channel.one_end(VAL_CH_STATE_RECV) + _recv_start = time.time() msg, _ = recv_wrapper(self, channel, end) + # agg→trainer delivery + payload transfer (leg i); the first phase term. + self._phase_times["mqtt_fetch_s"] = time.time() - _recv_start if not msg: logger.info(f"NO msg received for trainer_id {self.trainer_id}") @@ -219,6 +244,25 @@ def _fetch_weights(self, tag: str) -> None: if MessageType.ROUND in msg: self._round = msg[MessageType.ROUND] + # D1 (§H #8): emit task_recv carrying sim_send_ts so field_coverage's INV + # rung + K6 (sim_send_ts correctness) have the field they expect. fwdllm + # overrode _fetch_weights and dropped the base trainer's emission; restore + # it here. None in real mode (SIM_SEND_TS absent) -> real~=sim comparable. + if telemetry.is_enabled(): + try: + _avl = getattr(getattr(self, "avl_state", None), "value", None) + ev, fields = build_task_recv( + round_num=int(self._round), + trainer_id=str(getattr(self, "trainer_id", "")), + time_mode=getattr(self, "time_mode", "real"), + sim_send_ts=(float(self._sim_send_ts) + if self._sim_send_ts is not None else None), + avl_state=_avl, + ) + telemetry.emit(ev, **fields) + except Exception as e: # telemetry must never break training + logger.debug(f"task_recv telemetry emit failed: {e}") + logger.info( f"Checking DataID: {self.data_id}| MessageType.DATA_ID in msg: {msg.get(MessageType.DATA_ID)}| IterationPerDataID: {self.iteration_per_data_id}| MessageType.ITERATION_PER_DATA_ID in msg: {msg.get(MessageType.ITERATION_PER_DATA_ID)}" ) @@ -299,13 +343,15 @@ def _fetch_weights(self, tag: str) -> None: # Update the model logger.info(f"Weights received: # {msg[MessageType.WEIGHTS]}") self.weights = # weights_to_model_device(msg[MessageType.WEIGHTS], self.model) - trainable_weights = weights_to_model_device( - msg[MessageType.WEIGHTS], self.model - ) - full_state_dict = self.model.state_dict() - full_state_dict.update(trainable_weights) - self.weights = full_state_dict - self._update_model() + with self._phase("weights_to_ram_s"): + trainable_weights = weights_to_model_device( + msg[MessageType.WEIGHTS], self.model + ) + full_state_dict = self.model.state_dict() + full_state_dict.update(trainable_weights) + self.weights = full_state_dict + with self._phase("weights_to_gpu_s"): + self._update_model() if MessageType.MODEL_VERSION in msg: self._model_version = msg[MessageType.MODEL_VERSION] diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index 19f88d163..8eebf272a 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -374,6 +374,94 @@ def test_intervals_present_even_without_captured_ts(self, tmp_path): telemetry.shutdown() +class TestPerRoundWallDecomposition: + """Stage A2: agg_round carries the per-round wall breakdown feeding #6 -- + aggregate_fedavg_s + eval_s always; barrier_wait_s/drain_tail_s when the + dispatch/last-grad wall stamps were captured (else null, rung SKIPs).""" + + def test_fedavg_and_eval_present_on_pass(self, tmp_path): + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=True) + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + # fedavg wall is measured around aggregate() and is non-negative + assert r["aggregate_fedavg_s"] is not None + assert r["aggregate_fedavg_s"] >= 0.0 + # eval ran (variance passed) -> eval_s measured, non-negative + assert r["eval_s"] is not None and r["eval_s"] >= 0.0 + finally: + telemetry.shutdown() + + def test_eval_s_null_on_variance_fail(self, tmp_path): + """eval_model only runs on the pass path, so eval_s is null on a FAIL.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["eval_s"] is None + assert r["aggregate_fedavg_s"] is not None # aggregate always runs + finally: + telemetry.shutdown() + + def test_barrier_and_drain_from_wall_stamps(self, tmp_path): + """When the dispatch + last-grad wall stamps exist, barrier_wait_s = + last_grad - dispatch and drain_tail_s = commit - last_grad.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + # Stamps the barrier collection would have set (dispatch then last + # grad, both in the past relative to the commit inside the method). + import time as _t + now = _t.time() + agg._round_dispatch_wall_ts = now - 5.0 + agg._last_grad_wall_ts = now - 2.0 + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert abs(r["barrier_wait_s"] - 3.0) < 0.5 # (now-2) - (now-5) + assert r["drain_tail_s"] >= 0.0 # commit is after last grad + finally: + telemetry.shutdown() + + def test_barrier_drain_null_without_stamps(self, tmp_path): + """No wall stamps captured (test double / async path) -> null, not a + crash; the rung SKIPs rather than the field being absent.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["barrier_wait_s"] is None + assert r["drain_tail_s"] is None + finally: + telemetry.shutdown() + + class _UtilityFakeChannel: """Generic fake for _process_single_trainer_message's channel calls -- stores per-end properties in a dict, doesn't care about specific PROP_* @@ -416,6 +504,7 @@ class _UtilityFakeAggregator: trainers is a no-op) since it's irrelevant to the telemetry under test.""" process = TopAggregator._process_single_trainer_message + _release_end_on_return = TopAggregator._release_end_on_return # K-D19 def __init__(self, model_version=5, data_id=3, iteration_per_data_id=0, is_async=False): diff --git a/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py b/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py index 460dcd6b3..fffa0d02b 100644 --- a/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py +++ b/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py @@ -30,10 +30,15 @@ def __init__( max_runtime_s=None, agg_start_time_ts=None, is_async=False, + simulated=False, + vclock_now=0.0, + sim_wall_ceiling_s=None, ): self._work_done = False self.data_id = data_id self.is_async = is_async + self.simulated = simulated + self._vclock = SimpleNamespace(now=vclock_now) self.agg_start_time_ts = ( agg_start_time_ts if agg_start_time_ts is not None else time.time() ) @@ -41,6 +46,7 @@ def __init__( hyperparameters=SimpleNamespace( max_data_id_progress=max_data_id_progress, max_runtime_s=max_runtime_s, + sim_wall_ceiling_s=sim_wall_ceiling_s, ) ) @@ -105,6 +111,45 @@ def test_whichever_fires_first_runtime_before_data_id(self): agg.check() assert agg._work_done is True + def test_sim_max_runtime_uses_vclock_not_wall(self): + """SIM mode (root #6 methodology): max_runtime_s is a VIRTUAL-clock + budget, not wall. vclock below budget => keep running even though wall + has long passed it (the sim runs faster than real).""" + agg = _FakeAggregator( + max_runtime_s=3600.0, simulated=True, vclock_now=100.0, + agg_start_time_ts=time.time(), # ~0 wall elapsed + ) + agg.check() + assert agg._work_done is False # vclock 100 < budget 3600 + + def test_sim_max_runtime_stops_when_vclock_reaches_budget(self): + agg = _FakeAggregator( + max_runtime_s=3600.0, simulated=True, vclock_now=3600.0, + agg_start_time_ts=time.time(), + ) + agg.check() + assert agg._work_done is True + + def test_sim_wall_ceiling_stops_runaway_undermodeled_vclock(self): + """The root-#6 failsafe: even with vclock below budget, a sim whose WALL + exceeds the ceiling (default = budget) stops -- an under-modeled vclock + must not run the sim forever.""" + agg = _FakeAggregator( + max_runtime_s=3600.0, simulated=True, vclock_now=80.0, # under budget + agg_start_time_ts=time.time() - 3601.0, # wall > ceiling + ) + agg.check() + assert agg._work_done is True + + def test_real_max_runtime_still_uses_wall(self): + """Real mode unchanged: wall-clock elapsed drives the cap.""" + agg = _FakeAggregator( + max_runtime_s=1.0, simulated=False, + agg_start_time_ts=time.time() - 2.0, + ) + agg.check() + assert agg._work_done is True + def test_already_work_done_is_a_noop(self): """Once stopped, the check must not re-derive or overwrite state.""" agg = _FakeAggregator(data_id=5, max_data_id_progress=10) diff --git a/lib/python/tests/mode/test_fwdllm_sct_model.py b/lib/python/tests/mode/test_fwdllm_sct_model.py new file mode 100644 index 000000000..9bef9dda4 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sct_model.py @@ -0,0 +1,104 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Stage B -- sct-model folds (K-D20 #6), all config-gated OFF => byte-identical. + +B1 eval_s onto the vclock (aggregator): after a committed data_id's eval, + advance the vclock by the measured eval wall when simModelEvalTime is set. +B2 per-trainer straggler spread (trainer): a stable offset in [0, spread) added + to the modeled delay in SIM only. +B3 WAN transfer knob: documented, default 0 (verified inert here). + +Only the MECHANISM + the flag-off byte-identical invariant is unit-tested; +whether the folds drive wall_disparity->~0 is an EMERGENT run quantity. +""" + +import os +import sys +from types import SimpleNamespace + +sys.path.insert( + 0, + os.path.join( + os.path.dirname(os.path.abspath(__file__)), "..", "..", "examples", "fwdllm", + "trainer", "forward_training", + ), +) +from FedSgdTrainer import FedSGDTrainer # noqa: E402 + + +class _Host: + """Binds the straggler-offset helper onto a minimal stand-in.""" + + _sim_straggler_offset_s = FedSGDTrainer._sim_straggler_offset_s + + def __init__(self, trainer_id="3", simulated=True, spread=0.0): + self.trainer_id = trainer_id + self.simulated = simulated + self.config = SimpleNamespace( + hyperparameters=SimpleNamespace(sim_straggler_spread_s=spread) + ) + + +class TestStragglerSpreadB2: + def test_zero_spread_is_no_offset(self): + assert _Host(spread=0.0)._sim_straggler_offset_s() == 0.0 + + def test_real_mode_never_offsets(self): + # real mode gets dispersion from GPU contention, not this knob + assert _Host(simulated=False, spread=5.0)._sim_straggler_offset_s() == 0.0 + + def test_offset_within_spread_and_stable(self): + h = _Host(trainer_id="7", spread=2.3) + o1 = h._sim_straggler_offset_s() + o2 = h._sim_straggler_offset_s() + assert 0.0 <= o1 < 2.3 + assert o1 == o2 # deterministic per trainer (reproducible) + + def test_offset_varies_across_trainers(self): + offs = { + _Host(trainer_id=str(i), spread=2.3)._sim_straggler_offset_s() + for i in range(1, 11) + } + # crc32-derived fractions spread the cohort (not all identical) + assert len(offs) > 1 + + +class TestEvalOnVclockB1: + """B1 mechanism: the aggregator advances the vclock by the measured eval + wall only when simModelEvalTime is set (byte-identical off).""" + + def _run(self, flag): + from flame import telemetry + from tests.mode.test_fwdllm_agg_telemetry import ( + _FakeAggregator, _FakeChannel, + ) + from datetime import timedelta + + agg = _FakeAggregator(contributors=["t1"], var_good_enough=True) + agg.simulated = True + # sim path calls the boundary slot-release hook; not under test here. + agg._release_sim_slots_at_agg_goal = lambda *a, **k: None + agg._vclock = SimpleNamespace( + now=100.0, + advance=lambda ts: setattr(agg._vclock, "now", max(agg._vclock.now, ts)), + ) + agg.config.hyperparameters.sim_model_eval_time = flag + # eval_model in the fake returns instantly, so measured eval_s ~ 0; force + # a nonzero measured eval by making eval_model sleep a hair. + import time as _t + orig_eval = agg.eval_model + + def _slow_eval(): + _t.sleep(0.02) + return orig_eval() + + agg.eval_model = _slow_eval + channel = _FakeChannel(durations={"t1": timedelta(seconds=2)}) + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + return agg._vclock.now + + def test_flag_off_does_not_advance_vclock(self): + assert self._run(flag=False) == 100.0 # byte-identical: vclock untouched + + def test_flag_on_charges_eval_wall(self): + assert self._run(flag=True) > 100.0 # vclock advanced by measured eval_s diff --git a/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py b/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py index 8c36cff03..2527a3d5e 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py +++ b/lib/python/tests/mode/test_fwdllm_sim_drain_and_repick.py @@ -144,6 +144,7 @@ class _RepickAgg: stamp lands before it) and telemetry stays disabled by default.""" process = TopAggregator._process_single_trainer_message + _release_end_on_return = TopAggregator._release_end_on_return # K-D19 def __init__(self, residence=True, simulated=True, curr_ver=(5, 2, 1)): self.simulated = simulated diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py index 01f2b2af9..61a486d0d 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py @@ -68,6 +68,8 @@ class _FakeGradAgg: _sim_recv_grace_s = _SyncBase._sim_recv_grace_s # fwdllm overrides felix's hold with the Option-A two-lifetime split (K-D16). _sim_hold_busy_slots = TopAggregator._sim_hold_busy_slots + # The return-path guard/slot release (K-D19: defers to COMMIT in sim residence). + _release_end_on_return = TopAggregator._release_end_on_return # grace-window class knobs the base method reads off self SIM_RECV_GRACE_FLOOR_S = 0.0 SIM_RECV_GRACE_FACTOR = 0.0 @@ -223,6 +225,15 @@ def add_msg(self, end, sct, budget=None, release_at=0): super().add_msg(end, sct, budget, release_at) self._ends.setdefault(end, _FakeEnd()) + # Faithful mirror of channel.cleanup_provided_ends -> selector + # _cleanup_provided_ends: drop the end from the re-pick guard + slot. + def cleanup_provided_ends(self, end): + self._selector.all_selected.pop(end, None) + self._selector.selected_ends[self._selector.requester].discard(end) + + def cleanup_recvd_end(self, end): # sync path (random selector) + self.cleanup_provided_ends(end) + class TestAsyncBoundaryReleasesSlots: def test_async_boundary_frees_every_committed_slot(self): diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py index b2593ac3d..707b0cce3 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -114,6 +114,47 @@ def test_no_same_trainer_reselected_while_in_flight(self): assert ch._selector.selected_ends["agg"] == {"B", "C"} +class TestReturnPathGuardHeldToCommit: + """K-D19 (the R1 regression K-D17b left open): the guard release on grad + RETURN. The prior residence tests poked `_release_sim_slots_at_agg_goal` / + `_sim_hold_busy_slots` directly and NEVER exercised the per-message return + path (`_release_end_on_return`, called from `_process_single_trainer_message`) + -- which is exactly why R1=44.7% shipped with `tests/mode` green. The bug: + the async accept path called `channel.cleanup_provided_ends(end)` on physical + return, tearing the trainer out of `all_selected` while its carried grad had + not committed in virtual time -> re-selectable -> re-dispatch-while-in-flight. + """ + + def test_guard_held_on_return_in_sim_residence(self): + """async + sim + residence: return must NOT release the re-pick guard + (held to COMMIT by _sim_hold_busy_slots). This is the regression guard.""" + agg = _residence_agg(residence=True) + agg.is_async = True + ch = _FakeSelChannel(["A", "B", "C"]) # all dispatched + in flight + agg._release_end_on_return(ch, "A") # A's grad returns (carried) + # A stays in the guard -> cannot be re-picked while still outstanding. + assert "A" in ch._selector.all_selected + assert "A" in ch._selector.selected_ends["agg"] + + def test_guard_released_on_return_when_residence_off(self): + """async WITHOUT residence: legacy behavior -- release immediately + (return ~= commit), so the fix is byte-identical off the residence path.""" + agg = _residence_agg(residence=False) + agg.is_async = True + ch = _FakeSelChannel(["A", "B", "C"]) + agg._release_end_on_return(ch, "A") + assert "A" not in ch._selector.all_selected + + def test_sync_return_uses_recvd_cleanup(self): + """sync (random selector, is_async=False): unchanged cleanup_recvd_end + path -- releases on return (barrier re-selects the whole cohort).""" + agg = _residence_agg(residence=True) # residence flag is inert when sync + agg.is_async = False + ch = _FakeSelChannel(["A", "B", "C"]) + agg._release_end_on_return(ch, "A") + assert "A" not in ch._selector.all_selected + + class TestVirtualInflightSlotHold: """K-D17b (felix-aligned, supersedes K-D16 Option-A): a returned-but- uncommitted trainer is still in flight in VIRTUAL time (its grad commits diff --git a/lib/python/tests/mode/test_fwdllm_sync_scarcity_wait.py b/lib/python/tests/mode/test_fwdllm_sync_scarcity_wait.py new file mode 100644 index 000000000..b9fa90451 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sync_scarcity_wait.py @@ -0,0 +1,182 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Stage C -- real-mode sync-barrier liveness under availability scarcity. + +`_await_dispatchable_under_scarcity` keeps the cohort == `agg_goal` and +sleep-to-next-avail (parity-faithful, matches the sim vclock-jump) instead of +hot-spin-dispatching when a trace keeps the eligible pool below `agg_goal`. + +Invariants under test: +- no-op on the sim path and when availability tracking is off (byte-identical), +- no-op during startup join-lag (fewer than `agg_goal` trainers joined), +- returns immediately when an available un-contributed trainer exists, +- waits (bounded) while the pool is scarce, then proceeds once it recovers, +- self-terminates via `_check_early_stop_conditions` at the wall budget. +""" + +from types import SimpleNamespace + +import pytest + +from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator + + +class _FakeAggregator: + """Minimal stand-in exposing only what the scarcity gate touches.""" + + def __init__( + self, + *, + simulated=False, + trainer_event_dict=None, + all_trainers=None, + agg_goal=10, + agg_goal_cnt=0, + per_agg_trainer_list=None, + unavail_sequence=None, + ): + self.simulated = simulated + self.trainer_event_dict = trainer_event_dict + self.all_trainers = set(all_trainers or []) + self._agg_goal = agg_goal + self._agg_goal_cnt = agg_goal_cnt + self._per_agg_trainer_list = list(per_agg_trainer_list or []) + self._round = 0 + self.data_id = 0 + self._work_done = False + self.config = SimpleNamespace(hyperparameters=SimpleNamespace(scarcity_poll_s=0.0)) + # Each poll pops the next unavailable-set from this sequence (the last + # one repeats), letting a test model availability recovering over time. + self._unavail_sequence = list(unavail_sequence or [[]]) + self.slept = 0 + self.stop_checks = 0 + + def get_curr_unavail_trainers(self): + if len(self._unavail_sequence) > 1: + return self._unavail_sequence.pop(0) + return self._unavail_sequence[0] + + def _check_early_stop_conditions(self): + self.stop_checks += 1 + + _await_dispatchable_under_scarcity = ( + TopAggregator._await_dispatchable_under_scarcity + ) + + +@pytest.fixture(autouse=True) +def _no_real_sleep(monkeypatch): + import flame.mode.horizontal.syncfl.fwdllm_aggregator as mod + + def _count_sleep(_s): + _count_sleep.calls += 1 + + _count_sleep.calls = 0 + monkeypatch.setattr(mod.time, "sleep", _count_sleep) + return _count_sleep + + +class TestScarcityWaitNoOp: + def test_sim_path_never_waits(self, _no_real_sleep): + agg = _FakeAggregator( + simulated=True, + trainer_event_dict={"1": []}, + all_trainers=[str(i) for i in range(10)], + unavail_sequence=[[str(i) for i in range(10)]], # all unavail + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 0 + + def test_tracking_off_never_waits(self, _no_real_sleep): + agg = _FakeAggregator( + trainer_event_dict=None, # availability tracking disabled + all_trainers=[str(i) for i in range(10)], + unavail_sequence=[[str(i) for i in range(10)]], + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 0 + + def test_join_lag_does_not_wait(self, _no_real_sleep): + # Only 3 of the 10 have joined -> startup, not scarcity: do not block. + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=["0", "1", "2"], + agg_goal=10, + unavail_sequence=[["0", "1", "2"]], + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 0 + + def test_available_trainer_returns_without_wait(self, _no_real_sleep): + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=[str(i) for i in range(10)], + unavail_sequence=[["0", "1", "2"]], # 7 available, none contributed + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 0 + + def test_barrier_already_met_returns(self, _no_real_sleep): + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=[str(i) for i in range(10)], + agg_goal=3, + agg_goal_cnt=3, # cohort complete + unavail_sequence=[[str(i) for i in range(10)]], + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 0 + + +class TestScarcityWaitEngages: + def test_waits_then_proceeds_when_availability_recovers(self, _no_real_sleep): + allt = [str(i) for i in range(10)] + # Scarce for two polls (all unavail), then trainer "5" comes back. + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=allt, + unavail_sequence=[list(allt), list(allt), [x for x in allt if x != "5"]], + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 2 # slept exactly across the two scarce polls + assert agg.stop_checks == 2 # re-checked early-stop each scarce poll + + def test_already_contributed_pool_still_scarce(self, _no_real_sleep): + # The available trainers have ALL already contributed this cycle, and the + # one still-uncontributed trainer ("5") is unavailable -> no dispatchable + # end -> must wait, not spin-redispatch the already-used ones. On the next + # poll "5" recovers -> dispatchable -> proceed. + allt = [str(i) for i in range(10)] + used = [x for x in allt if x != "5"] # the 9 available already contributed + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=allt, + agg_goal=10, + agg_goal_cnt=9, + per_agg_trainer_list=used, + unavail_sequence=[["5"], []], # "5" unavail, then recovers + ) + agg._await_dispatchable_under_scarcity("train") + assert _no_real_sleep.calls == 1 # waited one poll, then "5" recovered + + def test_self_terminates_at_budget(self, _no_real_sleep): + allt = [str(i) for i in range(10)] + + agg = _FakeAggregator( + trainer_event_dict={"1": []}, + all_trainers=allt, + unavail_sequence=[list(allt)], # permanently scarce + ) + + # Model the wall budget firing after 3 polls. + orig = agg._check_early_stop_conditions + + def _stop_after_3(): + orig() + if agg.stop_checks >= 3: + agg._work_done = True + + agg._check_early_stop_conditions = _stop_after_3 + agg._await_dispatchable_under_scarcity("train") + assert agg._work_done is True + assert _no_real_sleep.calls == 3 # bounded, then stopped diff --git a/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py new file mode 100644 index 000000000..8cac23e21 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py @@ -0,0 +1,150 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Stage A1 -- trainer per-phase wall timing. + +The fwdllm trainer emitted no per-phase breakdown, so the 8 phase rungs +(mqtt_fetch_s / weights_to_ram_s / weights_to_gpu_s / pre_train_s / +gpu_compute_s / post_train_s / training_budget_s / trainer_phase) SKIPed. This +covers: +- the `_phase` context manager accumulates into `_phase_times` (fwdllm_trainer), +- `FedSgdTrainer.train_with_data_id` drains those + the pre/gpu/post/budget/ + phase terms into the emitted `trainer_round` event. +""" + +import json +import os +import sys + +import pytest + +from flame import telemetry +from flame.mode.horizontal.syncfl.fwdllm_trainer import Trainer as FwdLLMTrainer + +_EXAMPLE = os.path.join( + os.path.dirname(__file__), + "..", "..", "examples", "fwdllm", "trainer", "forward_training", +) +sys.path.insert(0, os.path.abspath(_EXAMPLE)) +import FedSgdTrainer as _fedsgd_mod # noqa: E402 + +FedSgdTrainer = _fedsgd_mod.FedSGDTrainer + +_PHASE_KEYS = [ + "mqtt_fetch_s", + "weights_to_ram_s", + "weights_to_gpu_s", + "pre_train_s", + "gpu_compute_s", + "post_train_s", + "training_budget_s", + "trainer_phase", +] + + +class _PhaseHost: + """Minimal host for the (abstract) fwdllm Trainer's _phase method.""" + + _phase = FwdLLMTrainer._phase + + def __init__(self): + self._phase_times = {} + + +class TestPhaseContextManager: + def test_accumulates(self): + t = _PhaseHost() + with t._phase("gpu_compute_s"): + pass + with t._phase("gpu_compute_s"): + pass + assert "gpu_compute_s" in t._phase_times + # two entries under one name accumulate, not overwrite + assert t._phase_times["gpu_compute_s"] >= 0.0 + assert isinstance(t._phase_times["gpu_compute_s"], float) + + +class _AvlState: + value = "AVL_TRAIN" + + +class _FakeFedSgd: + """Binds the real train_with_data_id onto a stand-in that stubs only the + heavy compute (perform_training / delay / availability); the phase-timing + + telemetry-emit logic under test runs for real.""" + + train_with_data_id = FedSgdTrainer.train_with_data_id + + def __init__(self, phase_times=None, delay_s=1.5): + self._round = 7 + self.data_id = 3 + self.iteration_per_data_id = 2 + self._model_version = 5 + self.trainer_id = "t1" + self.simulated = False + self.abort_training = False + self.avl_state = _AvlState() + self.dataset_size = 128 + self._stat_utility = 0.9 + self._sim_send_ts = None + self.sim_completion_leg_s = 0.0 + self._sim_completion_ts = None + self._sim_round_duration_s = None + # Populated by _fetch_weights in a live run; pre-seeded here. + self._phase_times = dict(phase_times or {}) + self._delay_s = delay_s + + def _check_availability(self): + return True + + def _perform_training(self): + pass + + def _emulate_training_delay(self): + return self._delay_s + + +class TestTrainWithDataIdEmitsPhases: + def test_phase_fields_present(self, tmp_path): + telemetry.configure(role="trainer", run_dir=str(tmp_path)) + try: + t = _FakeFedSgd( + phase_times={ + "mqtt_fetch_s": 0.4, + "weights_to_ram_s": 0.2, + "weights_to_gpu_s": 0.1, + } + ) + t.train_with_data_id() + + lines = (tmp_path / "trainer.jsonl").read_text().splitlines() + events = [json.loads(l) for l in lines] + rounds = [e for e in events if e["event"] == "trainer_round"] + assert len(rounds) == 1 + ev = rounds[0] + for k in _PHASE_KEYS: + assert k in ev, f"missing phase field {k}" + # the three _fetch_weights phases flowed through verbatim + assert ev["mqtt_fetch_s"] == 0.4 + assert ev["weights_to_ram_s"] == 0.2 + assert ev["weights_to_gpu_s"] == 0.1 + # training_budget_s is the modeled additive delay + assert ev["training_budget_s"] == 1.5 + # trainer_phase encodes round/data_id/iteration identity + assert ev["trainer_phase"] == "7/3/2" + # pre/post are real non-negative wall slivers + assert ev["pre_train_s"] >= 0.0 + assert ev["post_train_s"] >= 0.0 + finally: + telemetry.shutdown() + + def test_aborted_round_emits_nothing(self, tmp_path): + telemetry.configure(role="trainer", run_dir=str(tmp_path)) + try: + t = _FakeFedSgd() + t.abort_training = True + t.train_with_data_id() + assert not (tmp_path / "trainer.jsonl").exists() or not ( + tmp_path / "trainer.jsonl" + ).read_text().strip() + finally: + telemetry.shutdown() diff --git a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py index 94baa1e9c..1c16fe7b0 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py @@ -33,6 +33,7 @@ class _FakeTrainer: touches; binds the real method under test.""" _emulate_training_delay = FedSGDTrainer._emulate_training_delay + _sim_straggler_offset_s = FedSGDTrainer._sim_straggler_offset_s def __init__(self, training_delay_enabled, training_delay_s=0.0, training_delay_factor=1.0, speedup_factor=1.0, simulated=False): @@ -164,8 +165,9 @@ class TestSimCompletionStampIsAdditive: def test_additive_round_duration_and_completion_ts(self, monkeypatch): monkeypatch.setattr(_fst_module.telemetry, "is_enabled", lambda: False) - # round_start=100.0, gpu-end=100.5 -> real_gpu = 0.5s. - monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.5])) + # ticks: phase_entry=100.0, round_start=100.0, gpu-end=100.5 (+clamp) + # -> real_gpu = 0.5s. (Stage A1 added the phase_entry tick.) + monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.0, 100.5])) t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=0.0) t.train_with_data_id() @@ -176,7 +178,7 @@ def test_additive_round_duration_and_completion_ts(self, monkeypatch): assert t._sim_completion_ts == 12.5 def test_completion_ts_includes_leg(self, monkeypatch): - monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.5])) + monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.0, 100.5])) t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=1.5) t.train_with_data_id() diff --git a/lib/python/tests/mode/test_fwdllm_trainer_task_recv.py b/lib/python/tests/mode/test_fwdllm_trainer_task_recv.py new file mode 100644 index 000000000..076ca8491 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_trainer_task_recv.py @@ -0,0 +1,104 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Stage D1 (§H #8): the fwdllm trainer overrode _fetch_weights and dropped the +base trainer's task_recv emission, so field_coverage's sim_send_ts INV rung + K6 +had no field to read (null/null both modes). Restored here -- task_recv carries +sim_send_ts (the aggregator's dispatch vclock), null in real mode. +""" + +import json + +from flame import telemetry +from flame.mode.horizontal.syncfl.fwdllm_trainer import Trainer +from flame.mode.message import MessageType + + +class _FakeSelector: + def __init__(self): + self.ordered_updates_recv_ends = [] + + +class _FakeChannel: + def __init__(self, msg): + self._msg = msg + self._selector = _FakeSelector() + + def await_join(self): + pass + + def one_end(self, state): + return "end_1" + + def recv(self, end_id): + return self._msg, None + + def cleanup_recvd_ends(self): + pass + + +class _FakeChannelManager: + def __init__(self, channel): + self._channel = channel + + def get_by_tag(self, tag): + return self._channel + + +class _FakeTrainer: + _fetch_weights = Trainer._fetch_weights + + def __init__(self, channel, time_mode="real"): + self.cm = _FakeChannelManager(channel) + self.trainer_id = "trainer_1" + self.fetch_success = False + self._work_done = False + self.data_id = None + self.iteration_per_data_id = None + self._round = 1 + self._model_version = 0 + self.time_mode = time_mode + + +def _task_recv_events(tmp_path): + lines = (tmp_path / "trainer.jsonl").read_text().splitlines() + return [json.loads(l) for l in lines if json.loads(l)["event"] == "task_recv"] + + +class TestTaskRecvEmission: + def test_sim_send_ts_emitted_in_sim(self, tmp_path): + telemetry.configure(role="trainer", run_dir=str(tmp_path)) + try: + channel = _FakeChannel( + {MessageType.ROUND: 3, MessageType.SIM_SEND_TS: 42.5} + ) + t = _FakeTrainer(channel, time_mode="simulated") + t._fetch_weights("fetch") + + evs = _task_recv_events(tmp_path) + assert len(evs) == 1 + assert evs[0]["sim_send_ts"] == 42.5 + assert evs[0]["time_mode"] == "simulated" + assert evs[0]["trainer_id"] == "trainer_1" + finally: + telemetry.shutdown() + + def test_sim_send_ts_null_in_real(self, tmp_path): + telemetry.configure(role="trainer", run_dir=str(tmp_path)) + try: + # real mode: aggregator stamps no SIM_SEND_TS -> field is null, which + # is what makes real~=sim task_recv directly comparable. + channel = _FakeChannel({MessageType.ROUND: 3}) + t = _FakeTrainer(channel, time_mode="real") + t._fetch_weights("fetch") + + evs = _task_recv_events(tmp_path) + assert len(evs) == 1 + assert evs[0]["sim_send_ts"] is None + finally: + telemetry.shutdown() + + def test_noop_when_telemetry_disabled(self, tmp_path): + assert not telemetry.is_enabled() + channel = _FakeChannel({MessageType.ROUND: 3, MessageType.SIM_SEND_TS: 1.0}) + _FakeTrainer(channel)._fetch_weights("fetch") + assert not (tmp_path / "trainer.jsonl").exists() diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 01a808f6e..d1a77d135 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -390,6 +390,141 @@ def test_normal_fl_still_keyed_on_round(self): assert r["sim_rounds"] == 10, r assert r["ok"], r + # --- Stage A3: the 4 advance rungs (K3/K4/K3a/K3b) re-key too, via + # _per_round_advances. Keyed on `round` they saw <2 units for fwdllm and + # SKIPed ("<2 sim rounds"); on data_id they measure real advances. --- + + def test_advance_rung_measures_on_data_id_axis(self): + # Matched 10 s/data_id on both sides -> K3 has advances and PASSes, + # instead of SKIP-ing for "<2 rounds" (round pinned at 1). + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 10)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 10), + ts=float(d)) for d in range(10)]) + r = pc.per_round_advance_parity(real, sim, ks_tol=0.2, mean_tol_rel=0.15) + assert "run too short" not in r.get("note", ""), r + assert r["ok"], r + + def test_advance_rung_catches_data_id_rate_gap(self): + # sim 26 s/data_id vclock vs real 15 s/data_id wall -> divergence caught. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 15)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 26), + ts=float(d)) for d in range(10)]) + r = pc.per_round_advance_parity(real, sim, ks_tol=0.2, mean_tol_rel=0.15) + assert not r["ok"], r + + def test_advance_mean_on_round_axis_unchanged(self): + # async_cifar10 (round-advancing): the sim mean advance is measured on + # `round` exactly as before the re-key (+7 vclock per round). + real = _agg(agg_rounds=[_round(r, ["a"], [0], ts=float(r * 7)) + for r in range(1, 8)]) + sim = _agg(agg_rounds=[_round(r, ["a"], [0], vclock=float(r * 7), + ts=float(r)) for r in range(1, 8)]) + r = pc.per_round_advance_parity(real, sim, ks_tol=0.2, mean_tol_rel=0.15) + assert r["sim_mean_advance_s"] == 7.0, r + assert r["real_mean_advance_s"] == 7.0, r + + def test_advance_mean_on_data_id_when_round_static(self): + # fwdllm: round pinned at 1 -> mean advance is measured per data_id + # (+4 vclock per committed data_id), not collapsed to a single stuck bin. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 4)) + for d in range(5)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 4), + ts=float(d)) for d in range(5)]) + r = pc.per_round_advance_parity(real, sim, ks_tol=0.2, mean_tol_rel=0.15) + assert r["sim_mean_advance_s"] == 4.0, r + + +class TestWallDisparity: + """Stage A4 / K-D20 #6: a DIAG rung reporting |real_wall − sim_vclock| per + matched progress unit. Never gates; surfaces the residual to drive to ~0.""" + + def test_zero_disparity_when_clocks_match(self): + # real advances 10 wall-s/data_id, sim 10 vclock-s/data_id -> residual 0. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 10)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 10), + ts=float(d)) for d in range(10)]) + r = pc.wall_disparity(real, sim) + assert r["axis"] == "data_id" + assert r["mean_abs_disparity_s"] == 0.0, r + assert r["max_abs_disparity_s"] == 0.0, r + assert r["n_matched_units"] == 10 + + def test_surfaces_the_rate_gap(self): + # real 35 wall-s/data_id vs sim 8 vclock-s/data_id -> growing residual, + # but the rung still "ok" (DIAG never fails). + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 35)) + for d in range(10)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 8), + ts=float(d)) for d in range(10)]) + r = pc.wall_disparity(real, sim) + assert r["ok"] is True # DIAG: informational only + # by the last of 10 data_ids: |9*35 - 9*8| = 243 + assert r["max_abs_disparity_s"] > 100.0, r + assert r["mean_abs_disparity_s"] > 0.0 + + def test_aligns_when_run_does_not_start_at_unit_zero(self): + # matched units start at data_id 3; cumulative-from-first-matched keeps + # residual 0 despite the nonzero vclock/ts origin. + real = _agg(agg_rounds=[_fwd_round(d, ["a"], ts=float(d * 10)) + for d in range(3, 8)]) + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 10), + ts=float(d)) for d in range(3, 8)]) + r = pc.wall_disparity(real, sim) + assert r["mean_abs_disparity_s"] == 0.0, r + + def test_skips_when_too_few_matched_units(self): + real = _agg(agg_rounds=[_fwd_round(0, ["a"], ts=0.0)]) + sim = _agg(agg_rounds=[_fwd_round(0, ["a"], vclock=0.0, ts=0.0)]) + r = pc.wall_disparity(real, sim) + assert r.get("status") == "SKIP", r + assert r["ok"] is True + + def test_round_axis_for_normal_fl(self): + real = _agg(agg_rounds=[_round(r, ["a"], [0], ts=float(r * 5)) + for r in range(1, 6)]) + sim = _agg(agg_rounds=[_round(r, ["a"], [0], vclock=float(r * 5), + ts=float(r)) for r in range(1, 6)]) + r = pc.wall_disparity(real, sim) + assert r["axis"] == "round" + assert r["mean_abs_disparity_s"] == 0.0, r + + +class TestFailsafeRealComputeSim: + """Stage D2 / §H #9: a real-compute sim (fwdllm runs the real forward-grad + pass in sim mode) has sim wall ≫ vclock by construction, so K5 must compare + wall against the RUN wall budget, not the vclock (else it false-fails).""" + + def test_real_compute_sim_skips_without_run_budget(self): + # data_id axis auto-detects real_compute_sim; wall ≫ vclock but no run + # budget passed -> SKIP, not a false INV failure. + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 8), + ts=float(d * 35)) for d in range(6)]) + r = pc.failsafe_ok(sim, budget_s=None) + assert r.get("status") == "SKIP", r + assert r["ok"] is True + + def test_real_compute_sim_uses_run_budget_when_given(self): + # 6 data_ids, wall ends at 5*35=175s; run budget 3600s -> within budget. + sim = _agg(agg_rounds=[_fwd_round(d, ["a"], vclock=float(d * 8), + ts=float(d * 35)) for d in range(6)]) + r = pc.failsafe_ok(sim, budget_s=3600.0) + assert r.get("status") != "SKIP", r + assert r["real_compute_sim"] is True + assert r["ok"] is True # 175s wall << 3600s run budget + + def test_cheap_compute_sim_keeps_vclock_fallback(self): + # round-advancing (async_cifar10): wall≈vclock, no run budget -> vclock + # fallback unchanged (byte-identical). + sim = _agg(agg_rounds=[_round(r, ["a"], [0], vclock=float(r * 10), + ts=float(r * 10)) for r in range(1, 7)]) + r = pc.failsafe_ok(sim, budget_s=None) + assert r["real_compute_sim"] is False + assert r.get("status") != "SKIP" + assert r["ok"] is True # wall == vclock -> 0 overshoot + class TestFieldCoverageAlias: """§H open-root #3: fwdllm's trainer emits gpu/budget under different names; From e35433d35905982b792608eea10d0431b1649983 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 13:18:46 -0400 Subject: [PATCH 17/56] fwdllm runner: availability params respected end-to-end (K-D22) Phase-1 now defaults to syn_0 and the pre-flight table shows the RESOLVED config actually patched into each launched cfg (print == run), not a hardcoded default that masked the yaml's own mode. - run_sequential.sh: --avail-trace defaults to syn_0 (Phase 1), so patch() always sets trainer availability.mode + aggregator trackTrainerAvail.trace + client_notify.trace EXPLICITLY on every baseline. The trace row + a new per-baseline `avail` column are read back from the patched cfg. New feasibility gate BLOCKS a full-participation sync barrier under a non-syn_0 trace (sync agg_goal >= n_trainers + unavailability -> can't assemble -> stall; --force to override). - fwdllm_plus + fluxtune yamls (real+sim): mode flipped mobiperf_* -> syn_0; switch to a mobiperf trace via --avail-trace only for Phase-2. - simulate_fwdllm.md: K-D22. Verified: generated launch cfg carries syn_0 in all three availability fields; --avail-trace mobiperf_2st --only fwdllm_plus blocks at pre-flight (exit 2). Co-Authored-By: Claude Opus 4.8 --- .../expt_scripts/fluxtune_n10_smoke.yaml | 9 ++-- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 9 ++-- .../expt_scripts/fwdllm_plus_n10_smoke.yaml | 10 ++-- .../fwdllm_plus_n10_smoke_sim.yaml | 10 ++-- .../fwdllm/expt_scripts/run_sequential.sh | 54 +++++++++++++++++-- lib/python/examples/fwdllm/simulate_fwdllm.md | 14 +++++ 6 files changed, 85 insertions(+), 21 deletions(-) diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index 0f1039530..628de0d8c 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -32,10 +32,11 @@ experiments: # file exists (or ever will) for agnews; skips that lookup entirely. path_style: true availability: - # Cosmetic here -- fluxtune's 3-tier signal comes from - # trainer.client_notify (set in the baseline), not this mode. Kept - # consistent with the baseline's mobiperf_3st_50 trace. - mode: mobiperf_3st_50 + # Phase 1 is syn_0 (100% availability): client_notify then reports all + # trainers available. Switch to mobiperf_3st_50 (via --avail-trace) only + # for the Phase-2 unavailability runs. run_sequential.sh's --avail-trace + # patches trainer mode + client_notify + trackTrainerAvail together. + mode: syn_0 # FedFwd has no simulated-clock support; always real. time_mode: real # Skip the registry's 4-18s artificial delay for a fast smoke run. diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 828e9abc2..37d4f487b 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -37,10 +37,11 @@ experiments: # file exists (or ever will) for agnews; skips that lookup entirely. path_style: true availability: - # Cosmetic here -- fluxtune's 3-tier signal comes from - # trainer.client_notify (set in the baseline), not this mode. Kept - # consistent with the baseline's mobiperf_3st_50 trace. - mode: mobiperf_3st_50 + # Phase 1 is syn_0 (100% availability): client_notify then reports all + # trainers available. Switch to mobiperf_3st_50 (via --avail-trace) only + # for the Phase-2 unavailability runs. run_sequential.sh's --avail-trace + # patches trainer mode + client_notify + trackTrainerAvail together. + mode: syn_0 # Batch 1: drive the simulated virtual clock instead of real sleeps. time_mode: simulated # Skip the registry's 4-18s artificial delay for a fast smoke run. diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml index fcfa7143d..42dcdb686 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke.yaml @@ -31,10 +31,12 @@ experiments: # file exists (or ever will) for agnews; skips that lookup entirely. path_style: true availability: - # ORACULAR reads trainer 1-10's mobiperf_2st trace from _metadata -- - # mode here just keeps the trainer-side avl_events_* fields - # consistent with what the aggregator reads. - mode: mobiperf_2st + # Phase 1 is syn_0 (100% availability); the ORACULAR aggregator then + # reads all-available, so the sync barrier assembles the full agg_goal=10 + # cohort. Switch to a mobiperf_* trace (via --avail-trace) only for the + # Phase-2 unavailability runs. Keep trainer + aggregator on the SAME + # trace (run_sequential.sh's --avail-trace patches both). + mode: syn_0 # FedFwd has no simulated-clock support; always real. time_mode: real # Skip the registry's 4-18s artificial delay for a fast smoke run. diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml index 25372126b..8e9231450 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -37,10 +37,12 @@ experiments: # file exists (or ever will) for agnews; skips that lookup entirely. path_style: true availability: - # ORACULAR reads trainer 1-10's mobiperf_2st trace from _metadata -- - # mode here just keeps the trainer-side avl_events_* fields - # consistent with what the aggregator reads. - mode: mobiperf_2st + # Phase 1 is syn_0 (100% availability); the ORACULAR aggregator then + # reads all-available, so the sync barrier assembles the full agg_goal=10 + # cohort. Switch to a mobiperf_* trace (via --avail-trace) only for the + # Phase-2 unavailability runs. Keep trainer + aggregator on the SAME + # trace (run_sequential.sh's --avail-trace patches both). + mode: syn_0 # Batch 1: drive the simulated virtual clock instead of real sleeps. time_mode: simulated # Skip the registry's 4-18s artificial delay for a fast smoke run. diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index c1b337574..49c09e798 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -230,7 +230,15 @@ LOGDIR = env("LOGDIR"); MANIFEST = env("MANIFEST") DRY_RUN = env("DRY_RUN") == "1"; SHOW_ALL = env("SHOW_ALL") == "1" delays_on = (DELAYS == "on") -traces = [t for t in (env("TRACE_CSV") or "").replace(",", " ").split()] or [""] +# Availability trace(s). Default to syn_0 (Phase-1, 100% availability) when the +# operator passes no --avail-trace, so patch() ALWAYS sets the mode EXPLICITLY on +# every baseline (trainer availability.mode + aggregator trackTrainerAvail + +# client_notify -- lines below) rather than silently inheriting each yaml's own +# `mode:`. This is what makes the printed "trace" row match what actually runs: +# the resolved value is patched into the launched cfg, not just displayed. +_trace_raw = [t for t in (env("TRACE_CSV") or "").replace(",", " ").split()] +trace_set = bool(_trace_raw) # operator passed --avail-trace(s)? +traces = _trace_raw or ["syn_0"] # Phase-1 default: 100% availability multi_trace = len(traces) > 1 variants = {"real": 0, "sim": 1} if MODE == "both" else {MODE: (0 if MODE == "real" else 1)} @@ -328,6 +336,9 @@ for trace in traces: "n_gpus": e0.get("execution", {}).get("num_gpus"), "partition": h0.get("partition_method"), "delays": e0["trainer"].get("enable_training_delays"), + # RESOLVED availability mode read back from the PATCHED cfg (what + # actually launches), so the table can't show a stale default. + "avail": e0["trainer"].get("availability", {}).get("mode"), "async": (run_key == "fluxtune"), }) @@ -358,8 +369,18 @@ def scalar_row(label, val, overridden, note=None, review=False): tiers = [] # ① review every run -trace_overridden = any(traces) -trace_val = " ".join(traces) if trace_overridden else "syn_0" +# The trace row reflects the RESOLVED per-baseline availability (read back from +# the patched cfgs), NOT a hardcoded default -- so "what is printed" == "what +# runs". If every baseline resolved to the same mode, show it; otherwise flag +# the divergence and defer to the per-baseline table (tier ②). +trace_overridden = trace_set +_resolved_avails = {b.get("avail") for b in per_baseline.values() if b.get("avail")} +if len(_resolved_avails) == 1: + trace_val = next(iter(_resolved_avails)) +elif _resolved_avails: + trace_val = "MIXED: " + ",".join(sorted(a or "?" for a in _resolved_avails)) + " (see ②)" +else: + trace_val = " ".join(traces) # mode: single-sided always warns (parity needs both), regardless of override. if MODE != "both": mode_row = {"label": "mode", "value": MODE, "level": "warn", "note": "single-sided: parity needs both"} @@ -373,7 +394,9 @@ tier1 = {"name": "① REVIEW EVERY RUN", "rows": [ scalar_row("max_data_id_progress", MAX_DATA_ID, MAX_DATA_ID_SET, note="STOP condition: stop when data_id reaches this (--max-data-id)"), scalar_row("trace", trace_val, trace_overridden, - note=("Phase 1 is syn_0 (100% avail)" if any(t and t != "syn_0" for t in traces) else "100% availability")), + note=("resolved availability actually patched into each launched cfg; " + + ("100% availability (Phase 1)" if _resolved_avails == {"syn_0"} + else "NON-syn_0 — unavailability (Phase 2+)"))), scalar_row("enable_training_delays", str(delays_on).lower(), DELAYS_SET, note=f"modeled training delay {'ON (D>0)' if delays_on else 'OFF (D=0)'}; matched on BOTH sides — K-D8"), # var_threshold / max_iterations_per_data_id vary with data heterogeneity -> @@ -394,7 +417,7 @@ tiers.append(tier1) tier2_cols = [ ("c", "c"), ("agg_goal", "agg_goal"), ("k", "k"), ("min_init", "minInit"), ("n_trainers", "n_trainers"), - ("n_gpus", "n_gpus"), ("partition", "part"), + ("n_gpus", "n_gpus"), ("partition", "part"), ("avail", "avail"), ] overridden2 = [] if bool(SEL_C) or bool(SEL_C_ASYNC): overridden2.append("c") @@ -404,6 +427,7 @@ if bool(MIN_INIT): overridden2.append("min_init") if bool(NUM_TRAINERS): overridden2.append("n_trainers") if bool(NUM_GPUS): overridden2.append("n_gpus") if bool(PART): overridden2.append("partition") +if trace_set: overridden2.append("avail") rows2 = [] for rk in (r[0] for r in runs): b = per_baseline.get(rk, {}) @@ -411,6 +435,7 @@ for rk in (r[0] for r in runs): "c": b.get("c"), "agg_goal": b.get("agg_goal"), "k": b.get("k"), "min_init": b.get("min_init"), "n_trainers": b.get("n_trainers"), "n_gpus": b.get("n_gpus"), "partition": b.get("partition"), + "avail": b.get("avail"), }}) tiers.append({"name": "② PER-BASELINE (moderate)", "table": {"columns": tier2_cols, "rows": rows2, @@ -438,6 +463,25 @@ for rk in (r[0] for r in runs): "detail": f"agg_goal={g} > c={c} — selected trainers would be stranded"}) else: checks.append({"name": f"agg_goal <= c ({rk})", "level": "ok", "detail": f"agg_goal={g} c={c}"}) +# Availability consistency + sync-barrier liveness: surface the RESOLVED per- +# baseline trace (what actually runs), and BLOCK a full-participation sync +# barrier under a non-syn_0 trace -- agg_goal == n_trainers can never assemble if +# any trainer is unavailable, so the real barrier waits to the wall cap (the +# fwdllm_plus / K-D20 stall; Stage C's wait bounds the log but still can't +# complete when full participation is required under scarcity). +for rk in (r[0] for r in runs): + b = per_baseline.get(rk, {}) + av, g, n, is_async = b.get("avail"), b.get("agg_goal"), b.get("n_trainers"), b.get("async") + if av and av != "syn_0": + if (not is_async) and isinstance(g, int) and isinstance(n, int) and g >= n: + checks.append({"name": f"availability liveness ({rk})", "level": "error", + "detail": f"trace={av} + sync agg_goal={g} >= n_trainers={n}: " + f"barrier can't assemble under unavailability → stall"}) + else: + checks.append({"name": f"availability ({rk})", "level": "warn", + "detail": f"trace={av} — non-syn_0 unavailability (Phase 2+); confirm intended"}) + else: + checks.append({"name": f"availability ({rk})", "level": "ok", "detail": f"trace={av}"}) # (No k-vs-agg_goal check: in the random selector, send-side selection/concurrency # is driven by `c` (required_trainers = min(len(ends), c - in_use)); `k` is the # RECV-side batch size (num_ends_to_remove = min(..., self.k)), NOT a selection diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ddb7e1f25..f01866c4a 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -1326,6 +1326,20 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent (real-transport artifacts, ≥28.7 s/run) gated `if not self.simulated` — real byte-identical, sim skips pure wall overhead. Full pytest suite re-run proves grad values + cadence unchanged. +- **K-D22 Availability params respected end-to-end — Phase-1 defaults to syn_0; the pre-flight table shows the + RESOLVED config, not a hardcoded default.** Two consistency bugs in `run_sequential.sh`: (1) the "trace" row + PRINTED `syn_0` whenever no `--avail-trace` was passed, but `patch()` only sets the mode when a trace is + truthy — so with no flag the yaml's OWN `mode:` ran (fwdllm_plus `mobiperf_2st`, fluxtune `mobiperf_3st_50`) + while the table claimed syn_0 (the K-D20 stall, mislabeled). (2) The yamls' source-of-truth `mode:` was a + mobiperf trace, not syn_0. **Fixes:** the availability trace now DEFAULTS to `syn_0` (Phase-1) so `patch()` + always sets trainer `availability.mode` + aggregator `trackTrainerAvail.trace` + `client_notify.trace` + EXPLICITLY on every baseline; the table's trace row + a new per-baseline `avail` column are read BACK from the + patched cfg (what actually launches), so print == run; the 4 source yamls flipped to `mode: syn_0` (switch to a + mobiperf trace via `--avail-trace` only for Phase-2); and a new **feasibility gate BLOCKS** a full-participation + sync barrier under a non-syn_0 trace (sync `agg_goal >= n_trainers` + unavailability = can't assemble → stall; + `--force` to override). Verified: generated launch cfg carries `syn_0` in all three availability fields; the + guard blocks `--avail-trace mobiperf_2st --only fwdllm_plus` at pre-flight (exit 2). + --- ## §L Phase-1 sign-off remediation — async grad-path residence & carry-over (Batch 2.5, staged) From 37430c184643678a4033d2ae22bf2e1f2f61c81f Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 16:43:30 -0400 Subject: [PATCH 18/56] fwdllm sim: Phase 1 speedup instrumentation + Phase 3 config-flow fan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-sign-off work (simulate_fwdllm.md §H). The 2026-07-04 sign-off run was unusable: every sim hit [SIM_WALL_CEILING] with vclock << wall (sim_rate~0.37x = a SLOWDOWN, root #13), real/sim reached mismatched data_id. Roots traced to (a) the sim still paying real transport waits it must skip (#13) and (b) the config flow (#12). This lands the two prerequisite phases; Phase 2 (the leak fix) and Phase 4 (stopping rule) remain. Phase 1 — speedup instrumentation (make the slowdown observable): - fwdllm agg_round now emits wall_elapsed_s (both modes) + sim_rate=vclock/wall (sim), and a live throttled [VCLOCK_PROGRESS] log (the inherited base log never fired on fwdllm's composer path). - new sim_speedup DIAG parity rung asserts the principle-#13 invariant (sim_rate >= 1) and reports wall_speedup = real_wall/sim_wall. K7 sim_rate only checks a sane range and silently passes a slowdown; this catches it. Validated on the banked 134801 sim: sim_rate=0.377, wall_speedup=0.986. - Phase 1d PDF plot deferred (metric already in telemetry+log+rung). Phase 3 — config-flow correctness (#12, framework fan chosen): - runner._build_aggregator_config fans exp.trainer.enable_training_delays (+training_delay_factor) into the aggregator hyperparameters as the final layer (single source, both roles agree), with a honored-100% tripwire. Safe for async_cifar10: training_delay_enabled is read only by the trainer, never the aggregator base, so the fanned field is truthful-but-inert. - effective delay flags banked in execution_config (were absent -> a --delays run "looked" off post-hoc). - run_sequential.sh --delay-factor knob (default trainer_base=10 shrinks the 4-18s registry delay /10; pass 1 for the full modeled delay). Principles #12 (consult async_cifar10 PARITY.md vclock rules) and #13 (vclock is virtual wall-time; sim must produce speedup, vclock >= physical-wall) locked in the doc, plus the phased plan and a Phase-2 cold-start leak inventory. Tests: launch 120, parity engine 208 (incl. async_cifar10), fwdllm/telemetry mode 228 — all green. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 55 ++++ .../async_cifar10/scripts/parity_checks.py | 1 + .../fwdllm/expt_scripts/run_sequential.sh | 14 +- lib/python/examples/fwdllm/simulate_fwdllm.md | 234 +++++++++++++++++- .../launch/execution_config_generator.py | 10 + lib/python/flame/launch/runner.py | 40 +++ .../horizontal/syncfl/fwdllm_aggregator.py | 35 +++ .../tests/launch/test_config_generator.py | 36 +++ lib/python/tests/launch/test_runner_paths.py | 46 ++++ .../tests/mode/test_fwdllm_agg_telemetry.py | 50 ++++ lib/python/tests/mode/test_parity_checks.py | 56 +++++ 11 files changed, 571 insertions(+), 6 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 22982e757..138b9985a 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -2009,6 +2009,59 @@ def wall_disparity(real: dict, sim: dict) -> dict: } +def _wall_span_s(agg: dict) -> Optional[float]: + """Physical wall seconds spanned by a run's agg_round events. Prefers the + emitted `wall_elapsed_s` (measured from the re-anchored agg start, excludes + the join wait) when present; falls back to the ts epoch span.""" + evs = [e for e in agg["agg_rounds"] if e.get("event") == "agg_round"] + we = [e.get("wall_elapsed_s") for e in evs if e.get("wall_elapsed_s") is not None] + if we: + return max(we) + ts = [e["ts"] for e in evs if e.get("ts") is not None] + if len(ts) >= 2: + return max(ts) - min(ts) + return None + + +def sim_speedup(real: dict, sim: dict, min_rate: float = 0.98) -> dict: + """sim_speedup [DIAG]: the sim must be a SPEEDUP, not a slowdown (principle + #13 / §H #13). Two reported numbers: + - sim_rate = final_vclock / sim_wall (virtual-s per wall-s). The + operator's invariant is sim_rate >= 1 (vclock advances at least as fast + as physical wall). K7 `sim_rate` only checks the sane range [0.01,100], + so a 0.37x SLOWDOWN passes it silently -- this rung is the invariant. + - wall_speedup = real_wall / sim_wall (how many times faster the sim + finishes the same work than the real run; > 1 is the whole point). + DIAG: surfaces every run, does not gate the ladder (a violation is root #13, + the speedup-leak work, not a downstream ladder failure). For a real-compute + sim (fwdllm) the GPU pass is irreducible wall, so once the transport waits + are skipped sim_rate -> (gpu+D)/gpu >= 1.""" + vclock_vals = [e.get("vclock_now") for e in sim["agg_rounds"] + if e.get("vclock_now") is not None] + sim_wall = _wall_span_s(sim) + real_wall = _wall_span_s(real) + if not vclock_vals or not sim_wall or sim_wall <= 0: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no sim vclock_now or zero sim wall span"} + final_vclock = max(vclock_vals) + sim_rate = final_vclock / sim_wall + wall_speedup = (real_wall / sim_wall) if (real_wall and sim_wall > 0) else None + ok = sim_rate >= min_rate + return { + "ok": ok, # DIAG but ok reflects the #13 invariant so it shows red + "tier": "DIAG", + "sim_rate": round(sim_rate, 4), + "is_speedup": sim_rate >= min_rate, + "wall_speedup": round(wall_speedup, 3) if wall_speedup is not None else None, + "final_vclock_s": round(final_vclock, 1), + "sim_wall_s": round(sim_wall, 1), + "real_wall_s": round(real_wall, 1) if real_wall else None, + "min_rate": min_rate, + "note": ("SLOWDOWN — sim_rate < 1, the sim is broken (root #13)" + if not ok else "speedup healthy"), + } + + def overlap_factor(real: dict, sim: dict, tol: float = 0.3) -> dict: """K4 [DIAG]: async overlap factor diagnostic. @@ -4078,6 +4131,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["per_round_advance"] = per_round_advance_parity(real_agg, sim_agg) results["throughput"] = throughput_parity(real_agg, sim_agg) results["wall_disparity"] = wall_disparity(real_agg, sim_agg) + results["sim_speedup"] = sim_speedup(real_agg, sim_agg) # ── Stage 2 Availability ── results["avail_composition"] = avail_composition_parity(real_agg, sim_agg) @@ -4203,6 +4257,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "per_round_advance": {"stage": 1, "role": "EMERGENT", "deps": ("overhead_residual",)}, "throughput": {"stage": 1, "role": "EMERGENT", "deps": ("per_round_advance",)}, "wall_disparity": {"stage": 1, "role": "DIAG", "deps": ("throughput",)}, + "sim_speedup": {"stage": 1, "role": "DIAG", "deps": ("sim_rate",)}, # ── Stage 2 Availability ── "avail_composition": {"stage": 2, "role": "MECHANISM", "deps": ()}, "eligibility": {"stage": 2, "role": "MECHANISM", "deps": ("avail_composition",)}, diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index b3015ca7f..cfa698c1c 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -79,6 +79,7 @@ per_round_advance_parity, overlap_factor, wall_disparity, + sim_speedup, total_commits_parity, terminal_state_parity, budget_not_cap, diff --git a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh index 49c09e798..dcafd9dbb 100755 --- a/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh +++ b/lib/python/examples/fwdllm/expt_scripts/run_sequential.sh @@ -107,6 +107,7 @@ AVAIL_TRACES="" PARTITION_METHOD="" VAR_THRESHOLD="" # variance-pass gate threshold; varies with data heterogeneity -> review every run MAX_ITER_PER_DATA_ID="" # force-commit cap (max_iterations_per_data_id); review every run +DELAY_FACTOR="" # training_delay_factor: divides the registry 4-18s delay. Default (trainer_base) is 10 (=> 0.4-1.8s); pass 1 for the FULL modeled delay (simulate_fwdllm.md #12). Fans to BOTH roles via runner.py. ONLY="" AFTER="" # comma list of post-launch hooks: parity,sanity,plot (see after_* below) DRY_RUN=0 @@ -118,7 +119,7 @@ usage() { echo "usage: $0 [--mode sim|real|both] [--delays on|off] [--max-runtime-s S] [--max-data-id N]" >&2 echo " [--num-trainers N] [--num-gpus N] [--c C] [--c-async C] [--k K] [--agg-goal N]" >&2 echo " [--min-initial-trainers N] [--partition-method NAME]" >&2 - echo " [--var-threshold F] [--max-iter-per-data-id N]" >&2 + echo " [--var-threshold F] [--max-iter-per-data-id N] [--delay-factor F]" >&2 echo " [--avail-trace NAME | --avail-traces N1,N2] [--only n1,n2] [--stop-on-fail]" >&2 echo " [--dry-run] [--yes] [--force] [--show-all]" >&2 exit 2 @@ -142,6 +143,7 @@ while [[ $# -gt 0 ]]; do --partition-method) PARTITION_METHOD="$2"; shift 2 ;; --var-threshold) VAR_THRESHOLD="$2"; shift 2 ;; --max-iter-per-data-id) MAX_ITER_PER_DATA_ID="$2"; shift 2 ;; + --delay-factor) DELAY_FACTOR="$2"; shift 2 ;; --only) ONLY="$2"; shift 2 ;; --after) AFTER="$2"; shift 2 ;; --stop-on-fail) STOP_ON_FAIL=1; shift ;; @@ -204,7 +206,7 @@ MODE="$MODE" DELAYS="$DELAYS" MAX_RUNTIME_S="$MAX_RUNTIME_S" MAX_DATA_ID="$MAX_D NUM_TRAINERS="$NUM_TRAINERS" NUM_GPUS="$NUM_GPUS" SEL_C="$SEL_C" SEL_C_ASYNC="$SEL_C_ASYNC" \ SEL_K="$SEL_K" AGG_GOAL="$AGG_GOAL" MIN_INIT_TRAINERS="$MIN_INIT_TRAINERS" \ PARTITION_METHOD="$PARTITION_METHOD" TRACE_CSV="$TRACE_CSV" GPUS_VISIBLE="$GPUS_VISIBLE" \ -VAR_THRESHOLD="$VAR_THRESHOLD" MAX_ITER_PER_DATA_ID="$MAX_ITER_PER_DATA_ID" \ +VAR_THRESHOLD="$VAR_THRESHOLD" MAX_ITER_PER_DATA_ID="$MAX_ITER_PER_DATA_ID" DELAY_FACTOR="$DELAY_FACTOR" \ MODE_SET="$MODE_SET" DELAYS_SET="$DELAYS_SET" MAX_RUNTIME_S_SET="$MAX_RUNTIME_S_SET" MAX_DATA_ID_SET="$MAX_DATA_ID_SET" \ LOGDIR="$LOGDIR" MANIFEST="$MANIFEST" RUN_TSV="$RUN_TSV" DRY_RUN="$DRY_RUN" SHOW_ALL="$SHOW_ALL" \ EXAMPLE_DIR="$EXAMPLE_DIR" AC10_DIR="$AC10_DIR" \ @@ -222,6 +224,7 @@ SEL_C = env("SEL_C") or ""; SEL_C_ASYNC = env("SEL_C_ASYNC") or ""; SEL_K = env( AGG_GOAL = env("AGG_GOAL") or ""; MIN_INIT = env("MIN_INIT_TRAINERS") or "" PART = env("PARTITION_METHOD") or "" VAR_THRESHOLD = env("VAR_THRESHOLD") or ""; MAX_ITER = env("MAX_ITER_PER_DATA_ID") or "" +DELAY_FACTOR = env("DELAY_FACTOR") or "" # "was it passed on the command line?" (override -> green) for the defaulted flags MODE_SET = env("MODE_SET") == "1"; DELAYS_SET = env("DELAYS_SET") == "1" MAX_RUNTIME_S_SET = env("MAX_RUNTIME_S_SET") == "1"; MAX_DATA_ID_SET = env("MAX_DATA_ID_SET") == "1" @@ -261,6 +264,13 @@ def patch(exp, run_key, variant, trace): h["max_data_id_progress"] = MAX_DATA_ID # enable_training_delays: SAME on both sides of a pair (K-D8). exp["trainer"]["enable_training_delays"] = delays_on + # training_delay_factor (simulate_fwdllm.md #12): divides the registry 4-18s + # delay (trainer_base default 10 => 0.4-1.8s). Set it on the TRAINER + # hyperparameters; runner.py fans the same value into the aggregator so both + # roles agree. Only patched when explicitly passed (else the base default). + if DELAY_FACTOR: + exp["trainer"].setdefault("hyperparameters", {}) + exp["trainer"]["hyperparameters"]["training_delay_factor"] = float(DELAY_FACTOR) if PART: h["partition_method"] = PART exp["trainer"]["config_overrides"]["hyperparameters"]["partition_method"] = PART diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index f01866c4a..fb57822a5 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -430,6 +430,33 @@ from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **M roots first; when a fluxtune fix or learning could apply to the sync baselines (e.g. the #6/#1 shared time-base question, a selector-ignores-hold-set bug, a duration-input anchor), note it in §B.1/§K and check the others from THEIR banked telemetry before assuming it's async-only. +12. **Consult async_cifar10/PARITY.md's vclock principles BEFORE touching any sim/vclock behavior.** The virtual + clock is a shared flame-core capability that async_cifar10 already exercises correctly across sync AND async; + its rules are settled there. Before changing how the sim clock advances/increments/orders in any scenario, + read the relevant PARITY.md principle and mirror it — deviate only with a logged §K rationale (DESIGN + PRINCIPLE at the top). The settled rules (anchors verified 2026-07-04): + - **Clock is a monotone `max`.** `VirtualClock.advance(ts)` moves forward only (`flame/sim/virtual_clock.py:32-37`); + `_advance_sim_clock(sct)` = `vclock = max(vclock, sct)` (`syncfl/top_aggregator.py:331-343`). K1 + `sim_commit_order_monotone` asserts `vclock_now` non-decreasing. + - **NEVER put overhead on the vclock.** `_sim_commit_overhead_s` defaults 0.0; PARITY.md dead-end forbids >0 + ("masks & drifts; clock must `= max(vclock, sct)`"). K3b `overhead_residual` catches an uncharged/overcharged term. + - **sct = `sim_send_ts + max(gpu, D) + leg`** (`async_cifar10/.../main.py:831-845`); the `leg` goes on the vclock + but NOT on `trainer_speed_s`/utility/gate inputs (keep those pure compute). + - **sync charges MAX-of-K sct; async charges the K-th-fastest** (streaming) — the one real sync/async increment + difference (PARITY.md K3a). Starvation = a clock JUMP to the next availability event, never a spin. + - **The sim SKIPS real waits and reconstructs order from sct** (`SimReorderBuffer`, `virtual_clock.py:52-60`); + `mqtt_fetch_s` (re-selection wait) and `realDistributeSettleSeconds` are deliberately OFF the vclock. +13. **The vclock is virtual wall-time; the sim MUST produce SPEEDUP (vclock ≥ physical-wall-elapsed).** (Operator's + model, locked.) If a trainer's work is 10 real wall-seconds, the vclock advances 10s while the sim executes it in + far less wall — same events, same order, same computation (fidelity preserved), but faster in time (the whole + reason the sim exists). Therefore, **for a healthy sim, `vclock.now ≥ physical_wall_elapsed` at all times**, and + `sim_rate = vclock/wall ≥ 1` and should GROW. A `sim_rate < 1` (as in the 2026-07-04 runs, 0.37×) means the sim is + a slowdown = BROKEN: either it under-charges the vclock (small/zero D) or it fails to skip a real wait (the leak), + or both. **fwdllm nuance:** the forward-grad "train" is a REAL GPU pass (~2s) that MUST run in sim for grad + mode-invariance — that GPU wall is irreducible. So sim wall ≈ gpu + (skippable transport); speedup comes from + skipping the transport/inter-round waits, NOT from skipping compute. `sim_rate` is the top-line health metric to + emit and drive EVERY run — async_cifar10 already logs it (`[VCLOCK_PROGRESS]`, `syncfl/top_aggregator.py:1189-1203`; + K7 `sim_rate_ok`); fwdllm must too (root S3 / issue #13). --- @@ -466,6 +493,57 @@ so what we keep is (1) the current per-baseline state table, (2) the **open root ledger (what we tried or believed that was wrong, so we don't retry it). Per-run detail lives in the JSON + git history; the as-built rationale lives in §K.* +**Status (2026-07-04 PM → the K-D21 Phase-1 sign-off run RAN but every pair is UNUSABLE — all 6 processes hit +the 600s WALL, none reached a natural stop, and no `parity_*.json` was generated.** `bash run_sequential.sh +--mode both --delays on --max-runtime-s 600` produced `run_20260704_133627..143445` (real+sim × 3 baselines). +Two upstream mis-configs made the run non-comparable (analysis-from-telemetry, principle #11a — NOT re-run): +> - **(root S1 — the "timeout") Every SIM died on `[SIM_WALL_CEILING]`, not the intended vclock budget.** The sim +> does REAL forward-grad GPU compute for every dispatched trainer, so physical wall (566–588s) hit the 600s +> ceiling while the vclock only reached **fwdllm 213 / fwdllm_plus 221 / fluxtune 85 s** (0.14–0.37× of the 600 +> budget). Piece A set `sim_wall_ceiling_s = max_runtime_s = 600`, so for a real-compute sim (wall ≫ vclock by +> construction, #6/#9) the ceiling ALWAYS fires before the vclock-budget stop it was paired with — exactly what +> piece A's own comment warned ("a vclock-bounded sim over-runs to the wall ceiling"). Real stopped on +> `max_runtime_s` wall too. So the two sides stop at DIFFERENT progress and cannot be compared: +> | baseline | real rounds·data_id·wall/rnd | sim rounds·data_id·vclock(wall)·wall/rnd | stop | +> |---|---|---|---| +> | fwdllm | 36 · **13** · 15.5s | 66 · **23** · 213s(566s) · 8.6s vc-3.2s | real max_runtime / sim CEILING | +> | fwdllm_plus | 8 · **4** · 59.7s | 69 · **24** · 221s(588s) · 8.5s vc-3.2s | real max_runtime / sim CEILING | +> | fluxtune | 133 · **15** · 3.6s | 103 · **9** · 85s(579s) · 5.6s vc-0.8s | real max_runtime / sim CEILING | +> - **(root S2 — CORRECTED after codepath trace) `--delays on` WAS honored by the trainer, but D was tiny (0.4–1.8s, +> not the 4–18s registry delay) and the aggregator sim-folds were never wired.** The literal "delays didn't reach +> the trainer" was WRONG: real trainers slept 0.4–1.8s (`FedSgdTrainer.py:530`), sim trainers modeled 1.0–1.3s and +> charged it to the vclock (`:524`). Three real gaps: (1) `training_delay_factor=10` (hardcoded +> `configs/trainer_base.yaml:96`) silently divides the registry 4–18s delay ÷10 → 0.4–1.8s; (2) the aggregator's +> logged `training_delay_enabled=False` is an **orphaned field the fwdllm aggregator never reads** (the sim D flows +> from the trainer-reported `sim_completion_ts`, not an agg hyperparam) — a misleading artifact; (3) the sim sct-model +> folds (`sim_model_eval_time`, `sim_straggler_spread_s`) were never patched by the launcher (they live only in +> `aggregator.config_overrides.hyperparameters`, which `--delays` never writes); (4) the banked +> `execution_config.yaml` OMITS `enable_training_delays`, so post-hoc the run looked off. Reframed **open issue #12**. +> NOTE: even a correct large D does NOT by itself give speedup — see root S3. +> - **(root S3 — THE headline: the sim delivers a SLOWDOWN, not speedup — it still pays real waits it must skip).** +> The whole point of the sim is speedup: do the SAME events in the SAME logical order but advance the vclock +> (virtual wall-time) FASTER than physical wall by not waiting. Measured, the sim runs at **sim_rate ≈ 0.37×** +> (vclock 3.2 v-s/round vs physical wall 8.6 s/round) — SLOWER than wall. Root (codepath trace): the sim trainer +> still physically blocks on `await_join()` + real MQTT `recv_wrapper` every round (`fwdllm_trainer.py:213,218`, +> `mqtt_fetch_s`~1.8s steady / ~18s round 1), the aggregator still pays grace-bounded `recv_fifo` (≥2.0s floor, +> `syncfl/top_aggregator.py:347`) + `await_join` on distribute (`fwdllm_aggregator.py:2280/2427`), and ungated +> `time.sleep(1)` backoffs — **none gated `and not self.simulated`**. Real GPU compute (~2s) is irreducible in sim +> (needed for grad mode-invariance), so speedup REQUIRES skipping the transport waits (async_cifar10's base does; +> fwdllm's per-round sync trainer fetch does not). The invariant to restore: **vclock ≥ physical-wall-elapsed** +> (operator's model). New **open issue #13**; tracked via a `sim_rate` metric fwdllm does not yet emit. +> - **fwdllm_plus real is pathologically slow even at syn_0 (no scarcity): 59.7 s/round (8 rounds, data_id 4)** vs +> fwdllm real 15.5 s/round — ~4×. This is NOT the Stage-C scarcity liveness (#7, syn_0 has 100% avail); it is the +> `reselect_each_iteration=True` per-iteration re-selection + oracular read overhead (29 selection events vs +> fwdllm's 1). Confirms the "why is fwdllm_plus real slower/round" half of #7 as a distinct real-config cost, and +> guarantees fwdllm_plus real can never reach the sim's data_id in a wall budget (#4 real-side truncation). +> **Principled fix (design, not yet implemented — awaiting go-ahead):** for a valid parity/sign-off run the +> stopping rule must be a **matched committed-`data_id` target** (finite `--max-data-id`, e.g. 10), NOT a wall or +> vclock budget — a time budget desyncs wall-bound real from ceiling-truncated sim (the overnight 10-`data_id` +> runs passed for exactly this reason). Couple that with: (1) fix the `--delays on` wiring so D>0 reaches BOTH the +> trainer sleep AND the sim sct model (#12); (2) decouple `sim_wall_ceiling_s` from the vclock budget (generous +> multiple / separate `--max-wall-s`) so the ceiling is an outer safety, not the primary stop; (3) resolve +> fwdllm_plus real slowness (#7) so its real side can reach the matched data_id. See open roots S1/#12 below. + **Status (2026-07-05 → overnight 10-`data_id` grounding runs BANKED; parity read below (K-D18). The fluxtune R1 residence regression (K-D17b) is STILL the top blocker — do it first (fluxtune ran real-only overnight, so its sim is unchanged from run `020254`, R1=44.7%).** New findings from the 10-`data_id` runs (bigger than the @@ -543,7 +621,12 @@ PRE-K-D16 baseline the re-run is measured against (the K-D16 run is void — it *Discipline: keep exactly two columns per baseline — the latest parity JSON and the one before it — so drift is visible without a growing log. `run_parity.py` regenerates both from the banked dirs.* -| baseline | penultimate (run · pass/fail/skip) | LATEST (run · pass/fail/skip) | Δ | failing rungs (latest) | +*NOTE: the 2026-07-04 PM sign-off run (`run_20260704_133627..143445`) produced NO parity JSON — every sim was +wall-ceiling-truncated and real/sim reached mismatched `data_id` (root S1 above), so a parity read would be +meaningless. The LATEST column below is still the last VALID banked pair (the overnight 10-`data_id` runs); it +does not advance until a matched-`data_id` sign-off run is banked.* + +| baseline | penultimate (run · pass/fail/skip) | LATEST valid (run · pass/fail/skip) | Δ | failing rungs (latest) | |---|---|---|---|---| | **fwdllm** | `0703_224641` (3 `data_id`) · 37/5/32 | `0704_022610` (10 `data_id`) · **37/5/32** | =0 | field_coverage, throughput, total_commits, terminal_state, failsafe | | **fwdllm_plus** | `0703_225614` (3 `data_id`) · 30/11/32 | `0704_033206` (10 `data_id`, real stalled) · **31/10/32** | +1 | above + eligibility, selection_detail, v1, v2, g2_grad_pool_size | @@ -576,8 +659,11 @@ summary. Detailed rationale lives in the "Open roots" prose below + §K.* | # | issue (1 sentence) | baseline(s) | principled fix / next step | |---|---|---|---| -| **#6** ⭐ | sim vclock under-models real wall — 3.0 vclock-s vs 13.4 wall-s/round. **Decomp (K-D20): only 8.7 s/round is GENUINE** (gap ~2.9×, not 4.5×); 4.7 s/round is harness artifact | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **PRIORITY.** Fold the two genuine unmodeled terms into `sct`: **`eval_s`** (+3.34 s/round, server eval) + **widen the barrier straggler spread** to real `trainer_speed_s` (+2.3 s/round). Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). Recurring sanity = `\|real_wall − sim_vclock\|/data_id`→~0. Piece A (vclock-budget stop) **landed**. | -| **#7** | fwdllm_plus real STALLED — 5 agg-rounds then dead-spun to the 1 h cap (log ballooned to 292 MB) | fwdllm_plus | **ROOT-CAUSED (K-D20):** oracular `mobiperf_2st` drops avail to 1/10 < `agg_goal=10`; real sync barrier can't assemble the cohort and spins (`wait_until_next_avl=False`). Genuine unavailability exposing a **real-mode liveness bug** (sim jumps its vclock past it → completes). Fix real-mode: relax goal to available-count OR honor `wait_until_next_avl`. Not a sim bug. | +| **#13** ⭐⭐ | **sim delivers a SLOWDOWN not speedup (`sim_rate`≈0.37×): the sim path still pays real waits it must skip** — trainer `await_join`+MQTT `recv_wrapper` per round (`fwdllm_trainer.py:213,218`), agg grace `recv_fifo` (≥2s floor) + `await_join` (`fwdllm_aggregator.py:2280/2427`), ungated `sleep(1)` backoffs | fwdllm, fwdllm_plus (+ fluxtune) | **THE root of S1/#6.** Gate the sim critical-path real-waits `and not self.simulated` / drive trainer progress off the vclock, so sim wall ≈ real GPU only and `vclock ≥ wall`. Mirror asyncfl base's skip pattern (principle #12/#13). Shared-base blast radius → full `tests/` (principle #9). Phase 2 of the plan below. | +| **S1** ⭐ | sign-off run unusable: every sim died on `[SIM_WALL_CEILING]` (wall≫vclock for a real-compute sim), real died on `max_runtime_s` wall, so real/sim reached mismatched `data_id` — no comparison possible | all (2026-07-04 PM) | **DOWNSTREAM of #13:** once the sim runs faster than wall (`sim_rate>1`) the vclock-budget stop becomes reachable within the ceiling. Also: **stop the parity run on a matched committed-`data_id` target** (finite `--max-data-id`), not a wall/vclock budget; **decouple `sim_wall_ceiling_s`** to an outer safety. Confirms #6, not a new mechanism bug. | +| **#12** | `--delays on` WAS honored by the trainer (real slept 0.4–1.8s; sim modeled it), but (a) `training_delay_factor=10` (`trainer_base.yaml:96`) shrank the 4–18s registry delay ÷10, (b) the agg-side sim-folds (`sim_model_eval_time`/`sim_straggler_spread_s`) were never wired from the launcher, (c) agg `training_delay_enabled` is an orphaned/misleading field, (d) banked `execution_config.yaml` omits the flag | all | **Config-flow fix (Phase 3 below).** Fan `--delays`/folds into `config_overrides.hyperparameters` for BOTH roles (mirror `agg_goal` single-source); expose `training_delay_factor` as a knob; bank the effective values + a post-launch assertion that banked config == requested flags; retire/wire the orphan. Extensible to async_cifar10 (same `runner.py`). | +| **#6** ⭐ | sim vclock under-models real wall — **CONFIRMED again by the sign-off run** (fwdllm 3.2 vclock-s vs 15.5 wall-s/round; fluxtune 0.8 vs 3.6; fwdllm_plus 3.2 vs 59.7). **Decomp (K-D20): only 8.7 s/round is GENUINE** (gap ~2.9×); 4.7 s/round is harness artifact. NOTE: measured at D≈0 (#12), so the gap is inflated vs a D>0 run | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **PRIORITY.** Fold the two genuine unmodeled terms into `sct`: **`eval_s`** (+3.34 s/round, server eval) + **widen the barrier straggler spread** to real `trainer_speed_s` (+2.3 s/round). Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). Recurring sanity = `\|real_wall − sim_vclock\|/data_id`→~0. Piece A (vclock-budget stop) **landed but is unreachable until S1+#12 fixed** (ceiling truncates first). | +| **#7** | fwdllm_plus real STALLED — 5 agg-rounds then dead-spun to the 1 h cap (log ballooned to 292 MB); **AND (new, syn_0) real is 4× slower/round even with NO scarcity** (59.7 vs fwdllm 15.5 s/round, 8 rounds/data_id 4 in 600s) | fwdllm_plus | **Scarcity half ROOT-CAUSED (K-D20) + FIXED (Stage C):** oracular `mobiperf_2st` avail 1/10 < `agg_goal=10`; real sync barrier spun. **Slowness half (NEW, syn_0):** `reselect_each_iteration=True` per-iteration re-select + oracular read (29 sel events vs fwdllm's 1) — a distinct real-config cost, profile from the banked per-phase log; it caps fwdllm_plus real's data_id below sim's (#4 real-side truncation). Not a sim bug. | | **#8** | `field_coverage` INV fail: `task_recv.sim_send_ts` absent both modes | fwdllm, fwdllm_plus | Emit `task_recv.sim_send_ts` (over-instrument) or add it to the coverage alias tuple; unblocks K6. | | **#9** | `failsafe`/K5 false-positive: sim WALL ≫ vclock (real forward-grad compute) | fwdllm, fwdllm_plus | K5 should compare sim wall vs the RUN wall budget, not the vclock, for a real-compute sim (checker fix). Partly reframed by piece A's `sim_wall_ceiling_s`. | | **#10** | 12 rungs SKIP as rigor gaps (4 advance rungs wrong-axis, 8 phase rungs missing telemetry) | fwdllm, fwdllm_plus | Re-key K3a/K3b/K3/K4 to `data_id` (shared engine → full suite); add per-phase timing telemetry (= step 2, feeds #6). | @@ -591,7 +677,147 @@ summary. Detailed rationale lives in the "Open roots" prose below + §K.* | ~~#4~~ | V1/V2/g2 short-run truncation | fwdllm (cleared), fwdllm_plus | **CLEARED for fwdllm** at 10 `data_id`s; for fwdllm_plus it's real-side truncation → folds into #7. | | ~~#5~~ | fluxtune `selection_detail` in_flight (2.7 vs 9.75) | fluxtune | **MOSTLY FIXED K-D17b** — hold the compute slot to COMMIT so `len(selected_ends)` = virtual in-flight. | -### Pre-next-run implementation plan (staged — the "definition of done" before spending another run) +### ⭐ Next implementation plan (2026-07-04, post-sign-off) — SPEEDUP + CONFIG-FLOW + PARITY, phased + +*This supersedes the "Pre-next-run implementation plan (K-D21)" below (that plan's sign-off run RAN and was +unusable — see the 2026-07-04 status block at the top of §H). The sign-off run surfaced THREE root causes that +must be fixed, pytest-gated, BEFORE any next launch (principle #11). Derived from three codepath traces +(async_cifar10 vclock principles; the bash→yaml→runtime config flow; the fwdllm sim vclock/wait machinery). +Phases 0→1 are prerequisites; Phases 2 (speedup) and 3 (config-flow) are INDEPENDENT parallel threads; Phase 4 +depends on Phase 2. Every phase ships telemetry + pytest and ends pytest-green (no run in the inner loop).* + +**PROGRESS (2026-07-04, update in place):** `✅ Phase 0` principles #12/#13 locked · `✅ Phase 1` speedup +instrumentation — `wall_elapsed_s`+`sim_rate` on agg_round, live `[VCLOCK_PROGRESS]` log, `sim_speedup` DIAG rung +(VALIDATED on the banked `134801` sim: sim_rate=0.377, wall_speedup=0.986, "SLOWDOWN"), rung+emit pytests green; +**Phase 1d PDF plot DEFERRED** (metric already in telemetry+log+rung; the analyze_run.py plotter add is low-value, +do with Phase 4) · `✅ Phase 3` config-flow (operator chose 3b framework fan + 3c factor knob) — runner.py fans +`enable_training_delays`(+factor) into the aggregator (single source, both roles agree; safe: `training_delay_enabled` +is read ONLY by the trainer, never the agg base, so async_cifar10 is behavior-neutral), honored-100% tripwire in +`_build_aggregator_config`, effective flags banked in `execution_config`, `--delay-factor` CLI, fan+bank pytests green +· ⏳ **Phase 2 (speedup leak)** and **Phase 4 (stopping rule + sct model)** NOT STARTED — Phase 2 is the prize (needs +the 2a design decision from banked `mqtt_fetch_s` telemetry first). Full `tests/` gate pending. + +**Scope decisions (which issues live / die):** +- **KEPT & PROMOTED — #13 (the sim is a slowdown, not speedup).** New headline root. Was implicit in #6; now + first-class per the operator's vclock-is-virtual-wall-time model (principle #13). This is the prize. +- **REFRAMED — S1 & #6** are now understood as *downstream* of #13: the `[SIM_WALL_CEILING]` truncation and the + "vclock ≪ wall/round" gap are the SYMPTOM of the speedup leak, not independent bugs. Fixing #13 makes the + vclock-budget stop reachable and shrinks the #6 residual to the genuine unmodeled terms (eval_s, straggler). +- **REFRAMED — #12** is a config-flow bug (Phase 3), not a "flag dropped to trainer" bug (the trainer honored it). +- **My earlier S1 "matched-`data_id` stopping rule"** is DEMOTED to a Phase-4 convenience, not the primary fix — + the real fix is #13 (make the sim fast so a time budget is honored on both sides). Kept, subordinated. + +--- + +#### Phase 0 — Principles & invariant (DONE in this doc edit; no code) +Locked **principle #12** (consult async_cifar10/PARITY.md vclock rules before any sim-clock change) and +**principle #13** (vclock = virtual wall-time; sim must produce speedup, `vclock ≥ physical_wall_elapsed`, +`sim_rate ≥ 1`). These are the acceptance criteria the rest of the plan is measured against. *No pytest — doc only.* + +#### Phase 1 — Speedup instrumentation (make the slowdown OBSERVABLE; prereq for 2 & 4) +*You cannot fix or verify #13 without a `sim_rate` metric — fwdllm emits `vclock_now` but no paired wall stamp +(principle #11b: ship the instrument with the mechanism). Mirror async_cifar10 which already has this.* +- **1a** Emit a per-round **`wall_elapsed_s`** (= `time.time() − agg_start_ts`) alongside `vclock_now` in the + fwdllm `agg_round` `extra` (`fwdllm_aggregator.py` `build_agg_round`, ~:1627-1655), so `sim_rate` is joinable + per round without external data. +- **1b** Add the **`[VCLOCK_PROGRESS] … sim_rate=…`** periodic log to the fwdllm sync/async loops (mirror + `syncfl/top_aggregator.py:1189-1203`; confirm whether fwdllm inherits/drives it — if not, wire it). +- **1c** New parity DIAG rung **`sim_speedup`** (`sim_rate = final_vclock / sim_wall`, and `wall_speedup = + real_wall / sim_wall` at matched `data_id`) in the shared engine; + re-use async_cifar10's **K7 `sim_rate_ok`** + and **K1 monotonic-vclock** for fwdllm (they exist; ensure they run on the fwdllm axis). Shared engine → + full-suite gate (principle #9). +- **1d** A speedup plot (sim_rate over `data_id`) in the run's `plots/`. +- **Pytest:** synthetic agg_round series → assert `sim_rate` computed correctly, vclock monotonic, rung PASS/FAIL + on known-fast/known-slow fixtures. **Exit:** running Phase-1 on the BANKED 2026-07-04 sim dirs reproduces + `sim_rate≈0.37×` from telemetry alone (validates the instrument before any code that changes behavior). + +#### Phase 2 — Fix the speedup leak (#13; THE core work) — make the sim skip real waits +*Goal: sim wall/round → real GPU only (~2s); `vclock ≥ wall`; `sim_rate > 1` and growing. Each sub-step is +`and not self.simulated`-gated or vclock-driven so REAL mode is byte-identical (principle #8), and shared-base +edits (`syncfl/`/`asyncfl/top_aggregator.py`) carry async_cifar10-regression risk → FULL `tests/` (principle #9).* + +> **COLD-START LEAK INVENTORY (from the 2026-07-04 codepath trace — don't re-derive).** The sim critical path +> pays these real waits (ranked by cost); each is the Phase-2 work. `agg` = `fwdllm_aggregator.py`, +> `fwd-tr` = `syncfl/fwdllm_trainer.py`, `FedSgd` = `trainer/forward_training/FedSgdTrainer.py`, +> `sync-base` = `syncfl/top_aggregator.py`, `async-base` = `asyncfl/top_aggregator.py`. +> 1. **Trainer inter-round fetch (dominant, ~1.8s steady / ~18s round-1 = `mqtt_fetch_s`):** `fwd-tr:213` +> `await_join()` + `fwd-tr:217-218` `recv_wrapper` real blocking MQTT recv — UNGATED. (2a; needs the +> barrier-vs-wire localization first.) +> 2. **Aggregator commit/barrier grace recv:** grad path `agg:756-758` `recv_fifo(..., timeout=grace)` looped to +> `RECV_TIMEOUT_WAIT_S=30` (`async-base:66`); sync barrier `sync-base:386-398`; `grace` has a **2.0s floor ×4 +> EMA** (`sync-base:347-352`). (2b.) +> 3. **Aggregator `await_join` on distribute:** `agg:2280` (sync) / `agg:2427` (async) — UNGATED. (2b.) +> 4. **Ungated `time.sleep(1)` backoffs / avail spin:** `fwd-tr:205`, `fwd-tr:230`, `FedSgd:457`. (2c.) +> +> **Already correctly skipped in sim — do NOT touch (byte-identical real; re-gating would be a no-op or a bug):** +> post-distribute settle pads `agg:2295-2297`/`agg:2434-2436` (`if not self.simulated: time.sleep(0.1)`); +> scarcity poll `_await_dispatchable_under_scarcity` early-returns in sim (`agg:2232`). And the base async +> pattern to MIRROR: starvation = a vclock JUMP to the next avail event (`async-base:648-672`), the sim SKIPS the +> real delay-sleep while charging `sct` (that's the speedup), `SimReorderBuffer` reconstructs order from `sct` +> not wall (`virtual_clock.py:52-60`). fwdllm's deviation is that its SYNC per-round trainer fetch (leak #1) has +> no async_cifar10 analog and was never gated — that's the core of #13. +- **2a (dominant leak)** The sim trainer `_fetch_weights` blocks on `await_join()` + real MQTT `recv_wrapper` + every round (`fwdllm_trainer.py:213,218`; `mqtt_fetch_s`~1.8s steady/~18s round-1). **DECISION REQUIRED** + (design, not obvious): this inter-round fetch wait must become ~free in sim WITHOUT breaking real message + transport (grads/weights are real). Options to evaluate against async_cifar10's pattern: (i) the wait is really + *barrier* wait (trainer idle until the aggregator distributes) → it should overlap in virtual time, charged to + the vclock not physical wall; (ii) on localhost the transport is ms — confirm the ~1.8s is barrier-wait not wire + time, then the fix is to not physically block the trainer between rounds. Localize from the banked `mqtt_fetch_s` + + phase telemetry FIRST (principle #11a). +- **2b** Aggregator `await_join` on distribute (`fwdllm_aggregator.py:2280/2427`) and the grace-bounded + `recv_fifo` (≥2.0s floor, `syncfl/top_aggregator.py:347-352`; `agg:756`) in the sim commit/barrier loops → + gate/shrink the grace to ~0 in sim (the sct reorder buffer, not a wall timeout, is the sim's ordering source — + principle #8 "real-transport artifact vs algorithmic property"). +- **2c** Ungated `time.sleep(1)` backoffs / avail spins (`fwdllm_trainer.py:205,230`; `FedSgdTrainer.py:457`) → + gate `and not self.simulated`. +- **Pytest:** spy/mocked-channel test asserting the sim path issues NO blocking real-wait on the critical path + (or a bounded ~0 grace); a driven test asserting `sim_rate > 1` on synthetic timings; **async_cifar10 + byte-identical** assertion for every shared-base edit. **Exit:** `sim_rate > 1` on a synthetic fwdllm round; + full `tests/` green. + +#### Phase 3 — Config-flow correctness (#12; INDEPENDENT thread) — flag → banked config → runtime, honored 100% +*Trace result: `--delays` reaches the trainer via a bespoke bridge (`runner.py:250`) but NOT the aggregator/sim +folds; there are ~10 layers that can set/override a delay value with disagreeing defaults; the effective value is +not banked. Design goal: one flag → one canonical sink → fanned to both roles, asserted post-launch, reusable by +async_cifar10 (same `runner.py`/harness).* +- **3a** Make the launcher patch write the aggregator-side knobs too: `--delays`/folds → + `exp["aggregator"]["config_overrides"]["hyperparameters"]` (`sim_model_eval_time`, `sim_straggler_spread_s`, + and `trainingDelayEnabled` if kept), symmetric to how `max_runtime_s` is already written there + (`run_sequential.sh:260`). Smallest correct fix. +- **3b** Better/extensible: fan the resolved flag into BOTH roles' `config_overrides.hyperparameters` as the last + merge layer in `runner.py` (mirror the `agg_goal` single-source pattern, `runner.py:531-543`), removing the + bespoke trainer-only bridge as a special case. **DECISION: 3a (local, fast) vs 3b (framework, extensible) — + choose per appetite; 3b is the principled one and helps async_cifar10.** +- **3c** Expose `training_delay_factor` (hardcoded `"10"` in `trainer_base.yaml:96`) as a launcher knob so + "delays on" can mean the full registry delay, not a silent ÷10. +- **3d** Bank the effective values: include `enable_training_delays`/folds in `create_execution_config`'s output; + add a **post-launch assertion** (in `expt_runner.render_and_gate` or a new `assert_banked_config`) that the + banked `execution_config.yaml` + `aggregator_config.json` equal the requested flags — BLOCK the run otherwise. +- **3e** Retire or wire the orphaned aggregator `training_delay_enabled` (the fwdllm aggregator reads it nowhere) + so the log stops misleading. +- **Pytest:** a launcher/render test that patches a config with `--delays on`, renders the yaml, loads it through + `runner.py`, and asserts both roles' runtime `Hyperparameters` reflect the flag + folds; a negative test that + the post-launch assertion BLOCKS on a mismatch. Extend the same test to an async_cifar10 config. + +#### Phase 4 — Parity stopping rule + #6 sct model (depends on Phase 2) +*Only meaningful once the sim is fast (`sim_rate>1`). Then:* +- **4a** Decouple `sim_wall_ceiling_s` from the vclock budget (generous multiple / separate `--max-wall-s`) → an + outer runaway safety, not the primary stop (which is vclock reaching `max_runtime_s`). +- **4b** For the parity comparison, stop both sides at a **matched committed-`data_id`** target (finite + `--max-data-id`), so C1/C2/V1/V2 compare at identical progress (the overnight 10-`data_id` runs passed for + exactly this reason). Demoted from "primary fix" to convenience. +- **4c** Calibrate B1/B2 folds (`eval_s` +3.34s, straggler-spread +2.3s) so `wall_disparity/data_id → ~0` (the + genuine #6 residual once the artifact waits from Phase 2 are gone). B3 WAN knob stays doc-only. +- **Pytest:** ceiling-decouple + matched-data_id stop unit tests; `wall_disparity` rung on synthetic. **Exit:** + gated by the sign-off run (below), not pytest alone. + +#### THEN (not before): the Phase-1 sign-off RUN +Only after Phases 0–4 are pytest-green + full `tests/` + shared-parity suite green: smoke → convergence per +baseline at a matched `--max-data-id`, D>0 (full factor), **with `sim_rate>1` confirmed live**. It banks: the +12 un-skipped rungs, `sim_rate`/`wall_speedup`, `wall_disparity→~0`, fwdllm_plus completing the matched data_id, +and the fluxtune R1 emergent smoke. Record in the scoreboard. + +### Pre-next-run implementation plan (staged — the "definition of done" before spending another run) — ⚠️ SUPERSEDED by the plan above (its sign-off run was unusable; see 2026-07-04 status) *Goal: land ALL dev that lets the NEXT parity run extract maximum information — un-skip the 12 rigor-gap rungs, make #6/#7 first-class in telemetry, and close the sct model — so the run measures a nearly-complete ladder, diff --git a/lib/python/flame/launch/execution_config_generator.py b/lib/python/flame/launch/execution_config_generator.py index 61afc1c2e..b1362137c 100644 --- a/lib/python/flame/launch/execution_config_generator.py +++ b/lib/python/flame/launch/execution_config_generator.py @@ -130,6 +130,16 @@ def create_execution_config( }, "battery_threshold": exp_config.trainer.battery_threshold, "time_mode": exp_config.trainer.time_mode, + # Effective training-delay config, banked so a run can be + # audited post-hoc (previously absent -> a --delays run "looked" + # off, simulate_fwdllm.md #12). training_delay_factor is the + # trainer override if set, else the trainer_base default applies. + "enable_training_delays": exp_config.trainer.enable_training_delays, + "training_delay_factor": ( + (exp_config.trainer.hyperparameters or {}).get( + "training_delay_factor" + ) + ), }, "execution": { "num_gpus": exp_config.execution.num_gpus, diff --git a/lib/python/flame/launch/runner.py b/lib/python/flame/launch/runner.py index 6bb0d2059..e625f2f62 100644 --- a/lib/python/flame/launch/runner.py +++ b/lib/python/flame/launch/runner.py @@ -542,7 +542,47 @@ def _build_aggregator_config( }, )) + # Training-delay config is a single source of truth: exp.trainer's + # flags govern the whole run. Fan them into the AGGREGATOR hyperparameters + # too (final, highest-precedence layer) so both roles agree. Previously + # only the trainer bridge (spawn config_overrides, see start_experiment) + # set training_delay_enabled, leaving the aggregator's trainingDelayEnabled + # at its pydantic default (False) -- a misleading orphan for fwdllm (whose + # sim D flows from the trainer-reported completion, not this field) and an + # actual real<->sim desync risk for examples whose aggregator DOES read it + # (async_cifar10). See simulate_fwdllm.md #12 / principle #13. + _delay_fan: dict = { + "trainingDelayEnabled": bool(exp.trainer.enable_training_delays), + } + # training_delay_factor: if the experiment set it on the trainer, fan the + # same value to the aggregator so a launcher knob (e.g. --delay-factor) + # reaches both roles from one place instead of the hardcoded per-role + # trainer_base default. + _tr_hp = exp.trainer.hyperparameters or {} + if "training_delay_factor" in _tr_hp: + _delay_fan["trainingDelayFactor"] = _tr_hp["training_delay_factor"] + layers.append(( + "experiment.trainer.training_delay (fanned to aggregator)", + {"hyperparameters": _delay_fan}, + )) + merged, provenance = merge_with_provenance(layers) + + # Honored-100% tripwire: the delay fan is the final layer, so the merged + # aggregator config MUST reflect the requested flag. If a later refactor + # reorders layers or a higher-precedence override shadows it, fail loudly + # here rather than silently running with a stale/default value (the exact + # class of bug behind simulate_fwdllm.md #12). + _eff = merged.get("hyperparameters", {}).get("trainingDelayEnabled") + _req = bool(exp.trainer.enable_training_delays) + if _eff is not None and bool(_eff) != _req: + raise ValueError( + "training-delay config did not flow to the aggregator: requested " + f"enable_training_delays={_req} but merged aggregator " + f"trainingDelayEnabled={_eff!r}. Check the config-override layer " + "order in _build_aggregator_config (simulate_fwdllm.md #12)." + ) + return merged, provenance def _create_experiment_directory(self, exp: ExperimentConfig) -> Path: diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 4975315b5..b202a5b55 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -1613,6 +1613,36 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): self._is_model_updated = False if telemetry.is_enabled(): + # Speedup instrumentation (§H issue #13 / principle #13): the sim + # must run virtual time FASTER than physical wall (sim_rate >= 1). + # fwdllm emitted vclock_now but no paired wall stamp, so the + # slowdown (sim_rate~0.37x in the 2026-07-04 runs) was invisible. + # Emit wall_elapsed_s in BOTH modes (real needs it for wall_speedup + # = real_wall/sim_wall) and sim_rate = vclock/wall in sim only. + # agg_start_time_ts is re-anchored past the join wait (syncfl base), + # so this excludes the initial ~join stall. + _wall_elapsed_s = time.time() - getattr( + self, "agg_start_time_ts", time.time() + ) + _sim_rate = None + if self.simulated and getattr(self, "_vclock", None) is not None: + _sim_rate = ( + float(self._vclock.now) / _wall_elapsed_s + if _wall_elapsed_s > 0 + else None + ) + # Live speedup log (throttled ~30s). The inherited base + # [VCLOCK_PROGRESS] lives in increment_round, which fwdllm's + # composer loop bypasses, so it never fired — emit it here. + _last = getattr(self, "_last_vclock_log_wall_ts", 0.0) + if time.time() - _last >= 30.0 and _sim_rate is not None: + logger.info( + f"[VCLOCK_PROGRESS] vclock={float(self._vclock.now):.1f}s " + f"wall={_wall_elapsed_s:.1f}s sim_rate={_sim_rate:.3f} " + f"(virtual-s/wall-s; <1 == SLOWDOWN) round={self._round} " + f"data_id={self.data_id}" + ) + self._last_vclock_log_wall_ts = time.time() try: ev, fields = build_agg_round( round_num=self._round, @@ -1652,6 +1682,11 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): "drain_tail_s": _drain_tail_s, "aggregate_fedavg_s": _aggregate_fedavg_s, "eval_s": _eval_s, + # Speedup metric (§H #13): wall in both modes; sim_rate + # = vclock/wall (sim only, None in real). sim_rate < 1 + # means the sim is a SLOWDOWN (broken, principle #13). + "wall_elapsed_s": _wall_elapsed_s, + "sim_rate": _sim_rate, }, ) telemetry.emit(ev, **fields) diff --git a/lib/python/tests/launch/test_config_generator.py b/lib/python/tests/launch/test_config_generator.py index 8686fc595..bc21a0e50 100644 --- a/lib/python/tests/launch/test_config_generator.py +++ b/lib/python/tests/launch/test_config_generator.py @@ -412,3 +412,39 @@ def test_generate_trainer_config_differs_across_trainers(self, gen): cfg_054["hyperparameters"]["avl_events_syn_20"] != cfg_005["hyperparameters"]["avl_events_syn_20"] ) + + +class TestExecutionConfigBanksDelays: + """Phase 3d (simulate_fwdllm.md #12): the effective training-delay config is + banked in the execution config so a run can be audited post-hoc. Previously + absent -> a `--delays on` run "looked" off because nothing recorded it.""" + + def _exp(self, enable, factor=None): + from flame.launch.experiment_config import ExperimentConfig, TrainerConfig + tr_hp = {"training_delay_factor": factor} if factor is not None else None + return ExperimentConfig( + name="bank_test", + trainer=TrainerConfig( + enable_training_delays=enable, hyperparameters=tr_hp + ), + ) + + def test_banks_enable_and_factor(self, tmp_path): + from flame.launch.execution_config_generator import create_execution_config + cfg = create_execution_config( + self._exp(enable=True, factor=3), + aggregator_config_path=tmp_path / "agg.json", + ) + tr = cfg["experiment"]["trainer"] + assert tr["enable_training_delays"] is True + assert tr["training_delay_factor"] == 3 + + def test_banks_disabled(self, tmp_path): + from flame.launch.execution_config_generator import create_execution_config + cfg = create_execution_config( + self._exp(enable=False), + aggregator_config_path=tmp_path / "agg.json", + ) + assert cfg["experiment"]["trainer"]["enable_training_delays"] is False + # factor unset -> banked as None (trainer_base default applies) + assert cfg["experiment"]["trainer"]["training_delay_factor"] is None diff --git a/lib/python/tests/launch/test_runner_paths.py b/lib/python/tests/launch/test_runner_paths.py index 012ace1c5..4e8df310b 100644 --- a/lib/python/tests/launch/test_runner_paths.py +++ b/lib/python/tests/launch/test_runner_paths.py @@ -11,6 +11,7 @@ ExampleConfig, ExperimentConfig, MetadataPaths, + TrainerConfig, ) from flame.launch.runner import ExperimentRunner @@ -24,6 +25,51 @@ def fake_example_dir(tmp_path): return ex +class TestTrainingDelayFan: + """Phase 3b (simulate_fwdllm.md #12): the trainer's enable_training_delays + is a single source of truth fanned into the AGGREGATOR config too, so both + roles agree. Before this, only the trainer bridge set it and the aggregator + defaulted to False -- a misleading orphan / real<->sim desync risk.""" + + def _agg_hp(self, fake_example_dir, enable, tr_hp=None, agg_overrides=None): + runner = ExperimentRunner(fake_example_dir) + exp = ExperimentConfig( + name="x", + trainer=TrainerConfig( + enable_training_delays=enable, hyperparameters=tr_hp + ), + aggregator=AggregatorConfig(config_overrides=agg_overrides), + ) + merged, _ = runner._build_aggregator_config(exp, None, None) + return merged.get("hyperparameters", {}) + + def test_enabled_fans_to_aggregator(self, fake_example_dir): + hp = self._agg_hp(fake_example_dir, enable=True) + assert hp["trainingDelayEnabled"] is True + + def test_disabled_fans_to_aggregator(self, fake_example_dir): + # The key case: without the fan the aggregator would default False + # regardless of the request; with it, it reflects the actual value. + hp = self._agg_hp(fake_example_dir, enable=False) + assert hp["trainingDelayEnabled"] is False + + def test_factor_fans_when_set(self, fake_example_dir): + hp = self._agg_hp( + fake_example_dir, enable=True, + tr_hp={"training_delay_factor": 3}, + ) + assert hp["trainingDelayFactor"] == 3 + + def test_fan_wins_over_conflicting_config_override(self, fake_example_dir): + # The fan is the final, highest-precedence layer: a stale aggregator + # config_override cannot shadow the run-wide request. + hp = self._agg_hp( + fake_example_dir, enable=True, + agg_overrides={"hyperparameters": {"trainingDelayEnabled": False}}, + ) + assert hp["trainingDelayEnabled"] is True + + class TestPathResolution: def test_defaults(self, fake_example_dir): runner = ExperimentRunner(fake_example_dir) diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index 8eebf272a..df7519b34 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -13,6 +13,7 @@ it isn't. """ +import time from datetime import timedelta import torch @@ -202,6 +203,55 @@ def test_emitted_every_cycle_regardless_of_variance_outcome(self, tmp_path): finally: telemetry.shutdown() + def test_speedup_fields_emitted(self, tmp_path): + """§H #13: agg_round carries wall_elapsed_s in both modes and, in sim, + sim_rate = vclock/wall so the slowdown is observable in telemetry.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + # Put the aggregator on the sim path with a virtual clock ahead of + # a known wall span. + agg.simulated = True + agg._vclock = type("V", (), {"now": 120.0})() + agg.agg_start_time_ts = time.time() - 60.0 # ~60 wall-s elapsed + # The sim boundary hook is exercised elsewhere; no-op it here so this + # test isolates the speedup-telemetry emission. + agg._release_sim_slots_at_agg_goal = lambda *a, **k: None + channel = _FakeChannel( + durations={"t1": timedelta(seconds=1)}, utilities={"t1": 0.1} + ) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["wall_elapsed_s"] > 0 + # vclock 120 over ~60 wall-s -> sim_rate ~2 (a speedup); must be present + assert r["sim_rate"] is not None and r["sim_rate"] > 1.0 + finally: + telemetry.shutdown() + + def test_wall_elapsed_emitted_in_real_mode_sim_rate_none(self, tmp_path): + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1"], var_good_enough=False) + # default fake is real mode (simulated=False) + agg.agg_start_time_ts = time.time() - 5.0 + channel = _FakeChannel( + durations={"t1": timedelta(seconds=1)}, utilities={"t1": 0.1} + ) + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + assert r["wall_elapsed_s"] > 0 + assert r["sim_rate"] is None # real mode has no virtual clock rate + finally: + telemetry.shutdown() + def test_staleness_computed_against_pre_cycle_model_version(self, tmp_path): """staleness = the model version this cycle aggregated against, minus each contributor's last-known trained-on version -- captured BEFORE diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index d1a77d135..71dbf2d07 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -213,6 +213,62 @@ def test_no_vclock_fails(self): assert not r["ok"] and "K10" in r.get("note", "") +class TestSimSpeedup: + """sim_speedup [DIAG] asserts the principle-#13 invariant: the sim must run + virtual time at least as fast as physical wall (sim_rate >= 1). K7 sim_rate + only checks the sane range [0.01,100], so it PASSES a slowdown — this rung is + the one that catches it (§H #13).""" + + def test_healthy_speedup_passes(self): + # sim: vclock 0..100 virtual-s in 0..10 wall-s (10x speedup); + # real: same work took 0..100 wall-s. + sim = _agg(agg_rounds=[ + _round(r, ["a"], [0], vclock=float(r * 10), ts=float(r)) + for r in range(1, 11)]) + real = _agg(agg_rounds=[ + _round(r, ["a"], [0], ts=float(r * 10)) for r in range(1, 11)]) + r = pc.sim_speedup(real, sim) + assert r["ok"] and r["is_speedup"], r + assert r["sim_rate"] >= 1.0 + assert r["wall_speedup"] > 1.0 # sim finished faster than real + + def test_slowdown_fails(self): + # The 2026-07-04 bug shape: vclock reaches only ~213 while wall burns + # ~566 (sim_rate ~0.38 < 1) — a SLOWDOWN, sim is broken (root #13). + sim = _agg(agg_rounds=[ + _round(r, ["a"], [0], vclock=float(r * 213.0 / 24), + ts=float(r * 566.0 / 24)) + for r in range(1, 25)]) + real = _agg(agg_rounds=[ + _round(r, ["a"], [0], ts=float(r * 558.0 / 14)) + for r in range(1, 15)]) + r = pc.sim_speedup(real, sim) + assert not r["ok"] and not r["is_speedup"], r + assert r["sim_rate"] < 1.0 + assert "SLOWDOWN" in r["note"] + + def test_prefers_wall_elapsed_s_over_ts_span(self): + # When agg_round carries wall_elapsed_s (the re-anchored measure), it is + # used instead of the ts epoch span. + sim_rounds = [ + _round(r, ["a"], [0], vclock=float(r * 10), ts=float(r * 1000 + r)) + for r in range(1, 11)] + for e in sim_rounds: + e["wall_elapsed_s"] = float(e["round"]) # 1..10, unlike the ts span + sim = _agg(agg_rounds=sim_rounds) + real = _agg(agg_rounds=[ + _round(r, ["a"], [0], ts=float(r * 10)) for r in range(1, 11)]) + r = pc.sim_speedup(real, sim) + assert r["sim_wall_s"] == 10.0 # from wall_elapsed_s, not the huge ts span + assert r["sim_rate"] >= 1.0 + + def test_no_vclock_skips(self): + real = _agg(agg_rounds=[_round(1, ["a"], [0], ts=10.0)]) + sim = _agg(agg_rounds=[_round(1, ["a"], [0], ts=5.0)]) # no vclock + r = pc.sim_speedup(real, sim) + assert r.get("status") == "SKIP" + + class TestPerRoundAdvanceParity: def test_matched_passes(self): # Both advance 10s per round From 5fce3811e3304d79efa58b188f7dab124c2d5063 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 17:20:19 -0400 Subject: [PATCH 19/56] fwdllm sim: Phase 2 (speedup-leak skips) + Phase 4 (ceiling decouple + sct folds) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pre-next-run dev for the fwdllm real+sim, telemetry-grounded (principle #11a). Phase 2 (root #13, K-D23) — sim was a slowdown because it paid per-round wall with no fidelity value. Banked `134801` telemetry + async_cifar10 map REFUTED the doc's "trainer fetch is the dominant leak" hypothesis: the trainer recv is barrier-wait overlapping irreducible agg eval+GPU (must stay). The skippable per-round wall was two ungated sleeps, gated `and not self.simulated` (fwdllm-only, real byte-identical): - trainer `pause_execution` (time.sleep(1) at the tail of every loop iter) - `_check_availability` avail-spin (would freeze the vclock in sim) Phase 4 (K-D24): - 4a (root S1): decouple `sim_wall_ceiling_s` default from 1× max_runtime_s to `max_runtime_s × SIM_WALL_CEILING_FACTOR` (=20) so a real-compute sim (wall >> vclock by construction) isn't truncated before its data_id/vclock stop; explicit override honored. - 4c (#6): enable the two genuine unmodeled sct terms in the 3 sim yamls — B1 `sim_model_eval_time` (self-calibrating on measured eval_s) and B2 `sim_straggler_spread_s=0.9`. `wall_disparity` DIAG rung drives the residual to ~0; final B2 value calibrated at the sign-off run. Tests: test_fwdllm_sim_speedup_waits.py (6, new); early-stop ceiling cases updated (+2). `tests/mode -k fwdllm` 148 green. fwdllm-only blast radius. Co-Authored-By: Claude Opus 4.8 --- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 8 ++ .../expt_scripts/fwdllm_n10_smoke_sim.yaml | 11 ++ .../fwdllm_plus_n10_smoke_sim.yaml | 8 ++ lib/python/examples/fwdllm/simulate_fwdllm.md | 91 +++++++++++++- .../trainer/forward_training/FedSgdTrainer.py | 9 ++ .../horizontal/syncfl/fwdllm_aggregator.py | 29 ++++- .../mode/horizontal/syncfl/fwdllm_trainer.py | 10 +- .../mode/test_fwdllm_early_stop_conditions.py | 36 +++++- .../mode/test_fwdllm_sim_speedup_waits.py | 119 ++++++++++++++++++ 9 files changed, 304 insertions(+), 17 deletions(-) create mode 100644 lib/python/tests/mode/test_fwdllm_sim_speedup_waits.py diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 37d4f487b..9a18faffc 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -56,6 +56,11 @@ experiments: # Uniform/IID partition for smoke tests -- avoids the niid_label # baseline default's data-skew confound while validating mechanics. partition_method: uniform + # Phase 4c (B2, #6): sim barrier straggler spread to match real's + # per-trainer completion dispersion (uniform[0,spread), spread ≈ + # real-compute-std·√12). Active with --delays on; calibrate to + # wall_disparity→~0 at the sign-off run. + sim_straggler_spread_s: 0.9 aggregator: config_template: configs/aggregator_base.json @@ -83,6 +88,9 @@ experiments: # data-id completions). max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time + # Phase 4c (B1, #6): charge the measured server eval_s onto the vclock + # so it models real wall (self-calibrating). Delay-independent. + sim_model_eval_time: true data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 partition_method: uniform diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml index 4c93ad40f..a513a81d4 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml @@ -56,6 +56,12 @@ experiments: # default adds a data-skew confound we don't want while validating # the launcher mechanics. partition_method: uniform + # Phase 4c (B2, #6): widen the sim barrier's k-th-smallest sct to + # match real's per-trainer completion dispersion (banked real compute + # std ~0.26s; uniform[0,spread) offset ⇒ spread ≈ 0.26·√12). Only + # active with --delays on (convergence run); inert at D=0. Calibrate + # to wall_disparity→~0 at the sign-off run. + sim_straggler_spread_s: 0.9 client_notify: trace: syn_0 @@ -76,6 +82,11 @@ experiments: # data-id completions). max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time + # Phase 4c (B1, #6): charge the MEASURED server eval_s (~13s/data_id -- + # the syn_0 wall dominator) onto the vclock so it models real wall and + # sim_rate approaches/exceeds 1 (self-calibrating: uses the actual + # eval wall, not a constant). Delay-independent; the point of Phase 2/4. + sim_model_eval_time: true data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 partition_method: uniform diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml index 8e9231450..45bb27a95 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -57,6 +57,11 @@ experiments: # Uniform/IID partition for smoke tests -- avoids the niid_label # baseline default's data-skew confound while validating mechanics. partition_method: uniform + # Phase 4c (B2, #6): sim barrier straggler spread to match real's + # per-trainer completion dispersion (uniform[0,spread), spread ≈ + # real-compute-std·√12). Active with --delays on; calibrate to + # wall_disparity→~0 at the sign-off run. + sim_straggler_spread_s: 0.9 aggregator: config_template: configs/aggregator_base.json @@ -74,6 +79,9 @@ experiments: # data-id completions). max_data_id_progress: 9999 # effectively unbounded; runs governed by max_runtime_s (pass --max-data-id for a short run) max_runtime_s: 1800 # ...or after 30 minutes wall time + # Phase 4c (B1, #6): charge the measured server eval_s onto the vclock + # so it models real wall (self-calibrating). Delay-independent. + sim_model_eval_time: true data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 partition_method: uniform diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index fb57822a5..158fd8999 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -659,10 +659,10 @@ summary. Detailed rationale lives in the "Open roots" prose below + §K.* | # | issue (1 sentence) | baseline(s) | principled fix / next step | |---|---|---|---| -| **#13** ⭐⭐ | **sim delivers a SLOWDOWN not speedup (`sim_rate`≈0.37×): the sim path still pays real waits it must skip** — trainer `await_join`+MQTT `recv_wrapper` per round (`fwdllm_trainer.py:213,218`), agg grace `recv_fifo` (≥2s floor) + `await_join` (`fwdllm_aggregator.py:2280/2427`), ungated `sleep(1)` backoffs | fwdllm, fwdllm_plus (+ fluxtune) | **THE root of S1/#6.** Gate the sim critical-path real-waits `and not self.simulated` / drive trainer progress off the vclock, so sim wall ≈ real GPU only and `vclock ≥ wall`. Mirror asyncfl base's skip pattern (principle #12/#13). Shared-base blast radius → full `tests/` (principle #9). Phase 2 of the plan below. | -| **S1** ⭐ | sign-off run unusable: every sim died on `[SIM_WALL_CEILING]` (wall≫vclock for a real-compute sim), real died on `max_runtime_s` wall, so real/sim reached mismatched `data_id` — no comparison possible | all (2026-07-04 PM) | **DOWNSTREAM of #13:** once the sim runs faster than wall (`sim_rate>1`) the vclock-budget stop becomes reachable within the ceiling. Also: **stop the parity run on a matched committed-`data_id` target** (finite `--max-data-id`), not a wall/vclock budget; **decouple `sim_wall_ceiling_s`** to an outer safety. Confirms #6, not a new mechanism bug. | +| **#13** ⭐⭐ | **sim delivers a SLOWDOWN not speedup (`sim_rate`≈0.37×).** Phase-2 skips **LANDED (K-D23)**: the doc's ranking was INVERTED by the `134801` telemetry + async_cifar10 map — the trainer `await_join`/`recv_wrapper` is barrier-wait that OVERLAPS irreducible agg eval+GPU (recv MUST stay), and the agg grace/`sleep(0.1)`/`await_join` were ALREADY sim-correct. The genuinely skippable per-round wall was the trainer `pause_execution` throttle + the `_check_availability` avail-spin. | fwdllm, fwdllm_plus (+ fluxtune) | **Phase-2 skips DONE (fwdllm-only, `and not self.simulated`, 146 green).** REMAINING for real-run `sim_rate>1`: **#6/Phase-4c** must fold `eval_s` (~13s/data_id, the syn_0 wall dominator) into the vclock, then the sign-off run confirms `vclock ≥ wall` live. Phase 2 exit (`sim_rate>1` on SYNTHETIC timings) is met. | +| **S1** ⭐ | sign-off run unusable: every sim died on `[SIM_WALL_CEILING]` (wall≫vclock for a real-compute sim), real died on `max_runtime_s` wall, so real/sim reached mismatched `data_id` — no comparison possible | all (2026-07-04 PM) | **FIXED (Phase 4a / K-D24):** `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` (was 1×), so the ceiling is a runaway OUTER safety, not the primary stop — a real-compute sim is no longer truncated before its vclock/data_id stop. Sign-off run should also **stop on a matched `--max-data-id`** (present: `max_data_id_progress`), not a wall/vclock budget. | | **#12** | `--delays on` WAS honored by the trainer (real slept 0.4–1.8s; sim modeled it), but (a) `training_delay_factor=10` (`trainer_base.yaml:96`) shrank the 4–18s registry delay ÷10, (b) the agg-side sim-folds (`sim_model_eval_time`/`sim_straggler_spread_s`) were never wired from the launcher, (c) agg `training_delay_enabled` is an orphaned/misleading field, (d) banked `execution_config.yaml` omits the flag | all | **Config-flow fix (Phase 3 below).** Fan `--delays`/folds into `config_overrides.hyperparameters` for BOTH roles (mirror `agg_goal` single-source); expose `training_delay_factor` as a knob; bank the effective values + a post-launch assertion that banked config == requested flags; retire/wire the orphan. Extensible to async_cifar10 (same `runner.py`). | -| **#6** ⭐ | sim vclock under-models real wall — **CONFIRMED again by the sign-off run** (fwdllm 3.2 vclock-s vs 15.5 wall-s/round; fluxtune 0.8 vs 3.6; fwdllm_plus 3.2 vs 59.7). **Decomp (K-D20): only 8.7 s/round is GENUINE** (gap ~2.9×); 4.7 s/round is harness artifact. NOTE: measured at D≈0 (#12), so the gap is inflated vs a D>0 run | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **PRIORITY.** Fold the two genuine unmodeled terms into `sct`: **`eval_s`** (+3.34 s/round, server eval) + **widen the barrier straggler spread** to real `trainer_speed_s` (+2.3 s/round). Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). Recurring sanity = `\|real_wall − sim_vclock\|/data_id`→~0. Piece A (vclock-budget stop) **landed but is unreachable until S1+#12 fixed** (ceiling truncates first). | +| **#6** ⭐ | sim vclock under-models real wall — genuine unmodeled terms are `eval_s` (server eval, the syn_0 wall dominator) + barrier straggler under-spread. | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **FOLDS ENABLED (Phase 4c / K-D24):** B1 `sim_model_eval_time:true` charges the MEASURED eval_s to the vclock (self-calibrating, no constant); B2 `sim_straggler_spread_s:0.9` widens the barrier spread — both live in the 3 sim yamls. Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). `wall_disparity` DIAG rung (data_id axis) drives the residual → ~0; **final B2 value is calibrated FROM the sign-off run** (emergent). Ceiling decoupled (4a) so the vclock-budget stop is now reachable. | | **#7** | fwdllm_plus real STALLED — 5 agg-rounds then dead-spun to the 1 h cap (log ballooned to 292 MB); **AND (new, syn_0) real is 4× slower/round even with NO scarcity** (59.7 vs fwdllm 15.5 s/round, 8 rounds/data_id 4 in 600s) | fwdllm_plus | **Scarcity half ROOT-CAUSED (K-D20) + FIXED (Stage C):** oracular `mobiperf_2st` avail 1/10 < `agg_goal=10`; real sync barrier spun. **Slowness half (NEW, syn_0):** `reselect_each_iteration=True` per-iteration re-select + oracular read (29 sel events vs fwdllm's 1) — a distinct real-config cost, profile from the banked per-phase log; it caps fwdllm_plus real's data_id below sim's (#4 real-side truncation). Not a sim bug. | | **#8** | `field_coverage` INV fail: `task_recv.sim_send_ts` absent both modes | fwdllm, fwdllm_plus | Emit `task_recv.sim_send_ts` (over-instrument) or add it to the coverage alias tuple; unblocks K6. | | **#9** | `failsafe`/K5 false-positive: sim WALL ≫ vclock (real forward-grad compute) | fwdllm, fwdllm_plus | K5 should compare sim wall vs the RUN wall budget, not the vclock, for a real-compute sim (checker fix). Partly reframed by piece A's `sim_wall_ceiling_s`. | @@ -694,8 +694,32 @@ do with Phase 4) · `✅ Phase 3` config-flow (operator chose 3b framework fan + `enable_training_delays`(+factor) into the aggregator (single source, both roles agree; safe: `training_delay_enabled` is read ONLY by the trainer, never the agg base, so async_cifar10 is behavior-neutral), honored-100% tripwire in `_build_aggregator_config`, effective flags banked in `execution_config`, `--delay-factor` CLI, fan+bank pytests green -· ⏳ **Phase 2 (speedup leak)** and **Phase 4 (stopping rule + sct model)** NOT STARTED — Phase 2 is the prize (needs -the 2a design decision from banked `mqtt_fetch_s` telemetry first). Full `tests/` gate pending. +· `✅ Phase 2` **speedup leak (root #13) — DONE, pytest-green (K-D23).** 2a decision made FROM the banked +`134801` telemetry (principle #11a): the trainer inter-round `mqtt_fetch_s` is barrier-wait realized by the +**blocking recv** (~15s at each data_id boundary OVERLAPS the aggregator's irreducible server eval ~13s + +real GPU — NOT an independent skippable sleep; recv MUST stay, it delivers the real weights for grad +mode-invariance, exactly as async_cifar10 keeps real MQTT in sim). The only additive, fidelity-free per-round +wall the sim skips is the **trainer `pause_execution` throttle** (`time.sleep(1)` at the tail of EVERY loop +iteration, "don't overwhelm mqtt" — a real-transport artifact, principle #8) + the **`_check_availability` +avail-spin** (`while UN_AVL: sleep(1)`, which would freeze the vclock in sim; avail is agg-side enforced). Both +gated `and not self.simulated`; real byte-identical; `tests/mode -k fwdllm` 146 green (+6 new, +`test_fwdllm_sim_speedup_waits.py`). **Turned out fwdllm-ONLY (no shared-base edit)** — the aggregator was +already sim-correct (`_sync_sim_recv_first_k` 2s grace not 30/90s; `sleep(0.1)` pads gated real-only, Stage E; +`await_join` kept per async_cifar10; agg `pause_execution` not in the loop), so per principle #9 the scoped +suite is the gate, not full `tests/`. **REFRAME:** the leak inventory ranked "trainer await_join+recv" as +DOMINANT (#1) and "sleep(1) backoffs" LAST (#4); the telemetry + async_cifar10 map INVERT this. **Phase 2 alone +does NOT push the REAL-run `sim_rate>1`** — at syn_0 the sim wall is eval-dominated (#6/Phase-4c folds `eval_s` +into the vclock) — its exit is `sim_rate>1` on SYNTHETIC timings + pytest, which is met. · `✅ Phase 4` +**stopping rule + #6 sct model — DEV DONE, pytest-green (K-D24); emergent calibration is the sign-off run's +job.** 4a: `sim_wall_ceiling_s` DECOUPLED from the vclock budget (default `max_runtime_s × SIM_WALL_CEILING_ +FACTOR=20`, explicit override honored) so a real-compute sim (wall ≫ vclock by construction) is no longer +truncated before its vclock/data_id stop — root S1's direct fix; `test_fwdllm_early_stop_conditions.py` 19 +green. 4b: matched committed-`data_id` stop already present (`max_data_id_progress`; yamls 9999, `--max-data-id` +governs). 4c: the B1 eval-fold (`sim_model_eval_time`, self-calibrating on the MEASURED eval_s) + B2 straggler +spread (`sim_straggler_spread_s=0.9 ≈ real-compute-std·√12`) ENABLED in all 3 sim yamls (operator chose B1+B2); +`wall_disparity` DIAG rung (already landed, keyed on the fwdllm `data_id` axis) drives the residual → ~0 at the +run. **fwdllm-only edit (4a) → `tests/mode -k fwdllm` 148 green is the gate.** **NEXT: the Phase-1 sign-off +RUN** — matched `--max-data-id`, `--delays on`, `sim_rate>1` confirmed live. **Scope decisions (which issues live / die):** - **KEPT & PROMOTED — #13 (the sim is a slowdown, not speedup).** New headline root. Was implicit in #6; now @@ -968,6 +992,17 @@ gaps). Attack #6 and #1's wall gap together — they are the same time-base ques ### Fixes landed (what worked — do not redo) +- **Phase 4 stopping rule + #6 folds (K-D24)** — 4a: `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` + (class const `SIM_WALL_CEILING_FACTOR`, explicit override honored) so a real-compute sim isn't wall-truncated + before its data_id/vclock stop (root S1). 4c: B1 `sim_model_eval_time:true` (self-calibrating eval_s fold) + + B2 `sim_straggler_spread_s:0.9` ON in all 3 sim yamls. `test_fwdllm_early_stop_conditions.py` 19 green; + `tests/mode -k fwdllm` 148. Do NOT re-derive the ceiling from 1× budget — it truncates real-compute sims. +- **Phase 2 speedup-leak skips (K-D23)** — gated the trainer per-round `pause_execution` `time.sleep(1)` + throttle and the `_check_availability` avail-spin `and not self.simulated` (fwdllm-only; real byte-identical). + The trainer `recv`/`await_join` and the aggregator grace/`await_join` are LEFT AS-IS (recv delivers the real + weights; agg already uses the 2s sim grace + real-only `sleep(0.1)` pads). `test_fwdllm_sim_speedup_waits.py` + (6), `tests/mode -k fwdllm` 146 green. Do NOT try to short-circuit the trainer recv — it's barrier-wait over + irreducible agg eval+GPU, not a skippable sleep (K-D23). Real-run `sim_rate>1` still needs #6/Phase-4c. - **Pre-next-run plan A+B+C+D+E (K-D21)** — Stage C scarcity-wait liveness; A1 trainer phase timing; A2 agg wall-decomp; A3 advance-rung re-key (shared engine, async byte-identical); A4 `wall_disparity` DIAG rung; B1/B2/ B3 sct-model folds (config-gated OFF ⇒ byte-identical); D1 `task_recv.sim_send_ts`; D2 K5 real-compute exemption; @@ -1565,6 +1600,52 @@ decision. Keep appending; do not rewrite history (supersede with a new dated ent sync barrier under a non-syn_0 trace (sync `agg_goal >= n_trainers` + unavailability = can't assemble → stall; `--force` to override). Verified: generated launch cfg carries `syn_0` in all three availability fields; the guard blocks `--avail-trace mobiperf_2st --only fwdllm_plus` at pre-flight (exit 2). +- **K-D23 Phase 2 (speedup leak, root #13) — the 2a "trainer inter-round fetch" hypothesis was REFUTED by + the telemetry; the real skippable wall was two per-round sleeps, gated fwdllm-only.** The pre-plan leak + inventory ranked the trainer's `await_join`+`recv_wrapper` fetch as the DOMINANT leak (#1) and "ungated + `sleep(1)` backoffs" LAST (#4). Localizing from the banked `134801` sim (principle #11a) + a full + async_cifar10 sim-path map INVERTED that ranking. **What the telemetry showed:** the trainer's + `mqtt_fetch_s` splits by `trainer_phase` into ~1s steady (iter `/1`) and ~15–18s at each data_id boundary + (iter `/0`); the ~15s OVERLAPS the aggregator's `agg_round` `eval_s` (~13s server eval) + real GPU — i.e. + it is **barrier-wait for irreducible real work, not an independent trainer sleep.** **2a decision:** do + NOT short-circuit the trainer `recv` — it delivers the real weights the forward-grad pass needs for grad + mode-invariance, and async_cifar10 confirms the sim keeps real MQTT recv (the map: "sim trainer physically + blocks on real MQTT for the fetch — intentional and cheap; do not expect a gate here to port"). The genuine, + additive, fidelity-free per-round wall was: **(1)** the trainer `pause_execution` throttle (`time.sleep(1)` + chained at the tail of EVERY loop iteration — `compose(): ... >> task_put_grad >> pause_exec`; comment + ":494 don't overwhelm mqtt" = a real-transport artifact, principle #8); **(2)** the `_check_availability` + avail-spin (`while UN_AVL: time.sleep(1)`, which in sim would FREEZE the vclock — sim time can't advance + while a trainer blocks — and is redundant because sim availability is enforced agg-side, mirroring + async_cifar10's gated trainer avail wait). Both gated `and not self.simulated`; real byte-identical. + **Aggregator side needed NO change** — verified already sim-correct: `_sync_sim_recv_first_k` uses the 2s + grace floor (not the 30/90s real timeout), the `sleep(0.1)` distribute pads are already gated real-only + (Stage E), `await_join` is kept (cheap, round-1, matches async_cifar10), and the agg's own + `pause_execution` is defined but NOT in its loop. **Blast radius:** fwdllm-only (`fwdllm_trainer.py` + `pause_execution` + example `FedSgdTrainer._check_availability`), so per principle #9 `tests/mode -k fwdllm` + (146 green, +6 in `test_fwdllm_sim_speedup_waits.py`) is the gate, NOT full `tests/` — the original plan + assumed shared-base edits that turned out unnecessary. **Scope note:** Phase 2 alone does NOT make the + REAL-run `sim_rate>1` — at syn_0 the sim wall is dominated by server `eval_s` (~13s/data_id), which is + #6/Phase-4c's job to fold into the vclock; Phase 2's own exit (`sim_rate>1` on SYNTHETIC timings + pytest) + is met. See §H issue #13 + the Phase-2 PROGRESS marker. +- **K-D24 Phase 4 (parity stopping rule + #6 sct model) — ceiling decoupled, eval/straggler folds enabled + for the sign-off run; final calibration is the run's job (not pytest).** Three pieces: **(4a, root S1)** + `sim_wall_ceiling_s` defaulted to `1× max_runtime_s`, which ALWAYS truncated a real-compute sim (wall ≫ + vclock by construction) before its paired vclock/data_id stop — the 2026-07-04 sign-off void. Decoupled to + `max_runtime_s × SIM_WALL_CEILING_FACTOR` (=20, class const, overridable by explicit `sim_wall_ceiling_s`) + so it is a runaway OUTER safety, not the primary stop; fwdllm-only (`_check_early_stop_conditions`), 19 + tests. **(4b)** the matched committed-`data_id` stop already existed (`max_data_id_progress`; the sign-off + run passes `--max-data-id N` as the primary stop so real↔sim compare at identical progress, NOT a + time budget — a wall/vclock budget desyncs wall-bound real from the sim). **(4c, #6)** the two GENUINE + unmodeled `sct` terms (Stage B, landed config-gated in K-D21) turned ON in the 3 sim yamls per the + operator: **B1** `sim_model_eval_time:true` charges the MEASURED server `eval_s` (~13s/data_id, the syn_0 + wall dominator) to the vclock — self-calibrating (uses the real eval wall, no magic constant), delay- + independent, and the term that actually lifts `sim_rate` toward/over 1; **B2** `sim_straggler_spread_s:0.9` + (≈ banked real-compute std 0.26s · √12, since the offset is uniform`[0,spread)`) widens the barrier's + k-th-smallest sct to real's completion dispersion — active only with `--delays on`. The `wall_disparity` + DIAG rung (A4, keyed on the fwdllm `data_id` axis) surfaces `|real_wall − sim_vclock|/data_id` every run; + the **B2 value is calibrated to drive that → ~0 FROM the sign-off run** (emergent, per the doc — do NOT + tune it blind, principle #3/#8). Blast radius: 4a is fwdllm-only → `tests/mode -k fwdllm` 148 green is the + gate. This closes all PRE-RUN dev; the Phase-1 sign-off run is next. --- diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index 8c181b50d..1973b50fa 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -454,6 +454,15 @@ def _check_availability(self): f"Trainer id {self.trainer_id} is not available to train. Waiting for it to be available" ) while self.avl_state != TrainerAvailState.AVL_TRAIN: + # Sim availability is enforced AGGREGATOR-side (ClientAvailability + # send-gate / vclock-jump to the next avail event), never by a + # trainer wall sleep: sim time cannot advance while a trainer is + # blocked on time.sleep, so this spin would freeze the virtual + # clock (root #13). The sim therefore never spins here -- it + # proceeds and lets the agg-side gate withhold. Mirrors + # async_cifar10's gated trainer avail wait; real byte-identical. + if self.simulated: + break time.sleep(1) logger.info( f"Trainer id {self.trainer_id} is back to available to train." diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index b202a5b55..81d188275 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -239,6 +239,16 @@ class TopAggregator(AsyncTopAgg): """Top level Aggregator implements an ML aggregation role.""" + # Phase 4a (root S1): the sim runs REAL forward-grad GPU + server eval, so + # its physical WALL legitimately exceeds the vclock budget by construction + # (#6/#13). A wall ceiling = 1× max_runtime_s therefore ALWAYS truncated the + # sim before its vclock reached the budget (the 2026-07-04 sign-off void). + # The ceiling is decoupled to a GENEROUS multiple of the budget so it is a + # true runaway OUTER safety, not the primary stop -- the primary stops are + # max_data_id_progress (matched-data_id, 4b) and vclock >= max_runtime_s. + # Override with an explicit `sim_wall_ceiling_s` for a tighter outer bound. + SIM_WALL_CEILING_FACTOR = 20.0 + def internal_init(self) -> None: """Initialize internal state for role.""" super().internal_init() @@ -2659,18 +2669,25 @@ def _check_early_stop_conditions(self) -> None: ) self._work_done = True return - # Sim wall failsafe: a well-behaved (correctly-clocked) sim finishes - # in <= real wall, so cap sim WALL at `sim_wall_ceiling_s` (default = - # the budget) so an under-modeled vclock (root #6) can't run away. + # Sim wall failsafe (Phase 4a, root S1): the sim does REAL GPU + + # server eval, so its WALL legitimately exceeds the vclock budget -- + # a 1× ceiling truncated every sim before the vclock stop it was + # paired with. Decoupled to `max_runtime_s × SIM_WALL_CEILING_FACTOR` + # (a runaway OUTER safety), overridable by an explicit + # `sim_wall_ceiling_s`. Primary stops stay max_data_id_progress / + # vclock >= max_runtime_s. if self.simulated: _wall = time.time() - self.agg_start_time_ts _ceil = getattr(self.config.hyperparameters, "sim_wall_ceiling_s", None) - _ceil = float(_ceil) if _ceil is not None else float(max_runtime_s) + _ceil = (float(_ceil) if _ceil is not None + else float(max_runtime_s) * self.SIM_WALL_CEILING_FACTOR) if _wall > _ceil: logger.warning( f"[SIM_WALL_CEILING] wall={_wall:.0f}s > ceiling={_ceil:.0f}s " - f"(vclock={self._vclock.now:.0f}s) -- sim slower than budget " - f"OR vclock under-modeled (root #6); stopping run." + f"(={self.SIM_WALL_CEILING_FACTOR:g}× budget {max_runtime_s}s; " + f"vclock={self._vclock.now:.0f}s) -- runaway outer safety " + f"tripped (a healthy sim stops on max_data_id/vclock first); " + f"stopping run." ) self._work_done = True diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index 509a2c31a..d7e195846 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -833,7 +833,15 @@ def reset_stat_utility(self) -> None: @timer_decorator def pause_execution(self): - time.sleep(1) + # Per-round MQTT throttle ("don't overwhelm mqtt", :494) chained at the + # tail of the trainer loop (compose(): ...>> task_put_grad >> pause_exec). + # It is a real-transport artifact with NO sim analog (principle #8): the + # sim's inter-round barrier is the blocking recv in _fetch_weights + the + # sct reorder buffer, not a wall pause. Charging 1 wall-s/round to the + # sim is pure slowdown with zero fidelity value (root #13, Phase 2). + # Gate off in sim; real mode byte-identical. + if not getattr(self, "simulated", False): + time.sleep(1) return def compose(self) -> None: diff --git a/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py b/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py index fffa0d02b..bbea82a6e 100644 --- a/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py +++ b/lib/python/tests/mode/test_fwdllm_early_stop_conditions.py @@ -56,6 +56,8 @@ def _distribute_weights_sync(self, tag, task_to_perform="train"): def _distribute_weights_async(self, tag, task_to_perform="train"): pass + # Phase 4a: the ceiling default reads self.SIM_WALL_CEILING_FACTOR. + SIM_WALL_CEILING_FACTOR = TopAggregator.SIM_WALL_CEILING_FACTOR check = TopAggregator._check_early_stop_conditions _check_early_stop_conditions = TopAggregator._check_early_stop_conditions distribute = TopAggregator._distribute_weights @@ -130,13 +132,37 @@ def test_sim_max_runtime_stops_when_vclock_reaches_budget(self): agg.check() assert agg._work_done is True - def test_sim_wall_ceiling_stops_runaway_undermodeled_vclock(self): - """The root-#6 failsafe: even with vclock below budget, a sim whose WALL - exceeds the ceiling (default = budget) stops -- an under-modeled vclock - must not run the sim forever.""" + def test_sim_wall_below_decoupled_ceiling_keeps_running(self): + """Phase 4a (root S1): the default ceiling is now a GENEROUS multiple of + the budget (SIM_WALL_CEILING_FACTOR), NOT 1×. A sim whose WALL exceeds + the budget but is well under budget×factor must KEEP running -- the sim + legitimately uses more wall than vclock (real GPU + eval). This is the + S1 truncation the old 1× ceiling caused.""" agg = _FakeAggregator( max_runtime_s=3600.0, simulated=True, vclock_now=80.0, # under budget - agg_start_time_ts=time.time() - 3601.0, # wall > ceiling + agg_start_time_ts=time.time() - 7200.0, # wall 2× budget + ) + agg.check() + assert agg._work_done is False # 7200s < 20× × 3600s ceiling + + def test_sim_wall_ceiling_stops_true_runaway(self): + """The outer safety still fires on a genuine runaway: wall beyond + budget × SIM_WALL_CEILING_FACTOR stops the sim.""" + over = 3600.0 * TopAggregator.SIM_WALL_CEILING_FACTOR + 10.0 + agg = _FakeAggregator( + max_runtime_s=3600.0, simulated=True, vclock_now=80.0, + agg_start_time_ts=time.time() - over, + ) + agg.check() + assert agg._work_done is True + + def test_explicit_sim_wall_ceiling_overrides_default(self): + """An explicit `sim_wall_ceiling_s` is honored verbatim (tighter outer + bound) -- not multiplied by the factor.""" + agg = _FakeAggregator( + max_runtime_s=3600.0, simulated=True, vclock_now=80.0, + sim_wall_ceiling_s=100.0, + agg_start_time_ts=time.time() - 101.0, # wall > explicit ceiling ) agg.check() assert agg._work_done is True diff --git a/lib/python/tests/mode/test_fwdllm_sim_speedup_waits.py b/lib/python/tests/mode/test_fwdllm_sim_speedup_waits.py new file mode 100644 index 000000000..52636997f --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_sim_speedup_waits.py @@ -0,0 +1,119 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Phase 2 (speedup leak, root #13): the simulated clock must NOT pay +real-transport wall waits that have no fidelity value -- the sim's job is to +advance the virtual clock FASTER than physical wall (sim_rate >= 1), so every +skippable per-round sleep on the sim critical path is a slowdown. + +Telemetry localization (banked run_20260704_134801 sim): the trainer's +inter-round `mqtt_fetch_s` is barrier-wait realized by the blocking recv in +_fetch_weights (which MUST stay -- it delivers the real weights the forward-grad +pass needs for grad mode-invariance; async_cifar10 keeps real MQTT recv in sim +too). The only additive, fidelity-free per-round wall the sim can skip is: + + (1) the trainer's `pause_execution` throttle -- a `time.sleep(1)` chained at + the tail of EVERY trainer loop iteration ("don't overwhelm mqtt", a + real-transport artifact, principle #8); and + (2) the `_check_availability` avail-spin -- a `while UN_AVL: time.sleep(1)` + that, in sim, would freeze the virtual clock (sim time can't advance while + a trainer blocks on time.sleep); sim availability is enforced agg-side. + +Both are gated so REAL mode is byte-identical and SIM mode never wall-sleeps. +""" + +import os +import sys + +sys.path.insert( + 0, + os.path.join( + os.path.dirname(os.path.abspath(__file__)), "..", "..", "examples", "fwdllm", + "trainer", "forward_training", + ), +) + +import FedSgdTrainer as _fst_module # noqa: E402 +from FedSgdTrainer import FedSGDTrainer # noqa: E402 +from flame.config import TrainerAvailState # noqa: E402 +from flame.mode.horizontal.syncfl import fwdllm_trainer as _tr_module # noqa: E402 + + +class _PauseTrainer: + """Minimal stand-in binding the base Trainer.pause_execution under test. + timer_decorator reads its own runtime.time (untouched here) and only a + best-effort self.fwd_llm_stage, so no other state is needed.""" + + pause_execution = _tr_module.Trainer.pause_execution + + def __init__(self, simulated): + self.simulated = simulated + + +class TestPauseExecutionGatedInSim: + def test_real_mode_pauses_one_second(self, monkeypatch): + slept = [] + monkeypatch.setattr(_tr_module.time, "sleep", lambda s: slept.append(s)) + _PauseTrainer(simulated=False).pause_execution() + assert slept == [1] # real byte-identical: the MQTT throttle stays + + def test_sim_mode_does_not_pause(self, monkeypatch): + slept = [] + monkeypatch.setattr(_tr_module.time, "sleep", lambda s: slept.append(s)) + _PauseTrainer(simulated=True).pause_execution() + assert slept == [] # root #13: no per-round wall charged to the sim + + def test_sim_default_missing_attr_is_treated_real(self, monkeypatch): + """A stand-in with no `simulated` attr (getattr default False) must keep + the real throttle -- the gate must never silently skip in real mode.""" + slept = [] + monkeypatch.setattr(_tr_module.time, "sleep", lambda s: slept.append(s)) + t = _PauseTrainer(simulated=False) + del t.simulated + t.pause_execution() + assert slept == [1] + + +class _AvailTrainer: + """Binds the real _check_availability onto a minimal stand-in.""" + + _check_availability = FedSGDTrainer._check_availability + + def __init__(self, simulated, avl_state, wait_until_next_avl=True): + self.simulated = simulated + self.avl_state = avl_state + self.wait_until_next_avl = wait_until_next_avl + self.trainer_id = "t1" + + +class TestCheckAvailabilityGatedInSim: + def test_sim_never_spins_and_proceeds(self, monkeypatch): + """UN_AVL trainer in sim: must NOT time.sleep (would freeze the vclock) + and must return True so it proceeds -- the agg-side send-gate handles + the withhold.""" + def _boom(_s): + raise AssertionError("time.sleep called on the simulated avail path") + monkeypatch.setattr(_fst_module.time, "sleep", _boom) + t = _AvailTrainer(simulated=True, avl_state=TrainerAvailState.UN_AVL) + assert t._check_availability() is True + + def test_real_mode_still_spins_until_available(self, monkeypatch): + """Real mode is byte-identical: it wall-sleeps while UN_AVL and returns + True once the trainer flips back to AVL_TRAIN.""" + t = _AvailTrainer(simulated=False, avl_state=TrainerAvailState.UN_AVL) + slept = [] + + def _sleep_then_avail(s): + slept.append(s) + t.avl_state = TrainerAvailState.AVL_TRAIN # becomes available after 1 tick + + monkeypatch.setattr(_fst_module.time, "sleep", _sleep_then_avail) + assert t._check_availability() is True + assert slept == [1] + + def test_real_unavailable_no_wait_exits(self, monkeypatch): + """wait_until_next_avl=False is unchanged in both modes: skip training.""" + monkeypatch.setattr(_fst_module.time, "sleep", + lambda s: (_ for _ in ()).throw(AssertionError("no sleep"))) + t = _AvailTrainer(simulated=False, avl_state=TrainerAvailState.UN_AVL, + wait_until_next_avl=False) + assert t._check_availability() is False From cc5569c073d4ea97e558319edaadab3859357901 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 22:21:15 -0400 Subject: [PATCH 20/56] fwdllm sim: #6 checker-anchor fix (K-D25) + #1c R1 root-cause & vclock-timeout (K-D26) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #6 (K-D25) — root-caused the clock-rate "gap" to a CHECKER-ANCHOR artifact, not a sim under-charge: the rungs compared sim-Δvclock (genuine algorithmic time) vs real-Δts (full wall = genuine + a ~constant ~7.7s/round localhost transport artifact the sim correctly omits). Aggregator now emits intrinsic_span_s (barrier + eval, mirroring the sim vclock composition; fedavg excluded); the 5 clock-rate rungs (K2/K3/K3b/K8/U2) + wall_disparity anchor real on its cumulative intrinsic clock, ts fallback -> async_cifar10 byte-identical. Validated on banked fwdllm: throughput 0.47->0.048, overhead 21->1.1, wall_disparity 90->1.6. Root B (K-D25) — phase rungs compared a mode-dependent phase: post_train now stamped AFTER the delay (real stops bundling the slept delay); B2 straggler moved from _delay_s into the sct so training_budget_s is a mode-invariant input. Validated: post_train 0/0, training_budget 1.14/1.14. #1c (K-D26) — root-caused fluxtune R1 (60.4%) to a physical-wall vs vclock desync in the shared async_oort re-pick guard, exposed by the slow sim (wall >> vclock): the 90s SEND_TIMEOUT_WAIT_S abandon-timeout keyed on time.time() evicts a still-outstanding trainer from all_selected -> re-dispatch. Corrects K-D19 (fixed the wrong RETURN path; K-D17b's NONE hypothesis was right). Fix (part a): _abandon_clock_now runs the timeout on the vclock in sim; real -> time.time() -> async_cifar10 byte-identical. Sustained NONE-delete path (part b) is the same slow-sim root -> tracked as #13. Tests: intrinsic-span anchor + emit, straggler-separation, post_train, vclock abandon-timeout; full tests/ 879 passed / 7 skipped. Doc trimmed 1817->408 lines with a standing maintenance rule; K-D25/K-D26 logged. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 122 +- lib/python/examples/fwdllm/simulate_fwdllm.md | 2148 +++-------------- .../trainer/forward_training/FedSgdTrainer.py | 36 +- .../horizontal/syncfl/fwdllm_aggregator.py | 29 + lib/python/flame/selector/async_oort.py | 33 +- .../tests/mode/test_async_sim_ordering.py | 20 + .../tests/mode/test_fwdllm_agg_telemetry.py | 27 + .../mode/test_fwdllm_trainer_phase_timing.py | 33 + .../mode/test_fwdllm_trainer_sim_duration.py | 3 + lib/python/tests/mode/test_parity_checks.py | 48 + 10 files changed, 678 insertions(+), 1821 deletions(-) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 138b9985a..d50670924 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -401,11 +401,36 @@ def _per_round_max_speed(agg_rounds: list) -> dict: return out +def _real_intrinsic_clock(agg_rounds: list) -> Optional[dict]: + """Real's GENUINE-time coordinate for the #6 clock-rate family, or None. + + When the aggregator emits ``intrinsic_span_s`` (fwdllm), returns a dict + {id(agg_round_event): cumulative_intrinsic_s} -- the running sum of per-cycle + algorithmic spans (barrier + fedavg + eval), the real analog of the sim's + vclock. Real's raw wall Δts bundles a ~constant inter-round transport ARTIFACT + (mqtt re-fetch / redistribute / drain-tail / sleeps) the sim omits by design + (principle #1), so anchoring the clock-rate rungs (K2/K3/K3b/K8/U2 + wall_ + disparity) on Δts spuriously fails #6; anchoring on this intrinsic clock + compares real-genuine vs sim-vclock like for like. Returns None when + ``intrinsic_span_s`` is absent (async_cifar10 -> callers fall back to ``ts`` + -> byte-identical).""" + evs = [e for e in agg_rounds if e.get("event") == "agg_round"] + if not any(e.get("intrinsic_span_s") is not None for e in evs): + return None + coord, run = {}, 0.0 + for e in evs: + run += (e.get("intrinsic_span_s") or 0.0) + coord[id(e)] = run + return coord + + def _per_round_advances(agg_rounds: list, use_vclock: bool) -> list: """Compute per-progress-unit time advances (Stage A3 re-key). use_vclock=True: Δvclock_now between consecutive units (sim mode). - use_vclock=False: Δts (wall) between consecutive units (real mode). + use_vclock=False: real mode -- Δ(intrinsic algorithmic clock) when the + aggregator emits ``intrinsic_span_s`` (#6 anchor, see _real_intrinsic_clock), + else Δts (wall) -> async_cifar10 byte-identical. Returns list of positive advances. Keyed on the run's TRUE progress axis (_progress_axis), not raw `round`: @@ -419,6 +444,8 @@ def _per_round_advances(agg_rounds: list, use_vclock: bool) -> list: rounds_sorted = sorted(by_round.keys()) if len(rounds_sorted) < 2: return [] + # REAL (#6): prefer real's intrinsic algorithmic clock over raw wall ts. + real_coord = None if use_vclock else _real_intrinsic_clock(agg_rounds) advances = [] for i in range(1, len(rounds_sorted)): e_prev = by_round[rounds_sorted[i - 1]] @@ -426,15 +453,15 @@ def _per_round_advances(agg_rounds: list, use_vclock: bool) -> list: if use_vclock: v_prev = e_prev.get("vclock_now") v_curr = e_curr.get("vclock_now") - if v_prev is None or v_curr is None: - continue - adv = v_curr - v_prev + elif real_coord is not None: + v_prev = real_coord.get(id(e_prev)) + v_curr = real_coord.get(id(e_curr)) else: - t_prev = e_prev.get("ts") - t_curr = e_curr.get("ts") - if t_prev is None or t_curr is None: - continue - adv = t_curr - t_prev + v_prev = e_prev.get("ts") + v_curr = e_curr.get("ts") + if v_prev is None or v_curr is None: + continue + adv = v_curr - v_prev if adv > 0: advances.append(adv) return advances @@ -1888,11 +1915,20 @@ def throughput_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: n_sim_rounds = len(sim_by_round) sim_throughput = n_sim_rounds / final_vclock if final_vclock > 0 else 0.0 - real_ts = [e["ts"] for e in real["agg_rounds"] if e.get("ts") is not None] - if not real_ts or len(real_ts) < 2: - return {"ok": True, "tier": "EXACT", "status": "SKIP", - "note": "insufficient real ts data (< 2 agg_round events)"} - wall_elapsed = max(real_ts) - min(real_ts) + # REAL (#6): denominator is real's GENUINE algorithmic time -- total + # intrinsic span (barrier+fedavg+eval) when emitted, else wall ts span + # (async_cifar10 byte-identical). Excludes real's inter-round transport + # artifact so rounds-per-genuine-second compares like-for-like vs the sim's + # rounds-per-vclock-second. + real_coord = _real_intrinsic_clock(real["agg_rounds"]) + if real_coord is not None: + wall_elapsed = max(real_coord.values()) if real_coord else 0.0 + else: + real_ts = [e["ts"] for e in real["agg_rounds"] if e.get("ts") is not None] + if not real_ts or len(real_ts) < 2: + return {"ok": True, "tier": "EXACT", "status": "SKIP", + "note": "insufficient real ts data (< 2 agg_round events)"} + wall_elapsed = max(real_ts) - min(real_ts) real_by_round = _per_progress_last_event(real["agg_rounds"], axis) n_real_rounds = len(real_by_round) real_throughput = n_real_rounds / wall_elapsed if wall_elapsed > 0 else 0.0 @@ -1969,32 +2005,38 @@ def per_round_advance_parity(real: dict, sim: dict, def wall_disparity(real: dict, sim: dict) -> dict: - """wall_disparity [DIAG]: |real_wall − sim_vclock| per matched progress unit + """wall_disparity [DIAG]: |real_genuine − sim_vclock| per matched progress unit (Stage A4 / K-D20 #6). The recurring sanity metric to drive to ~0 -- surfaces - the gap between real's per-unit WALL and the sim's per-unit VCLOCK every run, - without gating (a persistent residual is root #6, the sct-model work, not a - ladder failure). Both clocks are measured cumulatively from the FIRST matched - unit (real: ts − ts0; sim: vclock_now − v0) so a run that does not start at - unit 0 still aligns. Keyed on the run's progress axis (data_id for fwdllm, - round otherwise). Never fails.""" + the gap between real's per-unit GENUINE algorithmic time and the sim's per-unit + VCLOCK every run, without gating. Real's coordinate is the cumulative intrinsic + span (barrier+fedavg+eval) when the aggregator emits ``intrinsic_span_s`` -- + NOT raw wall, which bundles the inter-round transport artifact the sim omits + (principle #1); driving THIS residual to ~0 is the correct #6 target (chasing + real's full wall would over-charge the vclock). Falls back to wall ts for async + (byte-identical). Both clocks cumulative from the FIRST matched unit. Keyed on + the progress axis (data_id for fwdllm, round otherwise). Never fails.""" axis = "data_id" if "data_id" in (_progress_axis(sim["agg_rounds"]), _progress_axis(real["agg_rounds"])) else "round" real_by = _per_progress_last_event(real["agg_rounds"], axis) sim_by = _per_progress_last_event(sim["agg_rounds"], axis) + # REAL (#6): genuine algorithmic clock when emitted, else raw wall ts. + real_coord = _real_intrinsic_clock(real["agg_rounds"]) + _real_t = ((lambda e: real_coord.get(id(e))) if real_coord is not None + else (lambda e: e.get("ts"))) matched = [k for k in sorted(set(real_by) & set(sim_by)) - if real_by[k].get("ts") is not None + if _real_t(real_by[k]) is not None and sim_by[k].get("vclock_now") is not None] if len(matched) < 2: return {"ok": True, "tier": "DIAG", "status": "SKIP", - "note": "fewer than 2 matched units with both real ts and " + "note": "fewer than 2 matched units with both real time and " "sim vclock_now"} - ts0 = real_by[matched[0]]["ts"] + ts0 = _real_t(real_by[matched[0]]) v0 = sim_by[matched[0]]["vclock_now"] residuals, per_unit = [], {} for k in matched: - real_wall = real_by[k]["ts"] - ts0 + real_genuine = _real_t(real_by[k]) - ts0 sim_vclock = sim_by[k]["vclock_now"] - v0 - resid = abs(real_wall - sim_vclock) + resid = abs(real_genuine - sim_vclock) residuals.append(resid) per_unit[k] = round(resid, 2) mean_resid = sum(residuals) / len(residuals) @@ -2002,6 +2044,7 @@ def wall_disparity(real: dict, sim: dict) -> dict: "ok": True, # DIAG: informational, never gates the ladder "tier": "DIAG", "axis": axis, + "anchor": "intrinsic_span" if real_coord is not None else "wall_ts", "mean_abs_disparity_s": round(mean_resid, 2), "max_abs_disparity_s": round(max(residuals), 2), "n_matched_units": len(residuals), @@ -2137,8 +2180,17 @@ def total_commits_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: real_ts = [e["ts"] for e in real["agg_rounds"] if e.get("ts") is not None] if not real_ts: return {"ok": False, "tier": "EXACT", "note": "no ts in real events"} + # REAL (#6): matched-budget window on real's GENUINE algorithmic clock + # (cumulative intrinsic span) when emitted, else raw wall ts (async byte- + # identical). _real_time(e) is 0-based cumulative-intrinsic OR ts-real_t0. + real_coord = _real_intrinsic_clock(real["agg_rounds"]) real_t0 = min(real_ts) - final_real_wall = max(real_ts) - real_t0 + if real_coord is not None: + _real_time = lambda e: real_coord.get(id(e)) + final_real_wall = max(real_coord.values()) if real_coord else 0.0 + else: + _real_time = lambda e: (e["ts"] - real_t0) if e.get("ts") is not None else None + final_real_wall = max(real_ts) - real_t0 V = min(final_sim_vclock, final_real_wall) if V <= 0: return {"ok": True, "tier": "EXACT", "status": "SKIP", @@ -2152,13 +2204,13 @@ def total_commits_parity(real: dict, sim: dict, tol_rel: float = 0.05) -> dict: if axis == "round": n_sim = sum(1 for e in sim["agg_rounds"] if (e.get("vclock_now") or 0) <= V + 1e-9) n_real = sum(1 for e in real["agg_rounds"] - if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9) + if _real_time(e) is not None and _real_time(e) <= V + 1e-9) else: sim_units = _per_progress_last_event(sim["agg_rounds"], axis) real_units = _per_progress_last_event(real["agg_rounds"], axis) n_sim = sum(1 for e in sim_units.values() if (e.get("vclock_now") or 0) <= V + 1e-9) n_real = sum(1 for e in real_units.values() - if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9) + if _real_time(e) is not None and _real_time(e) <= V + 1e-9) if max(n_sim, n_real, 1) == 0: return {"ok": True, "tier": "EXACT", "note": "no commits in V window"} rel_diff = abs(n_sim - n_real) / max(n_sim, n_real) @@ -2190,8 +2242,16 @@ def terminal_state_parity(real: dict, sim: dict, real_ts_all = [e["ts"] for e in real["agg_rounds"] if e.get("ts") is not None] if not real_ts_all: return {"ok": False, "tier": "EXACT", "note": "no ts in real events"} + # REAL (#6): matched-budget window on real's GENUINE algorithmic clock (see + # total_commits_parity); falls back to wall ts for async (byte-identical). + real_coord = _real_intrinsic_clock(real["agg_rounds"]) real_t0 = min(real_ts_all) - final_real_wall = max(real_ts_all) - real_t0 + if real_coord is not None: + _real_time = lambda e: real_coord.get(id(e)) + final_real_wall = max(real_coord.values()) if real_coord else 0.0 + else: + _real_time = lambda e: (e["ts"] - real_t0) if e.get("ts") is not None else None + final_real_wall = max(real_ts_all) - real_t0 V = min(final_sim_vclock, final_real_wall) if V <= 0: return {"ok": True, "tier": "EXACT", "status": "SKIP", @@ -2208,7 +2268,7 @@ def terminal_state_parity(real: dict, sim: dict, sim_rounds_at_V = {r for r, e in sim_by_round.items() if (e.get("vclock_now") or 0) <= V + 1e-9} real_rounds_at_V = {r for r, e in real_by_round.items() - if e.get("ts") is not None and (e["ts"] - real_t0) <= V + 1e-9} + if _real_time(e) is not None and _real_time(e) <= V + 1e-9} def _trainers(agg_rounds, unit_set): ts = set() diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 158fd8999..7a6b77383 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -1,1816 +1,408 @@ -# High-Fidelity Simulator for FwdLLM -- Design & Staged Build Plan - -**Active build (branch `dg/fwdllm_sim_unavail`).** This document is the plan for building a -high-fidelity **simulated-clock** runner for the `fwdllm` example (FedFwd / forward-gradient FL) that -reaches **real<->sim parity** across the **fluxtune / fwdllm / fwdllm++** baselines, at -**100% availability (syn_0)** first, then under **unavailability (syn_20, syn_50, mobiperf)**, then -**beyond syn_0 traces**. - -> **DESIGN PRINCIPLE -- concept-parity with async_cifar10, but deviate where the workload demands it.** -> We deliberately **reuse async_cifar10's concepts and machinery** (virtual clock, sct reorder buffer, -> in-flight gate, availability substrate, the parity ladder) wherever they transfer -- that shared -> substrate is the whole point and keeps the two examples auditable against each other. **But fwdllm is a -> fundamentally different workload** (aggregates GRADIENTS not weights, endogenous variance-gated -> dynamic-K commit cadence, `data_id` progress axis, one-message-per-call grad loop, rollback across -> agg-goal cycles), so a verbatim port is sometimes WRONG. When such a fork appears we **(a) evaluate the -> options carefully and make an explicit decision** (do not silently copy async_cifar10, and do not -> silently invent something new either), **and (b) APPEND that decision + its rationale to this document** -> -- so a later reader can tell an intentional fwdllm-specific divergence from an accidental discrepancy, -> and trace any real<->sim gap back to the choice that caused it. Deviation log lives in **§K** (running, -> newest-last); the **curated at-a-glance delta table is §B.1** (kept current, unlike §K's append-only log); -> locked cross-cutting ones also surface in §F "Locked principles" / §J.4. - -**Current state -- Phase 1 Batches 1-2 COMPLETE (checker + telemetry); Phase-1 syn_0 sign-off run is NEXT.** -Batch 1 landed the two structural clock ports; Batch 2 landed the variance-cadence rung layer + its -telemetry, pytest-gated. **Batch 1 done:** trainer sim path (`time_mode`/`simulated`, additive -`_sim_completion_ts`, no-sleep on the sim path), the async grad loop on the vclock (`_sim_recv_min_grad` -sct reorder buffer + in-flight gate + agg-goal-boundary rollback cleanup), the sync barrier -(`_sync_sim_recv_first_k` first-k-smallest-sct + U6 lags), the sim launchers, and the flag-off -byte-identical regression. **Batch 2 done:** the V1-V5 / DK1-DK3 / G1-G2 rungs in the shared parity engine -(`parity/checks.py` + `run_all_parity` + `CHECK_META`), the cycle-relative cadence telemetry they read -(`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size` added to the agg_round `extra`, -**§K-D9**), and their pytest gate (known-PASS + known-broken cadence fixtures + a driven emission test); -DK3/G1 emit deferred with a logged SKIP (**§K-D10**). fluxtune oort selection-fidelity is a **validation** -concern (the selection rungs already exist in the STAGE map, §I.4) folded into the sign-off run. Build -detail is history (code is the source of truth; deviation rationale in **§K**). **Still to build:** the -**Phase-1 syn_0 sign-off run** (smoke -> convergence per baseline -> `parity_checks` full battery, §I.6); -**(Batch 3)** the availability *effect* path + `EVENT_AVAIL_CHANGE` emission (Phase 2); **(Batch 4)** full -ladder under unavailability (Phase 3). See **§E** for the batch list and **§I** for the Batch-2 map. - -**Prerequisites (read first):** -- [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) -- the parity methodology (the ladder §1, roles/ - tiers/dependency-gating, workflow policy, run-length budget, landed sim mechanisms §3). **fwdllm's - own rung catalog is PARITY.md §F** (modified rungs, the variance-cadence layer). This doc references - §F for rung *definitions* and does not duplicate them; it states the fwdllm **build deltas**. -- [async_cifar10/UNAVAILABILITY_DESIGN.md](../async_cifar10/UNAVAILABILITY_DESIGN.md) -- the availability - substrate this reuses wholesale (ClientAvailability mixin, trace-read effect path, the two flag axes, - send-gate/deliver-late-stale, two-ledger discipline, starvation self-termination, the ground-truth - fidelity rungs A6/A7/A8/K11). fwdllm inherits it via the aggregator class chain (§A). - -**Starting reality (why this is real work, not a re-label):** FedFwd has **no simulated-clock support -today** -- every launcher yaml is `time_mode: real`, and the trainer emulates delay with a real -`time.sleep` (`trainer/forward_training/FedSgdTrainer.py:481` `_emulate_training_delay`, called at -`:523`; explicit note `trainer/main.py:57-78`). The aggregator machinery for a virtual clock is -*inherited but not driven* (§A). Adding the sim clock + wiring unavailability into the -variance-gated gradient loop is the build. - -**Decisions locked (kickoff):** -- A trusted **real** wall-clock run is the reference per baseline (as in async_cifar10). Parity != tuning - sim to real; real must pass `validate_real` admissibility first. -- The parity checker **extends the shared `async_cifar10/scripts/parity` engine** -- fwdllm rungs are - *added* (PARITY.md §F is their catalog), the engine is reused. -- The fluxtune / fwdllm / fwdllm++ -> config mapping is **resolved** from the landed launcher configs - (§C); it is no longer a blocking gap. +# High-Fidelity Simulator for FwdLLM — Real↔Sim Parity + +**Active build (branch `dg/fwdllm_sim_unavail`).** A simulated-clock runner for the `fwdllm` example +(FedFwd / forward-gradient FL) that reaches **real↔sim parity** across the **fluxtune / fwdllm / fwdllm++** +baselines — at **100% availability (syn_0)** first (Phase 1), then under **unavailability** +(syn_20/50/mobiperf, Phase 2), then **beyond syn_0** (Phase 3). fwdllm has no native sim clock; the build +wires flame-core's virtual clock + availability substrate into fwdllm's variance-gated gradient loop. + +> **DESIGN PRINCIPLE — reuse async_cifar10's concepts, deviate where the workload demands.** We reuse its +> virtual clock, sct reorder buffer, in-flight gate, availability substrate, and parity ladder wherever they +> transfer. But fwdllm is a different workload (aggregates **gradients** not weights; endogenous +> **variance-gated dynamic-K** commit cadence; **`data_id`** progress axis; one-message-per-call grad loop; +> rollback across agg-goal cycles), so a verbatim port is sometimes wrong. When a fork appears, **decide +> explicitly** (don't silently copy or silently invent) and **log it in §K**. The at-a-glance delta table is +> **§B.1** (curated, current); §K is the one-line-per-decision rationale log. + +> **DOC MAINTENANCE — standing instruction (keep this doc rich but CRISP).** This is a living status doc, not +> a changelog. Every edit obeys: +> - **Fold, don't append.** A decision that has landed is **one line** (anchor + what/why). A mechanism that +> **worked** is **one sentence** in §G Fixes-landed. Something that **didn't work / was believed-then-refuted** +> goes to §H Dead-ends as one line. No multi-paragraph appends — update the relevant section **in place**. +> - **Status sections are rewritten in place, not grown.** §A (current state), the scoreboard, the open-issues +> index, and open-roots reflect the CURRENT truth only; per-run history lives in git + the parity JSONs. +> - **§K is one entry per deviation.** Keep the anchor (referenced from §B.1) + a one-line rationale once folded; +> expand only an entry that is still open/contested. +> - **Delete stale scaffolding.** Cold-start build maps, superseded plans, and finished batch checklists are +> removed once landed (the code is the source of truth), leaving a one-line pointer if still referenced. + +**Prerequisites:** [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) — the parity methodology (ladder, +roles/tiers/gating, run-length budget, landed sim mechanisms); **fwdllm's rung catalog is PARITY.md §F**. +[async_cifar10/UNAVAILABILITY_DESIGN.md](../async_cifar10/UNAVAILABILITY_DESIGN.md) — the availability substrate +fwdllm inherits via its aggregator class chain (ClientAvailability mixin, trace-read effect path, two-ledger +discipline, starvation self-termination, A6/A7/A8/K11 ground-truth rungs). --- -## §A Already on the class (inherited) vs. NOT yet wired - -The virtual clock is a **flame-core** capability. fwdllm's aggregator is -`examples/fwdllm/aggregator/FedSgdAggregator.py` -> `flame/mode/horizontal/syncfl/fwdllm_aggregator.py:: -TopAggregator(AsyncTopAgg)` -> `asyncfl.TopAggregator` -> `syncfl.TopAggregator(ClientAvailability, Role)`. -So both the vclock machinery **and** the availability substrate are already **on the instance** -- the -work is **wiring them into fwdllm's gradient loop + trainer**, not re-implementing them. - -### A.1 Aggregator spine -- clock machinery NOW DRIVEN (Batch 1) -| Mechanism (from `asyncfl/top_aggregator.py`) | Status for fwdllm | -|---|---| -| Virtual clock `_vclock`, `_advance_sim_clock` (`vclock = max(vclock, sct)`) | **DRIVEN** by the grad loop (Batch 1) | -| sct-ordered drain + in-flight gate | **DRIVEN** via purpose-built `_sim_recv_min_grad` (K-D4) / sync `_sync_sim_recv_first_k` | -| one-in-flight residence `_sim_hold_busy_slots` | **DRIVEN** at the agg-goal boundary (`_release_sim_slots_at_agg_goal`, K-D5) | -| §4.5/§4.9 sct-gated pool exclusion / carry-over (oort overlay) | available if a baseline selects via oort (fluxtune); **Batch-2 validation** | -| `ClientAvailability` substrate (trace-read gate, two ledgers, proactive evict, starvation advance) | inherited; **NOT yet wired** into the grad loop -- **Batch 3 (Phase 2)** | - -### A.2 Selector + duration signal (already wired, baseline-agnostic) -- §S.dur duration helper `client_duration.py::real_client_task_train_duration` is already called in - `_process_single_trainer_message` to set `PROP_CLIENT_TASK_TRAIN_DURATION`. -- Oort selector (`flame/selector/oort.py`, faithful pacer/UCB/D1-D8) is usable verbatim by fluxtune - (`async_oort`) -- **Batch 2 validates its rungs fire**; inert for the two `random`-selector baselines. -- Dynamic-KC controller/policy is integrated (`_build_dynamic_kc_metrics`, `_dynamic_kc_controller.step()`); - fluxtune leaves it **disabled** (fixed K/C). See [docs/dynamic_kc_design.md](docs/dynamic_kc_design.md). -- Analysis `scripts/analysis/analyze_run.py` is fwdllm-aware via `telemetry_manifest.yaml`. +## §A Current status + +**Ground truth (2026-07-04 PM): the Phase-1 sign-off run RAN and is USABLE — the first matched-`data_id` +real↔sim pair for all three baselines.** Command: +`run_sequential.sh --mode both --delays on --max-data-id 10 --max-runtime-s 3600 --yes` +(runs `run_20260704_172155..181912`). **Config sanity PASS** (flags honored bash→yaml→banked config→runtime; +delays matched per-trainer real-sleep = sim-modeled; caveat: `training_delay_factor=None` → the hardcoded ÷10, +so D≈0.4–1.8s, not the full 4–18s — pass `--delay-factor` for the full delay, issue #12c). + +**The S1 blocker is gone:** all 6 processes stopped cleanly on **matched `data_id=10`**, none on +`[SIM_WALL_CEILING]` — the Phase-4a ceiling-decouple + matched-`data_id` stop + Stage-C fwdllm_plus liveness +all worked (fwdllm_plus real completed 10, was stalling at ≈3). + +**Headline results:** (1) **#6 clock-rate ROOT-CAUSED + FIXED (K-D25).** The "sim under-models real wall 2×" +gap was a **checker-anchor artifact, not a sim under-charge** — the rungs compared sim-vclock (genuine +algorithmic time) against real's FULL wall, which carries a ~constant ~7.7s/round localhost transport ARTIFACT +(mqtt re-fetch/redistribute/drain-tail/sleeps) the sim correctly omits. Fix: real now anchors on its intrinsic +algorithmic clock; throughput 0.47→0.048, wall_disparity 90→1.6s (landed, 878-green). The **phase rungs (Root B)** +are fixed too (real's slept delay excluded from post_train; B2 straggler out of the emitted training_budget). +(2) **fluxtune R1 residence REGRESSED to 60.4%** — the K-D19 "fixed in code" claim is **refuted** by this +emergent smoke (was 44.7%); `[SIM_R1_DISPATCH]` fires despite `_release_end_on_return`. Now the top blocker. +(3) **#13 sim_rate** for the sync baselines is near-1 (0.82/0.85) and Root A shows the vclock is correct — the +residual is a small physical overhead; the severe case (fluxtune 0.26) is **R1-confounded** → fix #1c first. + +### Per-baseline ground state +| baseline | stop | R1 | cadence (V1/V2/g2/U3/S2/conv) | sim_rate | surviving fails | verdict | +|---|---|---|---|---|---|---| +| **fwdllm** (sync) | data_id=10 both | **0% PASS** | **all PASS** | 0.82 | #6 clock-RATE family (throughput/commits/terminal + advance/phase rungs now un-skipped), gpu_budget, training_budget | **44/9/21.** Cadence CLEAN; fails all trace to #6 rate gap, no cadence bug | +| **fwdllm_plus** (sync) | data_id=10 both | 0% PASS | PASS (real completes now) | 0.85 | above + eligibility/selection_detail (real sees 4.9 eligible vs sim 9.6 at syn_0), avail_timebase | **40/12/21.** Real liveness FIXED (Stage C); new open: real/sim selection divergence #7 | +| **fluxtune** (async) | data_id ≥9 both | **60.4% FAIL** ⛔ | V2 PASS | **0.26** | R1, staleness, selection_detail (chosen 1.68 vs 1.19), avail_composition + #6 family | **39/15/19.** K-D19 R1 fix REFUTED — top blocker | -### A.3 Remaining structural work (post-Batch-1) -1. **Endogenous commit cadence** -- variance-gated dynamic-K (PARITY.md §F.1): the emergent layer the - async_cifar10 ladder does not model. The clock is now driven; **Batch 2 adds the V/DK/G rung layer** that - verifies the *cadence* on top of it. Most of the per-cycle telemetry it needs already lands via - `build_agg_round` (§I). -2. **Availability telemetry + effect path.** The fwdllm trainer emits **no `EVENT_AVAIL_CHANGE`** - (`telemetry_manifest.yaml`: availability = *partial*), so A6/A7/A8 ground-truth rungs can't run yet, and - the send-gate/delivery-buffer effect path is not wired. Builders exist in `flame/telemetry/events.py`. - **Batch 3 (Phase 2).** +### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) +| baseline | penultimate (10-data_id, K-D18) | **LATEST (sign-off `_1721..1819`)** | Δpass | key failing rungs | +|---|---|---|---|---| +| **fwdllm** | `0704_022610` · 37/5/32 | **44 / 9 / 21** | +7 (skips −11) | overhead_residual, per_round_advance, throughput, total_commits, terminal_state, training_budget, gpu_budget | +| **fwdllm_plus** | `0704_033206` (real stalled) · 31/10/32 | **40 / 12 / 21** | +9 | above + eligibility, avail_timebase, selection_detail | +| **fluxtune** | `0704_020254` (R1 44.7%) · — | **39 / 15 / 19** | first valid pair since K-D19 | **r1_inflight_overlap**, staleness, avail_composition, selection_detail + #6 family | + +*Pass counts rose / skips fell because the 12 rigor-gap rungs (#10) un-skipped; several FAIL on the #6 clock-rate +gap made observable, not new bugs. **Those fails (throughput/overhead/per_round/total_commits/terminal + +training_budget/phase_post_train) are now FIXED IN CODE (K-D25), validated on this banked pair — the scoreboard +counts above are the pre-fix JSON and advance only when the next run is banked.*** + +### Next run — expected per-baseline deltas (K-D25 baseline-agnostic; K-D26 fluxtune-only) +*A FRESH run is required for ALL THREE — `intrinsic_span_s` is a new agg emit; the checker's #6 anchor only +engages on runs that carry it (old banked runs fall back to raw `ts`). Consider `--max-data-id 20` to clear +fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 min/10 data_ids fits in 3600s.* +- **fwdllm** — clock-rate (throughput/overhead/per_round/commits/terminal) + phase (training_budget/post_train) + flip to PASS (~44→~50). The **3 marginal rungs** (per_round KS, total_commits/terminal off-by-one) stay + borderline at exactly 10 data_id; clear at a longer run. `gpu_budget` may persist (mis-applied to fwdllm). +- **fwdllm_plus** — same clock-rate + phase improvement (real completes 10, slowly). **eligibility/ + selection_detail divergence (#7)** persists — separate root, not touched. +- **fluxtune** — clock-rate + phase improve; R1's **round-1 trigger gone (K-D26a)** but the **sustained + NONE-delete path (part b) not fixed** → R1 partially recovers, throughput stays R1-confounded. **This run + measures how much (a) alone recovers R1** → decides whether #13 (or a NONE-race fix) is required. + +### Open issues — master index (OPEN top; 1-line issue + next step) +| # | issue | baseline(s) | next step | +|---|---|---|---| +| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. | fluxtune | **Fix (a) LANDED (K-D26):** vclock-key the abandon-timeout in sim (`_abandon_clock_now`, stamp+check on `vclock_now`); async_cifar10 byte-identical (vclock≈wall). **Fix (b) = the sustained NONE path** is the same slow-sim root → cured by **#13** (tracked), or a follow-up aggregator-side race fix. Re-run confirms how much (a) alone recovers. | +| **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | +| **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | +| **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | +| **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only 4.9 eligible vs sim 9.6 | fwdllm_plus | Real liveness FIXED (Stage C — completes 10). Remaining: profile the per-iteration reselection + oracular read cost from the banked per-phase log; explain the real/sim eligible-count gap at 100% avail. Not a sim bug. | +| **#12c** | `--delays on` uses the hardcoded `training_delay_factor=10` ÷10 shrink unless `--delay-factor` passed | all | Optional: pass `--delay-factor` for the full 4–18s registry delay. Config-flow otherwise correct (K-D24/K-D21). | +| **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits, real-only; one-grad-per-poll drain tail) | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run to validate the MQTT settle timing (principle #8/#11c). Never change grad values/cadence. | + +### Open roots — ranked lowest-rung-first +1. **#1c R1 residence (fluxtune)** — ROOT-CAUSED (K-D26): a wall-vs-vclock desync in the shared async_oort + re-pick guard. Wall-timeout path FIXED; the sustained NONE-delete path is the same slow-sim root → **#13**. +2. **#13 slow sim (fluxtune wall ≫ vclock)** — promoted: it is what makes the async_oort guard machinery misfire + (#1c) AND the headline slowdown. Cure the fluxtune inter-round physical wall → wall≈vclock (async_cifar10). +3. **#7 fwdllm_plus real speed / selection divergence** — real completes now; explain the 4× slowness and the + syn_0 eligible-count gap from banked telemetry. +3. *(#6 clock-rate — RESOLVED as a checker anchor, K-D25; sim vclock was correct. Only the 3 marginal rungs + + the B2 straggler calibration remain, both emergent at the longer sign-off run.)* + +### SKIP audit (19–21 skips; ~16 legit) +Legit (mechanism genuinely absent at Phase-1 syn_0 + `random` selector): 7 availability ground-truth rungs + +4 delivery/withheld (Phase-2 effect path, not built) + 3 DynamicKC (disabled by design) + 2 oort-only +(`random` baselines) + `residence` (async telemetry). The 12 former rigor-gap skips (4 advance rungs, 8 +phase-timing rungs) are now **un-skipped** (K-D21 A1/A3) — they run and mostly FAIL on #6, as intended. --- -## §B How FwdLLM differs structurally - -Full detail (the crux that makes the ladder applicable, and the variance-cadence rung layer it forces) is -**PARITY.md §F.1**. In one line: fwdllm aggregates **gradients** (JVPs) not weights, its commit cadence is -**endogenous** (variance-gated dynamic-K), and its progress axis is **`data_id`** (committed variance -passes), not raw update count. Gradient *values* are mode-invariant given identical input+perturbation -seed, so fwdllm parity reduces to clock+selection+ordering parity **plus** a new variance-cadence layer. - -Concrete anchors (`flame/mode/horizontal/syncfl/fwdllm_aggregator.py`): `aggregate()` var gate + -rollback/`cached_v` at each `_agg_goal` boundary; `total_data_bins=150` (`:271`); force-commit cap -`_max_iter_per_data_id` (`:276`, config key `max_iterations_per_data_id`); `_reselect_each_iteration` -(`:295`, per-iteration reselection for fwdllm++); sync path `_aggregate_grads_sync` (`:1354`). +## §B How fwdllm differs structurally -### §B.1 Curated real<->sim design deltas vs async_cifar10 -- **KEEP CURRENT** +fwdllm aggregates **gradients** (JVPs) not weights; its commit cadence is **endogenous** (variance-gated +dynamic-K); its progress axis is committed **`data_id`** (variance passes), not update count. Gradient values +are mode-invariant given identical input+perturbation seed, so parity reduces to **clock + selection + ordering +parity plus a variance-cadence layer**. Anchors (`flame/mode/horizontal/syncfl/fwdllm_aggregator.py`): +`aggregate()` var gate + rollback/`cached_v` at each `_agg_goal` boundary; `total_data_bins=150`; +force-commit cap `max_iterations_per_data_id`; `reselect_each_iteration` (fwdllm++ per-iteration reselection); +sync path `_aggregate_grads_sync`. Full detail: PARITY.md §F.1. -Every place fwdllm's real+sim design intentionally diverges from async_cifar10, in one at-a-glance table -so a reviewer can separate an **intentional fwdllm-specific choice** from an accidental discrepancy without -reading the whole §K log. **This table is CURATED (rewritten in place to reflect the current design); -§K is the append-only rationale log.** *Maintenance rule (per the DESIGN PRINCIPLE at the top): when a -deviation lands or changes, append its rationale to §K **and** update the matching row here so this stays -the true current picture.* The `§K` column points at the full rationale. +### §B.1 Real↔sim design deltas vs async_cifar10 — **CURATED, KEEP CURRENT** +Separates an **intentional fwdllm choice** from an accidental discrepancy. Update a row when a deviation +lands/changes; the §K column points at the one-line rationale. -| # | Axis | async_cifar10 | fwdllm | Why fwdllm differs | §K | +| # | Axis | async_cifar10 | fwdllm | Why | §K | |---|---|---|---|---|---| -| 1 | Aggregated object | model **weights** | **gradients** (forward-grad JVPs) | grad values are mode-invariant given identical input+perturbation seed, so parity reduces to clock+order+selection parity **plus** the variance-cadence layer | §F.1 | -| 2 | Progress axis | update / round count | committed **`data_id`** (variance passes) | cadence (updates-per-data_id) is an **output to match**, not an input to assume | principle #2 | -| 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer the cifar ladder doesn't model; V/DK/G rungs verify it | §F.1 | -| 4 | sct delay model | `sct = send + max(gpu, D)` (sleep-to-fill-budget) | `sct = send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU time; copying cifar's `max()` would desync real<->sim | **K-D2** | -| 5 | Per-eval sct | distinct eval sct, ~20x eval speedup | **collapses to the train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20x factor); trainer eval msg is a utility report, not a clocked commit | **K-D3** | -| 6 | Slot release | per-commit (inside `_sim_recv_min`), hold slot to COMMIT | **same as felix — hold slot to COMMIT** (`selected_ends` AND `all_selected` held for every dispatched-but-not-committed trainer, released on commit) | a returned-but-uncommitted trainer is still in flight in VIRTUAL time (grad commits when vclock reaches sct), so its slot is occupied; K-D16 briefly freed it on physical RETURN → undercounted `in_flight` 3× (K-D17b reverted, realigned with felix) | **K-D5**/**K-D16**/**K-D17b**, principle #4 | -| 7 | Buffered-but-uncommitted grad on rollback | carried across the barrier | **carried** (async/fluxtune, K-D12) — commit-then-carry + Option-A slot/guard split (K-D16); **dropped** stays correct for sync (c≈agg_goal, no surplus) | drop was benign only for `\|selected\|≈agg_goal` (sync); fluxtune (c=10≫agg_goal=3) dropped ~7 grads/cycle → 2× passes → reversed to carry; the carry then needed the slot-release timing fix (K-D16) | **K-D6**, **K-D12**, **K-D16**, §L | -| 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` (reuse the primitives, fork the orchestration) | `_sim_recv_min`'s per-commit slot release + withheld/staggered paths key on WEIGHTS semantics -- wrong for a grad pool released on the agg-goal boundary | **K-D4** | -| 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's entire config corpus is `time_mode: real` and shipped with no sim path; a "simulated" default risks silently half-activating an unbuilt path | **K-D1** | -| 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | the post-mutation `data_id`/`iteration_per_data_id` advance BEFORE the event emits, so binning by them is off-by-one; snapshot before the pass/fail branch makes V1 exact | **K-D9** | -| 11 | Availability tracking (v1) | all `trace_read` | **mixed**: fwdllm unaware, fwdllm_plus `oracular`, fluxtune `client_notify` mapped onto approx `trace_read` | baselines carry different tracking models; first-class `client_notify` deferred to Stage H to keep Phase 2 tractable | **D1** | -| 12 | Launch tooling | single parity template (sim+real pairs, `baseline:` field) driven by `debug_run.sh` | per-baseline yamls + **separate `_sim` files** driven by `run_sequential.sh` | different config models; **both drivers now source the shared harness `examples/scripts/expt_runner.{sh,py}`** (conda activation, launch+ticker, log asserts, pre-flight + tiered hyperparam display) so only the config-discovery/patch adapter differs per example | this work | +| 1 | Aggregated object | model **weights** | **gradients** (JVPs) | grad values mode-invariant → parity = clock+order+selection + variance-cadence | §F.1 | +| 2 | Progress axis | update/round count | committed **`data_id`** | cadence (updates-per-data_id) is an **output to match**, not an input | principle #2 | +| 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer cifar doesn't model; V/DK/G rungs verify it | §F.1 | +| 4 | sct delay model | `send + max(gpu, D)` | `send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU; `max()` would desync | K-D2 | +| 5 | Per-eval sct | distinct, ~20× faster | **collapses to train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20× factor) | K-D3 | +| 6 | Slot residence | per-commit release | **hold slot to COMMIT** (felix-aligned) — every dispatched-but-uncommitted trainer held in `selected_ends`+`all_selected`, released on commit | a returned-but-uncommitted grad is still in flight in VIRTUAL time (commits when vclock reaches sct) | K-D5/K-D17b, principle #4 | +| 7 | Surplus grad on rollback | carried | **carried** for async (fluxtune, c≫agg_goal); **drop** stays correct for sync (c≈agg_goal) | drop was benign only for sync; fluxtune dropped ~7/cycle → 2× passes | K-D6/K-D12 | +| 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` | cifar's per-commit release + withheld paths key on WEIGHTS semantics | K-D4 | +| 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's whole config corpus is `real`; a `simulated` default risks half-activating an unbuilt path | K-D1 | +| 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | post-mutation `data_id` advances before emit → off-by-one; snapshot makes V1 exact | K-D9 | +| 11 | Availability tracking (v1) | all `trace_read` | **mixed**: fwdllm unaware, fwdllm_plus `oracular`, fluxtune `client_notify`→approx `trace_read` | baselines carry different models; first-class `client_notify` deferred to Stage H | D1 | +| 12 | Launch tooling | single parity template (`debug_run.sh`) | per-baseline yamls + `_sim` files (`run_sequential.sh`) | different config models; both source the shared harness `examples/scripts/expt_runner.{sh,py}` | this work | --- -## §C Baseline matrix -- RESOLVED (from the landed launcher configs) - -Read from `expt_scripts/{fluxtune,fwdllm,fwdllm_plus}_n10_smoke.yaml`. Canonical copy + taxonomy mapping -is **PARITY.md §F.2**; repeated here for the build plan. - -| baseline | sync/async | selector | agg | tracking_mode / avail | reselection | K (agg_goal) | dynamic_kc | +## §C Baseline matrix (resolved from the launcher configs) +| baseline | sync/async | selector | agg | tracking / avail | reselection | K (agg_goal) | dynamic_kc | |---|---|---|---|---|---|---|---| -| **fluxtune** | async | `async_oort` | fedbuff (+server LR, JVP) | `client_notify` (3-tier, mobiperf_3st_50) | -- | 3 | disabled (fixed K/C) | -| **fwdllm** | sync | `random` | fedavg | `default` (unaware) | per-round | 10 (=c; all selected required) | -- | -| **fwdllm_plus** | sync | `random` | fedavg | `oracular` (`_metadata`, mobiperf_2st) | per-iteration (`reselect_each_iteration=True`) | 2 | -- | +| **fluxtune** | async | `async_oort` | fedbuff (+server LR, JVP) | `client_notify`→`trace_read` v1 (aware-at-select, reactive-90s) | — | 3 | disabled | +| **fwdllm** | sync | `random` | fedavg | `default` unaware (reactive-90s) | per-round | 10 (=c) | — | +| **fwdllm_plus** | sync | `random` | fedavg | `oracular` (aware-at-select via trace read) | per-iteration | **10** (yaml is source of truth; §C draft read 2) | — | -**Availability-taxonomy mapping** (UNAVAILABILITY_DESIGN.md two axes): -- **fwdllm** -- unaware at selection (`avail_select_filter=False`), reactive-90s in-flight - (`proactive_inflight_evict=False`). -- **fwdllm_plus** -- aware at selection via `oracular` trace read (`avail_select_filter=True`), - reactive-90s in-flight. -- **fluxtune** -- async, aware via `client_notify` (message-transport), reactive-90s in-flight. - `client_notify` is async_cifar10's **deferred Stage-H** tracking model -> **decision D1**. - -This decides: Stage 3 oort rungs run **only** for fluxtune; Stage 5 sync-barrier rungs run for -**fwdllm/fwdllm_plus**; V1/V4 distributions are shaped by each baseline's `max_iterations_per_data_id` / -`var_threshold`. Parity is staged **one baseline at a time** (PARITY.md workflow rule). +Phase order (locked): Phase-1 syn_0 → Phase-2 unavailability → Phase-3 beyond syn_0. One baseline at a time. +Stage 3 oort rungs run **only** for fluxtune; sync-barrier rungs run for **fwdllm/fwdllm_plus**. --- -## §D The parity ladder for fwdllm - -**Rung definitions live in PARITY.md §F** (modified rungs §F.3; variance-cadence / dynamic-K/C / -forward-grad rungs §F.4). This section states only which rungs apply and how they gate. - -- **Reused verbatim** (PARITY.md §2): Stage 0 TC1/K10; Stage 1 P3/K1/K7; Stage 2 A1/A2 (+ A3 time-base - CONTROL, + A6/A7/A8 ground-truth once telemetry is ported); Stage 3 S3/4, A2c and the oort stack - (Sx/Sd/S2) **only for fluxtune**; Stage 4 T2, per-phase timing, K6, T_mqtt; Stage 8 C1; Stage 9 budget/stop. -- **Modified** for variance-gated dynamic-K: K3a/K3b/K2/U3/K8/U2 -> PARITY.md §F.3 (all re-keyed to the - **variance-pass boundary** / committed **data_id**). -- **New** -- the variance-cadence layer (the prize): V1-V5 (Stage 6'), DK1-DK3 (Stage 3'), G1-G2 (Stage 7') - -> PARITY.md §F.4. Localize down, never fix an EMERGENT rung directly; `var_threshold` / - `max_iterations_per_data_id` are config knobs, not parity levers. -- **Availability** rungs (A1-A5, A6/A7/A8/K11, withheld_delivery, abandon_timeout, starvation_advance, - eligible_pool_reduction) are inherited from the async_cifar10 substrate and apply once §Stage-Avail - (below) wires the effect path + telemetry. +## §D Parity ladder for fwdllm (rung defs live in PARITY.md §F) +- **Reused verbatim:** Stage 0 TC1/K10; Stage 1 P3/K1/K7; Stage 2 A1–A3 (+A6/A7/A8 once avail telemetry lands); + Stage 3 S3/4, A2c + the oort stack (fluxtune only); Stage 4 T2/K6/T_mqtt; Stage 8 C1; Stage 9 budget/stop. +- **Modified** for variance-gated dynamic-K: K3a/K3b/K2/U3/K8/U2 → re-keyed to the variance-pass boundary / + committed `data_id` (PARITY.md §F.3). +- **New** variance-cadence layer: V1–V5, DK1–DK3, G1–G2 (PARITY.md §F.4). Localize down; never fix an EMERGENT + rung directly. `var_threshold` / `max_iterations_per_data_id` are baseline knobs, not parity levers. +- **Availability** rungs (A1–A5, A6/A7/A8/K11, withheld/abandon/starvation) inherited; apply once Phase-2 + wires the effect path + telemetry. --- -## §E Implementation plan -- 4 pytest-gated batches across 3 phases - -**Batching principle (why this shape).** Each batch is a *large dev push* that lands a complete -mechanism cluster **plus its pytest coverage together**, ending in ONE "pause to test" that is -**pytest-only** -- no experiment run in the inner loop. Experiment runs (smoke -> convergence) happen -**once per phase, gated behind green pytests**, never as the dev loop. This is affordable because the -parity **rungs are pure functions over telemetry** (`parity_checks.run_all_parity`): we test them by -constructing synthetic real/sim telemetry pairs with known-correct and known-broken deltas and asserting -PASS/FAIL; and the mechanisms (sct reorder buffer, one-in-flight hold, delivery buffering, sync barrier) -are driven with synthetic message sequences. Every batch is **config-gated: flag-off => byte-identical** -to today; the flag-off regression is itself a pytest assertion. - -**Phase order (locked at kickoff):** Phase 1 = **100% availability (syn_0) parity for all three -baselines**; Phase 2 = **unavailability (syn_20/50/mobiperf)**; Phase 3 = **beyond syn_0**. Do not start a -phase until the prior phase's sign-off run is banked in §H. - -**Two staging changes from the original stage list (below):** (1) the **sync-path port** -(`_aggregate_grads_sync`, old "Stage 5") moves **into Phase 1 Batch 1** -- fwdllm/fwdllm_plus are *sync*, -so their primary commit path is the barrier; they cannot reach syn_0 parity without it. (2) old "Stage 0" -is a *validation gate*, not a batch -- its telemetry is already landed (see Current-state callout). - ---- - -### PHASE 1 -- 100% availability (syn_0), all three baselines - -#### Batch 1 -- both structural clock ports: trainer sim path + async grad loop + sync barrier -- ✅ DONE -Landed & pytest-gated (75 green). Trainer additive `_sim_completion_ts` + no-sleep; async -`_sim_recv_min_grad` sct buffer + in-flight gate + agg-goal-boundary rollback cleanup; sync -`_sync_sim_recv_first_k` first-k-smallest + U6 lags; sim launchers. **Build detail is history** -- see -the code (`ef4cb70e` +) and the deviation rationale **§K-D1..D8**. Cold-start map for the code lives in -**§I** (now repurposed as the Batch-2 map). U6 telemetry *emission* deferred (K-D7); the seeded mini-run -feeding `parity_checks` is folded into the Phase-1 sign-off run. - -#### Batch 2 -- variance-cadence layer + fluxtune selection fidelity -- ✅ DONE (checker + telemetry, pytest-gated) -Landed: V1-V5 / DK1-DK3 / G1-G2 in `parity/checks.py` (registered in `run_all_parity` + `CHECK_META`), -the cycle-relative cadence telemetry they read (**§K-D9**), DK3/G1 emit deferred with a logged SKIP -(**§K-D10**). Pytest: `test_parity_checks.py` (per-rung known-PASS/known-broken + `run_all_parity` -fwdllm smoke + non-fwdllm-SKIP) and `test_fwdllm_agg_telemetry.py` (driven pre-mutation snapshot). The -fluxtune selection-fidelity rungs already exist in the STAGE map (§I.4) -> they fire in the sign-off run; -no new checker code. Original Batch-2 plan below kept for the rung-by-rung rationale. - -- **Variance-cadence rungs** (PARITY.md §F.4): implement V1-V5 / DK1-DK3 / G1-G2 in the shared engine. - Most dev is checker + the telemetry to feed it: per-cycle `var` trajectory (at each agg-goal), - iterations-per-`data_id` (realized dynamic-K), force-commit (`max_iterations_per_data_id`) bypass rate, - and a `cached_v` carry-over diagnostic (V3). Likely roots, confirm via the *lowest broken rung*: - contributing-set/order divergence (U5/S2 upstream) -> V1 -> K2; grad-pool accumulation order - (`cached_v` carry-over, `grad_pool.append` order) -> V2 with matched inputs = a true sim bug; - force-commit rate (V4) = chronic variance divergence, not a separate bug. **DynamicKC: DK3 (policy - *input*) before DK1/DK2** (inert unless a baseline enables dynamic_kc; fluxtune leaves it off). -- **Selection fidelity (fluxtune only, `async_oort`):** validate A2c/Sx/Sd/S2 + the §S.pacer/§S.temporal/ - §S.dur stack (already landed in `flame/selector/oort.py`) -- validation + any fwdllm-specific wiring, not - a rebuild. Inert for the two `random`-selector baselines (S-rungs WARN/skip). - -**Pause to test (pytest-only):** unit-test each new rung function against **constructed real/sim telemetry -pairs** with known cadence deltas (known-PASS fixtures + known-broken force-commit-rate / var-trajectory -fixtures). Then a seeded mini-run per baseline -> `parity_checks` -> assert applicable rungs PASS. **Exit -rungs:** V1/V2/V5 PASS; K2 (committed-data_ids/vsec) PASS; A2c PASS + Sd binding real~=sim + no pacer -ratchet (fluxtune); DK tracks if enabled. - -#### Phase-1 sign-off run (the one time-consuming step of the phase) -Only after Batches 1-2 are green in pytest: one smoke (5 min) then one convergence run **per baseline** at -syn_0 -> `parity_checks` full battery + **C1/C2 at matched `data_id`**. Record in §H. Gate to Phase 2. - ---- - -### PHASE 2 -- unavailability (syn_20/50/mobiperf) - -#### Batch 3 (large dev) -- wire the ClientAvailability effect path into the grad loop -- **Effect path:** send-time gate (real) / `delivery_ts = max(sct, next_avail)` buffering (sim); the two - ledgers; reactive-90s in-flight for all three baselines (no proactive-evict baseline here); starvation - vclock-advance under scarcity; per-baseline `avail_select_filter` (fwdllm off; fwdllm_plus via the landed - oracular read; **fluxtune via `trace_read` for v1 -- D1 resolved (b), `client_notify` deferred to - Stage H**). -- **Availability telemetry (D2):** emit `EVENT_AVAIL_CHANGE` (trainer state machine) + - `agg_belief_change`/`send_gate_wait` (aggregator belief hooks). All three builders exist in - `flame/telemetry/events.py`; this unlocks A6/A7/A8/K11. -- **D3 interaction (over-instrument BEFORE trusting cadence):** a withheld/late grad meets the variance - gate -- does it roll into `cached_v` on a rollback? Does a late grad against a stale `model_version` - inflate the `var` signal (and thus dynamic-K)? Genuinely new vs async_cifar10 (which commits weights, - not a variance-gated pool). - -**Pause to test (pytest-only):** adapt async_cifar10's availability test patterns -(`scripts/parity/test_availability_rungs.py`, `test_delivery_ledger.py`, `test_starvation_termination.py`) -for fwdllm; unit-test delivery buffering, withheld-then-delivered (not dropped), starvation self-stop -(`"stopping run"`, no `SIM_WALL_CEILING`), and the **syn_0 gate-ON-vs-OFF byte-identical** invariant. -`test_eot_avail_catchup.py` already covers part of this. **Exit rungs:** A1-A5 + A6/A7/A8 PASS. +## §E Roadmap — remaining phases -#### Phase-2 sign-off run: one run per (baseline x trace) -> A1-A5 + A6/A7/A8 PASS; self-stops; withheld -grads delivered not dropped. Record in §H. Gate to Phase 3. +**Phase 1 (syn_0) — the sign-off run is banked (§A); close-out work:** +1. **#1c fluxtune R1** — root-cause + fix (top blocker). +2. **#13/#6 clock-rate** — confirm the eval fold fires; drive `sim_rate>1` and `wall_disparity`→~0. +3. **#7 fwdllm_plus** — explain real slowness + selection divergence from telemetry. +Then C1/C2 convergence at matched `data_id` per baseline → gate to Phase 2. ---- - -### PHASE 3 -- beyond syn_0 (the added-complexity layer, last) +**Phase 2 — unavailability (syn_20/50/mobiperf).** Wire the ClientAvailability effect path into the grad loop: +send-time gate (real) / `delivery_ts = max(sct, next_avail)` buffering (sim); two ledgers; reactive-90s +in-flight; starvation vclock-advance; per-baseline `avail_select_filter`. Emit `EVENT_AVAIL_CHANGE` + +`agg_belief_change`/`send_gate_wait` (builders exist in `flame/telemetry/events.py`) → unlocks A6/A7/A8/K11. +**D3 watch:** does a withheld/late grad roll into `cached_v` on rollback, or a stale-version late grad inflate +`var` (and thus dynamic-K)? Over-instrument before trusting cadence. Pytest: adapt async_cifar10's availability +test patterns; syn_0 gate-ON-vs-OFF byte-identical. Exit: A1–A5 + A6/A7/A8 PASS; self-stops; withheld grads +delivered not dropped. -#### Batch 4 -- full ladder under syn_20/50/mobiperf + convergence sign-off -Mostly runs + checker, minimal new dev. V/DK rungs under scarcity (force-commit rate shifts), K8/U2 -terminal-state @ matched `data_id`, C1/C2. **Bin V1/V2 by run-fraction** to separate a *constant* mix bias -from a *compounding* variance-feedback loop (the headline fwdllm risk, §G). **Min:** full (3-4 h+). -**Exit:** curves within tolerance at matched data_id; K8/U2 rel within bar; V1/V2 binned residual flat. +**Phase 3 — beyond syn_0.** Full ladder under scarcity; bin V1/V2 by run-fraction to separate a constant mix +bias from a compounding variance-feedback loop (§G). Exit: curves within tolerance at matched `data_id`; K8/U2 +within bar; V1/V2 binned residual flat. --- -## §F Key design decisions (open -- resolved at implementation) - -- **D1 -- tracking-mode strategy. RESOLVED: option (b) -- map fluxtune onto `trace_read` for the v1 parity - pass; `client_notify` is deferred to Stage H.** fluxtune's landed config uses `tracking_mode: - client_notify`, which UNAVAILABILITY_DESIGN.md treats as the **deferred Stage-H** message-transport model - (async_cifar10 v1 is all `trace_read`). We take the smaller Phase-2 scope: fluxtune runs an approximated - `trace_read`-shaped availability model in v1 (aware-at-selection via trace read, reactive-90s in-flight), - and first-class `client_notify` becomes a later stage once syn_0->unavailability parity is banked. - fwdllm_plus's `oracular` and fwdllm's unaware paths are already `trace_read`-shaped, so this makes all - three baselines share one substrate for v1. *(Original doc leaned (a); reversed at kickoff to keep - Phase 2 tractable.)* -- **D2 -- availability telemetry port.** The fwdllm trainer emits no `EVENT_AVAIL_CHANGE`; the aggregator - emits no `agg_belief_change`/`send_gate_wait`. All three builders exist in `flame/telemetry/events.py` - -- port emission (trainer state machine + aggregator belief hooks) so A6/A7/A8/K11 light up. Prereq for - Stage Avail's exit. -- **D3 -- variance-cadence x withheld/late grads.** Does a withheld grad roll into `cached_v` on rollback? - Does a late grad against a stale `model_version` distort the `var` signal (and thus dynamic-K)? New to - fwdllm; over-instrument in Stage Avail before Stage 3. -- **D4 -- eval-delay factor.** FedFwd eval is a forward pass (~= train cost), unlike async_cifar10's - ~20x-faster eval. Confirm the factor per baseline before Stage 1's per-eval `sct` stamp. -- **D6 -- D-e fix: how to decouple compute-slot release from the re-pick guard. RESOLVED → Option A - (K-D16).** *Decision gate answered:* async_oort's triplet filter (`async_oort.py:1660`) skips only when - `trainer_version_states[end] == agg_version_state`, i.e. WITHIN the current cycle; since the triplet - advances every agg-goal boundary it does NOT block re-pick of a *carried* trainer across the boundary, and - `_trainer_state_dict` was never even populated — so **Option B is unsafe** and A is required. Landed: free - `selected_ends` (compute slot) on RETURN, hold `all_selected` (re-pick guard) to COMMIT, carry the grad, - survive the rollback boundary; triplet guard populated on grad RETURN (**corrected in K-D17** — the K-D16 - as-built stamped it at DISPATCH, which froze the pool before the first commit and deadlocked the re-run). - See §H open-root #1/#1b for the root-cause and K-D16/K-D17 for the as-built. *Original options analysis - kept below for the record.* The bug: the - K-D12 commit-then-carry fix leaves a - returned-but-uncommitted trainer in the selector's `all_selected`, which the dispatch top-up reads as an - occupied compute slot (`extra = c − |all_selected|`, `count_avl_train` excludes `all_selected`, - `async_oort.py:1580`) → sim refills the pipeline by ~1/cycle → concurrency collapses to ~1.5 vs real ~5.8. - `all_selected` conflates two lifetimes that must split: **compute-slot occupancy** (must end at grad - **RETURN** — a returned trainer is idle, its slot should refill to keep C computing, exactly what real's - channel does on receipt) vs the **re-pick guard** (must hold until **COMMIT** — don't re-select the same - trainer for the same `(model_version, data_id, iteration)` triplet while its grad is still carried). - Two implementation strategies: - - **Option A — two explicit sets (faithful, more code).** Add a distinct in-flight/computing set: a trainer - enters at dispatch, leaves at grad RETURN; the top-up uses `extra = c − |in_flight_computing|` so the - pipeline refills to C on return. A separate guard set holds the carried-but-uncommitted trainers - un-re-pickable until commit, then clears. *Pros:* mirrors async_cifar10 exactly (§3.resid frees on - receipt + §4.9 `simInflightCarryover` carries the grad); explicit, auditable, unit-testable in isolation - (assert concurrency ≈ C and R1 overlap == 0%). *Cons:* new state to keep consistent with the carry buffer - **across variance-FAIL rollbacks** at the agg-goal boundary (K-D5) — the main correctness surface to test. - - **Option B — release the slot at return, reuse the existing triplet-version filter as the guard - (minimal change).** Only fix WHEN the slot frees: remove a returned trainer from `all_selected` at grad - receipt (so `count_avl_train`/`extra` see it free), and rely on the EXISTING `trainer_version_states` - triplet filter (`async_oort.py:1660`, skips a trainer already holding the current triplet) to block - re-pick until commit advances the version. *Pros:* smallest diff, no new bookkeeping, leans on selector - machinery already present. *Cons:* the residence guarantee now depends on selector-internal filtering - rather than an explicit set (harder to reason about / test alone); correctness hinges on the triplet - filter covering the ENTIRE return→commit window **including across a variance rollback** (where - model_version/data_id/iter can shift and the guard could leak → re-pick a still-carried trainer → R1 - violation). **Must prove R1 stays 0% under rollback before choosing B.** - - **Leaning A** (explicit + testable + faithful) unless a scoping pass shows the triplet filter provably - covers the return→commit window across rollbacks — then B is the cheaper equivalent. **Decision gate:** - does `trainer_version_states` block re-pick for a returned-uncommitted trainer through a variance-FAIL - rollback? Answer that first; it picks A vs B. Either way: `fwdllm_aggregator.py` + fluxtune yaml only, - sync path untouched (K-D5/K-D11), shared weight path untouched (principle #8); log the chosen option in - §K as the D-e resolution. - -### Locked principles (from async_cifar10, carried over) +## §F Locked principles (from async_cifar10, carried over) 1. **Sim does real forward-grad compute, charges modeled time.** GPU runs the real JVP; the agg stamps - `sct = sim_send_ts + max(real_gpu_s, D) + leg`. Makes gradient values mode-invariant. - **Never** put overhead on the virtual clock (`vclock = max(vclock, sct)` only). -2. **Progress axis is `data_id`** (committed variance passes). Update count per data_id is the dynamic-K - random variable V1 validates -- an output to match, not an input to assume. -3. **Variance is an emergent gate; localize, never tune it.** V-rungs gate on U5/S2/V1. `var_threshold` is - a baseline-defining config knob. -4. **Slot residence must survive variance rollbacks.** Release keys on the agg-goal boundary; the sct - reorder buffer must not strand a grad across a rollback (the fwdllm-specific extension of §3.resid). -5. **DynamicKC: validate the input before the policy.** DK3 (CONTROL) before DK1/DK2 (MECHANISM). The - controller is shared/baseline-agnostic -- do not fork it per baseline. -6. **Real is the reference only after admissibility.** A real<->sim gap has two fix directions; if a - selection-mix residual appears, check whether the **real** input is the divergent side. -7. **syn_0 byte-identical gate OFF; unavailability config-gated.** Reuse the async_cifar10 flag axes and - the `simUnavailability` gate; do not delete eligibility plumbing. -8. **Fix the concept, not the symptom -- and never regress a working real<->sim example.** Before patching - a sim divergence, ask *what the mechanism is for*, not just how to get past the failure: - - **Classify the mechanism: real-transport artifact vs. algorithmic property.** Devices that exist only - to manage the MQTT/channel message queue (incremental "commit 1 per pass" draining, `recv_fifo` - timeouts, "waiting to clear up queue", persistence of uncommitted messages across calls) are - **real-transport artifacts with no sim analog** -- the sim commit path is the *barrier* (`sct`-ordered, - single-pass, slot-release at the agg-goal boundary), not a trickling queue. Applying a real-transport - discipline to the sim barrier is a category error. Gate such logic `and not self.simulated`; do **not** - re-derive it in the sim path. - - **Prefer the byte-identical-for-the-working-side fix.** When one side already works (usually real), - the correct fix leaves that side provably unchanged. `and not self.simulated` on a real-only clamp is - safe; globally reverting a flag that a prior stall-hotfix overloaded (e.g. `ends_not_selected_yet`'s - `len(ends) >= agg_goal` clause from #42) is *not* -- it risks re-opening the bug that hotfix closed. - A greedy fix in one direction is still greedy. - - **Scope-check before editing shared code.** `grep` the symbol: if it lives only in - `fwdllm_aggregator.py` the blast radius is fwdllm; if it's in `top_aggregator.py` / the shared - `parity` engine / `_sim_recv_min`, a change can silently break **async_cifar10**, the previously-green - baseline. Never trade a fwdllm fix for an async_cifar10 regression. (See principle #6: a real<->sim - gap has two fix directions -- pick the one that doesn't move the already-admissible side.) -9. **Test cadence -- match the pytest scope to the blast radius; full-suite after a cross-stack change, - not before every run.** The parity rungs + sim mechanisms are pure/deterministic over synthetic - telemetry, so pytest is the fast gate -- but scope it to what you touched: - - **fwdllm-only edit** (`fwdllm_aggregator.py`/`fwdllm_trainer.py`): `pytest tests/mode -k fwdllm`. - - **shared parity engine / `parity_checks` shim** (also feeds async_cifar10): add - `pytest examples/async_cifar10/scripts/parity` + `tests/mode -k parity`. - - **shared stack** (`flame/launch/*`, `flame/telemetry/*`, `flame/config.py`, optimizer/selector, any - `top_aggregator.py`): run the **full `pytest tests/`** -- these fan out across BOTH examples and every - aggregator/trainer, so a subset can miss the regression they'd cause. - You do **NOT** need full `pytest tests/` before every real<->sim run -- the per-scope subset is the - inner-loop gate. Run the full suite **once after a major set of changes that spans the code stack / - could affect other aggregators/trainers/examples** (this Batch 2.5 is exactly that: it touched - `flame/launch/*` + the shared `parity` engine), and bank that green before launching. A cluster run is - far more expensive than the full unit suite (minutes, no GPU/MQTT), so the insurance is cheap. -10. **Comment discipline -- comment the WHY, crisply; skip the trivial.** Write a comment when it carries - knowledge a later reader (or a later you) would otherwise have to re-derive or re-learn the hard way: - a conceptual/algorithmic choice, a baseline/architectural decision, a real<->sim divergence and its - rationale, or a failure mode + why the fix takes the shape it does (so the mistake isn't repeated). - Do **not** comment trivial mechanical edits (snake_case→camelCase, a rename, an obvious guard). When you - do comment, be **crisp** -- one or two tight sentences, not a paragraph restating the code. The deeper - rationale belongs in §K (append-only) with a one-line pointer from the code; the code comment states the - decision, not its whole history. -11. **Telemetry-FIRST, then instrument, then (rarely) run — do not launch runs to guess-and-check a - hypothesis.** A cluster run is the single most expensive and slowest step in this loop; treat it as the - LAST resort, never the debugger. (Codifies async_cifar10/PARITY.md "Workflow policy" #1–#2 for fwdllm.) - - **(a) Validate/refute a hypothesis from the telemetry ALREADY ON DISK before considering a run.** Every - banked run dir carries per-commit/per-round/per-phase telemetry + the aggregator log; most roots - (the R1 overlap in `[SIM_GRAD_RECV]`, the #6 rate gap in the per-phase timings, the #7 real-speed surplus - in the `runtime.py` decorator lines) are already visible there. If you catch yourself saying "let's run it - and see," stop — first name the exact stored field/line that would confirm or kill the hypothesis, and go - read it. A run that only re-confirms what the logs already show is wasted. - - **(b) When you implement anything new, ship its telemetry + plot + pytest IN THE SAME CHANGE.** Over- - instrument pre-emptively (cheap to log, expensive to re-run for): add the per-event field that makes the - new mechanism's correctness DIRECTLY observable in the checker (not inferred), a plot if it has a - trajectory, and a pytest that drives it on synthetic telemetry and asserts the invariant. This is what - lets (a) work next time: the R1 regression was catchable only because the diagnostic line existed. A fix - with no new instrument/test is not done. - - **(c) A run is justified only to observe an EMERGENT quantity that no stored telemetry can yield** (fresh - convergence, a concurrency/wall number after a mechanism change, a cadence trajectory over more - `data_id`s than any banked run reached) — and then run the SHORTEST length that exhibits it - (PARITY.md run-length budget), smoke first, one mechanism per round. - - **(d) Cross-pollinate fluxtune ↔ fwdllm/fwdllm_plus.** fluxtune is furthest from parity, so it surfaces - roots first; when a fluxtune fix or learning could apply to the sync baselines (e.g. the #6/#1 shared - time-base question, a selector-ignores-hold-set bug, a duration-input anchor), note it in §B.1/§K and - check the others from THEIR banked telemetry before assuming it's async-only. -12. **Consult async_cifar10/PARITY.md's vclock principles BEFORE touching any sim/vclock behavior.** The virtual - clock is a shared flame-core capability that async_cifar10 already exercises correctly across sync AND async; - its rules are settled there. Before changing how the sim clock advances/increments/orders in any scenario, - read the relevant PARITY.md principle and mirror it — deviate only with a logged §K rationale (DESIGN - PRINCIPLE at the top). The settled rules (anchors verified 2026-07-04): - - **Clock is a monotone `max`.** `VirtualClock.advance(ts)` moves forward only (`flame/sim/virtual_clock.py:32-37`); - `_advance_sim_clock(sct)` = `vclock = max(vclock, sct)` (`syncfl/top_aggregator.py:331-343`). K1 - `sim_commit_order_monotone` asserts `vclock_now` non-decreasing. - - **NEVER put overhead on the vclock.** `_sim_commit_overhead_s` defaults 0.0; PARITY.md dead-end forbids >0 - ("masks & drifts; clock must `= max(vclock, sct)`"). K3b `overhead_residual` catches an uncharged/overcharged term. - - **sct = `sim_send_ts + max(gpu, D) + leg`** (`async_cifar10/.../main.py:831-845`); the `leg` goes on the vclock - but NOT on `trainer_speed_s`/utility/gate inputs (keep those pure compute). - - **sync charges MAX-of-K sct; async charges the K-th-fastest** (streaming) — the one real sync/async increment - difference (PARITY.md K3a). Starvation = a clock JUMP to the next availability event, never a spin. - - **The sim SKIPS real waits and reconstructs order from sct** (`SimReorderBuffer`, `virtual_clock.py:52-60`); - `mqtt_fetch_s` (re-selection wait) and `realDistributeSettleSeconds` are deliberately OFF the vclock. -13. **The vclock is virtual wall-time; the sim MUST produce SPEEDUP (vclock ≥ physical-wall-elapsed).** (Operator's - model, locked.) If a trainer's work is 10 real wall-seconds, the vclock advances 10s while the sim executes it in - far less wall — same events, same order, same computation (fidelity preserved), but faster in time (the whole - reason the sim exists). Therefore, **for a healthy sim, `vclock.now ≥ physical_wall_elapsed` at all times**, and - `sim_rate = vclock/wall ≥ 1` and should GROW. A `sim_rate < 1` (as in the 2026-07-04 runs, 0.37×) means the sim is - a slowdown = BROKEN: either it under-charges the vclock (small/zero D) or it fails to skip a real wait (the leak), - or both. **fwdllm nuance:** the forward-grad "train" is a REAL GPU pass (~2s) that MUST run in sim for grad - mode-invariance — that GPU wall is irreducible. So sim wall ≈ gpu + (skippable transport); speedup comes from - skipping the transport/inter-round waits, NOT from skipping compute. `sim_rate` is the top-line health metric to - emit and drive EVERY run — async_cifar10 already logs it (`[VCLOCK_PROGRESS]`, `syncfl/top_aggregator.py:1189-1203`; - K7 `sim_rate_ok`); fwdllm must too (root S3 / issue #13). + `sct = sim_send_ts + max(real_gpu_s, D) + leg`. **Never** put overhead on the vclock (`vclock = max(vclock, sct)`). +2. **Progress axis is `data_id`.** Updates-per-data_id is the dynamic-K random variable V1 validates — an output + to match, not an input to assume. +3. **Variance is an emergent gate; localize, never tune it.** `var_threshold` / `max_iterations_per_data_id` are + baseline-defining config knobs. +4. **Slot residence must survive variance rollbacks.** Release keys on the agg-goal boundary; the sct reorder + buffer must not strand a grad across a rollback. +5. **DynamicKC: validate the input before the policy** (DK3 CONTROL before DK1/DK2). The controller is shared; + don't fork it per baseline. +6. **Real is the reference only after admissibility.** A real↔sim gap has two fix directions — check whether the + **real** input is the divergent side before tuning sim. +7. **syn_0 byte-identical gate OFF; unavailability config-gated.** Reuse the async_cifar10 flag axes; don't delete + eligibility plumbing. +8. **Fix the concept, not the symptom — never regress a working example.** Classify a mechanism as + **real-transport artifact** (incremental drain, `recv_fifo` timeouts, queue persistence — no sim analog; gate + `and not self.simulated`) vs **algorithmic property**. Prefer the byte-identical-for-the-working-side fix. + Scope-check before editing shared code: `fwdllm_aggregator.py` = fwdllm blast radius; `top_aggregator.py` / + shared `parity` engine / `_sim_recv_min` can silently break async_cifar10. +9. **Match pytest scope to blast radius.** fwdllm-only edit → `pytest tests/mode -k fwdllm`; shared parity engine + → add `examples/async_cifar10/scripts/parity` + `tests/mode -k parity`; shared stack (`flame/launch/*`, + `flame/telemetry/*`, `config.py`, optimizer/selector, `top_aggregator.py`) → full `pytest tests/`. Full suite + once after a cross-stack change, not before every run. +10. **Comment the WHY, crisply.** A conceptual/architectural choice, a real↔sim divergence + rationale, or a + failure mode + why the fix takes its shape — one or two tight sentences. Deeper rationale → §K. +11. **Telemetry-FIRST, then instrument, then (rarely) run.** (a) Validate/refute a hypothesis from telemetry + ALREADY ON DISK before considering a run — name the exact field/line first. (b) Ship telemetry + plot + pytest + IN THE SAME CHANGE as any new mechanism. (c) A run is justified only to observe an EMERGENT quantity no stored + telemetry can yield (convergence, a concurrency/wall number, a longer cadence trajectory) — then run the + SHORTEST length that exhibits it, smoke first, one mechanism per round. (d) Cross-pollinate fluxtune↔sync + baselines — fluxtune surfaces roots first. +12. **Consult PARITY.md vclock principles BEFORE any sim-clock change.** Settled rules: clock is a monotone `max` + (`_advance_sim_clock = vclock = max(vclock, sct)`); NEVER put overhead on the vclock; `sct = send + max(gpu,D) + + leg` (leg NOT on `trainer_speed`/utility); sync charges MAX-of-K sct, async the K-th-fastest; the sim SKIPS + real waits and reconstructs order from sct (`SimReorderBuffer`). +13. **The vclock is virtual wall-time; the sim MUST produce SPEEDUP (`vclock ≥ physical_wall_elapsed`, + `sim_rate = vclock/wall ≥ 1`).** fwdllm nuance: the forward-grad "train" is a REAL GPU pass (~2s) that MUST run + in sim for grad mode-invariance — that GPU wall is irreducible; speedup comes from skipping transport/inter-round + waits, NOT compute. `sim_rate<1` (as in the sign-off run, 0.26–0.85) means the sim under-charges the vclock or + fails to skip a real wait. `sim_rate` is the top-line health metric — emit it every run (`[VCLOCK_PROGRESS]`). + +**Open design decisions:** D2 (avail telemetry port — Phase 2); D3 (variance-cadence × withheld/late grads — +Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 resolved (see §K). --- -## §G Open questions / risks / dead-ends - -**Open (toward Stage 0):** -- D1-D4 above. -- Which real datasets/traces to use for the fwdllm reference runs (agnews H5 is the smoke default). - -**Risks specific to fwdllm:** -- **Variance-cadence is feedback-compounding** -- a tiny per-cycle grad-pool order difference compounds into - a different iterations-per-data_id (like the async clock residuals that surfaced only at 3 h). Bin V1/V2 - by run-fraction to separate a *constant* mix bias from a *compounding* feedback loop. -- **`cached_v` carry-over** is stateful across cycles; a divergence looks like a variance bug but is - bookkeeping (V3 DIAG localizes it). -- **One-message-per-call grad loop** vs asyncfl's batch -- slot-hold/release wiring is genuinely different; - do not copy async_cifar10's release points blindly. -- **Withheld grad x variance gate** (D3) -- a stale late grad may distort `var` and the dynamic-K decision. - -**Pre-emptive dead-ends (from PARITY.md / UNAVAILABILITY_DESIGN.md):** overhead > 0 on the vclock; -prediction-only gates that never block; tuning a scalar redispatch/overhead knob instead of fixing the -mechanism; letting eval commits into the train/agg stream; expressing "busy" via the unavailable list. -**fwdllm-new:** do **not** tune `var_threshold`/`max_iterations_per_data_id` to force cadence parity -- -those are baseline-defining knobs; a cadence gap is an upstream set/order/clock divergence. +## §G Fixes landed (what worked — one line each; do not redo) +- **#1c abandon-timeout vclock-key (K-D26, part a)** — the shared `async_oort` 90s `SEND_TIMEOUT_WAIT_S` + in-flight abandon-timeout was keyed on `time.time()` (physical wall); in a slow sim (wall ≫ vclock) it evicted + still-outstanding trainers from `all_selected` → R1 re-dispatch. Now runs on the vclock in sim + (`_abandon_clock_now`: stash `channel_props["vclock_now"]` per-select, stamp + check on it); None in real → + `time.time()` → async_cifar10 byte-identical (its vclock≈wall). `test_sim_vclock_keeps_virtually_recent_end`. + *(Sustained NONE-delete path is the same slow-sim root → #13.)* +- **#6 clock-rate anchor + Root B phase rungs (K-D25)** — root-caused #6 to a checker anchor (sim vclock is + correct; real's full wall carried a ~7.7s/round transport artifact). Agg emits `intrinsic_span_s` (barrier = + max committed intrinsic duration + eval, mirroring the sim vclock composition, **fedavg excluded**); the 5 + clock-rate rungs (K2/K3/K3b/K8/U2) + `wall_disparity` anchor real on the cumulative intrinsic clock, `ts` + fallback → async_cifar10 byte-identical. Root B: `post_train_s` stamped after the delay (pure post-proc); B2 + straggler moved from `_delay_s` into the sct only (`training_budget_s` = base delay). Validated on banked data + (throughput 0.47→0.048, wall_disparity 90→1.6, post_train 0/0, budget 1.14/1.14); full `tests/` 878-green. +- **Phase 4 stopping rule + folds (K-D24)** — `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` (fixed root + S1, no more wall-truncation of a real-compute sim); B1 `sim_model_eval_time` + B2 `sim_straggler_spread_s:0.9` + ON in the 3 sim yamls; matched `--max-data-id` is the primary stop. *(Sign-off run confirms S1 fixed; folds + did NOT yet lift `sim_rate>1` — see #13.)* +- **Phase 2 speedup-leak skips (K-D23)** — gated the trainer per-round `pause_execution` `sleep(1)` throttle + + `_check_availability` avail-spin `and not self.simulated` (fwdllm-only). The trainer `recv`/`await_join` and + agg grace/`await_join` are LEFT AS-IS (barrier-wait over irreducible agg eval+GPU, not skippable sleeps). +- **Pre-run instrumentation A–E (K-D21)** — Stage-C fwdllm_plus scarcity-wait liveness; A1 trainer phase timing; + A2 agg wall-decomp; A3 advance-rung re-key to `data_id` (shared engine, async byte-identical); A4 + `wall_disparity` DIAG rung; B1/B2/B3 sct folds (config-gated); D1 `task_recv.sim_send_ts`; D2 K5 real-compute + exemption; E `sleep(0.1)`-pad gating. Un-skipped the 12 rigor-gap rungs (#10). +- **Availability params end-to-end (K-D22)** — Phase-1 defaults to syn_0; the pre-flight table reads back the + RESOLVED config (print == run); feasibility gate blocks a full-participation sync barrier under a non-syn_0 trace. +- **Trainer sim clock (Batch 1, K-D2/K-D3)** — additive `sct = send + gpu + D`, no-sleep on the sim path. +- **Async grad loop (Batch 1, K-D4/K-D5)** — `_sim_recv_min_grad` sct reorder buffer + in-flight gate + + agg-goal-boundary rollback cleanup. +- **Sync barrier (K-D11)** — `_sync_sim_recv_first_k` first-k-smallest; the `ends_not_selected_yet` "commit-1-per- + pass" clamp gated real-only so the sim drains the full dynamic-K cohort in one pass. +- **Variance-cadence rung layer (Batch 2, K-D9/K-D10)** — V1–V5 / DK1–DK3 / G1–G2 + the pre-mutation + `cycle_data_id` snapshot that makes V1 exact; DK3/G1 emit deferred with a logged SKIP. +- **Residence: commit-then-carry + R1/W1 rungs (Batch 2.5, K-D12/K-D14)** — killed the 2× recompute; R1 sourced + from an echoed per-contribution interval (immutable across re-dispatch). fwdllm/fwdllm_plus R1 exact 0%. +- **Slot residence realigned to felix (K-D17/K-D17b)** — sim drain keys on `_sim_buffer` not channel RECV; the + re-pick triplet stamped on grad RETURN not dispatch; hold the compute slot to COMMIT so `len(selected_ends)` = + virtual-time in-flight (fixed the in_flight mis-measurement, 2.7→9.5 vs real 9.75). +- **staleness_policy wired from config (K-D15/K-D13)** — was silently `none` for every run; fluxtune set to + `fedbuff` staleness-weighted accept (real+sim identical, a definition not a lever). +- **Clock family re-keyed to `data_id` (#2)** — `_progress_axis`/`_per_progress_last_event`; async_cifar10 + auto-detects `round` → byte-identical. field_coverage accepts fwdllm field aliases (#3). W1 made asymmetric + (flags only sim EXCESS). Launcher fail-fast on early aggregator death (D-d). --- -## §H Status - -*Discipline (from PARITY.md §Status / §Settled-roots / §Dead-ends): this section is the **consolidated -ground state**, rewritten in place — NOT an append-per-run log. The end goal is end-to-end real↔sim parity, -so what we keep is (1) the current per-baseline state table, (2) the **open roots** ranked lowest-rung-first, -(3) a **fixes-landed** ledger (what worked, so we don't redo it), and (4) a **dead-ends & corrections** -ledger (what we tried or believed that was wrong, so we don't retry it). Per-run detail lives in the JSON + -git history; the as-built rationale lives in §K.* - -**Status (2026-07-04 PM → the K-D21 Phase-1 sign-off run RAN but every pair is UNUSABLE — all 6 processes hit -the 600s WALL, none reached a natural stop, and no `parity_*.json` was generated.** `bash run_sequential.sh ---mode both --delays on --max-runtime-s 600` produced `run_20260704_133627..143445` (real+sim × 3 baselines). -Two upstream mis-configs made the run non-comparable (analysis-from-telemetry, principle #11a — NOT re-run): -> - **(root S1 — the "timeout") Every SIM died on `[SIM_WALL_CEILING]`, not the intended vclock budget.** The sim -> does REAL forward-grad GPU compute for every dispatched trainer, so physical wall (566–588s) hit the 600s -> ceiling while the vclock only reached **fwdllm 213 / fwdllm_plus 221 / fluxtune 85 s** (0.14–0.37× of the 600 -> budget). Piece A set `sim_wall_ceiling_s = max_runtime_s = 600`, so for a real-compute sim (wall ≫ vclock by -> construction, #6/#9) the ceiling ALWAYS fires before the vclock-budget stop it was paired with — exactly what -> piece A's own comment warned ("a vclock-bounded sim over-runs to the wall ceiling"). Real stopped on -> `max_runtime_s` wall too. So the two sides stop at DIFFERENT progress and cannot be compared: -> | baseline | real rounds·data_id·wall/rnd | sim rounds·data_id·vclock(wall)·wall/rnd | stop | -> |---|---|---|---| -> | fwdllm | 36 · **13** · 15.5s | 66 · **23** · 213s(566s) · 8.6s vc-3.2s | real max_runtime / sim CEILING | -> | fwdllm_plus | 8 · **4** · 59.7s | 69 · **24** · 221s(588s) · 8.5s vc-3.2s | real max_runtime / sim CEILING | -> | fluxtune | 133 · **15** · 3.6s | 103 · **9** · 85s(579s) · 5.6s vc-0.8s | real max_runtime / sim CEILING | -> - **(root S2 — CORRECTED after codepath trace) `--delays on` WAS honored by the trainer, but D was tiny (0.4–1.8s, -> not the 4–18s registry delay) and the aggregator sim-folds were never wired.** The literal "delays didn't reach -> the trainer" was WRONG: real trainers slept 0.4–1.8s (`FedSgdTrainer.py:530`), sim trainers modeled 1.0–1.3s and -> charged it to the vclock (`:524`). Three real gaps: (1) `training_delay_factor=10` (hardcoded -> `configs/trainer_base.yaml:96`) silently divides the registry 4–18s delay ÷10 → 0.4–1.8s; (2) the aggregator's -> logged `training_delay_enabled=False` is an **orphaned field the fwdllm aggregator never reads** (the sim D flows -> from the trainer-reported `sim_completion_ts`, not an agg hyperparam) — a misleading artifact; (3) the sim sct-model -> folds (`sim_model_eval_time`, `sim_straggler_spread_s`) were never patched by the launcher (they live only in -> `aggregator.config_overrides.hyperparameters`, which `--delays` never writes); (4) the banked -> `execution_config.yaml` OMITS `enable_training_delays`, so post-hoc the run looked off. Reframed **open issue #12**. -> NOTE: even a correct large D does NOT by itself give speedup — see root S3. -> - **(root S3 — THE headline: the sim delivers a SLOWDOWN, not speedup — it still pays real waits it must skip).** -> The whole point of the sim is speedup: do the SAME events in the SAME logical order but advance the vclock -> (virtual wall-time) FASTER than physical wall by not waiting. Measured, the sim runs at **sim_rate ≈ 0.37×** -> (vclock 3.2 v-s/round vs physical wall 8.6 s/round) — SLOWER than wall. Root (codepath trace): the sim trainer -> still physically blocks on `await_join()` + real MQTT `recv_wrapper` every round (`fwdllm_trainer.py:213,218`, -> `mqtt_fetch_s`~1.8s steady / ~18s round 1), the aggregator still pays grace-bounded `recv_fifo` (≥2.0s floor, -> `syncfl/top_aggregator.py:347`) + `await_join` on distribute (`fwdllm_aggregator.py:2280/2427`), and ungated -> `time.sleep(1)` backoffs — **none gated `and not self.simulated`**. Real GPU compute (~2s) is irreducible in sim -> (needed for grad mode-invariance), so speedup REQUIRES skipping the transport waits (async_cifar10's base does; -> fwdllm's per-round sync trainer fetch does not). The invariant to restore: **vclock ≥ physical-wall-elapsed** -> (operator's model). New **open issue #13**; tracked via a `sim_rate` metric fwdllm does not yet emit. -> - **fwdllm_plus real is pathologically slow even at syn_0 (no scarcity): 59.7 s/round (8 rounds, data_id 4)** vs -> fwdllm real 15.5 s/round — ~4×. This is NOT the Stage-C scarcity liveness (#7, syn_0 has 100% avail); it is the -> `reselect_each_iteration=True` per-iteration re-selection + oracular read overhead (29 selection events vs -> fwdllm's 1). Confirms the "why is fwdllm_plus real slower/round" half of #7 as a distinct real-config cost, and -> guarantees fwdllm_plus real can never reach the sim's data_id in a wall budget (#4 real-side truncation). -> **Principled fix (design, not yet implemented — awaiting go-ahead):** for a valid parity/sign-off run the -> stopping rule must be a **matched committed-`data_id` target** (finite `--max-data-id`, e.g. 10), NOT a wall or -> vclock budget — a time budget desyncs wall-bound real from ceiling-truncated sim (the overnight 10-`data_id` -> runs passed for exactly this reason). Couple that with: (1) fix the `--delays on` wiring so D>0 reaches BOTH the -> trainer sleep AND the sim sct model (#12); (2) decouple `sim_wall_ceiling_s` from the vclock budget (generous -> multiple / separate `--max-wall-s`) so the ceiling is an outer safety, not the primary stop; (3) resolve -> fwdllm_plus real slowness (#7) so its real side can reach the matched data_id. See open roots S1/#12 below. - -**Status (2026-07-05 → overnight 10-`data_id` grounding runs BANKED; parity read below (K-D18). The fluxtune -R1 residence regression (K-D17b) is STILL the top blocker — do it first (fluxtune ran real-only overnight, so -its sim is unchanged from run `020254`, R1=44.7%).** New findings from the 10-`data_id` runs (bigger than the -prior 3-`data_id` runs — the operator intended 1 h but the `--max-data-id` default was 10, now raised to 9999, -K-D18): -> - **(a) fwdllm sync cadence is HEALTHY at 10 `data_id`s.** V1/V2/g2/R1/W1/U3/S2/conv/conv_loss all PASS — -> the #4 short-run truncation cleared exactly as predicted (37 pass / 5 fail). -> - **(b) fwdllm's surviving fails are the clock-RATE family** (throughput/total_commits/terminal_state), NOT -> the #2 axis re-key: at a matched virtual budget sim charges **~7.8 vclock-s/round** while real spends -> **~34.9 wall-s/round** (both completed all 10 `data_id`s). New **open root #6** — must classify real's -> ~27 s/round surplus as real-transport overhead (correct to exclude, principle #1/#8) vs a sim D under-charge, -> FROM the existing per-phase timing telemetry, before touching the clock. **Shared-shaped with the fluxtune -> 2.4× wall gap** — the cross-baseline learning to watch. -> - **(c) fwdllm_plus real is pathologically slow.** It ran the FULL 1 h (`max_runtime_s=3600` cap) and reached -> only **`data_id`≈3** (86 wall-s/round; sim did all 10 in 81 vclock-s / 274 wall-s). So its V1/V2/g2/ -> eligibility/selection fails are REAL-side **truncation** (comparing sim's 10 full `data_id`s to real's 3), -> plus a genuine **real-speed root #7** (why is fwdllm_plus real ~2.5× slower/round than fwdllm real? — the -> per-iteration reselection is the suspect; localize from telemetry, not a re-run). -> - **(d) field_coverage now fails on a real telemetry GAP** — `task_recv.sim_send_ts` is absent in both modes -> (this is NOT the #3 gpu/budget alias, which passes). New **open root #8** (INV). `failsafe`/K5 also fires -> falsely for fwdllm (sim WALL ≫ vclock because forward-grad compute is REAL) — a checker exemption, root #9. -> -> _Original 2026-07-04 EOD state: K-D17 + K-D17b landed; in_flight measurement FIXED (sim 9.49 vs real 9.75) -> but K-D17b introduced the R1 residence regression (0%→44.7%). Resume from the "NEXT SESSION" block below._ -> **NEXT SESSION (2026-07-05) pickup — start here.** -> **UPDATE: #1c (R1) is now FIXED in code — K-D19.** The prime hypothesis below was **wrong on the mechanism** -> (async_oort *does* exclude `all_selected`; the NONE reset is a red herring). The real root was the guard -> being released on physical grad RETURN before the carried grad committed; fixed fwdllm-only via -> `_release_end_on_return` (defer release to COMMIT), + a `[SIM_R1_DISPATCH]` tripwire + a RETURN-path pytest, -> `tests/mode -k fwdllm` 110 green. **The only step left on #1c is the emergent smoke** (`bash -> run_sequential.sh --mode both --delays on --max-runtime-s 3600 --only fluxtune --yes`, or a shorter -> `--max-data-id 3` smoke first) → `run_parity.py --baselines fluxtune`, confirm R1≤2% + `in_flight`~9.5. -> _The original hypothesis + fix-options are kept below only as the record of what was refuted (K-D19)._ -> **[REFUTED] R1 prime hypothesis:** `_sim_hold_busy_slots` resets buffered ends to `KEY_END_STATE=NONE`… but -> async_oort selects by avail/end-state and does NOT exclude `all_selected`, so a NONE-state buffered trainer -> is re-selectable → R1 overlap. **Refuted:** async_oort excludes `all_selected` (`async_oort.py:1586-87`) and -> the channel passes the full pool regardless of `KEY_END_STATE`; exclusion is by `all_selected` membership -> alone, and the trainer left the guard on RETURN. See K-D19. -> **Then:** re-check `selection_detail.chosen` (sim 0.83 vs real 1.76 — likely follows R1). The **longer run -> is already banked** (the overnight 10-`data_id` runs, K-D18) and cleared #4 for fwdllm; for fluxtune the -> fresh sim re-run is still owed (overnight was real-only). When re-running, the `--max-data-id` default is now -> **9999** (K-D18) so `--max-runtime-s` governs; pass `--max-data-id 10` only for a deliberately short run. -> Launch (after R1 is clean): `bash run_sequential.sh --mode both --delays on --max-runtime-s 3600 --only -> fluxtune --yes`. **First-principle before any launch: exhaust the existing telemetry** (principle #11) — the -> R1 root is already visible in run `020254`'s `[SIM_GRAD_RECV]` log; validate the fix there + by pytest, not -> by a launch. -> **Do NOT redo:** the in_flight measurement fix (K-D17b) is correct and validated — keep the hold-to-commit; -> only the state-reset that re-opens selection is at fault. The temporary `[SIM_GRAD_RECV] ... inflight_exp=/ -> sel_ends=` diagnostic in `_sim_recv_min_grad` is useful — keep it until R1 is closed, then optionally trim. - -**Status (2026-07-04 → K-D17 landed; the K-D16 re-run DEADLOCKED, root-caused to two bugs, now fixed — -awaiting a fresh fluxtune re-run).** Batches 1–2 + §L Batch-2.5 + Option-A (K-D16) + clock-family re-key (#2) -+ field alias (#3) + the **K-D17 drain-gate + triplet-at-return fixes** are landed & pytest-green (full -`tests/mode` 314 passed / 7 skipped). The **post-K-D16 fluxtune smoke (`run_20260703_230407`) did NOT -recover — it deadlocked** (1 cohort dispatched, only 2 grads committed, 0 `agg_round` events, in-flight -0.28, 1663 empty select-rounds). Root cause = **two independent bugs (K-D17)**: (A) the sim drain loop -gated on channel RECV state instead of the `_sim_buffer` — the greedy first drain emptied RECV and stranded -the other ~8 already-received grads (`"no ends yet"` ×1681), so agg_goal=3 was never met from a full cohort; -(B) K-D16 stamped the async_oort re-pick triplet at DISPATCH, freezing the whole eligible pool before any -commit could advance the version (bootstrap case the D6 note missed). Both fixed: drain now keys on the -buffer/in-flight set; the triplet is stamped on grad RETURN. Residence (R1) exact on all three; grad values -mode-invariant. **Remaining action: a fresh fluxtune re-run** to bank concurrency (vs real ~5.8) / wall / U3 -/ V1 / selection, then the **longer run (`max_data_id_progress=10`)** to clear #4. The table below is the -PRE-K-D16 baseline the re-run is measured against (the K-D16 run is void — it deadlocked). - -**Per-baseline ground state (2026-07-05, overnight 10-`data_id` runs; K-D18):** -| baseline | run (real→sim) | R1 | V1/V2/g2 cadence | S2/U3/conv | surviving fails | status | -|---|---|---|---|---|---|---| -| **fwdllm** (sync) | 10 `data_id`s both; wall 349→184s | PASS (0%) | **PASS / PASS / PASS** | PASS / PASS / PASS | throughput·commits·terminal (#6 clock-RATE), field_coverage (#8), failsafe (#9 false-pos) | **37 pass / 5 fail.** Cadence CLEAN at 10 `data_id`s (#4 cleared). Remaining fails all trace to the RATE gap #6 + two checker/telemetry gaps — no cadence bug | -| **fwdllm_plus** (sync) | sim 10 `data_id`s (274s wall); **real hit the 1 h cap at `data_id`≈3** | PASS (0%) | FAIL / FAIL / FAIL | PASS / PASS / PASS | +eligibility, selection_detail (real_chosen 4.88 vs sim 4.39) | **31 pass / 10 fail.** Cadence fails are **REAL-side truncation** (3 vs 10 `data_id`s) driven by real being ~2.5× slower/round (#7); NOT confirmed a sim bug until real completes 10 | -| **fluxtune** (async) | overnight **real-only** (10 `data_id`s); sim unchanged from `020254` | **FIXED in code (K-D19)**; was 44.7% | V2 PASS / — | — | (re-run owed) | **#1c FIXED + pytest-green.** Root = guard released on grad RETURN, not the doc's `all_selected` hypothesis; deferred to COMMIT via `_release_end_on_return`. in_flight measurement (9.49 vs 9.75) preserved. **Emergent-only step left:** one sim smoke to bank R1≤2% | - -### Parity scoreboard — LATEST vs PENULTIMATE only (rewrite in place each run; do NOT append history) - -*Discipline: keep exactly two columns per baseline — the latest parity JSON and the one before it — so drift is -visible without a growing log. `run_parity.py` regenerates both from the banked dirs.* - -*NOTE: the 2026-07-04 PM sign-off run (`run_20260704_133627..143445`) produced NO parity JSON — every sim was -wall-ceiling-truncated and real/sim reached mismatched `data_id` (root S1 above), so a parity read would be -meaningless. The LATEST column below is still the last VALID banked pair (the overnight 10-`data_id` runs); it -does not advance until a matched-`data_id` sign-off run is banked.* - -| baseline | penultimate (run · pass/fail/skip) | LATEST valid (run · pass/fail/skip) | Δ | failing rungs (latest) | -|---|---|---|---|---| -| **fwdllm** | `0703_224641` (3 `data_id`) · 37/5/32 | `0704_022610` (10 `data_id`) · **37/5/32** | =0 | field_coverage, throughput, total_commits, terminal_state, failsafe | -| **fwdllm_plus** | `0703_225614` (3 `data_id`) · 30/11/32 | `0704_033206` (10 `data_id`, real stalled) · **31/10/32** | +1 | above + eligibility, selection_detail, v1, v2, g2_grad_pool_size | -| **fluxtune** | `0704_020254` (sim, R1=44.7%) · — | overnight **real-only → no sim pair** · — | — | (R1 fixed in code K-D19; re-run owed) | - -### SKIP audit — 32 skips, only ~16 are legitimate (rest are rigor gaps to close) - -The user rule: skips are OK only for a true **example/config disparity** (the mechanism genuinely isn't in this -baseline); otherwise add the equivalent fwdllm test. Breakdown of the 32: -- **~16 LEGIT (Phase-1 syn_0 + `random` selector; correct to skip):** 7 availability ground-truth rungs - (`*_trace_fidelity`, `agg_belief_fidelity_*`, `send_gate_wait_*`) + 4 delivery/withheld - (`duty_cycle*`, `abandon_timeout`, `withheld_delivery`, `commit_promptness`, `state_timeline_agreement`) — - the **availability effect path is Phase 2 (Batch 3)**, not built; + 3 DynamicKC (`dk1/dk2/dk3`) disabled by - design; + 2 oort-only (`selector_score`, `preferred_duration`) — fwdllm uses `random`; + `residence` (async - telemetry). No equivalent test exists because the mechanism isn't present. *(These un-skip as Phase 2 lands.)* -- **4 clock-advance rungs skip on the WRONG AXIS → RIGOR GAP (root #10):** `modeled_compute_advance` (K3a), - `overhead_residual` (K3b), `per_round_advance` (K3), `overlap_factor` (K4) report "fewer than 2 sim rounds" - even though the sim did 26 — they're still keyed on `round` (fwdllm stays at round=1, advancing `data_id`), - the exact bug #2 fixed for throughput but never applied to these. **Re-key to the `data_id` axis** (extend the - `_progress_axis` helper) → they run and directly validate root #6. *(shared parity engine → full-suite gate.)* -- **8 per-phase timing rungs skip on MISSING telemetry → RIGOR GAP = the root-#6 instrument (step 2):** - `training_budget_s`, `gpu_compute_s`, `mqtt_fetch_s`, `weights_to_gpu_s`, `weights_to_ram_s`, `pre_train_s`, - `post_train_s`, `trainer_phase`. Adding the per-phase wall telemetry un-skips them AND enables the #6 - decomposition. *(root #10 / step 2.)* - -### Open issues — cross-baseline master index (OPEN top, RESOLVED moved down; 1-sentence issue + fix) - -*Rewrite in place: when an issue resolves, move its row into the RESOLVED block with a crisp principled-fix -summary. Detailed rationale lives in the "Open roots" prose below + §K.* - -| # | issue (1 sentence) | baseline(s) | principled fix / next step | -|---|---|---|---| -| **#13** ⭐⭐ | **sim delivers a SLOWDOWN not speedup (`sim_rate`≈0.37×).** Phase-2 skips **LANDED (K-D23)**: the doc's ranking was INVERTED by the `134801` telemetry + async_cifar10 map — the trainer `await_join`/`recv_wrapper` is barrier-wait that OVERLAPS irreducible agg eval+GPU (recv MUST stay), and the agg grace/`sleep(0.1)`/`await_join` were ALREADY sim-correct. The genuinely skippable per-round wall was the trainer `pause_execution` throttle + the `_check_availability` avail-spin. | fwdllm, fwdllm_plus (+ fluxtune) | **Phase-2 skips DONE (fwdllm-only, `and not self.simulated`, 146 green).** REMAINING for real-run `sim_rate>1`: **#6/Phase-4c** must fold `eval_s` (~13s/data_id, the syn_0 wall dominator) into the vclock, then the sign-off run confirms `vclock ≥ wall` live. Phase 2 exit (`sim_rate>1` on SYNTHETIC timings) is met. | -| **S1** ⭐ | sign-off run unusable: every sim died on `[SIM_WALL_CEILING]` (wall≫vclock for a real-compute sim), real died on `max_runtime_s` wall, so real/sim reached mismatched `data_id` — no comparison possible | all (2026-07-04 PM) | **FIXED (Phase 4a / K-D24):** `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` (was 1×), so the ceiling is a runaway OUTER safety, not the primary stop — a real-compute sim is no longer truncated before its vclock/data_id stop. Sign-off run should also **stop on a matched `--max-data-id`** (present: `max_data_id_progress`), not a wall/vclock budget. | -| **#12** | `--delays on` WAS honored by the trainer (real slept 0.4–1.8s; sim modeled it), but (a) `training_delay_factor=10` (`trainer_base.yaml:96`) shrank the 4–18s registry delay ÷10, (b) the agg-side sim-folds (`sim_model_eval_time`/`sim_straggler_spread_s`) were never wired from the launcher, (c) agg `training_delay_enabled` is an orphaned/misleading field, (d) banked `execution_config.yaml` omits the flag | all | **Config-flow fix (Phase 3 below).** Fan `--delays`/folds into `config_overrides.hyperparameters` for BOTH roles (mirror `agg_goal` single-source); expose `training_delay_factor` as a knob; bank the effective values + a post-launch assertion that banked config == requested flags; retire/wire the orphan. Extensible to async_cifar10 (same `runner.py`). | -| **#6** ⭐ | sim vclock under-models real wall — genuine unmodeled terms are `eval_s` (server eval, the syn_0 wall dominator) + barrier straggler under-spread. | fwdllm, fwdllm_plus (+ fluxtune 2.4× wall) | **FOLDS ENABLED (Phase 4c / K-D24):** B1 `sim_model_eval_time:true` charges the MEASURED eval_s to the vclock (self-calibrating, no constant); B2 `sim_straggler_spread_s:0.9` widens the barrier spread — both live in the 3 sim yamls. Do NOT add drain-tail/sleeps/localhost-MQTT (artifact, principle #1). `wall_disparity` DIAG rung (data_id axis) drives the residual → ~0; **final B2 value is calibrated FROM the sign-off run** (emergent). Ceiling decoupled (4a) so the vclock-budget stop is now reachable. | -| **#7** | fwdllm_plus real STALLED — 5 agg-rounds then dead-spun to the 1 h cap (log ballooned to 292 MB); **AND (new, syn_0) real is 4× slower/round even with NO scarcity** (59.7 vs fwdllm 15.5 s/round, 8 rounds/data_id 4 in 600s) | fwdllm_plus | **Scarcity half ROOT-CAUSED (K-D20) + FIXED (Stage C):** oracular `mobiperf_2st` avail 1/10 < `agg_goal=10`; real sync barrier spun. **Slowness half (NEW, syn_0):** `reselect_each_iteration=True` per-iteration re-select + oracular read (29 sel events vs fwdllm's 1) — a distinct real-config cost, profile from the banked per-phase log; it caps fwdllm_plus real's data_id below sim's (#4 real-side truncation). Not a sim bug. | -| **#8** | `field_coverage` INV fail: `task_recv.sim_send_ts` absent both modes | fwdllm, fwdllm_plus | Emit `task_recv.sim_send_ts` (over-instrument) or add it to the coverage alias tuple; unblocks K6. | -| **#9** | `failsafe`/K5 false-positive: sim WALL ≫ vclock (real forward-grad compute) | fwdllm, fwdllm_plus | K5 should compare sim wall vs the RUN wall budget, not the vclock, for a real-compute sim (checker fix). Partly reframed by piece A's `sim_wall_ceiling_s`. | -| **#10** | 12 rungs SKIP as rigor gaps (4 advance rungs wrong-axis, 8 phase rungs missing telemetry) | fwdllm, fwdllm_plus | Re-key K3a/K3b/K3/K4 to `data_id` (shared engine → full suite); add per-phase timing telemetry (= step 2, feeds #6). | -| **#11** | real-mode critical-path waste surfaced by profiling: fixed `time.sleep(0.1)` pads (≥28.7 s/run, `fwdllm_aggregator.py:2209/2338`) + one-grad-per-poll-tick drain tail (2.48 s/round) | fwdllm, fwdllm_plus (real) | Correctness-preserving optimization (profile-driven): verify each sleep's purpose, gate/remove off the critical path; drain-tail is the real 1-per-pass clamp (leave the sim all-k barrier). Watch GPU-shared trainer compute. Do NOT change grad values / cadence. | -| — RESOLVED ↓ — | | | | -| ~~#1c~~ | R1 in-flight overlap 0%→44.7% (fluxtune) | fluxtune | **FIXED K-D19** — root was guard released on grad RETURN, not `all_selected`; deferred release to COMMIT (`_release_end_on_return`), fwdllm-only. Emergent smoke owed. | -| ~~#1~~ | async concurrency starved to ~1.5 vs real ~5.8 (D-e) | fluxtune | **FIXED K-D16** — Option-A two-lifetime split (free slot on return, hold guard to commit). | -| ~~#1b~~ | fluxtune K-D16 re-run deadlocked | fluxtune | **FIXED K-D17** — drain keys on `_sim_buffer` not RECV; triplet stamped at RETURN not dispatch. | -| ~~#2~~ | clock family collapsed to `sim_rounds=1` | all | **FIXED** — re-key throughput/commits/terminal to the `data_id` axis (`_progress_axis`). Exposed #6. | -| ~~#3~~ | field_coverage false-fail on gpu/budget field names | all | **FIXED** — coverage spec accepts fwdllm aliases (tuple of names). | -| ~~#4~~ | V1/V2/g2 short-run truncation | fwdllm (cleared), fwdllm_plus | **CLEARED for fwdllm** at 10 `data_id`s; for fwdllm_plus it's real-side truncation → folds into #7. | -| ~~#5~~ | fluxtune `selection_detail` in_flight (2.7 vs 9.75) | fluxtune | **MOSTLY FIXED K-D17b** — hold the compute slot to COMMIT so `len(selected_ends)` = virtual in-flight. | - -### ⭐ Next implementation plan (2026-07-04, post-sign-off) — SPEEDUP + CONFIG-FLOW + PARITY, phased - -*This supersedes the "Pre-next-run implementation plan (K-D21)" below (that plan's sign-off run RAN and was -unusable — see the 2026-07-04 status block at the top of §H). The sign-off run surfaced THREE root causes that -must be fixed, pytest-gated, BEFORE any next launch (principle #11). Derived from three codepath traces -(async_cifar10 vclock principles; the bash→yaml→runtime config flow; the fwdllm sim vclock/wait machinery). -Phases 0→1 are prerequisites; Phases 2 (speedup) and 3 (config-flow) are INDEPENDENT parallel threads; Phase 4 -depends on Phase 2. Every phase ships telemetry + pytest and ends pytest-green (no run in the inner loop).* - -**PROGRESS (2026-07-04, update in place):** `✅ Phase 0` principles #12/#13 locked · `✅ Phase 1` speedup -instrumentation — `wall_elapsed_s`+`sim_rate` on agg_round, live `[VCLOCK_PROGRESS]` log, `sim_speedup` DIAG rung -(VALIDATED on the banked `134801` sim: sim_rate=0.377, wall_speedup=0.986, "SLOWDOWN"), rung+emit pytests green; -**Phase 1d PDF plot DEFERRED** (metric already in telemetry+log+rung; the analyze_run.py plotter add is low-value, -do with Phase 4) · `✅ Phase 3` config-flow (operator chose 3b framework fan + 3c factor knob) — runner.py fans -`enable_training_delays`(+factor) into the aggregator (single source, both roles agree; safe: `training_delay_enabled` -is read ONLY by the trainer, never the agg base, so async_cifar10 is behavior-neutral), honored-100% tripwire in -`_build_aggregator_config`, effective flags banked in `execution_config`, `--delay-factor` CLI, fan+bank pytests green -· `✅ Phase 2` **speedup leak (root #13) — DONE, pytest-green (K-D23).** 2a decision made FROM the banked -`134801` telemetry (principle #11a): the trainer inter-round `mqtt_fetch_s` is barrier-wait realized by the -**blocking recv** (~15s at each data_id boundary OVERLAPS the aggregator's irreducible server eval ~13s + -real GPU — NOT an independent skippable sleep; recv MUST stay, it delivers the real weights for grad -mode-invariance, exactly as async_cifar10 keeps real MQTT in sim). The only additive, fidelity-free per-round -wall the sim skips is the **trainer `pause_execution` throttle** (`time.sleep(1)` at the tail of EVERY loop -iteration, "don't overwhelm mqtt" — a real-transport artifact, principle #8) + the **`_check_availability` -avail-spin** (`while UN_AVL: sleep(1)`, which would freeze the vclock in sim; avail is agg-side enforced). Both -gated `and not self.simulated`; real byte-identical; `tests/mode -k fwdllm` 146 green (+6 new, -`test_fwdllm_sim_speedup_waits.py`). **Turned out fwdllm-ONLY (no shared-base edit)** — the aggregator was -already sim-correct (`_sync_sim_recv_first_k` 2s grace not 30/90s; `sleep(0.1)` pads gated real-only, Stage E; -`await_join` kept per async_cifar10; agg `pause_execution` not in the loop), so per principle #9 the scoped -suite is the gate, not full `tests/`. **REFRAME:** the leak inventory ranked "trainer await_join+recv" as -DOMINANT (#1) and "sleep(1) backoffs" LAST (#4); the telemetry + async_cifar10 map INVERT this. **Phase 2 alone -does NOT push the REAL-run `sim_rate>1`** — at syn_0 the sim wall is eval-dominated (#6/Phase-4c folds `eval_s` -into the vclock) — its exit is `sim_rate>1` on SYNTHETIC timings + pytest, which is met. · `✅ Phase 4` -**stopping rule + #6 sct model — DEV DONE, pytest-green (K-D24); emergent calibration is the sign-off run's -job.** 4a: `sim_wall_ceiling_s` DECOUPLED from the vclock budget (default `max_runtime_s × SIM_WALL_CEILING_ -FACTOR=20`, explicit override honored) so a real-compute sim (wall ≫ vclock by construction) is no longer -truncated before its vclock/data_id stop — root S1's direct fix; `test_fwdllm_early_stop_conditions.py` 19 -green. 4b: matched committed-`data_id` stop already present (`max_data_id_progress`; yamls 9999, `--max-data-id` -governs). 4c: the B1 eval-fold (`sim_model_eval_time`, self-calibrating on the MEASURED eval_s) + B2 straggler -spread (`sim_straggler_spread_s=0.9 ≈ real-compute-std·√12`) ENABLED in all 3 sim yamls (operator chose B1+B2); -`wall_disparity` DIAG rung (already landed, keyed on the fwdllm `data_id` axis) drives the residual → ~0 at the -run. **fwdllm-only edit (4a) → `tests/mode -k fwdllm` 148 green is the gate.** **NEXT: the Phase-1 sign-off -RUN** — matched `--max-data-id`, `--delays on`, `sim_rate>1` confirmed live. - -**Scope decisions (which issues live / die):** -- **KEPT & PROMOTED — #13 (the sim is a slowdown, not speedup).** New headline root. Was implicit in #6; now - first-class per the operator's vclock-is-virtual-wall-time model (principle #13). This is the prize. -- **REFRAMED — S1 & #6** are now understood as *downstream* of #13: the `[SIM_WALL_CEILING]` truncation and the - "vclock ≪ wall/round" gap are the SYMPTOM of the speedup leak, not independent bugs. Fixing #13 makes the - vclock-budget stop reachable and shrinks the #6 residual to the genuine unmodeled terms (eval_s, straggler). -- **REFRAMED — #12** is a config-flow bug (Phase 3), not a "flag dropped to trainer" bug (the trainer honored it). -- **My earlier S1 "matched-`data_id` stopping rule"** is DEMOTED to a Phase-4 convenience, not the primary fix — - the real fix is #13 (make the sim fast so a time budget is honored on both sides). Kept, subordinated. +## §H Dead-ends & corrections — do NOT retry +- **"K-D19 fixed fluxtune R1 / the NONE reset is a red herring."** REFUTED (K-D26). R1 stayed 60.4%. K-D19 + fixed the wrong release path (the aggregator's RETURN-path `cleanup_provided_ends`) and dismissed K-D17b's + NONE hypothesis by checking the *selection filter* (async_oort:1586, which excludes `all_selected` members) — + but the guard fails because `all_selected` MEMBERSHIP is deleted first, by (a) the 90s wall-timeout and (b) + async_oort reading the aggregator's `KEY_END_STATE=NONE` slot-hold as "trainer left." K-D17b was right. + *Lessons:* (i) an R1 fix is not done until the emergent smoke banks R1≤2%; (ii) when a guard "should exclude" + but doesn't, check whether the member is being *deleted* upstream, not just whether the filter reads it. +- **"K-D16 Option-A: free the compute slot on physical RETURN."** WRONG for virtual time — a returned-but- + uncommitted trainer is still in flight until the vclock reaches its sct; freeing the slot undercounted + `in_flight` 3×. Reverted to hold-to-COMMIT (felix-aligned, K-D17b). +- **"K-D16 fixed fluxtune; the re-run just banks numbers."** WRONG — it DEADLOCKED (K-D17): the sim drain was + gated on channel RECV state (not the buffer), and the re-pick triplet was stamped at DISPATCH (freezing the pool + before the first commit). The guard belongs at RETURN; the sim commit path keys on its OWN reorder buffer. +- **K-D6 "drop stranded grads at the agg-goal boundary."** REVERSED for async (K-D12) — the "|selected|≈agg_goal" + premise holds only for sync; fluxtune (c=10≫agg_goal=3) dropped ~7 grads/cycle → residence violation → 2× passes. + Commit-then-carry replaced it. Drop stays correct for sync. +- **"sim wall ≈ real, comparable."** WRONG — always check avg in-flight concurrency, not just total pass counts + (the concurrency collapse hid behind a similar pass count). +- **"D=0 smoke, so clock-family fails are artifacts."** WRONG — the runs are D>0; the fails traced to the rung + being keyed on `round` not `data_id` (#2). +- **Tuning `var_threshold` / `max_iterations_per_data_id` to close a cadence gap** — rejected; baseline-defining + knobs, not parity levers. A cadence gap is an upstream set/order/clock divergence. +- **Adding a scalar overhead to the vclock to close #6** — forbidden (principle #1/#12); the vclock is + `max(vclock, sct)`. Fold only genuine unmodeled compute terms (eval_s, straggler spread); artifacts stay off. +- **"Drive `wall_disparity`→0 by folding more terms into the vclock" (K-D24's #6 plan).** WRONG target — the + eval fold was the LAST genuine term; the residual gap was real's ~7.7s/round localhost transport ARTIFACT, so + folding further would over-charge the vclock (principle #1). #6 was a CHECKER-ANCHOR problem (K-D25): anchor + real on its intrinsic algorithmic clock, and retarget `wall_disparity` to |sim_vclock − real_intrinsic|. +- **Including the FedAvg merge in `intrinsic_span_s`.** Made real intrinsic overshoot the sim vclock by + ~fedavg×cycles (wall_disparity 1.6→5.5); the sim vclock charges barrier+eval but NOT fedavg, so intrinsic must + mirror that composition. (Re-add only if fedavg is ever folded into the sim vclock.) +- **Subtracting the slept delay from `post_train_s` to exclude it (Root B first attempt).** Went NEGATIVE when + the delay wall wasn't in the post window (fake-time tests / sim never sleeps). Correct fix: stamp + `_phase_post_start` AFTER the delay so post_train is pure post-proc, never negative. --- -#### Phase 0 — Principles & invariant (DONE in this doc edit; no code) -Locked **principle #12** (consult async_cifar10/PARITY.md vclock rules before any sim-clock change) and -**principle #13** (vclock = virtual wall-time; sim must produce speedup, `vclock ≥ physical_wall_elapsed`, -`sim_rate ≥ 1`). These are the acceptance criteria the rest of the plan is measured against. *No pytest — doc only.* - -#### Phase 1 — Speedup instrumentation (make the slowdown OBSERVABLE; prereq for 2 & 4) -*You cannot fix or verify #13 without a `sim_rate` metric — fwdllm emits `vclock_now` but no paired wall stamp -(principle #11b: ship the instrument with the mechanism). Mirror async_cifar10 which already has this.* -- **1a** Emit a per-round **`wall_elapsed_s`** (= `time.time() − agg_start_ts`) alongside `vclock_now` in the - fwdllm `agg_round` `extra` (`fwdllm_aggregator.py` `build_agg_round`, ~:1627-1655), so `sim_rate` is joinable - per round without external data. -- **1b** Add the **`[VCLOCK_PROGRESS] … sim_rate=…`** periodic log to the fwdllm sync/async loops (mirror - `syncfl/top_aggregator.py:1189-1203`; confirm whether fwdllm inherits/drives it — if not, wire it). -- **1c** New parity DIAG rung **`sim_speedup`** (`sim_rate = final_vclock / sim_wall`, and `wall_speedup = - real_wall / sim_wall` at matched `data_id`) in the shared engine; + re-use async_cifar10's **K7 `sim_rate_ok`** - and **K1 monotonic-vclock** for fwdllm (they exist; ensure they run on the fwdllm axis). Shared engine → - full-suite gate (principle #9). -- **1d** A speedup plot (sim_rate over `data_id`) in the run's `plots/`. -- **Pytest:** synthetic agg_round series → assert `sim_rate` computed correctly, vclock monotonic, rung PASS/FAIL - on known-fast/known-slow fixtures. **Exit:** running Phase-1 on the BANKED 2026-07-04 sim dirs reproduces - `sim_rate≈0.37×` from telemetry alone (validates the instrument before any code that changes behavior). - -#### Phase 2 — Fix the speedup leak (#13; THE core work) — make the sim skip real waits -*Goal: sim wall/round → real GPU only (~2s); `vclock ≥ wall`; `sim_rate > 1` and growing. Each sub-step is -`and not self.simulated`-gated or vclock-driven so REAL mode is byte-identical (principle #8), and shared-base -edits (`syncfl/`/`asyncfl/top_aggregator.py`) carry async_cifar10-regression risk → FULL `tests/` (principle #9).* - -> **COLD-START LEAK INVENTORY (from the 2026-07-04 codepath trace — don't re-derive).** The sim critical path -> pays these real waits (ranked by cost); each is the Phase-2 work. `agg` = `fwdllm_aggregator.py`, -> `fwd-tr` = `syncfl/fwdllm_trainer.py`, `FedSgd` = `trainer/forward_training/FedSgdTrainer.py`, -> `sync-base` = `syncfl/top_aggregator.py`, `async-base` = `asyncfl/top_aggregator.py`. -> 1. **Trainer inter-round fetch (dominant, ~1.8s steady / ~18s round-1 = `mqtt_fetch_s`):** `fwd-tr:213` -> `await_join()` + `fwd-tr:217-218` `recv_wrapper` real blocking MQTT recv — UNGATED. (2a; needs the -> barrier-vs-wire localization first.) -> 2. **Aggregator commit/barrier grace recv:** grad path `agg:756-758` `recv_fifo(..., timeout=grace)` looped to -> `RECV_TIMEOUT_WAIT_S=30` (`async-base:66`); sync barrier `sync-base:386-398`; `grace` has a **2.0s floor ×4 -> EMA** (`sync-base:347-352`). (2b.) -> 3. **Aggregator `await_join` on distribute:** `agg:2280` (sync) / `agg:2427` (async) — UNGATED. (2b.) -> 4. **Ungated `time.sleep(1)` backoffs / avail spin:** `fwd-tr:205`, `fwd-tr:230`, `FedSgd:457`. (2c.) -> -> **Already correctly skipped in sim — do NOT touch (byte-identical real; re-gating would be a no-op or a bug):** -> post-distribute settle pads `agg:2295-2297`/`agg:2434-2436` (`if not self.simulated: time.sleep(0.1)`); -> scarcity poll `_await_dispatchable_under_scarcity` early-returns in sim (`agg:2232`). And the base async -> pattern to MIRROR: starvation = a vclock JUMP to the next avail event (`async-base:648-672`), the sim SKIPS the -> real delay-sleep while charging `sct` (that's the speedup), `SimReorderBuffer` reconstructs order from `sct` -> not wall (`virtual_clock.py:52-60`). fwdllm's deviation is that its SYNC per-round trainer fetch (leak #1) has -> no async_cifar10 analog and was never gated — that's the core of #13. -- **2a (dominant leak)** The sim trainer `_fetch_weights` blocks on `await_join()` + real MQTT `recv_wrapper` - every round (`fwdllm_trainer.py:213,218`; `mqtt_fetch_s`~1.8s steady/~18s round-1). **DECISION REQUIRED** - (design, not obvious): this inter-round fetch wait must become ~free in sim WITHOUT breaking real message - transport (grads/weights are real). Options to evaluate against async_cifar10's pattern: (i) the wait is really - *barrier* wait (trainer idle until the aggregator distributes) → it should overlap in virtual time, charged to - the vclock not physical wall; (ii) on localhost the transport is ms — confirm the ~1.8s is barrier-wait not wire - time, then the fix is to not physically block the trainer between rounds. Localize from the banked `mqtt_fetch_s` - + phase telemetry FIRST (principle #11a). -- **2b** Aggregator `await_join` on distribute (`fwdllm_aggregator.py:2280/2427`) and the grace-bounded - `recv_fifo` (≥2.0s floor, `syncfl/top_aggregator.py:347-352`; `agg:756`) in the sim commit/barrier loops → - gate/shrink the grace to ~0 in sim (the sct reorder buffer, not a wall timeout, is the sim's ordering source — - principle #8 "real-transport artifact vs algorithmic property"). -- **2c** Ungated `time.sleep(1)` backoffs / avail spins (`fwdllm_trainer.py:205,230`; `FedSgdTrainer.py:457`) → - gate `and not self.simulated`. -- **Pytest:** spy/mocked-channel test asserting the sim path issues NO blocking real-wait on the critical path - (or a bounded ~0 grace); a driven test asserting `sim_rate > 1` on synthetic timings; **async_cifar10 - byte-identical** assertion for every shared-base edit. **Exit:** `sim_rate > 1` on a synthetic fwdllm round; - full `tests/` green. - -#### Phase 3 — Config-flow correctness (#12; INDEPENDENT thread) — flag → banked config → runtime, honored 100% -*Trace result: `--delays` reaches the trainer via a bespoke bridge (`runner.py:250`) but NOT the aggregator/sim -folds; there are ~10 layers that can set/override a delay value with disagreeing defaults; the effective value is -not banked. Design goal: one flag → one canonical sink → fanned to both roles, asserted post-launch, reusable by -async_cifar10 (same `runner.py`/harness).* -- **3a** Make the launcher patch write the aggregator-side knobs too: `--delays`/folds → - `exp["aggregator"]["config_overrides"]["hyperparameters"]` (`sim_model_eval_time`, `sim_straggler_spread_s`, - and `trainingDelayEnabled` if kept), symmetric to how `max_runtime_s` is already written there - (`run_sequential.sh:260`). Smallest correct fix. -- **3b** Better/extensible: fan the resolved flag into BOTH roles' `config_overrides.hyperparameters` as the last - merge layer in `runner.py` (mirror the `agg_goal` single-source pattern, `runner.py:531-543`), removing the - bespoke trainer-only bridge as a special case. **DECISION: 3a (local, fast) vs 3b (framework, extensible) — - choose per appetite; 3b is the principled one and helps async_cifar10.** -- **3c** Expose `training_delay_factor` (hardcoded `"10"` in `trainer_base.yaml:96`) as a launcher knob so - "delays on" can mean the full registry delay, not a silent ÷10. -- **3d** Bank the effective values: include `enable_training_delays`/folds in `create_execution_config`'s output; - add a **post-launch assertion** (in `expt_runner.render_and_gate` or a new `assert_banked_config`) that the - banked `execution_config.yaml` + `aggregator_config.json` equal the requested flags — BLOCK the run otherwise. -- **3e** Retire or wire the orphaned aggregator `training_delay_enabled` (the fwdllm aggregator reads it nowhere) - so the log stops misleading. -- **Pytest:** a launcher/render test that patches a config with `--delays on`, renders the yaml, loads it through - `runner.py`, and asserts both roles' runtime `Hyperparameters` reflect the flag + folds; a negative test that - the post-launch assertion BLOCKS on a mismatch. Extend the same test to an async_cifar10 config. - -#### Phase 4 — Parity stopping rule + #6 sct model (depends on Phase 2) -*Only meaningful once the sim is fast (`sim_rate>1`). Then:* -- **4a** Decouple `sim_wall_ceiling_s` from the vclock budget (generous multiple / separate `--max-wall-s`) → an - outer runaway safety, not the primary stop (which is vclock reaching `max_runtime_s`). -- **4b** For the parity comparison, stop both sides at a **matched committed-`data_id`** target (finite - `--max-data-id`), so C1/C2/V1/V2 compare at identical progress (the overnight 10-`data_id` runs passed for - exactly this reason). Demoted from "primary fix" to convenience. -- **4c** Calibrate B1/B2 folds (`eval_s` +3.34s, straggler-spread +2.3s) so `wall_disparity/data_id → ~0` (the - genuine #6 residual once the artifact waits from Phase 2 are gone). B3 WAN knob stays doc-only. -- **Pytest:** ceiling-decouple + matched-data_id stop unit tests; `wall_disparity` rung on synthetic. **Exit:** - gated by the sign-off run (below), not pytest alone. - -#### THEN (not before): the Phase-1 sign-off RUN -Only after Phases 0–4 are pytest-green + full `tests/` + shared-parity suite green: smoke → convergence per -baseline at a matched `--max-data-id`, D>0 (full factor), **with `sim_rate>1` confirmed live**. It banks: the -12 un-skipped rungs, `sim_rate`/`wall_speedup`, `wall_disparity→~0`, fwdllm_plus completing the matched data_id, -and the fluxtune R1 emergent smoke. Record in the scoreboard. - -### Pre-next-run implementation plan (staged — the "definition of done" before spending another run) — ⚠️ SUPERSEDED by the plan above (its sign-off run was unusable; see 2026-07-04 status) - -*Goal: land ALL dev that lets the NEXT parity run extract maximum information — un-skip the 12 rigor-gap rungs, -make #6/#7 first-class in telemetry, and close the sct model — so the run measures a nearly-complete ladder, -not a half-instrumented one. Rung math: 74 total; today **42 active / 32 skip**; after Stage A → **~54 active / -20 skip** (the remaining 20 skips are the legit Phase-2/config ones). Each task ships telemetry + pytest -(principle #11). Anchors in **K-D20**.* - -**PROGRESS (update markers in place as stages land):** -`✅ piece A` vclock-budget stop (landed, 17 tests) · `✅ K-D20` decomposition + anchors · -`✅ A1` trainer phase timing · `✅ A2` agg wall-decomp · `✅ A3` re-key advance rungs · `✅ A4` wall_disparity rung · -`✅ B1/B2` sct model (eval_s + straggler spread, config-gated) · `✅ B3` WAN knob (doc-only) · `✅ C` fwdllm_plus liveness · -`✅ D1/D2` task_recv.sim_send_ts + K5 real-compute exemption · `✅ E` sleep(0.1)-pad gating (#11). -**ALL landed & pytest-green (K-D21). NEXT: the Phase-1 sign-off RUN** — smoke → convergence per baseline; -it (a) exercises the 12 un-skipped rungs, (b) drives `wall_disparity`→~0 and calibrates B1/B2's `simModelEvalTime`/ -`simStragglerSpreadS`, (c) confirms fwdllm_plus completes 10 `data_id`s, (d) banks the fluxtune R1 emergent smoke. - -- **Stage A — instrumentation (un-skip 12 rungs; make #6/#7 measurable).** Depends on nothing. - - **A1** Trainer per-phase timing → un-skips the 8 phase rungs. Populate `_phase_times` in fwdllm's - `_fetch_weights` override (`fwdllm_trainer.py:168`) + emit `mqtt_fetch_s`/`weights_to_ram_s`/ - `weights_to_gpu_s`/`pre_train_s`/`gpu_compute_s`/`post_train_s`/`training_budget_s`/`trainer_phase` in - `FedSgdTrainer.train_with_data_id` extra (`:598-602`). Mirror `async_cifar10 main.py:784-908`. + manifest + pytest. - - **A2** Aggregator per-round wall decomposition → `barrier_wait_s` (dispatch→last grad), `drain_tail_s` - (last grad→commit, the artifact), `aggregate_fedavg_s`, `eval_s` (from the `eval_model` decorator) in - `agg_round`. Makes phases (b)–(e) telemetry, not log-grep. + pytest. - - **A3** Re-key the 4 advance rungs (K3a/K3b/K3/K4) to the `data_id` axis (extend `_progress_axis`) so they - stop reporting "<2 sim rounds." **Shared parity engine → FULL `tests/` + `examples/async_cifar10/scripts/ - parity` gate** (principle #9); assert async_cifar10 byte-identical (auto-detects `round`). - - **A4** New DIAG rung `wall_disparity` = `|real_wall − sim_vclock|` per matched `data_id` — the recurring - sanity metric to drive → ~0 (surfaced every run). + pytest. -- **Stage B — #6 `sct` model (piece B), reads A2's telemetry.** Fold the two GENUINE unmodeled terms into the - vclock; leave artifacts off (principle #1). - - **B1** `eval_s` per `data_id` onto the vclock (+3.34 s/round). - - **B2** Widen the sim barrier straggler completion-spread to match real `trainer_speed_s` dispersion - (2.66–4.05 s), so the k-th-smallest `sct` reflects real (+~2.3 s/round). + pytest. - - **B3** WAN payload-transfer term (1.96 MB up / 3.45 MB down) — NOT measurable on localhost; add a documented - config knob, **do not enable this run** (no ground truth). **Exit: `wall_disparity/data_id` → residual ≈ the - 4.7 s/round artifact, and throughput/K2/K3b within tolerance.** -- **Stage C — #7 fwdllm_plus real liveness** (so its 10-`data_id` run actually completes). Real-mode sync barrier - under scarcity: relax the agg-goal to available-count OR honor `wait_until_next_avl` (sleep-to-next-avail) - instead of spin-to-cap; bound the log. + pytest. Independent of A/B. -- **Stage D — cheap checker/telemetry gaps.** **D1** #8 emit `task_recv.sim_send_ts` (or alias). **D2** #9 - `failsafe`/K5 compares sim wall vs the RUN wall budget (not vclock) for a real-compute sim. -- **Stage E — optimization pass (#11; correctness-preserving, opportunistic).** Audit the `time.sleep(0.1)` pads - (`fwdllm_aggregator.py:2209/2338`, ≥28.7 s/run) — verify each one's purpose, gate/remove off the critical - path; profile GPU-shared trainer compute (A1's `gpu_compute_s` per trainer) for waste. **Re-run pytest to - prove grad values + cadence unchanged** (never trade correctness for wall). - -**Definition of done:** A + B + C + D pytest-green; **full `tests/` + shared parity suite green** (A3 spans the -stack); E opportunistic. THEN one run per baseline banks: 12 un-skipped rungs live, `wall_disparity`→~0, -fwdllm_plus completing 10 `data_id`s, and the fluxtune R1 emergent smoke. - -### Open roots (fix lowest-rung-first) - -**TOP: #1c (R1 residence regression) is now FIXED in code (K-D19) + pytest-green; the only thing owed on it is -ONE fluxtune sim smoke to bank the emergent R1/in_flight numbers — the LAST step, not the debugger (principle -#11c).** Roots #1 (D-e), #1b (K-D17 deadlock), #3 (field_coverage alias) FIXED; **#2 (clock-family AXIS) fixed -but exposed the RATE gap #6.** **Ranking of what's OPEN:** bank the #1c re-run → then the sync-baseline work -#6 (clock-RATE, shared with fluxtune #1) → #7 (fwdllm_plus real speed, gates #4) → #8/#9 (telemetry/checker -gaps). Attack #6 and #1's wall gap together — they are the same time-base question on the async vs sync path. - -1c. **[FIXED in code — K-D19; emergent re-run pending] R1 in-flight overlap 0% → 44.7% (fluxtune).** Root was - NOT the doc's hypothesis (async_oort *does* exclude `all_selected`) but the guard being released on physical - grad **RETURN** (`_process_single_trainer_message` → `cleanup_provided_ends`) while the carried grad hadn't - committed in virtual time. Fixed fwdllm-only via `_release_end_on_return` (defers the async+sim+residence - release to COMMIT, felix-aligned); + a `[SIM_R1_DISPATCH]` tripwire + `TestReturnPathGuardHeldToCommit` - (the RETURN-path blind spot). `tests/mode -k fwdllm` 110 green. Full rationale **K-D19**. **Remaining:** one - fluxtune sim smoke to confirm R1≤2% + `in_flight`~9.5 (emergent — measured, not assertable from banked - telemetry; the `020254` raw dir was cleaned). - -1. **[FIXED — Option A / K-D16] D-e async concurrency starvation (fluxtune).** *Symptom (pre-fix):* sim kept - ~1.5 trainers computing vs real's ~5.8, did fewer forward passes (89 vs 109) yet took **2.4× the wall** - (304 vs 125s) — re-dispatching ~1 trainer/agg-goal-cycle. *Root:* K-D12 parked a returned-but-uncommitted - trainer in async_oort's `selected_ends`, which drives `extra = c − len(selected_ends)`, so top-up refilled - ~1/cycle. `selected_ends`/`all_selected` conflated compute-slot occupancy with the re-pick guard. *Fix:* - Option A two-lifetime split — free `selected_ends` on RETURN (slot reopens → a DIFFERENT trainer refills - C, mirroring real's channel freeing on receipt), hold `all_selected` (guard) to COMMIT so - one-in-flight-per-trainer survives the carry + rollback. Details K-D16 / §F-D6. **NOTE:** the K-D16 re-run - surfaced root #1b (K-D17) — the concurrency-recovery numbers are only measurable AFTER the K-D17 fixes. - -1b. **[FIXED — K-D17] Fluxtune deadlock in the K-D16 re-run (drain-gate + triplet-at-dispatch).** The K-D16 - fluxtune smoke deadlocked: 2/10 grads committed, 0 `agg_round`, in-flight 0.28, 1663 empty select-rounds. - Two bugs: **(A)** `_aggregate_grads_async` gated the sim drain on `channel.ends(VAL_CH_STATE_RECV)` (empty - after the first greedy buffer-fill) → `"no ends yet"` ×1681 → the ~8 buffered grads never popped → agg_goal - never met; **(B)** K-D16 stamped the async_oort triplet at DISPATCH → whole pool matched `agg_version_state` - → `filtered_ends=0` → no re-dispatch, and the version never advanced to unfreeze it (bootstrap case D6 - missed). *Fix:* drain keys on `_sim_buffer`/`_sim_inflight_expected` (not RECV); triplet stamped on grad - RETURN (not dispatch). Full `tests/mode` green (314/7). Details **K-D17**. **Remaining: the fresh fluxtune - re-run** confirms recovery (concurrency, wall, U3, V1, selection) — emergent, measured not asserted. - -2. **[AXIS FIXED — #2; but exposed the RATE gap #6] Clock family re-keyed to the `data_id` progress axis.** - `throughput_parity` / `total_commits_parity` / `terminal_state_parity` now count units on the axis the run - advances (a unified `_progress_axis` / `_per_progress_last_event` helper): `round` for normal FL, committed - `data_id` for fwdllm (was collapsing to `sim_rounds=1`). async_cifar10 auto-detects `round` → byte-identical. - The re-key was correct, but with the axis fixed the 10-`data_id` run shows these rungs **still FAIL on a - genuine per-round TIME-BASE gap** (sim vclock-s/round ≪ real wall-s/round) — that residual is **new root #6**, - not this checker fix. - -3. **[FIXED — #3] field_coverage accepts fwdllm field aliases.** The coverage spec rows for - `gpu_compute_s` / `training_budget_s` now accept the fwdllm spellings `real_gpu_time_s` / - `sim_round_duration_s` (a spec field may be a tuple of accepted names). No more false-FAIL / benign SKIP. - -4. **[CLEARED for fwdllm; now REAL-side for fwdllm_plus] V1/V2/V5/g2 short-run truncation.** Per-`data_id` - series match on every FULL `data_id` and diverge only on the last, truncated one. **fwdllm: CLEARED** — at - 10 `data_id`s V1/V2/g2 all PASS (K-D18). **fwdllm_plus: now the truncation is on the REAL side** — sim did - 10 `data_id`s but real reached only ≈3 (1 h cap, root #7), so KS blows up comparing 10 vs 3. Fix is to make - real reach 10 (i.e. resolve #7), NOT code and NOT a knob. The `--max-data-id` default is now 9999 (K-D18) so - `--max-runtime-s` governs — but for fwdllm_plus even 1 h wasn't enough, so #7 must be understood first. - -5. **[MOSTLY FIXED — K-D17b] fluxtune selection_detail (in_flight).** Was NOT an oort-fidelity gap: the - `in_flight` metric = `len(selected_ends)`, and K-D16 freed `selected_ends` on physical RETURN, so it - measured physically-COMPUTING (~2.7) not virtual-time in-flight (~7.8, measured — close to real 9.75). - K-D17b holds the slot to COMMIT (felix-aligned) so `len(selected_ends)` = virtual in-flight. Residual - ~7.8 vs 9.75 (~20%) + preferred_duration to re-validate on the fresh re-run. - -6. **[OPEN — sync clock-RATE gap; new] throughput/commits/terminal: sim charges far less time/round than real.** - At 10 `data_id`s (both sides complete), fwdllm sim advances **~7.8 vclock-s/round** vs real **~34.9 - wall-s/round** (rel 0.78); fwdllm_plus **~8.1** vs **~86** (but real truncated, #7 confounds). Do NOT read - this as a cadence bug (V1/V2/g2 PASS — the WORK per `data_id` matches). It is a **time-base** question: - real's per-round WALL bundles MQTT/dispatch/aggregation/eval overhead that the sim (correctly, principle #1) - does NOT put on the vclock (`vclock = max(vclock, sct)`), and real emits **no `vclock_now`** so the checker - uses real WALL as real's axis. **First move (telemetry, not a run):** decompose real's per-round wall into - D + per-phase transport overhead using the existing `trainer_phase` / `mqtt_fetch_s` / per-phase splits, and - compare against the sim `sct` model. Two outcomes: (i) the surplus is real-transport overhead → the rung is - comparing WALL-vs-VCLOCK and should anchor real on an intrinsic per-round span (like async_cifar10's - `WALL_SEND−WALL_RECV` duration lesson), a checker fix; (ii) part is modeled D the sim under-charges → a sim - `sct` fix. **Shared-shaped with the fluxtune 2.4× wall gap (#1)** — resolve the classification once, apply to - both. Dead-end guard: do NOT add a scalar overhead to the vclock (principle #1, async_cifar10 dead-ends). - -7. **[FIXED in code — Stage C / K-D21; emergent re-run pending] fwdllm_plus real spun to the cap under - `mobiperf_2st` scarcity (agg_goal=10 > avail).** ROOT was the real sync barrier hot-re-dispatching when the - eligible pool < `agg_goal` (292 MB log, reached `data_id`≈3). Fixed: `_await_dispatchable_under_scarcity` - sleep-to-next-avail (WAIT/accumulate — cohort stays == agg_goal, parity-faithful vs the sim vclock-jump; - operator-chosen over relax-to-available). Self-terminates at `max_runtime_s`; one warn per stall. **Remaining: - the sign-off run confirms fwdllm_plus now completes 10 `data_id`s** (emergent). The separate question of WHY - fwdllm_plus real is slower/round (per-iteration reselection + oracular read) is a real-config profiling item, - localizable from the banked per-phase log — now that the run can complete, it is judgeable. - -8. **[FIXED — Stage D1 / K-D21] field_coverage INV `task_recv.sim_send_ts` absent.** `fwdllm_trainer._fetch_ - weights` re-emits `task_recv` carrying `sim_send_ts` (the override had dropped the base trainer's emission); - null in real, non-null in sim. Un-SKIPs K6. `test_fwdllm_trainer_task_recv.py`. - -9. **[FIXED — Stage D2 / K-D21] `failsafe`/K5 false-positive for fwdllm.** K5 flags sim WALL overshooting - the virtual budget by >20% (184s wall vs 78s vclock). For fwdllm this is EXPECTED and benign: the sim does - **real forward-grad GPU compute** (many passes), so sim wall ≫ vclock by construction — unlike async_cifar10 - where compute is cheap. FIXED: `failsafe_ok` now compares sim wall against the RUN wall budget (`max_runtime_s`) - for a real-compute sim (auto-detected via the progress axis) and SKIPs when no run budget is available, instead - of the wall-vs-vclock fallback. `TestFailsafeRealComputeSim`. - -### Fixes landed (what worked — do not redo) - -- **Phase 4 stopping rule + #6 folds (K-D24)** — 4a: `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` - (class const `SIM_WALL_CEILING_FACTOR`, explicit override honored) so a real-compute sim isn't wall-truncated - before its data_id/vclock stop (root S1). 4c: B1 `sim_model_eval_time:true` (self-calibrating eval_s fold) + - B2 `sim_straggler_spread_s:0.9` ON in all 3 sim yamls. `test_fwdllm_early_stop_conditions.py` 19 green; - `tests/mode -k fwdllm` 148. Do NOT re-derive the ceiling from 1× budget — it truncates real-compute sims. -- **Phase 2 speedup-leak skips (K-D23)** — gated the trainer per-round `pause_execution` `time.sleep(1)` - throttle and the `_check_availability` avail-spin `and not self.simulated` (fwdllm-only; real byte-identical). - The trainer `recv`/`await_join` and the aggregator grace/`await_join` are LEFT AS-IS (recv delivers the real - weights; agg already uses the 2s sim grace + real-only `sleep(0.1)` pads). `test_fwdllm_sim_speedup_waits.py` - (6), `tests/mode -k fwdllm` 146 green. Do NOT try to short-circuit the trainer recv — it's barrier-wait over - irreducible agg eval+GPU, not a skippable sleep (K-D23). Real-run `sim_rate>1` still needs #6/Phase-4c. -- **Pre-next-run plan A+B+C+D+E (K-D21)** — Stage C scarcity-wait liveness; A1 trainer phase timing; A2 agg - wall-decomp; A3 advance-rung re-key (shared engine, async byte-identical); A4 `wall_disparity` DIAG rung; B1/B2/ - B3 sct-model folds (config-gated OFF ⇒ byte-identical); D1 `task_recv.sim_send_ts`; D2 K5 real-compute exemption; - E `sleep(0.1)`-pad gating. Full `tests/` 854 passed / 7 skipped. Emergent calibration (wall_disparity→~0, B1/B2 - values, fwdllm_plus 10-`data_id` completion, fluxtune R1) is the sign-off run's job. - -- **Trainer sim clock** — additive `sct = send + gpu + D`, no-sleep on the sim path (Batch 1, K-D2/K-D3). -- **Async grad loop on the vclock** — purpose-built `_sim_recv_min_grad` sct reorder buffer + in-flight gate - + agg-goal-boundary rollback cleanup (Batch 1, K-D4/K-D5). -- **Sync barrier** — `_sync_sim_recv_first_k` (first-k-smallest); the `ends_not_selected_yet` "commit-1-per- - pass" clamp gated **real-only** so the sim barrier drains the full dynamic-K cohort in one pass (K-D11). -- **Variance-cadence rung layer** — V1-V5 / DK1-DK3 / G1-G2 + the pre-mutation `cycle_data_id` snapshot that - makes V1 exact (Batch 2, K-D9/K-D10). -- **Residence: commit-then-carry + R1/W1 rungs** — killed the 2× recompute (228→~89 passes); R1 in-flight - overlap exact 0% on all three (K-D12/K-D14). *(This fix also introduced D-e — the carry was right, the - slot-release timing was not; now resolved by K-D16 below.)* -- **D-e: Option A two-lifetime split (K-D16)** — *SUPERSEDED by K-D17b.* Freed the compute slot - (`selected_ends`) on physical RETURN while holding the `all_selected` guard to COMMIT. The slot-on-return - half was wrong for virtual time (undercounted `in_flight` 3×); reverted in K-D17b. The `all_selected` - guard-to-commit half survives. -- **K-D17: drain-gate + triplet-at-return (unblocks the K-D16 deadlock)** — (A) `_aggregate_grads_async` - now drains while `_sim_buffer`/`_sim_inflight_expected` is non-empty even when the channel has no RECV end - (the sct buffer, not RECV state, is the sim source of truth); (B) the async_oort re-pick triplet is stamped - on grad RETURN (in `_process_single_trainer_message`), not at dispatch, so the eligible pool isn't frozen - before the first commit. `test_fwdllm_sim_drain_and_repick.py`. -- **K-D17b: hold the compute slot to COMMIT (felix-aligned; fixes the in_flight mis-measurement)** — - `_sim_hold_busy_slots` now holds EVERY dispatched-but-not-committed trainer (computing ∪ carried) in BOTH - `selected_ends` and `all_selected` until commit, called per-commit in `_sim_recv_min_grad`. `in_flight` - (= `len(selected_ends)`) and `extra` now reflect virtual-time in-flight, matching felix - (`asyncfl::_sim_hold_busy_slots`/`_sim_recv_min`). Reverts K-D16's slot-on-return; safe now that K-D17 - fixed the drain. Full `tests/mode` green (315/7); `TestVirtualInflightSlotHold`. Concurrency/downstream - recovery measured by the fresh fluxtune re-run. -- **K-D19: R1 regression fixed — guard release deferred from RETURN to COMMIT (`_release_end_on_return`).** - Root was NOT the recorded `all_selected` hypothesis (async_oort *does* exclude it) but - `_process_single_trainer_message` freeing the re-pick guard on physical grad RETURN while the carried grad - was still uncommitted in virtual time. Fixed fwdllm-only (no shared `async_oort`/`asyncfl` edit — principle - #8); the async+sim+residence release now happens at COMMIT via `_sim_hold_busy_slots`, felix-aligned. Shipped - WITH a `[SIM_R1_DISPATCH]` dispatch-time tripwire + `TestReturnPathGuardHeldToCommit` (closes the RETURN-path - test blind spot). `tests/mode -k fwdllm` 110 green. Emergent R1/in_flight numbers pending one fluxtune smoke. -- **Piece A: vclock-budget stop (matched-budget convergence axis; step 1 toward #6).** `_check_early_stop_ - conditions` now interprets `max_runtime_s` as **wall in real / `vclock.now` in sim** (mirrors base - `max_experiment_runtime_s`, `top_aggregator.py:1140`), so one budget = "3600 wall-s of real work" AND "3600 - virtual-s of modeled work". A `sim_wall_ceiling_s` failsafe (default = budget) stops a runaway when the vclock - is under-modeled (root #6) — that ceiling firing IS the #6 signal until piece B lands. `tests/mode/ - test_fwdllm_early_stop_conditions.py` 17 green (4 new sim/vclock cases). **Does NOT alone fix #6** — the - vclock must first model real wall (step 2) or a vclock-bounded sim over-runs to the wall ceiling. -- **Clock-family re-key to `data_id` (#2)** — unified `_progress_axis`/`_per_progress_last_event` helper; - `throughput`/`total_commits`/`terminal_state` now count on the axis the run advances (fwdllm `data_id`, - normal FL `round`). async_cifar10 byte-identical (auto-detects `round`). `TestProgressAxisRekey`. -- **field_coverage aliases (#3)** — spec rows accept a tuple of field names; `gpu_compute_s`/ - `training_budget_s` also match fwdllm's `real_gpu_time_s`/`sim_round_duration_s`. `TestFieldCoverageAlias`. -- **W1 made asymmetric** — flags only a sim EXCESS over real (the 2× recompute it was built for); benign sim - under-compute no longer fires. -- **staleness_policy wired from config** — was silently `none` for every run (K-D15); + fedbuff staleness- - weighted accept for fluxtune (K-D13). Confirmed both sides now log `fedbuff`/`exact`/`round_data_id`. -- **camelCase key-collision** — fluxtune yaml carried both `stalenessPolicy:none` (base) and - `staleness_policy:fedbuff` (override); base won at pydantic resolution → ran as `none`. Fixed to camelCase. -- **vclock_now emitted** on agg_round (mirrors asyncfl) — confirmed present (sim 6/6/26 agg_rounds). -- **fwdllm_plus crash** — `read_trainer_unavailability` kwarg rename (`base_dir`→`metadata_dir`); no longer - crashes in `internal_init`. -- **Launcher fail-fast (D-d)** — detect early aggregator death / traceback; terminate trainers on non-zero - agg exit instead of burning the 30s-per-trainer EOT grace. - -### Dead ends & corrections — do NOT retry - -- **"K-D16 fixed fluxtune; the re-run just banks numbers" (believed at K-D16 landing).** WRONG — the K-D16 - re-run DEADLOCKED (2/10 grads, 0 `agg_round`). Two bugs (K-D17): the sim drain was gated on channel RECV - state (not the buffer), and K-D16 stamped the re-pick triplet at DISPATCH (freezing the pool pre-commit). - *Lesson:* stamping a "already contributed this triplet" guard at DISPATCH conflates in-flight with - contributed and, since the agg version only advances on commit, freezes the pool at bootstrap. The guard - belongs at RETURN. And the sim commit path's readiness must key on its OWN reorder buffer, never on the - real transport's RECV bookkeeping (principle #8). -- **K-D6 "drop stranded grads at the agg-goal boundary"** — REVERSED for async (K-D12). The - "|selected| ≈ agg_goal" premise holds only for the sync baselines; for c ≫ agg_goal (fluxtune) it dropped - ~7 grads/cycle → residence violation. Commit-then-carry replaced it. (Drop stays correct for sync.) -- **"sim wall ≈ real, comparable" (claimed after the first post-residence re-run).** WRONG — fluxtune sim is - **2.4× real wall** (D-e). The earlier "82 vs 120 passes, comparable" read the pass *count* and missed the - concurrency *collapse* behind it. Always check avg in-flight concurrency, not just total passes. -- **"D=0 smoke, so the clock-family fails are artifacts" (believed briefly this session).** WRONG — the runs - are **D>0** (real trainers sleep ≈1.13s/pass; sim charges ≈1.12s to the vclock). The clock-family fails are - real, and trace to the rung being keyed on `round` not `data_id` (open root #2) — not to D being off. -- **Tuning `var_threshold` / `max_iterations_per_data_id` to close a cadence gap** — pre-emptively rejected - (§G): these are baseline-defining knobs, not parity levers. A cadence gap is an upstream set/order/clock - divergence (here, D-e). - ---- - -## §I Batch 2 implementation map (code-level -- cold-start; NEXT to build) - -**Purpose:** durable engineering map so Batch 2 can start cold in a new context. Batch 2 = the -**variance-cadence rung layer** (V1-V5 / DK1-DK3 / G1-G2 in the shared parity engine) + **fluxtune oort -selection-fidelity validation**. It is a **checker + telemetry** batch, NOT a clock port -- the vclock is -already driven (Batch 1). Config-gated nothing new here (the rungs are pure functions over telemetry; they -only run when a `parity_checks` battery is invoked). **Re-grep line numbers before editing.** - -### I.1 The prize + the localization discipline (read first) -fwdllm's headline risk (§G) is that the **variance-gated cadence is feedback-compounding**: a tiny -per-cycle grad-pool order difference -> a different `iterations_per_data_id` -> a different committed- -data_id throughput, which may only surface at 3 h. The V/DK/G rungs exist to **catch and localize** that. -**Never fix an EMERGENT rung directly** (PARITY.md §1 / §F.4): walk down to the lowest variance-cadence -rung whose *inputs* are matched. `var_threshold` / `max_iterations_per_data_id` are **baseline-defining -config knobs, not parity levers** -- a cadence gap is always an upstream set/order/clock divergence. Rung -definitions + deps live in **PARITY.md §F.3 (modified K3a/K3b/K2/U3/K8/U2) and §F.4 (new V/DK/G)**; this -section states only the fwdllm **build deltas** (where the checker + telemetry go). - -### I.2 Where rungs plug in -- the shared parity engine (VERIFIED) -- Engine: `examples/async_cifar10/scripts/parity/checks.py`. Each rung is a **check function** - `f(real_agg, sim_agg, ...) -> result_dict`, registered two places: - 1. **`run_all_parity(...)`** (`checks.py:~3431`): add `results["v1_iter_per_data_id"] = v1_...(real_agg, - sim_agg)` etc. in a new **"Stage 6' variance cadence"** block (after Stage 5 updates, before/near the - Stage 6 clock block -- deps are Stage-5 ordering + Stage-1 clock). - 2. **The STAGE map** (`checks.py:~3560`, the `{ "rung_name": {"stage":, "role":, "deps": ()} }` dict): - add each new rung with its role/tier + deps from PARITY.md §F.4 (e.g. `v2_var_trajectory` role - MECHANISM deps `("v1_iter_per_data_id",)`; `v5_pass_ratio` EMERGENT; `dk3_eligible_metric` CONTROL - before `dk1_k_trajectory`/`dk2_c_trajectory`; `g1_grad_norm`/`g2_grad_pool_size` EMERGENT). -- Parsed telemetry shape (VERIFIED, `cli.py:74`): the loader hands each check `real_agg["agg_rounds"]` / - `sim_agg["agg_rounds"]` = the list of **agg_round events**, plus `selection_train`, trainer dicts, etc. - So the V/DK rungs read the per-cycle agg_round series. -- Tests: `tests/mode/test_parity_checks.py` (the existing 40-test fwdllm/shared battery). Add per-rung - tests the **same way the batch was gated for Batch 1**: construct synthetic real/sim `agg_rounds` pairs - with a **known-PASS** fixture and a **known-broken** fixture (e.g. a force-commit-rate skew, a var- - trajectory divergence) and assert PASS/FAIL. **Pytest-only, no run.** - -### I.3 Telemetry sources -- what the emitted agg_round ALREADY carries (VERIFIED) -The per-cycle agg_round event is emitted at `fwdllm_aggregator.py:~1372` (`build_agg_round`, -`flame/telemetry/events.py:108`). **VERIFIED** its `extra` dict carries exactly: -`data_id`, `iteration_per_data_id`, `var`, `var_threshold`, `var_good_enough`, `force_commit_planned`, -`is_async` (plus the generic `agg_goal`, `agg_goal_count`, `staleness`, `trainer_speed_s`, -`contributing_trainers`, `agg_observed_s`). NOTE: `_build_dynamic_kc_metrics` (`:~1190`) computes MORE -(`var_pass_rate`, `max_iter_per_data_id`, `n_eligible_train/eval`, `target_iter_per_data_id`) but that -dict feeds the **dynamic_kc controller**, and is **NOT** in the emitted event -- add fields to the `extra` -dict if a rung needs them. Mapping to rungs: -- **V1 (iterations-per-data_id dist / realized dynamic-K):** derive from the emitted - `(data_id, iteration_per_data_id)` series -- max `iteration_per_data_id` reached before each `data_id` - advances. **Present, no new emit.** -- **V2 (per-cycle `var` trajectory):** the emitted `var` per cycle. **Present.** -- **V4 (force-commit / `max_iterations_per_data_id` bypass rate):** the emitted `force_commit_planned` - boolean (aggregator computes `_force_commit_this_cycle` at `:~1266`, `iter+1 >= max_iter`). **Present -- - V4 = mean(`force_commit_planned`) over cycles. No new emit** (correcting an earlier assumption). -- **V5 (variance-pass ratio per window):** derive from the emitted `var_good_enough` boolean series - (pass ratio = fraction True per window). **Present** (or add `var_pass_rate` to `extra` if you want it - precomputed). -- **V3 (`cached_v` pool size over time, DIAG) + G2 (grad_pool size at commit):** `grad_pool` (`:286`) and - `cached_shared_grad_pool_trainable` (fwdllm `cached_v`, `:287`) sizes are **NOT emitted** -> **NEW emit**: - add `grad_pool_size` + `cached_v_size` to the agg_round `extra` if V3/G2 are exercised. -- **DK1/DK2 (`_agg_goal` / `dynamic_c` trajectory):** inert for all three current baselines (dynamic_kc - disabled) -> **DK rungs WARN/skip**. **DK3** (CONTROL, eligible-ends metric fed to the policy) lives in - the controller metrics dict, not the event -> add `n_eligible_train`/`n_eligible_eval` to `extra` to - wire DK3. Do NOT fork the shared controller. -- **G1 (per-update grad/JVP norm or SNR):** NOT emitted per update. Grad values are mode-invariant given - identical input+perturbation seed, so G1 should be ~0 -- a FAIL means a perturbation seed/order leaked. - **NEW per-update emit** (trainer-side grad norm, into `trainer_round` or a new event) IF G1 is exercised; - else defer with a **logged skip** (§G "no silent caps"), do not silently drop. - -### I.4 fluxtune oort selection-fidelity -- VALIDATION, not a rebuild -The oort stack (`flame/selector/oort.py`, faithful pacer/UCB/D1-D8) is landed and the selection rungs -already exist in the engine STAGE map (`selection_detail`, `residence`, `selection_bias`, `selector_score`, -`preferred_duration`, `participation`, `decision_determinism`, `selection` -- stage 3). Batch-2 work: -**confirm they fire for fluxtune** (`async_oort` selector) and are correctly **inert (WARN/skip) for the -two `random`-selector baselines** (fwdllm/fwdllm_plus). Any gap is fwdllm-specific *wiring* (e.g. a -selection event field the fwdllm aggregator does not stamp), NOT an oort rebuild. Exit: A2c PASS + -`preferred_duration` (Sd) binding real~=sim + no pacer ratchet (fluxtune); S-rungs WARN for `random`. - -### I.5 Batch-2 pytest gate (pytest-only, no run) + exit rungs -1. Per new rung: known-PASS + known-broken synthetic `agg_rounds` fixtures in `test_parity_checks.py` - (force-commit-rate skew, var-trajectory divergence, iter-per-data_id divergence). -2. Any new emitted field (V4 `force_commit`, V3 `grad_pool_size`/`cached_v_size`, G1 grad norm): a small - emission test asserting the field lands in the agg_round/trainer_round event. -3. Full `test_fwdllm_*` + `test_parity_checks` green; flag-off byte-identical (adding fields must not - change real-mode behavior). -**Exit rungs (PARITY.md §F.4):** V1/V2/V5 PASS; K2 (committed-data_ids/vsec) PASS; V4 bypass-rate ~= real; -A2c PASS + Sd binding + no pacer ratchet (fluxtune); DK tracks-or-skips as configured. - -### I.6 After Batch 2 -- the Phase-1 sign-off run (first §H entry) -Only after Batches 1-2 are pytest-green: one smoke (~5 min) then one convergence run **per baseline** at -syn_0 (the `*_n10_smoke_sim.yaml` launchers + their real siblings) -> `parity_checks` full battery + -**C1/C2 at matched `data_id`**. **Enable modeled delays (D>0) in BOTH real and sim together** for the -convergence run (K-D8: the sim smokes keep D=0 for mechanics-only comparability; the parity run needs -D>0). Record in §H; gate to Phase 2 (Batch 3, availability). - -**How to launch (tooling landed post-Batch-2).** `expt_scripts/run_sequential.sh` drives the real<->sim -pairs via the shared harness `examples/scripts/expt_runner.{sh,py}` (§B.1 row 12): `--mode {sim|real|both}` -pairs each baseline and tags run dirs `_real`/`_sim` so `scripts.parity.cli` globs the pair; `--delays -{on|off}` sets `enable_training_delays` identically on both sides (K-D8); a pre-flight gate prints the -hyperparameters in three volatility tiers (① review-every-run, ② per-baseline, ③ config-baked) and blocks -infeasible configs (`agg_goal>c`, `num_gpus>visible`, `num_trainers --max-data-id ` (D>0 both sides). -Parity: `python expt_scripts/run_parity.py` (the repeatable post-fix checker) — it picks the LATEST -real/sim pair per baseline (parsing the baseline as an exact dir-name token, so `fwdllm` never captures -`fwdllm_plus`; the raw `scripts.parity.cli --batch` glob does), reads agg_goal from each run's config, -prints the pairs to confirm, then a compact cross-baseline rung summary. `--yes` skips the prompt; -`--validate` also reports live `staleness_policy` + `vclock_now` presence. - ---- - -## §K DEVIATION LOG -- fwdllm-specific divergences from async_cifar10 (running, newest-last) - -Per the DESIGN PRINCIPLE at the top: every place we chose NOT to copy async_cifar10 verbatim (or chose to -copy it despite a workload difference) is logged here with the options weighed + why. One entry per -decision. Keep appending; do not rewrite history (supersede with a new dated entry instead). - -- **K-D1 time_mode default = `"real"` (trainer getattr fallback).** async_cifar10 defaults `time_mode` to - `"simulated"` (`trainer/pytorch/main.py:174`). **Options:** (a) mirror "simulated" default; (b) default - "real". **Chose (b):** fwdllm's entire existing config corpus is `time_mode: real` and FedFwd shipped - with no sim path at all, so a "simulated" default risks silently flipping a manual/legacy config into a - half-built sim path. Sim variants set `time_mode: simulated` explicitly. **Rationale/where:** J.1(1), - `FedSgdTrainer.__init__` `getattr(..., "time_mode", "real")`. Revisit if a sim run without an explicit - yaml value is ever wanted. -- **K-D2 Additive delay model `sim_round_duration = gpu + D`, NOT cifar's `max(gpu, D)`.** cifar "sleeps - to fill a budget" (wall = max(gpu, D)); fwdllm's real mode sleeps D *on top of* GPU time (flat additive, - `_emulate_training_delay` docstring + existing `sim_round_duration_s = real_gpu + delay`). **Chose - additive** so sim sct matches what real wall-clock actually does for THIS workload -- copying cifar's - max() would desync real<->sim. **Where:** §I CRITICAL CORRECTION, J.1(1) `train_with_data_id`. (This is a - workload fact, not a preference -- logged so the sct formula difference is traceable.) -- **K-D3 Per-eval sct collapses to the same-iteration train sct (D4).** cifar stamps a DISTINCT eval sct - = `send + max(gpu, D_eval)` with a ~20x eval speedup (`main.py:1086-1096`). **Options:** (a) port a - distinct per-eval sct; (b) reuse the train sct. **Chose (b):** in fwdllm eval lives on the AGGREGATOR - (`eval_model` on the global model); the trainer loop runs `train_with_data_id >> _send_grads` every - iteration so `_sim_completion_ts` is always fresh, and the trainer's eval message is only a utility - report, not a separately-clocked commit. No 20x factor (forward-grad "train" IS a forward pass). **Where:** - D4, J.1(1). Revisit only if a trainer-side distinct eval task is added. -- **K-D4 Grad loop uses a purpose-built `_sim_recv_min_grad`, NOT asyncfl `_sim_recv_min` verbatim.** - **Options:** (a) call `_sim_recv_min` directly (it already handles ingest/gate/advance); (b) write a - grad-specific drain reusing the same primitives (`SimReorderBuffer`, `_advance_sim_clock`, - `_sim_inflight_expected`). **Chose (b):** `_sim_recv_min` does per-commit slot release via - `_sim_pending_commit`/`sel.all_selected` (asyncfl:615-627) and its withheld/staggered paths key on WEIGHTS - semantics -- both wrong for fwdllm, which must release slots on the AGG-GOAL boundary and commits grads. - Reused the primitives, forked the orchestration. **Where:** §J.2, J.1(3). -- **K-D5 Slot release + committed/buffer clear on the AGG-GOAL boundary, not per-commit.** async_cifar10 - releases a slot the moment its update commits (inside `_sim_recv_min`). **Chose boundary release** - (`_release_sim_slots_at_agg_goal`) because fwdllm commits one grad per call and a `data_id` spans many - agg-goal cycles with variance-FAIL rollbacks; per-commit release would strand/re-skip a re-contributing - trainer across a rollback (the §I.6 rollback hole). **Where:** §I.5, §J.2, J.1(3). Locked principle #4. -- **K-D6 `_sim_buffer.clear()` at the boundary DROPS stranded arrived-but-uncommitted grads.** async path - in cifar carries buffered futures across the barrier. **Chose drop** for rollback-cleanliness (a stranded - grad was trained on a pre-rollback model_version -> stale next cycle anyway). Benign at syn_0 - (|selected| ~= agg_goal). **OPEN watch-point:** if a rung shows lost updates, switch to commit-then-carry. - **Where:** J.4, J.2. **⚠ SUPERSEDED by K-D12 (2026-07-03):** the watch-point TRIGGERED for fluxtune - (async, c=10 ≫ agg_goal=3) — the "|selected| ≈ agg_goal" premise holds only for the sync baselines; - async drops ~7 grads/cycle → residence violation → switch to commit-then-carry (see K-D12, §L). -- **K-D7 U6 sync-barrier visibility-lag telemetry DEFERRED (compute+stash now, emit later).** The base - syncfl `_aggregate_weights` finalizes `update_visibility_lag_s` from `_barrier_anchored_lags(_real_round_durs)` - into a `_round_update_values` dict it then emits (`syncfl/top_aggregator.py:742-786`). fwdllm's - `sync_collect_and_accumulate_grads` has NO such per-round telemetry struct. **Options:** (a) port the whole - `_round_update_values` visibility-lag telemetry plumbing into fwdllm's sync path now; (b) compute the - barrier lags now, stash on `self._sync_barrier_lags_s` + log, defer the telemetry-event emission. - **Chose (b)** to keep Batch 1 the clock-drive port, not a telemetry port -- the value is COMPUTED and - available (so the U6 exit-rung wiring in the pytest/parity step can read it), just not yet emitted as an - event. **Where:** J.1(4). **TODO before the Batch-1 U6 exit-rung check:** either emit `_sync_barrier_lags_s` - via the telemetry manifest or have `parity_checks` read the stashed value. -- **K-D8 sim smoke launchers keep `enable_training_delays: false` (D=0).** The sim variants - (`*_n10_smoke_sim.yaml`) mirror the real smokes exactly except `time_mode: simulated` + `_sim` job id. - **Options:** (a) enable modeled delays (D>0) in the sim smoke so the vclock advances by the modeled - budget (the "interesting" sim behavior); (b) keep D=0 as in the real smoke. **Chose (b)** for step 5: - with D=0 the sct = dispatch + real GPU time, which STILL drives the reorder buffer + vclock advance + - sct ordering -- enough to validate the Batch-1 MECHANICS (does the sim path run/advance/not crash), and - it keeps the sim smoke byte-for-byte comparable to its real sibling. Modeled-delay-on (D>0) belongs to - the later convergence/parity runs and MUST be enabled in BOTH the real reference and the sim run together - (else real sleeps D while sim charges D to the clock -> that's the point; mismatched enable would be a - false divergence). **Where:** the three `*_n10_smoke_sim.yaml` headers. -- **K-D9 Cycle-relative cadence telemetry (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/ - `cached_v_size`) added to the agg_round `extra`, NOT reconstructed from the post-mutation series - (Batch 2).** §I.3 assumed V1 needs "no new emit" -- derive iterations-per-data_id from the emitted - `(data_id, iteration_per_data_id)`. But those fields are emitted **post-mutation**: on a variance PASS the - branch advances `data_id += 1` and zeroes `iteration_per_data_id` BEFORE `build_agg_round`, so a commit - event carries the **next** data_id (and iter 0). Binning cycles by the emitted `data_id` therefore - attributes each commit to the following bin (off-by-one), and a first-try pass emits nothing under its own - data_id. **Options:** (a) keep "no new emit" and reconstruct with off-by-one accounting (commit-maps-to- - previous-data_id, force-commit vs natural-pass both counted) -- fragile, and fwdllm's headline risk (§G) is - exactly this cadence being feedback-compounding and hard to debug; (b) snapshot the cycle identity BEFORE - the pass/fail branch and emit it unambiguously. **Chose (b):** `cycle_data_id`/`cycle_iteration` name the - data_id this cycle worked on + its 0-based attempt index, so **V1 = count(cycles) grouped by - `cycle_data_id`** is exact for both natural-pass and force-commit paths. The existing post-mutation - `data_id`/`iteration_per_data_id` are LEFT UNCHANGED (the analyzer's `progress_key` depends on them -- - adding fields, not changing them, keeps real-mode behavior byte-identical). `grad_pool_size` (G2) and - `cached_v_size` (V3) are snapshotted at the same point (before `_update_state_after_payload_prepared` - clears `grad_pool` on a commit). All four are `getattr`-guarded like `var_threshold` so test doubles - without the pools still emit. **Where:** `fwdllm_aggregator.py::_process_aggregation_goal_met` (snapshot - block before the `if self.var_good_enough:` branch + the `build_agg_round` `extra`); - `parity/checks.py::_iters_per_data_id` / the V/G rung functions; `tests/mode/test_fwdllm_agg_telemetry.py` - (`test_cadence_fields_snapshot_pre_mutation`, `test_cycle_data_id_is_pre_advance_on_commit`). -- **K-D10 DK3 (`n_eligible_train`/`n_eligible_eval`) and G1 (per-update grad/JVP norm) emit DEFERRED -- - checker reads-if-present, logged SKIP otherwise (Batch 2).** §I.3 flagged both as candidate NEW emits. - DK3's eligible-ends metric feeds the DynamicKC policy, which is **disabled for all three current - baselines** (fluxtune fixed K/C; fwdllm/fwdllm_plus have no controller), so the metric is never consumed; - G1's per-update grad norm should be ~mode-invariant (grad values depend only on input+perturbation seed) - and needs a **trainer-side per-update** emit, a larger surface than this checker+agg-telemetry batch. - **Options:** (a) wire both emits now (add the eligible-count loop to every agg_round; add a per-update - trainer emit) despite no active consumer; (b) implement the **checker** rungs to read the fields when - present and **SKIP with a §K-D10-referencing note** when absent, deferring the emit. **Chose (b)** -- - matches the §G "no silent caps" discipline (a SKIP with a logged reason, not a dropped rung) and avoids - adding inert per-cycle overhead / a trainer-telemetry port to a checker batch. The DK3/G1 check functions - are still fully unit-tested (synthetic fixtures supply the fields for the PASS/FAIL paths + the absent- - field SKIP path). DK1 (K trajectory) similarly SKIPs when K is constant across both modes (fixed-K - baselines) rather than trivially passing. **Wire the emits when a DynamicKC baseline (DK3) or a G1 - exercise is added.** **Where:** `parity/checks.py::eligible_ends_metric_parity` / `grad_norm_parity` / - `agg_goal_trajectory_parity`; `tests/mode/test_parity_checks.py` (`TestDK3EligibleEndsMetric`, - `TestG1GradNorm`, `TestDK1AggGoalTrajectory`). -- **K-D11 `ends_not_selected_yet` "commit 1 per pass" clamp gated real-only (`and not self.simulated`).** - Surfaced by the first Phase-1 syn_0 sign-off smoke (`--mode both`, 2026-07-03): the **real** run aggregated - and hit `max_data_id`; the **sim** run deadlocked to `max_runtime` with 0 aggregations. Root cause: in - `sync_collect_and_accumulate_grads` (`:~1486`) the flag clamps `num_min_req` (the barrier `first_k`) to 1. - That clamp is a **real-transport draining discipline** — it only works because uncommitted messages persist - in the channel queue and are re-collected next pass (real accumulates 1→agg_goal over ~10 passes). The sim - barrier `_sync_sim_recv_first_k` is single-pass: it drains the whole cohort, commits `first_k`, and **drops - the rest** (slots release at the agg-goal boundary, K-D5). With `first_k=1` it committed 1 and discarded 9; - trainers had already sent once and block in `_fetch_weights`, so the queue never refilled → goal never met. - The `len(ends) >= agg_goal` clause that trips the flag on a *full* fresh cohort is itself an overload from a - real-mode stall hotfix (#42); the flag's original meaning was just "no cohort selected yet". **Options:** - (a) revert the #42 overload globally — risks re-opening the real stall; (b) gate the clamp real-only. - **Chose (b):** real path byte-identical (`self.simulated=False`), async_cifar10 untouched (flag is - fwdllm-only), sim keeps `num_min_req=min(agg_goal, len(recv_ends))` and commits the full dynamic-K cohort - in one barrier pass. Instance of locked principle **#8** (real-transport artifact, no sim analog). **Where:** - `fwdllm_aggregator.py::sync_collect_and_accumulate_grads`. **Verify on re-run:** sim `[SIM_BARRIER]` shows - `first_k=10` + non-zero `buf_depth`, `committed` reaches agg_goal, `data_id` advances to `max_data_id` - (stops on the cap, not `max_runtime`), eval metrics emit; then parity-compare vs the real sibling. -- **K-D12 fluxtune async grad path: commit-then-CARRY the surplus buffer + hold residence (REVERSES - K-D6 for the async baseline).** Surfaced by the 2026-07-03 3-baseline smoke (§H, §L): fluxtune sim ran - 2× the forward passes of real. **Options:** (a) keep K-D6 drop (rollback-clean, but for c≫agg_goal it - discards ~7 arrived grads/cycle → residence violation → 2× compute → cadence divergence); (b) carry the - arrived-but-uncommitted grads across the agg-goal boundary and hold their trainers busy until commit — - the fwdllm analog of felix `simInflightResidence` (§3.resid) + oort `simInflightCarryover` (§4.9). - **Chose (b) for async (fluxtune); (a) stays correct for the sync baselines** (fwdllm/fwdllm_plus have - c≈agg_goal, no surplus). K-D6's "benign at syn_0" premise held only for `\|selected\|≈agg_goal`; async - breaks it. Full remediation design + staging is **§L**. **Where:** `_release_sim_slots_at_agg_goal` - (clear-before-hold ordering bug) + `_sim_hold_busy_slots` + the `sim_inflight_residence` gate. Instance - of locked principle #4 (slot residence must survive the boundary) and #8 (don't apply a sync-barrier - discipline to the async path). -- **K-D14 R1/W1 residence telemetry sourced from an ECHOED per-contribution interval, not the aggregator's - per-end dispatch stamp (Batch 2.5).** R1 (in-flight overlap) needs each grad's exact `[dispatch, - completion]` interval. **Options:** (a) read dispatch from the aggregator's `PROP_SIM_SEND_TS` end - property at commit time; (b) have the trainer ECHO `SIM_SEND_TS` back in the grad message (it already - stores the received stamp) and capture `[SIM_SEND_TS, SIM_COMPLETION_TS]` in - `_process_single_trainer_message`. **Chose (b):** the property (a) is overwritten on a re-dispatch, so if - residence is broken (exactly the case R1 must catch) it would read the NEW dispatch and hide the overlap; - the echoed pair is immutable per contribution. Emitted per cycle as `contributor_intervals` on the - agg_round event; real mode falls back to the wall interval (`PROP_ROUND_START_TIME`..receipt). W1 - (compute-conservation) reads forward passes from `trainer_round` counts vs committed contributors — both - already emitted. The rungs SKIP cleanly on the sync baselines (no re-selection ⇒ no ≥2 intervals/trainer) - and non-fwdllm runs (field absent). R1 is wired UPSTREAM of V1 in `CHECK_META` so the dep chain proves a - cadence divergence is downstream of a residence violation. **Where:** `fwdllm_trainer.py` (SIM_SEND_TS - echo), `fwdllm_aggregator.py` (`_sim_contrib_intervals` capture + `contributor_intervals` emit), - `parity/checks.py::inflight_overlap_parity`/`compute_conservation_parity`, `validate_real.py:: - check_grad_residence`, `tests/mode/test_fwdllm_sim_grad_residence.py` + R1/W1 fixtures. -- **K-D15 `self.staleness_policy` was never wired from config — the gate silently ran as `none` for every - run (as-built fix, Batch 2.5).** Implementing D-c (fedbuff staleness-accept) surfaced that - `_process_single_trainer_message` read `getattr(self, "staleness_policy", "none")` but NOTHING ever - assigned `self.staleness_policy` — so `round_data_id`/`exact`/`fedbuff` configs were all inert (always - "none"). **Fix:** set `self.staleness_policy` in `internal_init` from - `config.hyperparameters.staleness_policy`, and recognize `fedbuff` as an explicit ACCEPT policy (no reject, - no "unrecognized" warning) so its carried stale grads flow into the EXISTING fedbuff down-weighting - (`aggregate_grads_from_trainers`, `rate = 1/sqrt(1+staleness)` when `_weighted_aggregation_enabled`). A - guard warns if `staleness_policy=fedbuff` is set without a grad-aggregation-weighting optimizer (rate - stays 1.0). The U3 staleness telemetry (`_cycle_staleness` on agg_round) already emitted, so no new emit. - **Where:** `fwdllm_aggregator.py::internal_init` + `_process_single_trainer_message`; fluxtune yamls - (`staleness_policy: fedbuff`, real+sim identical); `tests/mode/test_fwdllm_agg_telemetry.py:: - TestStalenessPolicy`. -- **K-D13 fluxtune staleness_policy = fedbuff staleness-weighted accept (resolves §L D5; provisional).** - fluxtune's yaml leaves `staleness_policy` unset → defaults `none` (no staleness handling), wrong for an - async fedbuff baseline whose carried grads (K-D12) are stale by construction. **Options:** (a) fedbuff - staleness-weighted accept (down-weight by `V′−V`); (b) `round_data_id` reject+record (fwdllm_plus's sync - policy); (c) keep `none`. **Chose (a)** — fluxtune IS async fedbuff, so accept-and-down-weight is the - faithful treatment and feeds a non-trivial U3 staleness signal; **provisional** (may revisit the policy - later). Set **identically in real and sim** (a baseline definition, not a parity lever — never tuned to - close a gap; PARITY.md §F.4 / principle #3). **Obligation:** confirm the fedbuff optimizer weighting path - is active and `staleness = V′−V` emits on the commit event (U3). **Where:** `fluxtune_n10_smoke*.yaml` - `staleness_policy`; fedbuff optimizer weighting; commit-event `staleness` emit. Implement in §L step 5. -- **K-D16 D-e resolution — Option A two-lifetime split (free compute slot on RETURN, hold re-pick guard to - COMMIT).** The K-D12 commit-then-carry fix parked a returned-but-uncommitted trainer in async_oort's - `selected_ends`, which drives the dispatch top-up (`extra = c − len(selected_ends)`), so the pipeline - refilled ~1/cycle and sim wall blew up 2.4× (§H open-root D-e). **Decision gate (D6) answered by reading - the code:** async_oort's triplet filter (`async_oort.py:1660`, skip if `trainer_version_states[end] == - agg_version_state`) is a WITHIN-cycle guard only — the triplet advances every agg-goal cycle, so it does - NOT block re-pick of a *carried* trainer across the boundary (its recorded triplet ≠ the new current one). - So **Option B is unsafe** (its fallback guard doesn't cover the carry window), and `_trainer_state_dict` - was in fact **never populated** (filter inert). **Chose Option A** (§F-D6): async_oort already tracks the - two lifetimes in SEPARATE ledgers — `selected_ends` = compute-slot occupancy (drives `extra`), - `all_selected` = re-pick guard (drives `filtered_ends`) — they were merely conflated. Override fwdllm's - `_sim_hold_busy_slots` to hold a RETURNED/carried trainer in `all_selected` ONLY (frees its slot so a - DIFFERENT trainer refills concurrency to C, mirroring real freeing on channel receipt) while - still-computing trainers hold BOTH; the guard clears on COMMIT so one-in-flight-per-trainer survives the - carry boundary + variance-FAIL rollback (principle #4). ALSO populate `_trainer_state_dict` at dispatch = - `(model_version, data_id, iteration)` and prune to the busy set at the boundary, so the triplet filter has - real state (the within-cycle guard the user asked to re-key) and the R1 rung can prove no per-triplet - re-pick. **Options weighed:** (a) explicit two-set split [CHOSEN — the ledgers already exist; smallest - faithful change]; (b) reuse the triplet filter as the guard [REJECTED — proven not to cover the carry - window across rollbacks]. **Scope:** `fwdllm_aggregator.py` override + dispatch (fwdllm blast radius); - async_oort/oort and the shared asyncfl `_sim_hold_busy_slots` UNTOUCHED (principle #8); sync path untouched - (K-D5/K-D11). **Where:** `fwdllm_aggregator.py::_sim_hold_busy_slots` (override) + - `_release_sim_slots_at_agg_goal` + `_distribute_weights_async` (triplet emit); - `tests/mode/test_fwdllm_sim_grad_residence.py::TestOptionASlotGuardSplit`. **Concurrency is emergent** — - the split restores dispatch top-up; the actual in-flight number (vs real ~5.8) is measured by the §L step-6 - re-run, not asserted analytically. - -- **K-D17 Two bugs that deadlocked the K-D16 fluxtune re-run (drain-gate + triplet-at-dispatch).** The - post-K-D16 fluxtune syn_0 smoke (`run_20260703_230407`) did NOT recover — it DEADLOCKED: 1 cohort of 10 - dispatched, **only 2 grads ever committed** (agg_goal=3 never met once), **0 `agg_round` events** (so - `vclock_telemetry`/`throughput`/`total_commits`/`terminal_state` all can't run), in-flight collapsed to - **0.28** (worse than pre-fix ~1.5), and the agg spun 1663 empty select-rounds (`feasible_extra: 0` ×1617) - until `max_runtime_s`. Two independent bugs, one masked the other: - - **Bug A — sim drain gated on channel RECV state, not on `_sim_buffer`.** `_aggregate_grads_async` - early-returned (`"no ends yet"`, **1681/1729** loop passes) whenever `channel.ends(VAL_CH_STATE_RECV) is - None`. But `_sim_recv_min_grad` greedily drains ALL ready channel messages into `_sim_buffer` on its - FIRST call (`recv_fifo(..., first_k=len(to_probe))`), which empties RECV — so the 8-9 already-received - grads sat in the buffer and were never popped (`buf_depth=9` then stuck). The commit path's readiness - must key on its OWN reorder buffer / in-flight set, not the real transport's RECV bookkeeping - (principle #8: RECV gating is a real-transport artifact, no sim analog). *Fix:* fall through to - `_sim_recv_min_grad` when `len(_sim_buffer) > 0 or _sim_inflight_expected` even with RECV empty; real - path unchanged (guarded on `self.simulated`). - - **Bug B — the K-D16 triplet was stamped at DISPATCH.** Stamping the whole cohort at the current - `_curr_agg_version` made every dispatched trainer match `agg_version_state`, so async_oort's filter - (`async_oort.py:1660`, skip if equal) excluded the ENTIRE pool → `filtered_ends=0` → no re-dispatch. The - version only advances at a commit boundary, which (via Bug A) is never reached → permanent freeze. **This - is the bootstrap case the D6 analysis missed:** D6 reasoned the triplet is safe because "it advances - every agg-goal boundary" — true only once commits flow; BEFORE the first commit (or whenever the - in-flight set alone can't reach agg_goal), it freezes the pool. Conceptually the triplet means "already - CONTRIBUTED this triplet," a property of a RETURNED trainer; an in-flight-but-not-returned trainer is - already guarded by its compute slot (`selected_ends`). *Fix:* stamp `_trainer_state_dict[end] = - _curr_agg_version` on grad RETURN (in `_process_single_trainer_message`, residence-gated), NOT at - dispatch. Preserves every K-D16 benefit (slot frees on return, guard holds return→commit, R1 intact) and - unfreezes the top-up. - Either fix alone breaks the deadlock (A lets the initial cohort drain to a commit which advances the - version and unfreezes the triplet; B restores the stream of RECV transitions that masked A pre-K-D16) — - both landed for correctness. **Where:** `fwdllm_aggregator.py::_aggregate_grads_async` (drain gate) + - `_process_single_trainer_message` (triplet on return) + `_distribute_weights_async` (dispatch stamp - removed); `tests/mode/test_fwdllm_sim_drain_and_repick.py` (`TestDrainGateNotBlockedByEmptyRecv` + - `TestRepickTripletStampedOnReturn`). **Scope:** fwdllm class only; async_oort/oort/asyncfl + sync path - untouched (empty triplet map ⇒ inert). Full `tests/mode` green (314 passed / 7 skipped). Concurrency/wall - recovery is emergent — measured by the fluxtune re-run, not asserted. - -- **K-D17b The K-D17 fluxtune concurrency "gap" was mostly a MEASUREMENT artifact — revert the K-D16 - Option-A slot split, realign with felix (hold the slot to COMMIT).** The post-K-D17 fluxtune run was - healthy (no deadlock) but `selection_detail` still FAILed: reported `in_flight` sim **3.38** vs real 9.75. - A diagnostic (`inflight_exp` = dispatched-not-committed vs `sel_ends` = `len(selected_ends)`) settled it: - the sim keeps **~7.8** trainers in flight in VIRTUAL time (close to real's 9.75), but the `in_flight` - telemetry reads `len(selected_ends)` ≈ **2.73** — trainers physically COMPUTING right now. *Root:* the - `in_flight` metric is `len(selected_ends)` (`selector/__init__.py:161`), and K-D16 Option A freed - `selected_ends` on **physical RETURN** — a wall event with no virtual-time meaning. In virtual time a - returned-but-uncommitted trainer is still in flight (its grad commits only when the vclock reaches its - sct), so its slot is genuinely occupied. Freeing it undercounts `in_flight` 3× AND makes - `extra = c − len(selected_ends)` read false free capacity (the selector churns `desired_extra` 7–10, - blocked by the guard). *Confirmed against felix:* `asyncfl/top_aggregator.py::_sim_hold_busy_slots` - (:1453) holds the FULL dispatched-but-not-committed set in `selected_ends` and `_sim_recv_min` (:606-627) - releases only on COMMIT; its docstring names freeing-the-slot-on-return as the "over-selection bug." So - fwdllm's Option-A split was a deviation from the proven reference; freeing the slot manifested as an - UNDER-count only because fwdllm's extra `all_selected` guard blocked the over-selection felix warns of. - *Fix (K-D17b):* rewrite `_sim_hold_busy_slots` to hold EVERY outstanding trainer (`_sim_inflight_expected` - ∪ buffered surplus, minus committed) in BOTH `selected_ends` and `all_selected` until commit, and call it - per-commit inside `_sim_recv_min_grad` (mirroring felix's per-commit reset, since recv_fifo marks - freshly-buffered ends RECVD and the channel strips their slots). Now `len(selected_ends)` = virtual-time - in-flight, so both `in_flight` and `extra` are correct. Safe to revert Option A *now* only because K-D17 - fixed the drain (the pre-K-D16 "concurrency 1.5" was the drain stall, not holding-to-commit). **Where:** - `_sim_hold_busy_slots` + `_sim_recv_min_grad` (per-commit hold) + `_release_sim_slots_at_agg_goal` - (docstring); `tests/mode/test_fwdllm_sim_grad_residence.py::TestVirtualInflightSlotHold` (+ updated - `TestCommitThenCarryResidenceOn`). Full `tests/mode` green (315/7). - - **VALIDATION RESULT (run `020254`, 2026-07-04) — PARTIAL WIN + A NEW R1 REGRESSION (do not ship as-is):** - - ✅ **in_flight measurement FIXED.** `sel_ends` 2.73 → **7.78**, matching `inflight_exp` (8.10). In the - parity battery `selection_detail.in_flight` = sim **9.49** vs real 9.75 (**rel_diff 0.027**, was 0.65). - The core goal is met — the telemetry now reports virtual-time in-flight. - - ✅ **V2 var-trajectory now PASSES** (sim 0.894 vs real 1.008, KS 0.177<0.20); U3/S2/conv/W1 pass. - - ❌ **R1 residence REGRESSED: 0.0% → 44.7% overlap** (real 1.8%, tol 2%). K-D17b re-dispatches a trainer - while its prior grad is still in flight. **Root lead:** `_sim_hold_busy_slots` resets buffered-but- - uncommitted ends to `KEY_END_STATE=NONE` (to stop the RECVD-strip of their slot, mirroring felix), but - **async_oort's selection filter keys on avail-state/end-state, NOT on `all_selected`** (`async_oort.py` - ~1643-1695 filters by `count_avl_train`+triplet; `all_selected` at :405 is only *logged*). So a NONE-state - buffered trainer looks idle/available -> re-selected -> overlapping in-flight interval -> residence - violation. fwdllm's async_oort does not exclude `all_selected` the way felix's path effectively does. - - ❌ still failing: `selection_detail.chosen` (sim 0.83 vs real 1.76), throughput/total_commits/terminal - (sim commits faster — 3 vs 1 at matched virtual budget 27.2s; partly short-run #4), field_coverage, - avail_composition, V1/g2 (#4 truncation). - - **STOPPED before the overnight run** — R1 is a correctness regression; banking 6h of runs on it would be - wrong. Resume steps are in **§H -> "NEXT SESSION (2026-07-05) pickup"**. - -- **K-D18 Overnight 10-`data_id` grounding runs + the `--max-data-id` default raised to 9999.** The operator - launched `run_sequential.sh --mode both --max-runtime-s 3600 --only fwdllm,fwdllm_plus` (then fluxtune - real-only), intending 1 h runs, but `run_sequential.sh`'s `--max-data-id` **defaulted to 10** and its - `patch()` always overrides the yaml, so every run stopped at `data_id=10` instead of at 1 h. This was a - useful accident — 10 `data_id`s is >3× the prior banked runs and cleared the #4 truncation for fwdllm — but - the trap is real, so the default is now **9999** (effectively unbounded; `--max-runtime-s` governs) in both - `run_sequential.sh` and the six `expt_scripts/*_n10_smoke*.yaml`; pass `--max-data-id 10` (or 3) for a - deliberately short run. **Parity read** (`run_parity.py --baselines fwdllm fwdllm_plus`, JSONs - `parity_fwdllm_20260704_022610` / `parity_fwdllm_plus_20260704_033206`): *fwdllm* 37 pass / 5 fail — cadence - (V1/V2/g2/R1/W1/U3/S2/conv) CLEAN, surviving fails = the clock-RATE family (open root #6), field_coverage - (#8), failsafe (#9); *fwdllm_plus* 31 pass / 10 fail — but its **real run hit the 1 h cap at `data_id`≈3** - (86 wall-s/round; sim did 10 in 81 vclock-s), so the cadence/eligibility/selection fails are REAL-side - truncation (#4) driven by real-speed root #7, NOT a confirmed sim bug. *fluxtune* ran real-only (no sim), so - no parity — its sim is unchanged from `020254` (R1=44.7%, #1c). **Discipline applied (principle #11):** all of - the above was read from the ALREADY-banked telemetry + logs (stop-reason lines, per-round `runtime.py` - timings, the parity JSONs); no new run was launched to reach these conclusions. - -- **K-D19 The R1 residence regression (#1c) — root-caused to guard-release-on-RETURN, fixed fwdllm-only + - pytest + tripwire. (Corrects the K-D17b/§H "NEXT SESSION" hypothesis.)** The hypothesis on record was - "`_sim_hold_busy_slots` resets buffered ends to `KEY_END_STATE=NONE` and async_oort doesn't exclude - `all_selected`, so they're re-selectable." **That was wrong on the mechanism:** async_oort *does* exclude - `all_selected` (`async_oort.py:1586-1587`) and the channel passes the full pool regardless of `KEY_END_STATE` - (`channel.py:221-240`) — so the NONE reset is a red herring for re-selection; exclusion depends solely on - `all_selected` membership. **Real root:** on the async accept path, `_process_single_trainer_message` called - `channel.cleanup_provided_ends(end)` on **physical grad RETURN, per-message, mid-cycle** - (`fwdllm_aggregator.py:1259-60` pre-fix) → `async_oort._cleanup_provided_ends` (:889-916) deletes the trainer - from `all_selected`. But in sim residence the grad is **carried** (its `sct` is in the future), so the trainer - is torn out of the re-pick guard while still in flight in VIRTUAL time → any `_distribute_weights_async` - before its commit re-dispatches it → R1 overlap 0%→44.7%. K-D17b's per-**commit** `_sim_hold_busy_slots` - re-asserts the guard, but too late — it can't cover the RETURN→COMMIT window. felix (`asyncfl:1316`) releases - the guard only at the agg-goal boundary, never per-return — the divergence. *This is the D6 "carry was right, - release TIMING was wrong" pattern applied to the guard (`all_selected`), not just the compute slot.* **Fix - (fwdllm-only, principle #8 clean — no shared `async_oort`/`asyncfl` edit):** extracted the release into - `_release_end_on_return`, which in async+sim+residence **defers the release to COMMIT** (owned by - `_sim_hold_busy_slots`); real mode / non-residence / sync unchanged (return≈commit → byte-identical). - **Over-instrument (principle #11):** a dispatch-time `[SIM_R1_DISPATCH]` tripwire warns the instant a still- - outstanding trainer is re-dispatched (violation visible live, not reconstructed offline). **Pytest:** - `TestReturnPathGuardHeldToCommit` in `test_fwdllm_sim_grad_residence.py` drives the RETURN path directly — - the blind spot that let R1=44.7% ship with `tests/mode` green (the prior residence tests only poked - `_sim_hold_busy_slots`/`_release_sim_slots_at_agg_goal`, never `_release_end_on_return`). `tests/mode -k - fwdllm` 110 green. **Remaining = emergent-only validation** (principle #11c): one fluxtune sim smoke to - confirm R1≤2% with `in_flight` still ~9.5 — measured, not assertable from banked telemetry (the `020254` raw - dir was cleaned). - -- **K-D20 Real per-round wall decomposition — grounds #6 (`sct` model) and root-causes #7. (telemetry-only, - no run.)** For fwdllm sync the real aggregator spends **13.4 s/agg-round**, the sim vclock **3.0 s** — but the - gap is NOT all a modeling deficit. Barrier-anchored decomposition (dispatch → 1st grad → 10th grad → commit → - eval, from `runtime.py` decorator lines + `RECV_FIFO` markers + `agg_round`/`agg_eval` telemetry): **(a)** - trainer compute + round-trip 3.96 s (gpu 1.05 s; `sim_round_duration_s` 2.19 s already modeled), **(b)** - straggler/sync-barrier wait 3.06 s (GENUINE), **(c)** aggregator drain tail 2.48 s (**ARTIFACT** — real clamps - `num_min_req=min(agg_goal,1)` and acks one grad per poll-tick; the sim's all-k barrier is the correct model), - **(d)** `eval_model` 8.68 s/`data_id` = 3.34 s/round amortized (GENUINE server eval, the largest unmodeled - term), **(e)** FedAvg + redispatch ~0.6 s (mostly the `time.sleep(0.1)` pads, ARTIFACT). **Net: ~8.7 s/round - genuine vs ~4.7 s/round artifact → the true sct gap is ~2.9×, not 4.5×.** So piece B folds ONLY `eval_s` - (+3.34) + a widened straggler spread (+~2.3) into the vclock; the drain tail, the ≥28.7 s of `sleep(0.1)` - pads, and localhost-MQTT/GIL stay OFF the clock (principle #1) — and become optimization targets (#11). **#7 - root cause:** fwdllm_plus real spun to the 1 h cap (log → 292 MB) because oracular `mobiperf_2st` drops avail - to 1/10 < `agg_goal=10` and the real sync barrier can't assemble the cohort (`wait_until_next_avl=False`); the - SIM completes the same trace (vclock jumps past the unavailability window), proving it's a **real-mode liveness - bug**, not a sim/model bug. Full anchors (trainer `_phase_times` already exist at `syncfl/trainer.py:143-150` - but fwdllm's `_fetch_weights` override drops them; per-field wrap points; the `async_cifar10 main.py:784-908` - reference) are in the Stage-A/A1 plan above. Feeds the entire "Pre-next-run implementation plan." - -- **K-D21 Pre-next-run implementation plan LANDED — Stages A + B + C + D + E (pytest-gated, config-gated; - emergent calibration deferred to the sign-off run).** All dev that lets the next run measure a nearly-complete - ladder is in code + pytest-green; nothing here needed a run (principle #11a/b — telemetry-first). - - **Stage C (fwdllm_plus liveness, decision: WAIT/accumulate — parity-faithful, operator-chosen over relax):** - `_await_dispatchable_under_scarcity` (real mode) sleep-to-next-avail instead of hot-re-dispatching when the - eligible pool < `agg_goal` — KEEPS cohort == `agg_goal` (matches the sim's vclock-jump, so cohort size stays - identical real↔sim; relaxing real's cohort would desync cadence). Self-terminates via - `_check_early_stop_conditions` at `max_runtime_s`; one warn per stall (bounds the 292 MB log). Byte-identical - when availability tracking is off / on the sim path. `test_fwdllm_sync_scarcity_wait.py` (8). *Note:* §C's - fwdllm_plus `agg_goal` row read 2; the landed yaml is **10** (all clients) — the yaml is source of truth. - - **Stage A1 (trainer per-phase timing):** `_phase`/`_phase_times` added to `fwdllm_trainer` (+ `mqtt_fetch_s`/ - `weights_to_{ram,gpu}_s` wrapped in `_fetch_weights`); `FedSGDTrainer.train_with_data_id` drains those + - `pre_train_s`/`gpu_compute_s`/`post_train_s`/`training_budget_s`/`trainer_phase` into `trainer_round`. Un-skips - the 8 phase rungs. Manifest note updated (the phase fields ARE emitted for the #6 decomposition; the DERIVED - budget-slack/overrun plots stay not_populated — fwdllm's delay is a flat additive sleep). Reversed the Part-6 - "don't emit training_budget_s" note for the decomposition (not for budget-slack). - - **Stage A2 (agg per-round wall decomposition):** `barrier_wait_s` (dispatch→last grad), `drain_tail_s` (last - grad→commit, the K-D20(c) ARTIFACT), `aggregate_fedavg_s`, `eval_s` in `agg_round` — via a dispatch wall stamp - (`_round_dispatch_wall_ts`) + a last-grad stamp (`_last_grad_wall_ts`) + timers around `aggregate()`/ - `eval_model()`. Null-safe (rung SKIPs, no crash). `TestPerRoundWallDecomposition` (4). - - **Stage A3 (advance-rung re-key — SHARED engine):** `_per_round_advances` (feeds K3/K4/K3a/K3b) now keys on - `_progress_axis` via `_per_progress_last_event` — fwdllm advances on `data_id` (was <2 `round`s → SKIP); - async_cifar10 (round-advancing) falls through to `_per_round_last_event` → byte-identical. Full `tests/` + - `examples/async_cifar10/scripts/parity` re-run green. - - **Stage A4 (`wall_disparity` DIAG rung):** new `|real_wall − sim_vclock|` per matched progress unit - (cumulative-from-first-matched so a non-zero-start run still aligns); registered in `run_all_parity`/ - `CHECK_META`/shim; never gates. The recurring #6 sanity metric → drive to ~0. `TestWallDisparity` (5). - - **Stage B (sct-model folds — CONFIG-GATED, flag-off ⇒ byte-identical; emergent calibration = the run):** - **B1** `simModelEvalTime` charges the MEASURED server-eval wall to the vclock after each committed data_id - (consistent with the existing real-gpu-on-vclock model; the largest genuine unmodeled term). **B2** - `simStragglerSpreadS` adds a STABLE per-trainer (crc32) offset in [0, spread) to the modeled delay in SIM - only, widening the barrier's k-th sct to real dispersion. **B3** `simWanTransferS` — documented knob, left 0 - (no localhost ground truth; do NOT enable). All default OFF. `test_fwdllm_sct_model.py` (7) covers mechanism + - flag-off byte-identical. **Whether the folds drive `wall_disparity`→~0 is the sign-off run's job**, not - asserted here (principle #11c). - - **Stage D (checker/telemetry gaps):** **D1** `fwdllm_trainer._fetch_weights` re-emits `task_recv` carrying - `sim_send_ts` (the override had dropped the base trainer's emission → field_coverage INV + K6 had null/null); - `test_fwdllm_trainer_task_recv.py` (3). **D2** `failsafe_ok`/K5 now compares sim wall against the RUN wall - budget for a real-compute sim (auto-detected via the progress axis) and SKIPs when no run budget is available, - instead of the wall-vs-vclock fallback that false-failed every fwdllm run (sim wall ≫ vclock BY construction — - real forward-grad compute); `TestFailsafeRealComputeSim` (3). - - **Stage E (optimization):** the two `_distribute_weights_{sync,async}` `time.sleep(0.1)` "busy wait" pads - (real-transport artifacts, ≥28.7 s/run) gated `if not self.simulated` — real byte-identical, sim skips pure - wall overhead. Full pytest suite re-run proves grad values + cadence unchanged. - -- **K-D22 Availability params respected end-to-end — Phase-1 defaults to syn_0; the pre-flight table shows the - RESOLVED config, not a hardcoded default.** Two consistency bugs in `run_sequential.sh`: (1) the "trace" row - PRINTED `syn_0` whenever no `--avail-trace` was passed, but `patch()` only sets the mode when a trace is - truthy — so with no flag the yaml's OWN `mode:` ran (fwdllm_plus `mobiperf_2st`, fluxtune `mobiperf_3st_50`) - while the table claimed syn_0 (the K-D20 stall, mislabeled). (2) The yamls' source-of-truth `mode:` was a - mobiperf trace, not syn_0. **Fixes:** the availability trace now DEFAULTS to `syn_0` (Phase-1) so `patch()` - always sets trainer `availability.mode` + aggregator `trackTrainerAvail.trace` + `client_notify.trace` - EXPLICITLY on every baseline; the table's trace row + a new per-baseline `avail` column are read BACK from the - patched cfg (what actually launches), so print == run; the 4 source yamls flipped to `mode: syn_0` (switch to a - mobiperf trace via `--avail-trace` only for Phase-2); and a new **feasibility gate BLOCKS** a full-participation - sync barrier under a non-syn_0 trace (sync `agg_goal >= n_trainers` + unavailability = can't assemble → stall; - `--force` to override). Verified: generated launch cfg carries `syn_0` in all three availability fields; the - guard blocks `--avail-trace mobiperf_2st --only fwdllm_plus` at pre-flight (exit 2). -- **K-D23 Phase 2 (speedup leak, root #13) — the 2a "trainer inter-round fetch" hypothesis was REFUTED by - the telemetry; the real skippable wall was two per-round sleeps, gated fwdllm-only.** The pre-plan leak - inventory ranked the trainer's `await_join`+`recv_wrapper` fetch as the DOMINANT leak (#1) and "ungated - `sleep(1)` backoffs" LAST (#4). Localizing from the banked `134801` sim (principle #11a) + a full - async_cifar10 sim-path map INVERTED that ranking. **What the telemetry showed:** the trainer's - `mqtt_fetch_s` splits by `trainer_phase` into ~1s steady (iter `/1`) and ~15–18s at each data_id boundary - (iter `/0`); the ~15s OVERLAPS the aggregator's `agg_round` `eval_s` (~13s server eval) + real GPU — i.e. - it is **barrier-wait for irreducible real work, not an independent trainer sleep.** **2a decision:** do - NOT short-circuit the trainer `recv` — it delivers the real weights the forward-grad pass needs for grad - mode-invariance, and async_cifar10 confirms the sim keeps real MQTT recv (the map: "sim trainer physically - blocks on real MQTT for the fetch — intentional and cheap; do not expect a gate here to port"). The genuine, - additive, fidelity-free per-round wall was: **(1)** the trainer `pause_execution` throttle (`time.sleep(1)` - chained at the tail of EVERY loop iteration — `compose(): ... >> task_put_grad >> pause_exec`; comment - ":494 don't overwhelm mqtt" = a real-transport artifact, principle #8); **(2)** the `_check_availability` - avail-spin (`while UN_AVL: time.sleep(1)`, which in sim would FREEZE the vclock — sim time can't advance - while a trainer blocks — and is redundant because sim availability is enforced agg-side, mirroring - async_cifar10's gated trainer avail wait). Both gated `and not self.simulated`; real byte-identical. - **Aggregator side needed NO change** — verified already sim-correct: `_sync_sim_recv_first_k` uses the 2s - grace floor (not the 30/90s real timeout), the `sleep(0.1)` distribute pads are already gated real-only - (Stage E), `await_join` is kept (cheap, round-1, matches async_cifar10), and the agg's own - `pause_execution` is defined but NOT in its loop. **Blast radius:** fwdllm-only (`fwdllm_trainer.py` - `pause_execution` + example `FedSgdTrainer._check_availability`), so per principle #9 `tests/mode -k fwdllm` - (146 green, +6 in `test_fwdllm_sim_speedup_waits.py`) is the gate, NOT full `tests/` — the original plan - assumed shared-base edits that turned out unnecessary. **Scope note:** Phase 2 alone does NOT make the - REAL-run `sim_rate>1` — at syn_0 the sim wall is dominated by server `eval_s` (~13s/data_id), which is - #6/Phase-4c's job to fold into the vclock; Phase 2's own exit (`sim_rate>1` on SYNTHETIC timings + pytest) - is met. See §H issue #13 + the Phase-2 PROGRESS marker. -- **K-D24 Phase 4 (parity stopping rule + #6 sct model) — ceiling decoupled, eval/straggler folds enabled - for the sign-off run; final calibration is the run's job (not pytest).** Three pieces: **(4a, root S1)** - `sim_wall_ceiling_s` defaulted to `1× max_runtime_s`, which ALWAYS truncated a real-compute sim (wall ≫ - vclock by construction) before its paired vclock/data_id stop — the 2026-07-04 sign-off void. Decoupled to - `max_runtime_s × SIM_WALL_CEILING_FACTOR` (=20, class const, overridable by explicit `sim_wall_ceiling_s`) - so it is a runaway OUTER safety, not the primary stop; fwdllm-only (`_check_early_stop_conditions`), 19 - tests. **(4b)** the matched committed-`data_id` stop already existed (`max_data_id_progress`; the sign-off - run passes `--max-data-id N` as the primary stop so real↔sim compare at identical progress, NOT a - time budget — a wall/vclock budget desyncs wall-bound real from the sim). **(4c, #6)** the two GENUINE - unmodeled `sct` terms (Stage B, landed config-gated in K-D21) turned ON in the 3 sim yamls per the - operator: **B1** `sim_model_eval_time:true` charges the MEASURED server `eval_s` (~13s/data_id, the syn_0 - wall dominator) to the vclock — self-calibrating (uses the real eval wall, no magic constant), delay- - independent, and the term that actually lifts `sim_rate` toward/over 1; **B2** `sim_straggler_spread_s:0.9` - (≈ banked real-compute std 0.26s · √12, since the offset is uniform`[0,spread)`) widens the barrier's - k-th-smallest sct to real's completion dispersion — active only with `--delays on`. The `wall_disparity` - DIAG rung (A4, keyed on the fwdllm `data_id` axis) surfaces `|real_wall − sim_vclock|/data_id` every run; - the **B2 value is calibrated to drive that → ~0 FROM the sign-off run** (emergent, per the doc — do NOT - tune it blind, principle #3/#8). Blast radius: 4a is fwdllm-only → `tests/mode -k fwdllm` 148 green is the - gate. This closes all PRE-RUN dev; the Phase-1 sign-off run is next. - ---- - -## §L Phase-1 sign-off remediation — async grad-path residence & carry-over (Batch 2.5, staged) - -**Status: IMPLEMENTED & pytest-gated (2026-07-03); the one re-run (step 6) is the only remaining step.** -All of §L.4 steps 1–5 + §L.5 landed and are green (`tests/mode` = 297 passed / 7 skipped; parity -sub-package = 115; the new suites: `test_fwdllm_sim_grad_residence.py`, R1/W1 fixtures in -`test_parity_checks.py`, contributor-interval + staleness-accept tests in `test_fwdllm_agg_telemetry.py`). -As-built decisions are logged in **§K-D14/D15**; the code is the source of truth. **Step 6 (the 3-baseline -syn_0 re-run → `parity_checks`) is the operator's next action** — it needs the cluster and cannot run in -pytest. The first real↔sim 3-baseline -syn_0 smoke (§H, 2026-07-03) surfaced three parity defects + one tooling defect. This section states the -**principled correctness model** (what BOTH real and sim must do), root-causes each defect against the -PARITY.md ladder, and gives the **staged, pytest-gated build plan** with exact code sites, new -telemetry/checks/tests, and exit criteria. This is a Batch inserted between Batch 2 and the Phase-1 -convergence sign-off (the mechanics must be right before convergence/parity numbers mean anything). - -### §L.1 The correctness model — what real AND sim must do (async fedbuff forward-grad, fluxtune) - -The execution contract, mode-invariant (this is the "right thing" both sides must obey): - -1. **Concurrency budget C.** The aggregator keeps up to C=`c` trainers in flight. A dispatched trainer is - **BUSY** — held in the selector's `selected_ends` (a *slot*, not the unavailable list) — from dispatch - until its update **commits**. -2. **One-in-flight-per-trainer (residence invariant, PARITY.md §3.resid).** A trainer is **re-pickable - ONLY after its update returns AND is committed.** It is never re-dispatched while a prior update is - outstanding, and never re-picked for the same `(round, data_id, iteration)` (fwdllm's tightening of the - "not twice per round" FL rule). Real satisfies this by construction (the channel holds a dispatched end - out of `VAL_CH_STATE_SEND` until aggregated → real overlap 0%). Sim must model it explicitly. -3. **Buffered accumulation (fedbuff).** Arriving grads accumulate in a buffer. When `agg_goal`=K grads are - available, the aggregator runs one fedbuff step (staleness-weighted server-LR SGD) + the variance gate. - The **surplus** in-flight trainers keep computing; a completed-but-not-yet-committed grad is **retained - in the buffer and applied at a later step (carry-over)** — it is **never silently dropped**. -4. **Staleness is handled, not ignored.** A grad trained on `model_version` V that lands after the model - advanced to V′>V is **stale by V′−V** and must be either (a) accepted with fedbuff staleness weighting, - or (b) explicitly rejected AND recorded (speed/utility props still logged — PARITY.md "stale-property - recording" lesson). "Silently drop" is neither and is wrong. -5. **Progress is `data_id`** (committed variance passes), not raw update count (§B, PARITY.md §F.1). - -**Consequence to hold as an invariant/telemetry check:** over a run, **total forward passes ≈ total -committed grads + (grads still in flight at stop) + (grads explicitly stale-rejected)**. A large -unexplained gap = wasted recompute = a residence/carry violation. Real 109 fwd / 63 commit and sim 228 / -66 both show a gap; **sim's is far larger AND its participation is lock-step even** — the tell. - -### §L.2 Root-caused defects (lowest-rung first) - -- **D-a — Residence hold is nullified on the grad path (the root).** `_release_sim_slots_at_agg_goal` - (`fwdllm_aggregator.py`) runs `self._sim_buffer.clear(); self._sim_inflight_expected.clear()` **before** - `self._sim_hold_busy_slots(channel)`, which holds exactly `pending_in_buffer (∪ _sim_inflight_expected)` - — both now empty → `held={}` → it **releases every trainer**. Plus `sim_inflight_residence=False` for - fluxtune, so even the buffered set wouldn't be augmented by the in-flight set. Net: all C trainers freed - every agg-goal cycle → re-dispatched → recompute. **Evidence:** even 22–23 rounds/trainer (sim) vs uneven - 7–14 (real); 228 vs 109 forward passes; identical per-pass GPU time. **Ladder:** this is the fwdllm async - analog of felix's overlapping-re-dispatch root (PARITY.md §3.resid, "measure overlap from intervals not - counters"). -- **D-b — K-D6 buffer drop discards arrived grads (compounds D-a).** `_sim_buffer.clear()` at the boundary - drops the ~7 arrived-but-uncommitted grads (c=10, K=3). Reverses to **commit-then-carry** (K-D12). For - the sync baselines c≈K so there's no surplus — the drop stays correct there. -- **D-c — fluxtune `staleness_policy` unset → defaults `none`.** The yaml never sets it (§H config dump). - For an async fedbuff baseline the surplus grads that carry over are stale-by-construction; `none` means - no principled handling. **RESOLVED (D5/K-D13): fedbuff staleness-weighted accept**, set identically - real+sim. NOT a parity lever to tune — a *definition* fixed once (may revisit the policy later). -- **D-d — launcher has no fail-fast (tooling).** fwdllm_plus crashed in `internal_init` at ~5s but the run - burned 86s: `aggregator_spawner.wait_until_ready` is a fixed 5s "process alive" heuristic (misses a - crash landing at ~5s), so trainers spawned anyway; then `runner.py` `wait_all(timeout_per_trainer=30.0)` - waits 30s for trainers that will never get an EOT. Waste the operator could spend fixing the error. - -### §L.3 What "correct" telemetry/checks look like — mirror the async_cifar10 lessons - -The PARITY.md discipline (§1 ladder, "growth rule: every root leaves behind the finest check that would -have localized it"): D-a would have been caught instantly by a **residence/overlap invariant** rung. Add -these, reusing the shared engine (`async_cifar10/scripts/parity/checks.py` + STAGE map, §I.2): - -| New/validate | Rung | Role/Tier | What it asserts | Telemetry it needs | -|---|---|---|---|---| -| **NEW** | **R1 in-flight overlap** | MECHANISM/**INV** | per-trainer dispatch→commit intervals do NOT overlap (one-in-flight); ≈0% both modes (felix metric, PARITY.md L295-298) | per-(trainer,cycle) `sim_send_ts`/dispatch ts + commit `vclock`/wall ts | -| **NEW** | **W1 compute conservation** | CONTROL/DIAG | `forward_passes ≈ commits + in_flight_at_stop + stale_rejected`, and the **real↔sim ratio ≈1**; localizes wasted recompute | count `trainer_round` events vs committed grads per mode (already emitted) | -| validate | **S2 participation** | EMERGENT/DIST | per-trainer chosen-count matches; the even-vs-uneven split is exactly this — must FAIL pre-fix, PASS post-fix (fluxtune only) | `selection_train` (exists) | -| validate | **V1 iters-per-data_id** | MECHANISM/DIST | data_id-2 80-vs-8 must flag; confirms V1 (Batch 2) fires and is **downstream** of R1 (dep chain proves it) | `cycle_data_id` (K-D9, exists) | -| validate | **U3 staleness** | MECHANISM/DIST | fedbuff staleness dist matched + non-trivial once D-c fixed | `staleness` in agg_round (exists) | - -**Sanity gate (validate_real FIRST — HARD GATE, PARITY.md principle #6 / "verify real is correct before -tuning sim").** Real also shows fwd(109) > commit(63). Before making sim match real, **confirm real is -admissible:** (1) real R1 overlap is truly ~0% (channel residence), (2) the real fwd−commit gap is -explained by end-of-run in-flight + any stale-rejections, NOT by real *also* dropping carried grads. **If -real is itself wrong (overlap > 0, or real drops carried grads), FIX REAL FIRST** — make real correct, -re-bank it as the reference, and only then bring sim to parity (D-b becomes a two-sided fix, real side -first). Never tune sim toward a real that has not passed its own admissibility. Add a `validate_real` -assertion for R1==0% and the W1 decomposition; it **blocks** the §L.4 step-4 sim mechanism change. - -### §L.4 Staged build (pytest-gated; flag-off ⇒ byte-identical; async-only, sync untouched) - -Ordering follows the ladder: instrument → establish real-correctness → fix the mechanism → prove with -tests → re-run. Each step gated by pytest before the one run. **Steps 1–5 DONE & green (2026-07-03); -step 6 is the operator re-run.** As-built notes in K-D14/D15. - -1. **✅ Telemetry (enables R1/W1).** Emit per-(trainer,cycle) dispatch ts + commit ts so overlap intervals - are reconstructable (extend the agg_round `extra` and/or a `SIM_GRAD_RECV`-adjacent field; - `sim_send_ts` + commit `vclock` already exist internally — surface them). Add nothing to the real hot - path beyond a timestamp. *Exit:* a driven emission test asserts the fields land (mirror - `test_fwdllm_agg_telemetry.py`). -2. **✅ Checks R1 + W1 in the shared engine** (`checks.py` `run_all_parity` + STAGE map; deps: R1 under - Stage 3 residence, W1 CONTROL feeding V1/K2). *Exit:* known-PASS + known-broken synthetic fixtures - (an overlapping-interval fixture FAILS R1; a 2×-recompute fixture FAILS W1) in `test_parity_checks.py`. -3. **✅ validate_real sanity** (R1==0% + W1 decomposition on the real dir) — run against the **already-banked - 2026-07-03 real run dirs** (checker-side, no cluster run; PARITY.md "checker fixes validate instantly"). - Decides whether D-b is one-sided or two-sided. -4. **✅ Mechanism fix (D-a + D-b), async grad path only.** In `_release_sim_slots_at_agg_goal`: (i) **hold - before clear** — compute the held set from the pre-clear buffer/in-flight, release only the *committed* - subset, keep the surplus buffered (commit-then-carry); (ii) turn on residence for fluxtune - (`sim_inflight_residence=True` in `fluxtune_n10_smoke_sim.yaml`, and confirm `_sim_hold_busy_slots` holds - `pending_in_buffer ∪ _sim_inflight_expected`). Reference implementations: felix `_sim_hold_busy_slots` - (`asyncfl/top_aggregator.py:1453`, §3.resid) and oort `simInflightCarryover` (§4.9). **Scope-guard - (principle #8):** the change is in `fwdllm_aggregator.py` (fwdllm blast radius) + a fluxtune yaml; do - NOT touch the shared `_sim_recv_min`/`asyncfl` weight path (async_cifar10 is green). Sync path - (`_sync_sim_recv_first_k`, K-D5/K-D11) unchanged — K-D6 drop stays correct there. *Exit:* a - `test_fwdllm_sim_grad_residence.py` (mirror `test_async_inflight_residence.py`): drive c>K synthetic - grads, assert surplus carried (not dropped), busy trainers not re-selected, R1==0%; + flag-off - byte-identical. -5. **✅ Staleness definition (D-c) — D5 LOCKED: fedbuff-accept (K-D13/K-D15).** Wire fluxtune to fedbuff - staleness-weighted accept: set `staleness_policy` in all fluxtune yamls (real+sim identical) so a - carried stale grad is **accepted and down-weighted by `V′−V`**, and confirm the fedbuff optimizer's - weighting path is active + emits `staleness` on the commit event (for U3). *Exit:* a unit test that a - carried stale grad is accepted+weighted (never silently dropped) and that `staleness=V′−V` is emitted. -6. **✅ First re-run (2026-07-03).** Confirmed W1→PASS (sim under-computes, not over), R1==0% both modes, - fwdllm_plus no longer crashes. **Surfaced D-e (concurrency starvation): the commit-then-carry fix freed - the compute slot at COMMIT instead of at RETURN, collapsing sim in-flight concurrency to ~1.5 vs real - ~5.8 → fluxtune sim wall 2.4× real, U3 low.** -7. **✅ D-e fixed — Option A two-lifetime split (K-D16).** Free `selected_ends` (compute slot) on RETURN, - hold `all_selected` (re-pick guard) to COMMIT, carry + rollback survive; triplet populated at dispatch. - Landed with the clock-family `data_id` re-key (#2) and field-coverage aliases (#3); pytest-green - (`tests/mode` 421 passed / 7 skipped incl. parity). Current ground state in **§H**. -8. **▶ IN FLIGHT (operator launched the post-fix re-run, 2026-07-03).** One `--mode both` 3-baseline run at syn_0 with the yaml - default `max_data_id_progress=10` (do NOT cap at 3 — clears the #4 truncation) → `run_parity.py`. Confirm: - fluxtune concurrency recovers toward real ~5.8, wall gap closes, U3 non-trivial, clock-family rungs now - read on the `data_id` axis, V1/V2 within tolerance on the longer run. Then the Phase-1 convergence - sign-off (§I.6). - -### §L.5 ✅ Launcher fail-fast (D-d) — parallel tooling task (no parity coupling) — DONE - -Independent of the aggregation work; do in the harness/launcher. Two changes in `flame/launch/`: -- **Detect early aggregator death before committing to trainers.** `aggregator_spawner.wait_until_ready` - (`:135`) should treat a process exit during the window as `False` immediately (it already checks - `is_running()` each second — return `False` on death instead of the 5s "alive ⇒ ready" heuristic; better, - watch the agg log for a readiness marker / a traceback). If not ready → abort the run, do NOT - `spawn_all` trainers. -- **On aggregator non-zero exit, terminate trainers immediately.** `runner.py:324-327`: when - `agg_rc not in (0, None)`, skip `wait_all(timeout_per_trainer=30.0)` and call the trainer spawner's - terminate path now (the run has failed; the 30s EOT grace is only meaningful on a clean finish). -*Exit:* a crashed-aggregator run returns in ~seconds, not ~90s; the harness health line already reads -CRASH (done this session). Optional: surface the aggregator's traceback tail in the runner output so the -operator sees the cause without opening the log. - -### §L.6 Locked decisions - -- **D5 — fluxtune staleness treatment: RESOLVED → (a) fedbuff staleness-weighted accept (K-D13).** A - carried grad trained on `model_version` V that commits at V′ is **accepted and down-weighted by `V′−V`** - (fedbuff default, faithful to an async buffered aggregator). Rejected: (b) `round_data_id` reject+record - (fwdllm_plus's sync policy — discards work an async baseline is designed to consume); (c) unset→`none` - (no staleness signal for U3). **Provisional lock** — keep fedbuff-accept for the Phase-1 async parity - pass; MAY revisit later. It is a baseline *definition*, not a parity lever (PARITY.md §F.4 / principle - #3): **set identically real+sim, never tuned to close a gap.** Obligation: confirm the fedbuff optimizer - weighting path is wired and `staleness = V′−V` emits on the commit event so U3 can score it. -- **Verify-real-first is a HARD GATE.** Real correctness is established BEFORE any sim mechanism change - (§L.3 sanity gate blocks §L.4 step 4). If real is itself wrong (R1 overlap > 0, or real drops carried - grads), we **fix real first**, re-bank it as the reference, then bring sim to parity — never tune sim to - an inadmissible real (PARITY.md "verify real is correct first — a gap has two fix directions"). +## §K Deviation log — one line per decision (anchor + rationale; referenced from §B.1) +*Folded/landed entries are one line. Expand only an entry still open/contested.* + +- **K-D1** — `time_mode` default `"real"` (not cifar's `"simulated"`): fwdllm's whole corpus is `real`; a + `simulated` default risks half-activating an unbuilt path. Sim variants set it explicitly. +- **K-D2** — additive `sct = send + gpu + D` (not `max`): fwdllm's real mode sleeps D on top of GPU time. +- **K-D3** — per-eval sct collapses to the train sct (D4): eval lives on the aggregator; forward-grad "train" is a + forward pass (no 20× factor); the trainer eval message is a utility report, not a clocked commit. +- **K-D4** — purpose-built `_sim_recv_min_grad` (not `_sim_recv_min` verbatim): the latter's per-commit release + + withheld paths key on WEIGHTS semantics; reuse the primitives, fork the orchestration. +- **K-D5** — slot release + buffer clear on the AGG-GOAL boundary (not per-commit): a `data_id` spans many cycles + with rollbacks; per-commit release would strand a re-contributing trainer (principle #4). +- **K-D6** — boundary buffer-drop DROPPED stranded grads. *Superseded by K-D12* (benign only for sync). +- **K-D7** — U6 sync-barrier visibility-lag telemetry: compute+stash now, emit later (kept Batch 1 a clock port). +- **K-D8** — sim smoke launchers keep D=0 for mechanics-only comparability; D>0 belongs to convergence/parity runs, + enabled in BOTH real and sim together (else mismatched enable is a false divergence). +- **K-D9** — cadence telemetry is a **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/ + `grad_pool_size`/`cached_v_size`): the post-mutation `data_id` advances before emit → off-by-one; snapshot makes + V1 exact for natural-pass and force-commit paths. Existing post-mutation fields unchanged (byte-identical real). +- **K-D10** — DK3 / G1 emit DEFERRED (checker reads-if-present, logged SKIP otherwise): DynamicKC disabled for all + three baselines; G1 needs a trainer-side per-update emit. Wire when a DynamicKC/G1 baseline is added. +- **K-D11** — `ends_not_selected_yet` "commit-1-per-pass" clamp gated real-only: it's a real-transport draining + discipline (queue re-collection); the sim barrier is single-pass (principle #8). +- **K-D12** — fluxtune async: commit-then-CARRY the surplus + hold residence (reverses K-D6 for async); drop stays + correct for sync (c≈agg_goal). +- **K-D13/K-D15** — fluxtune `staleness_policy = fedbuff` staleness-weighted accept (was silently `none` — never + wired from config); set identically real+sim (a definition, not a lever). Provisional lock. +- **K-D14** — R1/W1 sourced from an ECHOED per-contribution interval (not the agg's per-end dispatch stamp, which + is overwritten on re-dispatch — exactly when residence is broken). +- **K-D16** — D-e resolution Option A (two-lifetime split). *Slot-on-return half superseded by K-D17b; the + hold-guard-to-COMMIT half survives.* +- **K-D17** — two bugs that deadlocked the K-D16 re-run: (A) drain gated on channel RECV not `_sim_buffer`; + (B) triplet stamped at DISPATCH froze the pool pre-commit. Fixed: drain keys on the buffer; triplet on RETURN. +- **K-D17b** — hold the compute slot to COMMIT (felix-aligned): `len(selected_ends)` now = virtual-time in-flight + (fixed in_flight 2.7→9.5). Reverts K-D16's slot-on-return. +- **K-D18** — overnight 10-`data_id` grounding runs; `--max-data-id` default raised to 9999 (was 10, silently + capping 1h runs); cleared #4 truncation for fwdllm. +- **K-D19** — R1 regression fix: defer the re-pick guard release from physical RETURN to COMMIT + (`_release_end_on_return`, fwdllm-only) + a `[SIM_R1_DISPATCH]` tripwire + RETURN-path pytest. **⚠ REFUTED by + the 2026-07-04 sign-off smoke** — R1 still 60.4% (§H dead-ends, #1c reopened); the guard leaks despite the defer. + Root-cause from the tripwire lines next. +- **K-D20** — real per-round wall decomposition (telemetry-only): ~8.7 s/round GENUINE (eval_s 3.34 + straggler + 3.06 + compute) vs ~4.7 s/round ARTIFACT (drain tail 2.48 + `sleep(0.1)` pads) → fold ONLY eval_s + straggler + spread; artifacts stay off the vclock and become optimization targets (#11). Root-caused #7 (real-mode liveness). +- **K-D21** — pre-run instrumentation A–E landed & pytest-green (see §G). +- **K-D22** — availability params respected end-to-end; Phase-1 syn_0 default; print==run (see §G). +- **K-D23** — Phase-2 speedup skips; the "trainer fetch is the dominant leak" hypothesis was REFUTED by telemetry + (it's barrier-wait over irreducible agg eval+GPU); the real skippable wall was two per-round sleeps (see §G). +- **K-D24** — Phase-4 ceiling decouple (×20) + B1/B2 folds ON for the sign-off run (see §G). Sign-off confirmed S1 + fixed. *(#6-via-folds superseded by K-D25 — the residual gap was a checker anchor, not a missing fold.)* +- **K-D25** — #6 is a CHECKER-ANCHOR bug, not a sim under-charge (root-caused from the sign-off #6 telemetry). + The clock-rate rungs compared sim-Δvclock (genuine algorithmic time) against real-Δts (FULL wall = genuine + + a ~constant ~7.7s/round localhost transport artifact real trainers pay via pipelined inter-round fetch; the + agg `barrier_wait_s` reads ~0 in real for the same pipelining reason). Proof: on a non-eval round real burns + 10.9s wall vs the sim's 3.5s genuine, all artifact; on eval rounds real genuine (barrier+eval 15.8) ≈ sim + vclock 15.86 exactly. Fix (fwdllm agg emit + shared checker, async byte-identical): emit `intrinsic_span_s` = + max(committed intrinsic duration) + eval (mirrors the sim vclock; **fedavg excluded**, else overshoot); the 5 + clock-rate rungs + `wall_disparity` anchor real on its cumulative intrinsic clock. **Root B (bundled):** + `post_train_s` stamped after the delay (real stops bundling the slept delay); B2 straggler moved from + `_delay_s` into the sct so `training_budget_s` is a mode-invariant input. `TestIntrinsicSpanAnchor` + emit + + straggler-separation + post_train tests; full `tests/` 878-green. Residual (3 marginal rungs + B2 calibration) + is emergent at the longer sign-off run. #11 real-side waste deferred (zero parity impact; needs a real run). +- **K-D26** — #1c R1 root-caused to a physical-wall vs vclock desync in the SHARED `async_oort` re-pick guard + (`all_selected`), exposed by fluxtune's slow sim (sim_rate 0.26, wall ≫ vclock) and masked in async_cifar10 + (wall ≈ vclock). Evidence (banked sign-off log, principle #11a): trainer 370 dispatched at vclock ~0, its grad + commits at vclock ~7, but the sim burns ~99s wall first → at 91s wall async_oort's 90s `SEND_TIMEOUT_WAIT_S` + (keyed on `time.time()`) evicts it from `all_selected` → re-dispatch → R1. Two deletion paths: **(a)** the 90s + wall-timeout (round-1 trigger, 10 evictions); **(b)** the aggregator marks carried grads `KEY_END_STATE=NONE` + to keep their `selected_ends` slot (`_sim_hold_busy_slots`), and async_oort's recv-handler reads `NONE` as + "left/rejoined" → deletes from `all_selected` (sustained, 113 violations). Corrects K-D19 (fixed the RETURN + path; dismissed the NONE hypothesis by checking the selection FILTER not the DELETION) — K-D17b was right. + **Fix (a) LANDED** (operator: "both — direct fix now + #13 tracked"): `_abandon_clock_now()` runs the timeout + on the vclock in sim — `select()` stashes `channel_props["vclock_now"]`; the dispatch STAMP and the timeout + CHECK both use it (90 = 90 *virtual* s); None in real → `time.time()` → async_cifar10 byte-identical. + `test_sim_vclock_keeps_virtually_recent_end` + reclaim regressions green. **Fix (b)** = the sustained NONE path, + same slow-sim root → cured by **#13** (make the fluxtune sim fast so physical receipt ≈ virtual commit, as in + async_cifar10) or a follow-up aggregator-side re-assertion; the emergent re-run measures how much (a) recovers. diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index 1973b50fa..0afccca2c 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -540,7 +540,10 @@ def _emulate_training_delay(self): f"Delayed eval time for trainer " f"{self.trainer_id} by {eval_delay}s. Sleeping for {_delay_s}s." ) - _delay_s += self._sim_straggler_offset_s() + # Returns the BASE modeled delay (identical real↔sim). The B2 straggler + # spread is NOT added here -- it belongs in the sct (sim_round_duration_s, + # below) so the emitted training_budget_s stays a mode-invariant INPUT + # (T2), while the sct still carries the per-trainer dispersion (#6/Root B). return _delay_s return 0.0 @@ -584,12 +587,18 @@ def train_with_data_id(self): _pre_train_s = _round_start_ts - _phase_entry self._perform_training() _real_gpu_time_s = time.time() - _round_start_ts - _phase_post_start = time.time() # emulate delays in training (due to compute resource and/or # dataset size and/or network latency) _delay_s = self._emulate_training_delay() + # post_train phase starts AFTER the modeled delay (Root B / #6): real + # SLEEPS _delay_s above (the modeled-latency term, compared via + # training_budget_s), so stamping here EXCLUDES it -> post_train_s is pure + # post-processing, mode-comparable (~0 both modes; sim never slept). + # Without this, phase_post_train saw real ~1.14 vs sim 0.0. + _phase_post_start = time.time() + # Sim-mode stamps (Batch 1): the modeled round duration is ADDITIVE # (real_gpu + D), matching real mode's sleep-D-on-top-of-GPU semantics # -- NOT cifar10's max(gpu, D) "sleep to fill a budget" model. The sct @@ -606,7 +615,14 @@ def train_with_data_id(self): float(getattr(_hp, "sim_wan_transfer_s", 0.0) or 0.0) if (self.simulated and _hp is not None) else 0.0 ) - self._sim_round_duration_s = _real_gpu_time_s + _delay_s + _wan_s + # B2 straggler spread lives HERE (in the sct / modeled round duration), + # NOT in _delay_s, so the emitted training_budget_s (the modeled-delay + # INPUT) stays identical real↔sim (T2) while the sct still carries the + # per-trainer completion dispersion the sync barrier needs (#6/Root B). + # sim-only (the offset is 0 in real). + self._sim_round_duration_s = ( + _real_gpu_time_s + _delay_s + self._sim_straggler_offset_s() + _wan_s + ) if self.simulated: _leg = self.sim_completion_leg_s _base = self._sim_send_ts if self._sim_send_ts is not None else time.time() @@ -625,16 +641,18 @@ def train_with_data_id(self): _stat_utility = float(self._stat_utility) except (TypeError, ValueError): _stat_utility = None - # Post-compute overhead (delay emulation + telemetry build); the last - # trainer-side phase term for the #6 wall decomposition (Stage A1). + # Post-compute overhead (telemetry build); the last trainer-side phase + # term for the #6 wall decomposition (Stage A1). The modeled delay is + # EXCLUDED via the _phase_post_start stamp position (after the delay) -- + # see Root B note above. _post_train_s = time.time() - _phase_post_start ev, fields = build_trainer_round( round_num=int(self._round), real_gpu_time_s=_real_gpu_time_s, - # real GPU compute + the emulated delay (0 if disabled) + any - # sim sct-model folds (B2 straggler spread in _delay_s, B3 WAN) -- - # NOT a budget-vs-actual quantity (fwdllm has no sleep-to-fill- - # budget model); the total modeled wall this round took. + # real GPU compute + the emulated base delay (0 if disabled) + the + # sim sct-model folds (B2 straggler spread, B3 WAN) -- NOT a + # budget-vs-actual quantity (fwdllm has no sleep-to-fill-budget + # model); the total modeled wall this round took. sim_round_duration_s=self._sim_round_duration_s, avail_state=self.avl_state.value, dataset_size=self.dataset_size, diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 81d188275..0f3b14c6b 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -1653,6 +1653,30 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): f"data_id={self.data_id}" ) self._last_vclock_log_wall_ts = time.time() + + # Intrinsic algorithmic span of this cycle (#6 anchor): the GENUINE + # per-cycle work the sim charges to the vclock -- the barrier (slowest + # committed trainer's intrinsic compute+delay) plus the server eval + # (commit cycles only). Must mirror the sim vclock's COMPOSITION exactly + # (barrier sct + eval fold): the FedAvg merge is deliberately EXCLUDED + # because the sim does NOT charge it to the vclock -- including it made + # real intrinsic overshoot sim by ~fedavg×cycles (wall_disparity 5.5->1.6 + # once dropped). If fedavg is ever folded into the sim vclock, add it + # here too. Emitted in BOTH modes so the clock-rate rungs (K2/K3/K3b/K8/ + # U2 + wall_disparity) anchor REAL on its intrinsic time instead of raw + # wall Δts: real's wall bundles a ~constant inter-round transport ARTIFACT + # (mqtt re-fetch/redistribute/drain-tail/sleeps) the sim (correctly, + # principle #1) omits, which spuriously fails #6. The barrier uses the + # trainer INTRINSIC duration (PROP_CLIENT_TASK_TRAIN_DURATION = WALL_SEND- + # WALL_RECV, _cycle_speed_s) not the agg-side barrier_wait_s, which reads + # ~0 in real because real trainers pipeline (grads pre-queued at dispatch). + # max() = the sync barrier (MAX-of-K sct, principle #12). + _barrier_span_s = max(_cycle_speed_s) if _cycle_speed_s else None + _intrinsic_span_s = ( + _barrier_span_s + (_eval_s or 0.0) + if _barrier_span_s is not None + else None + ) try: ev, fields = build_agg_round( round_num=self._round, @@ -1692,6 +1716,11 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): "drain_tail_s": _drain_tail_s, "aggregate_fedavg_s": _aggregate_fedavg_s, "eval_s": _eval_s, + # #6 anchor: real's GENUINE per-cycle algorithmic time + # (barrier+fedavg+eval), the like-for-like counterpart to + # the sim's Δvclock. Lets the clock-rate rungs exclude + # real's inter-round transport artifact. See computation above. + "intrinsic_span_s": _intrinsic_span_s, # Speedup metric (§H #13): wall in both modes; sim_rate # = vclock/wall (sim only, None in real). sim_rate < 1 # means the sim is a SLOWDOWN (broken, principle #13). diff --git a/lib/python/flame/selector/async_oort.py b/lib/python/flame/selector/async_oort.py index c14bbe030..c18cfac67 100644 --- a/lib/python/flame/selector/async_oort.py +++ b/lib/python/flame/selector/async_oort.py @@ -63,6 +63,11 @@ def __init__(self, **kwargs): """Initailize instance.""" super().__init__(**kwargs) + # #1c: the abandon-timeout clock — set per-select() from + # channel_props["vclock_now"] (sim) or left None (real -> wall). See + # _abandon_clock_now. + self._sim_now_s = None + ml_framework_in_use = get_ml_framework_in_use() if ml_framework_in_use != MLFramework.PYTORCH: raise NotImplementedError( @@ -323,6 +328,18 @@ def select( if self.requester not in self.selected_ends: self.selected_ends[self.requester] = set() + # #1c: the in-flight abandon-timeout (SEND_TIMEOUT_WAIT_S) must run on the + # SAME clock the trainer commits on. In sim that is the virtual clock + # (vclock_now, plumbed via channel_props), NOT physical wall: a slow sim + # (fwdllm fluxtune sim_rate 0.26) runs ~90 wall-s while only ~7 vclock-s + # elapse, so a wall-keyed 90s timeout evicts a still-outstanding trainer + # from all_selected -> re-dispatch -> R1 residence violation. Stash the sim + # clock so the dispatch STAMP (process_chosen_candidate_dict/_handle_recv_ + # state) and the CHECK (_handle_send_state) use it consistently. None in + # real -> falls back to time.time() -> byte-identical (async_cifar10's fast + # sim has vclock≈wall, so it is unaffected either way). + self._sim_now_s = channel_props.get("vclock_now") + # TODO: (DG) Is explicit round tracking required here? round = # channel_props["round"] if "round" in channel_props else 0 # logger.debug(f"let's select {num_of_ends} ends for new round @@ -1392,6 +1409,16 @@ def _select_candidates_prioritiseUnavail( return candidates, exploit_end_ids + def _abandon_clock_now(self) -> float: + """Clock for the in-flight abandon-timeout (SEND_TIMEOUT_WAIT_S): the + VIRTUAL clock in sim (vclock_now stashed per-select), physical wall in + real. Keeping the STAMP (all_selected[end]) and the CHECK on the same + clock makes the 90s timeout mean 90 *virtual* seconds in sim, so a slow + sim no longer evicts a still-outstanding trainer from the re-pick guard + (#1c). Real / async_cifar10 fast-sim: unchanged (None -> wall / vclock≈wall).""" + sim_now = getattr(self, "_sim_now_s", None) + return sim_now if sim_now is not None else time.time() + def _handle_send_state( self, ends: dict[str, End], @@ -1425,7 +1452,7 @@ def _handle_send_state( # deadlock this caused). curr_all_selected_ends = list(self.all_selected.keys()) for end in curr_all_selected_ends: - current_time_s = time.time() + current_time_s = self._abandon_clock_now() # vclock in sim, wall in real (#1c) if end in self.all_selected.keys(): # Check again to avoid possible case of race condition # when all_selected has been updated from another @@ -2010,7 +2037,7 @@ def _handle_recv_state( for selected_end in selected_ends: # Add to all_selected. {key: end, val: TS epoch (s)} - self.all_selected[selected_end] = time.time() + self.all_selected[selected_end] = self._abandon_clock_now() # #1c logging.debug( f"self.all_selected {self.all_selected} after combining with " f"selected_ends {selected_ends}" @@ -2084,7 +2111,7 @@ def process_chosen_candidate_dict( for candidate_end in candidates: # Add to all_selected. {key: end, val: TS epoch (s)} - self.all_selected[candidate_end] = time.time() + self.all_selected[candidate_end] = self._abandon_clock_now() # #1c logging.debug( f"self.all_selected {self.all_selected} after combining" f" with candidates {candidates}" diff --git a/lib/python/tests/mode/test_async_sim_ordering.py b/lib/python/tests/mode/test_async_sim_ordering.py index 8d0d33654..edbc99d62 100644 --- a/lib/python/tests/mode/test_async_sim_ordering.py +++ b/lib/python/tests/mode/test_async_sim_ordering.py @@ -427,6 +427,26 @@ def test_fresh_end_not_reclaimed_still_short_circuits(self): assert result == {} assert not pacer_called + + def test_sim_vclock_keeps_virtually_recent_end_despite_old_wall(self): + """#1c: in sim the abandon-timeout runs on the VCLOCK, not physical wall. + A trainer stamped at a small vclock (recent in VIRTUAL time) is NOT + reclaimed even though a slow sim has burned >90 physical wall-seconds -- + which was evicting still-outstanding trainers from all_selected and + causing R1 re-dispatch. Contrast test_stale_end_* (wall) which reclaims.""" + sel = self._stub_selector(stale=True) # wall-old stamp from the helper... + # ...but drive the timeout on the VIRTUAL clock instead: stamp = vclock 5 + # at dispatch, sim-now = vclock 10 -> delta 5 < SEND_TIMEOUT_WAIT_S (90). + sel.all_selected = {self.STALE_END: 5.0} + sel._sim_now_s = 10.0 # would come from channel_props["vclock_now"] + pacer_called = [] + sel.pacer = lambda: pacer_called.append(True) + + result = self._call(sel, concurrency=1) + + # Guard held: still in all_selected, concurrency saturated -> short-circuit. + assert self.STALE_END in sel.all_selected + assert result == {} and not pacer_called assert self.STALE_END in sel.all_selected assert self.STALE_END in sel.selected_ends["agg"] diff --git a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py index df7519b34..dafd7965a 100644 --- a/lib/python/tests/mode/test_fwdllm_agg_telemetry.py +++ b/lib/python/tests/mode/test_fwdllm_agg_telemetry.py @@ -233,6 +233,33 @@ def test_speedup_fields_emitted(self, tmp_path): finally: telemetry.shutdown() + def test_intrinsic_span_is_barrier_plus_eval_excludes_fedavg(self, tmp_path): + """#6 anchor: intrinsic_span_s = the barrier (MAX committed-cohort + duration) + eval_s (commit only), EXCLUDING the FedAvg merge -- it must + mirror the sim vclock's composition (barrier sct + eval fold) so the + clock-rate rungs anchor REAL like-for-like. A variance-FAIL cycle runs no + eval, so intrinsic collapses to exactly the barrier -- which also proves + max() (not min/sum) and that fedavg is not folded in.""" + telemetry.configure(role="aggregator", run_dir=str(tmp_path)) + try: + agg = _FakeAggregator(contributors=["t1", "t2"], + var_good_enough=False) + channel = _FakeChannel(durations={"t1": timedelta(seconds=5), + "t2": timedelta(seconds=3)}) + + agg._process_aggregation_goal_met(tag="aggregate", channel=channel) + + import json + events = [json.loads(l) for l in + (tmp_path / "aggregator.jsonl").read_text().splitlines()] + r = [e for e in events if e["event"] == "agg_round"][0] + # barrier = max(5, 3) = 5; no eval on the fail path; fedavg (stubbed + # ~0) is excluded regardless -> intrinsic == the barrier. + assert r["intrinsic_span_s"] is not None + assert abs(r["intrinsic_span_s"] - 5.0) < 0.5, r + finally: + telemetry.shutdown() + def test_wall_elapsed_emitted_in_real_mode_sim_rate_none(self, tmp_path): telemetry.configure(role="aggregator", run_dir=str(tmp_path)) try: diff --git a/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py index 8cac23e21..85a41d093 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py @@ -73,6 +73,9 @@ class _FakeFedSgd: + telemetry-emit logic under test runs for real.""" train_with_data_id = FedSgdTrainer.train_with_data_id + # train_with_data_id now folds the B2 straggler into the sct (#6/Root B); + # no config -> spread 0 -> offset 0 -> phase/duration values preserved. + _sim_straggler_offset_s = FedSgdTrainer._sim_straggler_offset_s def __init__(self, phase_times=None, delay_s=1.5): self._round = 7 @@ -129,6 +132,8 @@ def test_phase_fields_present(self, tmp_path): assert ev["weights_to_gpu_s"] == 0.1 # training_budget_s is the modeled additive delay assert ev["training_budget_s"] == 1.5 + # post_train excludes the delay (stamped after it) -> pure post-proc. + assert ev["post_train_s"] >= 0.0 and ev["post_train_s"] < 0.5 # trainer_phase encodes round/data_id/iteration identity assert ev["trainer_phase"] == "7/3/2" # pre/post are real non-negative wall slivers @@ -137,6 +142,34 @@ def test_phase_fields_present(self, tmp_path): finally: telemetry.shutdown() + def test_straggler_in_sct_not_in_training_budget(self, tmp_path): + """#6/Root B: the B2 straggler spread is folded into the sct + (sim_round_duration_s) but NOT into the emitted training_budget_s -- so + training_budget stays a mode-invariant INPUT (T2 passes) while the sync + barrier still gets its per-trainer dispersion.""" + from types import SimpleNamespace + telemetry.configure(role="trainer", run_dir=str(tmp_path)) + try: + t = _FakeFedSgd(delay_s=1.5) + t.simulated = True + t._sim_send_ts = 100.0 + t.config = SimpleNamespace( + hyperparameters=SimpleNamespace(sim_straggler_spread_s=0.9)) + offset = t._sim_straggler_offset_s() + assert offset > 0.0, "t1 should have a non-zero stable offset" + t.train_with_data_id() + + ev = [json.loads(l) for l in + (tmp_path / "trainer.jsonl").read_text().splitlines() + if json.loads(l)["event"] == "trainer_round"][0] + # training_budget = the BASE delay; straggler EXCLUDED. + assert ev["training_budget_s"] == 1.5 + # sct carries the straggler: duration - budget == gpu(~0) + offset. + assert (ev["sim_round_duration_s"] - ev["training_budget_s"]) == \ + pytest.approx(offset, abs=0.05) + finally: + telemetry.shutdown() + def test_aborted_round_emits_nothing(self, tmp_path): telemetry.configure(role="trainer", run_dir=str(tmp_path)) try: diff --git a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py index 1c16fe7b0..9c674c5c6 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py @@ -133,6 +133,9 @@ class _StampTrainer: heavy compute so only the sim-stamp arithmetic is exercised.""" train_with_data_id = FedSGDTrainer.train_with_data_id + # train_with_data_id now folds the B2 straggler into the sct (#6/Root B); + # no config -> spread 0 -> offset 0 -> additive duration preserved. + _sim_straggler_offset_s = FedSGDTrainer._sim_straggler_offset_s def __init__(self, sim_send_ts, delay_d, leg_s=0.0): self.simulated = True diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 71dbf2d07..6ea14c55a 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -213,6 +213,54 @@ def test_no_vclock_fails(self): assert not r["ok"] and "K10" in r.get("note", "") +class TestIntrinsicSpanAnchor: + """#6: the clock-rate rungs anchor REAL on `intrinsic_span_s` (genuine + algorithmic time = barrier+eval) instead of raw wall ts, which bundles a + fixed inter-round transport artifact the sim (correctly) omits. Same run, + with vs without the field, must flip the verdict -- proving the anchor IS the + fix and that async_cifar10 (no field) is byte-identical (raw-ts fallback).""" + + def _pair(self, with_intrinsic): + # 10 committed data_ids. GENUINE work = 20 s/data_id (sim vclock advances + # 20/unit). REAL wall carries +15 s/data_id transport artifact (35/unit), + # but real's intrinsic_span_s reports the clean 20. + real_rounds, sim_rounds = [], [] + for d in range(1, 11): + re = {"event": "agg_round", "round": 1, "ts": float(d * 35), + "data_id": d, "cycle_data_id": d, + "contributing_trainers": ["a"], "staleness": [0], + "agg_goal_count": 1} + if with_intrinsic: + re["intrinsic_span_s"] = 20.0 + real_rounds.append(re) + sim_rounds.append({"event": "agg_round", "round": 1, "ts": float(d), + "vclock_now": float(d * 20), "data_id": d, + "cycle_data_id": d, "contributing_trainers": ["a"], + "staleness": [0], "agg_goal_count": 1, + "intrinsic_span_s": 20.0}) + return _agg(agg_rounds=real_rounds), _agg(agg_rounds=sim_rounds) + + def test_intrinsic_anchor_passes(self): + real, sim = self._pair(with_intrinsic=True) + assert pc.throughput_parity(real, sim, tol_rel=0.05)["ok"] + assert pc.per_round_advance_parity(real, sim)["ok"] + assert pc.total_commits_parity(real, sim)["ok"] + assert pc.terminal_state_parity(real, sim)["ok"] + wd = pc.wall_disparity(real, sim) + assert wd["anchor"] == "intrinsic_span" + assert wd["mean_abs_disparity_s"] < 1.0, wd + + def test_raw_wall_fallback_fails_and_is_byte_identical(self): + # Field absent -> real anchors on raw ts (35/unit) vs sim vclock (20/unit) + # -> the #6 gap the anchor exists to remove. Also the async_cifar10 path. + real, sim = self._pair(with_intrinsic=False) + assert not pc.throughput_parity(real, sim, tol_rel=0.05)["ok"] + assert not pc.per_round_advance_parity(real, sim)["ok"] + wd = pc.wall_disparity(real, sim) + assert wd["anchor"] == "wall_ts" + assert wd["max_abs_disparity_s"] > 100.0, wd # 15 s/unit artifact, cumulative + + class TestSimSpeedup: """sim_speedup [DIAG] asserts the principle-#13 invariant: the sim must run virtual time at least as fast as physical wall (sim_rate >= 1). K7 sim_rate From 3c052445bf39731d2d18053374e8a7bf3f58806f Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sat, 4 Jul 2026 23:26:45 -0400 Subject: [PATCH 21/56] fwdllm sim: fix MQTT join-notify startup race (#14) + parity update (K-D25 validated) backend/mqtt.py: join() now waits for the MQTT connection before subscribing and announcing presence. Previously, when the caller thread beat the async on_connect callback, both the topic subscriptions and the JOIN notify were silently dropped (notify's _is_connected guard) with no retry, so the end never registered with peers and the run hung below minInitialTrainers. This flakily lost trainer #379 in the fluxtune sim -> 9/10 ends -> zero selections. _wait_for_connect() is a no-op once connected, so async_cifar10 and all other examples are unchanged. Full pytest tests/ 879-green. simulate_fwdllm.md: fresh 2026-07-04 PM pairs validate K-D25/K-D26 clock-rate + phase fixes on both sync baselines (fwdllm 44->49, fwdllm_plus 40->45 pass); surviving fails characterized (marginal-KS, mis-applied gpu_budget, #7 selection divergence, stop-mismatch boundary). #14 logged and fixed. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 39 +++++++++++++------ lib/python/flame/backend/mqtt.py | 26 +++++++++++++ 2 files changed, 54 insertions(+), 11 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 7a6b77383..c3185c256 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -58,24 +58,35 @@ emergent smoke (was 44.7%); `[SIM_R1_DISPATCH]` fires despite `_release_end_on_r (3) **#13 sim_rate** for the sync baselines is near-1 (0.82/0.85) and Root A shows the vclock is correct — the residual is a small physical overhead; the severe case (fluxtune 0.26) is **R1-confounded** → fix #1c first. +**UPDATE (2026-07-04 later PM, `run_20260704_2224..2258`, data_id=20): K-D25/K-D26 clock-rate+phase fixes +VALIDATED on fresh banked pairs — for BOTH sync baselines.** fwdllm rose **44→49 pass**, fwdllm_plus **40→45**: +the 5 clock-rate rungs (throughput/overhead/per_round base/commits/terminal) + phase all **flipped to PASS** +exactly as predicted. fwdllm's 4 survivors are benign (2 marginal-KS, 2 mis-applied gpu_budget). fwdllm_plus's 7 +survivors are the #7 selection divergence (eligibility/selection_detail/avail_timebase — real 4.9 vs sim 9.6 +eligible at syn_0), gpu_budget, and a 13-vs-14 total_commits/terminal off-by-one (real capped at +`max_runtime_s=1800s`, data_id<20; sim finished 20 — a stop-mismatch + boundary, not a bug). **fluxtune sim did +NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/10 ends → async_oort never reached +`minInitialTrainers=10` → hung with **zero selections** (#14, **now FIXED** — `join()` waits for connect; +full `tests/` 879-green). fluxtune real is a clean reference (10 ends, data_id=20); re-run the fluxtune sim. + ### Per-baseline ground state | baseline | stop | R1 | cadence (V1/V2/g2/U3/S2/conv) | sim_rate | surviving fails | verdict | |---|---|---|---|---|---|---| -| **fwdllm** (sync) | data_id=10 both | **0% PASS** | **all PASS** | 0.82 | #6 clock-RATE family (throughput/commits/terminal + advance/phase rungs now un-skipped), gpu_budget, training_budget | **44/9/21.** Cadence CLEAN; fails all trace to #6 rate gap, no cadence bug | -| **fwdllm_plus** (sync) | data_id=10 both | 0% PASS | PASS (real completes now) | 0.85 | above + eligibility/selection_detail (real sees 4.9 eligible vs sim 9.6 at syn_0), avail_timebase | **40/12/21.** Real liveness FIXED (Stage C); new open: real/sim selection divergence #7 | -| **fluxtune** (async) | data_id ≥9 both | **60.4% FAIL** ⛔ | V2 PASS | **0.26** | R1, staleness, selection_detail (chosen 1.68 vs 1.19), avail_composition + #6 family | **39/15/19.** K-D19 R1 fix REFUTED — top blocker | +| **fwdllm** (sync) | data_id=20 both | **0% PASS** | **all PASS** | ~0.8 | per_round_advance + phase_gpu_compute (marginal KS 0.211/0.257 vs 0.20/0.25, means match 17.85≈17.48 / 1.03≈1.13), gpu_budget_real/sim (~40% overrun **symmetric** both modes — mis-applied invariant, not a divergence) | **49/4/21.** K-D25/K-D26 clock-rate+phase rungs VALIDATED PASS (44→49). Cadence CLEAN; 4 surviving fails all benign | +| **fwdllm_plus** (sync) | real→wall-cap 1800s (data_id<20); sim→data_id=20 | 0% PASS | **all PASS** | ~0.85 | #7 eligibility/selection_detail/avail_timebase (real 4.9/4.87 vs sim 9.6 eligible/chosen; in_flight 10=10), gpu_budget (symmetric ~40%), total_commits/terminal (13 vs 14 off-by-one, stop-mismatch) | **45/7/21.** Clock-rate rungs VALIDATED PASS (40→45). Fails = #7 + gpu_budget + stop-mismatch boundary; real ~4× slow (#7) | +| **fluxtune** (async) | real→data_id=20; **sim FAILED to launch** ⛔ | — (no sim) | — | — | **#14 sim join-race: 9/10 ends, no selection** (see below); prior pair: R1 60.4%, staleness, selection_detail, avail_composition | **No valid pair this run** — sim hung at startup. Prior banked pair (`_1721..`): 39/15/19, R1 refuted (#1c) | ### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) -| baseline | penultimate (10-data_id, K-D18) | **LATEST (sign-off `_1721..1819`)** | Δpass | key failing rungs | +| baseline | penultimate (sign-off `_1721..1819`) | **LATEST (`_2224..2258`, data_id=20)** | Δpass | key failing rungs | |---|---|---|---|---| -| **fwdllm** | `0704_022610` · 37/5/32 | **44 / 9 / 21** | +7 (skips −11) | overhead_residual, per_round_advance, throughput, total_commits, terminal_state, training_budget, gpu_budget | -| **fwdllm_plus** | `0704_033206` (real stalled) · 31/10/32 | **40 / 12 / 21** | +9 | above + eligibility, avail_timebase, selection_detail | -| **fluxtune** | `0704_020254` (R1 44.7%) · — | **39 / 15 / 19** | first valid pair since K-D19 | **r1_inflight_overlap**, staleness, avail_composition, selection_detail + #6 family | +| **fwdllm** | `0704_173105` · 44/9/21 | **49 / 4 / 21** | +5 (clock-rate+phase→PASS) | per_round_advance (KS 0.211), phase_gpu_compute (KS 0.257), gpu_budget_real/sim (~40% symmetric overrun) | +| **fwdllm_plus** | `0704_180511` · 40/12/21 | **45 / 7 / 21** | +5 (clock-rate→PASS) | #7 eligibility/selection_detail/avail_timebase, gpu_budget_real/sim, total_commits/terminal (13 vs 14, stop-mismatch) | +| **fluxtune** | `0704_181912` · 39/15/19 | **sim FAILED to launch** (#14) | — | no valid pair — 9/10 ends, zero selections (join race) | -*Pass counts rose / skips fell because the 12 rigor-gap rungs (#10) un-skipped; several FAIL on the #6 clock-rate -gap made observable, not new bugs. **Those fails (throughput/overhead/per_round/total_commits/terminal + -training_budget/phase_post_train) are now FIXED IN CODE (K-D25), validated on this banked pair — the scoreboard -counts above are the pre-fix JSON and advance only when the next run is banked.*** +*fwdllm's +5 is the K-D25/K-D26 validation: the 5 clock-rate rungs (throughput/overhead/per_round base/commits/ +terminal) + phase (training_budget/post_train) all flipped to PASS on this fresh `intrinsic_span_s`-carrying +pair. The 4 survivors are benign — 2 marginal-KS (means match) + the mis-applied gpu_budget invariant (symmetric +real/sim overrun, not a divergence). **fluxtune could not be scored** — see #14.* ### Next run — expected per-baseline deltas (K-D25 baseline-agnostic; K-D26 fluxtune-only) *A FRESH run is required for ALL THREE — `intrinsic_span_s` is a new agg emit; the checker's #6 anchor only @@ -93,6 +104,7 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi ### Open issues — master index (OPEN top; 1-line issue + next step) | # | issue | baseline(s) | next step | |---|---|---|---| +| **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | | **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. | fluxtune | **Fix (a) LANDED (K-D26):** vclock-key the abandon-timeout in sim (`_abandon_clock_now`, stamp+check on `vclock_now`); async_cifar10 byte-identical (vclock≈wall). **Fix (b) = the sustained NONE path** is the same slow-sim root → cured by **#13** (tracked), or a follow-up aggregator-side race fix. Re-run confirms how much (a) alone recovers. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | @@ -245,6 +257,11 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso --- ## §G Fixes landed (what worked — one line each; do not redo) +- **#14 MQTT join-notify startup race (2026-07-04)** — `backend/mqtt.py` `join()` fired `notify(JOIN)` + fire-and-forget; when the MainThread beat the async `on_connect`, the subscriptions AND the JOIN were dropped + (notify's own `_is_connected` guard) with no retry → the end never registered → async_oort hung below + `minInitialTrainers`. Fix: `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before + subscribe+notify; no-op when already connected → async_cifar10/all examples unchanged. Full `tests/` 879-green. - **#1c abandon-timeout vclock-key (K-D26, part a)** — the shared `async_oort` 90s `SEND_TIMEOUT_WAIT_S` in-flight abandon-timeout was keyed on `time.time()` (physical wall); in a slow sim (wall ≫ vclock) it evicted still-outstanding trainers from `all_selected` → R1 re-dispatch. Now runs on the vclock in sim diff --git a/lib/python/flame/backend/mqtt.py b/lib/python/flame/backend/mqtt.py index 8142d6d82..0845eff1a 100644 --- a/lib/python/flame/backend/mqtt.py +++ b/lib/python/flame/backend/mqtt.py @@ -194,8 +194,34 @@ def _topics_for_notify(self, channel: Channel) -> list[str]: return notify_topics + def _wait_for_connect(self, timeout: float = DEFAULT_RUN_ASYNC_WAIT_TIME) -> bool: + """Block until the MQTT client reports connected (bounded by timeout). + + `connect()` is async (on_connect fires on the run_forever thread), while + join()/subscribe()/notify() run on the caller thread. When join() wins the + race, subscribe() would target an unconnected client and notify(JOIN) is + dropped by its own connection guard -> this end never registers with peers + and hangs the run (a startup race that flakily loses a trainer). Waiting + here makes both the subscription and the JOIN announcement happen only + after the connection is up; when on_connect already fired (the common + case, e.g. async_cifar10) this returns immediately -> behavior unchanged. + """ + deadline = time.time() + timeout + while not self._is_connected and time.time() < deadline: + time.sleep(0.01) + if not self._is_connected: + logger.warning( + f"join proceeding without a confirmed MQTT connection " + f"after {timeout}s; presence notify may be dropped" + ) + return self._is_connected + def join(self, channel: Channel) -> None: """Join a channel by subscribing to topics.""" + # ensure the connection is up before subscribing/announcing, else the + # subscriptions and the JOIN notify can be silently dropped (startup race) + self._wait_for_connect() + for topic in self._topics_for_notify(channel): self.subscribe(topic) From 8151b942d223fabae1cdd50229e10fdaa23abade Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 00:14:05 -0400 Subject: [PATCH 22/56] =?UTF-8?q?fwdllm=20sim:=20bank=20fluxtune=20post-#1?= =?UTF-8?q?4=20parity=20(39/14/19)=20=E2=80=94=20R1=20part-a=20insufficien?= =?UTF-8?q?t?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The #14 join-race fix let the fluxtune sim run to data_id=20 (10 ends). The banked pair confirms K-D26's prediction: part (a) (vclock-keyed abandon-timeout) alone recovered ~0 of R1 residence — sim still 62.9% (215 tripwire fires) at sim_rate 0.29. The sustained NONE-delete path (part b) / #13 slow-sim is required, not optional. All fluxtune clock/commit/staleness fails are downstream of this R1+#13 desync. Doc §A/scoreboard/#1c updated in place. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index c3185c256..23a3d88d6 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -66,22 +66,25 @@ survivors are the #7 selection divergence (eligibility/selection_detail/avail_ti eligible at syn_0), gpu_budget, and a 13-vs-14 total_commits/terminal off-by-one (real capped at `max_runtime_s=1800s`, data_id<20; sim finished 20 — a stop-mismatch + boundary, not a bug). **fluxtune sim did NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/10 ends → async_oort never reached -`minInitialTrainers=10` → hung with **zero selections** (#14, **now FIXED** — `join()` waits for connect; -full `tests/` 879-green). fluxtune real is a clean reference (10 ends, data_id=20); re-run the fluxtune sim. +`minInitialTrainers=10` → hung with **zero selections** (#14, **FIXED** — `join()` waits for connect; full +`tests/` 879-green). The re-run (`_2327` sim) then banked cleanly (10 ends, data_id=20) → **39/14/19**, and +**answers the K-D26 open question: part (a) alone recovered ~0 of R1** — sim R1 residence is still **62.9%** +(215 `SIM_R1_DISPATCH` fires) at sim_rate 0.29, so the sustained NONE-delete path / **#13 slow sim is required** +(part-b), not optional. Every fluxtune clock/commit/staleness fail is downstream of this R1+#13 desync. ### Per-baseline ground state | baseline | stop | R1 | cadence (V1/V2/g2/U3/S2/conv) | sim_rate | surviving fails | verdict | |---|---|---|---|---|---|---| | **fwdllm** (sync) | data_id=20 both | **0% PASS** | **all PASS** | ~0.8 | per_round_advance + phase_gpu_compute (marginal KS 0.211/0.257 vs 0.20/0.25, means match 17.85≈17.48 / 1.03≈1.13), gpu_budget_real/sim (~40% overrun **symmetric** both modes — mis-applied invariant, not a divergence) | **49/4/21.** K-D25/K-D26 clock-rate+phase rungs VALIDATED PASS (44→49). Cadence CLEAN; 4 surviving fails all benign | | **fwdllm_plus** (sync) | real→wall-cap 1800s (data_id<20); sim→data_id=20 | 0% PASS | **all PASS** | ~0.85 | #7 eligibility/selection_detail/avail_timebase (real 4.9/4.87 vs sim 9.6 eligible/chosen; in_flight 10=10), gpu_budget (symmetric ~40%), total_commits/terminal (13 vs 14 off-by-one, stop-mismatch) | **45/7/21.** Clock-rate rungs VALIDATED PASS (40→45). Fails = #7 + gpu_budget + stop-mismatch boundary; real ~4× slow (#7) | -| **fluxtune** (async) | real→data_id=20; **sim FAILED to launch** ⛔ | — (no sim) | — | — | **#14 sim join-race: 9/10 ends, no selection** (see below); prior pair: R1 60.4%, staleness, selection_detail, avail_composition | **No valid pair this run** — sim hung at startup. Prior banked pair (`_1721..`): 39/15/19, R1 refuted (#1c) | +| **fluxtune** (async) | data_id=20 both | **62.9% FAIL** ⛔ | V2 PASS; V1/g2 marginal (10.9 vs 9.1) | **0.29** | **R1** (part-a alone recovered ~0), throughput/overhead/per_round/total_commits/terminal/staleness (all R1+#13-confounded), selection_detail (1.68 vs 1.34), gpu_budget, convergence (1 eval) | **39/14/19.** #14 fix let the sim RUN (10 ends, data_id=20). R1 still 62.9% (215 tripwire fires) → **#1c part (b)/#13 required**, K-D26 confirmed | ### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) | baseline | penultimate (sign-off `_1721..1819`) | **LATEST (`_2224..2258`, data_id=20)** | Δpass | key failing rungs | |---|---|---|---|---| | **fwdllm** | `0704_173105` · 44/9/21 | **49 / 4 / 21** | +5 (clock-rate+phase→PASS) | per_round_advance (KS 0.211), phase_gpu_compute (KS 0.257), gpu_budget_real/sim (~40% symmetric overrun) | | **fwdllm_plus** | `0704_180511` · 40/12/21 | **45 / 7 / 21** | +5 (clock-rate→PASS) | #7 eligibility/selection_detail/avail_timebase, gpu_budget_real/sim, total_commits/terminal (13 vs 14, stop-mismatch) | -| **fluxtune** | `0704_181912` · 39/15/19 | **sim FAILED to launch** (#14) | — | no valid pair — 9/10 ends, zero selections (join race) | +| **fluxtune** | `0704_181912` · 39/15/19 | **39 / 14 / 19** (`_2327` sim, post-#14) | ≈0 | **r1_inflight_overlap** (sim 62.9% vs real 0.2%), throughput/overhead/per_round/total_commits/terminal/staleness (R1+#13-confounded), selection_detail, gpu_budget, v1/g2, convergence | *fwdllm's +5 is the K-D25/K-D26 validation: the 5 clock-rate rungs (throughput/overhead/per_round base/commits/ terminal) + phase (training_budget/post_train) all flipped to PASS on this fresh `intrinsic_span_s`-carrying @@ -105,7 +108,7 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | # | issue | baseline(s) | next step | |---|---|---|---| | **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | -| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. | fluxtune | **Fix (a) LANDED (K-D26):** vclock-key the abandon-timeout in sim (`_abandon_clock_now`, stamp+check on `vclock_now`); async_cifar10 byte-identical (vclock≈wall). **Fix (b) = the sustained NONE path** is the same slow-sim root → cured by **#13** (tracked), or a follow-up aggregator-side race fix. Re-run confirms how much (a) alone recovers. | +| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **MEASURED (`_2327` re-run): part (a) alone recovered ~0 — R1 still 62.9% (was 60.4%), 215 tripwire fires at sim_rate 0.29 → part (b)/#13 is REQUIRED, not optional.** | fluxtune | **Fix (a) LANDED (K-D26)** but insufficient alone (see MEASURED). **Now the top blocker = fix (b): the sustained NONE-delete path**, same slow-sim root → cure via **#13** (make the fluxtune sim fast so physical receipt ≈ virtual commit) or a follow-up aggregator-side re-assertion of `all_selected` membership on the NONE slot-hold. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | | **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | From 552edf7e6d2180052d4485e54466e76f643e9785 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 00:24:25 -0400 Subject: [PATCH 23/56] fwdllm sim: root-cause fluxtune R1 to a two-ledger split (K-D27) Deep telemetry investigation of the post-#14 fluxtune sim (_2327). R1 part (a) recovered ~0 (still 62.9%, 215 SIM_R1_DISPATCH). Root cause: async_oort decides selection eligibility solely from all_selected (pruned on physical events - recv-fifo 2s re-select, RECVD/NONE cleanup) and never consults the aggregator's virtual in-flight set (_sim_inflight_expected u _sim_buffer). The intended bridge _agg_pending_commit_ref is read for telemetry but never assigned/filtered; _sim_pending_commit is accumulated but never bridged (felix's asyncfl maintains it). In the slow sim's large returned-but-uncommitted window, any physical prune frees a still-in-flight trainer -> re-dispatch -> R1. Real/async_cifar10 mask it via wall~=vclock. Evidence: num_eligible=10 while in_flight=10; in_pending_commit =0/879; CHANNEL_CLEANUP freed=0 x218; 0 fires of 90s/removed-ends prunes. Doc #1c + K-D27 updated; fix proposed (wire the bridge, exclude in filtered_ends). Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 23a3d88d6..21e28bee1 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -108,7 +108,7 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | # | issue | baseline(s) | next step | |---|---|---|---| | **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | -| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **MEASURED (`_2327` re-run): part (a) alone recovered ~0 — R1 still 62.9% (was 60.4%), 215 tripwire fires at sim_rate 0.29 → part (b)/#13 is REQUIRED, not optional.** | fluxtune | **Fix (a) LANDED (K-D26)** but insufficient alone (see MEASURED). **Now the top blocker = fix (b): the sustained NONE-delete path**, same slow-sim root → cure via **#13** (make the fluxtune sim fast so physical receipt ≈ virtual commit) or a follow-up aggregator-side re-assertion of `all_selected` membership on the NONE slot-hold. | +| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26)** but insufficient. **Robust fix (b) = wire the virtual in-flight set into selection eligibility:** assign `_agg_pending_commit_ref` = aggregator's `_sim_inflight_expected ∪ _sim_buffer` (kept in sync: add on dispatch, discard on COMMIT) and add `end not in _agg_pending_commit_ref` to `filtered_ends` — sim-only (∅ in real → async_cifar10 byte-identical). Drives `SIM_R1_DISPATCH→0`. Underlying window also collapses if #13 makes the sim fast. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | | **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | @@ -426,3 +426,18 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso `test_sim_vclock_keeps_virtually_recent_end` + reclaim regressions green. **Fix (b)** = the sustained NONE path, same slow-sim root → cured by **#13** (make the fluxtune sim fast so physical receipt ≈ virtual commit, as in async_cifar10) or a follow-up aggregator-side re-assertion; the emergent re-run measures how much (a) recovers. +- **K-D27** — #1c R1 DEEPER root-cause from the post-#14 `_2327` telemetry (part (a) recovered ~0; R1 still + 62.9%, 215 `SIM_R1_DISPATCH`). The NONE-delete (K-D26 part b) is one symptom of a **two-ledger split**: + async_oort decides selection eligibility SOLELY from `all_selected` (`filtered_ends`, async_oort:1607), a + ledger it prunes on PHYSICAL events (the recv-fifo 2s re-select loop, RECVD/NONE cleanup) — and NEVER consults + the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The intended bridge + `_agg_pending_commit_ref` is read for telemetry (async_oort:423) but **never assigned** (∅) and never filtered + on; `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared or bridged (felix's asyncfl + DOES maintain it, asyncfl:618/1490). fwdllm ported felix's `_sim_hold_busy_slots` (mirror the truth INTO + all_selected) but not the selector-side exclusion, so any physical-schedule prune in the (large, slow-sim) + returned-but-uncommitted window frees a still-in-flight trainer → own select() re-dispatches → R1. Real/ + async_cifar10 mask it via wall≈vclock (window≈0), not by design. **Telemetry proof:** `num_eligible=10` while + `in_flight=10`; `in_pending_commit=0/879` (dead hook); `[CHANNEL_CLEANUP] freed=0`×218; 0 fires of the 90s / + removed-ends prune paths (part (a) confirmed dead). **Fix (proposed):** wire `_agg_pending_commit_ref` to the + aggregator's virtual in-flight set (add on dispatch, discard on COMMIT) + exclude it in `filtered_ends`; sim- + only (∅ in real → async_cifar10 byte-identical). Controlling metric: `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. From c002c7fc13b36d3355d683b10e125104959121b9 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 00:50:59 -0400 Subject: [PATCH 24/56] fwdllm sim: fix fluxtune R1 via the pending-commit bridge (K-D27) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause (K-D27): async_oort gated selection eligibility solely on all_selected (pruned on physical events - recv-fifo 2s re-select, RECVD/NONE cleanup) and never consulted the aggregator's virtual in-flight set, so in the slow sim a returned-but-uncommitted trainer got re-dispatched (R1 62.9%, 215 SIM_R1_DISPATCH). Fix (felix-parallel with asyncfl's _sim_pending_commit, plus the selector-side exclusion fwdllm needs because async_oort + the slow sim prune all_selected mid-window): - fwdllm_aggregator: maintain _sim_pending_commit as the live in-flight set - add on dispatch (2627), discard on commit (811), in-place reconcile to the still-outstanding set in _sim_hold_busy_slots (clear+update, never rebind; replaces the |= accumulate that would starve committed trainers), clear on the legacy-drop boundary. Bind sel._agg_pending_commit_ref to the live set. - async_oort: exclude _agg_pending_commit_ref from filtered_ends (both the eligible-set list and the candidate loop). Sim-only: the set is empty in real and felix leaves the ref unset, so real / async_cifar10 are byte-identical. Tests: TestPendingCommitExcludedFromSelection (selector) + TestPendingCommitBridge (aggregator). Full pytest tests/ 886-green. Doc: #1c + K-D27 + §G updated to Fix LANDED. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 21 ++++-- .../horizontal/syncfl/fwdllm_aggregator.py | 25 ++++++- lib/python/flame/selector/async_oort.py | 18 ++++- .../mode/test_fwdllm_sim_grad_residence.py | 67 +++++++++++++++++++ .../tests/selector/test_oort_selector.py | 65 ++++++++++++++++++ 5 files changed, 189 insertions(+), 7 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 21e28bee1..bc1932a18 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -108,7 +108,7 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | # | issue | baseline(s) | next step | |---|---|---|---| | **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | -| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26)** but insufficient. **Robust fix (b) = wire the virtual in-flight set into selection eligibility:** assign `_agg_pending_commit_ref` = aggregator's `_sim_inflight_expected ∪ _sim_buffer` (kept in sync: add on dispatch, discard on COMMIT) and add `end not in _agg_pending_commit_ref` to `filtered_ends` — sim-only (∅ in real → async_cifar10 byte-identical). Drives `SIM_R1_DISPATCH→0`. Underlying window also collapses if #13 makes the sim fast. | +| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 886-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | | **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | @@ -260,6 +260,13 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso --- ## §G Fixes landed (what worked — one line each; do not redo) +- **#1c/K-D27 R1 two-ledger bridge (2026-07-05)** — async_oort gated selection eligibility ONLY on `all_selected` + (a physical-event-pruned ledger) and never on the aggregator's virtual in-flight set, so a slow-sim returned- + but-uncommitted trainer got re-dispatched (R1 62.9%). Fix: fwdllm agg maintains `_sim_pending_commit` felix- + style (add@dispatch, discard@commit, in-place reconcile in `_sim_hold_busy_slots`, clear on legacy-drop) and + binds it live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). Reconcile + SHRINKS (committed → re-pickable); the old `|= outstanding` would starve. Sim-only (∅ in real → async_cifar10 + byte-identical). `TestPendingCommitExcludedFromSelection` + `TestPendingCommitBridge`; full `tests/` 886-green. - **#14 MQTT join-notify startup race (2026-07-04)** — `backend/mqtt.py` `join()` fired `notify(JOIN)` fire-and-forget; when the MainThread beat the async `on_connect`, the subscriptions AND the JOIN were dropped (notify's own `_is_connected` guard) with no retry → the end never registered → async_oort hung below @@ -438,6 +445,12 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso returned-but-uncommitted window frees a still-in-flight trainer → own select() re-dispatches → R1. Real/ async_cifar10 mask it via wall≈vclock (window≈0), not by design. **Telemetry proof:** `num_eligible=10` while `in_flight=10`; `in_pending_commit=0/879` (dead hook); `[CHANNEL_CLEANUP] freed=0`×218; 0 fires of the 90s / - removed-ends prune paths (part (a) confirmed dead). **Fix (proposed):** wire `_agg_pending_commit_ref` to the - aggregator's virtual in-flight set (add on dispatch, discard on COMMIT) + exclude it in `filtered_ends`; sim- - only (∅ in real → async_cifar10 byte-identical). Controlling metric: `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. + removed-ends prune paths (part (a) confirmed dead). **Fix LANDED:** the fwdllm aggregator maintains + `_sim_pending_commit` felix-style (add on dispatch @2627, discard on COMMIT @811, in-place reconcile to the + still-outstanding set in `_sim_hold_busy_slots`, clear on the legacy-drop boundary) and BINDS it live to the + selector (`sel._agg_pending_commit_ref = self._sim_pending_commit`); async_oort's `filtered_ends` excludes + `_agg_pending_commit_ref` in BOTH spots (list @1607 + loop @1627). The reconcile SHRINKS (a committed trainer + becomes re-pickable) — the old `|= outstanding` accumulate would have starved every committed trainer. Sim-only + (∅ in real; felix leaves the ref unset → async_cifar10 byte-identical). Tests: `TestPendingCommitExcludedFrom + Selection` (selector) + `TestPendingCommitBridge` (aggregator); full `tests/` 886-green. Controlling metric: + `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. **Awaiting the fluxtune sim re-run to bank the recovery.** diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 0f3b14c6b..f51a361c4 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -810,6 +810,10 @@ def _sim_recv_min_grad(self, channel, recv_ends): self._advance_sim_clock(_advance_to) self._sim_committed.add(_end) self._sim_inflight_expected.pop(_end, None) + # K-D27 (felix-parallel, asyncfl:618): the grad committed -> the trainer is + # no longer in flight in virtual time -> drop it from pending so it is + # re-pickable. (_sim_hold_busy_slots reconciles right after, but be explicit.) + self._sim_pending_commit.discard(_end) # Learn this end's MODELED budget (contention-free lower bound) so the # gate fires on genuine stragglers for not-yet-observed trainers. _b = m.get(MessageType.TRAINING_BUDGET_S) if isinstance(m, dict) else None @@ -868,6 +872,10 @@ def _release_sim_slots_at_agg_goal(self, channel, is_async): self._sim_committed.clear() self._sim_buffer.clear() self._sim_inflight_expected.clear() + # K-D27: legacy-drop path abandons all in-flight -> nothing is pending. + # (async path clears via _sim_hold_busy_slots' reconcile; sync barriers use + # the random selector where the ref is inert, but keep the set bounded.) + self._sim_pending_commit.clear() if is_async: self._sim_hold_busy_slots(channel) @@ -905,7 +913,17 @@ def _sim_hold_busy_slots(self, channel) -> None: buffered = set(self._sim_buffer.pending_ends()) # returned, grad carried # Outstanding = still in flight in virtual time (not yet committed). outstanding = (set(self._sim_inflight_expected) | buffered) - self._sim_committed - self._sim_pending_commit |= outstanding + # K-D27: `_sim_pending_commit` is the aggregator's authoritative VIRTUAL + # in-flight set; reconcile it to `outstanding` IN PLACE (clear+update, never + # rebind -- the selector holds a live reference) so a committed trainer + # drops out and becomes re-pickable, while a returned-but-uncommitted one + # stays. Bind the reference so async_oort's eligibility filter excludes it + # (see async_oort `_pending`), making a still-outstanding trainer + # un-re-pickable regardless of all_selected churn. `|=` (the old accumulate) + # would never shrink -> a committed trainer would be starved forever. + self._sim_pending_commit.clear() + self._sim_pending_commit.update(outstanding) + sel._agg_pending_commit_ref = self._sim_pending_commit # Prune the (model_version, data_id, iteration) triplet guard to the # still-outstanding set (a trainer whose grad committed is re-pickable). self._trainer_state_dict = { @@ -2625,6 +2643,11 @@ def _distribute_weights_async( # running min) so the gate never laps a not-yet-committed trainer. _budget = self._sim_trainer_budget.get(end, self._sim_budget_min) self._sim_inflight_expected[end] = _round_now + _budget + # K-D27 (felix-parallel, asyncfl:1490): the instant a trainer is + # dispatched it is in flight in virtual time -> add to the pending- + # commit set so the selector's eligibility filter excludes it until + # its grad COMMITS (discarded in _sim_recv_min_grad). + self._sim_pending_commit.add(end) # NOTE: the async_oort re-pick triplet is stamped on grad RETURN # (in _process_single_trainer_message), NOT here at dispatch -- # stamping the whole cohort at the current _curr_agg_version froze diff --git a/lib/python/flame/selector/async_oort.py b/lib/python/flame/selector/async_oort.py index c18cfac67..c5b8d472c 100644 --- a/lib/python/flame/selector/async_oort.py +++ b/lib/python/flame/selector/async_oort.py @@ -1599,19 +1599,33 @@ def _handle_send_state( count_avl_eval = 0 count_ineligible = 0 + # SIM R1 guard (K-D27): async_oort releases `all_selected` on PHYSICAL + # events (the recv-fifo 2s re-select loop, RECVD/NONE cleanup). In a slow + # sim (fluxtune wall >> vclock) a grad stays returned-but-uncommitted for a + # long VIRTUAL window during which the aggregator still models the trainer + # as in flight; a physical prune in that window frees a still-outstanding + # trainer -> the aggregator's own select() re-dispatches it -> R1 residence + # violation. Additionally exclude the aggregator's virtual in-flight set + # (bound live via `_agg_pending_commit_ref` = its `_sim_pending_commit`) so + # a trainer is un-re-pickable until its grad COMMITS, regardless of + # all_selected churn. Empty (default) in real / async_cifar10 (felix keeps + # this ref unset; its wall~=vclock makes the window ~0) -> byte-identical. + _pending = getattr(self, "_agg_pending_commit_ref", None) or set() + # Check the eligible set first. Out of the ends, how many are # not in all_selected? Only those are eligible since the rest # have weights already sent to them for either train/eval # task. count_eligible_set_to_check = [ - end for end in ends if end not in self.all_selected + end for end in ends + if end not in self.all_selected and end not in _pending ] logger.debug( f"Before creating filtered_ends. count_eligible_set_to_check: {len(count_eligible_set_to_check)} from total {len(ends)} ends." ) for end_id in ends: - if end_id not in self.all_selected.keys(): + if end_id not in self.all_selected.keys() and end_id not in _pending: logger.debug( f"Creating filtered ends. Checking end id {end_id}, avl_state = {ends[end_id].get_property(PROP_AVL_STATE)}" ) diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py index 707b0cce3..18e24a85d 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -259,3 +259,70 @@ def test_not_simulated_is_noop(self): assert agg._sim_committed == {"X"} assert len(agg._sim_buffer) == 1 assert agg._sim_inflight_expected == {"X": 1.0} + + +class TestPendingCommitBridge: + """K-D27: the fwdllm aggregator maintains its VIRTUAL in-flight set + (`_sim_pending_commit`) and BINDS it to the selector's `_agg_pending_commit_ref` + so async_oort's eligibility filter excludes a returned-but-uncommitted trainer + regardless of `all_selected` churn. Felix-parallel (asyncfl maintains + `_sim_pending_commit` add@dispatch/discard@commit); the reconcile must SHRINK + (a committed trainer becomes re-pickable), which the old `|= outstanding` + accumulate did not -> it would starve every committed trainer forever. + """ + + def test_hold_reconciles_pending_to_outstanding_and_binds_ref(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B", "C"]) + agg._sim_committed = {"A"} # committed this cycle + agg._sim_buffer.add("B", 20.0, None) # returned, carried (uncommitted) + agg._sim_inflight_expected = {"C": 30.0} # still computing + + agg._sim_hold_busy_slots(ch) + + # pending == still-outstanding (carried B ∪ computing C); committed A dropped. + assert agg._sim_pending_commit == {"B", "C"} + # bound to the SAME live object the selector reads (never rebind). + assert ch._selector._agg_pending_commit_ref is agg._sim_pending_commit + + def test_committed_trainer_drops_out_not_starved(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B"]) + agg._sim_inflight_expected = {"A": 10.0, "B": 20.0} + agg._sim_hold_busy_slots(ch) + assert agg._sim_pending_commit == {"A", "B"} + + # A commits -> leaves in-flight, marked committed. + agg._sim_inflight_expected = {"B": 20.0} + agg._sim_committed = {"A"} + agg._sim_hold_busy_slots(ch) + + # A is re-pickable again (dropped from pending); `|=` would have kept it. + assert agg._sim_pending_commit == {"B"} + + def test_commit_discards_from_pending(self): + agg = _residence_agg(residence=True) + ch = _FakeSelChannel([]) + for e, sct in zip(["A", "B"], (10.0, 20.0)): + ch.add_msg(e, sct) + agg._sim_inflight_expected = {"A": 10.0, "B": 20.0} + agg._sim_pending_commit = {"A", "B"} + + agg._drain(ch, ["A", "B"], 1) # commit the smallest-sct grad (A) + + assert "A" not in agg._sim_pending_commit # discarded on COMMIT + assert "B" in agg._sim_pending_commit # still in flight + + def test_legacy_drop_path_clears_pending(self): + # residence OFF -> the boundary drops all in-flight; pending must clear + # too (else the sync-barrier set grows unbounded). + agg = _residence_agg(residence=False) + ch = _FakeSelChannel([]) + agg._sim_committed = {"X"} + agg._sim_buffer.add("Y", 5.0, None) + agg._sim_inflight_expected = {"Z": 9.0} + agg._sim_pending_commit = {"X", "Y", "Z"} + + agg._release_sim_slots_at_agg_goal(ch, is_async=True) + + assert agg._sim_pending_commit == set() diff --git a/lib/python/tests/selector/test_oort_selector.py b/lib/python/tests/selector/test_oort_selector.py index 5fe76243a..39506fc76 100644 --- a/lib/python/tests/selector/test_oort_selector.py +++ b/lib/python/tests/selector/test_oort_selector.py @@ -404,3 +404,68 @@ def test_fallback_to_eligible_when_no_connected_pool( ends=make_ends(["t1"]), concurrency=1, channel_props={}, ) assert async_oort.selected_ends["agg"] == {"t1"} + + +class TestPendingCommitExcludedFromSelection: + """K-D27: async_oort must exclude the aggregator's VIRTUAL in-flight set + (`_agg_pending_commit_ref`, bound live to the fwdllm aggregator's + `_sim_pending_commit`) from selection eligibility -- so a returned-but- + uncommitted trainer is NEVER re-dispatched even when `all_selected` has been + pruned by a physical event (the recv-fifo 2s re-select loop / RECVD-NONE + cleanup that a slow sim triggers). This is the fwdllm-side hardening over + felix's all_selected-only guard (felix leaves the ref unset -> byte-identical). + """ + + def test_pending_end_excluded_even_when_all_selected_empty( + self, async_oort, make_ends + ): + async_oort.requester = "agg" + async_oort.selected_ends = {"agg": set()} + async_oort.all_selected = {} # guard pruned by a physical event + async_oort._agg_pending_commit_ref = {"t3"} # agg still holds t3 in flight + ends = make_ends(count=5, prefix="t") + + result = async_oort._handle_send_state( + ends=ends, concurrency=5, channel_props={"round": 1}, + trainer_unavail_list=[], task_to_perform="train", + agg_version_state=(1, 0, 0), trainer_version_states={}, + connected_ends=ends, + ) + + # t3 is still outstanding in virtual time -> must not be re-picked. + assert "t3" not in result + # the other four are eligible -> selection still works (no over-restriction). + assert len(result) >= 1 + assert set(result).issubset({"t0", "t1", "t2", "t4"}) + + def test_all_pending_yields_no_selection(self, async_oort, make_ends): + async_oort.requester = "agg" + async_oort.selected_ends = {"agg": set()} + async_oort.all_selected = {} + ends = make_ends(count=4, prefix="t") + async_oort._agg_pending_commit_ref = set(ends) # every trainer in flight + + result = async_oort._handle_send_state( + ends=ends, concurrency=4, channel_props={"round": 1}, + trainer_unavail_list=[], task_to_perform="train", + agg_version_state=(1, 0, 0), trainer_version_states={}, + connected_ends=ends, + ) + assert result == {} # nobody eligible -> no re-dispatch-while-in-flight + + def test_empty_ref_is_byte_identical_noop(self, async_oort, make_ends): + """Real / async_cifar10 never populate the ref -> selection is unchanged.""" + async_oort.requester = "agg" + async_oort.selected_ends = {"agg": set()} + async_oort.all_selected = {} + # no _agg_pending_commit_ref attribute at all -> getattr default {} + ends = make_ends(count=5, prefix="t") + + result = async_oort._handle_send_state( + ends=ends, concurrency=5, channel_props={"round": 1}, + trainer_unavail_list=[], task_to_perform="train", + agg_version_state=(1, 0, 0), trainer_version_states={}, + connected_ends=ends, + ) + assert len(result) >= 1 + assert set(result).issubset(set(ends)) From b61c7933ea027a19acdbef36306f01da54df43ce Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 01:26:29 -0400 Subject: [PATCH 25/56] =?UTF-8?q?fwdllm=20sim:=20fix=20K-D27=20R1=20regres?= =?UTF-8?q?sion=20=E2=80=94=20drop=20stale=20=5Fsim=5Fcommitted=20from=20o?= =?UTF-8?q?utstanding=20(K-D27b)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The K-D27 bind alone left R1 at 67.6% on the _005222 re-run (238 SIM_R1_DISPATCH). Telemetry proved the selector filter worked (in_pending_commit populated; chosen ∩ pending = 0), so the re-dispatched trainer was in _sim_inflight_expected but NOT in pending -- only possible via _sim_committed. Root: the reconcile computed outstanding = (inflight ∪ buffer) − _sim_committed, but _sim_committed is a stale cross-cycle marker (cleared only at the agg-goal boundary). A trainer that committed then was legitimately re-picked + re-dispatched (re-added to _sim_inflight_expected) got dropped from outstanding by that subtraction -> re-pickable while its new dispatch was still in flight -> R1. Fix: outstanding = inflight ∪ buffer (drop − _sim_committed). A this-cycle commit is already popped from both sets, so the subtraction was redundant for it and harmful for the re-dispatch case. Regression test test_recommitted_trainer_stays_pending_despite_stale_committed. Full tests/ 887-green. Also banks the _005222 parity (38/15/19, R1 67.6%) that surfaced this. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 23 ++++++++++++---- .../horizontal/syncfl/fwdllm_aggregator.py | 26 ++++++++++++------- .../mode/test_fwdllm_sim_grad_residence.py | 23 ++++++++++++++++ 3 files changed, 58 insertions(+), 14 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index bc1932a18..ff7198e61 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -84,7 +84,7 @@ NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/ |---|---|---|---|---| | **fwdllm** | `0704_173105` · 44/9/21 | **49 / 4 / 21** | +5 (clock-rate+phase→PASS) | per_round_advance (KS 0.211), phase_gpu_compute (KS 0.257), gpu_budget_real/sim (~40% symmetric overrun) | | **fwdllm_plus** | `0704_180511` · 40/12/21 | **45 / 7 / 21** | +5 (clock-rate→PASS) | #7 eligibility/selection_detail/avail_timebase, gpu_budget_real/sim, total_commits/terminal (13 vs 14, stop-mismatch) | -| **fluxtune** | `0704_181912` · 39/15/19 | **39 / 14 / 19** (`_2327` sim, post-#14) | ≈0 | **r1_inflight_overlap** (sim 62.9% vs real 0.2%), throughput/overhead/per_round/total_commits/terminal/staleness (R1+#13-confounded), selection_detail, gpu_budget, v1/g2, convergence | +| **fluxtune** | `_2327` sim · 39/14/19 (R1 62.9%) | **38 / 15 / 19** (`_005222`, K-D27 bind) | ≈0 | **r1_inflight_overlap sim 67.6%** (K-D27 bind alone insufficient — stale-`_sim_committed` bug, fixed K-D27b, re-run pending), + throughput/overhead/commits/staleness/selection_detail/gpu_budget/v1/g2/convergence (R1+#13-confounded) | *fwdllm's +5 is the K-D25/K-D26 validation: the 5 clock-rate rungs (throughput/overhead/per_round base/commits/ terminal) + phase (training_budget/post_train) all flipped to PASS on this fresh `intrinsic_span_s`-carrying @@ -108,7 +108,7 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | # | issue | baseline(s) | next step | |---|---|---|---| | **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | -| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 886-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | +| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27+K-D27b):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). **K-D27b:** the bind alone left R1 at 67.6% (`_005222`) — the reconcile subtracted the STALE `_sim_committed`, dropping re-dispatched-after-commit trainers; fixed to `outstanding = inflight ∪ buffer`. Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 887-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | | **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | @@ -265,8 +265,11 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso but-uncommitted trainer got re-dispatched (R1 62.9%). Fix: fwdllm agg maintains `_sim_pending_commit` felix- style (add@dispatch, discard@commit, in-place reconcile in `_sim_hold_busy_slots`, clear on legacy-drop) and binds it live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). Reconcile - SHRINKS (committed → re-pickable); the old `|= outstanding` would starve. Sim-only (∅ in real → async_cifar10 - byte-identical). `TestPendingCommitExcludedFromSelection` + `TestPendingCommitBridge`; full `tests/` 886-green. + SHRINKS (committed → re-pickable); the old `|= outstanding` would starve. K-D27b: `outstanding = inflight ∪ + buffer` (NOT `− _sim_committed`, a stale cross-cycle marker that dropped re-dispatched-after-commit trainers → + R1 stayed 67.6% on the bind-only attempt). Sim-only (∅ in real → async_cifar10 byte-identical). `TestPending + CommitExcludedFromSelection` + `TestPendingCommitBridge` (incl. `..._stays_pending_despite_stale_committed`); + full `tests/` 887-green. - **#14 MQTT join-notify startup race (2026-07-04)** — `backend/mqtt.py` `join()` fired `notify(JOIN)` fire-and-forget; when the MainThread beat the async `on_connect`, the subscriptions AND the JOIN were dropped (notify's own `_is_connected` guard) with no retry → the end never registered → async_oort hung below @@ -453,4 +456,14 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso becomes re-pickable) — the old `|= outstanding` accumulate would have starved every committed trainer. Sim-only (∅ in real; felix leaves the ref unset → async_cifar10 byte-identical). Tests: `TestPendingCommitExcludedFrom Selection` (selector) + `TestPendingCommitBridge` (aggregator); full `tests/` 886-green. Controlling metric: - `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. **Awaiting the fluxtune sim re-run to bank the recovery.** + `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. + **CORRECTION (K-D27b, `_005222` re-run): the bind alone was INSUFFICIENT — R1 stayed 67.6% (238 tripwire).** + Telemetry proved the filter worked (`in_pending_commit` now 5755 rows; chosen ∩ pending = 0), so the re- + dispatched trainer was in `_sim_inflight_expected` but NOT in pending — only possible if it was in + `_sim_committed`. Root: the reconcile computed `outstanding = (inflight ∪ buffer) − _sim_committed`, but + `_sim_committed` is a STALE cross-cycle marker (cleared only at the boundary). A trainer that committed then + was legitimately re-picked + re-dispatched (re-added to `_sim_inflight_expected`) got dropped from `outstanding` + by that subtraction → re-pickable while its NEW dispatch was in flight → R1. Fix: `outstanding = + inflight ∪ buffer` (drop `− _sim_committed`; a this-cycle commit is already popped from both sets, so the + subtraction was redundant and harmful). `test_recommitted_trainer_stays_pending_despite_stale_committed`; full + `tests/` 887-green. **Awaiting the next fluxtune sim re-run to bank the recovery.** diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index f51a361c4..74c534788 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -911,15 +911,23 @@ def _sim_hold_busy_slots(self, channel) -> None: selected_ends = getattr(sel, "selected_ends", None) buffered = set(self._sim_buffer.pending_ends()) # returned, grad carried - # Outstanding = still in flight in virtual time (not yet committed). - outstanding = (set(self._sim_inflight_expected) | buffered) - self._sim_committed - # K-D27: `_sim_pending_commit` is the aggregator's authoritative VIRTUAL - # in-flight set; reconcile it to `outstanding` IN PLACE (clear+update, never - # rebind -- the selector holds a live reference) so a committed trainer - # drops out and becomes re-pickable, while a returned-but-uncommitted one - # stays. Bind the reference so async_oort's eligibility filter excludes it - # (see async_oort `_pending`), making a still-outstanding trainer - # un-re-pickable regardless of all_selected churn. `|=` (the old accumulate) + # Outstanding = still in flight in virtual time. Membership in + # `_sim_inflight_expected` (popped on commit @811) or `_sim_buffer` (popped + # on commit @793) IS the "not yet committed" truth. Do NOT subtract + # `_sim_committed` (K-D27 fix): it is a STALE, cross-cycle marker cleared + # only at the agg-goal boundary, so a trainer that committed and was then + # legitimately re-picked + re-dispatched (re-added to `_sim_inflight_expected`) + # would be wrongly dropped from `outstanding` -> re-pickable while its NEW + # dispatch is still in flight -> R1 residence violation. A trainer that + # committed THIS cycle is already absent from both sets, so the subtraction + # was redundant for it and harmful for the re-dispatch case. + outstanding = set(self._sim_inflight_expected) | buffered + # `_sim_pending_commit` is the aggregator's authoritative VIRTUAL in-flight + # set; reconcile it to `outstanding` IN PLACE (clear+update, never rebind -- + # the selector holds a live reference) so a genuinely-committed trainer drops + # out (re-pickable) while a dispatched-but-uncommitted one stays. Bind the + # reference so async_oort's eligibility filter excludes it (see async_oort + # `_pending`) regardless of all_selected churn. `|=` (the old accumulate) # would never shrink -> a committed trainer would be starved forever. self._sim_pending_commit.clear() self._sim_pending_commit.update(outstanding) diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py index 18e24a85d..de1dc6943 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_residence.py @@ -326,3 +326,26 @@ def test_legacy_drop_path_clears_pending(self): agg._release_sim_slots_at_agg_goal(ch, is_async=True) assert agg._sim_pending_commit == set() + + def test_recommitted_trainer_stays_pending_despite_stale_committed(self): + """The K-D27 R1 regression: `_sim_committed` is a STALE cross-cycle marker + (cleared only at the boundary). A trainer that committed then was re-picked + + re-dispatched is back in `_sim_inflight_expected`; the reconcile must NOT + drop it from pending just because it lingers in `_sim_committed` -- else it + is re-pickable while its NEW dispatch is still in flight -> R1 (the 67.6% + overlap the first K-D27 attempt still showed). `outstanding` therefore keys + on inflight/buffer membership only, never `- _sim_committed`.""" + agg = _residence_agg(residence=True) + ch = _FakeSelChannel(["A", "B"]) + # A committed earlier this cycle (still in the stale marker) AND has been + # re-dispatched -> back in flight. B is a first-time in-flight trainer. + agg._sim_committed = {"A"} + agg._sim_inflight_expected = {"A": 40.0, "B": 20.0} + + agg._sim_hold_busy_slots(ch) + + # A is genuinely in flight again -> stays pending + held (un-re-pickable). + assert "A" in agg._sim_pending_commit + assert "B" in agg._sim_pending_commit + assert "A" in ch._selector.all_selected + assert "A" in ch._selector.selected_ends["agg"] From 35417519221c8c09731c399596730b16b565e9e3 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 02:33:04 -0400 Subject: [PATCH 26/56] fwdllm sim: root-cause #13 drain stall + lock the felix-port design (handoff) Verified against async_cifar10 (felix asyncfl::_sim_recv_min): fwdllm's async drain _sim_recv_min_grad is an early port missing ALL of felix's straggler refinements (grep-confirmed felix=N/fwdllm=0): stuck-end eviction on deadline (inflight_expected.pop(_stuck_end)), probe-ceiling + _sim_end_has_ready_msg ready-gating, drain_ready/_sim_sct_ordered_drain direct ingest, _sim_free_slot_ts freed-slot refill. So on the 30s RECV_TIMEOUT_WAIT_S deadline fwdllm just breaks and leaves the stuck trainer in _sim_inflight_expected forever -> earlier_stuck re-fires the full 30s every commit cycle -> pipeline starvation (data_id=1 in ~9min, sim_rate 0.06, 205 recv 2s-timeouts). R1 is FIXED (K-D27b, 238->0); fixing it unmasked this drain stall. felix's _sim_recv_min IS the intended slot-based discrete-event model (dispatch c slots -> batch-collect cohort replies -> reorder by sct at the aggregator -> skip vclock to next sct -> free one slot -> re-select). Doc: new "#13 DESIGN" block with operator-confirmed design choices (invisible-until-sct residence, modeled-fastest-k set, variance rollbacks untouched, sim-only) + a 4-step incremental port plan + expected result. #13 promoted to top blocker; R1/#1c marked FIXED in the roots/scoreboard. Design locked for a fresh-context impl. Co-Authored-By: Claude Opus 4.8 --- lib/python/examples/fwdllm/simulate_fwdllm.md | 61 +++++++++++++++++-- 1 file changed, 55 insertions(+), 6 deletions(-) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ff7198e61..bcf553165 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -77,7 +77,7 @@ NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/ |---|---|---|---|---|---|---| | **fwdllm** (sync) | data_id=20 both | **0% PASS** | **all PASS** | ~0.8 | per_round_advance + phase_gpu_compute (marginal KS 0.211/0.257 vs 0.20/0.25, means match 17.85≈17.48 / 1.03≈1.13), gpu_budget_real/sim (~40% overrun **symmetric** both modes — mis-applied invariant, not a divergence) | **49/4/21.** K-D25/K-D26 clock-rate+phase rungs VALIDATED PASS (44→49). Cadence CLEAN; 4 surviving fails all benign | | **fwdllm_plus** (sync) | real→wall-cap 1800s (data_id<20); sim→data_id=20 | 0% PASS | **all PASS** | ~0.85 | #7 eligibility/selection_detail/avail_timebase (real 4.9/4.87 vs sim 9.6 eligible/chosen; in_flight 10=10), gpu_budget (symmetric ~40%), total_commits/terminal (13 vs 14 off-by-one, stop-mismatch) | **45/7/21.** Clock-rate rungs VALIDATED PASS (40→45). Fails = #7 + gpu_budget + stop-mismatch boundary; real ~4× slow (#7) | -| **fluxtune** (async) | data_id=20 both | **62.9% FAIL** ⛔ | V2 PASS; V1/g2 marginal (10.9 vs 9.1) | **0.29** | **R1** (part-a alone recovered ~0), throughput/overhead/per_round/total_commits/terminal/staleness (all R1+#13-confounded), selection_detail (1.68 vs 1.34), gpu_budget, convergence (1 eval) | **39/14/19.** #14 fix let the sim RUN (10 ends, data_id=20). R1 still 62.9% (215 tripwire fires) → **#1c part (b)/#13 required**, K-D26 confirmed | +| **fluxtune** (async) | R1 FIXED; sim **STALLS** on #13 (killed early, data_id=1) | **0% (FIXED, K-D27b)** | V2 PASS; V1/g2 marginal | **0.06 ⛔** | **#13 drain stall** (30s/cycle, pipeline starvation — see "#13 DESIGN") + throughput/overhead/commits/staleness (all #13-confounded), gpu_budget | **R1 FIXED** (`SIM_R1_DISPATCH` 238→0, `_012922`). #13 drain redesign is NEXT (port felix `_sim_recv_min`). Can't bank a valid pair until the sim runs at speed | ### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) | baseline | penultimate (sign-off `_1721..1819`) | **LATEST (`_2224..2258`, data_id=20)** | Δpass | key failing rungs | @@ -111,21 +111,70 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27+K-D27b):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). **K-D27b:** the bind alone left R1 at 67.6% (`_005222`) — the reconcile subtracted the STALE `_sim_committed`, dropping re-dispatched-after-commit trainers; fixed to `outstanding = inflight ∪ buffer`. Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 887-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | -| **#13** ⭐ | sim is physically slow (`sim_rate` 0.26 fluxtune / 0.82–0.85 sync). **Now promoted:** the slow sim (wall ≫ vclock) is the ROOT that makes async_oort's wall-keyed guard machinery misfire → #1c (both the 90s timeout and the NONE-delete race). | all (fluxtune worst) | Cure the fluxtune inter-round physical wall (the recv_fifo/MQTT waits, #11-adjacent) so wall≈vclock like async_cifar10 — then the #1c NONE-delete sustained path collapses on its own. Tracked as the deeper follow-up per the #1c "both" decision. | +| **#13** ⭐⭐ | **sim physically slow (`sim_rate` 0.06–0.29 fluxtune) — ROOT-CAUSED (2026-07-05): fwdllm's async drain `_sim_recv_min_grad` is an EARLY port that lacks ALL of felix's straggler-handling refinements, so a stuck/idle in-flight trainer blocks every commit for the full 30s `RECV_TIMEOUT_WAIT_S`, forever.** Exposed once #1c/R1 stopped the (incorrect) re-dispatch churn that was masking it. | fluxtune (all) | **Port felix's drain (design locked below — "#13 DESIGN").** The `_sim_recv_min` reference in `asyncfl/top_aggregator.py` IS the intended event-driven model; fwdllm has the skeleton but not the refinements. | | **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only 4.9 eligible vs sim 9.6 | fwdllm_plus | Real liveness FIXED (Stage C — completes 10). Remaining: profile the per-iteration reselection + oracular read cost from the banked per-phase log; explain the real/sim eligible-count gap at 100% avail. Not a sim bug. | | **#12c** | `--delays on` uses the hardcoded `training_delay_factor=10` ÷10 shrink unless `--delay-factor` passed | all | Optional: pass `--delay-factor` for the full 4–18s registry delay. Config-flow otherwise correct (K-D24/K-D21). | | **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits, real-only; one-grad-per-poll drain tail) | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run to validate the MQTT settle timing (principle #8/#11c). Never change grad values/cadence. | ### Open roots — ranked lowest-rung-first -1. **#1c R1 residence (fluxtune)** — ROOT-CAUSED (K-D26): a wall-vs-vclock desync in the shared async_oort - re-pick guard. Wall-timeout path FIXED; the sustained NONE-delete path is the same slow-sim root → **#13**. -2. **#13 slow sim (fluxtune wall ≫ vclock)** — promoted: it is what makes the async_oort guard machinery misfire - (#1c) AND the headline slowdown. Cure the fluxtune inter-round physical wall → wall≈vclock (async_cifar10). +1. **#13 slow sim / drain stall (fluxtune)** — ROOT-CAUSED + design LOCKED (see "#13 DESIGN"): fwdllm's + `_sim_recv_min_grad` lacks felix's straggler refinements → a stuck in-flight trainer blocks every commit the + full 30s → starvation. NEXT to implement (port felix's `_sim_recv_min`). Now the top blocker. +2. **#1c R1 residence (fluxtune)** — FIXED (K-D27/b): `SIM_R1_DISPATCH` 238→0. The pending-commit bridge + + dropping the stale `_sim_committed` subtraction. (Was masking #13; fixing it exposed the drain stall.) 3. **#7 fwdllm_plus real speed / selection divergence** — real completes now; explain the 4× slowness and the syn_0 eligible-count gap from banked telemetry. 3. *(#6 clock-rate — RESOLVED as a checker anchor, K-D25; sim vclock was correct. Only the 3 marginal rungs + the B2 straggler calibration remain, both emergent at the longer sign-off run.)* +### #13 DESIGN — event-driven drain redesign (design LOCKED 2026-07-05; NEXT to implement) +*R1 is FIXED (K-D27/b: `SIM_R1_DISPATCH` 238→0). But fixing R1 unmasked #13: the `_2327`→`_005222`→`_012922` +runs got progressively slower (`sim_rate` 0.29→0.06), `data_id=1` in ~9 min, `aggregate` calls taking exactly +30.0s. The R1-violation churn had been masking a broken drain by (incorrectly) keeping trainers busy.* + +**Root cause (verified by side-by-side with felix `asyncfl/top_aggregator.py::_sim_recv_min`).** fwdllm's +`_sim_recv_min_grad` is an EARLY port that lacks every one of felix's straggler-handling refinements +(grep-confirmed felix=N / fwdllm=0): `_stuck_end`/`_sim_gate_failsafe`/`inflight_expected.pop(_stuck_end)`, +`probe_ceiling` + `_sim_end_has_ready_msg` ready-gating, `drain_ready`/`_sim_sct_ordered_drain` direct ingest, +`_sim_free_slot_ts` freed-slot refill. Consequence: on the 30s `RECV_TIMEOUT_WAIT_S` deadline fwdllm just +`break`s and **leaves the stuck trainer in `_sim_inflight_expected` forever**, so `earlier_stuck` re-fires the +FULL 30s every commit cycle. Evidence: trainer idle in `_fetch_weights` (already replied, awaiting reselect) sits +in the expected set; the composer loop is frozen 30s in the drain so freed slots never re-dispatch → pipeline +starvation (only 5/10 trainers active, 205 recv 2s-timeouts = 74% of wall). + +**Confirmed pathway = felix's model (this is what SHOULD happen, verified in async_cifar10).** Slot-based +discrete-event sim: dispatch to fill `c` slots → trainers compute the REAL grad in parallel (~4s) and reply +carrying their modeled `sct` (the aggregator only learns an `sct` on receipt) → the recv window batch-collects +the cohort's replies at once → **reorder by `sct` AT THE AGGREGATOR** → **skip the vclock to the next visible +`sct`** (accounting jump, ~0 wall), at which one update commits, ONE slot frees (`_sim_free_slot_ts` stamps the +freed vclock), and the aggregator **selects again** to refill. Physical wall = GPU compute only; the vclock +charges GPU + the SKIPPED modeled delays → `sim_rate>1`. + +**Design choices (operator-confirmed):** (1) residence = **invisible-until-`sct`, released-at-boundary** — a +replied trainer is NOT eagerly re-dispatched; it and its update stay invisible until vclock reaches `sct`, then +the update releases and the trainer is re-usable only AFTER the round/bin/iteration boundary. (2) committed SET = +**modeled-fastest-k** (strict `sct` order kept; do NOT relax to arrival order). (3) **variance-BAD rollbacks are +NORMAL** (happen in real too — most cycles var isn't met, discarded until finally met); do NOT touch variance — +the fix is purely drain timing. (4) sim-only, real-mode byte-identical. + +**Plan (incremental, drain is the K-D16/K-D17 deadlock zone — one step per short-sim validation):** +1. **Stuck-end eviction** (felix `_sim_recv_min:436-442`): on the deadline, `pop(_stuck_end)` from + `_sim_inflight_expected` so a lost/idle straggler can't block every future cycle. *Suspected to break the 30s + stall alone.* Validate with a short fluxtune sim + the residence pytests. +2. **Probe-ceiling + ready-gating** (felix `:399-411`, `_sim_end_has_ready_msg`): only probe an in-flight end if + its rxq is physically ready OR its `exp ≤ buffered_min + slack` — stop burning 2s grace on far-future or + not-arrived stragglers. +3. **Direct `drain_ready` ingest** (felix `_sim_sct_ordered_drain`, `:375-390`): drain each live end's rxq + directly (no recv_fifo streamer) so a delivered-but-stranded update is a COMPLETE buffer snapshot (the streamer + can strand a delivered grad out of the buffer's view → the gate waits on a grad it already has). +4. **Freed-slot refill stamp** (felix `_sim_free_slot_ts`, `:507-520`): stamp each train-commit's freed vclock and + refill the slot's next dispatch at THAT vclock, so re-selection is prompt + correctly timed. + Keep the fwdllm-specific grad orchestration (one-grad-per-call loop, agg-goal-boundary carry, K-D12/K-D17b). + +**Expected result:** `aggregate` wall per cycle 30s→~GPU-compute (seconds); slots stay full; `data_id` progresses +at ~real speed; `sim_rate`→≥1 once `--delay-factor` gives the vclock delay-headroom (#12c). R1 stays 0 (residence +unchanged). Real byte-identical. Watch: throughput/staleness/commits rungs should ease (they were #13-confounded). + ### SKIP audit (19–21 skips; ~16 legit) Legit (mechanism genuinely absent at Phase-1 syn_0 + `random` selector): 7 availability ground-truth rungs + 4 delivery/withheld (Phase-2 effect path, not built) + 3 DynamicKC (disabled by design) + 2 oort-only From 32575d1b10f988632f680b3a19b011d3aa828d62 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 04:06:33 -0400 Subject: [PATCH 27/56] =?UTF-8?q?fwdllm=20sim:=20fix=20#13=20drain=20?= =?UTF-8?q?=E2=80=94=20event-driven=20port=20of=20felix's=20=5Fsim=5Frecv?= =?UTF-8?q?=5Fmin=20(K-D28/b/c)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ports felix's straggler-handling refinements into fwdllm's async grad drain (_sim_recv_min_grad), fixing the #13 "sim physically slow / 30s drain stall". All sim-only, fwdllm-scoped; async_cifar10 byte-identical. - Step 1 (K-D28): stuck-end eviction on the RECV_TIMEOUT_WAIT_S deadline so a lost/idle straggler can't re-fire the full 30s every cycle; + fwdllm recv-grace floor 2->5s (SIM_RECV_GRACE_FLOOR_S, tunable — the forward-grad GPU can take ~4s). - Step 2 (K-D28b): probe-ceiling + ready-gating — probe a non-recv_end in-flight end only if physically ready or exp <= buffered_min+slack (stop burning grace on far-future stragglers). - Step 3 (K-D28c): direct drain_ready ingest (flag sim_sct_ordered_drain) — sweep each in-flight rxq directly instead of the recv_fifo streamer, which blocked the full grace per not-ready end (the 12-34s blocks) and could strand a delivered grad (past-dated commit). Validated on fluxtune smoke (run_..._024637 -> 032729): sim_rate 0.06->0.29, aggregate blocks >=5s 54->8, 30s stall gone, R1 stays 0. Step 4 (K-D28d, freed-slot staggered re-dispatch) is implemented + tested but INVESTIGATED-NEUTRAL and left DISABLED (see §H): the residual 11-12s holds are the drain correctly waiting (strict sct order) for the earliest-sct in-flight straggler — inherent to real-GPU + strict-order + trickle dispatch, not a dispatch-timing artifact. The residual sim_rate<1 is #12c-bound (no --delay-factor -> ~no vclock headroom); measure with --delay-factor before any further drain work. Tests: TestStuckEndEviction, TestProbeCeilingReadyGating, TestSctOrderedDrain, TestFreedSlotRefill; full tests/mode -k fwdllm green (168). Co-Authored-By: Claude Opus 4.8 --- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 16 ++ lib/python/examples/fwdllm/simulate_fwdllm.md | 146 ++++++++-- .../horizontal/syncfl/fwdllm_aggregator.py | 152 ++++++++-- .../tests/mode/test_fwdllm_sim_grad_loop.py | 259 ++++++++++++++++++ 4 files changed, 526 insertions(+), 47 deletions(-) diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 9a18faffc..97b5565d0 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -78,6 +78,22 @@ experiments: # + re-dispatching them (the 2x-forward-pass residence bug, K-D12). # Real satisfies this by channel construction, so it's inert there. sim_inflight_residence: true + # #13 step 3: drain each in-flight end's rxq DIRECTLY (channel.drain_ready) + # instead of the recv_fifo streamer. The streamer blocks the full recv + # grace PER not-ready end (the 4-10s inter-burst gaps that dominated the + # post-step-2 wall) and can strand a delivered grad out of the buffer's + # view (past-dated commit). drain_ready sweeps ready rxqs non-blocking and + # returns on the first arrival. SIM-only (real never enters _sim_recv_min_grad). + sim_sct_ordered_drain: true + # #13 step 4: freed-slot staggered re-dispatch. INVESTIGATED + DISABLED + # (K-D28d, run_..._034550): neutral-to-slightly-worse for fwdllm (sim_rate + # 0.289->0.274, the 11-12s holds persisted). The holds are the drain + # correctly waiting (strict sct order) for the earliest-sct in-flight + # straggler, NOT a dispatch-bunching artifact; staggering feeds an OLD + # freed-slot vclock as send_ts, making the gate expect trainers even + # earlier -> more holds. Code kept (flag-gated) for a possible reworked + # attempt; see §H. Leave OFF -> validated steps 1-3 drain. + sim_staggered_redispatch: false # Async fedbuff: accept stale carried grads + down-weight by (V'-V), # not reject (K-D13). Set identically in the real sibling; never a # parity lever. camelCase MUST match the base key to override it — diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index bcf553165..ea3195755 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -36,6 +36,21 @@ discipline, starvation self-termination, A6/A7/A8/K11 ground-truth rungs). ## §A Current status +**Ground truth (2026-07-05): the fluxtune #13 drain is FIXED (steps 1-3, K-D28/b/c) — the sim runs at speed and +the 30s-forever stall is gone.** Landed + validated across `run_..._024637 → 032729`: stuck-end eviction + +fwdllm recv-grace floor 2→5s (tunable) + probe-ceiling/ready-gating + `drain_ready` direct ingest → `sim_rate` +0.06→0.29, blocks≥5s 54→8, `SIM_GRAD_STUCK_EVICT` fires once (not spinning), R1 stays 0, async_cifar10 +byte-identical (all fwdllm-only methods). **Step 4 (freed-slot staggering, K-D28d) was INVESTIGATED-NEUTRAL → +DISABLED (§H):** the residual 11-12s drain holds are the gate correctly waiting (strict sct order) for the +earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU + strict-order + trickle +dispatch, not a dispatch-timing artifact. **NEXT (top actionable): a fluxtune `--delay-factor` sim run (#12c)** — +the residual `sim_rate<1` is dominated by the missing vclock delay-headroom (D≈0.4-1.8s now → the vclock charges +~only GPU), NOT the drain. With D≈4-18s the vclock jumps by gpu+D per commit while wall pays only the skipped GPU +→ expect `sim_rate` toward/past 1; only then bank the first valid fluxtune real↔sim pair. R1 (#1c) FIXED earlier +(K-D27/b, `SIM_R1_DISPATCH` 238→0). *(Sync baselines fwdllm 49/4/21, fwdllm_plus 45/7/21 unchanged — see below.)* + +--- + **Ground truth (2026-07-04 PM): the Phase-1 sign-off run RAN and is USABLE — the first matched-`data_id` real↔sim pair for all three baselines.** Command: `run_sequential.sh --mode both --delays on --max-data-id 10 --max-runtime-s 3600 --yes` @@ -77,7 +92,7 @@ NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/ |---|---|---|---|---|---|---| | **fwdllm** (sync) | data_id=20 both | **0% PASS** | **all PASS** | ~0.8 | per_round_advance + phase_gpu_compute (marginal KS 0.211/0.257 vs 0.20/0.25, means match 17.85≈17.48 / 1.03≈1.13), gpu_budget_real/sim (~40% overrun **symmetric** both modes — mis-applied invariant, not a divergence) | **49/4/21.** K-D25/K-D26 clock-rate+phase rungs VALIDATED PASS (44→49). Cadence CLEAN; 4 surviving fails all benign | | **fwdllm_plus** (sync) | real→wall-cap 1800s (data_id<20); sim→data_id=20 | 0% PASS | **all PASS** | ~0.85 | #7 eligibility/selection_detail/avail_timebase (real 4.9/4.87 vs sim 9.6 eligible/chosen; in_flight 10=10), gpu_budget (symmetric ~40%), total_commits/terminal (13 vs 14 off-by-one, stop-mismatch) | **45/7/21.** Clock-rate rungs VALIDATED PASS (40→45). Fails = #7 + gpu_budget + stop-mismatch boundary; real ~4× slow (#7) | -| **fluxtune** (async) | R1 FIXED; sim **STALLS** on #13 (killed early, data_id=1) | **0% (FIXED, K-D27b)** | V2 PASS; V1/g2 marginal | **0.06 ⛔** | **#13 drain stall** (30s/cycle, pipeline starvation — see "#13 DESIGN") + throughput/overhead/commits/staleness (all #13-confounded), gpu_budget | **R1 FIXED** (`SIM_R1_DISPATCH` 238→0, `_012922`). #13 drain redesign is NEXT (port felix `_sim_recv_min`). Can't bank a valid pair until the sim runs at speed | +| **fluxtune** (async) | R1 FIXED; sim **STALLS** on #13 (killed early, data_id=1) | **0% (FIXED, K-D27b)** | V2 PASS; V1/g2 marginal | **0.29 ⛔** (was 0.06; steps 1-3 recovered ~5×) | **#13 drain FIXED** (steps 1-3); residual `sim_rate<1` is #12c delay-headroom-bound + inherent strict-order straggler holds (step-4 staggering neutral, OFF) + throughput/overhead/commits/staleness (#13/#12c-confounded), gpu_budget | **R1 FIXED** (`SIM_R1_DISPATCH` 238→0). #13 **steps 1-3 landed+validated (K-D28/b/c)**; NEXT a `--delay-factor` run before banking a pair | ### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) | baseline | penultimate (sign-off `_1721..1819`) | **LATEST (`_2224..2258`, data_id=20)** | Δpass | key failing rungs | @@ -111,16 +126,23 @@ fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 mi | **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27+K-D27b):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). **K-D27b:** the bind alone left R1 at 67.6% (`_005222`) — the reconcile subtracted the STALE `_sim_committed`, dropping re-dispatched-after-commit trainers; fixed to `outstanding = inflight ∪ buffer`. Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 887-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | | **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | | **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | -| **#13** ⭐⭐ | **sim physically slow (`sim_rate` 0.06–0.29 fluxtune) — ROOT-CAUSED (2026-07-05): fwdllm's async drain `_sim_recv_min_grad` is an EARLY port that lacks ALL of felix's straggler-handling refinements, so a stuck/idle in-flight trainer blocks every commit for the full 30s `RECV_TIMEOUT_WAIT_S`, forever.** Exposed once #1c/R1 stopped the (incorrect) re-dispatch churn that was masking it. | fluxtune (all) | **Port felix's drain (design locked below — "#13 DESIGN").** The `_sim_recv_min` reference in `asyncfl/top_aggregator.py` IS the intended event-driven model; fwdllm has the skeleton but not the refinements. | +| **#13** ⭐⭐ | **sim physically slow (`sim_rate` 0.06–0.29 fluxtune) — ROOT-CAUSED (2026-07-05): fwdllm's async drain `_sim_recv_min_grad` is an EARLY port that lacks ALL of felix's straggler-handling refinements, so a stuck/idle in-flight trainer blocks every commit for the full 30s `RECV_TIMEOUT_WAIT_S`, forever.** Exposed once #1c/R1 stopped the (incorrect) re-dispatch churn that was masking it. | fluxtune (all) | **Drain fixed — steps 1-3 LANDED+VALIDATED (K-D28/b/c): `sim_rate` 0.06→0.29, blocks≥5s 54→8, stall broken, R1=0.** Step 4 staggering INVESTIGATED-NEUTRAL → OFF (K-D28d/§H): residual 11-12s holds are inherent strict-sct-order straggler waits. **NEXT: `--delay-factor` run (#12c)** — the `sim_rate<1` residual is delay-headroom-bound, not drain. | | **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only 4.9 eligible vs sim 9.6 | fwdllm_plus | Real liveness FIXED (Stage C — completes 10). Remaining: profile the per-iteration reselection + oracular read cost from the banked per-phase log; explain the real/sim eligible-count gap at 100% avail. Not a sim bug. | | **#12c** | `--delays on` uses the hardcoded `training_delay_factor=10` ÷10 shrink unless `--delay-factor` passed | all | Optional: pass `--delay-factor` for the full 4–18s registry delay. Config-flow otherwise correct (K-D24/K-D21). | | **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits, real-only; one-grad-per-poll drain tail) | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run to validate the MQTT settle timing (principle #8/#11c). Never change grad values/cadence. | ### Open roots — ranked lowest-rung-first -1. **#13 slow sim / drain stall (fluxtune)** — ROOT-CAUSED + design LOCKED (see "#13 DESIGN"): fwdllm's - `_sim_recv_min_grad` lacks felix's straggler refinements → a stuck in-flight trainer blocks every commit the - full 30s → starvation. NEXT to implement (port felix's `_sim_recv_min`). Now the top blocker. -2. **#1c R1 residence (fluxtune)** — FIXED (K-D27/b): `SIM_R1_DISPATCH` 238→0. The pending-commit bridge + +1. **#12c `--delay-factor` measurement (fluxtune) — NEW TOP ACTIONABLE.** #13 drain is FIXED (steps 1-3); the + residual `sim_rate<1` (0.29) is dominated by the missing vclock delay-headroom, NOT the drain (`training_delay_ + factor=None` → the hardcoded ÷10 → D≈0.4-1.8s, so each commit charges ~only GPU to the vclock while wall pays + GPU + the inherent strict-order holds). **NEXT:** re-run fluxtune sim with `--delay-factor` (sct = send + gpu + + D, D≈4-18s) so the vclock jumps by gpu+D while wall pays only the skipped GPU → expect `sim_rate` toward/past 1. + Only after this is the drain proven end-to-end; then bank the first valid real↔sim pair. +2. **#13 slow sim / drain stall (fluxtune) — FIXED (K-D28/b/c).** stuck-evict + recv-grace 2→5s + drain_ready + direct ingest → `sim_rate` 0.06→0.29, blocks≥5s 54→8, 30s stall gone, R1=0. Step 4 freed-slot staggering was + INVESTIGATED-NEUTRAL → DISABLED (§H; the residual 11-12s holds are inherent strict-sct-order straggler waits, + not a dispatch-timing artifact). +3. **#1c R1 residence (fluxtune)** — FIXED (K-D27/b): `SIM_R1_DISPATCH` 238→0. The pending-commit bridge + dropping the stale `_sim_committed` subtraction. (Was masking #13; fixing it exposed the drain stall.) 3. **#7 fwdllm_plus real speed / selection divergence** — real completes now; explain the 4× slowness and the syn_0 eligible-count gap from banked telemetry. @@ -158,18 +180,38 @@ NORMAL** (happen in real too — most cycles var isn't met, discarded until fina the fix is purely drain timing. (4) sim-only, real-mode byte-identical. **Plan (incremental, drain is the K-D16/K-D17 deadlock zone — one step per short-sim validation):** -1. **Stuck-end eviction** (felix `_sim_recv_min:436-442`): on the deadline, `pop(_stuck_end)` from - `_sim_inflight_expected` so a lost/idle straggler can't block every future cycle. *Suspected to break the 30s - stall alone.* Validate with a short fluxtune sim + the residence pytests. -2. **Probe-ceiling + ready-gating** (felix `:399-411`, `_sim_end_has_ready_msg`): only probe an in-flight end if - its rxq is physically ready OR its `exp ≤ buffered_min + slack` — stop burning 2s grace on far-future or - not-arrived stragglers. -3. **Direct `drain_ready` ingest** (felix `_sim_sct_ordered_drain`, `:375-390`): drain each live end's rxq - directly (no recv_fifo streamer) so a delivered-but-stranded update is a COMPLETE buffer snapshot (the streamer - can strand a delivered grad out of the buffer's view → the gate waits on a grad it already has). -4. **Freed-slot refill stamp** (felix `_sim_free_slot_ts`, `:507-520`): stamp each train-commit's freed vclock and - refill the slot's next dispatch at THAT vclock, so re-selection is prompt + correctly timed. - Keep the fwdllm-specific grad orchestration (one-grad-per-call loop, agg-goal-boundary carry, K-D12/K-D17b). +1. **Stuck-end eviction** (felix `_sim_recv_min:436-442`) — **LANDED (K-D28), re-run pending.** On the deadline + the drain now tracks `_stuck_end` and `pop`s it from `_sim_inflight_expected` (+`_sim_gate_failsafe` counter + + `[SIM_GRAD_STUCK_EVICT]`) so a lost/idle straggler can't re-fire the full 30s every future cycle. Also bumped + the fwdllm-scoped recv-grace floor 2→**5s** (`SIM_RECV_GRACE_FLOOR_S`, fwdllm class only; async_cifar10 stays + 2.0) — fwdllm's forward-grad GPU can take ~4s so a 2s window closes before the grad reassembles (TUNABLE: pull + as low as correctness allows). *Suspected to break the 30s stall alone.* **NEXT: short fluxtune sim to confirm + `aggregate` wall 30s→seconds.** `TestStuckEndEviction` + full `tests/` green. +2. **Probe-ceiling + ready-gating** (felix `:399-411`, `_sim_end_has_ready_msg`) — **LANDED (K-D28b), + re-run pending.** The drain now probes a non-recv_end in-flight end only if its rxq is physically ready OR + `exp ≤ buffered_min + slack`, instead of blocking the full 5s grace on every in-flight end each pass. Targets + the ~42 gaps of 4–10s that dominate the post-step-1 wall (`sim_rate` 0.20). Safe vs the HOLD gate (any end that + could trigger `earlier_stuck` satisfies `exp ≤ bmin + slack` → always probed). `TestProbeCeilingReadyGating`. +3. **Direct `drain_ready` ingest** (felix `_sim_sct_ordered_drain`, `:375-390`) — **LANDED (K-D28c), re-run + pending.** Flag-gated (`sim_sct_ordered_drain: true` in the fluxtune sim yaml): the drain sweeps each live + in-flight end's rxq directly via `channel.drain_ready` (non-blocking, returns on FIRST arrival) instead of the + recv_fifo streamer that BLOCKS the full 5s grace PER not-ready end. **The `run_..._030300` trace proved this is + the dominant residual cost** — a `[RECV_FIFO] timeout (5.0s)` fired on every not-ready recv_end each pass. Also + yields a COMPLETE buffer snapshot (the streamer can strand a delivered grad out of the buffer's view → + past-dated commit). `TestSctOrderedDrain`. +4. **Freed-slot refill stamp** (felix `_sim_free_slot_ts`, `:507-520`) — **IMPLEMENTED but INVESTIGATED-NEUTRAL, + DISABLED (K-D28d, `run_..._034550`).** Code landed + flag-gated (`sim_staggered_redispatch`, default OFF) + + `TestFreedSlotRefill`, but it did NOT help: `sim_rate` 0.289→0.274, the 11-12s holds persisted. **Root-cause of + the residual holds (from the step-3 disk telemetry, `sct`/`T_v` around every hold):** the drain is correctly + waiting (strict sct order) for the EARLIEST-sct in-flight straggler while ~7 higher-sct grads sit buffered + (each hold commits a grad whose sct is 0.1-0.4s below the buffered min). Both wall (the straggler's real + arrival) and vclock (clock-jump clamp pins it to the straggler's `min_future+slack`) bottleneck on that one + trainer. This is INHERENT (strict-sct-order + real-GPU + one-in-flight trickle dispatch), not a dispatch- + bunching artifact — so staggering can't fix it, and feeding an OLD freed-slot vclock as `send_ts` makes the + gate expect trainers even earlier → more holds. **The real `sim_rate` lever is #12c** (D≈0.4-1.8s here → almost + no vclock headroom to charge; `--delay-factor` gives `sct = send + gpu + D` so the vclock jumps by gpu+D while + wall pays only the skipped GPU). Steps 1-3 captured the fixable artifacts; the residual is #12c-bound + real + GPU. See §H dead-ends. **Expected result:** `aggregate` wall per cycle 30s→~GPU-compute (seconds); slots stay full; `data_id` progresses at ~real speed; `sim_rate`→≥1 once `--delay-factor` gives the vclock delay-headroom (#12c). R1 stays 0 (residence @@ -371,6 +413,14 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso --- ## §H Dead-ends & corrections — do NOT retry +- **"#13 step 4 (freed-slot staggered re-dispatch) closes the residual 11-12s drain holds."** NEUTRAL/REFUTED + (K-D28d, `run_..._034550`): `sim_rate` 0.289→0.274, holds persisted. The holds are the drain correctly waiting + (strict sct order) for the earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU + + strict-order + trickle dispatch, not a dispatch-bunching artifact. Staggering feeds an OLD freed-slot vclock + as `send_ts` → gate expects trainers even earlier → more holds. *Lesson:* the fwdllm `sim_rate<1` residual after + steps 1-3 is dominated by **#12c** (no `--delay-factor` → ~no vclock headroom to charge), NOT the drain; measure + with `--delay-factor` before any further drain work. Code kept flag-gated OFF; don't re-enable without a reworked + send_ts model (stamp = actual dispatch vclock, not the old freed-slot vclock). - **"K-D19 fixed fluxtune R1 / the NONE reset is a red herring."** REFUTED (K-D26). R1 stayed 60.4%. K-D19 fixed the wrong release path (the aggregator's RETURN-path `cleanup_provided_ends`) and dismissed K-D17b's NONE hypothesis by checking the *selection filter* (async_oort:1586, which excludes `all_selected` members) — @@ -516,3 +566,63 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso inflight ∪ buffer` (drop `− _sim_committed`; a this-cycle commit is already popped from both sets, so the subtraction was redundant and harmful). `test_recommitted_trainer_stays_pending_despite_stale_committed`; full `tests/` 887-green. **Awaiting the next fluxtune sim re-run to bank the recovery.** +- **K-D28** — #13 drain-stall step 1 (of the 4-step felix port): fwdllm's `_sim_recv_min_grad` hit the 30s + `RECV_TIMEOUT_WAIT_S` deadline and just `break`, leaving the earliest-expected straggler in + `_sim_inflight_expected` forever → `earlier_stuck` re-fired the full 30s every drain cycle → composer froze → + pipeline starvation (`sim_rate` 0.06). Fix (felix `_sim_recv_min:436-442`): on the deadline track `_stuck_end` + and `pop` it from the expected set (+`_sim_gate_failsafe` counter + `[SIM_GRAD_STUCK_EVICT]`), then commit the + buffered min. Paired knob: fwdllm-scoped `SIM_RECV_GRACE_FLOOR_S` 2→**5s** (override on the fwdllm agg class + only; async_cifar10's base stays 2.0) — the forward-grad GPU can take ~4s so a 2s per-pass recv window closes + before the grad reassembles. **TUNABLE: keep as LOW as correctness allows** (re-measure real GPU wall, pull + toward observed-max + slack). Sim-only path (real never enters the failsafe). `TestStuckEndEviction`; full + `tests/` green. **VALIDATED (`run_..._024637`): stuck-evict fired exactly ONCE (not spinning), R1 stayed 0, + `sim_rate` 0.06→0.20 (~3×), data_id 1→3, pipeline 8-9 deep.** The residual wall is ~42 gaps of 4-10s where the + drain BLOCKS waiting for freshly-dispatched GPU while the vclock idles → steps 2-4. +- **K-D28b** — #13 step 2: probe-ceiling + ready-gating (felix `_sim_recv_min:399-411`). Post-step-1 the drain + still blocked the full 5s grace every pass on every in-flight end, incl. far-future/not-arrived stragglers (the + 4-10s inter-burst gaps). Fix: probe a non-recv_end in-flight end only if physically ready + (`_sim_end_has_ready_msg`, inherited from asyncfl) OR `exp ≤ buffered_min + slack`. Provably safe vs the HOLD + gate (an end triggering `earlier_stuck` at `exp < bmin - slack` always satisfies `exp ≤ bmin + slack`, so it is + never skipped). Sim-only. `TestProbeCeilingReadyGating`. **VALIDATED (`run_..._030300`): no regression (evict=1, + R1=0), `sim_rate` 0.20→0.23, data_id 3→5.** But the trace showed the dominant residual is recv_fifo's per-end 5s + timeout on freshly-dispatched (recv_end) trainers — which step 2 can't touch (recv_ends are always probed). +- **K-D28c** — #13 step 3: direct `drain_ready` ingest (felix `_sim_recv_min:375-390`), flag-gated on + `sim_sct_ordered_drain` (ON in the fluxtune sim yaml). The `run_..._030300` trace root-caused the residual 4-10s + gaps to `recv_fifo`'s streamer BLOCKING the full 5s grace PER not-ready end (`[RECV_FIFO] timeout (5.0s)` fired + on the 2 not-ready ends of every 3-end probe while 1 arrived at ~4.4s GPU). Fix: when on, the fwdllm drain + ingests via `channel.drain_ready(live_inflight, timeout=grace)` — a non-blocking rxq sweep that returns on the + FIRST arrival (no per-end timeout) and is a COMPLETE buffer snapshot (the streamer's background task can strand a + delivered grad → past-dated commit). No ready-gating on this path (non-blocking). Flag-off keeps the step-2 + recv_fifo path; async_cifar10 untouched (fwdllm-only method). Sim-only. `TestSctOrderedDrain`; full + fwdllm+async+parity suites green. **PARTIALLY VALIDATED (`run_..._032235`, killed early during startup): the + drain now CAPS every block at the 5s grace — max aggregate block 34.5s→5.0s, 12-34s recv_fifo blocks GONE, 0 + `[RECV_FIFO] timeout` / streamer lines. Active-loop gaps ~4.3s (pure GPU) once grads flow.** The run only + reached data_id=0 because ~2min was trainer cold-start (267MB distilbert fetch/load over MQTT) — a one-time + startup cost, not the drain; `sim_rate` 0.058 is that unamortized first emit. **VALIDATED (`run_..._032729`, + 6.5min): `sim_rate` 0.23→0.286 (climbing), mean gap 3.19→2.47s, blocks≥5s 54→8, data_id→4, R1=0.** Remaining + wall = 1 stuck-evict (36s, expected) + ~4 multi-pass gate-holds (11-12s) waiting for an earlier-expected + straggler → the **step-4 (freed-slot refill) target**. +- **K-D28d** — #13 step 4: freed-slot refill stamp (felix `_sim_recv_min:513-520` + `_distribute_weights` + staggered path). After step 3 the drain still HELD across 2-3 grace windows (the 11-12s blocks) for an + earlier-expected straggler, because fwdllm stamped every cohort's `sim_send_ts` at one frozen `_round_now`, so + `_sim_inflight_expected` bunched all expected completions together and the min-sct gate waited on the whole + batch. Fix: each grad commit appends the just-advanced vclock to `_sim_free_slot_ts` (inherited deque); + `_distribute_weights_async` pops it (`_pop_free_slot_ts`, FIFO + clamped ≤ live vclock) as each re-dispatched + end's per-end `sim_send_ts` (and `_sim_inflight_expected[end] = sst + budget`), rebuilding a shallow per-end + payload copy carrying its own `SIM_SEND_TS`. Spreads expected completions across the timeline (matching real's + staggered returns) so the gate stops holding for same-expected batches. Composes WITH sct_ordered_drain for + fwdllm (both flags ON) — felix's "supersedes, don't combine" note is async_cifar10-specific; fwdllm's c≫agg_goal + one-grad-per-call loop needs both, and the mechanisms are orthogonal (drain = ingest primitive; staggered = + dispatch-timing, only changing the values the gate reads). fwdllm overrides dispatch (`_distribute_weights_async`) + + drain (`_sim_recv_min_grad`), so felix's async_cifar10 paths are untouched. Flag-off = Batch-1 byte-identical; + real never stamps (`_round_now` None-guarded). `TestFreedSlotRefill`; full fwdllm+async+parity suites green. + **OUTCOME (`run_..._034550`): NEUTRAL — flag DISABLED.** `sim_rate` 0.289→0.274; the 11-12s holds persisted. + The premise was WRONG: disk telemetry (step-3 run, `sct`/`T_v` at every hold) shows each hold commits a grad + whose sct is 0.1-0.4s below the buffered min with ~7 higher-sct grads buffered throughout — the drain is + correctly holding for the earliest-sct in-flight STRAGGLER (strict order), not a same-expected cohort, and the + clock-jump clamp pins the vclock to that straggler's `min_future+slack`. Both wall + vclock bottleneck on the + slowest in-flight trainer: INHERENT to real-GPU + strict-order + trickle dispatch (residence frees 1-2 + slots/commit). Staggering makes it worse (old freed-slot vclock → gate expects trainers even earlier). Code + + test kept, flag OFF. The `sim_rate<1` residual after steps 1-3 is #12c-bound (no `--delay-factor`); measure with + `--delay-factor` before any further drain work. See §H. A reworked attempt would stamp the ACTUAL dispatch + vclock (not the old freed-slot one). diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 74c534788..2396bda63 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -249,6 +249,19 @@ class TopAggregator(AsyncTopAgg): # Override with an explicit `sim_wall_ceiling_s` for a tighter outer bound. SIM_WALL_CEILING_FACTOR = 20.0 + # #13 recv-grace floor (fwdllm-scoped override of SyncTopAgg's 2.0). This is + # the per-pass recv_fifo window the drain waits for an in-flight grad to + # physically arrive. async_cifar10's 2.0s suffices there (weights come back + # fast); fwdllm's forward-grad "train" is a REAL GPU pass that can take ~4s, + # so a 2s window can close before the grad reassembles and force needless + # extra gate passes. 5s covers the slow GPU here. + # TUNABLE: keep this as LOW as possible without compromising correctness -- + # too low and a genuinely-in-flight grad is missed within the pass; too high + # and every stuck-end failsafe pass pays the full window. Re-measure the + # real per-trainer GPU wall (fluxtune telemetry) and pull it down toward the + # observed max compute + a small slack once the drain is validated. + SIM_RECV_GRACE_FLOOR_S = 5.0 + def internal_init(self) -> None: """Initialize internal state for role.""" super().internal_init() @@ -753,33 +766,74 @@ def _sim_recv_min_grad(self, channel, recv_ends): deadline = time.time() + RECV_TIMEOUT_WAIT_S for _pass in range(_SIM_GATE_MAX_PASSES): grace = self._sim_recv_grace_s() - # Probe live recv ends UNION the tracked in-flight set (an end that - # entered RECV after the snapshot is still drained), minus anything + # Base probe: the live recv_ends (always drained), minus anything # already buffered or committed this cycle. - to_probe = [ - e for e in set(live) | set(self._sim_inflight_expected) - if channel.has(e) - and not self._sim_buffer.has(e) - and e not in self._sim_committed + _base = [ + e for e in live + if not self._sim_buffer.has(e) and e not in self._sim_committed ] - if to_probe: - for m, md in channel.recv_fifo( - to_probe, first_k=len(to_probe), timeout=grace - ): - if m is None: # no more ready (grace expired or set drained) - break - _e = md[0] - _s = m.get(MessageType.SIM_COMPLETION_TS) - _s = float(_s) if _s is not None else self._vclock.now - self._sim_buffer.add(_e, _s, (m, md)) + _seen = set(_base) + if getattr(self, "_sim_sct_ordered_drain", False): + # #13 step 3 -- direct sct-ordered ingest (felix _sim_recv_min:375-390). + # Drain each live in-flight end's rx queue DIRECTLY (no recv_fifo + # streamer), taking the FULL live in-flight set. Two wins: (1) the + # buffer is a COMPLETE snapshot of every arrived grad -- the + # streamer's background task + shared queue can strand a delivered + # grad out of the buffer's view (is_rxq_empty true) and let the clock + # lap it (past-dated commit); (2) drain_ready sweeps all ready rxqs + # non-blocking and returns on the FIRST arrival, so it does NOT burn + # the full grace PER not-ready end -- the recv_fifo per-end timeout + # that dominated the post-step-2 wall (the 4-10s gaps, #13). No + # ready-gating needed here (non-blocking sweep + poll-to-first). + to_probe = _base + [ + e for e in self._sim_inflight_expected + if e not in _seen and channel.has(e) + and not self._sim_buffer.has(e) and e not in self._sim_committed + ] + if to_probe: + for m, md in channel.drain_ready(to_probe, timeout=grace): + _e = md[0] + _s = m.get(MessageType.SIM_COMPLETION_TS) + _s = float(_s) if _s is not None else self._vclock.now + self._sim_buffer.add(_e, _s, (m, md)) + else: + # #13 step 2 -- probe-ceiling + ready-gating (felix _sim_recv_min:399-411). + # ALSO probe an in-flight-expected end that is NOT already a recv_end only + # if it is physically READY (its grad arrived) OR its modeled completion + # `exp` is at/before the buffered minimum (+slack). Without this the drain + # blocked the FULL grace window every pass on far-future / not-yet-arrived + # stragglers -- the dominant inter-burst wall waste (#13, the 4-10s gaps). + # Safe against the HOLD gate below: any end that could trigger + # `earlier_stuck` (exp < bmin - slack) also satisfies exp <= bmin + slack, + # so the gate's stuck end is always in this probe set -- no new deadlock. + _bmin = self._sim_buffer.peek_min_ts() + _probe_ceiling = ( + _bmin + _SIM_ORDER_SLACK_S if _bmin is not None else float("inf") + ) + to_probe = _base + [ + e for e, exp in self._sim_inflight_expected.items() + if e not in _seen and channel.has(e) + and not self._sim_buffer.has(e) and e not in self._sim_committed + and (self._sim_end_has_ready_msg(channel, e) or exp <= _probe_ceiling) + ] + if to_probe: + for m, md in channel.recv_fifo( + to_probe, first_k=len(to_probe), timeout=grace + ): + if m is None: # no more ready (grace expired or set drained) + break + _e = md[0] + _s = m.get(MessageType.SIM_COMPLETION_TS) + _s = float(_s) if _s is not None else self._vclock.now + self._sim_buffer.add(_e, _s, (m, md)) # Gate: earliest expected completion among un-buffered in-flight ends. bmin = self._sim_buffer.peek_min_ts() - min_stuck = None + _stuck_end, min_stuck = None, None for e, exp in self._sim_inflight_expected.items(): if self._sim_buffer.has(e) or e in self._sim_committed: continue if min_stuck is None or exp < min_stuck: - min_stuck = exp + min_stuck, _stuck_end = exp, e earlier_stuck = ( bmin is not None and min_stuck is not None and min_stuck + _SIM_ORDER_SLACK_S < bmin @@ -789,6 +843,21 @@ def _sim_recv_min_grad(self, channel, recv_ends): if not earlier_stuck: break # the buffered minimum is the true next completion if time.time() >= deadline: + # #13 step 1 (felix _sim_recv_min:436-442): the earliest-expected + # in-flight trainer never physically arrived within the failsafe + # window. WITHOUT this eviction it stays in _sim_inflight_expected + # forever, so `earlier_stuck` re-fires the FULL deadline on every + # future drain cycle -> the composer freezes 30s/commit -> + # pipeline starvation. Treat the straggler as lost: drop it from + # the expected set so it can't block future commits, and commit + # the buffered min now. Sim-only path (real never enters here). + self._sim_gate_failsafe = getattr(self, "_sim_gate_failsafe", 0) + 1 + self._sim_inflight_expected.pop(_stuck_end, None) + logger.info( + f"[SIM_GRAD_STUCK_EVICT] round={getattr(self, '_round', -1)} " + f"end={str(_stuck_end)[-4:]} exp={min_stuck} bmin={bmin} " + f"failsafe={self._sim_gate_failsafe}" + ) break # failsafe: a stuck trainer never arrived; commit buffered popped = self._sim_buffer.pop_min() if popped is None: @@ -810,6 +879,18 @@ def _sim_recv_min_grad(self, channel, recv_ends): self._advance_sim_clock(_advance_to) self._sim_committed.add(_end) self._sim_inflight_expected.pop(_end, None) + # #13 step 4 -- freed-slot refill stamp (felix _sim_recv_min:513-520). This + # grad commit frees a compute slot; record the just-advanced vclock so the + # trainer that REFILLS the slot rides THIS vclock (popped FIFO in + # _distribute_weights_async) instead of the round-start frontier. Spreads + # each cohort's expected completions across the timeline (matching real's + # staggered returns) instead of collapsing them at one frozen `_round_now`, + # so the drain gate stops HOLDING for a batch of same-expected stragglers + # (the residual 11-12s multi-pass holds after step 3). fwdllm's grad loop is + # all-train (every commit frees a slot), so no train/eval discriminator is + # needed. Sim-only + flag-gated (real never enters this method). + if getattr(self, "_sim_staggered_redispatch", False): + self._sim_free_slot_ts.append(self._vclock.now) # K-D27 (felix-parallel, asyncfl:618): the grad committed -> the trainer is # no longer in flight in virtual time -> drop it from pending so it is # re-pickable. (_sim_hold_busy_slots reconciles right after, but be explicit.) @@ -2593,14 +2674,18 @@ def _distribute_weights_async( self._update_state_after_payload_prepared() - # Sim-clock dispatch stamp (Batch 1): all ends in this dispatch share the - # same virtual instant, so one _round_now stamp is injected into each - # payload variant. Unlike the sync barrier the async path also arms the - # in-flight gate (_sim_inflight_expected[end] = dispatch vclock + a - # lower-bound budget) so the reorder-buffer drain can't lap a trainer - # whose modeled completion is still in the future. Inert in real mode. + # Sim-clock dispatch stamp (Batch 1): the async path arms the in-flight gate + # (_sim_inflight_expected[end] = send vclock + a lower-bound budget) so the + # reorder-buffer drain can't lap a trainer whose modeled completion is still + # in the future. Inert in real mode. + # #13 step 4 (staggered): when on, each end's SEND vclock is popped from + # the freed-slot FIFO (_pop_free_slot_ts) instead of the shared round + # frontier, so a cohort's expected completions spread across the timeline + # (see the commit-side stamp). Off/real ⇒ one shared _round_now injected + # into the two payload variants (byte-identical to Batch 1). _round_now = self._vclock.now if self.simulated else None - if self.simulated: + _staggered = self.simulated and getattr(self, "_sim_staggered_redispatch", False) + if self.simulated and not _staggered: for _p in (payload_weights, payload_var_bad): if _p is not None: _p[MessageType.SIM_SEND_TS] = _round_now @@ -2645,12 +2730,21 @@ def _distribute_weights_async( f"[SIM_R1_DISPATCH] end={end} re-dispatched while still " f"outstanding (vclock={_round_now}) -- R1 residence violation" ) - channel.set_end_property(end, PROP_SIM_SEND_TS, _round_now) - # Expected completion = dispatch vclock + a lower-bound budget + # #13 step 4: SEND vclock = the freed-slot stamp (staggered) else the + # shared round frontier. _pop_free_slot_ts is FIFO + clamped <= now; + # an empty queue (cold start / no held slot) falls back to _round_now. + _sst = self._pop_free_slot_ts(_round_now) if _staggered else _round_now + channel.set_end_property(end, PROP_SIM_SEND_TS, _sst) + # Expected completion = SEND vclock + a lower-bound budget # (this end's own last-observed TRAINING_BUDGET_S, else the # running min) so the gate never laps a not-yet-committed trainer. _budget = self._sim_trainer_budget.get(end, self._sim_budget_min) - self._sim_inflight_expected[end] = _round_now + _budget + self._sim_inflight_expected[end] = _sst + _budget + # Staggered: this end's payload must carry its OWN SIM_SEND_TS, so + # rebuild a shallow copy (weights shared by ref; small vs GPU cost). + if _staggered and isinstance(payload, dict): + payload = dict(payload) + payload[MessageType.SIM_SEND_TS] = _sst # K-D27 (felix-parallel, asyncfl:1490): the instant a trainer is # dispatched it is in flight in virtual time -> add to the pending- # commit set so the selector's eligibility filter excludes it until diff --git a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py index 61a486d0d..01e655df2 100644 --- a/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py +++ b/lib/python/tests/mode/test_fwdllm_sim_grad_loop.py @@ -13,8 +13,10 @@ commit a grad). """ +from collections import deque from datetime import datetime +from flame.mode.horizontal.asyncfl.top_aggregator import TopAggregator as _AsyncBase from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator from flame.mode.horizontal.syncfl.top_aggregator import TopAggregator as _SyncBase from flame.mode.message import MessageType @@ -58,6 +60,19 @@ def recv_fifo(self, end_ids, first_k=None, timeout=None): self._delivered.add(e) yield self._msgs[e], (e, datetime.now()) + def drain_ready(self, end_ids, timeout=None): + """Non-blocking snapshot: return every arrived-but-undelivered message + for the given ends (mirrors channel.drain_ready's direct rxq sweep).""" + self._probe += 1 + out = [] + for e in end_ids: + if e in self._delivered or e not in self._msgs: + continue + if self._probe > self._release_at.get(e, 0): + self._delivered.add(e) + out.append((self._msgs[e], (e, datetime.now()))) + return out + class _FakeGradAgg: """Binds the real sim grad-loop methods onto a minimal stand-in.""" @@ -66,6 +81,10 @@ class _FakeGradAgg: _release_sim_slots_at_agg_goal = TopAggregator._release_sim_slots_at_agg_goal _advance_sim_clock = _SyncBase._advance_sim_clock _sim_recv_grace_s = _SyncBase._sim_recv_grace_s + # #13 step 2 ready-gating helper (inherited by the real fwdllm agg from asyncfl). + _sim_end_has_ready_msg = staticmethod(TopAggregator._sim_end_has_ready_msg) + # #13 step 4 freed-slot FIFO consumer (inherited from asyncfl). + _pop_free_slot_ts = _AsyncBase._pop_free_slot_ts # fwdllm overrides felix's hold with the Option-A two-lifetime split (K-D16). _sim_hold_busy_slots = TopAggregator._sim_hold_busy_slots # The return-path guard/slot release (K-D19: defers to COMMIT in sim residence). @@ -86,6 +105,8 @@ def __init__(self): self._sim_fill_ema = 0.0 self._sim_pending_commit = set() self._sim_inflight_residence = False + self._sim_staggered_redispatch = False # #13 step 4 (default off) + self._sim_free_slot_ts = deque(maxlen=128) self._trainer_state_dict = {} def _drain(self, channel, recv_ends, n): @@ -158,6 +179,244 @@ def test_holds_commit_for_an_earlier_expected_straggler(self): assert agg._vclock.now == 100.0 +class _RecordingChannel(_FakeGradChannel): + """Records the end sets passed to recv_fifo so a test can assert WHICH + in-flight ends the drain chose to probe (#13 step 2 ready-gating). Optionally + reports a subset of ends as physically READY (non-empty rxq) via _ready.""" + + def __init__(self, ends, ready=()): + super().__init__(ends) + self.probe_calls = [] # recv_fifo end-id lists + self.drain_calls = [] # drain_ready end-id lists + self._ready = set(ready) + + def recv_fifo(self, end_ids, first_k=None, timeout=None): + self.probe_calls.append(list(end_ids)) + yield from super().recv_fifo(end_ids, first_k=first_k, timeout=timeout) + + def drain_ready(self, end_ids, timeout=None): + self.drain_calls.append(list(end_ids)) + return super().drain_ready(end_ids, timeout=timeout) + + # Mirrors channel._ends[e].is_rxq_empty() as read by _sim_end_has_ready_msg. + class _E: + def __init__(self, empty): + self._empty = empty + + def is_rxq_empty(self): + return self._empty + + @property + def _ends(self): + return {e: _RecordingChannel._E(e not in self._ready) for e in self._ends_set} + + +class TestProbeCeilingReadyGating: + """#13 step 2 (felix _sim_recv_min:399-411): an in-flight end that is NOT a + recv_end is probed ONLY if it is physically ready OR its modeled `exp` is at/ + before the buffered minimum (+slack) -- so the drain stops burning the full + grace window on far-future / not-yet-arrived stragglers each pass.""" + + def test_far_future_straggler_is_not_probed(self): + agg = _FakeGradAgg() + # FAR is expected far past the buffered min and has no ready message; the + # buffered A (sct=10) is the true next completion. + agg._sim_inflight_expected = {"FAR": 1000.0} + ch = _RecordingChannel([]) + ch._ends_set.add("FAR") # in-flight but no message queued + agg._sim_buffer.add("A", 10.0, ({MessageType.SIM_COMPLETION_TS: 10.0}, ("A", datetime.now()))) + + msg, _ = agg._sim_recv_min_grad(ch, []) # A not a recv_end (already buffered) + assert msg[MessageType.SIM_COMPLETION_TS] == 10.0 + # FAR was never handed to recv_fifo -> no grace burned waiting on it. + assert all("FAR" not in call for call in ch.probe_calls) + + def test_near_expected_straggler_is_probed(self): + agg = _FakeGradAgg() + # NEAR's exp (11) is within bmin(10)+slack -> the gate may still wait for + # it, so it MUST be probed. + agg._sim_inflight_expected = {"NEAR": 11.0} + ch = _RecordingChannel([]) + ch.add_msg("NEAR", sct=11.0, release_at=0) + agg._sim_buffer.add("A", 10.0, ({MessageType.SIM_COMPLETION_TS: 10.0}, ("A", datetime.now()))) + + agg._sim_recv_min_grad(ch, []) + assert any("NEAR" in call for call in ch.probe_calls) + + def test_physically_ready_straggler_is_probed_regardless_of_exp(self): + agg = _FakeGradAgg() + # READY is expected far in the future BUT its grad has physically arrived + # (ready rxq) -> drain it now so it buffers as a future rather than being + # committed past-dated later. + agg._sim_inflight_expected = {"READY": 1000.0} + ch = _RecordingChannel([], ready={"READY"}) + ch.add_msg("READY", sct=1000.0, release_at=0) + agg._sim_buffer.add("A", 10.0, ({MessageType.SIM_COMPLETION_TS: 10.0}, ("A", datetime.now()))) + + agg._sim_recv_min_grad(ch, []) + assert any("READY" in call for call in ch.probe_calls) + + +class TestSctOrderedDrain: + """#13 step 3: with sim_sct_ordered_drain ON, the drain ingests via + channel.drain_ready (direct rxq sweep, no per-end grace timeout) instead of + the blocking recv_fifo streamer, while keeping sct-ordered commit.""" + + def test_commits_via_drain_ready_not_recv_fifo(self): + agg = _FakeGradAgg() + agg._sim_sct_ordered_drain = True + ch = _RecordingChannel([]) + ch.add_msg("A", sct=30.0) + ch.add_msg("B", sct=10.0) + + scts = agg._drain(ch, ["A", "B"], 2) + assert scts == [10.0, 30.0] # still committed in sct order + assert ch.drain_calls # drain_ready was used... + assert not ch.probe_calls # ...and recv_fifo was NOT + + def test_recv_fifo_path_when_flag_off(self): + agg = _FakeGradAgg() # flag defaults off + ch = _RecordingChannel([]) + ch.add_msg("A", sct=10.0) + + agg._drain(ch, ["A"], 1) + assert ch.probe_calls # recv_fifo used + assert not ch.drain_calls # drain_ready NOT used + + def test_drain_ready_still_honors_inflight_gate(self): + """A (sct=100) arrives; B (sct=50) is expected earlier and arrives on the + 2nd sweep -- the gate must still hold A and commit B first.""" + agg = _FakeGradAgg() + agg._sim_sct_ordered_drain = True + agg._sim_inflight_expected = {"A": 98.0, "B": 48.0} + ch = _RecordingChannel([]) + ch.add_msg("A", sct=100.0, release_at=0) + ch.add_msg("B", sct=50.0, release_at=1) + + first, _ = agg._sim_recv_min_grad(ch, ["A", "B"]) + assert first[MessageType.SIM_COMPLETION_TS] == 50.0 # B, not A + second, _ = agg._sim_recv_min_grad(ch, ["A", "B"]) + assert second[MessageType.SIM_COMPLETION_TS] == 100.0 + + +class TestFreedSlotRefill: + """#13 step 4: a grad commit stamps the freed-slot vclock into the FIFO; a + later dispatch pops it (oldest-first, clamped) as the re-dispatched trainer's + SEND vclock -- spreading expected completions instead of collapsing the cohort + at one round frontier (the residual multi-pass gate-holds after step 3).""" + + def test_commit_stamps_freed_slot_vclock_when_staggered(self): + agg = _FakeGradAgg() + agg._sim_staggered_redispatch = True + agg._sim_inflight_expected = {"X": 10.0} + ch = _FakeGradChannel([]) + ch.add_msg("X", sct=10.0) + + agg._sim_recv_min_grad(ch, ["X"]) + # vclock advanced to sct=10 on commit -> that freed-slot vclock is stamped. + assert list(agg._sim_free_slot_ts) == [10.0] + + def test_commit_does_not_stamp_when_flag_off(self): + agg = _FakeGradAgg() # staggered off (default) + agg._sim_inflight_expected = {"X": 10.0} + ch = _FakeGradChannel([]) + ch.add_msg("X", sct=10.0) + + agg._sim_recv_min_grad(ch, ["X"]) + assert len(agg._sim_free_slot_ts) == 0 + + def test_pop_free_slot_ts_is_fifo_and_clamped(self): + agg = _FakeGradAgg() + agg._sim_free_slot_ts.extend([5.0, 8.0]) + assert agg._pop_free_slot_ts(100.0) == 5.0 # oldest first + assert agg._pop_free_slot_ts(100.0) == 8.0 + assert agg._pop_free_slot_ts(100.0) == 100.0 # empty -> live frontier + agg._sim_free_slot_ts.append(50.0) + assert agg._pop_free_slot_ts(30.0) == 30.0 # min(stamp, round_now) + + def test_staggered_expected_completions_spread_not_bunched(self): + """End-to-end at the unit level: two commits free slots at vclock 10 and + 25; a subsequent 2-trainer refill pops those as SEND vclocks, so expected + completions (sst + budget) SPREAD (14, 29) instead of bunching at one + round frontier (which would give the same expected for both).""" + agg = _FakeGradAgg() + agg._sim_staggered_redispatch = True + agg._sim_budget_min = 4.0 + for e, s in [("A", 10.0), ("B", 25.0)]: + agg._sim_inflight_expected = {e: s} + ch = _FakeGradChannel([]) + ch.add_msg(e, sct=s) + agg._sim_recv_min_grad(ch, [e]) + assert list(agg._sim_free_slot_ts) == [10.0, 25.0] + + sst1 = agg._pop_free_slot_ts(100.0) + sst2 = agg._pop_free_slot_ts(100.0) + exp1 = sst1 + agg._sim_budget_min + exp2 = sst2 + agg._sim_budget_min + assert (exp1, exp2) == (14.0, 29.0) # spread, not (round_now+budget)×2 + + +class TestStuckEndEviction: + """#13 step 1 (felix _sim_recv_min:436-442): a trainer that is EXPECTED to + complete earlier than the buffered minimum but never physically arrives must + be evicted from _sim_inflight_expected on the recv failsafe deadline -- else + `earlier_stuck` re-fires the full 30s deadline on every future drain cycle + and the composer freezes (pipeline starvation, the #13 drain stall).""" + + def _immediate_deadline(self, monkeypatch): + # Fire the recv failsafe on the first pass (no real 30s wait). + import flame.mode.horizontal.syncfl.fwdllm_aggregator as fa + monkeypatch.setattr(fa, "RECV_TIMEOUT_WAIT_S", 0.0) + + def test_stuck_end_evicted_on_deadline_and_buffered_min_commits(self, monkeypatch): + self._immediate_deadline(monkeypatch) + agg = _FakeGradAgg() + # STUCK is expected early (10) but has no message; A arrived at sct=100. + agg._sim_inflight_expected = {"STUCK": 10.0, "A": 98.0} + ch = _FakeGradChannel([]) + ch.add_msg("A", sct=100.0, release_at=0) + + msg, _md = agg._sim_recv_min_grad(ch, ["A", "STUCK"]) + + # The buffered min commits despite the earlier-expected straggler... + assert msg[MessageType.SIM_COMPLETION_TS] == 100.0 + # ...and STUCK is dropped so it can't block future cycles. + assert "STUCK" not in agg._sim_inflight_expected + assert agg._sim_gate_failsafe == 1 + + def test_evicted_end_does_not_block_the_next_cycle(self, monkeypatch): + self._immediate_deadline(monkeypatch) + agg = _FakeGradAgg() + agg._sim_inflight_expected = {"STUCK": 10.0} + ch = _FakeGradChannel([]) + ch.add_msg("A", sct=100.0, release_at=0) + ch.add_msg("B", sct=200.0, release_at=0) + + # Cycle 1: STUCK holds the gate, hits the failsafe, is evicted; A commits. + m1, _ = agg._sim_recv_min_grad(ch, ["A", "B", "STUCK"]) + assert m1[MessageType.SIM_COMPLETION_TS] == 100.0 + assert "STUCK" not in agg._sim_inflight_expected + + # Cycle 2: with STUCK gone, B commits WITHOUT re-arming the failsafe. + m2, _ = agg._sim_recv_min_grad(ch, ["A", "B", "STUCK"]) + assert m2[MessageType.SIM_COMPLETION_TS] == 200.0 + assert agg._sim_gate_failsafe == 1 # not re-incremented + + def test_no_spurious_eviction_when_buffered_min_is_the_true_next(self, monkeypatch): + # STUCK expected LATER than the buffered min -> not `earlier_stuck` -> + # commit proceeds without touching the failsafe or the expected set. + self._immediate_deadline(monkeypatch) + agg = _FakeGradAgg() + agg._sim_inflight_expected = {"LATE": 500.0} + ch = _FakeGradChannel([]) + ch.add_msg("A", sct=100.0, release_at=0) + + msg, _ = agg._sim_recv_min_grad(ch, ["A", "LATE"]) + assert msg[MessageType.SIM_COMPLETION_TS] == 100.0 + assert agg._sim_inflight_expected == {"LATE": 500.0} # untouched + assert getattr(agg, "_sim_gate_failsafe", 0) == 0 + + class TestRollbackSafety: def test_no_double_commit_within_a_cycle(self): agg = _FakeGradAgg() From 47fa613f50bb074066c2859217bb096cd1b4c14c Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 14:08:19 -0400 Subject: [PATCH 28/56] fwdllm sim: logical-parity checker rungs (P1) + remainder-wait timing model (K-D29) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checkpoint of the logical real<->sim parity work. Root diagnosis: all 3 baselines diverge from ONE cause — sim commit ORDER != real arrival ORDER (fwdllm var is a split-half stat over the commit-ordered grad list -> wrong order -> var flips -> per-trainer RNG desyncs -> grads diverge). Grads are deterministic given matched order, so exact parity is achievable. Checks (P1-1/2/3, P1-6 partial, P1-8): - New enforced `cohort_sequence` rung (EXACT, ungated): per-cycle cohort set + receive-order + cadence + var value must be identical. Keyed on the fwdllm cadence stream (not `round`); runs on all 3 fwdllm baselines. - V2 var_trajectory mean-guard (catches the ~1% systematic offset KS misses). - `--max-bin` window threaded through the engine + run_parity.py + cli.py. - Both rungs now correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus 36/17/21, fluxtune 35/19/19. - Tests: TestCohortSequence x9, TestVarTrajectoryMeanGuard x2. Timing model (K-D29 / P2-1/3/5/6, reverses K-D2's additive model): - Remainder-wait: real sleeps max(0, D-gpu); sct = max(gpu, D). Per-trainer registry delays supply the completion spread -> order deterministic + identical real<->sim. Overrun (gpu>D) flagged (training_overran, [TIMING_OVERRUN]). - crc32 straggler offset disabled in sim yamls; perturbation_count made a knob. - GPU profiled: fwdllm/plus ~1.0s, fluxtune 7.57s (JVP 20 passes) — the fluxtune overrun watch. Trainer duration tests rewritten to the new contract. Suite green: 606 mode/telemetry/selector + 115 async_cifar10 parity. Deferred to next session (recorded in PARITY_LOGICAL_TASKS.md SESSION CHECKPOINT): P1-4 (async_cifar10 cohort adapter), P1-5 (enforce determinism rungs) — both skipped en route to Phase 2; the databin1 validation run (--delay-factor 2 --max-data-id 1); P0-2 grad-determinism confirmation; fluxtune perturbation tuning. Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 150 +++- .../async_cifar10/scripts/parity/cli.py | 9 +- .../async_cifar10/scripts/parity_checks.py | 1 + .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 189 +++++ .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 2 +- .../expt_scripts/fwdllm_n10_smoke_sim.yaml | 2 +- .../fwdllm_plus_n10_smoke_sim.yaml | 2 +- .../fwdllm/expt_scripts/logical_parity.py | 111 +++ .../fwdllm/expt_scripts/run_parity.py | 12 +- lib/python/examples/fwdllm/simulate_fwdllm.md | 801 +++++++----------- lib/python/examples/fwdllm/trainer/fl_main.py | 3 + .../trainer/forward_training/FedSgdTrainer.py | 106 ++- .../tc_transformer_trainer_distribute.py | 26 +- lib/python/examples/fwdllm/trainer/main.py | 3 + .../mode/test_fwdllm_trainer_phase_timing.py | 5 +- .../mode/test_fwdllm_trainer_sim_duration.py | 186 ++-- lib/python/tests/mode/test_parity_checks.py | 100 +++ 17 files changed, 1024 insertions(+), 684 deletions(-) create mode 100644 lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md create mode 100644 lib/python/examples/fwdllm/expt_scripts/logical_parity.py diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index d50670924..98ca4aef6 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -3688,14 +3688,23 @@ def _curve(evs): # cached_v_size). Inputs source: fwdllm_aggregator.py _process_aggregation_goal_met. -def _fwd_cadence_cycles(agg: dict) -> list: +def _fwd_cadence_cycles(agg: dict, max_bin: Optional[int] = None) -> list: """Ordered per-cycle agg_round events carrying fwdllm cadence fields. A cycle is one agg-goal boundary (a variance gate). Non-fwdllm runs (no `var_good_enough`/`cycle_data_id`) yield [] -> the V/DK/G rungs SKIP. + + `max_bin` restricts to cycles whose `cycle_data_id` <= max_bin (the + first-data-bin logical-parity window, simulate_fwdllm.md §A); None = all. + A cycle with no `cycle_data_id` is kept only when unwindowed. """ - return [e for e in agg.get("agg_rounds", []) - if "cycle_data_id" in e or "var_good_enough" in e] + out = [e for e in agg.get("agg_rounds", []) + if "cycle_data_id" in e or "var_good_enough" in e] + if max_bin is not None: + out = [e for e in out + if e.get("cycle_data_id") is not None + and e["cycle_data_id"] <= max_bin] + return out def _iters_per_data_id(cycles: list) -> dict: @@ -3714,7 +3723,8 @@ def _iters_per_data_id(cycles: list) -> dict: def iters_per_data_id_parity(real: dict, sim: dict, ks_tol: float = 0.2, - mean_tol_rel: float = 0.15) -> dict: + mean_tol_rel: float = 0.15, + max_bin: Optional[int] = None) -> dict: """V1 [DIST]: iterations-per-data_id distribution (realized dynamic K). The number of accumulation cycles a data_id needs to pass the variance gate. @@ -3722,7 +3732,7 @@ def iters_per_data_id_parity(real: dict, sim: dict, ks_tol: float = 0.2, the accumulated grad-pool composition — and thus the variance trajectory — differs. KS on the per-data_id iteration counts + a mean guard. """ - rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + rc, sc = _fwd_cadence_cycles(real, max_bin), _fwd_cadence_cycles(sim, max_bin) if not rc or not sc: return {"ok": True, "tier": "DIST", "status": "SKIP", "note": "no fwdllm cadence events (non-fwdllm run or telemetry absent)"} @@ -3750,15 +3760,20 @@ def iters_per_data_id_parity(real: dict, sim: dict, ks_tol: float = 0.2, } -def var_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: +def var_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2, + mean_tol_rel: float = 0.02, + max_bin: Optional[int] = None) -> dict: """V2 [DIST]: per-cycle `var` trajectory distribution. With V1's inputs matched, the variance *signal* itself must match; a divergence with matched iterations points at a grad-pool accumulation-order bug (a true sim bug, not an input divergence). KS over the per-cycle var - values (None dropped — a cycle before the first gate has no var). + values (None dropped — a cycle before the first gate has no var), PLUS a + relative-mean guard: KS alone is blind to a systematic offset (a uniform + ~1% shift barely moves the empirical CDFs → KS≈0), which is exactly the + grad-desync signature (simulate_fwdllm.md §A). The mean guard fails it. """ - rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + rc, sc = _fwd_cadence_cycles(real, max_bin), _fwd_cadence_cycles(sim, max_bin) r_var = [e["var"] for e in rc if e.get("var") is not None] s_var = [e["var"] for e in sc if e.get("var") is not None] if not r_var or not s_var: @@ -3767,11 +3782,15 @@ def var_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: ks = ks_stat(s_var, r_var) r_mean, _ = mean_std(r_var) s_mean, _ = mean_std(s_var) + mean_rel = (abs(r_mean - s_mean) / max(abs(r_mean), abs(s_mean)) + if max(abs(r_mean), abs(s_mean)) > 0 else 0.0) return { - "ok": ks <= ks_tol, + "ok": ks <= ks_tol and mean_rel <= mean_tol_rel, "tier": "DIST", "real_mean_var": round(r_mean, 6), "sim_mean_var": round(s_mean, 6), + "mean_rel_diff": round(mean_rel, 4), + "mean_tol_rel": mean_tol_rel, "ks_stat": round(ks, 3), "ks_tol": ks_tol, "n_real_cycles": len(r_var), @@ -3779,7 +3798,91 @@ def var_trajectory_parity(real: dict, sim: dict, ks_tol: float = 0.2) -> dict: } -def cached_v_pool_parity(real: dict, sim: dict, ks_tol: float = 0.25) -> dict: +def cohort_sequence_parity(real: dict, sim: dict, max_bin: Optional[int] = None, + var_rel_tol: float = 1e-3) -> dict: + """L1 [EXACT]: the ordered per-aggregation logical sequence is IDENTICAL. + + The strongest logical-parity rung (simulate_fwdllm.md §A / PARITY_LOGICAL_ + TASKS.md P1-1). Unlike `aggregation_sequence_parity` (per-`round`, gated to + WARN for stochastic selectors), this keys on the fwdllm variance-cadence + *cycle* stream (`_fwd_cadence_cycles`, ordered) and asserts, cycle-by-cycle: + + - COHORT SET : same trainers committed together (which is fluxtune's bug) + - COHORT ORDER : same receive/commit order — NOT benign; the fwdllm + variance is a split-half statistic over the commit-ORDERED grad list, so + a reshuffle changes `var` even for an identical set + - CADENCE : (cycle_data_id, iteration_per_data_id, agg_goal_count, + var_good_enough, force_commit_planned) identical + - VAR VALUE : per-cycle `var` matches within var_rel_tol (grads are + deterministic GIVEN matched order → var must match; a ~1% gap is the + RNG-desync tell) + + Real receive-order is deterministic in both modes by design, so exact match + is the correct target (NOT gated). Enforced EXACT: any divergence fails. + SKIPs cleanly on non-fwdllm runs (no cadence fields → async_cifar10 etc.). + `max_bin` restricts to the first-data-bin window. + """ + rc = _fwd_cadence_cycles(real, max_bin) + sc = _fwd_cadence_cycles(sim, max_bin) + if not rc or not sc: + return {"ok": True, "tier": "EXACT", "status": "SKIP", + "note": "no fwdllm cadence events (non-fwdllm run or telemetry absent)"} + + def cohort(e): # receive/commit-ordered contributing trainers + return list(e.get("contributing_trainers") or []) + + def cadence(e): + return (e.get("cycle_data_id"), e.get("iteration_per_data_id"), + e.get("agg_goal_count"), e.get("var_good_enough"), + e.get("force_commit_planned")) + + n = min(len(rc), len(sc)) + set_m = order_m = cad_m = var_m = 0 + first_div = None + for i in range(n): + r, s = rc[i], sc[i] + rc_ord, sc_ord = cohort(r), cohort(s) + set_ok = sorted(rc_ord) == sorted(sc_ord) + order_ok = rc_ord == sc_ord + cad_ok = cadence(r) == cadence(s) + rv, sv = r.get("var"), s.get("var") + var_ok = (rv is None and sv is None) or ( + rv is not None and sv is not None + and abs(rv - sv) <= var_rel_tol * max(abs(rv), abs(sv), 1e-9)) + set_m += set_ok; order_m += order_ok; cad_m += cad_ok; var_m += var_ok + if first_div is None and not (set_ok and order_ok and cad_ok and var_ok): + first_div = { + "cycle_index": i, + "real": {"data_id": r.get("cycle_data_id"), + "iter": r.get("iteration_per_data_id"), + "cohort": rc_ord, "var": rv, + "var_good": r.get("var_good_enough")}, + "sim": {"data_id": s.get("cycle_data_id"), + "iter": s.get("iteration_per_data_id"), + "cohort": sc_ord, "var": sv, + "var_good": s.get("var_good_enough")}, + "set_ok": set_ok, "order_ok": order_ok, + "cadence_ok": cad_ok, "var_ok": var_ok, + } + ok = (len(rc) == len(sc) and set_m == n and order_m == n + and cad_m == n and var_m == n) + return { + "ok": ok, + "tier": "EXACT", + "cycles_compared": n, + "n_real_cycles": len(rc), + "n_sim_cycles": len(sc), + "set_match_frac": round(set_m / n, 3) if n else None, + "order_match_frac": round(order_m / n, 3) if n else None, + "cadence_match_frac": round(cad_m / n, 3) if n else None, + "var_match_frac": round(var_m / n, 3) if n else None, + "max_bin": max_bin, + "first_divergence": first_div, + } + + +def cached_v_pool_parity(real: dict, sim: dict, ks_tol: float = 0.25, + max_bin: Optional[int] = None) -> dict: """V3 [DIAG]: `cached_v` (carried aggregated grad-pool) size over time. A rollback/cache bookkeeping divergence looks like a variance bug but is @@ -3787,7 +3890,7 @@ def cached_v_pool_parity(real: dict, sim: dict, ks_tol: float = 0.25) -> dict: variance-FAIL rollbacks and clears on a commit; its size trajectory should match once V1 matches. SKIP if the field was not emitted. """ - rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + rc, sc = _fwd_cadence_cycles(real, max_bin), _fwd_cadence_cycles(sim, max_bin) r_sz = [e["cached_v_size"] for e in rc if e.get("cached_v_size") is not None] s_sz = [e["cached_v_size"] for e in sc if e.get("cached_v_size") is not None] if not r_sz or not s_sz: @@ -3804,7 +3907,8 @@ def cached_v_pool_parity(real: dict, sim: dict, ks_tol: float = 0.25) -> dict: } -def force_commit_rate_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: +def force_commit_rate_parity(real: dict, sim: dict, tol: float = 0.05, + max_bin: Optional[int] = None) -> dict: """V4 [DIST]: force-commit frequency (max_iterations_per_data_id bypass rate). The fraction of cycles that hit the iteration cap and force-commit despite a @@ -3813,7 +3917,7 @@ def force_commit_rate_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: var_threshold / max_iterations_per_data_id are baseline-defining config knobs, NOT parity levers. """ - rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + rc, sc = _fwd_cadence_cycles(real, max_bin), _fwd_cadence_cycles(sim, max_bin) if not rc or not sc: return {"ok": True, "tier": "DIST", "status": "SKIP", "note": "no fwdllm cadence events"} @@ -3831,7 +3935,8 @@ def force_commit_rate_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: } -def variance_pass_ratio_parity(real: dict, sim: dict, tol: float = 0.05) -> dict: +def variance_pass_ratio_parity(real: dict, sim: dict, tol: float = 0.05, + max_bin: Optional[int] = None) -> dict: """V5 [DIST]: genuine variance-pass ratio (the rollup feeding DynamicKC). A *genuine* pass is var<=var_threshold — a force-commit (var>threshold, @@ -3850,7 +3955,7 @@ def _ratio(cycles: list): passes += 1 return (passes / n if n else None), n - rc, sc = _fwd_cadence_cycles(real), _fwd_cadence_cycles(sim) + rc, sc = _fwd_cadence_cycles(real, max_bin), _fwd_cadence_cycles(sim, max_bin) r_ratio, r_n = _ratio(rc) s_ratio, s_n = _ratio(sc) if r_ratio is None or s_ratio is None: @@ -4159,7 +4264,8 @@ def run_all_parity(real_agg: dict, sim_agg: dict, rounds_cap: Optional[int] = None, budget_s: Optional[float] = None, real_ground_truth: Optional[dict] = None, - sim_ground_truth: Optional[dict] = None) -> dict: + sim_ground_truth: Optional[dict] = None, + max_bin: Optional[int] = None) -> dict: """Run the full parity + invariant battery; returns {name: result_dict}. Ordered HIGH → MID → LOW so coarse failures surface first: @@ -4255,11 +4361,12 @@ def run_all_parity(real_agg: dict, sim_agg: dict, # Pure functions over the per-cycle agg_round series; SKIP cleanly on # non-fwdllm runs (no cadence fields emitted). V/G rungs feed off Stage-5 # ordering + Stage-1 clock; DK rungs are inert unless DynamicKC is enabled. - results["v1_iter_per_data_id"] = iters_per_data_id_parity(real_agg, sim_agg) - results["v2_var_trajectory"] = var_trajectory_parity(real_agg, sim_agg) - results["v3_cached_v_pool"] = cached_v_pool_parity(real_agg, sim_agg) - results["v4_force_commit_rate"] = force_commit_rate_parity(real_agg, sim_agg) - results["v5_variance_pass_ratio"] = variance_pass_ratio_parity(real_agg, sim_agg) + results["cohort_sequence"] = cohort_sequence_parity(real_agg, sim_agg, max_bin=max_bin) + results["v1_iter_per_data_id"] = iters_per_data_id_parity(real_agg, sim_agg, max_bin=max_bin) + results["v2_var_trajectory"] = var_trajectory_parity(real_agg, sim_agg, max_bin=max_bin) + results["v3_cached_v_pool"] = cached_v_pool_parity(real_agg, sim_agg, max_bin=max_bin) + results["v4_force_commit_rate"] = force_commit_rate_parity(real_agg, sim_agg, max_bin=max_bin) + results["v5_variance_pass_ratio"] = variance_pass_ratio_parity(real_agg, sim_agg, max_bin=max_bin) results["dk1_agg_goal_trajectory"] = agg_goal_trajectory_parity(real_agg, sim_agg) results["dk2_dynamic_c"] = dynamic_c_trajectory_parity(real_agg, sim_agg) results["dk3_eligible_ends_metric"] = eligible_ends_metric_parity(real_agg, sim_agg) @@ -4368,6 +4475,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "withheld_delivery": {"stage": 6, "role": "DIAG", "deps": ("staleness", "abandon_timeout")}, "commit_promptness": {"stage": 6, "role": "CONTROL", "deps": ("withheld_delivery",)}, "aggregation_sequence": {"stage": 6, "role": "EMERGENT", "deps": ("participation", "inter_arrival_order")}, + "cohort_sequence": {"stage": 6, "role": "EMERGENT", "deps": ("participation", "inter_arrival_order", "r1_inflight_overlap")}, "first_divergence_summary": {"stage": 6, "role": "DIAG", "deps": ()}, # ── Stage 3' FwdLLM async residence (R1/W1, simulate_fwdllm.md §L.3) ── # R1 is the residence INV the fluxtune 2x-recompute bug violated; W1 is the diff --git a/lib/python/examples/async_cifar10/scripts/parity/cli.py b/lib/python/examples/async_cifar10/scripts/parity/cli.py index 7a4b2bbc6..0f78e7a90 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/cli.py +++ b/lib/python/examples/async_cifar10/scripts/parity/cli.py @@ -45,7 +45,8 @@ def _run_pair(real_dir: str, sim_dir: str, agg_goal: int, rounds_cap, budget_s, strict: bool, lenient: bool, json_out, plot_out, - real_label: str = "", sim_label: str = "") -> bool: + real_label: str = "", sim_label: str = "", + max_bin=None) -> bool: """Load, check, and report one real/sim pair. Returns True if passed.""" # Import here to avoid circular import issues when run as __main__ import sys as _sys @@ -87,6 +88,7 @@ def _run_pair(real_dir: str, sim_dir: str, budget_s=budget_s, real_ground_truth=real_ground_truth, sim_ground_truth=sim_ground_truth, + max_bin=max_bin, ) # Add first_divergence as a diagnostic summary entry (always ok — index=0 is expected for async) @@ -127,6 +129,9 @@ def main() -> None: help="rounds cap from config (enables K9 truncation check)") parser.add_argument("--budget-s", type=float, default=None, help="max_experiment_runtime_s / sim_wall_ceiling_s (enables K5/K9)") + parser.add_argument("--max-bin", type=int, default=None, + help="restrict fwdllm cadence rungs (cohort_sequence/V*) to " + "cycle_data_id <= MAX_BIN (first-data-bin logical parity)") parser.add_argument("--strict", action="store_true", help="Treat WARN as FAIL") parser.add_argument("--lenient", action="store_true", @@ -194,6 +199,7 @@ def main() -> None: lenient=args.lenient, json_out=json_out, plot_out=plot_out, + max_bin=args.max_bin, ) # Collect for roll-up (re-load results from JSON if written) import json as _json @@ -224,6 +230,7 @@ def main() -> None: lenient=args.lenient, json_out=args.json_out, plot_out=args.plot_out, + max_bin=args.max_bin, ) sys.exit(0 if ok else 1) diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index cfa698c1c..d547275af 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -85,6 +85,7 @@ budget_not_cap, # §F fwdllm variance-cadence layer (V/DK/G rungs) _iters_per_data_id, + cohort_sequence_parity, iters_per_data_id_parity, var_trajectory_parity, cached_v_pool_parity, diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md new file mode 100644 index 000000000..90519d310 --- /dev/null +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -0,0 +1,189 @@ +# TEMP task tracker — logical real↔sim parity (delete when folded into simulate_fwdllm.md §G) + +**Goal.** Prove the sim takes the SAME logical steps in the SAME order as real (same cohorts, same receive +order, same variance cadence, same grads) up to data bin 1 across fwdllm / fwdllm_plus / fluxtune — and make the +parity CHECKS + PYTESTS actually *exhibit* these properties (they don't today). Only then chase the time +dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14/#16). + +--- + +## ⏸ SESSION CHECKPOINT (2026-07-05, paused for a break — resume here) + +**Code + tests are landed and GREEN; NO run has been launched yet.** Full suite: 606 mode/telemetry/selector + +115 async_cifar10 parity green; trainer files compile. NOTHING is half-edited. Nothing committed yet at pause → +**commit + push done at end of session** (this checkpoint is the resume anchor). + +**DONE (landed + tested):** +- **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both + new rungs correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus + 36/17/21, fluxtune 35/19/19. +- **P1-6 (partial)** — 11 rung/guard pytests. *Still missing: a LIVE sim==real grad-determinism test (needs P0-2).* +- **P1-8** — banked logs re-run through the upgraded checker. +- **P2-1/P2-3/P2-5/P2-6** — remainder-wait delay model (K-D29: real sleeps `max(0,D−gpu)`, sct `max(gpu,D)`, + overrun telemetry), crc32 straggler disabled, `perturbation_count` knob (default 10), overrun watch. +- **P2-2 (config)** — factor=2 will be applied via the launch flag (below), not yet run. +- **P2-4 (partial)** — GPU profiled: fwdllm/plus ~1.0s, **fluxtune 7.57s** (JVP 20 passes). Optimization deferred. + +**NOT DONE / DEFERRED (pick up here) — in priority order:** +1. **⚠ P1-4 and P1-5 were SKIPPED before jumping to Phase 2** (operator flagged this): + - **P1-4** — uniform cohort-record ADAPTER so the `cohort_sequence` rung also runs on **async_cifar10**'s + per-commit `agg_round` shape (fwdllm's 3 baselines already work; async_cifar10 does not yet). + - **P1-5** — enforce `selection`/`decision_determinism`/`selection_detail` where order is deterministic + (populate `DETERMINISTIC_SELECTORS` / per-baseline gate; today WARN-only via the empty set at `checks.py:882`). +2. **P2-7 — LAUNCH THE databin1 RUN** (the immediate experimental next step; command below), then re-run the + checker `--max-bin 1` and confirm the cascade: fwdllm/fwdllm_plus `cohort_sequence` should extend past bin 8; + fluxtune expected to still break (GPU overrun) → confirms P2-5 tuning is the fluxtune next step. +3. **P0-2** — controlled grad-determinism-given-order confirmation (also unblocks the P1-6 live test). +4. **P2-4 (optimize)** / **P2-5 (tune fluxtune `perturbation_count`)** — reduce fluxtune GPU below its D budget. +5. **Phase 3** — extend beyond bin 1 once bin-1 parity holds. + +**IMMEDIATE NEXT COMMAND (P2-7):** +``` +cd lib/python/examples/fwdllm/expt_scripts +bash run_sequential.sh --mode both --delays on --delay-factor 2 --max-data-id 1 --max-runtime-s 1800 --yes +python run_parity.py --yes --max-bin 1 # then inspect cohort_sequence per baseline +``` +Expected: fwdllm/fwdllm_plus (GPU≈1s < D/2 of 2–9s) → order deterministic → `cohort_sequence` improves; fluxtune +(GPU 7.57s > D/2) → `[TIMING_OVERRUN]` fires, order still flips → tune `perturbation_count` down next. + +**KEY LEARNINGS (durable):** +- The divergence root is **commit ORDER**, not nondeterminism: fwdllm var is a split-half stat over the + commit-ordered grad list → wrong order → wrong var → threshold flips → per-trainer RNG (seeded once, never + reset) desyncs → grads diverge ~1%. Grads ARE deterministic given matched order ⇒ exact parity is achievable. +- The order wasn't deterministic because D≈0.4s (÷10) ≪ GPU 1–1.7s → GPU-jitter-dominated. Fix = remainder-wait + `max(gpu,D)` + per-trainer D + D≫GPU (D/2) ⇒ order = D-order = deterministic. +- fwdllm is **forward-only** (FedFwd, no backprop); the GPU lever is `perturbation_count` (JVP passes), not a + backprop optimization. fluxtune's 20 passes (7.57s) is the acute cost. +- The old checker was **blind** to all this (cohort rung gated off, V2 KS-only). The new `cohort_sequence` + + V2-guard are the spec the model fix must satisfy. + +--- + +## ROOT DIAGNOSIS (from the 2026-07-05 investigation — supersedes "order is benign") + +**One root explains all three baselines: the sim's commit/receive ORDER ≠ real's actual arrival ORDER.** +- fwdllm's variance is a **split-half statistic over the commit-ORDERED grad list** (`fwdgrad_utils.py:133-158`, + appended in commit order at `fwdllm_aggregator.py:718-721`). Order matters even for an identical cohort SET. +- Sim commits in **sct order** (`_sim_recv_min_grad` / `_sync_sim_recv_first_k`); real commits in **physical + arrival order**. Different order → different split-half `var` → `var_good_enough` flips at a different + `iteration_per_data_id`. +- Each trainer's `torch.Generator` is seeded ONCE and **never reset** (`tc_transformer_trainer_distribute.py: + 222-225`); the perturbation for a given `(data_id, iter)` depends on how many prior forward passes that trainer + did. One extra iteration → every trainer's RNG desyncs from that point → all later grads differ (~1%). This is + the fwdllm bin-8 cadence break AND the fluxtune 0/17 cohort break — same cause. +- **fluxtune** (async, agg_goal=3 delay` handling. +- Sim sct adds a crc32 `_sim_straggler_offset_s` (spread 0.9s) + `_wan_s` that **real has no counterpart for** + (`FedSgdTrainer.py:550-565`, `:623-624`) → an extra sim-only reorder vs real. + +**Foundational question — likely ANSWERED by the code:** grads are deterministic GIVEN matched commit order +(batch selection is deterministic in `data_id`; seeds are mode-invariant; JVP is fixed). The ONLY divergence +sources are commit-order + retry-count, both downstream of order. ⇒ **it's a sim ORDER bug, not nondeterminism.** +Needs one controlled confirmation (P0-2). If confirmed, exact cadence parity IS achievable. + +**Checker is blind to all of this (Agent A):** `aggregation_sequence` (cohort set) is gated `ok=True` for +stochastic selectors (`DETERMINISTIC_SELECTORS=∅`, `checks.py:882`) and keys on `round` not `cycle_data_id`; +`inter_arrival_order` is WARN-only; `v2_var_trajectory` is **KS-only, no mean guard** (a 1% offset passes); no +`--max-bin`. Both divergences pass today. + +--- + +## PHASE 0 — settle decisions + the foundational question (BLOCKING) +- [x] **P0-1 Operator decisions — RESOLVED 2026-07-05** (see "Resolved decisions" below): remainder-wait delay; + D configurable, start at **D/2** (not full registry); **optimize GPU now** (prove forward < backward); + checker rungs **hard-FAIL**. New scope added: fluxtune JVP forward-pass-count knob + budget-overrun ordering + watch. +- [ ] **P0-2 Confirm grad-determinism-given-order.** Controlled check: force identical commit order in a sim and + a real short run (or two real runs) and diff per-iteration `var` + a grad norm. Expect bit-match if order + matches. Decides sim-bug (exact target) vs nondeterminism (distributional target). + +## PHASE 1 — make the CHECKS/PYTESTS exhibit the properties (TOP PRIORITY, test-driven) +*Build the failing check FIRST, confirm it catches the current divergence on banked logs, then fix the model.* +- [x] **P1-1 DONE — enforced `cohort_sequence` rung landed** (`parity/checks.py::cohort_sequence_parity`, tier + EXACT, ungated, keyed on the `_fwd_cadence_cycles` stream). Per cycle asserts cohort SET + receive-ORDER + + cadence tuple + `var` value (rel-tol 1e-3). Wired into `run_all_parity` + `CHECK_META` + the `run_parity.py` + headline. **Now FAILS all 3 banked pairs** (was invisible). +- [x] **P1-2 DONE — V2 mean-guard added** (`var_trajectory_parity`, `mean_tol_rel=0.02`). **Now FAILS all 3** + (KS-only used to pass the ~1% offset). +- [x] **P1-3 DONE — `--max-bin` window** threaded through `_fwd_cadence_cycles` → V1/V2/V3/V4/V5 + `cohort_sequence` + + `run_all_parity` + `run_parity.py` + `cli.py`. `run_parity.py --max-bin 1` verified. +- [ ] **P1-4 Uniform cohort-record adapter** so async (per-commit `agg_round`, `contributing=[end]`) and sync + (per-cycle cohort) run ONE cohort/order diff. NOTE: fluxtune already emits per-cycle cadence fields (it's the + fwdllm aggregator with `is_async`), so `cohort_sequence` ALREADY runs on all 3 fwdllm baselines. The adapter + is only needed to extend the rung to async_cifar10's per-commit shape — lower priority. +- [ ] **P1-5 Enforce selection/determinism rungs** where order is deterministic: populate + `DETERMINISTIC_SELECTORS` (or a per-baseline determinism gate) so `selection`/`decision_determinism`/ + `selection_detail` stop being WARN-only. +- [~] **P1-6 PARTIAL — rung + guard pytests landed** (`TestCohortSequence` ×9, `TestVarTrajectoryMeanGuard` ×2 in + `tests/mode/test_parity_checks.py`). STILL TODO: a LIVE sim==real grad-determinism test given matched order + (needs P0-2 infra). +- [ ] **P1-7 Cross-example + cross-baseline coverage.** Confirm the rung runs on async_cifar10's per-commit shape + (P1-4). fwdllm_plus & fluxtune already covered by the banked-log run. +- [x] **P1-8 DONE — banked logs re-run through the upgraded stack.** New ENFORCED reference (full run): + **fwdllm 41/13/21, fwdllm_plus 36/17/21, fluxtune 35/19/19** — each now lists `cohort_sequence` + + `v2_var_trajectory` in FAILS. At **`--max-bin 1`**: cadence (V1) passes but `cohort_sequence` still fails — + fwdllm shows `order_match_frac=0.0`, `var_match_frac=0.5` (real `var=0.371605` vs sim `0.371067` at cycle 0), + proving the order→split-half-var→grad-desync mechanism. Regression: 337 mode/telemetry/selector +115 + async_cifar10 parity tests green. + +## PHASE 2 — fix the timing/order MODEL until the (now-failing) checks pass +- [x] **P2-1 DONE — remainder-wait delay model** (`FedSgdTrainer._emulate_training_delay(gpu_time_s)` → + `(modeled_delay, remaining, overran)`; real sleeps `max(0, delay−gpu)`, sim skips; `[TIMING_OVERRUN]` log). + sct is now `max(gpu, delay)` (not `gpu+delay`). Supersedes K-D2 → **K-D29**. Tests rewritten + (`test_fwdllm_trainer_sim_duration.py`), full suite green. +- [x] **P2-2 DONE (config) — factor=2 (D/2)** via launch flag `--delay-factor 2` (knob stays live; `training_delay_ + factor` per-config). Applied at launch below. +- [x] **P2-3 DONE — crc32 straggler offset disabled** (`sim_straggler_spread_s: 0.0` in all 3 sim yamls). The + per-trainer registry delays now supply the completion spread; the offset would re-noise the deterministic + order. `_wan_s` already 0. (Helper kept flag-gated for its unit tests.) +- [~] **P2-4 PARTIAL — GPU profiled** (from telemetry): fwdllm/fwdllm_plus **~1.0s** (cos-sim path, ~1 forward + pass); **fluxtune 7.57s mean, max 59.6s** (JVP path, 2×10 = 20 passes). fwdllm is forward-ONLY (FedFwd, no + backprop) so "forward modeled sct, the update arrives + after the vclock passed its sct → out-of-order commit. Expect it to fire on fluxtune this run. +- [ ] **P2-7 Verify the cascade closes** — the launch below is the test: fwdllm/fwdllm_plus (GPU≈1s < D/2) + should extend cohort_sequence parity past bin 8; fluxtune (GPU overrun) is expected to still break → confirms + P2-5 is the fluxtune next step. + +## PHASE 3 — validate on data bin 1, then extend +- [ ] **P3-1 Fresh databin1 loop** (`--max-data-id 1`, delays configured per P2-2) across the 3 baselines. +- [ ] **P3-2 Run parity + sanity checks** on the databin1 pairs; require the logical rung GREEN for all three. +- [ ] **P3-3 Extend length** only after bin-1 parity holds; re-check where (if) it breaks; iterate. +- [ ] **P3-4 Fold outcomes** into simulate_fwdllm.md (§G one-liners, retire closed issues) and DELETE this tracker. + +--- + +## RESOLVED DECISIONS (operator, 2026-07-05) +1. **Delay model → remainder-wait.** `wall = delay` (GPU hidden inside), aligning fwdllm to async_cifar10. (P2-1) +2. **Delay magnitude → configurable, start at D/2** (factor=2), not full registry, not ÷10. Raise only if order is + still GPU-dominated. Tied to proving forward < backward and to the true JVP cost. (P2-2) +3. **GPU overhead → optimize NOW.** Prove forward-pass < backprop; root-cause the 5–8× vs cifar10. Forward latency + grows with model size, so this is a standing invariant to hold with margin. (P2-4) +4. **fluxtune JVP cost is under-counted → new scope.** 2 passes/perturbation × ~10 = ~20 passes; make the count a + configurable knob and measure the real full-JVP GPU cost into the sct. (P2-5) Watch budget-overrun→ordering as + the next fluxtune step. (P2-6) +5. **Checker enforcement → hard FAIL.** Cohort/order rung + V2 mean-guard are EXACT & enforced (ungated). The + banked pairs will fail until the model is fixed — that's the gate. (P1-1/P1-2/P1-8) + +## ANCHORS (for implementers) +- Split-half var: `examples/fwdllm/aggregator/.../fwdgrad_utils.py:133-158`; order append `fwdllm_aggregator.py:718-721`. +- RNG-once seed: `.../tc_transformer_trainer_distribute.py:222-225`, draws `:306/:345/:414`. +- Delay/sct: `FedSgdTrainer.py:510-538` (sleep), `:623-629` (sct), `:550-565` (straggler offset). +- Sim commit order: `fwdllm_aggregator.py:747-927` (`_sim_recv_min_grad`), sync `top_aggregator.py:360+`. +- Checker rungs: `async_cifar10/scripts/parity/checks.py` — `aggregation_sequence:1071`, `inter_arrival_order:1323`, + `v2 var_trajectory:3753`, `v1:3716`, gating `DETERMINISTIC_SELECTORS:882`, wiring `run_all_parity:4155`. +- Standalone logical diff: `expt_scripts/logical_parity.py`. diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 97b5565d0..9ad99a0b8 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -60,7 +60,7 @@ experiments: # per-trainer completion dispersion (uniform[0,spread), spread ≈ # real-compute-std·√12). Active with --delays on; calibrate to # wall_disparity→~0 at the sign-off run. - sim_straggler_spread_s: 0.9 + sim_straggler_spread_s: 0.0 # P2-3/K-D29: per-trainer registry delays now supply the completion spread (remainder-wait max(gpu,D)); the crc32 offset would re-noise the deterministic order aggregator: config_template: configs/aggregator_base.json diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml index a513a81d4..ee6255704 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_n10_smoke_sim.yaml @@ -61,7 +61,7 @@ experiments: # std ~0.26s; uniform[0,spread) offset ⇒ spread ≈ 0.26·√12). Only # active with --delays on (convergence run); inert at D=0. Calibrate # to wall_disparity→~0 at the sign-off run. - sim_straggler_spread_s: 0.9 + sim_straggler_spread_s: 0.0 # P2-3/K-D29: per-trainer registry delays now supply the completion spread (remainder-wait max(gpu,D)); the crc32 offset would re-noise the deterministic order client_notify: trace: syn_0 diff --git a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml index 45bb27a95..10f496558 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fwdllm_plus_n10_smoke_sim.yaml @@ -61,7 +61,7 @@ experiments: # per-trainer completion dispersion (uniform[0,spread), spread ≈ # real-compute-std·√12). Active with --delays on; calibrate to # wall_disparity→~0 at the sign-off run. - sim_straggler_spread_s: 0.9 + sim_straggler_spread_s: 0.0 # P2-3/K-D29: per-trainer registry delays now supply the completion spread (remainder-wait max(gpu,D)); the crc32 offset would re-noise the deterministic order aggregator: config_template: configs/aggregator_base.json diff --git a/lib/python/examples/fwdllm/expt_scripts/logical_parity.py b/lib/python/examples/fwdllm/expt_scripts/logical_parity.py new file mode 100644 index 000000000..2cf211422 --- /dev/null +++ b/lib/python/examples/fwdllm/expt_scripts/logical_parity.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python3 +"""Time-STRIPPED logical parity diff, real vs sim, up to a data-bin cap. + +Separates the two parity concerns (simulate_fwdllm.md principles #14/#15): +does the sim take the SAME steps in the SAME order as real, IGNORING every +wall/vclock timestamp? Only once logical parity holds do we chase the time +dimension (sim_rate). Reads the ALREADY-BANKED aggregator telemetry — no re-run. + +The authoritative record is the `agg_round` event stream (it carries data_id, +iteration_per_data_id, the receive-ordered contributing_trainers, and the +variance decision). We restrict to data_id <= --max-bin (default 1: bin 0 has +many iterations/aggregations already) and compare, per aggregation cycle: + - receive SET : which trainers committed together (async cohort composition) + - receive ORDER : the arrival order (only meaningful for async; a sync barrier + commits the whole cohort so order within a set is irrelevant) + - cadence tuple : (data_id, iter, agg_goal_count, var_good_enough, force_commit) +plus iterations-to-clear-bin-0 and the per-cycle variance trajectory. + +Usage: python logical_parity.py [--max-bin 1] [--baselines fluxtune ...] +""" +from __future__ import annotations +import argparse, glob, json, os +from pathlib import Path + +_EXP = Path(__file__).resolve().parent.parent / "experiments" + + +def _short(t): return t[-3:] if len(t) >= 3 else t + + +def _agg_trace(run_dir: Path, max_bin: int): + f = next(iter(glob.glob(str(run_dir / "telemetry" / "aggregator_*.jsonl"))), None) + out = [] + for line in open(f): + try: + e = json.loads(line) + except Exception: + continue + if e.get("event") != "agg_round": + continue + did = e.get("data_id") + if did is None or did > max_bin: + continue + out.append({ + "data_id": did, + "iter": e.get("iteration_per_data_id"), + "goal": e.get("agg_goal_count"), + "order": [_short(t) for t in (e.get("contributing_trainers") or [])], + "var_good": e.get("var_good_enough"), + "force": e.get("force_commit_planned"), + "var": e.get("var"), + }) + return out + + +def _find_pair(baseline: str): + def pick(variant): + cands = [p for p in glob.glob(str(_EXP / f"run_*_{baseline}_n*_smoke_*_{variant}")) + if f"_{baseline}_n" in os.path.basename(p)] # fwdllm !=> fwdllm_plus + return Path(sorted(cands)[-1]) if cands else None + return pick("real"), pick("sim") + + +def _iters_to_clear_bin0(trace): + return sum(1 for x in trace if x["data_id"] == 0) + + +def _cmp(baseline: str, max_bin: int): + rdir, sdir = _find_pair(baseline) + print(f"\n==================== {baseline} (data bin <= {max_bin}) ====================") + print(f" real: {rdir.name if rdir else None}") + print(f" sim : {sdir.name if sdir else None}") + if not (rdir and sdir): + print(" MISSING pair"); return + r, s = _agg_trace(rdir, max_bin), _agg_trace(sdir, max_bin) + + n = min(len(r), len(s)) + set_match = sum(1 for i in range(n) if sorted(r[i]["order"]) == sorted(s[i]["order"])) + ord_match = sum(1 for i in range(n) if r[i]["order"] == s[i]["order"]) + cad = lambda x: (x["data_id"], x["iter"], x["goal"], x["var_good"], x["force"]) + cad_match = sum(1 for i in range(n) if cad(r[i]) == cad(s[i])) + + print(f" aggregations compared: {n} (real={len(r)}, sim={len(s)})") + print(f" receive-SET identical: {set_match}/{n}") + print(f" receive-ORDER identical: {ord_match}/{n} (sync barrier: order-within-set is benign)") + print(f" cadence identical: {cad_match}/{n} (data_id,iter,goal,var_good,force)") + print(f" iters to clear data bin 0: real={_iters_to_clear_bin0(r)} sim={_iters_to_clear_bin0(s)}") + + verdict = ("LOGICAL PARITY" if cad_match == n and set_match == n else + "CADENCE PARITY (cohorts differ)" if cad_match == n else + "LOGICAL DIVERGENCE") + print(f" => {verdict}") + + if cad_match != n or set_match != n: + print(f" {'REAL':<44} | SIM") + for i in range(max(len(r), len(s))): + rr = (r[i]["data_id"], r[i]["iter"], tuple(sorted(r[i]["order"])), + r[i]["var_good"], round(r[i]["var"] or 0, 3)) if i < len(r) else "" + ss = (s[i]["data_id"], s[i]["iter"], tuple(sorted(s[i]["order"])), + s[i]["var_good"], round(s[i]["var"] or 0, 3)) if i < len(s) else "" + flag = "" if (rr and ss and rr[2] == ss[2]) else " <- cohort differs" + print(f" {str(rr):<44} | {ss}{flag}") + + +if __name__ == "__main__": + ap = argparse.ArgumentParser() + ap.add_argument("--max-bin", type=int, default=1) + ap.add_argument("--baselines", nargs="+", default=["fwdllm", "fwdllm_plus", "fluxtune"]) + a = ap.parse_args() + for b in a.baselines: + _cmp(b, a.max_bin) diff --git a/lib/python/examples/fwdllm/expt_scripts/run_parity.py b/lib/python/examples/fwdllm/expt_scripts/run_parity.py index 14ed8e836..10295348d 100644 --- a/lib/python/examples/fwdllm/expt_scripts/run_parity.py +++ b/lib/python/examples/fwdllm/expt_scripts/run_parity.py @@ -75,9 +75,9 @@ def _agg_goal(run_dir: str) -> int | None: # rungs shown in the summary (name -> short label), in ladder-ish order _HEADLINE = [ - ("vclock_telemetry", "vclock"), ("throughput", "thru"), - ("total_commits", "commits"), ("terminal_state", "terminal"), - ("r1_inflight_overlap", "R1"), ("w1_compute_conservation", "W1"), + ("cohort_sequence", "cohort"), ("vclock_telemetry", "vclock"), + ("throughput", "thru"), ("total_commits", "commits"), + ("terminal_state", "terminal"), ("r1_inflight_overlap", "R1"), ("v1_iter_per_data_id", "V1"), ("v2_var_trajectory", "V2"), ("staleness", "U3"), ("participation", "S2"), ("convergence", "conv"), ("convergence_loss", "conv_loss"), @@ -129,6 +129,9 @@ def main(argv=None) -> int: ap.add_argument("--yes", action="store_true", help="skip the confirm prompt") ap.add_argument("--validate", action="store_true", help="also report staleness_policy + vclock_now presence") + ap.add_argument("--max-bin", type=int, default=None, + help="restrict fwdllm cadence rungs (cohort_sequence/V*) to " + "cycle_data_id <= MAX_BIN (first-data-bin logical parity)") args = ap.parse_args(argv) from parity.checks import load_run_dir, run_all_parity # noqa: E402 @@ -183,7 +186,8 @@ def main(argv=None) -> int: goal = _agg_goal(rdir) or 0 real_agg, real_tr = load_run_dir(rdir) sim_agg, sim_tr = load_run_dir(sdir) - res = run_all_parity(real_agg, sim_agg, real_tr, sim_tr, agg_goal=goal) + res = run_all_parity(real_agg, sim_agg, real_tr, sim_tr, agg_goal=goal, + max_bin=args.max_bin) jpath = os.path.join(json_dir, f"parity_{label.replace('/', '_')}_{sts}.json") json.dump(res, open(jpath, "w"), indent=2, default=str) n_pass = sum(1 for v in res.values() diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index ea3195755..3c5d3fabf 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -3,30 +3,31 @@ **Active build (branch `dg/fwdllm_sim_unavail`).** A simulated-clock runner for the `fwdllm` example (FedFwd / forward-gradient FL) that reaches **real↔sim parity** across the **fluxtune / fwdllm / fwdllm++** baselines — at **100% availability (syn_0)** first (Phase 1), then under **unavailability** -(syn_20/50/mobiperf, Phase 2), then **beyond syn_0** (Phase 3). fwdllm has no native sim clock; the build -wires flame-core's virtual clock + availability substrate into fwdllm's variance-gated gradient loop. - -> **DESIGN PRINCIPLE — reuse async_cifar10's concepts, deviate where the workload demands.** We reuse its -> virtual clock, sct reorder buffer, in-flight gate, availability substrate, and parity ladder wherever they -> transfer. But fwdllm is a different workload (aggregates **gradients** not weights; endogenous -> **variance-gated dynamic-K** commit cadence; **`data_id`** progress axis; one-message-per-call grad loop; -> rollback across agg-goal cycles), so a verbatim port is sometimes wrong. When a fork appears, **decide -> explicitly** (don't silently copy or silently invent) and **log it in §K**. The at-a-glance delta table is -> **§B.1** (curated, current); §K is the one-line-per-decision rationale log. - -> **DOC MAINTENANCE — standing instruction (keep this doc rich but CRISP).** This is a living status doc, not -> a changelog. Every edit obeys: -> - **Fold, don't append.** A decision that has landed is **one line** (anchor + what/why). A mechanism that -> **worked** is **one sentence** in §G Fixes-landed. Something that **didn't work / was believed-then-refuted** -> goes to §H Dead-ends as one line. No multi-paragraph appends — update the relevant section **in place**. -> - **Status sections are rewritten in place, not grown.** §A (current state), the scoreboard, the open-issues -> index, and open-roots reflect the CURRENT truth only; per-run history lives in git + the parity JSONs. -> - **§K is one entry per deviation.** Keep the anchor (referenced from §B.1) + a one-line rationale once folded; -> expand only an entry that is still open/contested. -> - **Delete stale scaffolding.** Cold-start build maps, superseded plans, and finished batch checklists are -> removed once landed (the code is the source of truth), leaving a one-line pointer if still referenced. - -**Prerequisites:** [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) — the parity methodology (ladder, +(syn_20/50/mobiperf, Phase 2), then **beyond syn_0** (Phase 3). fwdllm has no native sim clock; the build wires +flame-core's virtual clock + availability substrate into fwdllm's variance-gated gradient loop. We reuse +async_cifar10's virtual clock, sct reorder buffer, in-flight gate, availability substrate, and parity ladder +wherever they transfer, and deviate where the workload demands (fwdllm aggregates **gradients** not weights; +**variance-gated dynamic-K** commit cadence; **`data_id`** progress axis; one-message-per-call grad loop; +rollback across agg-goal cycles). + +> ## PREAMBLE — how to maintain this doc (READ BEFORE EDITING) +> This is a **living status doc**, not a changelog. It must stay **rich but crisp**: enough to reconstruct *why* +> a decision was made, never a running history. Per-run history lives in git + the parity JSONs; the code is the +> source of truth for *what* the mechanism is. Every edit obeys: +> - **Current-truth only.** §A, the scoreboard, and the open-issues table describe the state **right now**. +> Rewrite them **in place** — never stack dated "UPDATE" blocks. When something closes, **delete it from the +> open list** and leave at most a one-line trace in §G (fixes) or §H (dead-ends). +> - **One line per landed item.** A fix that worked = one line in §G (**≤20 words problem + ≤20 words fix**). A +> belief that was refuted = one line in §H. A deviation = one line in §K (anchor + rationale). No paragraphs. +> - **Keep only what teaches.** Retain a big root-cause or a conceptual correction that would otherwise be +> re-litigated; **drop nitpicks** and mechanical scaffolding once landed (leave a pointer only if still +> referenced). If an entry no longer changes a future decision, delete it. +> - **No contradictions.** An issue is in exactly one place: OPEN (open table) **xor** CLOSED (§G/§H one-liner). +> Never both. Cross-link with §-anchors instead of restating. +> - **Decide forks explicitly.** When fwdllm diverges from async_cifar10, log the choice in §K and surface it in +> the §B.1 delta table — don't silently copy or silently invent. + +**Prerequisites:** [async_cifar10/PARITY.md](../async_cifar10/PARITY.md) — parity methodology (ladder, roles/tiers/gating, run-length budget, landed sim mechanisms); **fwdllm's rung catalog is PARITY.md §F**. [async_cifar10/UNAVAILABILITY_DESIGN.md](../async_cifar10/UNAVAILABILITY_DESIGN.md) — the availability substrate fwdllm inherits via its aggregator class chain (ClientAvailability mixin, trace-read effect path, two-ledger @@ -36,222 +37,150 @@ discipline, starvation self-termination, A6/A7/A8/K11 ground-truth rungs). ## §A Current status -**Ground truth (2026-07-05): the fluxtune #13 drain is FIXED (steps 1-3, K-D28/b/c) — the sim runs at speed and -the 30s-forever stall is gone.** Landed + validated across `run_..._024637 → 032729`: stuck-end eviction + -fwdllm recv-grace floor 2→5s (tunable) + probe-ceiling/ready-gating + `drain_ready` direct ingest → `sim_rate` -0.06→0.29, blocks≥5s 54→8, `SIM_GRAD_STUCK_EVICT` fires once (not spinning), R1 stays 0, async_cifar10 -byte-identical (all fwdllm-only methods). **Step 4 (freed-slot staggering, K-D28d) was INVESTIGATED-NEUTRAL → -DISABLED (§H):** the residual 11-12s drain holds are the gate correctly waiting (strict sct order) for the -earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU + strict-order + trickle -dispatch, not a dispatch-timing artifact. **NEXT (top actionable): a fluxtune `--delay-factor` sim run (#12c)** — -the residual `sim_rate<1` is dominated by the missing vclock delay-headroom (D≈0.4-1.8s now → the vclock charges -~only GPU), NOT the drain. With D≈4-18s the vclock jumps by gpu+D per commit while wall pays only the skipped GPU -→ expect `sim_rate` toward/past 1; only then bank the first valid fluxtune real↔sim pair. R1 (#1c) FIXED earlier -(K-D27/b, `SIM_R1_DISPATCH` 238→0). *(Sync baselines fwdllm 49/4/21, fwdllm_plus 45/7/21 unchanged — see below.)* +**All three baselines now run end-to-end at the 2700s budget — the startup race (#14), the R1 re-dispatch churn +(#1c) and the 30s drain stall (#13) are all FIXED and validated. Two standing fronts remain: (1) LOGICAL PARITY — +prove the sim takes the same steps in the same order as real on a matched scope (first data bin); (2) `sim_rate < 1` +— the sim burns more wall than the virtual time it models, so the sim job is not faster than the real job.** ---- +Latest pairs (`run_sequential.sh --mode both --delays on --max-runtime-s 2700`, `run_..._041051 → 080931`): -**Ground truth (2026-07-04 PM): the Phase-1 sign-off run RAN and is USABLE — the first matched-`data_id` -real↔sim pair for all three baselines.** Command: -`run_sequential.sh --mode both --delays on --max-data-id 10 --max-runtime-s 3600 --yes` -(runs `run_20260704_172155..181912`). **Config sanity PASS** (flags honored bash→yaml→banked config→runtime; -delays matched per-trainer real-sleep = sim-modeled; caveat: `training_delay_factor=None` → the hardcoded ÷10, -so D≈0.4–1.8s, not the full 4–18s — pass `--delay-factor` for the full delay, issue #12c). - -**The S1 blocker is gone:** all 6 processes stopped cleanly on **matched `data_id=10`**, none on -`[SIM_WALL_CEILING]` — the Phase-4a ceiling-decouple + matched-`data_id` stop + Stage-C fwdllm_plus liveness -all worked (fwdllm_plus real completed 10, was stalling at ≈3). - -**Headline results:** (1) **#6 clock-rate ROOT-CAUSED + FIXED (K-D25).** The "sim under-models real wall 2×" -gap was a **checker-anchor artifact, not a sim under-charge** — the rungs compared sim-vclock (genuine -algorithmic time) against real's FULL wall, which carries a ~constant ~7.7s/round localhost transport ARTIFACT -(mqtt re-fetch/redistribute/drain-tail/sleeps) the sim correctly omits. Fix: real now anchors on its intrinsic -algorithmic clock; throughput 0.47→0.048, wall_disparity 90→1.6s (landed, 878-green). The **phase rungs (Root B)** -are fixed too (real's slept delay excluded from post_train; B2 straggler out of the emitted training_budget). -(2) **fluxtune R1 residence REGRESSED to 60.4%** — the K-D19 "fixed in code" claim is **refuted** by this -emergent smoke (was 44.7%); `[SIM_R1_DISPATCH]` fires despite `_release_end_on_return`. Now the top blocker. -(3) **#13 sim_rate** for the sync baselines is near-1 (0.82/0.85) and Root A shows the vclock is correct — the -residual is a small physical overhead; the severe case (fluxtune 0.26) is **R1-confounded** → fix #1c first. - -**UPDATE (2026-07-04 later PM, `run_20260704_2224..2258`, data_id=20): K-D25/K-D26 clock-rate+phase fixes -VALIDATED on fresh banked pairs — for BOTH sync baselines.** fwdllm rose **44→49 pass**, fwdllm_plus **40→45**: -the 5 clock-rate rungs (throughput/overhead/per_round base/commits/terminal) + phase all **flipped to PASS** -exactly as predicted. fwdllm's 4 survivors are benign (2 marginal-KS, 2 mis-applied gpu_budget). fwdllm_plus's 7 -survivors are the #7 selection divergence (eligibility/selection_detail/avail_timebase — real 4.9 vs sim 9.6 -eligible at syn_0), gpu_budget, and a 13-vs-14 total_commits/terminal off-by-one (real capped at -`max_runtime_s=1800s`, data_id<20; sim finished 20 — a stop-mismatch + boundary, not a bug). **fluxtune sim did -NOT run** — a startup MQTT join-notify race dropped trainer #379's JOIN → 9/10 ends → async_oort never reached -`minInitialTrainers=10` → hung with **zero selections** (#14, **FIXED** — `join()` waits for connect; full -`tests/` 879-green). The re-run (`_2327` sim) then banked cleanly (10 ends, data_id=20) → **39/14/19**, and -**answers the K-D26 open question: part (a) alone recovered ~0 of R1** — sim R1 residence is still **62.9%** -(215 `SIM_R1_DISPATCH` fires) at sim_rate 0.29, so the sustained NONE-delete path / **#13 slow sim is required** -(part-b), not optional. Every fluxtune clock/commit/staleness fail is downstream of this R1+#13 desync. - -### Per-baseline ground state -| baseline | stop | R1 | cadence (V1/V2/g2/U3/S2/conv) | sim_rate | surviving fails | verdict | -|---|---|---|---|---|---|---| -| **fwdllm** (sync) | data_id=20 both | **0% PASS** | **all PASS** | ~0.8 | per_round_advance + phase_gpu_compute (marginal KS 0.211/0.257 vs 0.20/0.25, means match 17.85≈17.48 / 1.03≈1.13), gpu_budget_real/sim (~40% overrun **symmetric** both modes — mis-applied invariant, not a divergence) | **49/4/21.** K-D25/K-D26 clock-rate+phase rungs VALIDATED PASS (44→49). Cadence CLEAN; 4 surviving fails all benign | -| **fwdllm_plus** (sync) | real→wall-cap 1800s (data_id<20); sim→data_id=20 | 0% PASS | **all PASS** | ~0.85 | #7 eligibility/selection_detail/avail_timebase (real 4.9/4.87 vs sim 9.6 eligible/chosen; in_flight 10=10), gpu_budget (symmetric ~40%), total_commits/terminal (13 vs 14 off-by-one, stop-mismatch) | **45/7/21.** Clock-rate rungs VALIDATED PASS (40→45). Fails = #7 + gpu_budget + stop-mismatch boundary; real ~4× slow (#7) | -| **fluxtune** (async) | R1 FIXED; sim **STALLS** on #13 (killed early, data_id=1) | **0% (FIXED, K-D27b)** | V2 PASS; V1/g2 marginal | **0.29 ⛔** (was 0.06; steps 1-3 recovered ~5×) | **#13 drain FIXED** (steps 1-3); residual `sim_rate<1` is #12c delay-headroom-bound + inherent strict-order straggler holds (step-4 staggering neutral, OFF) + throughput/overhead/commits/staleness (#13/#12c-confounded), gpu_budget | **R1 FIXED** (`SIM_R1_DISPATCH` 238→0). #13 **steps 1-3 landed+validated (K-D28/b/c)**; NEXT a `--delay-factor` run before banking a pair | - -### Parity scoreboard — LATEST vs penultimate (rewrite in place; two columns only) -| baseline | penultimate (sign-off `_1721..1819`) | **LATEST (`_2224..2258`, data_id=20)** | Δpass | key failing rungs | +| baseline | real (data_id in ~2710s wall) | sim `sim_rate` | sim wall to real's data_id | R1 | verdict | +|---|---|---|---|---|---| +| **fwdllm** (sync) | 69 | **0.935** | 1345s (real 2707s → **sim ~2× faster/progress**) | 0% (sync) | runs clean; `sim_rate` just under 1 | +| **fwdllm_plus** (sync) | 18 | **0.948** | 346s (real 2719s → **sim ~8× faster/progress**) | 0% (sync) | runs clean; `sim_rate` just under 1 | +| **fluxtune** (async) | 69 | **0.303** ⛔ | 6699s (real 2704s → **sim ~2.5× slower**) | **0** ✓ (`SIM_R1_DISPATCH=0`, evict=1, 0 recv_fifo timeouts) | drain + R1 fixed; acute `sim_rate` case **+ a genuine logical divergence (#1d, async cohort order)** | + +**Why `sim_rate < 1` (the #12c root, not the drain).** With no `--delay-factor`, `training_delay_factor=None` → +the hardcoded ÷10 → modeled delay D≈0.4–1.8s. So each commit charges ~only the real GPU pass (which the sim +genuinely runs) to the vclock — there is almost no skipped transport wait to compress → `sim_rate` ceilings just +below 1 for the sync baselines, and fluxtune's async strict-sct-order straggler holds drag it to 0.30. Per +principle #13 the sim MUST deliver speedup (`sim_rate ≥ 1`); it does not yet. + +**The `sim_rate` lever — `--delay-factor` (#12c), applied within the first-data-bin scope below.** Re-run with the +full registry delay so `sct = send + gpu + D`, D≈4–18s → the vclock jumps by gpu+D per commit while wall pays only +the skipped GPU → expect `sim_rate` toward/past 1. Pair it with the #15 hard-wall-pause audit (the sim must not +sleep real seconds it should fast-forward). Sync baselines are already near-parity in rate; the same should push +them over 1. (Overall next step is first-data-bin logical parity — see STRATEGY / Next roots.) + +### Parity scoreboard — REFERENCE baseline (checker run on the pairs above; `expt_scripts/run_parity.py --yes`) +*These are the numbers we hold against until the open issues resolve — it will be a while before a longer run. +The stale pre-rewrite counts (49/4/21 etc.) are retired; do NOT reference them. Numbers below are the ENFORCED +counts AFTER the logical-parity rungs landed (`cohort_sequence` EXACT + V2 mean-guard, [PARITY_LOGICAL_TASKS.md](PARITY_LOGICAL_TASKS.md) +P1) — these two now correctly FAIL the order/grad divergence that used to be invisible.* + +| baseline | pass / fail / skip | JSON | +|---|---|---| +| **fwdllm/syn_0** | **41 / 13 / 21** | `experiments/_parity_reports/parity_fwdllm_syn_0_20260705_045724.json` | +| **fwdllm_plus/syn_0** | **36 / 17 / 21** | `parity_fwdllm_plus_syn_0_20260705_063350.json` | +| **fluxtune/syn_0** | **35 / 19 / 19** | `parity_fluxtune_syn_0_20260705_080931.json` | + +**Fails, categorized by blast radius (fix the SHARED roots first — principle #14).** +- **SHARED — all 3 (top priority):** `throughput` / `overhead_residual` / `per_round_advance` (the `sim_rate<1` + clock-rate family → #12c); `gpu_budget_real` / `gpu_budget_sim` (mis-applied invariant, symmetric real/sim + overrun — likely a rung-definition bug, not a divergence); `v1_iter_per_data_id` / `v5_variance_pass_ratio` / + `g2_grad_pool_size` / `utility` / `convergence` (cadence/emergent). **The bin-1 logical check (below) splits + these:** length-confound for fwdllm/fwdllm_plus (cadence identical on bin 1), but GENUINE for fluxtune (#1d — the + async cohort/cadence diverges even on bin 1). +- **fwdllm only:** `phase_gpu_compute` (marginal KS, means match). +- **fwdllm_plus only:** `eligibility` / `avail_timebase` / `selection_detail` (the #7 selection divergence); + `terminal_state` / `total_commits` (stop-mismatch from unmatched length). +- **fluxtune only:** `selection_detail` / `phase_gpu_compute` / `phase_weights_to_gpu`; `staleness`; + `terminal_state` / `total_commits` / `convergence_loss` (stop-mismatch + length). + +### STRATEGY — nail first-data-bin logical parity before any longer run +The right way to prove parity is **logical determinism, not aggregate curve-matching**: for a matched scope, the +sim must take **the same sequence of steps in the same order** as real — same trainers selected, same order of +update receipt, same aggregations/rollbacks — differing ONLY in wall-clock (the sim skips real waits). **Scope to +the first 1 data bin** (`--max-data-id 1` / `--max-bin 1`): bin 0 already contains many iterations, variance passes, +and aggregations, so it exercises the full cadence machinery while keeping the trace short, matched, and diffable. +Prove logical parity on bin 0/1 FIRST (from the already-banked telemetry — no re-run needed); only then chase the +time dimension. This is what separates the two concerns and isolates length-confound from genuine logic bugs. + +### Logical-parity check (TIME-STRIPPED, all available bins) — CURRENT REFERENCE +Tool: `expt_scripts/logical_parity.py [--max-bin N]` — diffs the `agg_round` event stream (data_id, iteration, +receive-ordered contributors, variance decision) real vs sim with every timestamp removed. Real receive-order is +DETERMINISTIC in both real and sim by design (operator-confirmed) → exact match is the correct target. + +| baseline | receive-SET (to real's max bin) | cadence | logical parity HOLDS TO | verdict | |---|---|---|---|---| -| **fwdllm** | `0704_173105` · 44/9/21 | **49 / 4 / 21** | +5 (clock-rate+phase→PASS) | per_round_advance (KS 0.211), phase_gpu_compute (KS 0.257), gpu_budget_real/sim (~40% symmetric overrun) | -| **fwdllm_plus** | `0704_180511` · 40/12/21 | **45 / 7 / 21** | +5 (clock-rate→PASS) | #7 eligibility/selection_detail/avail_timebase, gpu_budget_real/sim, total_commits/terminal (13 vs 14, stop-mismatch) | -| **fluxtune** | `_2327` sim · 39/14/19 (R1 62.9%) | **38 / 15 / 19** (`_005222`, K-D27 bind) | ≈0 | **r1_inflight_overlap sim 67.6%** (K-D27 bind alone insufficient — stale-`_sim_committed` bug, fixed K-D27b, re-run pending), + throughput/overhead/commits/staleness/selection_detail/gpu_budget/v1/g2/convergence (R1+#13-confounded) | - -*fwdllm's +5 is the K-D25/K-D26 validation: the 5 clock-rate rungs (throughput/overhead/per_round base/commits/ -terminal) + phase (training_budget/post_train) all flipped to PASS on this fresh `intrinsic_span_s`-carrying -pair. The 4 survivors are benign — 2 marginal-KS (means match) + the mis-applied gpu_budget invariant (symmetric -real/sim overrun, not a divergence). **fluxtune could not be scored** — see #14.* - -### Next run — expected per-baseline deltas (K-D25 baseline-agnostic; K-D26 fluxtune-only) -*A FRESH run is required for ALL THREE — `intrinsic_span_s` is a new agg emit; the checker's #6 anchor only -engages on runs that carry it (old banked runs fall back to raw `ts`). Consider `--max-data-id 20` to clear -fwdllm's 3 marginal rungs (finer KS + boundary washout); fwdllm_plus real ~26 min/10 data_ids fits in 3600s.* -- **fwdllm** — clock-rate (throughput/overhead/per_round/commits/terminal) + phase (training_budget/post_train) - flip to PASS (~44→~50). The **3 marginal rungs** (per_round KS, total_commits/terminal off-by-one) stay - borderline at exactly 10 data_id; clear at a longer run. `gpu_budget` may persist (mis-applied to fwdllm). -- **fwdllm_plus** — same clock-rate + phase improvement (real completes 10, slowly). **eligibility/ - selection_detail divergence (#7)** persists — separate root, not touched. -- **fluxtune** — clock-rate + phase improve; R1's **round-1 trigger gone (K-D26a)** but the **sustained - NONE-delete path (part b) not fixed** → R1 partially recovers, throughput stays R1-confounded. **This run - measures how much (a) alone recovers R1** → decides whether #13 (or a NONE-race fix) is required. - -### Open issues — master index (OPEN top; 1-line issue + next step) +| **fwdllm** | 194/194 identical (trivial: K=10=all) | 22/194 | **data_id 7** — breaks at **8** | **BREAKS @ bin 8** — sim inserts +1 iteration at data_id 7 | +| **fwdllm_plus** | 48/48 identical | **48/48 identical** | **data_id 18 (real's max)** — no break | **PARITY ✓ to bin 18** (extend when a longer real run exists) | +| **fluxtune** | **0/17 identical** | 7/17 | **breaks at bin 0** (cohort #1) | **DIVERGES from the first aggregation** — async cohort order | + +**ROOT (2026-07-05 code+telemetry investigation — supersedes the earlier "order is benign" reading):** all three +break from ONE cause — **the sim's commit/receive ORDER ≠ real's actual arrival ORDER.** fwdllm's variance is a +**split-half statistic over the commit-ORDERED grad list** (`fwdgrad_utils.py:133-158`), so order matters even when +the cohort SET is identical; the sim commits in sct order, real in physical-arrival order → different `var` → the +`var<0.3` gate flips at a different iteration → and since each trainer's `torch.Generator` is seeded once and +**never reset**, one extra iteration desyncs every trainer's RNG → grads diverge ~1% thereafter (the fwdllm bin-8 +break). For fluxtune (agg_goal=3 **STATUS (2026-07-05 checkpoint):** the checker now GATES on this (enforced `cohort_sequence` EXACT rung + +> V2 mean-guard, P1) and the timing-model fix has LANDED (**K-D29** remainder-wait `max(gpu,D)` + per-trainer D + +> crc32 straggler off + `perturbation_count` knob; 606+115 tests green). **Not yet run.** Pending before close: +> **P1-4/P1-5** (async_cifar10 cohort adapter + enforce determinism rungs — skipped en route to Phase 2) and the +> **databin1 validation run** (`--delay-factor 2 --max-data-id 1`). Resume steps: the tracker's SESSION CHECKPOINT. + +### Open issues (OPEN only — closed items live in §G/§H) | # | issue | baseline(s) | next step | |---|---|---|---| -| **#14** ✅ | **fluxtune sim hung at startup — a shared-MQTT join-notify race dropped 1/10 trainers' JOIN → no selection.** `backend/mqtt.py` registers an end ONLY on receipt of a `NotifyType.JOIN` (health-check `ON` is ignored for adds, mqtt.py:243). `join()` fired `notify(JOIN)` fire-and-forget; for trainer #379 the MainThread called `join`→`notify` **2ms before** the async `on_connect` set `_is_connected=True`, so notify returned False ("Cannot send notify: MQTT client not connected") and the JOIN was **dropped with no retry** (subscriptions too). Later STATE_UPDATE notifies don't add an unknown end → #379 invisible → async_oort stuck below `minInitialTrainers=10` → looped `distribute→ends:None→no ends yet`, zero selections, killed by timeout. Real dodged it (all 10 `on_connect` fired before `join`) → flaky startup race, not sim-logic. | all (MQTT) | **FIXED (robust):** `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before subscribe+notify, so both the subscriptions and the JOIN happen post-connect; no-op when already connected (async_cifar10 unchanged). Full `tests/` 879-green. **Re-run the fluxtune sim to bank the pair.** | -| **#1c** ⭐⭐ | **R1 residence 60.4% — ROOT-CAUSED (K-D26): a physical-wall vs vclock desync in the shared `async_oort` re-pick guard, exposed by the slow sim (#13).** Two paths drop a still-outstanding trainer from `all_selected`: (a) the 90s `SEND_TIMEOUT_WAIT_S` abandon-timeout keyed on `time.time()` (round-1 trigger); (b) the aggregator marks carried grads `KEY_END_STATE=NONE` to keep their slot, and async_oort reads `NONE` as "left" → deletes from `all_selected` (sustained). Both misfire only because wall ≫ vclock. K-D19 misdiagnosed (checked the selection filter, not these deletion paths); K-D17b's NONE hypothesis was right. **DEEPER ROOT-CAUSE (K-D27, telemetry `_2327`): the NONE-delete is just ONE symptom of a two-ledger split — async_oort's selection eligibility (`filtered_ends`, async_oort:1607) is gated SOLELY on `all_selected`, a PHYSICAL-event-pruned ledger (recv-fifo 2s re-select, RECVD/NONE cleanup), and NEVER consults the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The bridge that would fix this — `_agg_pending_commit_ref` — is read for telemetry (async_oort:423) but NEVER assigned (always ∅) and never used as a filter; the aggregator's `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared/bridged (contrast asyncfl:618/1490 where felix DOES maintain it). So in the slow sim a grad sits returned-but-uncommitted for a long virtual window; the selector frees it on a physical event while the aggregator still holds it → own select() re-dispatches → R1. Real/async_cifar10 dodge it ONLY because wall≈vclock (window≈0), not by design.** Evidence: R1 62.9% vs 0.2%, 215 `SIM_R1_DISPATCH`, `num_eligible=10` while `in_flight=10`, `in_pending_commit=0/879` (dead hook), `[CHANNEL_CLEANUP] freed=0`×218 + 0 fires of the 90s-timeout/removed-ends prunes. | fluxtune | **Fix (a) LANDED (K-D26); robust fix (b) LANDED (K-D27+K-D27b):** `_sim_pending_commit` maintained felix-style + bound live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). **K-D27b:** the bind alone left R1 at 67.6% (`_005222`) — the reconcile subtracted the STALE `_sim_committed`, dropping re-dispatched-after-commit trainers; fixed to `outstanding = inflight ∪ buffer`. Sim-only (∅ in real → async_cifar10 byte-identical); full `tests/` 887-green. **Re-run the fluxtune sim to confirm `SIM_R1_DISPATCH→0` / R1→~0.2%** (`--only fluxtune --mode sim`). Underlying window also collapses if #13 makes the sim fast. | -| **#6** ✅ | clock-rate rungs anchored real on FULL wall (genuine + ~7.7s/round transport artifact) vs sim-vclock — **checker-anchor bug, sim vclock is correct** | all | **FIXED (K-D25):** agg emits `intrinsic_span_s` (barrier+eval, mirrors the sim vclock composition, fedavg excluded); the 5 clock-rate rungs + `wall_disparity` anchor real on the cumulative intrinsic clock (async byte-identical). Validated: throughput 0.47→0.048, overhead 21→1.1, wall_disparity 90→1.6. **Residual:** `per_round_advance` KS + `total_commits`/`terminal_state` off-by-one at 10 data_id (small-sample + B2 straggler boundary) → clear at longer run / B2 calibration. | -| **~~phase~~** ✅ | `phase_post_train`/`training_budget` compared a mode-dependent phase (real slept the delay; sim carried B2 straggler in its emitted budget) | all | **FIXED (Root B, K-D25):** post_train stamped after the delay (pure post-proc ~0 both modes); B2 straggler moved from `_delay_s` into the sct only, so `training_budget_s` emits the base delay (identical real/sim). Validated 0/0 and 1.14/1.14. | -| **#13** ⭐⭐ | **sim physically slow (`sim_rate` 0.06–0.29 fluxtune) — ROOT-CAUSED (2026-07-05): fwdllm's async drain `_sim_recv_min_grad` is an EARLY port that lacks ALL of felix's straggler-handling refinements, so a stuck/idle in-flight trainer blocks every commit for the full 30s `RECV_TIMEOUT_WAIT_S`, forever.** Exposed once #1c/R1 stopped the (incorrect) re-dispatch churn that was masking it. | fluxtune (all) | **Drain fixed — steps 1-3 LANDED+VALIDATED (K-D28/b/c): `sim_rate` 0.06→0.29, blocks≥5s 54→8, stall broken, R1=0.** Step 4 staggering INVESTIGATED-NEUTRAL → OFF (K-D28d/§H): residual 11-12s holds are inherent strict-sct-order straggler waits. **NEXT: `--delay-factor` run (#12c)** — the `sim_rate<1` residual is delay-headroom-bound, not drain. | -| **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only 4.9 eligible vs sim 9.6 | fwdllm_plus | Real liveness FIXED (Stage C — completes 10). Remaining: profile the per-iteration reselection + oracular read cost from the banked per-phase log; explain the real/sim eligible-count gap at 100% avail. Not a sim bug. | -| **#12c** | `--delays on` uses the hardcoded `training_delay_factor=10` ÷10 shrink unless `--delay-factor` passed | all | Optional: pass `--delay-factor` for the full 4–18s registry delay. Config-flow otherwise correct (K-D24/K-D21). | -| **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits, real-only; one-grad-per-poll drain tail) | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run to validate the MQTT settle timing (principle #8/#11c). Never change grad values/cadence. | - -### Open roots — ranked lowest-rung-first -1. **#12c `--delay-factor` measurement (fluxtune) — NEW TOP ACTIONABLE.** #13 drain is FIXED (steps 1-3); the - residual `sim_rate<1` (0.29) is dominated by the missing vclock delay-headroom, NOT the drain (`training_delay_ - factor=None` → the hardcoded ÷10 → D≈0.4-1.8s, so each commit charges ~only GPU to the vclock while wall pays - GPU + the inherent strict-order holds). **NEXT:** re-run fluxtune sim with `--delay-factor` (sct = send + gpu + - D, D≈4-18s) so the vclock jumps by gpu+D while wall pays only the skipped GPU → expect `sim_rate` toward/past 1. - Only after this is the drain proven end-to-end; then bank the first valid real↔sim pair. -2. **#13 slow sim / drain stall (fluxtune) — FIXED (K-D28/b/c).** stuck-evict + recv-grace 2→5s + drain_ready - direct ingest → `sim_rate` 0.06→0.29, blocks≥5s 54→8, 30s stall gone, R1=0. Step 4 freed-slot staggering was - INVESTIGATED-NEUTRAL → DISABLED (§H; the residual 11-12s holds are inherent strict-sct-order straggler waits, - not a dispatch-timing artifact). -3. **#1c R1 residence (fluxtune)** — FIXED (K-D27/b): `SIM_R1_DISPATCH` 238→0. The pending-commit bridge + - dropping the stale `_sim_committed` subtraction. (Was masking #13; fixing it exposed the drain stall.) -3. **#7 fwdllm_plus real speed / selection divergence** — real completes now; explain the 4× slowness and the - syn_0 eligible-count gap from banked telemetry. -3. *(#6 clock-rate — RESOLVED as a checker anchor, K-D25; sim vclock was correct. Only the 3 marginal rungs + - the B2 straggler calibration remain, both emergent at the longer sign-off run.)* - -### #13 DESIGN — event-driven drain redesign (design LOCKED 2026-07-05; NEXT to implement) -*R1 is FIXED (K-D27/b: `SIM_R1_DISPATCH` 238→0). But fixing R1 unmasked #13: the `_2327`→`_005222`→`_012922` -runs got progressively slower (`sim_rate` 0.29→0.06), `data_id=1` in ~9 min, `aggregate` calls taking exactly -30.0s. The R1-violation churn had been masking a broken drain by (incorrectly) keeping trainers busy.* - -**Root cause (verified by side-by-side with felix `asyncfl/top_aggregator.py::_sim_recv_min`).** fwdllm's -`_sim_recv_min_grad` is an EARLY port that lacks every one of felix's straggler-handling refinements -(grep-confirmed felix=N / fwdllm=0): `_stuck_end`/`_sim_gate_failsafe`/`inflight_expected.pop(_stuck_end)`, -`probe_ceiling` + `_sim_end_has_ready_msg` ready-gating, `drain_ready`/`_sim_sct_ordered_drain` direct ingest, -`_sim_free_slot_ts` freed-slot refill. Consequence: on the 30s `RECV_TIMEOUT_WAIT_S` deadline fwdllm just -`break`s and **leaves the stuck trainer in `_sim_inflight_expected` forever**, so `earlier_stuck` re-fires the -FULL 30s every commit cycle. Evidence: trainer idle in `_fetch_weights` (already replied, awaiting reselect) sits -in the expected set; the composer loop is frozen 30s in the drain so freed slots never re-dispatch → pipeline -starvation (only 5/10 trainers active, 205 recv 2s-timeouts = 74% of wall). - -**Confirmed pathway = felix's model (this is what SHOULD happen, verified in async_cifar10).** Slot-based -discrete-event sim: dispatch to fill `c` slots → trainers compute the REAL grad in parallel (~4s) and reply -carrying their modeled `sct` (the aggregator only learns an `sct` on receipt) → the recv window batch-collects -the cohort's replies at once → **reorder by `sct` AT THE AGGREGATOR** → **skip the vclock to the next visible -`sct`** (accounting jump, ~0 wall), at which one update commits, ONE slot frees (`_sim_free_slot_ts` stamps the -freed vclock), and the aggregator **selects again** to refill. Physical wall = GPU compute only; the vclock -charges GPU + the SKIPPED modeled delays → `sim_rate>1`. - -**Design choices (operator-confirmed):** (1) residence = **invisible-until-`sct`, released-at-boundary** — a -replied trainer is NOT eagerly re-dispatched; it and its update stay invisible until vclock reaches `sct`, then -the update releases and the trainer is re-usable only AFTER the round/bin/iteration boundary. (2) committed SET = -**modeled-fastest-k** (strict `sct` order kept; do NOT relax to arrival order). (3) **variance-BAD rollbacks are -NORMAL** (happen in real too — most cycles var isn't met, discarded until finally met); do NOT touch variance — -the fix is purely drain timing. (4) sim-only, real-mode byte-identical. - -**Plan (incremental, drain is the K-D16/K-D17 deadlock zone — one step per short-sim validation):** -1. **Stuck-end eviction** (felix `_sim_recv_min:436-442`) — **LANDED (K-D28), re-run pending.** On the deadline - the drain now tracks `_stuck_end` and `pop`s it from `_sim_inflight_expected` (+`_sim_gate_failsafe` counter + - `[SIM_GRAD_STUCK_EVICT]`) so a lost/idle straggler can't re-fire the full 30s every future cycle. Also bumped - the fwdllm-scoped recv-grace floor 2→**5s** (`SIM_RECV_GRACE_FLOOR_S`, fwdllm class only; async_cifar10 stays - 2.0) — fwdllm's forward-grad GPU can take ~4s so a 2s window closes before the grad reassembles (TUNABLE: pull - as low as correctness allows). *Suspected to break the 30s stall alone.* **NEXT: short fluxtune sim to confirm - `aggregate` wall 30s→seconds.** `TestStuckEndEviction` + full `tests/` green. -2. **Probe-ceiling + ready-gating** (felix `:399-411`, `_sim_end_has_ready_msg`) — **LANDED (K-D28b), - re-run pending.** The drain now probes a non-recv_end in-flight end only if its rxq is physically ready OR - `exp ≤ buffered_min + slack`, instead of blocking the full 5s grace on every in-flight end each pass. Targets - the ~42 gaps of 4–10s that dominate the post-step-1 wall (`sim_rate` 0.20). Safe vs the HOLD gate (any end that - could trigger `earlier_stuck` satisfies `exp ≤ bmin + slack` → always probed). `TestProbeCeilingReadyGating`. -3. **Direct `drain_ready` ingest** (felix `_sim_sct_ordered_drain`, `:375-390`) — **LANDED (K-D28c), re-run - pending.** Flag-gated (`sim_sct_ordered_drain: true` in the fluxtune sim yaml): the drain sweeps each live - in-flight end's rxq directly via `channel.drain_ready` (non-blocking, returns on FIRST arrival) instead of the - recv_fifo streamer that BLOCKS the full 5s grace PER not-ready end. **The `run_..._030300` trace proved this is - the dominant residual cost** — a `[RECV_FIFO] timeout (5.0s)` fired on every not-ready recv_end each pass. Also - yields a COMPLETE buffer snapshot (the streamer can strand a delivered grad out of the buffer's view → - past-dated commit). `TestSctOrderedDrain`. -4. **Freed-slot refill stamp** (felix `_sim_free_slot_ts`, `:507-520`) — **IMPLEMENTED but INVESTIGATED-NEUTRAL, - DISABLED (K-D28d, `run_..._034550`).** Code landed + flag-gated (`sim_staggered_redispatch`, default OFF) + - `TestFreedSlotRefill`, but it did NOT help: `sim_rate` 0.289→0.274, the 11-12s holds persisted. **Root-cause of - the residual holds (from the step-3 disk telemetry, `sct`/`T_v` around every hold):** the drain is correctly - waiting (strict sct order) for the EARLIEST-sct in-flight straggler while ~7 higher-sct grads sit buffered - (each hold commits a grad whose sct is 0.1-0.4s below the buffered min). Both wall (the straggler's real - arrival) and vclock (clock-jump clamp pins it to the straggler's `min_future+slack`) bottleneck on that one - trainer. This is INHERENT (strict-sct-order + real-GPU + one-in-flight trickle dispatch), not a dispatch- - bunching artifact — so staggering can't fix it, and feeding an OLD freed-slot vclock as `send_ts` makes the - gate expect trainers even earlier → more holds. **The real `sim_rate` lever is #12c** (D≈0.4-1.8s here → almost - no vclock headroom to charge; `--delay-factor` gives `sct = send + gpu + D` so the vclock jumps by gpu+D while - wall pays only the skipped GPU). Steps 1-3 captured the fixable artifacts; the residual is #12c-bound + real - GPU. See §H dead-ends. - -**Expected result:** `aggregate` wall per cycle 30s→~GPU-compute (seconds); slots stay full; `data_id` progresses -at ~real speed; `sim_rate`→≥1 once `--delay-factor` gives the vclock delay-headroom (#12c). R1 stays 0 (residence -unchanged). Real byte-identical. Watch: throughput/staleness/commits rungs should ease (they were #13-confounded). +| **#1d** ⭐⭐ | **fluxtune LOGICAL divergence (not timing): async cohort composition differs every aggregation (0/17 receive-sets match, bin≤1).** The sim's sct-sorted receive order does not reproduce real's actual first-`agg_goal` arrival order → var<0.3 crossing shifts → sim needs +2 aggregations to clear bin 0 (V1 genuinely off). CORRECTNESS bug — fix BEFORE fluxtune's time dimension (principle #14/#16). | fluxtune | Diff per-trainer real-GPU time vs sim-sct order — does the sct use ACTUAL measured GPU or a nominal speed? First resolve the conceptual Q: is real's receive-order deterministic or wall-jittery (§A logical-parity check)? | +| **#12c** ⭐ | `sim_rate < 1` — no delay-headroom to compress. `--delays on` uses the hardcoded ÷10 shrink (D≈0.4–1.8s) unless `--delay-factor` is passed, so the vclock charges ~only GPU. | all (acute: fluxtune) | Re-run with `--delay-factor` (sct = send + gpu + D, D≈4–18s) → `sim_rate` toward/past 1. Then re-run the checker. Config-flow otherwise correct (K-D24/K-D21). | +| **#15** ⭐ | **Hard real-wall pauses in the sim.** fluxtune sim wall 8879s ≫ vclock 2694s (3× overhead) — the vclock must advance THROUGH the modeled wait after GPU compute (fast-forward), never make the sim PROCESS sleep real seconds. Suspect: drain holds waiting real-wall for the slowest straggler's GPU + any non-parallel GPU dispatch. | all (acute: fluxtune) | Audit every place the sim blocks real-wall (drain grace, recv_fifo, GPU dispatch serialization). Confirm the 10 trainers' GPU runs in PARALLEL and the only irreducible real wall is one GPU pass; everything else must be a vclock jump. | +| **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only ~4.9 eligible vs sim ~9.6. Not a sim bug. | fwdllm_plus | Profile per-iteration reselection + oracular-read cost from the banked per-phase log; explain the eligible-count gap at 100% avail. | +| **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits; one-grad-per-poll drain tail) — real-only. | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run (principle #8/#11c). Never change grad values/cadence. | + +### Next roots — ranked (correctness before time; SHARED before per-baseline — principle #14) +0. **DONE — first-data-bin logical-parity check ran.** fwdllm + fwdllm_plus are **logically clean on bin 1** + (cadence + cohorts identical) → their whole-run cadence fails are length-confound, and they are **cleared to + chase the time dimension**. fluxtune has a **genuine logical divergence (#1d)**. +1. **#1d fluxtune async cohort/receive-order (correctness, per-baseline) — TOP.** The one real logic bug. Root-cause + the sct receive-order vs real arrival order; resolve the deterministic-vs-jittery conceptual question first. +2. **#12c `--delay-factor` + #15 hard-wall-pause audit (time, SHARED).** Give the vclock its delay-headroom AND + stop the sim burning real wall it should skip → `sim_rate ≥ 1`. These are the `throughput`/`overhead`/`per_round` + fail family; safe to pursue for the two sync baselines now, and for fluxtune once #1d lands. +3. **#7 fwdllm_plus real speed / selection divergence (per-baseline).** `eligibility`/`selection_detail` — real + completes now; explain the 4× slowness + the syn_0 eligible-count gap from telemetry. Not a sim bug. +4. Then C1/C2 convergence at matched `data_id` per baseline → gate to Phase 2 (unavailability). ### SKIP audit (19–21 skips; ~16 legit) -Legit (mechanism genuinely absent at Phase-1 syn_0 + `random` selector): 7 availability ground-truth rungs + -4 delivery/withheld (Phase-2 effect path, not built) + 3 DynamicKC (disabled by design) + 2 oort-only -(`random` baselines) + `residence` (async telemetry). The 12 former rigor-gap skips (4 advance rungs, 8 -phase-timing rungs) are now **un-skipped** (K-D21 A1/A3) — they run and mostly FAIL on #6, as intended. +Legit at Phase-1 syn_0 + `random` selector: 7 availability ground-truth rungs + 4 delivery/withheld (Phase-2 +effect path, not built) + 3 DynamicKC (disabled by design) + 2 oort-only (`random` baselines) + `residence` +(async telemetry). The 12 former rigor-gap skips (4 advance, 8 phase-timing) are un-skipped (K-D21). --- ## §B How fwdllm differs structurally -fwdllm aggregates **gradients** (JVPs) not weights; its commit cadence is **endogenous** (variance-gated -dynamic-K); its progress axis is committed **`data_id`** (variance passes), not update count. Gradient values -are mode-invariant given identical input+perturbation seed, so parity reduces to **clock + selection + ordering -parity plus a variance-cadence layer**. Anchors (`flame/mode/horizontal/syncfl/fwdllm_aggregator.py`): -`aggregate()` var gate + rollback/`cached_v` at each `_agg_goal` boundary; `total_data_bins=150`; -force-commit cap `max_iterations_per_data_id`; `reselect_each_iteration` (fwdllm++ per-iteration reselection); -sync path `_aggregate_grads_sync`. Full detail: PARITY.md §F.1. +fwdllm aggregates **gradients** (JVPs) not weights; commit cadence is **endogenous** (variance-gated dynamic-K); +progress axis is committed **`data_id`** (variance passes), not update count. Gradient values are mode-invariant +given identical input+perturbation seed, so parity reduces to **clock + selection + ordering parity plus a +variance-cadence layer**. Anchors (`flame/mode/horizontal/syncfl/fwdllm_aggregator.py`): `aggregate()` var gate + +rollback/`cached_v` at each `_agg_goal` boundary; `total_data_bins=150`; force-commit cap +`max_iterations_per_data_id`; `reselect_each_iteration` (fwdllm++ per-iteration reselection); sync path +`_aggregate_grads_sync`. Full detail: PARITY.md §F.1. ### §B.1 Real↔sim design deltas vs async_cifar10 — **CURATED, KEEP CURRENT** -Separates an **intentional fwdllm choice** from an accidental discrepancy. Update a row when a deviation -lands/changes; the §K column points at the one-line rationale. +Separates an **intentional fwdllm choice** from an accidental discrepancy. The §K column points at the one-line +rationale. | # | Axis | async_cifar10 | fwdllm | Why | §K | |---|---|---|---|---|---| | 1 | Aggregated object | model **weights** | **gradients** (JVPs) | grad values mode-invariant → parity = clock+order+selection + variance-cadence | §F.1 | | 2 | Progress axis | update/round count | committed **`data_id`** | cadence (updates-per-data_id) is an **output to match**, not an input | principle #2 | | 3 | Commit cadence | fixed `agg_goal` | endogenous **variance-gated dynamic-K** | the emergent layer cifar doesn't model; V/DK/G rungs verify it | §F.1 | -| 4 | sct delay model | `send + max(gpu, D)` | `send + gpu + D` (**additive**) | fwdllm's real mode sleeps D *on top of* GPU; `max()` would desync | K-D2 | +| 4 | sct delay model | `send + max(gpu, D)` | `send + max(gpu, D)` (**remainder-wait, was additive**) | K-D29: real now sleeps `max(0,D−gpu)` (device wall = D, GPU hidden), so update order = per-trainer D order = deterministic & real↔sim identical. Reverses K-D2 | K-D2/**K-D29** | | 5 | Per-eval sct | distinct, ~20× faster | **collapses to train sct** | eval lives on the aggregator; forward-grad "train" IS a forward pass (no 20× factor) | K-D3 | -| 6 | Slot residence | per-commit release | **hold slot to COMMIT** (felix-aligned) — every dispatched-but-uncommitted trainer held in `selected_ends`+`all_selected`, released on commit | a returned-but-uncommitted grad is still in flight in VIRTUAL time (commits when vclock reaches sct) | K-D5/K-D17b, principle #4 | -| 7 | Surplus grad on rollback | carried | **carried** for async (fluxtune, c≫agg_goal); **drop** stays correct for sync (c≈agg_goal) | drop was benign only for sync; fluxtune dropped ~7/cycle → 2× passes | K-D6/K-D12 | +| 6 | Slot residence | per-commit release | **hold slot to COMMIT** (felix-aligned) — dispatched-but-uncommitted trainer held in `selected_ends`+`all_selected`, released on commit | a returned-but-uncommitted grad is still in flight in VIRTUAL time (commits when vclock reaches sct) | K-D5/K-D17b, principle #4 | +| 7 | Surplus grad on rollback | carried | **carried** for async (fluxtune, c≫agg_goal); **drop** stays correct for sync (c≈agg_goal) | drop was benign only for sync; fluxtune dropped ~7/cycle → 2× passes | K-D12 | | 8 | Async drain primitive | `_sim_recv_min` verbatim | purpose-built `_sim_recv_min_grad` / sync `_sync_sim_recv_first_k` | cifar's per-commit release + withheld paths key on WEIGHTS semantics | K-D4 | | 9 | `time_mode` default | `"simulated"` | `"real"` (getattr fallback) | fwdllm's whole config corpus is `real`; a `simulated` default risks half-activating an unbuilt path | K-D1 | | 10 | Cadence telemetry | n/a | **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/`grad_pool_size`/`cached_v_size`) | post-mutation `data_id` advances before emit → off-by-one; snapshot makes V1 exact | K-D9 | -| 11 | Availability tracking (v1) | all `trace_read` | **mixed**: fwdllm unaware, fwdllm_plus `oracular`, fluxtune `client_notify`→approx `trace_read` | baselines carry different models; first-class `client_notify` deferred to Stage H | D1 | +| 11 | Availability tracking (v1) | all `trace_read` | **mixed**: fwdllm unaware, fwdllm_plus `oracular`, fluxtune `client_notify`→approx `trace_read` | baselines carry different models; first-class `client_notify` deferred to Phase 2 | D1 | | 12 | Launch tooling | single parity template (`debug_run.sh`) | per-baseline yamls + `_sim` files (`run_sequential.sh`) | different config models; both source the shared harness `examples/scripts/expt_runner.{sh,py}` | this work | --- @@ -261,7 +190,7 @@ lands/changes; the §K column points at the one-line rationale. |---|---|---|---|---|---|---|---| | **fluxtune** | async | `async_oort` | fedbuff (+server LR, JVP) | `client_notify`→`trace_read` v1 (aware-at-select, reactive-90s) | — | 3 | disabled | | **fwdllm** | sync | `random` | fedavg | `default` unaware (reactive-90s) | per-round | 10 (=c) | — | -| **fwdllm_plus** | sync | `random` | fedavg | `oracular` (aware-at-select via trace read) | per-iteration | **10** (yaml is source of truth; §C draft read 2) | — | +| **fwdllm_plus** | sync | `random` | fedavg | `oracular` (aware-at-select via trace read) | per-iteration | 10 (yaml is source of truth) | — | Phase order (locked): Phase-1 syn_0 → Phase-2 unavailability → Phase-3 beyond syn_0. One baseline at a time. Stage 3 oort rungs run **only** for fluxtune; sync-barrier rungs run for **fwdllm/fwdllm_plus**. @@ -275,31 +204,29 @@ Stage 3 oort rungs run **only** for fluxtune; sync-barrier rungs run for **fwdll committed `data_id` (PARITY.md §F.3). - **New** variance-cadence layer: V1–V5, DK1–DK3, G1–G2 (PARITY.md §F.4). Localize down; never fix an EMERGENT rung directly. `var_threshold` / `max_iterations_per_data_id` are baseline knobs, not parity levers. -- **Availability** rungs (A1–A5, A6/A7/A8/K11, withheld/abandon/starvation) inherited; apply once Phase-2 - wires the effect path + telemetry. +- **Availability** rungs (A1–A5, A6/A7/A8/K11, withheld/abandon/starvation) inherited; apply once Phase-2 wires + the effect path + telemetry. --- ## §E Roadmap — remaining phases -**Phase 1 (syn_0) — the sign-off run is banked (§A); close-out work:** -1. **#1c fluxtune R1** — root-cause + fix (top blocker). -2. **#13/#6 clock-rate** — confirm the eval fold fires; drive `sim_rate>1` and `wall_disparity`→~0. -3. **#7 fwdllm_plus** — explain real slowness + selection divergence from telemetry. -Then C1/C2 convergence at matched `data_id` per baseline → gate to Phase 2. +**Phase 1 (syn_0) — CLOSE-OUT (near done):** #14/#1c/#13 fixed; the drain runs at speed. Remaining: (1) **#12c +`--delay-factor` run** to drive `sim_rate ≥ 1`, then re-run the checker + bank the three pairs; (2) **#7** +fwdllm_plus real slowness + selection divergence from telemetry; (3) C1/C2 convergence at matched `data_id` per +baseline → gate to Phase 2. **Phase 2 — unavailability (syn_20/50/mobiperf).** Wire the ClientAvailability effect path into the grad loop: -send-time gate (real) / `delivery_ts = max(sct, next_avail)` buffering (sim); two ledgers; reactive-90s -in-flight; starvation vclock-advance; per-baseline `avail_select_filter`. Emit `EVENT_AVAIL_CHANGE` + -`agg_belief_change`/`send_gate_wait` (builders exist in `flame/telemetry/events.py`) → unlocks A6/A7/A8/K11. -**D3 watch:** does a withheld/late grad roll into `cached_v` on rollback, or a stale-version late grad inflate -`var` (and thus dynamic-K)? Over-instrument before trusting cadence. Pytest: adapt async_cifar10's availability -test patterns; syn_0 gate-ON-vs-OFF byte-identical. Exit: A1–A5 + A6/A7/A8 PASS; self-stops; withheld grads +send-time gate (real) / `delivery_ts = max(sct, next_avail)` buffering (sim); two ledgers; reactive-90s in-flight; +starvation vclock-advance; per-baseline `avail_select_filter`. Emit `EVENT_AVAIL_CHANGE` + +`agg_belief_change`/`send_gate_wait` (builders in `flame/telemetry/events.py`) → unlocks A6/A7/A8/K11. **D3 watch:** +does a withheld/late grad roll into `cached_v` on rollback, or a stale-version late grad inflate `var` (thus +dynamic-K)? Over-instrument before trusting cadence. Exit: A1–A5 + A6/A7/A8 PASS; self-stops; withheld grads delivered not dropped. -**Phase 3 — beyond syn_0.** Full ladder under scarcity; bin V1/V2 by run-fraction to separate a constant mix -bias from a compounding variance-feedback loop (§G). Exit: curves within tolerance at matched `data_id`; K8/U2 -within bar; V1/V2 binned residual flat. +**Phase 3 — beyond syn_0.** Full ladder under scarcity; bin V1/V2 by run-fraction to separate a constant mix bias +from a compounding variance-feedback loop. Exit: curves within tolerance at matched `data_id`; K8/U2 within bar; +V1/V2 binned residual flat. --- @@ -320,149 +247,111 @@ within bar; V1/V2 binned residual flat. eligibility plumbing. 8. **Fix the concept, not the symptom — never regress a working example.** Classify a mechanism as **real-transport artifact** (incremental drain, `recv_fifo` timeouts, queue persistence — no sim analog; gate - `and not self.simulated`) vs **algorithmic property**. Prefer the byte-identical-for-the-working-side fix. - Scope-check before editing shared code: `fwdllm_aggregator.py` = fwdllm blast radius; `top_aggregator.py` / - shared `parity` engine / `_sim_recv_min` can silently break async_cifar10. + `and not self.simulated`) vs **algorithmic property**. Scope-check before editing shared code: + `fwdllm_aggregator.py` = fwdllm blast radius; `top_aggregator.py` / shared `parity` engine / `_sim_recv_min` + can silently break async_cifar10. 9. **Match pytest scope to blast radius.** fwdllm-only edit → `pytest tests/mode -k fwdllm`; shared parity engine - → add `examples/async_cifar10/scripts/parity` + `tests/mode -k parity`; shared stack (`flame/launch/*`, - `flame/telemetry/*`, `config.py`, optimizer/selector, `top_aggregator.py`) → full `pytest tests/`. Full suite - once after a cross-stack change, not before every run. -10. **Comment the WHY, crisply.** A conceptual/architectural choice, a real↔sim divergence + rationale, or a - failure mode + why the fix takes its shape — one or two tight sentences. Deeper rationale → §K. -11. **Telemetry-FIRST, then instrument, then (rarely) run.** (a) Validate/refute a hypothesis from telemetry - ALREADY ON DISK before considering a run — name the exact field/line first. (b) Ship telemetry + plot + pytest - IN THE SAME CHANGE as any new mechanism. (c) A run is justified only to observe an EMERGENT quantity no stored - telemetry can yield (convergence, a concurrency/wall number, a longer cadence trajectory) — then run the - SHORTEST length that exhibits it, smoke first, one mechanism per round. (d) Cross-pollinate fluxtune↔sync - baselines — fluxtune surfaces roots first. -12. **Consult PARITY.md vclock principles BEFORE any sim-clock change.** Settled rules: clock is a monotone `max` - (`_advance_sim_clock = vclock = max(vclock, sct)`); NEVER put overhead on the vclock; `sct = send + max(gpu,D) - + leg` (leg NOT on `trainer_speed`/utility); sync charges MAX-of-K sct, async the K-th-fastest; the sim SKIPS - real waits and reconstructs order from sct (`SimReorderBuffer`). -13. **The vclock is virtual wall-time; the sim MUST produce SPEEDUP (`vclock ≥ physical_wall_elapsed`, - `sim_rate = vclock/wall ≥ 1`).** fwdllm nuance: the forward-grad "train" is a REAL GPU pass (~2s) that MUST run - in sim for grad mode-invariance — that GPU wall is irreducible; speedup comes from skipping transport/inter-round - waits, NOT compute. `sim_rate<1` (as in the sign-off run, 0.26–0.85) means the sim under-charges the vclock or - fails to skip a real wait. `sim_rate` is the top-line health metric — emit it every run (`[VCLOCK_PROGRESS]`). + → add `examples/async_cifar10/scripts/parity` + `tests/mode -k parity`; shared stack → full `pytest tests/`. +10. **Comment the WHY, crisply.** A conceptual choice, a real↔sim divergence + rationale, or a failure mode + why + the fix takes its shape — one or two tight sentences. Deeper rationale → §K. +11. **Telemetry-FIRST, then instrument, then (rarely) run.** (a) Validate/refute from telemetry ALREADY ON DISK + first — name the exact field/line. (b) Ship telemetry + plot + pytest IN THE SAME CHANGE as any new mechanism. + (c) A run is justified only to observe an EMERGENT quantity no stored telemetry can yield — then run the + SHORTEST length that exhibits it, smoke first, one mechanism per round. (d) fluxtune surfaces roots first. +12. **Consult PARITY.md vclock rules BEFORE any sim-clock change.** Clock is a monotone `max` + (`vclock = max(vclock, sct)`); NEVER put overhead on it; `sct = send + max(gpu,D) + leg` (leg NOT on + `trainer_speed`/utility); sync charges MAX-of-K sct, async the K-th-fastest; the sim SKIPS real waits and + reconstructs order from sct (`SimReorderBuffer`). +13. **The vclock is virtual wall-time; the sim MUST produce SPEEDUP (`sim_rate = vclock/wall ≥ 1`).** The vclock + advances exactly as the real wall-clock WOULD after the trainer's simulated waits — but faster, because the sim + **elapses through** the modeled time remaining after GPU compute instead of actually waiting it out. fwdllm + nuance: the forward-grad "train" is a REAL GPU pass (~2s) that MUST run in sim for grad mode-invariance — that + ONE GPU pass (parallel across trainers) is the only irreducible real wall; transport/inter-round/delay waits + are vclock jumps, never process sleeps. `sim_rate < 1` means the sim under-charges the vclock OR is stalling on + a real wait it should skip (#15). Emit it every run (`[VCLOCK_PROGRESS]`). +14. **Correctness before speed; SHARED roots before per-baseline.** Fix major logical-correctness divergences + (wrong selection order, wrong receive order, wrong cadence) before any throughput/wall tuning. Rank a root by + blast radius: a bug that fails rungs across ≥2 baselines outranks a single-baseline one — fix the shared cause + once, then narrow to per-baseline residuals. Never chase an aggregate-curve rung while a logic rung is red. +15. **Logical determinism is the parity definition.** For a matched scope the sim must take the SAME sequence of + steps in the SAME order as real — same trainers selected, same order of update receipt, same aggregations and + rollbacks — differing ONLY in wall-clock. Prove this on the **first data bin** (`--max-data-id 1`: multiple + iterations + variance passes + aggregations, short + diffable) before extending length. A matched, ordered + trace is the proof; a matched summary statistic is not. +16. **Do the right thing — no hacks.** Ask as many conceptual questions as it takes to understand the real↔sim + divergence, but solve it at the root. A hack that moves a number without a correct mechanism is a regression in + disguise — it will not close parity and it will mask the real bug. When unsure, stop and ask. **Open design decisions:** D2 (avail telemetry port — Phase 2); D3 (variance-cadence × withheld/late grads — -Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 resolved (see §K). +Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 resolved (§K). --- -## §G Fixes landed (what worked — one line each; do not redo) -- **#1c/K-D27 R1 two-ledger bridge (2026-07-05)** — async_oort gated selection eligibility ONLY on `all_selected` - (a physical-event-pruned ledger) and never on the aggregator's virtual in-flight set, so a slow-sim returned- - but-uncommitted trainer got re-dispatched (R1 62.9%). Fix: fwdllm agg maintains `_sim_pending_commit` felix- - style (add@dispatch, discard@commit, in-place reconcile in `_sim_hold_busy_slots`, clear on legacy-drop) and - binds it live to `sel._agg_pending_commit_ref`; async_oort `filtered_ends` excludes it (both spots). Reconcile - SHRINKS (committed → re-pickable); the old `|= outstanding` would starve. K-D27b: `outstanding = inflight ∪ - buffer` (NOT `− _sim_committed`, a stale cross-cycle marker that dropped re-dispatched-after-commit trainers → - R1 stayed 67.6% on the bind-only attempt). Sim-only (∅ in real → async_cifar10 byte-identical). `TestPending - CommitExcludedFromSelection` + `TestPendingCommitBridge` (incl. `..._stays_pending_despite_stale_committed`); - full `tests/` 887-green. -- **#14 MQTT join-notify startup race (2026-07-04)** — `backend/mqtt.py` `join()` fired `notify(JOIN)` - fire-and-forget; when the MainThread beat the async `on_connect`, the subscriptions AND the JOIN were dropped - (notify's own `_is_connected` guard) with no retry → the end never registered → async_oort hung below - `minInitialTrainers`. Fix: `join()` calls `_wait_for_connect()` (spin on `_is_connected`, 10s bound) before - subscribe+notify; no-op when already connected → async_cifar10/all examples unchanged. Full `tests/` 879-green. -- **#1c abandon-timeout vclock-key (K-D26, part a)** — the shared `async_oort` 90s `SEND_TIMEOUT_WAIT_S` - in-flight abandon-timeout was keyed on `time.time()` (physical wall); in a slow sim (wall ≫ vclock) it evicted - still-outstanding trainers from `all_selected` → R1 re-dispatch. Now runs on the vclock in sim - (`_abandon_clock_now`: stash `channel_props["vclock_now"]` per-select, stamp + check on it); None in real → - `time.time()` → async_cifar10 byte-identical (its vclock≈wall). `test_sim_vclock_keeps_virtually_recent_end`. - *(Sustained NONE-delete path is the same slow-sim root → #13.)* -- **#6 clock-rate anchor + Root B phase rungs (K-D25)** — root-caused #6 to a checker anchor (sim vclock is - correct; real's full wall carried a ~7.7s/round transport artifact). Agg emits `intrinsic_span_s` (barrier = - max committed intrinsic duration + eval, mirroring the sim vclock composition, **fedavg excluded**); the 5 - clock-rate rungs (K2/K3/K3b/K8/U2) + `wall_disparity` anchor real on the cumulative intrinsic clock, `ts` - fallback → async_cifar10 byte-identical. Root B: `post_train_s` stamped after the delay (pure post-proc); B2 - straggler moved from `_delay_s` into the sct only (`training_budget_s` = base delay). Validated on banked data - (throughput 0.47→0.048, wall_disparity 90→1.6, post_train 0/0, budget 1.14/1.14); full `tests/` 878-green. -- **Phase 4 stopping rule + folds (K-D24)** — `sim_wall_ceiling_s` decoupled to `max_runtime_s × 20` (fixed root - S1, no more wall-truncation of a real-compute sim); B1 `sim_model_eval_time` + B2 `sim_straggler_spread_s:0.9` - ON in the 3 sim yamls; matched `--max-data-id` is the primary stop. *(Sign-off run confirms S1 fixed; folds - did NOT yet lift `sim_rate>1` — see #13.)* -- **Phase 2 speedup-leak skips (K-D23)** — gated the trainer per-round `pause_execution` `sleep(1)` throttle + - `_check_availability` avail-spin `and not self.simulated` (fwdllm-only). The trainer `recv`/`await_join` and - agg grace/`await_join` are LEFT AS-IS (barrier-wait over irreducible agg eval+GPU, not skippable sleeps). -- **Pre-run instrumentation A–E (K-D21)** — Stage-C fwdllm_plus scarcity-wait liveness; A1 trainer phase timing; - A2 agg wall-decomp; A3 advance-rung re-key to `data_id` (shared engine, async byte-identical); A4 - `wall_disparity` DIAG rung; B1/B2/B3 sct folds (config-gated); D1 `task_recv.sim_send_ts`; D2 K5 real-compute - exemption; E `sleep(0.1)`-pad gating. Un-skipped the 12 rigor-gap rungs (#10). -- **Availability params end-to-end (K-D22)** — Phase-1 defaults to syn_0; the pre-flight table reads back the - RESOLVED config (print == run); feasibility gate blocks a full-participation sync barrier under a non-syn_0 trace. -- **Trainer sim clock (Batch 1, K-D2/K-D3)** — additive `sct = send + gpu + D`, no-sleep on the sim path. -- **Async grad loop (Batch 1, K-D4/K-D5)** — `_sim_recv_min_grad` sct reorder buffer + in-flight gate + - agg-goal-boundary rollback cleanup. -- **Sync barrier (K-D11)** — `_sync_sim_recv_first_k` first-k-smallest; the `ends_not_selected_yet` "commit-1-per- - pass" clamp gated real-only so the sim drains the full dynamic-K cohort in one pass. -- **Variance-cadence rung layer (Batch 2, K-D9/K-D10)** — V1–V5 / DK1–DK3 / G1–G2 + the pre-mutation - `cycle_data_id` snapshot that makes V1 exact; DK3/G1 emit deferred with a logged SKIP. -- **Residence: commit-then-carry + R1/W1 rungs (Batch 2.5, K-D12/K-D14)** — killed the 2× recompute; R1 sourced - from an echoed per-contribution interval (immutable across re-dispatch). fwdllm/fwdllm_plus R1 exact 0%. -- **Slot residence realigned to felix (K-D17/K-D17b)** — sim drain keys on `_sim_buffer` not channel RECV; the - re-pick triplet stamped on grad RETURN not dispatch; hold the compute slot to COMMIT so `len(selected_ends)` = - virtual-time in-flight (fixed the in_flight mis-measurement, 2.7→9.5 vs real 9.75). -- **staleness_policy wired from config (K-D15/K-D13)** — was silently `none` for every run; fluxtune set to - `fedbuff` staleness-weighted accept (real+sim identical, a definition not a lever). -- **Clock family re-keyed to `data_id` (#2)** — `_progress_axis`/`_per_progress_last_event`; async_cifar10 - auto-detects `round` → byte-identical. field_coverage accepts fwdllm field aliases (#3). W1 made asymmetric - (flags only sim EXCESS). Launcher fail-fast on early aggregator death (D-d). +## §G Fixes landed (what worked — ≤20-word problem + ≤20-word fix; do not redo) +- **#13 drain stall (K-D28/b/c).** fwdllm's `_sim_recv_min_grad` left a stuck straggler in the expected set → + re-fired the full 30s `RECV_TIMEOUT` every cycle → pipeline starved (`sim_rate` 0.06). Fix: felix-port + stuck-end eviction + recv-grace 2→5s + probe-ceiling/ready-gating + `drain_ready` direct ingest → `sim_rate` + 0.06→0.30, 30s stall gone, R1=0. (Step-4 staggering NEUTRAL, OFF — §H; residual is #12c-bound.) +- **#1c R1 two-ledger bridge (K-D27/b).** async_oort gated selection ONLY on `all_selected` (physical-event-pruned) + and never on the aggregator's virtual in-flight set → slow-sim returned-but-uncommitted trainer re-dispatched + (R1 62.9%). Fix: agg maintains `_sim_pending_commit` felix-style, binds it live to `sel._agg_pending_commit_ref`, + async_oort's `filtered_ends` excludes it; `outstanding = inflight ∪ buffer` (K-D27b, NOT `− _sim_committed`). + Sim-only → async_cifar10 byte-identical. `SIM_R1_DISPATCH` 238→0, confirmed on the latest full run. +- **#14 MQTT join-notify startup race.** `backend/mqtt.py` `join()` fired `notify(JOIN)` fire-and-forget; when the + MainThread beat async `on_connect`, the JOIN + subscriptions dropped with no retry → end never registered → + async_oort hung below `minInitialTrainers`. Fix: `join()` `_wait_for_connect()` (10s bound) before subscribe+notify; + no-op when connected → all examples unchanged. +- **#6 clock-rate anchor + Root B phase rungs (K-D25).** The clock-rate rungs compared sim-vclock against real's + FULL wall, which carries a ~7.7s/round localhost transport ARTIFACT — a checker-anchor bug, the sim vclock was + correct. Fix: agg emits `intrinsic_span_s` (barrier+eval, fedavg excluded); rungs + `wall_disparity` anchor real + on its intrinsic clock (async byte-identical). Validated throughput 0.47→0.048, wall_disparity 90→1.6. +- **Phase-4 stopping rule (K-D24).** `sim_wall_ceiling_s` was truncating a real-compute sim. Fix: decoupled to + `max_runtime_s × 20`; matched `--max-data-id` is the primary stop. Fixed root S1. +- **Residence: commit-then-carry + felix realign (K-D12/K-D14/K-D17b).** Boundary-drop stranded ~7 async + grads/cycle → 2× passes; slot-on-return undercounted in_flight 3×. Fix: carry the surplus, hold the compute slot + to COMMIT (so `len(selected_ends)` = virtual-time in-flight), source R1 from an immutable echoed interval. +- **Clock family re-keyed to `data_id` (#2).** Rungs keyed on `round` mismeasured fwdllm's variance-pass progress. + Fix: `_progress_axis`/`_per_progress_last_event`; async_cifar10 auto-detects `round` → byte-identical. +- **Foundational sim mechanisms (Batch 1–2, K-D2/K-D3/K-D4/K-D5/K-D9/K-D11).** Additive `sct = send + gpu + D`, + no-sleep sim path; `_sim_recv_min_grad` sct reorder buffer + in-flight gate + agg-goal rollback cleanup; sync + `_sync_sim_recv_first_k` first-k-smallest (real-only commit-1-per-pass clamp); V1–V5/DK1–DK3/G1–G2 rungs + the + pre-mutation cycle snapshot that makes V1 exact. +- **Scaffolding (landed, pointer only).** Pre-run instrumentation A–E (K-D21); availability params end-to-end, + print==run (K-D22); Phase-2 real-only speedup-leak skips (K-D23); `staleness_policy` wired from config (K-D13/15). --- ## §H Dead-ends & corrections — do NOT retry - **"#13 step 4 (freed-slot staggered re-dispatch) closes the residual 11-12s drain holds."** NEUTRAL/REFUTED - (K-D28d, `run_..._034550`): `sim_rate` 0.289→0.274, holds persisted. The holds are the drain correctly waiting - (strict sct order) for the earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU - + strict-order + trickle dispatch, not a dispatch-bunching artifact. Staggering feeds an OLD freed-slot vclock - as `send_ts` → gate expects trainers even earlier → more holds. *Lesson:* the fwdllm `sim_rate<1` residual after - steps 1-3 is dominated by **#12c** (no `--delay-factor` → ~no vclock headroom to charge), NOT the drain; measure - with `--delay-factor` before any further drain work. Code kept flag-gated OFF; don't re-enable without a reworked - send_ts model (stamp = actual dispatch vclock, not the old freed-slot vclock). -- **"K-D19 fixed fluxtune R1 / the NONE reset is a red herring."** REFUTED (K-D26). R1 stayed 60.4%. K-D19 - fixed the wrong release path (the aggregator's RETURN-path `cleanup_provided_ends`) and dismissed K-D17b's - NONE hypothesis by checking the *selection filter* (async_oort:1586, which excludes `all_selected` members) — - but the guard fails because `all_selected` MEMBERSHIP is deleted first, by (a) the 90s wall-timeout and (b) - async_oort reading the aggregator's `KEY_END_STATE=NONE` slot-hold as "trainer left." K-D17b was right. - *Lessons:* (i) an R1 fix is not done until the emergent smoke banks R1≤2%; (ii) when a guard "should exclude" - but doesn't, check whether the member is being *deleted* upstream, not just whether the filter reads it. -- **"K-D16 Option-A: free the compute slot on physical RETURN."** WRONG for virtual time — a returned-but- - uncommitted trainer is still in flight until the vclock reaches its sct; freeing the slot undercounted - `in_flight` 3×. Reverted to hold-to-COMMIT (felix-aligned, K-D17b). -- **"K-D16 fixed fluxtune; the re-run just banks numbers."** WRONG — it DEADLOCKED (K-D17): the sim drain was - gated on channel RECV state (not the buffer), and the re-pick triplet was stamped at DISPATCH (freezing the pool - before the first commit). The guard belongs at RETURN; the sim commit path keys on its OWN reorder buffer. -- **K-D6 "drop stranded grads at the agg-goal boundary."** REVERSED for async (K-D12) — the "|selected|≈agg_goal" - premise holds only for sync; fluxtune (c=10≫agg_goal=3) dropped ~7 grads/cycle → residence violation → 2× passes. - Commit-then-carry replaced it. Drop stays correct for sync. -- **"sim wall ≈ real, comparable."** WRONG — always check avg in-flight concurrency, not just total pass counts - (the concurrency collapse hid behind a similar pass count). -- **"D=0 smoke, so clock-family fails are artifacts."** WRONG — the runs are D>0; the fails traced to the rung - being keyed on `round` not `data_id` (#2). -- **Tuning `var_threshold` / `max_iterations_per_data_id` to close a cadence gap** — rejected; baseline-defining + (K-D28d): `sim_rate` 0.289→0.274, holds persisted. The holds are the drain correctly waiting (strict sct order) + for the earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU + strict-order + + trickle dispatch, not a dispatch-bunching artifact. Feeding an OLD freed-slot vclock as `send_ts` makes the gate + expect trainers even earlier → more holds. *Lesson:* the residual after steps 1-3 is **#12c** (no delay-headroom), + NOT the drain. Code kept flag-gated OFF; a reworked attempt would stamp the ACTUAL dispatch vclock. +- **"K-D19 fixed fluxtune R1 / the NONE reset is a red herring."** REFUTED (K-D26). R1 stayed 60.4%: K-D19 fixed + the wrong release path and dismissed the NONE hypothesis by checking the *selection filter*, but `all_selected` + MEMBERSHIP was deleted upstream (90s wall-timeout + async_oort reading `KEY_END_STATE=NONE` as "left"). + *Lessons:* (i) an R1 fix isn't done until the smoke banks R1≤2%; (ii) when a guard "should exclude" but doesn't, + check whether the member is *deleted* upstream, not just whether the filter reads it. +- **"Free the compute slot on physical RETURN" (K-D16 Option-A).** WRONG for virtual time — a returned-but- + uncommitted trainer is still in flight until the vclock reaches its sct; freeing undercounted `in_flight` 3×. + Reverted to hold-to-COMMIT (K-D17b). (The K-D16 re-run also DEADLOCKED, K-D17: drain gated on channel RECV not + the buffer; triplet stamped at DISPATCH froze the pool pre-commit.) +- **"Drop stranded grads at the agg-goal boundary" (K-D6).** REVERSED for async (K-D12) — the "|selected|≈agg_goal" + premise holds only for sync; fluxtune (c=10≫agg_goal=3) dropped ~7/cycle → 2× passes. Drop stays correct for sync. +- **Adding a scalar overhead to the vclock to close #6.** Forbidden (principle #1/#12) — the vclock is + `max(vclock, sct)`. Fold only genuine unmodeled compute (eval_s, straggler spread); artifacts stay off. #6 was a + checker-anchor bug (K-D25), not a missing fold. (Including FedAvg in `intrinsic_span_s` over-charged it → excluded.) +- **Tuning `var_threshold` / `max_iterations_per_data_id` to close a cadence gap.** Rejected — baseline-defining knobs, not parity levers. A cadence gap is an upstream set/order/clock divergence. -- **Adding a scalar overhead to the vclock to close #6** — forbidden (principle #1/#12); the vclock is - `max(vclock, sct)`. Fold only genuine unmodeled compute terms (eval_s, straggler spread); artifacts stay off. -- **"Drive `wall_disparity`→0 by folding more terms into the vclock" (K-D24's #6 plan).** WRONG target — the - eval fold was the LAST genuine term; the residual gap was real's ~7.7s/round localhost transport ARTIFACT, so - folding further would over-charge the vclock (principle #1). #6 was a CHECKER-ANCHOR problem (K-D25): anchor - real on its intrinsic algorithmic clock, and retarget `wall_disparity` to |sim_vclock − real_intrinsic|. -- **Including the FedAvg merge in `intrinsic_span_s`.** Made real intrinsic overshoot the sim vclock by - ~fedavg×cycles (wall_disparity 1.6→5.5); the sim vclock charges barrier+eval but NOT fedavg, so intrinsic must - mirror that composition. (Re-add only if fedavg is ever folded into the sim vclock.) -- **Subtracting the slept delay from `post_train_s` to exclude it (Root B first attempt).** Went NEGATIVE when - the delay wall wasn't in the post window (fake-time tests / sim never sleeps). Correct fix: stamp - `_phase_post_start` AFTER the delay so post_train is pure post-proc, never negative. +- **"sim wall ≈ real, comparable" / "D=0 smoke ⇒ clock fails are artifacts".** WRONG — always check avg in-flight + concurrency (not just pass counts); the runs are D>0 and the fails traced to `round`-vs-`data_id` keying (#2). --- -## §K Deviation log — one line per decision (anchor + rationale; referenced from §B.1) -*Folded/landed entries are one line. Expand only an entry still open/contested.* - +## §K Deviation log — one line per decision (anchor + rationale; referenced from §B.1/§G) - **K-D1** — `time_mode` default `"real"` (not cifar's `"simulated"`): fwdllm's whole corpus is `real`; a - `simulated` default risks half-activating an unbuilt path. Sim variants set it explicitly. + `simulated` default risks half-activating an unbuilt path. - **K-D2** — additive `sct = send + gpu + D` (not `max`): fwdllm's real mode sleeps D on top of GPU time. - **K-D3** — per-eval sct collapses to the train sct (D4): eval lives on the aggregator; forward-grad "train" is a forward pass (no 20× factor); the trainer eval message is a utility report, not a clocked commit. @@ -470,159 +359,45 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso withheld paths key on WEIGHTS semantics; reuse the primitives, fork the orchestration. - **K-D5** — slot release + buffer clear on the AGG-GOAL boundary (not per-commit): a `data_id` spans many cycles with rollbacks; per-commit release would strand a re-contributing trainer (principle #4). -- **K-D6** — boundary buffer-drop DROPPED stranded grads. *Superseded by K-D12* (benign only for sync). -- **K-D7** — U6 sync-barrier visibility-lag telemetry: compute+stash now, emit later (kept Batch 1 a clock port). -- **K-D8** — sim smoke launchers keep D=0 for mechanics-only comparability; D>0 belongs to convergence/parity runs, - enabled in BOTH real and sim together (else mismatched enable is a false divergence). -- **K-D9** — cadence telemetry is a **pre-mutation** cycle snapshot (`cycle_data_id`/`cycle_iteration`/ - `grad_pool_size`/`cached_v_size`): the post-mutation `data_id` advances before emit → off-by-one; snapshot makes - V1 exact for natural-pass and force-commit paths. Existing post-mutation fields unchanged (byte-identical real). -- **K-D10** — DK3 / G1 emit DEFERRED (checker reads-if-present, logged SKIP otherwise): DynamicKC disabled for all - three baselines; G1 needs a trainer-side per-update emit. Wire when a DynamicKC/G1 baseline is added. -- **K-D11** — `ends_not_selected_yet` "commit-1-per-pass" clamp gated real-only: it's a real-transport draining - discipline (queue re-collection); the sim barrier is single-pass (principle #8). -- **K-D12** — fluxtune async: commit-then-CARRY the surplus + hold residence (reverses K-D6 for async); drop stays - correct for sync (c≈agg_goal). -- **K-D13/K-D15** — fluxtune `staleness_policy = fedbuff` staleness-weighted accept (was silently `none` — never - wired from config); set identically real+sim (a definition, not a lever). Provisional lock. +- **K-D9** — cadence telemetry is a **pre-mutation** cycle snapshot: post-mutation `data_id` advances before emit + → off-by-one; snapshot makes V1 exact. Existing post-mutation fields unchanged (byte-identical real). +- **K-D11** — `ends_not_selected_yet` "commit-1-per-pass" clamp gated real-only: a real-transport draining + discipline; the sim barrier is single-pass (principle #8). +- **K-D12** — fluxtune async: commit-then-CARRY the surplus + hold residence (reverses K-D6); drop stays correct + for sync (c≈agg_goal). +- **K-D13/K-D15** — fluxtune `staleness_policy = fedbuff` staleness-weighted accept (was silently `none`); set + identically real+sim (a definition, not a lever). - **K-D14** — R1/W1 sourced from an ECHOED per-contribution interval (not the agg's per-end dispatch stamp, which is overwritten on re-dispatch — exactly when residence is broken). -- **K-D16** — D-e resolution Option A (two-lifetime split). *Slot-on-return half superseded by K-D17b; the - hold-guard-to-COMMIT half survives.* -- **K-D17** — two bugs that deadlocked the K-D16 re-run: (A) drain gated on channel RECV not `_sim_buffer`; - (B) triplet stamped at DISPATCH froze the pool pre-commit. Fixed: drain keys on the buffer; triplet on RETURN. -- **K-D17b** — hold the compute slot to COMMIT (felix-aligned): `len(selected_ends)` now = virtual-time in-flight +- **K-D17b** — hold the compute slot to COMMIT (felix-aligned): `len(selected_ends)` = virtual-time in-flight (fixed in_flight 2.7→9.5). Reverts K-D16's slot-on-return. -- **K-D18** — overnight 10-`data_id` grounding runs; `--max-data-id` default raised to 9999 (was 10, silently - capping 1h runs); cleared #4 truncation for fwdllm. -- **K-D19** — R1 regression fix: defer the re-pick guard release from physical RETURN to COMMIT - (`_release_end_on_return`, fwdllm-only) + a `[SIM_R1_DISPATCH]` tripwire + RETURN-path pytest. **⚠ REFUTED by - the 2026-07-04 sign-off smoke** — R1 still 60.4% (§H dead-ends, #1c reopened); the guard leaks despite the defer. - Root-cause from the tripwire lines next. -- **K-D20** — real per-round wall decomposition (telemetry-only): ~8.7 s/round GENUINE (eval_s 3.34 + straggler - 3.06 + compute) vs ~4.7 s/round ARTIFACT (drain tail 2.48 + `sleep(0.1)` pads) → fold ONLY eval_s + straggler - spread; artifacts stay off the vclock and become optimization targets (#11). Root-caused #7 (real-mode liveness). -- **K-D21** — pre-run instrumentation A–E landed & pytest-green (see §G). -- **K-D22** — availability params respected end-to-end; Phase-1 syn_0 default; print==run (see §G). -- **K-D23** — Phase-2 speedup skips; the "trainer fetch is the dominant leak" hypothesis was REFUTED by telemetry - (it's barrier-wait over irreducible agg eval+GPU); the real skippable wall was two per-round sleeps (see §G). -- **K-D24** — Phase-4 ceiling decouple (×20) + B1/B2 folds ON for the sign-off run (see §G). Sign-off confirmed S1 - fixed. *(#6-via-folds superseded by K-D25 — the residual gap was a checker anchor, not a missing fold.)* -- **K-D25** — #6 is a CHECKER-ANCHOR bug, not a sim under-charge (root-caused from the sign-off #6 telemetry). - The clock-rate rungs compared sim-Δvclock (genuine algorithmic time) against real-Δts (FULL wall = genuine + - a ~constant ~7.7s/round localhost transport artifact real trainers pay via pipelined inter-round fetch; the - agg `barrier_wait_s` reads ~0 in real for the same pipelining reason). Proof: on a non-eval round real burns - 10.9s wall vs the sim's 3.5s genuine, all artifact; on eval rounds real genuine (barrier+eval 15.8) ≈ sim - vclock 15.86 exactly. Fix (fwdllm agg emit + shared checker, async byte-identical): emit `intrinsic_span_s` = - max(committed intrinsic duration) + eval (mirrors the sim vclock; **fedavg excluded**, else overshoot); the 5 - clock-rate rungs + `wall_disparity` anchor real on its cumulative intrinsic clock. **Root B (bundled):** - `post_train_s` stamped after the delay (real stops bundling the slept delay); B2 straggler moved from - `_delay_s` into the sct so `training_budget_s` is a mode-invariant input. `TestIntrinsicSpanAnchor` + emit + - straggler-separation + post_train tests; full `tests/` 878-green. Residual (3 marginal rungs + B2 calibration) - is emergent at the longer sign-off run. #11 real-side waste deferred (zero parity impact; needs a real run). -- **K-D26** — #1c R1 root-caused to a physical-wall vs vclock desync in the SHARED `async_oort` re-pick guard - (`all_selected`), exposed by fluxtune's slow sim (sim_rate 0.26, wall ≫ vclock) and masked in async_cifar10 - (wall ≈ vclock). Evidence (banked sign-off log, principle #11a): trainer 370 dispatched at vclock ~0, its grad - commits at vclock ~7, but the sim burns ~99s wall first → at 91s wall async_oort's 90s `SEND_TIMEOUT_WAIT_S` - (keyed on `time.time()`) evicts it from `all_selected` → re-dispatch → R1. Two deletion paths: **(a)** the 90s - wall-timeout (round-1 trigger, 10 evictions); **(b)** the aggregator marks carried grads `KEY_END_STATE=NONE` - to keep their `selected_ends` slot (`_sim_hold_busy_slots`), and async_oort's recv-handler reads `NONE` as - "left/rejoined" → deletes from `all_selected` (sustained, 113 violations). Corrects K-D19 (fixed the RETURN - path; dismissed the NONE hypothesis by checking the selection FILTER not the DELETION) — K-D17b was right. - **Fix (a) LANDED** (operator: "both — direct fix now + #13 tracked"): `_abandon_clock_now()` runs the timeout - on the vclock in sim — `select()` stashes `channel_props["vclock_now"]`; the dispatch STAMP and the timeout - CHECK both use it (90 = 90 *virtual* s); None in real → `time.time()` → async_cifar10 byte-identical. - `test_sim_vclock_keeps_virtually_recent_end` + reclaim regressions green. **Fix (b)** = the sustained NONE path, - same slow-sim root → cured by **#13** (make the fluxtune sim fast so physical receipt ≈ virtual commit, as in - async_cifar10) or a follow-up aggregator-side re-assertion; the emergent re-run measures how much (a) recovers. -- **K-D27** — #1c R1 DEEPER root-cause from the post-#14 `_2327` telemetry (part (a) recovered ~0; R1 still - 62.9%, 215 `SIM_R1_DISPATCH`). The NONE-delete (K-D26 part b) is one symptom of a **two-ledger split**: - async_oort decides selection eligibility SOLELY from `all_selected` (`filtered_ends`, async_oort:1607), a - ledger it prunes on PHYSICAL events (the recv-fifo 2s re-select loop, RECVD/NONE cleanup) — and NEVER consults - the aggregator's VIRTUAL in-flight truth (`_sim_inflight_expected ∪ _sim_buffer`). The intended bridge - `_agg_pending_commit_ref` is read for telemetry (async_oort:423) but **never assigned** (∅) and never filtered - on; `_sim_pending_commit` is accumulated (fwdllm_aggregator:908) but never cleared or bridged (felix's asyncfl - DOES maintain it, asyncfl:618/1490). fwdllm ported felix's `_sim_hold_busy_slots` (mirror the truth INTO - all_selected) but not the selector-side exclusion, so any physical-schedule prune in the (large, slow-sim) - returned-but-uncommitted window frees a still-in-flight trainer → own select() re-dispatches → R1. Real/ - async_cifar10 mask it via wall≈vclock (window≈0), not by design. **Telemetry proof:** `num_eligible=10` while - `in_flight=10`; `in_pending_commit=0/879` (dead hook); `[CHANNEL_CLEANUP] freed=0`×218; 0 fires of the 90s / - removed-ends prune paths (part (a) confirmed dead). **Fix LANDED:** the fwdllm aggregator maintains - `_sim_pending_commit` felix-style (add on dispatch @2627, discard on COMMIT @811, in-place reconcile to the - still-outstanding set in `_sim_hold_busy_slots`, clear on the legacy-drop boundary) and BINDS it live to the - selector (`sel._agg_pending_commit_ref = self._sim_pending_commit`); async_oort's `filtered_ends` excludes - `_agg_pending_commit_ref` in BOTH spots (list @1607 + loop @1627). The reconcile SHRINKS (a committed trainer - becomes re-pickable) — the old `|= outstanding` accumulate would have starved every committed trainer. Sim-only - (∅ in real; felix leaves the ref unset → async_cifar10 byte-identical). Tests: `TestPendingCommitExcludedFrom - Selection` (selector) + `TestPendingCommitBridge` (aggregator); full `tests/` 886-green. Controlling metric: - `SIM_R1_DISPATCH → 0`, R1 overlap → ~0.2%. - **CORRECTION (K-D27b, `_005222` re-run): the bind alone was INSUFFICIENT — R1 stayed 67.6% (238 tripwire).** - Telemetry proved the filter worked (`in_pending_commit` now 5755 rows; chosen ∩ pending = 0), so the re- - dispatched trainer was in `_sim_inflight_expected` but NOT in pending — only possible if it was in - `_sim_committed`. Root: the reconcile computed `outstanding = (inflight ∪ buffer) − _sim_committed`, but - `_sim_committed` is a STALE cross-cycle marker (cleared only at the boundary). A trainer that committed then - was legitimately re-picked + re-dispatched (re-added to `_sim_inflight_expected`) got dropped from `outstanding` - by that subtraction → re-pickable while its NEW dispatch was in flight → R1. Fix: `outstanding = - inflight ∪ buffer` (drop `− _sim_committed`; a this-cycle commit is already popped from both sets, so the - subtraction was redundant and harmful). `test_recommitted_trainer_stays_pending_despite_stale_committed`; full - `tests/` 887-green. **Awaiting the next fluxtune sim re-run to bank the recovery.** -- **K-D28** — #13 drain-stall step 1 (of the 4-step felix port): fwdllm's `_sim_recv_min_grad` hit the 30s - `RECV_TIMEOUT_WAIT_S` deadline and just `break`, leaving the earliest-expected straggler in - `_sim_inflight_expected` forever → `earlier_stuck` re-fired the full 30s every drain cycle → composer froze → - pipeline starvation (`sim_rate` 0.06). Fix (felix `_sim_recv_min:436-442`): on the deadline track `_stuck_end` - and `pop` it from the expected set (+`_sim_gate_failsafe` counter + `[SIM_GRAD_STUCK_EVICT]`), then commit the - buffered min. Paired knob: fwdllm-scoped `SIM_RECV_GRACE_FLOOR_S` 2→**5s** (override on the fwdllm agg class - only; async_cifar10's base stays 2.0) — the forward-grad GPU can take ~4s so a 2s per-pass recv window closes - before the grad reassembles. **TUNABLE: keep as LOW as correctness allows** (re-measure real GPU wall, pull - toward observed-max + slack). Sim-only path (real never enters the failsafe). `TestStuckEndEviction`; full - `tests/` green. **VALIDATED (`run_..._024637`): stuck-evict fired exactly ONCE (not spinning), R1 stayed 0, - `sim_rate` 0.06→0.20 (~3×), data_id 1→3, pipeline 8-9 deep.** The residual wall is ~42 gaps of 4-10s where the - drain BLOCKS waiting for freshly-dispatched GPU while the vclock idles → steps 2-4. -- **K-D28b** — #13 step 2: probe-ceiling + ready-gating (felix `_sim_recv_min:399-411`). Post-step-1 the drain - still blocked the full 5s grace every pass on every in-flight end, incl. far-future/not-arrived stragglers (the - 4-10s inter-burst gaps). Fix: probe a non-recv_end in-flight end only if physically ready - (`_sim_end_has_ready_msg`, inherited from asyncfl) OR `exp ≤ buffered_min + slack`. Provably safe vs the HOLD - gate (an end triggering `earlier_stuck` at `exp < bmin - slack` always satisfies `exp ≤ bmin + slack`, so it is - never skipped). Sim-only. `TestProbeCeilingReadyGating`. **VALIDATED (`run_..._030300`): no regression (evict=1, - R1=0), `sim_rate` 0.20→0.23, data_id 3→5.** But the trace showed the dominant residual is recv_fifo's per-end 5s - timeout on freshly-dispatched (recv_end) trainers — which step 2 can't touch (recv_ends are always probed). -- **K-D28c** — #13 step 3: direct `drain_ready` ingest (felix `_sim_recv_min:375-390`), flag-gated on - `sim_sct_ordered_drain` (ON in the fluxtune sim yaml). The `run_..._030300` trace root-caused the residual 4-10s - gaps to `recv_fifo`'s streamer BLOCKING the full 5s grace PER not-ready end (`[RECV_FIFO] timeout (5.0s)` fired - on the 2 not-ready ends of every 3-end probe while 1 arrived at ~4.4s GPU). Fix: when on, the fwdllm drain - ingests via `channel.drain_ready(live_inflight, timeout=grace)` — a non-blocking rxq sweep that returns on the - FIRST arrival (no per-end timeout) and is a COMPLETE buffer snapshot (the streamer's background task can strand a - delivered grad → past-dated commit). No ready-gating on this path (non-blocking). Flag-off keeps the step-2 - recv_fifo path; async_cifar10 untouched (fwdllm-only method). Sim-only. `TestSctOrderedDrain`; full - fwdllm+async+parity suites green. **PARTIALLY VALIDATED (`run_..._032235`, killed early during startup): the - drain now CAPS every block at the 5s grace — max aggregate block 34.5s→5.0s, 12-34s recv_fifo blocks GONE, 0 - `[RECV_FIFO] timeout` / streamer lines. Active-loop gaps ~4.3s (pure GPU) once grads flow.** The run only - reached data_id=0 because ~2min was trainer cold-start (267MB distilbert fetch/load over MQTT) — a one-time - startup cost, not the drain; `sim_rate` 0.058 is that unamortized first emit. **VALIDATED (`run_..._032729`, - 6.5min): `sim_rate` 0.23→0.286 (climbing), mean gap 3.19→2.47s, blocks≥5s 54→8, data_id→4, R1=0.** Remaining - wall = 1 stuck-evict (36s, expected) + ~4 multi-pass gate-holds (11-12s) waiting for an earlier-expected - straggler → the **step-4 (freed-slot refill) target**. -- **K-D28d** — #13 step 4: freed-slot refill stamp (felix `_sim_recv_min:513-520` + `_distribute_weights` - staggered path). After step 3 the drain still HELD across 2-3 grace windows (the 11-12s blocks) for an - earlier-expected straggler, because fwdllm stamped every cohort's `sim_send_ts` at one frozen `_round_now`, so - `_sim_inflight_expected` bunched all expected completions together and the min-sct gate waited on the whole - batch. Fix: each grad commit appends the just-advanced vclock to `_sim_free_slot_ts` (inherited deque); - `_distribute_weights_async` pops it (`_pop_free_slot_ts`, FIFO + clamped ≤ live vclock) as each re-dispatched - end's per-end `sim_send_ts` (and `_sim_inflight_expected[end] = sst + budget`), rebuilding a shallow per-end - payload copy carrying its own `SIM_SEND_TS`. Spreads expected completions across the timeline (matching real's - staggered returns) so the gate stops holding for same-expected batches. Composes WITH sct_ordered_drain for - fwdllm (both flags ON) — felix's "supersedes, don't combine" note is async_cifar10-specific; fwdllm's c≫agg_goal - one-grad-per-call loop needs both, and the mechanisms are orthogonal (drain = ingest primitive; staggered = - dispatch-timing, only changing the values the gate reads). fwdllm overrides dispatch (`_distribute_weights_async`) - + drain (`_sim_recv_min_grad`), so felix's async_cifar10 paths are untouched. Flag-off = Batch-1 byte-identical; - real never stamps (`_round_now` None-guarded). `TestFreedSlotRefill`; full fwdllm+async+parity suites green. - **OUTCOME (`run_..._034550`): NEUTRAL — flag DISABLED.** `sim_rate` 0.289→0.274; the 11-12s holds persisted. - The premise was WRONG: disk telemetry (step-3 run, `sct`/`T_v` at every hold) shows each hold commits a grad - whose sct is 0.1-0.4s below the buffered min with ~7 higher-sct grads buffered throughout — the drain is - correctly holding for the earliest-sct in-flight STRAGGLER (strict order), not a same-expected cohort, and the - clock-jump clamp pins the vclock to that straggler's `min_future+slack`. Both wall + vclock bottleneck on the - slowest in-flight trainer: INHERENT to real-GPU + strict-order + trickle dispatch (residence frees 1-2 - slots/commit). Staggering makes it worse (old freed-slot vclock → gate expects trainers even earlier). Code + - test kept, flag OFF. The `sim_rate<1` residual after steps 1-3 is #12c-bound (no `--delay-factor`); measure with - `--delay-factor` before any further drain work. See §H. A reworked attempt would stamp the ACTUAL dispatch - vclock (not the old freed-slot one). +- **K-D21** — pre-run instrumentation A–E landed & pytest-green; un-skipped the 12 rigor-gap rungs (§G). +- **K-D22** — availability params respected end-to-end; Phase-1 syn_0 default; print==run (§G). +- **K-D24** — Phase-4 ceiling decouple (×20) + B1/B2 sct folds; fixed root S1 (§G). +- **K-D25** — #6 is a CHECKER-ANCHOR bug, not a sim under-charge: agg emits `intrinsic_span_s` (barrier+eval, + fedavg excluded), rungs anchor real on its intrinsic clock; Root B stamps `post_train_s` after the delay and + moves the B2 straggler into the sct only. Async byte-identical (§G). +- **K-D26** — #1c R1 root-caused to a physical-wall vs vclock desync in the shared `async_oort` re-pick guard + (`all_selected`), exposed by the slow sim. Fix (a): `_abandon_clock_now()` runs the 90s timeout on the vclock in + sim (None in real → byte-identical). Corrected K-D19. Sustained NONE-delete path → deeper root K-D27. +- **K-D27/b** — the two-ledger split: async_oort's selection eligibility never consulted the aggregator's virtual + in-flight truth. Fix: agg maintains `_sim_pending_commit` felix-style + binds `sel._agg_pending_commit_ref`; + `filtered_ends` excludes it; `outstanding = inflight ∪ buffer` (K-D27b — the `− _sim_committed` subtraction + dropped re-dispatched-after-commit trainers, R1 stuck 67.6%). Sim-only. `SIM_R1_DISPATCH` 238→0 (§G). +- **K-D28/b/c** — #13 drain-stall felix port (3 steps landed): stuck-end eviction + recv-grace floor 2→5s; + probe-ceiling + ready-gating; direct `drain_ready` ingest (flag `sim_sct_ordered_drain`). `sim_rate` 0.06→0.30, + 30s stall gone, R1=0, 0 recv_fifo timeouts (§G). +- **K-D28d** — #13 step-4 freed-slot staggered re-dispatch: IMPLEMENTED but NEUTRAL → flag DISABLED. The residual + holds are inherent strict-sct-order straggler waits, not a dispatch artifact; residual `sim_rate` is #12c-bound + (§H). +- **K-D29** — **REMAINDER-WAIT delay model (reverses K-D2's additive decision).** The order→split-half-var→RNG- + desync root (§A) requires the sim's commit order to equal real's arrival order. That needs a deterministic + per-trainer arrival order, which the flat-additive `gpu+D` model didn't give (order was GPU-jitter-dominated). + Fix (aligned with async_cifar10): real sleeps `max(0, D−gpu)` so the device wall = D (GPU hidden inside); sct = + `send + max(gpu, D)`; per-trainer registry delays supply the completion spread → order = D-order = deterministic + & real↔sim identical. Overrun (gpu>D) is flagged (`training_overran`, `[TIMING_OVERRUN]`) — it un-determinises + order, so it's the fluxtune watch (its JVP GPU 7.57s overruns D/2). Crc32 straggler offset disabled + (`sim_straggler_spread_s=0`); `perturbation_count` made a knob (default 10). See PARITY_LOGICAL_TASKS.md P2. + +*Retired/superseded anchors (kept only as pointers): K-D6 (→K-D12), K-D7/K-D8/K-D10/K-D16/K-D18/K-D19/K-D20/K-D23 +— landed scaffolding or corrections, folded into §G/§H; see git history for detail.* diff --git a/lib/python/examples/fwdllm/trainer/fl_main.py b/lib/python/examples/fwdllm/trainer/fl_main.py index f92d6d8a7..2aab33cd0 100644 --- a/lib/python/examples/fwdllm/trainer/fl_main.py +++ b/lib/python/examples/fwdllm/trainer/fl_main.py @@ -154,6 +154,9 @@ def post_complete_message(tc_args): "var_control": config.hyperparameters.var_control, "perturbation_sampling": config.hyperparameters.perturbation_sampling, "select_perturbation_using_jvp": config.hyperparameters.select_perturbation_using_jvp, + # P2-5: forward-pass count knob (default 10 = historical behavior). + "perturbation_count": getattr( + config.hyperparameters, "perturbation_count", 10), } ) model_args.config["num_labels"] = num_labels diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index 0afccca2c..b0cb4779d 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -507,45 +507,57 @@ def _perform_training(self): self.jvp_for_snr_check = self.trainer.model_trainer.jvp_for_snr_check @timer_decorator - def _emulate_training_delay(self): - """Returns the modeled emulated-delay seconds D (0.0 if delay emulation - is disabled) -- unlike cifar10's trainer, this is a flat additive delay - on top of GPU time, not a budget-minus-actual "sleep to fill" model, so - there is no meaningful overrun/remaining_time_s/training_budget_s concept - here (see ../../../MIGRATING_TO_LAUNCHER.md §9). The caller adds this to - real_gpu_time_s to report sim_round_duration_s. - - In real mode the delay is realized by an actual time.sleep(D); in - simulated mode the sleep is SKIPPED (the aggregator advances a virtual - clock instead) but the same D is still RETURNED so the additive - sim_round_duration_s = real_gpu_time_s + D stays identical across modes. + def _emulate_training_delay(self, gpu_time_s: float = 0.0): + """REMAINDER-WAIT delay model (aligned with async_cifar10; supersedes the + old flat-additive model, K-D2 → K-D29). The modeled mobile device takes + `_delay_s = training_delay_s / factor / speedup`; on our GPU the forward + pass takes `gpu_time_s`, which SHOULD be << the device time (a faithful + emulation of a slow mobile client). So: + + - REAL mode sleeps ONLY the remainder max(0, _delay_s - gpu_time_s) → + real wall this round ≈ _delay_s (the mobile device wall), with GPU + compute hidden inside it. + - SIM mode skips the sleep (charged to the vclock); the sct round + duration is max(gpu, _delay_s) (see train_with_data_id), NOT + gpu + _delay_s. The per-trainer registry delays give the completion + SPREAD, so update ORDER = delay order = deterministic and identical + real↔sim (this is what makes cohort_sequence parity attainable). + + OVERRUN: if gpu_time_s > _delay_s the GPU is slower than the modeled + device (contention / too many trainers-per-GPU / delay_factor too big) — + the emulation is no longer faithful and update ORDER can flip, so we log + [TIMING_OVERRUN] and flag it in telemetry. Returns + (modeled_delay_s, remaining_s, overran). """ # config schema types training_delay_enabled as bool (default False) # but historical launcher yamls pass the string "True"; accept both so # the modeled delay is not silently dropped to 0. _enabled = self.training_delay_enabled in (True, "True", "true") - if _enabled: - # Eval is 3X faster than training on CPU - # Eval on NPUs is 10-50X is faster than training on CPUs. We could take 20X if we wanted to consider an all-NPU client cohort for Eval (NPUs don't support training) - eval_delay = self.training_delay_s / self.training_delay_factor - _delay_s = eval_delay / self.speedup_factor - if self.simulated: - logger.info( - f"time_mode=simulated: modeled eval delay for trainer " - f"{self.trainer_id} = {_delay_s}s (not slept; charged to vclock)." - ) - else: - time.sleep(_delay_s) - logger.info( - f"Delayed eval time for trainer " - f"{self.trainer_id} by {eval_delay}s. Sleeping for {_delay_s}s." - ) - # Returns the BASE modeled delay (identical real↔sim). The B2 straggler - # spread is NOT added here -- it belongs in the sct (sim_round_duration_s, - # below) so the emitted training_budget_s stays a mode-invariant INPUT - # (T2), while the sct still carries the per-trainer dispersion (#6/Root B). - return _delay_s - return 0.0 + if not _enabled: + return 0.0, 0.0, False + _delay_s = (self.training_delay_s / self.training_delay_factor) / self.speedup_factor + _remaining_s = max(0.0, _delay_s - gpu_time_s) + _overran = gpu_time_s > _delay_s + if _overran: + logger.warning( + f"[TIMING_OVERRUN] trainer {self.trainer_id} data_id={self.data_id} " + f"iter={self.iteration_per_data_id}: gpu={gpu_time_s:.2f}s > " + f"budget={_delay_s:.2f}s (excess={gpu_time_s - _delay_s:.2f}s) — " + f"emulation unfaithful, update order may flip. Reduce trainers/GPU " + f"or raise training_delay_factor." + ) + if self.simulated: + logger.info( + f"time_mode=simulated: modeled delay for trainer {self.trainer_id} " + f"= {_delay_s:.3f}s (not slept; charged to vclock; gpu={gpu_time_s:.3f}s)." + ) + elif _remaining_s > 0: + time.sleep(_remaining_s) + logger.info( + f"Trainer {self.trainer_id} slept remainder {_remaining_s:.3f}s " + f"(budget {_delay_s:.3f}s - gpu {gpu_time_s:.3f}s)." + ) + return _delay_s, _remaining_s, _overran def _sim_straggler_offset_s(self) -> float: """B2 (K-D20 #6): the modeled delay D is flat across trainers, so the @@ -588,9 +600,10 @@ def train_with_data_id(self): self._perform_training() _real_gpu_time_s = time.time() - _round_start_ts - # emulate delays in training (due to compute resource and/or - # dataset size and/or network latency) - _delay_s = self._emulate_training_delay() + # emulate the mobile-device delay via the REMAINDER-WAIT model: real + # sleeps max(0, delay - gpu); sim skips it. Returns the modeled budget, + # the remainder actually waited, and whether the GPU overran the budget. + _delay_s, _remaining_s, _overran = self._emulate_training_delay(_real_gpu_time_s) # post_train phase starts AFTER the modeled delay (Root B / #6): real # SLEEPS _delay_s above (the modeled-latency term, compared via @@ -615,13 +628,15 @@ def train_with_data_id(self): float(getattr(_hp, "sim_wan_transfer_s", 0.0) or 0.0) if (self.simulated and _hp is not None) else 0.0 ) - # B2 straggler spread lives HERE (in the sct / modeled round duration), - # NOT in _delay_s, so the emitted training_budget_s (the modeled-delay - # INPUT) stays identical real↔sim (T2) while the sct still carries the - # per-trainer completion dispersion the sync barrier needs (#6/Root B). - # sim-only (the offset is 0 in real). + # REMAINDER-WAIT sct (K-D29): the modeled round wall is max(gpu, delay), + # NOT gpu + delay — the mobile device's compute is HIDDEN inside its + # delay budget (real slept only the remainder above; sim charges the + # same max() to the vclock). The per-trainer registry delays supply the + # completion spread, so the legacy crc32 straggler offset is redundant + # (kept flag-gated at 0 in the sim yamls; supersedes #6/Root B B2). + # _wan_s stays a documented knob at 0 (no localhost ground truth). self._sim_round_duration_s = ( - _real_gpu_time_s + _delay_s + self._sim_straggler_offset_s() + _wan_s + max(_real_gpu_time_s, _delay_s) + self._sim_straggler_offset_s() + _wan_s ) if self.simulated: _leg = self.sim_completion_leg_s @@ -668,6 +683,11 @@ def train_with_data_id(self): "gpu_compute_s": _real_gpu_time_s, "post_train_s": _post_train_s, "training_budget_s": _delay_s, + # Remainder-wait model (K-D29): what real actually slept + + # whether the GPU overran the modeled device budget (P2-6: + # an overrun can flip update order → cohort_sequence break). + "remaining_time_s": _remaining_s, + "training_overran": _overran, "trainer_phase": ( f"{self._round}/{self.data_id}/{self.iteration_per_data_id}" ), diff --git a/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py b/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py index eba07e169..825268332 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py @@ -205,6 +205,18 @@ def __init__( if self.args.select_perturbation_using_jvp: self.select_perturbation_using_jvp = self.args.select_perturbation_using_jvp + # Number of candidate perturbations sampled per param (P2-5). Drives the + # forward-pass count: the select_perturbation_using_jvp path does 2 + # forward passes (JVP) PER perturbation, so N perturbations = ~2N passes + # → the dominant fluxtune GPU cost. Default 10 (byte-identical to the + # historical hardcode); a config knob so the JVP cost can be tuned to + # keep GPU << the modeled mobile delay (the remainder-wait invariant). + # Real and sim MUST use the same value (they read the same config). + try: + self.perturbation_count = int(getattr(self.args, "perturbation_count", 10) or 10) + except (TypeError, ValueError): + self.perturbation_count = 10 + # var control TODO: It is not layer id it is param id. Distilbert for eg # has only 6 layers. if self.args.model_type == "distilbert": @@ -303,9 +315,9 @@ def _select_optimal_perturbations(self, device, logging_state): if self.grad is not None and v.requires_grad: self.total_rng_iter += 1 shape = v.shape - candidate_v = _randn_wrapper((1 * 10, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) + candidate_v = _randn_wrapper((self.perturbation_count, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) logging.debug(f"Candidate v - random generation for layer - '{index}' layer shape {candidate_v.shape}") - # torch.randn((1 * 10, *shape), device="cpu", generator=self.torch_rng) + # torch.randn((self.perturbation_count, *shape), device="cpu", generator=self.torch_rng) target_grad = self.grad[index] target_grad = torch.flatten(target_grad) @@ -342,8 +354,8 @@ def _select_optimal_perturbations(device, logging_state): if self.grad is not None and v.requires_grad: self.total_rng_iter += 1 shape = v.shape - candidate_v = _randn_wrapper((1 * 10, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) - # torch.randn((1 * 10, *shape), device="cpu", generator=self.torch_rng) + candidate_v = _randn_wrapper((self.perturbation_count, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) + # torch.randn((self.perturbation_count, *shape), device="cpu", generator=self.torch_rng) target_grad = self.grad[index] target_grad = torch.flatten(target_grad) @@ -411,7 +423,7 @@ def _select_optimal_perturbations(device, logging_state ): if v.requires_grad: self.total_rng_iter += 1 shape = v.shape - candidate_v = _randn_wrapper((1 * 10, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) + candidate_v = _randn_wrapper((self.perturbation_count, *shape), device="cpu", generator=self.torch_rng, logging_state=logging_state, param_name=k) candidate_v = torch.flatten(candidate_v, start_dim=1) logging.info(f"len of candidate_v {len(candidate_v)}") @@ -430,7 +442,7 @@ def _select_optimal_perturbations(device, logging_state ): del candidate_v, target_grad, cos_sim, sorted_indices, shape else: v_buffer[index] = [ - candidate_v[i].reshape(v.shape) for i in range(0, 10) + candidate_v[i].reshape(v.shape) for i in range(0, self.perturbation_count) ] del candidate_v, shape index += 1 @@ -440,7 +452,7 @@ def _select_optimal_perturbations(device, logging_state ): jvp_all_perturbations = [] - for i in range(0,10): + for i in range(0, self.perturbation_count): v_params = _prepare_perturbation_tensors(device, v_buffer, i) loss, jvp = _compute_forward_jvp(device, x, labels, v_params) # logging.info(f"Jvp of option: {jvp}") diff --git a/lib/python/examples/fwdllm/trainer/main.py b/lib/python/examples/fwdllm/trainer/main.py index b821fb4fb..d85a51e76 100644 --- a/lib/python/examples/fwdllm/trainer/main.py +++ b/lib/python/examples/fwdllm/trainer/main.py @@ -118,6 +118,9 @@ def post_complete_message(tc_args): "var_control": config.hyperparameters.var_control, "perturbation_sampling": config.hyperparameters.perturbation_sampling, "select_perturbation_using_jvp": config.hyperparameters.select_perturbation_using_jvp, + # P2-5: forward-pass count knob (default 10 = historical behavior). + "perturbation_count": getattr( + config.hyperparameters, "perturbation_count", 10), } ) model_args.config["num_labels"] = num_labels diff --git a/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py index 85a41d093..5da839792 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_phase_timing.py @@ -102,8 +102,9 @@ def _check_availability(self): def _perform_training(self): pass - def _emulate_training_delay(self): - return self._delay_s + def _emulate_training_delay(self, gpu_time_s=0.0): + # K-D29 remainder-wait signature: (modeled_delay, remaining, overran). + return self._delay_s, max(0.0, self._delay_s - gpu_time_s), False class TestTrainWithDataIdEmitsPhases: diff --git a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py index 9c674c5c6..ffa497d78 100644 --- a/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py +++ b/lib/python/tests/mode/test_fwdllm_trainer_sim_duration.py @@ -1,16 +1,19 @@ # Copyright 2026 Cisco Systems, Inc. and its affiliates # SPDX-License-Identifier: Apache-2.0 -"""fwdllm's trainer (FedSgdTrainer.py) reported only real_gpu_time_s, not -sim_round_duration_s -- unlike async_cifar10's trainer, which reports total -round wall time (gpu + modeled delay). fwdllm has no budget-vs-actual -contention model (its delay is a flat additive sleep, not a sleep-to-fill- -budget pattern), so only sim_round_duration_s is added here -- NOT -training_budget_s/overran/remaining_time_s, which would need a budget -concept fwdllm doesn't have (see ../../examples/MIGRATING_TO_LAUNCHER.md §9). - -This covers _emulate_training_delay()'s return-value change: it now returns -the seconds actually slept (0.0 if delay emulation is disabled), which the -caller adds to real_gpu_time_s to report sim_round_duration_s. +"""fwdllm's trainer sim-duration / delay model (K-D29 REMAINDER-WAIT, replacing +the earlier flat-additive model, K-D2). + +The modeled mobile device takes ``_delay_s = training_delay_s/factor/speedup``. +On our GPU the forward pass takes ``gpu_time_s`` (SHOULD be << device time). So: + - REAL mode sleeps only the remainder ``max(0, _delay_s - gpu)`` -> real wall + ≈ _delay_s, GPU hidden inside it. + - SIM mode skips the sleep; the sct round duration is ``max(gpu, _delay_s)`` + (NOT gpu + _delay_s). Per-trainer registry delays supply the completion + SPREAD -> update order = delay order = deterministic + identical real↔sim. + - OVERRUN: gpu > _delay_s => emulation unfaithful; flagged (remaining==0). + +``_emulate_training_delay(gpu_time_s)`` returns +``(modeled_delay_s, remaining_s, overran)``. """ import os @@ -33,7 +36,6 @@ class _FakeTrainer: touches; binds the real method under test.""" _emulate_training_delay = FedSGDTrainer._emulate_training_delay - _sim_straggler_offset_s = FedSGDTrainer._sim_straggler_offset_s def __init__(self, training_delay_enabled, training_delay_s=0.0, training_delay_factor=1.0, speedup_factor=1.0, simulated=False): @@ -41,79 +43,81 @@ def __init__(self, training_delay_enabled, training_delay_s=0.0, self.training_delay_s = training_delay_s self.training_delay_factor = training_delay_factor self.speedup_factor = speedup_factor - # Batch 1: real mode (simulated=False) sleeps D; sim mode skips the - # sleep but still returns the same modeled D. self.simulated = simulated self.trainer_id = "t1" + self.data_id = 3 + self.iteration_per_data_id = 0 -class TestEmulateTrainingDelayReturnsSleptSeconds: - def test_returns_zero_when_disabled(self): +class TestEmulateTrainingDelayRemainderWait: + def test_returns_zero_tuple_when_disabled(self): t = _FakeTrainer(training_delay_enabled="False", training_delay_s=10.0) - assert t._emulate_training_delay() == 0.0 - - def test_returns_computed_delay_when_enabled(self): - t = _FakeTrainer( - training_delay_enabled="True", training_delay_s=3.0, - training_delay_factor=1.0, speedup_factor=1.0, - ) - assert t._emulate_training_delay() == 3.0 - - def test_speedup_factor_scales_the_returned_delay(self): - """The returned value must match what was actually slept (eval_delay - / speedup_factor), not the unscaled eval_delay -- otherwise - sim_round_duration_s would overstate the real wall time under a - speedup.""" - t = _FakeTrainer( - training_delay_enabled="True", training_delay_s=10.0, - training_delay_factor=2.0, speedup_factor=5.0, - ) - # eval_delay = 10.0 / 2.0 = 5.0; slept = 5.0 / 5.0 = 1.0 - assert t._emulate_training_delay() == 1.0 - - -class TestNoSleepOnSimPath: - """Batch 1 core invariant: in simulated mode the trainer must NOT - time.sleep to emulate the delay -- the aggregator advances a virtual clock - instead -- yet must STILL return the same modeled D so the additive - sim_round_duration_s = real_gpu + D is identical across modes.""" - - def test_real_mode_sleeps_the_modeled_delay(self, monkeypatch): + assert t._emulate_training_delay(0.2) == (0.0, 0.0, False) + + def test_modeled_delay_and_remainder_when_gpu_below_budget(self): + # delay = 4.0/2.0/1.0 = 2.0; gpu = 0.5 -> remaining = 1.5, no overrun. + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0) + modeled, remaining, overran = t._emulate_training_delay(0.5) + assert modeled == 2.0 and remaining == 1.5 and overran is False + + def test_speedup_factor_scales_the_modeled_delay(self): + # eval_delay = 10/2 = 5; modeled = 5/5 = 1.0; gpu 0.25 -> remaining 0.75. + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=10.0, + training_delay_factor=2.0, speedup_factor=5.0) + modeled, remaining, overran = t._emulate_training_delay(0.25) + assert modeled == 1.0 and remaining == 0.75 and overran is False + + def test_overrun_when_gpu_exceeds_budget(self): + # gpu 3.0 > budget 2.0 -> overran, remaining clamped to 0. + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0) + modeled, remaining, overran = t._emulate_training_delay(3.0) + assert modeled == 2.0 and remaining == 0.0 and overran is True + + +class TestSleepOnlyTheRemainderInRealMode: + def test_real_mode_sleeps_the_remainder(self, monkeypatch): + slept = [] + monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0, + simulated=False) + modeled, remaining, _ = t._emulate_training_delay(0.5) + assert modeled == 2.0 and remaining == 1.5 + assert slept == [1.5] # ONLY the remainder, not the full delay + + def test_real_mode_overrun_sleeps_nothing(self, monkeypatch): slept = [] monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) - t = _FakeTrainer( - training_delay_enabled="True", training_delay_s=4.0, - training_delay_factor=2.0, speedup_factor=1.0, simulated=False, - ) - d = t._emulate_training_delay() - # eval_delay = 4.0/2.0 = 2.0; slept = 2.0/1.0 = 2.0 - assert d == 2.0 - assert slept == [2.0] - - def test_sim_mode_does_not_sleep_but_still_returns_delay(self, monkeypatch): + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=2.0, + training_delay_factor=1.0, speedup_factor=1.0, + simulated=False) + t._emulate_training_delay(5.0) # gpu > budget + assert slept == [] # nothing to sleep; overran + + def test_sim_mode_does_not_sleep(self, monkeypatch): slept = [] monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) - t = _FakeTrainer( - training_delay_enabled="True", training_delay_s=4.0, - training_delay_factor=2.0, speedup_factor=1.0, simulated=True, - ) - d = t._emulate_training_delay() - assert d == 2.0 # same modeled D as real mode - assert slept == [] # but NOTHING was slept on the sim path - - def test_sim_mode_disabled_returns_zero_no_sleep(self, monkeypatch): + t = _FakeTrainer(training_delay_enabled="True", training_delay_s=4.0, + training_delay_factor=2.0, speedup_factor=1.0, + simulated=True) + modeled, remaining, _ = t._emulate_training_delay(0.5) + assert modeled == 2.0 and remaining == 1.5 # same modeled math as real + assert slept == [] # but NOTHING slept in sim + + def test_disabled_sim_no_sleep(self, monkeypatch): slept = [] monkeypatch.setattr(_fst_module.time, "sleep", lambda s: slept.append(s)) t = _FakeTrainer(training_delay_enabled="False", training_delay_s=9.0, simulated=True) - assert t._emulate_training_delay() == 0.0 + assert t._emulate_training_delay(0.1) == (0.0, 0.0, False) assert slept == [] class _FakeTime: - """Scripted time source so the additive-stamp arithmetic is deterministic. - Rebound only onto the FedSgdTrainer module's `time` name (not the shared - time module), so timer_decorator's own runtime.time is untouched.""" + """Scripted time source so the sct arithmetic is deterministic. Rebound only + onto FedSgdTrainer's `time` name (not the shared module).""" def __init__(self, ticks): self._ticks = list(ticks) @@ -130,11 +134,10 @@ def sleep(self, _s): # must never be called on the sim path class _StampTrainer: """Binds the real train_with_data_id onto a minimal stand-in, stubbing the - heavy compute so only the sim-stamp arithmetic is exercised.""" + heavy compute so only the sim-stamp arithmetic is exercised. No config -> + straggler spread 0 -> offset 0.""" train_with_data_id = FedSGDTrainer.train_with_data_id - # train_with_data_id now folds the B2 straggler into the sct (#6/Root B); - # no config -> spread 0 -> offset 0 -> additive duration preserved. _sim_straggler_offset_s = FedSGDTrainer._sim_straggler_offset_s def __init__(self, sim_send_ts, delay_d, leg_s=0.0): @@ -156,35 +159,38 @@ def _check_availability(self): def _perform_training(self): pass # no GPU work; wall time is scripted via _FakeTime - def _emulate_training_delay(self): - return self._delay_d # modeled D (no sleep in sim mode) + def _emulate_training_delay(self, gpu_time_s=0.0): + # modeled D, remainder (irrelevant in sim), no overrun + return self._delay_d, max(0.0, self._delay_d - gpu_time_s), False -class TestSimCompletionStampIsAdditive: - """The sct the aggregator orders by must be ADDITIVE - (sim_round_duration = real_gpu + D), matching fwdllm real mode's - sleep-D-on-top-of-GPU semantics -- NOT cifar10's max(gpu, D). And - _sim_completion_ts = _sim_send_ts + sim_round_duration + leg (K-D2).""" +class TestSimCompletionStampIsMaxGpuDelay: + """K-D29: the sct the aggregator orders by is ``max(gpu, D)`` (the mobile + device wall, GPU hidden inside), NOT the old additive gpu + D, and + _sim_completion_ts = _sim_send_ts + max(gpu, D) + leg.""" - def test_additive_round_duration_and_completion_ts(self, monkeypatch): + def test_max_round_duration_and_completion_ts(self, monkeypatch): monkeypatch.setattr(_fst_module.telemetry, "is_enabled", lambda: False) - # ticks: phase_entry=100.0, round_start=100.0, gpu-end=100.5 (+clamp) - # -> real_gpu = 0.5s. (Stage A1 added the phase_entry tick.) + # real_gpu = 0.5s; D = 2.0 -> max(0.5, 2.0) = 2.0 (additive would be 2.5). monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.0, 100.5])) t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=0.0) - t.train_with_data_id() + assert t._sim_round_duration_s == 2.0 + assert t._sim_completion_ts == 12.0 # 10.0 + 2.0 + 0.0 - # ADDITIVE: 0.5 (gpu) + 2.0 (D) = 2.5 (max(gpu,D) would be 2.0) - assert t._sim_round_duration_s == 2.5 - # completion = send(10.0) + duration(2.5) + leg(0.0) - assert t._sim_completion_ts == 12.5 + def test_gpu_dominates_when_over_budget(self, monkeypatch): + monkeypatch.setattr(_fst_module.telemetry, "is_enabled", lambda: False) + # real_gpu = 3.0s; D = 2.0 -> max = 3.0 (the overrun case). + monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.0, 103.0])) + t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=0.0) + t.train_with_data_id() + assert t._sim_round_duration_s == 3.0 + assert t._sim_completion_ts == 13.0 def test_completion_ts_includes_leg(self, monkeypatch): + monkeypatch.setattr(_fst_module.telemetry, "is_enabled", lambda: False) monkeypatch.setattr(_fst_module, "time", _FakeTime([100.0, 100.0, 100.5])) t = _StampTrainer(sim_send_ts=10.0, delay_d=2.0, leg_s=1.5) - t.train_with_data_id() - - assert t._sim_round_duration_s == 2.5 - assert t._sim_completion_ts == 14.0 # 10.0 + 2.5 + 1.5 + assert t._sim_round_duration_s == 2.0 + assert t._sim_completion_ts == 13.5 # 10.0 + 2.0 + 1.5 diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 6ea14c55a..29fb5f055 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -1157,3 +1157,103 @@ def test_present_in_run_all_and_meta(self): assert "w1_compute_conservation" in pc.CHECK_META assert "r1_inflight_overlap" in pc.CHECK_META["v1_iter_per_data_id"]["deps"] assert pc.CHECK_META["w1_compute_conservation"]["deps"] == ("r1_inflight_overlap",) + + +def _lcyc(data_id, iteration, cohort, var, var_good=False, force=False, goal=3): + """One fwdllm variance-cadence cycle event (cohort in receive/commit order).""" + return {"event": "agg_round", "round": 1, + "cycle_data_id": data_id, "iteration_per_data_id": iteration, + "contributing_trainers": list(cohort), "var": var, + "var_good_enough": var_good, "force_commit_planned": force, + "agg_goal_count": goal, "staleness": [0] * len(cohort)} + + +class TestCohortSequence: + """L1 cohort_sequence_parity: the ordered per-aggregation logical sequence + (set + receive-ORDER + cadence + var value) must be IDENTICAL. EXACT, ungated + (real receive-order is deterministic in both modes by design).""" + + def test_identical_sequence_passes(self): + cyc = [_lcyc(0, 1, ["a", "b", "c"], 0.9), + _lcyc(0, 2, ["a", "b", "c"], 0.28, var_good=True), + _lcyc(1, 1, ["a", "b", "c"], 0.5)] + r = pc.cohort_sequence_parity(_agg(agg_rounds=list(cyc)), + _agg(agg_rounds=list(cyc))) + assert r["ok"], r + assert r["order_match_frac"] == 1.0 and r["var_match_frac"] == 1.0 + + def test_reordered_cohort_same_set_FAILS(self): + # Same SET each cycle, different receive ORDER -> feeds split-half var. + real = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b", "c"], 0.9)]) + sim = _agg(agg_rounds=[_lcyc(0, 1, ["c", "a", "b"], 0.9)]) + r = pc.cohort_sequence_parity(real, sim) + assert not r["ok"], r + assert r["set_match_frac"] == 1.0 and r["order_match_frac"] == 0.0 + assert r["first_divergence"]["order_ok"] is False + + def test_different_cohort_FAILS(self): + real = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b", "c"], 0.9)]) + sim = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b", "d"], 0.9)]) + r = pc.cohort_sequence_parity(real, sim) + assert not r["ok"] and r["set_match_frac"] == 0.0 + + def test_var_divergence_FAILS_even_with_matched_order(self): + # Identical cohort+order, var off by >0.1% -> the RNG-desync tell. + real = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b", "c"], 0.371605)]) + sim = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b", "c"], 0.371067)]) + r = pc.cohort_sequence_parity(real, sim) + assert not r["ok"] and r["var_match_frac"] == 0.0 + assert r["first_divergence"]["var_ok"] is False + + def test_cadence_shift_FAILS(self): + real = _agg(agg_rounds=[_lcyc(0, 1, ["a"], 0.5), _lcyc(1, 0, ["a"], 0.2, var_good=True)]) + sim = _agg(agg_rounds=[_lcyc(0, 1, ["a"], 0.5), _lcyc(0, 2, ["a"], 0.4)]) # extra iter + r = pc.cohort_sequence_parity(real, sim) + assert not r["ok"] and r["cadence_match_frac"] < 1.0 + + def test_tier_exact_and_enforced(self): + # EXACT tier -> enforced FAIL even under --lenient (not a DIST/DIAG warn). + r = pc.cohort_sequence_parity( + _agg(agg_rounds=[_lcyc(0, 1, ["a"], 0.5)]), + _agg(agg_rounds=[_lcyc(0, 1, ["b"], 0.5)])) + assert r["tier"] == "EXACT" and r["ok"] is False + passed, roots, _down, _warn = pc.overall_verdict( + {"cohort_sequence": r}, lenient=True) + assert not passed and "cohort_sequence" in roots + + def test_skips_on_non_fwdllm(self): + # No cadence fields (async_cifar10 shape) -> clean SKIP, byte-identical. + plain = {"event": "agg_round", "round": 0, "ts": 0.0, + "contributing_trainers": ["a"], "staleness": [0]} + rd = _agg(agg_rounds=[plain]) + r = pc.cohort_sequence_parity(rd, rd) + assert r.get("status") == "SKIP" and r["ok"] + + def test_max_bin_windows_to_first_bin(self): + # Cohorts match on bin 0, diverge on bin 1 -> --max-bin 0 passes. + real = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b"], 0.5), _lcyc(1, 1, ["a", "b"], 0.5)]) + sim = _agg(agg_rounds=[_lcyc(0, 1, ["a", "b"], 0.5), _lcyc(1, 1, ["b", "a"], 0.5)]) + assert pc.cohort_sequence_parity(real, sim, max_bin=0)["ok"] + assert not pc.cohort_sequence_parity(real, sim)["ok"] + + def test_present_in_run_all_and_meta(self): + assert "cohort_sequence" in pc.CHECK_META + assert pc.CHECK_META["cohort_sequence"]["deps"] + + +class TestVarTrajectoryMeanGuard: + """V2 must fail a systematic mean offset that KS alone misses (a uniform ~1% + shift barely moves the CDF -> KS~0 but grads have desynced).""" + + def test_systematic_offset_fails_despite_low_ks(self): + base = [0.9, 0.5, 0.42, 0.31, 0.6, 0.48] + real = _agg(agg_rounds=[_lcyc(0, i, ["a"], v) for i, v in enumerate(base)]) + sim = _agg(agg_rounds=[_lcyc(0, i, ["a"], v * 1.05) for i, v in enumerate(base)]) + r = pc.var_trajectory_parity(real, sim) + assert r["mean_rel_diff"] > r["mean_tol_rel"], r + assert not r["ok"], r + + def test_matched_var_passes(self): + base = [0.9, 0.5, 0.42, 0.31] + agg = _agg(agg_rounds=[_lcyc(0, i, ["a"], v) for i, v in enumerate(base)]) + assert pc.var_trajectory_parity(agg, agg)["ok"] From e5006e62a93a18d92cd9e8a42cd496be6c432c5d Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 15:00:29 -0400 Subject: [PATCH 29/56] fwdllm sim: full-cohort determinism gate + timing_overrun + per-step GPU telemetry (K-D30) Pre-unavailability (Phase-1) checker rigor + GPU-optimization telemetry, ahead of the databin1 validation run. No workload behavior change; sim/checker-only, async_cifar10 byte-identical. Checker (P1-5 / K-D30): - Data-driven full-cohort determinism gate (_selection_is_deterministic / _full_cohort_selection): un-gates selection/aggregation_sequence/utility to GENUINE enforcement when num_chosen==num_candidates in both modes (fwdllm syn_0, K=all); fluxtune (agg_goal mechanical KS on fwdllm's data_id axis). - New timing_overrun DIAG rung: surfaces the P2-6 training_overran fraction + first-overrun bin (the K-D29 order-determinism tell). Wired + re-exported. - P1-4 (async_cifar10 cohort adapter) assessed redundant, not built. GPU telemetry (P2-4): - EVENT_STEP_TIMING + build_step_timing; timer_decorator emits a structured per-function record when a trainer fwd_llm_stage is present (no-op when telemetry off; guarded so nested helpers never mis-attribute). - plot_step_timing.py: fine step_timing + coarse trainer_round phases -> table + PNGs (exact-baseline discovery via _RUN_RE). Banked-log finding: real wall is dominated by mqtt_fetch_s (fluxtune 18.3s/round) >> gpu_compute. - train_with_data_id sets fwd_llm_stage AFTER the abort check so an aborted round stays a true no-op. Trainer: fixed two stale K-D29 comments (still described the reversed additive delay model). Tests: 15 new (determinism gate x6, timing_overrun x4, step_timing builder + decorator emit x5). 735 mode/telemetry/selector/parity green. Banked scoreboard stable (fwdllm 41/13, fwdllm_plus 36/17, fluxtune 35/19; +1 timing_overrun SKIP). Co-Authored-By: Claude Opus 4.8 --- .../async_cifar10/scripts/parity/checks.py | 181 +++++++++++++- .../async_cifar10/scripts/parity_checks.py | 4 + .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 59 +++-- .../fwdllm/expt_scripts/plot_step_timing.py | 222 ++++++++++++++++++ lib/python/examples/fwdllm/simulate_fwdllm.md | 34 ++- .../trainer/forward_training/FedSgdTrainer.py | 39 +-- lib/python/flame/monitor/runtime.py | 19 ++ lib/python/flame/telemetry/events.py | 31 +++ lib/python/tests/mode/test_parity_checks.py | 107 +++++++++ .../tests/telemetry/test_event_schema.py | 70 ++++++ 10 files changed, 713 insertions(+), 53 deletions(-) create mode 100644 lib/python/examples/fwdllm/expt_scripts/plot_step_timing.py diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index 98ca4aef6..99df020b4 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -875,10 +875,15 @@ def _by_round_selection(selection_train: list) -> dict: # Selectors whose per-round SET selection is a deterministic function of -# (candidate set, seed). Currently empty — every shipped selector samples -# from a join-order-dependent candidate list, making exact per-round set -# identity unattainable across real/sim. participation_parity is the -# enforced selection invariant for stochastic selectors. +# (candidate set, seed) INDEPENDENT of the draw. Currently empty — every +# shipped selector samples from a join-order-dependent candidate list, so a +# stochastic SUBSET draw is not per-round set-identical across real/sim. But +# set-identity is ALSO attainable without a deterministic selector when the +# run is FULL-COHORT (K >= candidate pool → everyone selected): that case is +# detected data-drivenly by `_full_cohort_selection` / `_selection_is_ +# deterministic`, which un-gates the set/sequence rungs for fwdllm (syn_0, +# K=all) while leaving fluxtune/fwdllm_plus gated. participation_parity is the +# enforced selection invariant for the remaining stochastic-subset case. DETERMINISTIC_SELECTORS: set = set() @@ -892,12 +897,76 @@ def _selector_name(*loaded: dict) -> str: return "" +def _has_cohort_counts(loaded: dict) -> bool: + """True iff selection telemetry carries the num_chosen/num_candidates fields + the full-cohort gate needs (real runs always do; synthetic/legacy may not).""" + for e in loaded.get("selection_train", []): + if e.get("num_candidates") is not None or e.get("num_chosen") is not None: + return True + return False + + +def _full_cohort_selection(loaded: dict) -> bool: + """True iff EVERY selection round chose the whole candidate pool + (num_chosen == num_candidates, pool > 0). + + When K >= the candidate pool the selected SET is the entire pool — a + deterministic function of the pool regardless of the stochastic draw — so + the set/sequence selection rungs become exact-enforceable (fwdllm at + syn_0, K=all). Under scarcity (K < pool) or ASYMMETRIC eligibility across + modes (fwdllm_plus #7: real ~4.9 vs sim ~9.6 eligible → num_chosen < + num_candidates in real) some round is not full-cohort → False → the rung + stays gated to a trivial pass rather than false-failing a genuinely + stochastic/divergent selection. Missing telemetry (either field absent) + also returns False — never assert determinism we cannot see. Data-driven, + so it self-disables under Phase-2 unavailability with no config change.""" + train = loaded.get("selection_train") or [] + saw = False + for e in train: + nchosen, ncand = e.get("num_chosen"), e.get("num_candidates") + if nchosen is None or ncand is None or ncand <= 0: + return False + saw = True + if nchosen != ncand: + return False + return saw + + +def _selection_is_deterministic(real: dict, sim: dict) -> bool: + """Whether the per-round selected SET is a deterministic function of the + candidate pool, so the set/sequence selection rungs should ENFORCE rather + than gate to a trivial pass. + + True when the selector is declared deterministic (DETERMINISTIC_SELECTORS) + OR the run is full-cohort in BOTH modes (_full_cohort_selection). This + un-gates fwdllm (syn_0 K=all) while keeping fluxtune (agg_goal=3 < K) and + asymmetric-eligibility fwdllm_plus (#7) gated. NOTE: the fwdllm variance- + cadence logical rung `cohort_sequence` is ungated-EXACT for ALL fwdllm + baselines (receive-order is deterministic BY DESIGN, operator-confirmed) — + it is intentionally NOT gated by this helper, so fluxtune's #1d cohort + divergence still FAILs there. + + Fallback: when neither mode carries num_chosen/num_candidates telemetry + (synthetic/legacy runs) the data-driven path is unavailable, so we revert to + the original selector-name rule — enforce on an unknown/deterministic + selector, gate a known stochastic one — to avoid silently WEAKENING a check + on telemetry that predates the count fields (principle #8).""" + selector = _selector_name(real, sim) + if selector and selector in DETERMINISTIC_SELECTORS: + return True + if _has_cohort_counts(real) and _has_cohort_counts(sim): + return _full_cohort_selection(real) and _full_cohort_selection(sim) + return not bool(selector) # legacy fallback = old `not gated` semantics + + def selection_parity(real: dict, sim: dict, max_rounds: Optional[int] = None, warn_jaccard: float = 0.7) -> dict: """S1/S2: Per-round selection overlap (Jaccard). - Enforced only for DETERMINISTIC_SELECTORS; gated to WARN for stochastic - selectors (participation_parity is the enforced invariant for those). + Enforced when selection is deterministic (`_selection_is_deterministic`: + a DETERMINISTIC_SELECTORS selector OR a full-cohort run); gated to a + trivial pass otherwise (participation_parity is the enforced invariant for + the stochastic-subset case). """ r = _by_round_selection(real["selection_train"]) s = _by_round_selection(sim["selection_train"]) @@ -912,7 +981,7 @@ def selection_parity(real: dict, sim: dict, max_rounds: Optional[int] = None, exact += 1 mean_j = sum(js) / len(js) if js else float("nan") selector = _selector_name(real, sim) - gated = bool(selector) and selector not in DETERMINISTIC_SELECTORS + gated = not _selection_is_deterministic(real, sim) enforced_ok = (not js) or mean_j >= warn_jaccard return { "ok": True if gated else enforced_ok, @@ -1072,9 +1141,11 @@ def aggregation_sequence_parity(real: dict, sim: dict, max_rounds: Optional[int] = None) -> dict: """P1 / U1: Per-round set of contributing trainers matches across modes. - Enforced only for DETERMINISTIC_SELECTORS; gated to WARN for stochastic - selectors. Exact per-round contributing-set identity is unattainable for a - stochastic, streaming, path-dependent selector (a trainer is chosen in + Enforced when selection is deterministic (`_selection_is_deterministic`: + a DETERMINISTIC_SELECTORS selector OR a full-cohort run — fwdllm syn_0); + gated to a trivial pass otherwise. Exact per-round contributing-set + identity is unattainable for a stochastic-SUBSET, streaming, path-dependent + selector (a trainer is chosen in *different* rounds across modes), the same reason S1 (selection_parity) is gated. The enforced selection invariants for stochastic selectors are participation_parity (S2) + the pooled distributions; this check stays as a @@ -1092,7 +1163,7 @@ def by_round(agg_rounds): rounds = [x for x in rounds if x <= max_rounds] matches = sum(1 for rd in rounds if r[rd] == s[rd]) selector = _selector_name(real, sim) - gated = bool(selector) and selector not in DETERMINISTIC_SELECTORS + gated = not _selection_is_deterministic(real, sim) enforced_ok = (not rounds) or matches == len(rounds) return { "ok": True if gated else enforced_ok, @@ -1456,6 +1527,13 @@ def class_share(counter): speed_class_tvd = 0.5 * sum(abs(rcs.get(b, 0) - scs.get(b, 0)) for b in buckets) selector = _selector_name(real, sim) + # NOT un-gated by the full-cohort rule: participation keys on `round`, which + # is CONSTANT for fwdllm (progress axis is data_id), so the matched-round + # window degenerates to nmatch=1 → a mechanical KS=1.0 that says nothing (the + # utility_parity "seen only 1-2 times" artifact). fwdllm's genuine per-cycle + # cohort enforcement is cohort_sequence; here the stochastic speed-class TVD + # branch stays the right call. Set-based round rungs (selection/aggregation_ + # sequence) ARE full-cohort-safe (all-K union both sides); count rungs aren't. gated = bool(selector) and selector not in DETERMINISTIC_SELECTORS tvd_tol = 0.15 if gated and speed_class_tvd is not None: @@ -1677,7 +1755,7 @@ def per_trainer_utils(agg_rounds): avg_mean_diff = sum(mean_diffs) / len(mean_diffs) if mean_diffs else float("nan") selector = _selector_name(real, sim) - gated = bool(selector) and selector not in DETERMINISTIC_SELECTORS + gated = not _selection_is_deterministic(real, sim) pooled_ok = math.isnan(pooled_ks) or pooled_ks <= max_ks per_trainer_ok = math.isnan(max_ks_val) or max_ks_val <= max_ks # Stochastic: enforce the pooled distribution only. Deterministic: also @@ -2391,6 +2469,83 @@ def gpu_budget_ok(trainers: dict, warn_overrun_frac: float = 0.25) -> dict: } +def _overrun_stats(trainers: dict) -> tuple: + """(#rounds, #overran, earliest (data_id, iter) overran) from trainer_round + telemetry. An event counts only when `training_overran` is present.""" + n = over = 0 + first = None + for _tid, d in trainers.items(): + for e in d.get("trainer_round", []): + if "training_overran" not in e: + continue + n += 1 + if e.get("training_overran"): + over += 1 + did = e.get("data_id") + if did is not None: + it = e.get("iteration_per_data_id") + cand = (did, it if it is not None else 0) + if first is None or cand < first: + first = cand + return n, over, first + + +def timing_overrun(real_trainers: dict, sim_trainers: dict, + warn_frac: float = 0.05) -> dict: + """Ovr [DIAG]: fraction of trainer rounds where the real GPU pass OVERRAN + the modeled mobile-device delay budget (K-D29 remainder-wait model, P2-6). + + Precondition-for-parity localizer, NOT a real↔sim diff. Under the K-D29 + model a trainer's arrival order is deterministic ONLY while gpu_time_s <= + the modeled delay D (the GPU hides inside the device wall). When gpu > D the + update completes AFTER the vclock passed its sct → the sim can commit it out + of order → the deterministic per-trainer arrival order that + `cohort_sequence`/`v2_var_trajectory` require is broken. So a cohort/var + divergence with a HIGH overrun fraction is a TIMING-MODEL limitation (GPU + too slow for the budget — fluxtune's 20-pass JVP at 7.57s vs a ~2-9s D/2 + budget), fixable by raising `delay_factor` or lowering `perturbation_count`, + NOT a sim ordering bug. A near-zero overrun fraction is the precondition for + exact cohort/var parity — expect ~0 for fwdllm/fwdllm_plus (GPU ~1s). + + DIAG (never fails the scoreboard); reports per-mode overrun fraction + the + earliest (data_id, iteration) an overrun occurs (where order first risks + flipping — cross-check against the cohort_sequence first_divergence). + SKIPs when `training_overran` is absent (delays disabled / pre-P2-6 run). + """ + rn, ro, rf = _overrun_stats(real_trainers) + sn, so, sf = _overrun_stats(sim_trainers) + if rn == 0 and sn == 0: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no training_overran telemetry (delays disabled or " + "pre-P2-6 run)"} + r_frac = ro / rn if rn else 0.0 + s_frac = so / sn if sn else 0.0 + worst = max(r_frac, s_frac) + if worst == 0.0: + verdict = ("no overrun — GPU within the modeled delay budget in both " + "modes; per-trainer arrival order is deterministic → exact " + "cohort/var parity is attainable") + elif worst < warn_frac: + verdict = (f"marginal overrun ({worst:.1%} < {warn_frac:.0%}) — order " + "determinism mostly holds; watch the cohort_sequence tail") + else: + verdict = (f"OVERRUN {worst:.1%} — GPU exceeds the modeled delay budget; " + "update order can flip → expect cohort_sequence/v2 breaks. " + "Raise delay_factor or lower perturbation_count (fluxtune).") + return { + "ok": True, + "tier": "DIAG", + "real_overrun_frac": round(r_frac, 4), + "sim_overrun_frac": round(s_frac, 4), + "real_overran": ro, "real_rounds": rn, + "sim_overran": so, "sim_rounds": sn, + "real_first_overrun": list(rf) if rf else None, + "sim_first_overrun": list(sf) if sf else None, + "warn_frac": warn_frac, + "verdict": verdict, + } + + _PHASE_FIELDS = ("pre_train_s", "gpu_compute_s", "mqtt_fetch_s", "weights_to_gpu_s", "weights_to_ram_s", "post_train_s") @@ -4340,6 +4495,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["trainer_phase"] = trainer_phase_parity(real_trainers, sim_trainers) results["gpu_budget_real"] = gpu_budget_ok(real_trainers) results["gpu_budget_sim"] = gpu_budget_ok(sim_trainers) + results["timing_overrun"] = timing_overrun(real_trainers, sim_trainers) results["sim_send_ts"] = sim_send_ts_ok(real_trainers, sim_trainers) # ── Stage 5 Update return & ordering ── @@ -4463,6 +4619,7 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "trainer_phase": {"stage": 4, "role": "DIAG", "deps": ()}, "gpu_budget_real": {"stage": 4, "role": "MECHANISM", "deps": ("training_budget",)}, "gpu_budget_sim": {"stage": 4, "role": "MECHANISM", "deps": ("training_budget",)}, + "timing_overrun": {"stage": 4, "role": "DIAG", "deps": ("gpu_budget_real", "gpu_budget_sim")}, "sim_send_ts": {"stage": 4, "role": "CONTROL", "deps": ("vclock_telemetry",)}, # ── Stage 5 Update return & ordering ── "inter_arrival_order": {"stage": 5, "role": "MECHANISM", "deps": ("per_round_advance", "selection_detail")}, diff --git a/lib/python/examples/async_cifar10/scripts/parity_checks.py b/lib/python/examples/async_cifar10/scripts/parity_checks.py index d547275af..258f6494d 100644 --- a/lib/python/examples/async_cifar10/scripts/parity_checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity_checks.py @@ -33,6 +33,9 @@ # selection constants DETERMINISTIC_SELECTORS, _selector_name, + _has_cohort_counts, + _full_cohort_selection, + _selection_is_deterministic, # §3.B selection selection_parity, # §3.D updates @@ -52,6 +55,7 @@ # §3.C sim invariants sim_send_ts_ok, gpu_budget_ok, + timing_overrun, trainer_phase_parity, # Stage 0 / 1 / 2 / 4 / 8 additions (causal ladder) field_coverage, diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md index 90519d310..9813b2a1c 100644 --- a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -17,6 +17,10 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 - **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both new rungs correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus 36/17/21, fluxtune 35/19/19. +- **P1-5 / P1-4** — full-cohort determinism gate (un-gates fwdllm selection/aggregation_sequence/utility; + fluxtune/fwdllm_plus stay gated); `timing_overrun` DIAG signal rung added; P1-4 assessed redundant. See the + DEFERRED list below for the full rationale. +9 pytests; banked scoreboard stable (only a new SKIP for the + overrun rung, which predates the banked logs). - **P1-6 (partial)** — 11 rung/guard pytests. *Still missing: a LIVE sim==real grad-determinism test (needs P0-2).* - **P1-8** — banked logs re-run through the upgraded checker. - **P2-1/P2-3/P2-5/P2-6** — remainder-wait delay model (K-D29: real sleeps `max(0,D−gpu)`, sct `max(gpu,D)`, @@ -25,11 +29,24 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 - **P2-4 (partial)** — GPU profiled: fwdllm/plus ~1.0s, **fluxtune 7.57s** (JVP 20 passes). Optimization deferred. **NOT DONE / DEFERRED (pick up here) — in priority order:** -1. **⚠ P1-4 and P1-5 were SKIPPED before jumping to Phase 2** (operator flagged this): - - **P1-4** — uniform cohort-record ADAPTER so the `cohort_sequence` rung also runs on **async_cifar10**'s - per-commit `agg_round` shape (fwdllm's 3 baselines already work; async_cifar10 does not yet). - - **P1-5** — enforce `selection`/`decision_determinism`/`selection_detail` where order is deterministic - (populate `DETERMINISTIC_SELECTORS` / per-baseline gate; today WARN-only via the empty set at `checks.py:882`). +1. **✅ P1-4 / P1-5 RESOLVED (this session):** + - **P1-5 DONE** — data-driven **full-cohort determinism gate** (`_selection_is_deterministic`: + `num_chosen==num_candidates` in both modes, else the DETERMINISTIC_SELECTORS name rule as a legacy fallback). + Un-gates `selection`/`aggregation_sequence`/`utility` for fwdllm (syn_0, K=all → GENUINELY enforced, no + longer a trivial gated pass) while self-keeping fluxtune (agg_goal=3) + fwdllm_plus (#7 asymmetric eligible) + gated. **participation deliberately NOT un-gated** — it keys on `round` (constant for fwdllm's data_id axis) + so it degenerates to a mechanical KS=1.0; cohort_sequence is fwdllm's per-cycle enforcement. Banked scoreboard + unchanged (41/13, 36/17, 35/19) — the un-gated rungs pass genuinely; utility's fail is a pre-existing pooled + KS=0.45 (bin-8 desync), not new. + - **P1-4 ASSESSED REDUNDANT (not built)** — extending `cohort_sequence` to async_cifar10's per-commit shape + yields a GATED trivial pass (stochastic subset, no `var`, exact order unattainable by design); its set+order + are already covered there by `aggregation_sequence` (gated) + `inter_arrival_order` + `first_divergence`. + Building the adapter adds a muddying no-signal rung (principle #16). Revisit only if async_cifar10 ever needs + an exact-ordered cohort rung. + - **NEW SIGNAL — `timing_overrun` DIAG rung** (`checks.py`, wired + tested): surfaces the P2-6 + `training_overran` fraction per mode + earliest `(data_id, iter)` overrun — the K-D29 order-determinism tell + (gpu > modeled D → arrival order can flip → cohort/var break is a TIMING-MODEL limit, not a sim bug). SKIPs on + the banked logs (predate P2-6); will populate on the databin1 run and tell us WHY fluxtune breaks. 2. **P2-7 — LAUNCH THE databin1 RUN** (the immediate experimental next step; command below), then re-run the checker `--max-bin 1` and confirm the cascade: fwdllm/fwdllm_plus `cohort_sequence` should extend past bin 8; fluxtune expected to still break (GPU overrun) → confirms P2-5 tuning is the fluxtune next step. @@ -41,7 +58,8 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 ``` cd lib/python/examples/fwdllm/expt_scripts bash run_sequential.sh --mode both --delays on --delay-factor 2 --max-data-id 1 --max-runtime-s 1800 --yes -python run_parity.py --yes --max-bin 1 # then inspect cohort_sequence per baseline +python run_parity.py --yes --max-bin 1 # then inspect cohort_sequence + timing_overrun per baseline +python plot_step_timing.py # NEW: per-step GPU breakdown (fine step_timing now populated) ``` Expected: fwdllm/fwdllm_plus (GPU≈1s < D/2 of 2–9s) → order deterministic → `cohort_sequence` improves; fluxtune (GPU 7.57s > D/2) → `[TIMING_OVERRUN]` fires, order still flips → tune `perturbation_count` down next. @@ -115,13 +133,16 @@ stochastic selectors (`DETERMINISTIC_SELECTORS=∅`, `checks.py:882`) and keys o (KS-only used to pass the ~1% offset). - [x] **P1-3 DONE — `--max-bin` window** threaded through `_fwd_cadence_cycles` → V1/V2/V3/V4/V5 + `cohort_sequence` + `run_all_parity` + `run_parity.py` + `cli.py`. `run_parity.py --max-bin 1` verified. -- [ ] **P1-4 Uniform cohort-record adapter** so async (per-commit `agg_round`, `contributing=[end]`) and sync - (per-cycle cohort) run ONE cohort/order diff. NOTE: fluxtune already emits per-cycle cadence fields (it's the - fwdllm aggregator with `is_async`), so `cohort_sequence` ALREADY runs on all 3 fwdllm baselines. The adapter - is only needed to extend the rung to async_cifar10's per-commit shape — lower priority. -- [ ] **P1-5 Enforce selection/determinism rungs** where order is deterministic: populate - `DETERMINISTIC_SELECTORS` (or a per-baseline determinism gate) so `selection`/`decision_determinism`/ - `selection_detail` stop being WARN-only. +- [x] **P1-4 ASSESSED REDUNDANT (not built).** Extending `cohort_sequence` to async_cifar10's per-commit shape is + a GATED trivial pass (stochastic subset, no `var`); async_cifar10's set+order are already covered by + `aggregation_sequence` + `inter_arrival_order` + `first_divergence`. No new signal → not built (principle #16). +- [x] **P1-5 DONE — data-driven full-cohort determinism gate** (`_selection_is_deterministic` / + `_full_cohort_selection` / `_has_cohort_counts`, `checks.py`). Un-gates `selection`/`aggregation_sequence`/ + `utility` when `num_chosen==num_candidates` in both modes (fwdllm syn_0) → GENUINELY enforced; fluxtune + + fwdllm_plus stay gated (subset / #7 asymmetric eligible); legacy no-count telemetry falls back to the old + selector-name rule (no regression). participation EXCLUDED (round-keyed → mechanical KS on fwdllm's data_id + axis). `decision_determinism` left DIAG (a localizer by design, not an enforce target). 6 gate pytests + + banked-log validation (scoreboard stable). NEW `timing_overrun` DIAG rung surfaces the P2-6 overrun signal. - [~] **P1-6 PARTIAL — rung + guard pytests landed** (`TestCohortSequence` ×9, `TestVarTrajectoryMeanGuard` ×2 in `tests/mode/test_parity_checks.py`). STILL TODO: a LIVE sim==real grad-determinism test given matched order (needs P0-2 infra). @@ -144,10 +165,14 @@ stochastic selectors (`DETERMINISTIC_SELECTORS=∅`, `checks.py:882`) and keys o - [x] **P2-3 DONE — crc32 straggler offset disabled** (`sim_straggler_spread_s: 0.0` in all 3 sim yamls). The per-trainer registry delays now supply the completion spread; the offset would re-noise the deterministic order. `_wan_s` already 0. (Helper kept flag-gated for its unit tests.) -- [~] **P2-4 PARTIAL — GPU profiled** (from telemetry): fwdllm/fwdllm_plus **~1.0s** (cos-sim path, ~1 forward - pass); **fluxtune 7.57s mean, max 59.6s** (JVP path, 2×10 = 20 passes). fwdllm is forward-ONLY (FedFwd, no - backprop) so "forward\d{8}_\d{6})_(?P.+)_n(?P\d+)_smoke" + r"(?:_(?P.+))?_(?Preal|sim)$" +) + +# Coarse trainer_round phase fields, in pipeline order. +_PHASES = ("mqtt_fetch_s", "weights_to_ram_s", "weights_to_gpu_s", + "pre_train_s", "gpu_compute_s", "post_train_s") + + +def _iter_trainer_events(run_dir: str): + """Yield (short_id, event_dict) for every trainer telemetry record.""" + tdir = pathlib.Path(run_dir) / "telemetry" + for f in sorted(tdir.glob("trainer_*.jsonl")): + sid = f.stem[-4:] + with open(f) as fp: + for line in fp: + line = line.strip() + if not line: + continue + try: + yield sid, json.loads(line) + except json.JSONDecodeError: + continue + + +def _collect(run_dir: str) -> dict: + """Aggregate step_timing (fine) + trainer_round phases (coarse) for a run.""" + step_tot: dict = collections.defaultdict(float) + step_n: dict = collections.defaultdict(int) + phase_tot: dict = collections.defaultdict(float) + phase_n: dict = collections.defaultdict(int) + n_rounds = 0 + for _sid, e in _iter_trainer_events(run_dir): + ev = e.get("event") + if ev == "step_timing": + fn, d = e.get("func"), e.get("duration_s") + if fn is not None and d is not None: + step_tot[fn] += float(d) + step_n[fn] += 1 + elif ev == "trainer_round": + n_rounds += 1 + for p in _PHASES: + v = e.get(p) + if v is not None: + phase_tot[p] += float(v) + phase_n[p] += 1 + return {"step_tot": dict(step_tot), "step_n": dict(step_n), + "phase_tot": dict(phase_tot), "phase_n": dict(phase_n), + "n_rounds": n_rounds} + + +def _fmt_table(label: str, agg: dict) -> str: + lines = [f"\n=== {label} ({agg['n_rounds']} trainer rounds) ==="] + # coarse phases (always present) + lines.append(" COARSE trainer_round phases (mean s/round | total s):") + ptot = agg["phase_tot"] + for p in _PHASES: + if p in ptot: + mean = ptot[p] / max(agg["phase_n"].get(p, 1), 1) + lines.append(f" {p:<18} {mean:8.3f} | {ptot[p]:9.1f}") + # fine steps (present only on runs with step_timing telemetry) + st = agg["step_tot"] + if st: + lines.append(" FINE step_timing (mean s/call | calls | total s), " + "by total desc:") + for fn in sorted(st, key=st.get, reverse=True): + n = agg["step_n"].get(fn, 0) + mean = st[fn] / max(n, 1) + lines.append(f" {fn:<30} {mean:7.3f} | {n:6d} | {st[fn]:8.1f}") + else: + lines.append(" FINE step_timing: NONE — run predates step_timing " + "telemetry (re-run to populate the per-step GPU breakdown).") + return "\n".join(lines) + + +def _plot(label: str, agg: dict, out_png: str) -> bool: + try: + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except Exception as exc: # pragma: no cover + print(f" [plot skipped] matplotlib unavailable: {exc}") + return False + + st = agg["step_tot"] + ptot = agg["phase_tot"] + fig, axes = plt.subplots(1, 2, figsize=(13, 5)) + + # left: coarse phase mean s/round + ph = [(p, ptot[p] / max(agg["phase_n"].get(p, 1), 1)) + for p in _PHASES if p in ptot] + if ph: + names, vals = zip(*ph) + axes[0].barh(range(len(names)), vals, color="#4C72B0") + axes[0].set_yticks(range(len(names))) + axes[0].set_yticklabels(names) + axes[0].invert_yaxis() + axes[0].set_xlabel("mean seconds / round") + axes[0].set_title(f"{label}: coarse phase (trainer_round)") + + # right: fine step_timing mean s/call, top 12 by total + ax = axes[1] + if st: + top = sorted(st, key=st.get, reverse=True)[:12] + means = [st[fn] / max(agg["step_n"].get(fn, 1), 1) for fn in top] + ax.barh(range(len(top)), means, color="#C44E52") + ax.set_yticks(range(len(top))) + ax.set_yticklabels(top, fontsize=8) + ax.invert_yaxis() + ax.set_xlabel("mean seconds / call") + ax.set_title(f"{label}: fine step (step_timing)") + else: + ax.text(0.5, 0.5, "no step_timing telemetry\n(re-run to populate)", + ha="center", va="center", transform=ax.transAxes) + ax.set_axis_off() + + fig.tight_layout() + fig.savefig(out_png, dpi=110) + plt.close(fig) + return True + + +def _discover_latest(experiments_dir: str, baselines, side: str) -> dict: + """{baseline -> newest run_dir} for the requested side, EXACT baseline match + (via _RUN_RE, so fwdllm != fwdllm_plus). ts is fixed-width → lexical == time.""" + want = set(baselines) + best: dict = {} + root = pathlib.Path(experiments_dir) + if not root.is_dir(): + return {} + for d in root.iterdir(): + if not d.is_dir(): + continue + m = _RUN_RE.match(d.name) + if not m or m["variant"] != side or m["baseline"] not in want: + continue + base = m["baseline"] + if base not in best or m["ts"] > best[base][0]: + best[base] = (m["ts"], str(d)) + return {b: p for b, (_ts, p) in best.items()} + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--run-dir", nargs="+", default=None, + help="explicit run dir(s); default = latest per baseline") + ap.add_argument("--experiments-dir", default=str(_DEFAULT_EXPERIMENTS)) + ap.add_argument("--baselines", nargs="+", + default=["fwdllm", "fwdllm_plus", "fluxtune"]) + ap.add_argument("--side", default="sim", choices=["sim", "real"], + help="which side to profile when auto-discovering (default sim)") + ap.add_argument("--out-dir", default=None, + help="where to write PNGs (default /_timing_plots)") + ap.add_argument("--no-plot", action="store_true", help="table only, no PNGs") + args = ap.parse_args(argv) + + if args.run_dir: + runs = {os.path.basename(d.rstrip("/")): d for d in args.run_dir} + else: + runs = _discover_latest(args.experiments_dir, args.baselines, args.side) + if not runs: + print("No run dirs found. Pass --run-dir or check --experiments-dir.") + return 1 + + out_dir = args.out_dir or os.path.join(args.experiments_dir, "_timing_plots") + if not args.no_plot: + os.makedirs(out_dir, exist_ok=True) + + for label, rdir in runs.items(): + agg = _collect(rdir) + print(_fmt_table(label, agg)) + if not args.no_plot: + png = os.path.join(out_dir, f"step_timing_{label}.png") + if _plot(label, agg, png): + print(f" -> {png}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 3c5d3fabf..2c4c740d0 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -70,9 +70,14 @@ P1) — these two now correctly FAIL the order/grad divergence that used to be i | baseline | pass / fail / skip | JSON | |---|---|---| -| **fwdllm/syn_0** | **41 / 13 / 21** | `experiments/_parity_reports/parity_fwdllm_syn_0_20260705_045724.json` | -| **fwdllm_plus/syn_0** | **36 / 17 / 21** | `parity_fwdllm_plus_syn_0_20260705_063350.json` | -| **fluxtune/syn_0** | **35 / 19 / 19** | `parity_fluxtune_syn_0_20260705_080931.json` | +| **fwdllm/syn_0** | **41 / 13 / 22** | `experiments/_parity_reports/parity_fwdllm_syn_0_20260705_045724.json` | +| **fwdllm_plus/syn_0** | **36 / 17 / 22** | `parity_fwdllm_plus_syn_0_20260705_063350.json` | +| **fluxtune/syn_0** | **35 / 19 / 20** | `parity_fluxtune_syn_0_20260705_080931.json` | + +*(skip +1 each vs the prior ref = the new `timing_overrun` DIAG rung, which SKIPs on these banked logs — they +predate its P2-6 `training_overran` telemetry; it populates on the databin1 run. Pass/fail unchanged: the K-D30 +full-cohort gate flipped fwdllm's `selection`/`aggregation_sequence`/`utility` from a trivial GATED pass to GENUINE +enforcement without moving the counts — they pass truthfully; `utility` still fails on a pre-existing pooled KS.)* **Fails, categorized by blast radius (fix the SHARED roots first — principle #14).** - **SHARED — all 3 (top priority):** `throughput` / `overhead_residual` / `per_round_advance` (the `sim_rate<1` @@ -121,8 +126,10 @@ exact cadence parity IS achievable once order matches. **Full diagnosis + fix pl > **STATUS (2026-07-05 checkpoint):** the checker now GATES on this (enforced `cohort_sequence` EXACT rung + > V2 mean-guard, P1) and the timing-model fix has LANDED (**K-D29** remainder-wait `max(gpu,D)` + per-trainer D + -> crc32 straggler off + `perturbation_count` knob; 606+115 tests green). **Not yet run.** Pending before close: -> **P1-4/P1-5** (async_cifar10 cohort adapter + enforce determinism rungs — skipped en route to Phase 2) and the +> crc32 straggler off + `perturbation_count` knob). **P1-4/P1-5 now RESOLVED** (K-D30): full-cohort determinism +> gate un-gates fwdllm's selection/aggregation_sequence/utility to GENUINE enforcement (fluxtune/fwdllm_plus stay +> gated); new `timing_overrun` DIAG rung surfaces the K-D29 overrun tell; P1-4 assessed redundant. 419 mode + 115 +> async_cifar10 parity tests green; banked scoreboard stable. **Not yet run.** Pending before close: the > **databin1 validation run** (`--delay-factor 2 --max-data-id 1`). Resume steps: the tracker's SESSION CHECKPOINT. ### Open issues (OPEN only — closed items live in §G/§H) @@ -147,10 +154,11 @@ exact cadence parity IS achievable once order matches. **Full diagnosis + fix pl completes now; explain the 4× slowness + the syn_0 eligible-count gap from telemetry. Not a sim bug. 4. Then C1/C2 convergence at matched `data_id` per baseline → gate to Phase 2 (unavailability). -### SKIP audit (19–21 skips; ~16 legit) +### SKIP audit (20–22 skips; ~17 legit) Legit at Phase-1 syn_0 + `random` selector: 7 availability ground-truth rungs + 4 delivery/withheld (Phase-2 effect path, not built) + 3 DynamicKC (disabled by design) + 2 oort-only (`random` baselines) + `residence` -(async telemetry). The 12 former rigor-gap skips (4 advance, 8 phase-timing) are un-skipped (K-D21). +(async telemetry) + `timing_overrun` (banked logs predate its P2-6 telemetry; populates on the next run). The 12 +former rigor-gap skips (4 advance, 8 phase-timing) are un-skipped (K-D21). --- @@ -399,5 +407,17 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso order, so it's the fluxtune watch (its JVP GPU 7.57s overruns D/2). Crc32 straggler offset disabled (`sim_straggler_spread_s=0`); `perturbation_count` made a knob (default 10). See PARITY_LOGICAL_TASKS.md P2. +- **K-D30** — **full-cohort determinism gate + `timing_overrun` signal (P1-5, closes the P1-4/P1-5 gap).** The + selection set/sequence rungs (`selection`/`aggregation_sequence`/`utility`) gated to a TRIVIAL pass for every + stochastic selector, so fwdllm's syn_0 selection was never actually checked. Fix: `_selection_is_deterministic` + un-gates when `num_chosen==num_candidates` in BOTH modes (K≥pool → set-deterministic) — data-driven, so it + enforces fwdllm (K=all), keeps fluxtune (agg_goal=3) + fwdllm_plus (#7 asymmetric eligible) gated, self-disables + under Phase-2 scarcity, and falls back to the old selector-name rule on count-less legacy telemetry (no + regression). `participation` EXCLUDED (round-keyed → mechanical KS on fwdllm's constant-`round`/`data_id` axis; + cohort_sequence is its per-cycle enforcement). New `timing_overrun` DIAG rung surfaces the K-D29 overrun tell + (P2-6 `training_overran` fraction + first-overrun bin → gpu>D flips order → cohort/var break is a timing-model + limit not a sim bug). P1-4 (async_cifar10 cohort adapter) assessed redundant, not built. Sim/checker-only → + async_cifar10 byte-identical; banked scoreboard stable. + *Retired/superseded anchors (kept only as pointers): K-D6 (→K-D12), K-D7/K-D8/K-D10/K-D16/K-D18/K-D19/K-D20/K-D23 — landed scaffolding or corrections, folded into §G/§H; see git history for detail.* diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index b0cb4779d..a405ca607 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -578,17 +578,20 @@ def _sim_straggler_offset_s(self) -> float: @timer_decorator def train_with_data_id(self): - # Create FwdLLMStage for timing/metrics logging - self.fwd_llm_stage = FwdLLMStage( - self._round, self.data_id, self.iteration_per_data_id, self.trainer_id - ) - if self.abort_training == True: logger.info( f"Aborting training for trainer id: {self.trainer_id} because it has already sent updates for iteration_per_data_id: {self.iteration_per_data_id}" ) return + # Create FwdLLMStage for timing/metrics logging. Set AFTER the abort + # check so an aborted round stays a true no-op (no fwd_llm_stage → the + # @timer_decorator emits no step_timing record for it, and nothing else + # fires either — see test_aborted_round_emits_nothing). + self.fwd_llm_stage = FwdLLMStage( + self._round, self.data_id, self.iteration_per_data_id, self.trainer_id + ) + # Phase-timing entry (Stage A1): everything up to the compute loop is # pre_train (avail check, FwdLLMStage setup, loader state). _phase_entry = time.time() @@ -612,14 +615,15 @@ def train_with_data_id(self): # Without this, phase_post_train saw real ~1.14 vs sim 0.0. _phase_post_start = time.time() - # Sim-mode stamps (Batch 1): the modeled round duration is ADDITIVE - # (real_gpu + D), matching real mode's sleep-D-on-top-of-GPU semantics - # -- NOT cifar10's max(gpu, D) "sleep to fill a budget" model. The sct - # (when this update COMMITS on the virtual clock) is the aggregator's - # dispatch stamp (SIM_SEND_TS, read in _fetch_weights) + that duration + - # the optional pre-commit holding leg. _send_grads sends these so the - # aggregator can order updates by _sim_completion_ts. In real mode these - # stay None and the aggregator falls back to arrival order (unchanged). + # Sim-mode stamps: the modeled round duration is the REMAINDER-WAIT + # max(real_gpu, D) (K-D29), matching real mode's sleep-the-remainder + # semantics (device wall = D, GPU hidden inside) -- NOT the old additive + # gpu + D (K-D2, reversed). The sct (when this update COMMITS on the + # virtual clock) is the aggregator's dispatch stamp (SIM_SEND_TS, read in + # _fetch_weights) + that duration + the optional pre-commit holding leg. + # _send_grads sends these so the aggregator can order updates by + # _sim_completion_ts. In real mode these stay None and the aggregator + # falls back to arrival order (unchanged). # B3 (K-D20 #6): WAN payload-transfer term (up+down). NOT measurable on # localhost (no ground truth), so this is a DOCUMENTED knob left at 0 -- # do NOT enable without a real WAN measurement. Byte-identical at 0. @@ -664,10 +668,11 @@ def train_with_data_id(self): ev, fields = build_trainer_round( round_num=int(self._round), real_gpu_time_s=_real_gpu_time_s, - # real GPU compute + the emulated base delay (0 if disabled) + the - # sim sct-model folds (B2 straggler spread, B3 WAN) -- NOT a - # budget-vs-actual quantity (fwdllm has no sleep-to-fill-budget - # model); the total modeled wall this round took. + # the total modeled wall this round took: max(real_gpu, D) under + # the K-D29 remainder-wait model (device wall = D, GPU hidden), + # + the sim sct-model folds (B2 straggler spread, B3 WAN, both 0 + # by default). training_budget_s below carries D separately so + # the overrun (gpu > D) is recoverable from telemetry. sim_round_duration_s=self._sim_round_duration_s, avail_state=self.avl_state.value, dataset_size=self.dataset_size, diff --git a/lib/python/flame/monitor/runtime.py b/lib/python/flame/monitor/runtime.py index 714079989..133d32ca0 100644 --- a/lib/python/flame/monitor/runtime.py +++ b/lib/python/flame/monitor/runtime.py @@ -40,6 +40,25 @@ def wrapper(*args, **kwargs): f"[decorator] Runtime of {func.__name__}: {duration:.6f}s " f"(Round={stage.round_id}, DataId={stage.data_id}, Iter={stage.iteration}, TrainerId={stage.trainer_id})" ) + # Structured, plottable companion to the log line: attribute this + # step's wall time to (func, data_id, iteration) so the GPU cost can + # be decomposed and optimized (P2-4). A no-op when telemetry is off, + # so non-fwdllm runs / unit tests pay nothing. Guarded on `stage` + # truthiness so nested helpers whose args[0] is NOT the trainer self + # (a torch device has no fwd_llm_stage) never emit a mis-attributed + # record. Best-effort: a telemetry hiccup must never break training. + try: + from flame import telemetry + if telemetry.is_enabled(): + from flame.telemetry.events import build_step_timing + ev, fields = build_step_timing( + func=func.__name__, duration_s=duration, + round_num=stage.round_id, data_id=stage.data_id, + iteration=stage.iteration, trainer_id=stage.trainer_id, + ) + telemetry.emit(ev, **fields) + except Exception: # pragma: no cover - telemetry must never fault training + logger.debug("step_timing telemetry emit failed", exc_info=True) else: logger.info( f"[decorator] Runtime of {func.__name__}: {duration:.6f}s (no stage info)" diff --git a/lib/python/flame/telemetry/events.py b/lib/python/flame/telemetry/events.py index 237f09d8c..b1bebc67e 100644 --- a/lib/python/flame/telemetry/events.py +++ b/lib/python/flame/telemetry/events.py @@ -29,6 +29,7 @@ EVENT_WITHHELD_DELIVERY = "withheld_delivery" # late stale delivery of a send-gated update EVENT_ABANDON_TIMEOUT = "abandon_timeout" # 90s vclock slot-free of a stalled trainer EVENT_AGG_BELIEF_CHANGE = "agg_belief_change" # aggregator's belief about a trainer's avail state +EVENT_STEP_TIMING = "step_timing" # per-function wall duration of a timed compute step KNOWN_EVENTS = frozenset( { @@ -47,6 +48,7 @@ EVENT_WITHHELD_DELIVERY, EVENT_ABANDON_TIMEOUT, EVENT_AGG_BELIEF_CHANGE, + EVENT_STEP_TIMING, } ) @@ -188,6 +190,35 @@ def build_trainer_round( return EVENT_TRAINER_ROUND, fields +def build_step_timing( + *, + func: str, + duration_s: float, + round_num: Optional[int] = None, + data_id: Optional[int] = None, + iteration: Optional[int] = None, + trainer_id: Optional[str] = None, +) -> tuple[str, dict[str, Any]]: + """Per-function wall duration of one timed compute step (`timer_decorator`). + + The fine-grained companion to `trainer_round`'s coarse phase split: it + attributes wall time to the individual forward-grad steps (functional-model + setup, perturbation selection, per-batch JVP, delay emulation) so the GPU + cost can be decomposed and optimized (P2-4, fluxtune's 20-pass JVP). Keyed by + (data_id, iteration) so a step's cost can be tracked across the cadence. + """ + fields: dict[str, Any] = {"func": func, "duration_s": duration_s} + for k, v in ( + ("round", round_num), + ("data_id", data_id), + ("iteration_per_data_id", iteration), + ("trainer_id", trainer_id), + ): + if v is not None: + fields[k] = v + return EVENT_STEP_TIMING, fields + + def build_util_disparity( *, round_num: int, diff --git a/lib/python/tests/mode/test_parity_checks.py b/lib/python/tests/mode/test_parity_checks.py index 29fb5f055..b5eeb9a9a 100644 --- a/lib/python/tests/mode/test_parity_checks.py +++ b/lib/python/tests/mode/test_parity_checks.py @@ -1257,3 +1257,110 @@ def test_matched_var_passes(self): base = [0.9, 0.5, 0.42, 0.31] agg = _agg(agg_rounds=[_lcyc(0, i, ["a"], v) for i, v in enumerate(base)]) assert pc.var_trajectory_parity(agg, agg)["ok"] + + +def _selc(round_, chosen, num_candidates, selector="random", ts=0.0): + """Selection event WITH the num_chosen/num_candidates count fields the + full-cohort determinism gate reads (real runs always emit these).""" + return {"event": "selection", "task": "train", "round": round_, "ts": ts, + "selector": selector, "chosen": list(chosen), + "num_chosen": len(chosen), "num_candidates": num_candidates} + + +class TestSelectionDeterminismGate: + """P1-5: set/sequence selection rungs ENFORCE when selection is provably + deterministic (full-cohort K>=pool, or a declared deterministic selector) + and gate to a trivial pass otherwise — data-driven from num_chosen/ + num_candidates so it self-splits fwdllm (enforce) vs fluxtune / fwdllm_plus + #7 (gate) and self-disables under Phase-2 scarcity.""" + + def test_full_cohort_helper(self): + full = _agg(selection=[_selc(1, ["a", "b", "c"], 3), + _selc(2, ["a", "b", "c"], 3)]) + subset = _agg(selection=[_selc(1, ["a", "b", "c"], 10)]) # fluxtune-like + assert pc._full_cohort_selection(full) + assert not pc._full_cohort_selection(subset) + + def test_full_cohort_ungates_and_enforces(self): + # fwdllm syn_0: everyone selected -> deterministic -> ENFORCED. A genuine + # per-round contributing-set divergence must now FAIL (was invisible). + real = _agg(selection=[_selc(1, ["a", "b", "c"], 3)], + agg_rounds=[_round(1, ["a", "b", "c"], [0, 0, 0])]) + sim = _agg(selection=[_selc(1, ["a", "b", "c"], 3)], + agg_rounds=[_round(1, ["a", "b", "x"], [0, 0, 0])]) # x != c + assert pc._selection_is_deterministic(real, sim) + agg_seq = pc.aggregation_sequence_parity(real, sim) + assert agg_seq["gated"] is False and agg_seq["ok"] is False + + def test_full_cohort_matched_is_genuine_pass(self): + a = _agg(selection=[_selc(1, ["a", "b", "c"], 3)], + agg_rounds=[_round(1, ["a", "b", "c"], [0, 0, 0])]) + agg_seq = pc.aggregation_sequence_parity(a, a) + assert agg_seq["gated"] is False and agg_seq["ok"] is True + + def test_subset_selection_stays_gated(self): + # fluxtune agg_goal=3 < pool=10: stochastic subset -> gated trivial pass + # even when the sim picks a DIFFERENT subset (cohort_sequence owns that). + real = _agg(selection=[_selc(1, ["a", "b", "c"], 10)], + agg_rounds=[_round(1, ["a", "b", "c"], [0, 0, 0])]) + sim = _agg(selection=[_selc(1, ["d", "e", "f"], 10)], + agg_rounds=[_round(1, ["d", "e", "f"], [0, 0, 0])]) + assert not pc._selection_is_deterministic(real, sim) + agg_seq = pc.aggregation_sequence_parity(real, sim) + assert agg_seq["gated"] is True and agg_seq["ok"] is True + + def test_asymmetric_eligibility_stays_gated(self): + # fwdllm_plus #7: real sees fewer eligible than the pool -> not full + # cohort in real -> gated (don't hard-fail a real-side eligibility gap). + real = _agg(selection=[_selc(1, ["a", "b"], 5)]) # 2 of 5 chosen + sim = _agg(selection=[_selc(1, ["a", "b", "c", "d", "e"], 5)]) # full + assert not pc._selection_is_deterministic(real, sim) + + def test_legacy_no_counts_falls_back_to_selector_rule(self): + # No count telemetry + unknown selector -> old behavior: ENFORCED (so the + # pre-existing disjoint-fails test semantics are preserved, no regression). + real = _agg(selection=[_sel(1, ["a", "b"])]) + sim = _agg(selection=[_sel(1, ["c", "d"])]) + assert pc._selection_is_deterministic(real, sim) # enforce on unknown + assert not pc.selection_parity(real, sim)["ok"] + + +def _tr_round(overran, data_id=0, it=0, gpu=1.0, budget=2.0): + return {"real_gpu_time_s": gpu, "training_budget_s": budget, + "training_overran": overran, "data_id": data_id, + "iteration_per_data_id": it} + + +class TestTimingOverrun: + """Ovr: the K-D29 order-determinism tell (P2-6). A high overrun fraction + means gpu > modeled D -> arrival order can flip -> cohort_sequence/v2 breaks + are a TIMING-MODEL limit, not a sim ordering bug. DIAG (never fails).""" + + def test_no_overrun_verdict(self): + tr = {"a": {"trainer_round": [_tr_round(False), _tr_round(False)]}} + r = pc.timing_overrun(tr, tr) + assert r["ok"] and r["tier"] == "DIAG" + assert r["real_overrun_frac"] == 0.0 + assert "attainable" in r["verdict"] + + def test_overrun_flags_and_reports_first_bin(self): + # fluxtune-like: some rounds overrun; earliest at (data_id, iter). + real = {"a": {"trainer_round": [ + _tr_round(False, 0, 0), _tr_round(True, 0, 1), _tr_round(True, 2, 0)]}} + sim = {"a": {"trainer_round": [_tr_round(False, 0, 0)]}} + r = pc.timing_overrun(real, sim) + assert r["ok"] # DIAG never fails the scoreboard + assert r["real_overrun_frac"] == pytest.approx(2 / 3, abs=1e-3) + assert r["real_first_overrun"] == [0, 1] + assert "OVERRUN" in r["verdict"] + + def test_skips_without_telemetry(self): + tr = {"a": {"trainer_round": [{"real_gpu_time_s": 1.0}]}} # no overran field + r = pc.timing_overrun(tr, tr) + assert r.get("status") == "SKIP" and r["ok"] + + def test_present_in_run_all_and_meta(self): + assert "timing_overrun" in pc.CHECK_META + tr = {"a": {"trainer_round": [_tr_round(False)]}} + res = pc.run_all_parity(_agg(), _agg(), tr, tr) + assert "timing_overrun" in res diff --git a/lib/python/tests/telemetry/test_event_schema.py b/lib/python/tests/telemetry/test_event_schema.py index 86748daa3..029dd687d 100644 --- a/lib/python/tests/telemetry/test_event_schema.py +++ b/lib/python/tests/telemetry/test_event_schema.py @@ -73,6 +73,18 @@ def test_agg_round_lists(self): assert f["staleness"] == [0, 1, 2] assert f["agg_goal"] == 3 + def test_step_timing_required_and_optional(self): + from flame.telemetry.events import EVENT_STEP_TIMING, build_step_timing + ev, f = build_step_timing( + func="_train_one_batch", duration_s=1.25, + round_num=1, data_id=0, iteration=2, trainer_id="t3") + assert ev == EVENT_STEP_TIMING + assert f["func"] == "_train_one_batch" and f["duration_s"] == 1.25 + assert f["data_id"] == 0 and f["iteration_per_data_id"] == 2 + # optional context omitted when absent + _ev2, f2 = build_step_timing(func="x", duration_s=0.1) + assert "data_id" not in f2 and "trainer_id" not in f2 + def test_util_disparity_ratio_and_fraction(self): ev, f = build_util_disparity( round_num=1, @@ -117,6 +129,64 @@ def test_full_pool_higher_utility_gives_ratio_below_one(self): assert f["utility_ratio"] < 1.0 +# --- timer_decorator step_timing emission ---------------------------------- + + +class TestTimerDecoratorTelemetry: + """timer_decorator emits a step_timing record for a genuine trainer step + (self has fwd_llm_stage) and stays silent for a nested helper whose first + arg is not the trainer (e.g. a torch device) — so no mis-attributed record. + """ + + def _timed(self): + from flame.monitor.runtime import timer_decorator + + @timer_decorator + def _train_one_batch(obj, x): + return x + 1 + + return _train_one_batch + + def test_emits_with_stage(self, tmp_path): + from flame.monitor.runtime import FwdLLMStage + + telemetry.configure(role="trainer", end_id="t9", run_dir=str(tmp_path)) + + class _Trainer: + fwd_llm_stage = FwdLLMStage(1, 0, 2, trainer_id="t9") + + assert self._timed()(_Trainer(), 41) == 42 + telemetry.shutdown() + recs = [r for r in _read(tmp_path / "trainer_t9.jsonl") + if r["event"] == "step_timing"] + assert len(recs) == 1 + r = recs[0] + assert r["func"] == "_train_one_batch" and r["data_id"] == 0 + assert r["iteration_per_data_id"] == 2 and "duration_s" in r + + def test_silent_without_stage(self, tmp_path): + telemetry.configure(role="trainer", end_id="t0", run_dir=str(tmp_path)) + + class _NoStage: # e.g. a nested helper's args[0] (a device) — no stage + pass + + self._timed()(_NoStage(), 1) + telemetry.shutdown() + recs = [r for r in _read(tmp_path / "trainer_t0.jsonl") + if r["event"] == "step_timing"] + assert recs == [] + + def test_noop_when_telemetry_disabled(self): + from flame.monitor.runtime import FwdLLMStage + + telemetry.shutdown() # unconfigured -> emit is a no-op, must not raise + + class _Trainer: + fwd_llm_stage = FwdLLMStage(1, 0, 0) + + assert self._timed()(_Trainer(), 7) == 8 + + # --- selector emission produces a consistent schema ------------------------ From a2146e17911be5813fe88879a7b1d606350eb374 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Sun, 5 Jul 2026 19:23:07 -0400 Subject: [PATCH 30/56] fwdllm: K-D31 canonical commit order + K-D32 fluxtune JVP perf-opt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit K-D31 (sync delay-tie): trainer stamps pure modeled delay D (MODELED_DELAY_S, both modes); aggregator canonicalizes each cohort's commit order by (D, str(end)) before the telemetry snapshot + aggregate(), so equal-D ties break by trainer_id identically in real and sim. Closes the sole databin1 residual — cohort_sequence order_match 0.5/0.0 -> 1.0 for the sync baselines, var/grads bit-identical. Validated: [COMMIT_CANON] fires in fluxtune (perms [2,0,1] etc); async_cifar10 byte-identical. K-D32 (fluxtune JVP perf-opt, config-gated `jvp_perf_opt`, all bit-identical): (a) trainable-only finite difference (calculate_jvp skips p-h*0 on the 98.5% frozen backbone, 1.26x + -251MB); (b) skip 3 diagnostic-only forward passes; (c) reuse the selected perturbation's cached JVP. fluxtune -37% GPU/batch to clear the delay_factor=1 timing overrun without changing fidelity. Default false (byte-identical); true in both fluxtune yamls (must match real<->sim). Profiled + validated by scripts/profile_jvp_opt.py (reuses real create_model + calc_jvp). Also: fluxtune_contributions.md (systems/ML contributions brief); simulate_fwdllm.md §L compute profile + K-D31/K-D32; 22 new pytests (commit_canon x9, jvp_perf_opt x4, + prior). 428 mode + 115 async_cifar10 parity green. Co-Authored-By: Claude Opus 4.8 --- .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 67 ++- .../examples/fwdllm/configs/trainer_base.yaml | 6 + .../expt_scripts/fluxtune_n10_smoke.yaml | 6 + .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 4 + .../examples/fwdllm/fluxtune_contributions.md | 175 +++++++ .../fwdllm/scripts/profile_jvp_opt.py | 436 ++++++++++++++++++ lib/python/examples/fwdllm/simulate_fwdllm.md | 93 +++- lib/python/examples/fwdllm/trainer/fl_main.py | 4 + .../trainer/forward_training/FedSgdTrainer.py | 10 + .../trainer/forward_training/fwdgrad_utils.py | 26 +- .../tc_transformer_trainer_distribute.py | 51 +- lib/python/examples/fwdllm/trainer/main.py | 4 + .../horizontal/syncfl/fwdllm_aggregator.py | 77 ++++ .../mode/horizontal/syncfl/fwdllm_trainer.py | 6 + lib/python/flame/mode/message.py | 9 + .../tests/mode/test_fwdllm_commit_canon.py | 140 ++++++ .../tests/mode/test_fwdllm_jvp_perf_opt.py | 66 +++ 17 files changed, 1143 insertions(+), 37 deletions(-) create mode 100644 lib/python/examples/fwdllm/fluxtune_contributions.md create mode 100644 lib/python/examples/fwdllm/scripts/profile_jvp_opt.py create mode 100644 lib/python/tests/mode/test_fwdllm_commit_canon.py create mode 100644 lib/python/tests/mode/test_fwdllm_jvp_perf_opt.py diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md index 9813b2a1c..eca2c0e07 100644 --- a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -7,11 +7,12 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 --- -## ⏸ SESSION CHECKPOINT (2026-07-05, paused for a break — resume here) +## ⏸ SESSION CHECKPOINT (2026-07-05 — DATABIN1 RUN LANDED; resume here) -**Code + tests are landed and GREEN; NO run has been launched yet.** Full suite: 606 mode/telemetry/selector + -115 async_cifar10 parity green; trainer files compile. NOTHING is half-edited. Nothing committed yet at pause → -**commit + push done at end of session** (this checkpoint is the resume anchor). +**Code + tests landed & GREEN; the P2-7 databin1 run is DONE and CHECKED.** Full suite: 606 mode/telemetry/ +selector + 115 async_cifar10 parity green. **Outcome (see P2-7 below):** K-D29 broke the RNG-desync root — sync +baselines are `var` bit-identical + `timing_overrun=0%` on bin 1; the only sync residual is a benign delay-tie +order swap (OPEN FORK P2-7a); fluxtune still overruns 38% as predicted → next is P2-5 (`perturbation_count`↓). **DONE (landed + tested):** - **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both @@ -47,20 +48,39 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 `training_overran` fraction per mode + earliest `(data_id, iter)` overrun — the K-D29 order-determinism tell (gpu > modeled D → arrival order can flip → cohort/var break is a TIMING-MODEL limit, not a sim bug). SKIPs on the banked logs (predate P2-6); will populate on the databin1 run and tell us WHY fluxtune breaks. -2. **P2-7 — LAUNCH THE databin1 RUN** (the immediate experimental next step; command below), then re-run the - checker `--max-bin 1` and confirm the cascade: fwdllm/fwdllm_plus `cohort_sequence` should extend past bin 8; - fluxtune expected to still break (GPU overrun) → confirms P2-5 tuning is the fluxtune next step. -3. **P0-2** — controlled grad-determinism-given-order confirmation (also unblocks the P1-6 live test). -4. **P2-4 (optimize)** / **P2-5 (tune fluxtune `perturbation_count`)** — reduce fluxtune GPU below its D budget. -5. **Phase 3** — extend beyond bin 1 once bin-1 parity holds. - -**IMMEDIATE NEXT COMMAND (P2-7):** +2. **✅ P2-7 DONE (databin1 run + `--max-bin 1` check).** Cascade verified — see P2-7 in PHASE 2. Sync RNG-desync + root broken (`var` bit-identical, `timing_overrun=0%`); fluxtune overruns 38% as predicted. +3. **✅ P2-7a CODE LANDED (K-D31; operator chose canonicalize-by-trainer_id) — PENDING VALIDATION RUN.** Trainer + stamps pure `D` (`MODELED_DELAY_S`, both modes); agg `_canonicalize_cohort_commit_order` sorts each cohort by + `(D, str(end))` before the telemetry snapshot + `aggregate()` (reorders `_per_agg_trainer_list` + trailing + grad/jvp slice in lockstep). No-op when delays off / already canonical. 428 mode + **9 new** + `test_fwdllm_commit_canon.py` + 115 async_cifar10 parity green; async byte-identical. **NEXT: the operator runs + the validation command below** → confirm sync `cohort_sequence` order_match→1.0 (var already bit-identical). + A dry `--only fwdllm,fwdllm_plus` attempt crashed on a **missing `h5py`** (environment, not code) → operator + will run in the correct env. +4. **✅ P2-4 GPU optimization — LANDED (K-D32, `jvp_perf_opt`, fluxtune-only, config-gated, bit-identical).** + Implemented in `calculate_jvp` (trainable_idx) + `tc_transformer_trainer_distribute._train_one_batch` + (skip 3 diagnostic passes + reuse winner JVP), threaded via main.py/fl_main.py, enabled in both fluxtune yamls + (default false in trainer_base). Startup `[JVP_PERF_OPT]` confirms 10/10 trainers True (aggregator's eval-only + trainer False, harmless). 13 pytests + 185 fwdllm mode + 115 parity green. NOT retained: vmap (fp32 FD + cancellation), fwd-AD (slower). **NEXT: the batch run** confirms fluxtune `timing_overrun`→0 + cohort set + recovers + real↔sim var parity at multi-bin. +5. **P2-5 (tune fluxtune `perturbation_count`)** — only if the bit-identical cuts above don't fully clear overrun. + Lower it so GPU < min cohort D. Changes the baseline algorithm (deferred, operator call). +5. **P0-2** — controlled grad-determinism-given-order confirmation (also unblocks the P1-6 live test). +6. **Phase 3** — extend beyond bin 1 once bin-1 parity holds. + +**IMMEDIATE NEXT COMMAND (P2-7a validation — confirm K-D31 closes the sync tie):** ``` cd lib/python/examples/fwdllm/expt_scripts -bash run_sequential.sh --mode both --delays on --delay-factor 2 --max-data-id 1 --max-runtime-s 1800 --yes -python run_parity.py --yes --max-bin 1 # then inspect cohort_sequence + timing_overrun per baseline -python plot_step_timing.py # NEW: per-step GPU breakdown (fine step_timing now populated) +# sync baselines only (fluxtune still overruns → deferred to P2-5): +bash run_sequential.sh --mode both --delays on --delay-factor 2 --max-data-id 1 \ + --max-runtime-s 600 --only fwdllm,fwdllm_plus --yes +python run_parity.py --yes --max-bin 1 --baselines fwdllm fwdllm_plus +# EXPECT: cohort_sequence order_match_frac → 1.0 (was 0.5 / 0.0); var still bit-identical. +# grep the agg log for [COMMIT_CANON] to see the tie reorder fire. ``` +(The dry attempt crashed on a missing `h5py` — environment, not code. Run in the env that has the fwdllm deps.) Expected: fwdllm/fwdllm_plus (GPU≈1s < D/2 of 2–9s) → order deterministic → `cohort_sequence` improves; fluxtune (GPU 7.57s > D/2) → `[TIMING_OVERRUN]` fires, order still flips → tune `perturbation_count` down next. @@ -180,9 +200,20 @@ stochastic selectors (`DETERMINISTIC_SELECTORS=∅`, `checks.py:882`) and keys o - [x] **P2-6 DONE — budget-overrun telemetry** (`training_overran` + `remaining_time_s` on `trainer_round`, `[TIMING_OVERRUN]` warning). This is the "keep a tab" watch: if actual GPU > modeled sct, the update arrives after the vclock passed its sct → out-of-order commit. Expect it to fire on fluxtune this run. -- [ ] **P2-7 Verify the cascade closes** — the launch below is the test: fwdllm/fwdllm_plus (GPU≈1s < D/2) - should extend cohort_sequence parity past bin 8; fluxtune (GPU overrun) is expected to still break → confirms - P2-5 is the fluxtune next step. +- [x] **P2-7 DONE — cascade VERIFIED on the databin1 run** (`--delay-factor 2 --max-data-id 1`, + `smoke_logs/20260705_150145` → `experiments/run_20260705_15*`; `run_parity.py --yes --max-bin 1`). + **RNG-desync root BROKEN for the sync baselines:** on bin 1 `cohort_sequence` shows `var` **bit-identical** + real↔sim (fwdllm `0.21480107…`, fwdllm_plus `0.37160512…`; banked had `var_match=0.5`, real≠sim) + + `set_match=1.0` + `cadence_match=1.0` + **`timing_overrun=0%`** ⇒ grads mode-invariant on bin 1. RESIDUAL: + `order_match` 0.5 (fwdllm) / 0.0 (fwdllm_plus) from ONE benign **delay-TIE** — trainers 3 & 9 (…372/…378) + both drew `training_delay_s=13.0` (`_metadata/trainer_registry.yaml`, → D=6.5); their sct ties, real breaks + it by physical arrival & sim by sct-sort, both in the SAME split-half ⇒ var/grads unchanged. **fluxtune:** + `timing_overrun=38%` (GPU≈4.2s > min cohort D 2.0/2.5/3.5s), `set_match=0.22`, `var` mean off 29% → genuine + break, timing-model-limited → P2-5 (`perturbation_count`↓ so GPUsim (both read this config). + jvp_perf_opt: false evaluate_during_training_steps: 100 frequency_of_the_test: 1 is_debug_mode: 0 diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml index 628de0d8c..e14c2b319 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke.yaml @@ -46,6 +46,12 @@ experiments: client_idx_modulo: 100 config_overrides: hyperparameters: + # Fluxtune JVP perf optimizations (simulate_fwdllm.md §L) — all + # BIT-IDENTICAL to the grads (scripts/profile_jvp_opt.py). Cuts + # fluxtune's per-batch forward passes ~37% (25->20) + trainable-only + # FD, so GPU drops under the modeled mobile delay. Set true on BOTH + # real+sim (must match). Set false to revert to the un-optimized path. + jvp_perf_opt: true data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 # Uniform/IID partition for smoke tests -- avoids the niid_label diff --git a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml index 9ad99a0b8..8bf173c7c 100644 --- a/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml +++ b/lib/python/examples/fwdllm/expt_scripts/fluxtune_n10_smoke_sim.yaml @@ -51,6 +51,10 @@ experiments: client_idx_modulo: 100 config_overrides: hyperparameters: + # Fluxtune JVP perf optimizations (simulate_fwdllm.md §L) — BIT-IDENTICAL + # grads; MUST match the real yaml (both sides read this config, so the + # optimization keeps real<->sim parity). Set false to revert. + jvp_perf_opt: true data_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/data_files/agnews_data.h5 partition_file_path: /coc/scratch/dgarg/fl_datasets/fwdllm/fednlp_data/partition_files/agnews_partition.h5 # Uniform/IID partition for smoke tests -- avoids the niid_label diff --git a/lib/python/examples/fwdllm/fluxtune_contributions.md b/lib/python/examples/fwdllm/fluxtune_contributions.md new file mode 100644 index 000000000..6821bb046 --- /dev/null +++ b/lib/python/examples/fwdllm/fluxtune_contributions.md @@ -0,0 +1,175 @@ +# Fluxtune: Systems & ML Contributions for On-Device LLM Fine-Tuning + +**Thesis.** Fluxtune makes federated LLM fine-tuning practical on *memory- and hardware-constrained* +devices by training with **forward-mode (backprop-free) gradients** over **parameter-efficient adapters**, +aggregated **asynchronously**, with an **informed (JVP-magnitude) perturbation selection**. The result is a +training regime whose **peak memory is independent of both model depth and the number of perturbations**, whose +compute is **pure forward inference** (no autograd graph, no backward GEMMs), and which therefore maps onto the +**inference-only accelerators** (mobile NPUs/DSPs) that cannot run backpropagation at all. This document states the +contributions precisely, quantifies them against baselines and alternatives, and separates what is *measured* from +what is *argued*. + +> Numbers below are from `scripts/profile_jvp_opt.py` (reuses the production `create_model` + `calculate_jvp`): +> DistilBERT-base + AdapterHub bottleneck adapters, batch 8, seq 192, fp16/autocast, NVIDIA A40. They are a *clean +> single-trainer* profile; a shared-GPU run multiplies wall-time by the contention factor, but **pass-counts, +> ratios, and memory transfer directly**. Mobile figures are *argued from structural properties*, not measured. + +--- + +## 1. The problem & why the baseline choice matters + +On-device fine-tuning of a transformer via **backpropagation** requires (i) a full **autograd graph** (stored +activations for the backward pass), (ii) **training-mode** kernels (backward GEMMs, transposes), and (iii) an +optimizer state. Mobile SoCs expose **inference-optimized** NPUs/DSPs (forward GEMMs, quantized, no autograd) and +tight memory budgets — so backprop fine-tuning is often infeasible on the device that owns the data. + +**Forward-gradient** methods (FedFwd / FwdLLM family) replace the backward pass with **directional derivatives** +estimated from *forward passes only*. Fluxtune is a system built on this idea. Its baselines in this repo are the +**sync** forward-grad variants (`fwdllm`: cosine-similarity perturbation selection; `fwdllm++`: per-iteration +reselection). Fluxtune's distinguishing choices — **async aggregation + JVP-magnitude selection + adapter PEFT + +server-side LR** — are what this document evaluates. + +--- + +## 2. The computation, precisely + +Each perturbation's gradient estimate is a **central finite-difference JVP** (`fwdgrad_utils.calculate_jvp`): + +``` +jvp(v) = ( f(θ + h·v) − f(θ − h·v) ) / (2h), h = 0.01, under autocast + no_grad +``` + +- `f` = one full forward pass of the (frozen-backbone) model + cross-entropy loss. **1 perturbation = 2 forward + passes.** No backward pass, no stored graph. +- **Fluxtune** evaluates `P` candidate perturbations (`perturbation_count`, default 10) and **selects the one with + the largest |jvp|** (the steepest measured descent direction) → `2P` forward passes. The **sync `fwdllm` + baseline** selects by cosine similarity to a cached gradient → **0 forward passes** for selection, 1 final JVP. +- **PEFT:** the DistilBERT backbone (66.4M params) is **frozen**; only **bottleneck adapters in every layer + the + head** are trainable — **1.04M / 67.4M = 1.5%**. The perturbation `v` is non-zero only on trainable params. + +--- + +## 3. Systems contributions + +### 3.1 Memory is flat — in perturbation count *and* in depth +Because there is **no autograd graph**, peak memory ≈ *model weights + one in-flight forward*. It does **not grow +with `P`** (perturbations are evaluated one at a time and discarded) and does **not accumulate activations for a +backward**. + +| regime | peak memory (measured) | +|---|---| +| forward-grad, any P (trainable-only, retained) | **3.19 GB** | +| forward-grad, any P (current) | 3.44 GB | +| backprop reference (1 fwd + 1 bwd) | 3.71 GB | + +At this favorable-to-backprop config (small batch, 98.5% frozen) the gap is ~14%; it **widens with batch size and +sequence length** (backprop's stored activations scale with both; forward-grad's do not) and with the **trainable +fraction**. The durable claim is structural: **forward-grad removes the autograd graph entirely**, so memory is +bounded by inference, not training. + +### 3.2 Compute is pure forward inference (hardware fit) +Every FLOP fluxtune spends is a **forward-pass GEMM** — the exact operator set an inference NPU/DSP is built for. +There are **no backward GEMMs, no transposed weight matmuls, no autograd bookkeeping**. This is the crux of the +**mobile-generality argument**: a device that can *run* the model can *train* it under fluxtune, with no autograd +runtime. (Argued from operator structure; not yet measured on-device.) + +### 3.3 Compute/latency cost — characterized honestly +Forward-grad trades memory for **time**: many forward passes instead of one forward+backward. The selection is the +surcharge, and it is linear in `P`: + +| path | fwd passes | ms/batch (clean A40) | +|---|---|---| +| sync `fwdllm` (opt) | 2 | 16 | +| **fluxtune P=1** | 2 | 16 — *equals sync* | +| fluxtune P=5 | 10 | 80 | +| **fluxtune P=10** | 20 | 159 | +| backprop reference | 1f+1b | 17 | + +Fluxtune's cost is **`2P × per-pass`** → **10× the sync compute at P=10**, collapsing to parity at `P=1`. The +JVP-selection is thus a **tunable accuracy/compute knob**, not a fixed tax. Per-pass ≈ 8–10 ms here; a shared-GPU +deployment multiplies wall-time by the contention factor (the real 10-trainer run saw ~0.21 s/pass). + +### 3.4 Communication +Only the **1.5% adapter** parameters are exchanged per round (PEFT), and async aggregation (`agg_goal=3 < K=10`) +commits updates **as stragglers arrive** with no synchronization barrier — directly targeting the intermittent +connectivity and device heterogeneity of the mobile setting. + +--- + +## 4. Machine-learning contributions + +### 4.1 Informed perturbation selection (JVP vs cosine / random) +Random-direction forward-grad (MeZO-style) and the cosine-similarity baseline pick a perturbation *without* +measuring its effect on the loss. Fluxtune **measures** each candidate's directional derivative and keeps the +steepest — a better single-sample gradient estimate per communication round, improving sample/round efficiency at +the cost of the `2P` forward passes (§3.3). The value of the knob `P` is an accuracy/compute trade the operator +controls. + +### 4.2 Finite-difference numerics & precision (a measured caution + an exactness result) +- The FD estimate subtracts two **O(1)** losses that differ by **O(h)** ≈ 1e-3. Under **fp16/fp32** this is + **catastrophic cancellation**: the JVP retains only ~1–2 significant figures, so the perturbation *ranking* is + mildly precision-limited. This is a genuine finding about forward-grad-by-finite-difference in mixed precision, + relevant to any deployment that lowers precision for the mobile NPU. +- **Batching is mathematically exact.** Vectorizing all `P` perturbations (`torch.func.vmap`) yields JVPs that are + **bit-identical to the sequential loop in fp64** and deterministic run-to-run — the fp32 divergence is *only* the + cancellation above, not a batching error. This bounds when the 2× batching speedup is safe to adopt. + +### 4.3 Tensor-operation profile +The finite-difference perturbation touches **only trainable tensors** (`p ± h·v` with `v=0` on frozen params, so +`p−0=p` exactly). The forward is otherwise identical inference. There is **no backward transpose-GEMM, no +grad-accumulation kernel**. This minimal op set is what makes the trainable-only optimization (§5) and the +inference-hardware mapping (§3.2) possible. + +--- + +## 5. Fidelity-preserving optimizations (retained) + +Validated **bit-identical** (`max|Δjvp| = 0`) by the profiler — they change *cost*, never the computed gradient, +so training fidelity and real↔sim simulator parity are untouched: + +1. **Trainable-only finite difference** — skip the `p − h·0 = p` arithmetic on the 98.5% frozen backbone: + **1.26× faster, −251 MB.** +2. **Remove redundant/diagnostic forward passes** — 3 passes that only fed a log line, plus (fluxtune) reusing the + selected perturbation's already-computed JVP: **fluxtune 25→20 passes, sync 5→2.** + +**Combined: sync −68%, fluxtune −37% GPU time, zero fidelity change.** This alone is expected to bring fluxtune's +per-batch compute under its modeled mobile-delay budget. + +**Deliberately *not* adopted** (they change fidelity / are inferior here): **vmap batching** (2× but re-baselines +the fp32 trajectory via §4.2 cancellation — exact only in fp64), **exact forward-mode AD** (0.5×, needs eager +attention, different math), **lowering `P`** (changes the algorithm — an accuracy knob, not a free optimization). + +--- + +## 6. Comparison summary + +| dimension | backprop FL | sync forward-grad (`fwdllm`/`++`) | **fluxtune** | +|---|---|---|---| +| training memory | activation graph (grows w/ batch·seq·depth) | flat (inference) | **flat (inference)** | +| hardware needed | autograd/training runtime | inference-only | **inference-only** | +| perturbation selection | n/a | cosine to cached grad (0 fwd) | **JVP magnitude (2P fwd, informed)** | +| aggregation | sync/async | **sync (barrier)** | **async (straggler-tolerant)** | +| compute/round | 1 fwd + 1 bwd | 1 JVP (2 fwd) | 2P fwd (tunable) | +| communication | full or PEFT | PEFT adapters | **PEFT adapters** | +| best when | server-class HW | homogeneous, fast clients | **memory/HW-constrained, heterogeneous, intermittent clients** | + +**Where fluxtune wins:** the on-device regime — constrained memory, inference-only accelerators, stragglers, and +intermittent availability — where backprop is infeasible and a synchronization barrier stalls on the slowest phone. +It buys a better per-round gradient (JVP selection) and straggler tolerance (async) for a forward-pass compute cost +that the §5 optimizations cut ~40% without any fidelity loss, and that `P` tunes directly. + +**Honest limits:** fluxtune is **compute-heavier** than the sync baselines (≈10× at P=10) and than backprop +per-round; its advantage is memory/hardware feasibility and robustness, not raw FLOPs. The FD JVP is +precision-sensitive in fp16 (§4.2). The memory gap over backprop is modest at small scale and grows with +batch/seq/trainable-fraction. + +--- + +## 7. Reproducibility + +All figures: `scripts/profile_jvp_opt.py` (env `test_fwdllm`), which reuses the production model builder and JVP +math so measured gains transfer directly to the trainer. It reports, per stage, forward-pass count, latency +(mean±std, warmup + `cuda.synchronize`), peak memory, speedup, and a BIT-IDENTICAL / WITHIN-TOL / DIVERGED verdict +against the ground-truth sequential path, plus an fp64 cancellation diagnosis (`--fp64-check`). The high-fidelity +real↔sim simulator that validates fluxtune's *training dynamics* under a virtual clock is documented separately in +`simulate_fwdllm.md` (compute profile persisted there in §L). diff --git a/lib/python/examples/fwdllm/scripts/profile_jvp_opt.py b/lib/python/examples/fwdllm/scripts/profile_jvp_opt.py new file mode 100644 index 000000000..cfe215d01 --- /dev/null +++ b/lib/python/examples/fwdllm/scripts/profile_jvp_opt.py @@ -0,0 +1,436 @@ +#!/usr/bin/env python3 +"""Fluxtune/fwdllm forward-grad JVP optimization profiler (TEMP, real-code reuse). + +Isolated latency + correctness profiler for the perturbation/JVP compute that +dominates fluxtune GPU time (`_train_one_batch` in +`trainer/forward_training/tc_transformer_trainer_distribute.py`). It measures +the GAIN and validates the numerical fidelity of each optimization stage BEFORE +we change the real trainer (simulate_fwdllm.md principle #11/#16: prove the +mechanism first). + +It deliberately REUSES the production code so a validated speedup transfers +directly: the real model (`expts.initializer.create_model` → DistilBERT-base + +AdapterHub bottleneck adapters, backbone frozen) and the real JVP math +(`fwdgrad_utils.calculate_jvp` / `functional_get_loss`). Only the current +fluxtune/fwdllm config is targeted (distilbert-base-uncased, agnews→4 labels, +seq 192, batch 8, adapter PEFT, perturbation_count 10) — not generic. + +STAGES (each vs Stage 0 as the numerical ground truth): + 0 baseline the real `calculate_jvp` in a Python loop over perturbations + (central finite difference, 2 forward passes each = the real code) + 1 vmap-FD the SAME finite-difference JVP, all perturbations batched via + torch.func.vmap (frozen params broadcast, only adapters vary) + 2 fwd-AD exact forward-mode-AD JVP (torch.func.jvp) — DIFFERENT math + (true directional derivative, not a finite difference); a small + systematic gap vs stage 0 is EXPECTED (the O(h^2) FD error). + +Per stage: forward-pass count, latency (mean±std over repeats, warmup + +cuda.sync), speedup vs baseline, max|Δjvp| / max rel-diff vs baseline, and a +verdict: BIT-IDENTICAL / WITHIN-TOL / DIVERGED. + +Run in the test_fwdllm env: + conda run -n test_fwdllm python scripts/profile_jvp_opt.py + conda run -n test_fwdllm python scripts/profile_jvp_opt.py --no-autocast +""" +from __future__ import annotations + +import argparse +import statistics +import sys +import time +from functools import partial +from pathlib import Path + +import torch + +# lib/python on the path so `examples.fwdllm...` imports resolve +_LIB_PYTHON = Path(__file__).resolve().parents[3] +if str(_LIB_PYTHON) not in sys.path: + sys.path.insert(0, str(_LIB_PYTHON)) + +import functorch as fc # noqa: E402 +from torch.func import jvp as func_jvp # noqa: E402 +from torch.func import vmap # noqa: E402 + +from examples.fwdllm.expts.initializer import create_model, set_seed # noqa: E402 +from examples.fwdllm.trainer.model.transformer.model_args import ( # noqa: E402 + ClassificationArgs, +) +from examples.fwdllm.trainer.forward_training.fwdgrad_utils import ( # noqa: E402 + calculate_jvp, + functional_get_loss, +) + + +# ─────────────────────────── real model (reused) ─────────────────────────── +def build_model(num_labels, seq, attn="sdpa"): + """Exactly the trainer's construction path (main.py): DistilBERT-base + + AdapterHub adapter, backbone frozen, only adapters + head trainable. + + `attn`: "sdpa" (the real default, fastest) or "eager" — forward-mode AD + (stage 2) is not implemented for scaled_dot_product_attention, so it needs + eager attention to run at all.""" + ma = ClassificationArgs() + ma.model_name = "distilbert-base-uncased" + ma.model_type = "distilbert" + ma.load(ma.model_name) # no-op for a hub name (no local model_args.json) + ma.num_labels = num_labels + ma.client_idx = 0 + ma.config["attn_implementation"] = attn + ma.update_from_dict( + { + "peft_method": "adapter", # BnConfig, output_adapter per layer + "do_lower_case": True, + "max_seq_length": seq, + "train_batch_size": 8, + "manual_seed": 42, + "var_control": True, + "perturbation_sampling": True, + "select_perturbation_using_jvp": True, # fluxtune JVP-selection path + "fp16": True, + "reprocess_input_data": False, + "overwrite_output_dir": True, + } + ) + ma.config["num_labels"] = num_labels + _, model, _ = create_model(ma, formulation="classification") + return model + + +# ─────────────────────────── JVP stage implementations ─────────────────────────── +def _loss_partial(fmodel, buffers, num_labels, x, labels): + # == _compute_forward_jvp's `f`: functional_get_loss(params, ...) + return partial( + functional_get_loss, + model=fmodel, + buffers=buffers, + num_classes=num_labels, + x=x, + t=labels, + ) + + +def stage0_baseline(f, params, V, h): + """The real code: calculate_jvp per perturbation (2 forward passes each).""" + jvps = [] + for v in V: # V: list[tuple(dir per param)] + _loss, jvp = calculate_jvp(f, params, v) + jvps.append(jvp) + return torch.stack([j.float().reshape(()) for j in jvps]) + + +def stage0b_trainable_only(fmodel, buffers, num_labels, x, labels, params, V, + train_idx, h, device): + """Same central-FD JVP but perturb ONLY the trainable params (frozen p-h*0=p + is skipped — the frozen tensors are reused as-is). Bit-identical to stage 0 + (p-0.0=p in IEEE), avoids copying the 98.5% frozen backbone twice/perturbation.""" + jvps = [] + with torch.no_grad(): + for v in V: + plus = list(params) + minus = list(params) + for i in train_idx: + plus[i] = params[i] + h * v[i] + minus[i] = params[i] - h * v[i] + with torch.autocast(device_type=device.type, enabled=(device.type == "cuda")): + lp = functional_get_loss(tuple(plus), model=fmodel, buffers=buffers, + num_classes=num_labels, x=x, t=labels) + lm = functional_get_loss(tuple(minus), model=fmodel, buffers=buffers, + num_classes=num_labels, x=x, t=labels) + jvps.append((lp - lm) / (2 * h)) + return torch.stack([j.float().reshape(()) for j in jvps]) + + +def stage1_vmap_fd(fmodel, buffers, num_labels, x, labels, params, train_idx, + V_train, h, autocast_on, device): + """Same central-FD JVP, all perturbations batched via vmap. Only the + trainable adapter dirs get a leading P dim (frozen params broadcast) so we + never materialise P copies of the 66M frozen backbone.""" + def jvp_one(vt_tuple): # vt_tuple: per-trainable dir (no P dim inside vmap) + plus = list(params) + minus = list(params) + for k, i in enumerate(train_idx): + plus[i] = params[i] + h * vt_tuple[k] + minus[i] = params[i] - h * vt_tuple[k] + with torch.autocast(device_type=device.type, enabled=autocast_on): + lp = functional_get_loss(tuple(plus), model=fmodel, buffers=buffers, + num_classes=num_labels, x=x, t=labels) + lm = functional_get_loss(tuple(minus), model=fmodel, buffers=buffers, + num_classes=num_labels, x=x, t=labels) + return ((lp - lm) / (2 * h)).float().reshape(()) + + in_dims = (tuple(0 for _ in train_idx),) + return vmap(jvp_one, in_dims=in_dims)(V_train) + + +def stage2_forward_mode(f, params, V, h): + """Exact forward-mode AD (torch.func.jvp): the true directional derivative, + not a finite difference. Different math -> a small O(h^2) gap vs stage 0 is + expected; answers 'is exact fwd-AD faster/more accurate than 2 FD passes?'.""" + jvps = [] + for v in V: + _loss, jv = func_jvp(f, (params,), (v,)) + jvps.append(jv) + return torch.stack([j.float().reshape(()) for j in jvps]) + + +# ─────────────────────────────── harness ─────────────────────────────── +def _time(fn, warmup, repeats, device): + for _ in range(warmup): + fn() + if device.type == "cuda": + torch.cuda.synchronize() + ts = [] + for _ in range(repeats): + t0 = time.perf_counter() + fn() + if device.type == "cuda": + torch.cuda.synchronize() + ts.append((time.perf_counter() - t0) * 1e3) + return statistics.mean(ts), (statistics.stdev(ts) if len(ts) > 1 else 0.0) + + +def _measure(fn, warmup, repeats, device): + """Latency (mean±std ms) AND peak CUDA memory (MB) for one call of fn.""" + for _ in range(warmup): + fn() + if device.type == "cuda": + torch.cuda.synchronize() + torch.cuda.empty_cache() + torch.cuda.reset_peak_memory_stats() + ts = [] + for _ in range(repeats): + t0 = time.perf_counter() + fn() + if device.type == "cuda": + torch.cuda.synchronize() + ts.append((time.perf_counter() - t0) * 1e3) + peak = (torch.cuda.max_memory_allocated() / 1024**2) if device.type == "cuda" else 0.0 + return statistics.mean(ts), (statistics.stdev(ts) if len(ts) > 1 else 0.0), peak + + +def backprop_ref(num_labels, seq, batch, h, seed, attn, autocast_on, device): + """Standard BACKPROP fine-tune step (1 forward + 1 backward on the trainable + adapters/head) — the memory/time contrast forward-grad is chosen to avoid. + Fresh model so functorch's make_functional does not interfere.""" + model = build_model(num_labels, seq, attn).to(device).eval() + g = torch.Generator(device="cpu").manual_seed(seed) + x = torch.randint(0, 30522, (batch, seq), generator=g).to(device) + labels = torch.randint(0, num_labels, (batch,), generator=g).to(device) + + def step(): + model.zero_grad(set_to_none=True) + with torch.autocast(device_type=device.type, enabled=autocast_on): + out = model(x) + logits = out[0] if not hasattr(out, "logits") else out.logits + loss = torch.nn.functional.cross_entropy(logits.float(), labels) + loss.backward() + return step + + +def _verdict(ref, got): + if ref.shape != got.shape: + return "SHAPE-MISMATCH", float("nan"), float("nan") + if torch.equal(ref, got): + return "BIT-IDENTICAL", 0.0, 0.0 + mad = (ref - got).abs().max().item() + rel = ((ref - got).abs() / (ref.abs() + 1e-12)).max().item() + tag = "WITHIN-TOL" if torch.allclose(ref, got, rtol=1e-3, atol=1e-4) else "DIVERGED" + return tag, mad, rel + + +def _fp64_check(attn, num_labels, seq, batch, P, h, seed, device): + """Prove the fp32 vmap-FD divergence is FINITE-DIFFERENCE CANCELLATION, not + a batching bug: in fp64 the sequential and vmap FD JVPs should agree to + ~machine eps. Also prints the JVP signal magnitude so 'DIVERGED' on tiny + (lp-lm) values is interpretable.""" + model = build_model(num_labels, seq, attn).to(device).double().eval() + mask = [p.requires_grad for p in model.parameters()] + fmodel, params, buffers = fc.make_functional_with_buffers(model) + train_idx = [i for i, m in enumerate(mask) if m] + g = torch.Generator(device="cpu").manual_seed(seed) + x = torch.randint(0, 30522, (batch, seq), generator=g).to(device) + labels = torch.randint(0, num_labels, (batch,), generator=g).to(device) + gp = torch.Generator(device=device).manual_seed(seed + 1) + V, per_train = [], [[] for _ in train_idx] + for _ in range(P): + v = [torch.randn(p.shape, generator=gp, device=device, dtype=p.dtype) + if mask[i] else torch.zeros_like(p) for i, p in enumerate(params)] + V.append(tuple(v)) + for k, i in enumerate(train_idx): + per_train[k].append(v[i]) + V_train = tuple(torch.stack(per_train[k]) for k in range(len(train_idx))) + f = _loss_partial(fmodel, buffers, num_labels, x, labels) + seq_jvp = stage0_baseline(f, params, V, h).double() + vmap_jvp = stage1_vmap_fd(fmodel, buffers, num_labels, x, labels, params, + train_idx, V_train, h, False, device).double() + tag, mad, rel = _verdict(seq_jvp, vmap_jvp) + print(f" fp64 seq-FD vs vmap-FD : {tag} max|Δ|={mad:.2e} rel={rel:.2e}") + print(f" jvp magnitude (fp64) : |jvp| in " + f"[{seq_jvp.abs().min():.2e}, {seq_jvp.abs().max():.2e}] " + f"(FD subtracts losses ~O(1) -> cancellation floors fp32 precision)") + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--perturbations", type=int, default=10) # perturbation_count + ap.add_argument("--batch", type=int, default=8) + ap.add_argument("--seq", type=int, default=192) + ap.add_argument("--labels", type=int, default=4) # agnews + ap.add_argument("--h", type=float, default=0.01) + ap.add_argument("--repeats", type=int, default=20) + ap.add_argument("--warmup", type=int, default=5) + ap.add_argument("--no-autocast", action="store_true", + help="fp32 (default: autocast, matching the real fp16 code)") + ap.add_argument("--attn", choices=["sdpa", "eager"], default="sdpa", + help="eager needed for forward-mode AD (stage 2); sdpa is the real default") + ap.add_argument("--fp64-check", action="store_true", + help="run the fp64 seq-vs-vmap cancellation diagnosis") + ap.add_argument("--seed", type=int, default=42) + args = ap.parse_args(argv) + + device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + autocast_on = (not args.no_autocast) and device.type == "cuda" + set_seed(args.seed) + + model = build_model(args.labels, args.seq, args.attn).to(device).eval() + trainable_mask = [p.requires_grad for p in model.parameters()] + fmodel, params, buffers = fc.make_functional_with_buffers(model) + train_idx = [i for i, m in enumerate(trainable_mask) if m] + + total = sum(p.numel() for p in params) + trainable = sum(params[i].numel() for i in train_idx) + + # fixed-seed inputs (the JVP-METHOD comparison is data-independent) — the + # real forward is model(params, buffers, x)[0]; x = token ids, no mask. + g = torch.Generator(device="cpu").manual_seed(args.seed) + x = torch.randint(0, 30522, (args.batch, args.seq), generator=g).to(device) + labels = torch.randint(0, args.labels, (args.batch,), generator=g).to(device) + + # perturbations: randn on trainable, zero on frozen (== _prepare_perturbation_tensors) + gp = torch.Generator(device=device).manual_seed(args.seed + 1) + V = [] + V_train = [] # trainable-only stacked view for vmap + per_train = [[] for _ in train_idx] + for _ in range(args.perturbations): + v = [] + for i, p in enumerate(params): + if trainable_mask[i]: + d = torch.randn(p.shape, generator=gp, device=device, dtype=p.dtype) + else: + d = torch.zeros_like(p) + v.append(d) + V.append(tuple(v)) + for k, i in enumerate(train_idx): + per_train[k].append(v[i]) + V_train = tuple(torch.stack(per_train[k]) for k in range(len(train_idx))) # each (P, *shape) + + f = _loss_partial(fmodel, buffers, args.labels, x, labels) + + print("=" * 94) + print("fluxtune/fwdllm JVP optimization profiler (TEMP — reuses create_model + calculate_jvp)") + print(f"device={device.type} autocast={autocast_on} seed={args.seed}") + print(f"model: distilbert-base + AdapterHub adapters | params={total/1e6:.1f}M " + f"trainable={trainable/1e6:.3f}M ({100*trainable/total:.1f}%) [{len(train_idx)} tensors]") + print(f"batch={args.batch} seq={args.seq} P(perturbations)={args.perturbations} " + f"h={args.h} repeats={args.repeats} (+{args.warmup} warmup)") + print("=" * 94) + print(f"{'stage':22s} {'fwd/batch':>9s} {'ms/batch':>13s} {'speedup':>8s} " + f"{'max|Δjvp|':>11s} {'max rel':>10s} verdict") + print("-" * 94) + + stages = [ + ("0 baseline (calc_jvp)", 2 * args.perturbations, + lambda: stage0_baseline(f, params, V, args.h)), + ("0b FD trainable-only", 2 * args.perturbations, + lambda: stage0b_trainable_only(fmodel, buffers, args.labels, x, labels, + params, V, train_idx, args.h, device)), + ("1 vmap-FD (all pert)", 2 * args.perturbations, + lambda: stage1_vmap_fd(fmodel, buffers, args.labels, x, labels, params, + train_idx, V_train, args.h, autocast_on, device)), + ("2 forward-mode AD", 1 * args.perturbations, + lambda: stage2_forward_mode(f, params, V, args.h)), + ] + + base = None + for name, n_pass, fn in stages: + try: + out = fn().detach().float() + ms, sd = _time(fn, args.warmup, args.repeats, device) + except Exception as e: + print(f"{name:22s} {n_pass:>9d} {'ERR':>13s} " + f"{type(e).__name__}: {str(e)[:44]}") + continue + if base is None: + base = (ms, out) + print(f"{name:22s} {n_pass:>9d} {ms:>9.2f}±{sd:>4.1f} {'1.00x':>8s} " + f"{'—':>11s} {'—':>10s} REFERENCE") + else: + verdict, mad, rel = _verdict(base[1], out) + print(f"{name:22s} {n_pass:>9d} {ms:>9.2f}±{sd:>4.1f} {base[0]/ms:>6.2f}x " + f"{mad:>11.2e} {rel:>10.2e} {verdict}") + + print("-" * 94) + + # ── COST COMPARISON: sync vs fluxtune (perturbation sweep) vs backprop ── + # Both baselines run the SAME _train_one_batch; the difference is the + # perturbation-SELECTION cost: fwdllm/sync uses cos-sim (0 forward passes, + # 1 final JVP=2 passes); fluxtune does JVP-selection (2*P passes) + 1 final. + # We retain ONLY fidelity-preserving (bit-identical) cuts: drop the 3 + # diagnostic-only passes (both) + reuse the winner JVP (fluxtune) => the + # per-batch forward-pass count below. + # Measure the two per-forward-pass unit costs + peak memory (memory is flat + # in P: no autograd graph, one forward held at a time). Totals are then + # passes x per-pass — transparent projection. + P = args.perturbations + fms, _, fmem = _measure(lambda: stage0_baseline(f, params, V[:P], args.h), + args.warmup, args.repeats, device) + oms, _, omem = _measure( + lambda: stage0b_trainable_only(fmodel, buffers, args.labels, x, labels, + params, V[:P], train_idx, args.h, device), + args.warmup, args.repeats, device) + full_pp, opt_pp = fms / (2 * P), oms / (2 * P) # ms per forward pass + print("COST MODEL — forward-grad (finite-difference JVP), per training batch") + print(f"per-pass: full-param={full_pp:.2f}ms trainable-only={opt_pp:.2f}ms " + f"(1.{round(100*(full_pp/opt_pp-1)):02d}x, BIT-IDENTICAL) | peak mem " + f"full={fmem:.0f}MB opt={omem:.0f}MB (flat in P)") + print(f"{'path':32s} {'fwd passes':>10s} {'ms/batch':>10s} note") + print("-" * 94) + # (passes, per-pass, note): current uses full-param; opt uses trainable-only + rows = [ + ("sync fwdllm (current)", 5, full_pp, "cos-sim select(0) + 1 JVP + 3 diag"), + ("sync fwdllm (opt)", 2, opt_pp, "1 JVP trainable-only; -3 diag [RETAIN]"), + ("fluxtune P=1 (opt)", 2, opt_pp, "== sync when P=1"), + ("fluxtune P=2 (opt)", 4, opt_pp, ""), + ("fluxtune P=5 (opt)", 10, opt_pp, ""), + (f"fluxtune P={P} (current)", 2 * P + 5, full_pp, "2P + 2 redundant + 3 diag"), + (f"fluxtune P={P} (opt)", 2 * P, opt_pp, "2P trainable-only; winner reused; -3 diag [RETAIN]"), + ] + for name, npass, pp, note in rows: + print(f"{name:32s} {npass:>10d} {npass*pp:>8.1f} {note}") + + bp = backprop_ref(args.labels, args.seq, args.batch, args.h, args.seed, + args.attn, autocast_on, device) + bms, bsd, bmem = _measure(bp, args.warmup, args.repeats, device) + print(f"{'backprop ref (1 fwd+1 bwd)':30s} {'1f+1b':>10s} {bms:>9.1f}±{bsd:>3.1f} {bmem:>9.0f}" + f" contrast: stores activations") + print("-" * 94) + print(f"forward-grad peak mem ≈ model+1 forward (no autograd graph); backprop " + f"stores activations ({bmem:.0f} MB).") + print("-" * 94) + + if args.fp64_check: + print("numerical diagnosis:") + _fp64_check(args.attn, args.labels, args.seq, args.batch, + args.perturbations, args.h, args.seed, device) + print("-" * 94) + print("stage 0 = numerical ground truth (the real calculate_jvp loop). Real fluxtune per") + print("batch also does +2 redundant + 3 diagnostic-only passes (removable, bit-identical)") + print("on top of P*2 selection passes. fwdllm uses cos-sim selection (no JVP-selection loop).") + print("forward-mode AD (stage 2) needs --attn eager (not impl for SDPA).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index 2c4c740d0..dcfa5da9c 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -124,13 +124,23 @@ not delay-deterministic; fwdllm also uses a flat-additive delay, not the remaind deterministic GIVEN matched order** (batch/seed mode-invariant) → this is a sim ORDER bug, NOT nondeterminism → exact cadence parity IS achievable once order matches. **Full diagnosis + fix plan: [PARITY_LOGICAL_TASKS.md](PARITY_LOGICAL_TASKS.md).** -> **STATUS (2026-07-05 checkpoint):** the checker now GATES on this (enforced `cohort_sequence` EXACT rung + -> V2 mean-guard, P1) and the timing-model fix has LANDED (**K-D29** remainder-wait `max(gpu,D)` + per-trainer D + -> crc32 straggler off + `perturbation_count` knob). **P1-4/P1-5 now RESOLVED** (K-D30): full-cohort determinism -> gate un-gates fwdllm's selection/aggregation_sequence/utility to GENUINE enforcement (fluxtune/fwdllm_plus stay -> gated); new `timing_overrun` DIAG rung surfaces the K-D29 overrun tell; P1-4 assessed redundant. 419 mode + 115 -> async_cifar10 parity tests green; banked scoreboard stable. **Not yet run.** Pending before close: the -> **databin1 validation run** (`--delay-factor 2 --max-data-id 1`). Resume steps: the tracker's SESSION CHECKPOINT. +> **STATUS (2026-07-05 — DATABIN1 RUN LANDED, `--delay-factor 2 --max-data-id 1`, `smoke_logs/20260705_150145`).** +> The K-D29 model fix **broke the RNG-desync root** for the sync baselines: on bin 1 `cohort_sequence` now shows +> **`var` BIT-IDENTICAL** real↔sim (fwdllm `0.21480107…`==, fwdllm_plus `0.37160512…`==; banked logs had +> `var_match_frac=0.5` with real≠sim) + `set_match=1.0` + `cadence_match=1.0`, and **`timing_overrun=0%`** — grads +> are now mode-invariant on bin 1. The SOLE residual is `order_match<1.0` from **one benign delay-TIE swap**: +> trainers 3 & 9 (…372/…378) both drew `training_delay_s=13.0` (registry, → D=6.5) so their sct ties; real breaks +> it by physical arrival, sim by sct-sort — but both land in the SAME split-half, so `var`/cadence/grads are +> unchanged. `cohort_sequence` (EXACT order) still FAILs on it. **fluxtune** behaves exactly as predicted: +> `timing_overrun=38%` (GPU≈4.2s > min cohort D of 2.0/2.5/3.5s), `set_match=0.22`, `var` mean off 29% → genuine +> break, **timing-model-limited** → fix is P2-5 (`perturbation_count`↓ to put GPU **SYNC TIE FIX LANDED (K-D31, operator chose canonicalize-by-trainer_id):** the trainer stamps its pure modeled +> delay `D` (`MODELED_DELAY_S`, both modes); the aggregator sorts each cohort's commits by `(D, str(end))` before +> the telemetry snapshot + `aggregate()`, so equal-D ties break by trainer_id IDENTICALLY in real and sim → +> receive order matches exactly while `var`/grads stay bit-identical (reorder only ever moves a grad WITHIN a +> split-half). 428 mode + 9 new canon + 115 async_cifar10 parity tests green; async byte-identical. +> **PENDING: the operator's re-run** (`--only fwdllm,fwdllm_plus --delay-factor 2 --max-data-id 1`) to confirm +> `cohort_sequence` order_match→1.0. Full diagnosis + numbers: [PARITY_LOGICAL_TASKS.md](PARITY_LOGICAL_TASKS.md) P2-7/P2-7a. ### Open issues (OPEN only — closed items live in §G/§H) | # | issue | baseline(s) | next step | @@ -419,5 +429,74 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso limit not a sim bug). P1-4 (async_cifar10 cohort adapter) assessed redundant, not built. Sim/checker-only → async_cifar10 byte-identical; banked scoreboard stable. +- **K-D31** — **canonical `(D, trainer_id)` cohort commit order (P2-7a; operator chose canonicalize-by-id).** The + databin1 run left ONE sync residual: two trainers sharing a registry delay (D=6.5) swap in receive order (real + breaks the sct tie by physical arrival, sim by sct-sort) — benign (both in the same split-half → `var`/grads + bit-identical) but the EXACT-order `cohort_sequence` rung flags it. Fix: trainer stamps pure `D` + (`MODELED_DELAY_S`, both modes, deterministic from the registry — unlike sct which folds in GPU jitter); agg + sorts each cohort by `(D, str(end))` in `_canonicalize_cohort_commit_order` before the telemetry snapshot + + `aggregate()`, reordering `_per_agg_trainer_list` + the trailing cohort slice of the grad/jvp lists in lockstep. + No-op when delays off / already canonical (real still receives in strict D-order, K-D29 — only ties move). + Sim/agg-only → async_cifar10 byte-identical; 428 mode + 9 canon + 115 parity green. + +- **K-D32** — **fluxtune JVP perf optimizations (`jvp_perf_opt`, §L; fluxtune-only, config-gated, all + BIT-IDENTICAL).** fluxtune's forward-grad JVP was ~10× the sync compute (2P=20 selection passes + 5 removable), + overrunning its modeled mobile delay → out-of-order commits (#1d). Fix (validated by + `scripts/profile_jvp_opt.py`): (a) trainable-only finite difference — `calculate_jvp(trainable_idx=…)` skips + `p−h·0=p` on the 98.5% frozen backbone (1.26× + −251 MB); (b) skip the 3 diagnostic-only forward passes; (c) + reuse the selected perturbation's cached JVP. Combined fluxtune −37% (sync would be −68% but is left OFF). + Gated on `jvp_perf_opt` (trainer_base default false = byte-identical; true in both fluxtune yamls, must match + real↔sim; revertible per-config). Startup `[JVP_PERF_OPT]` log confirms 10/10 trainers active. NOT adopted: vmap + (2× but fp32-diverges via FD cancellation), fwd-AD (slower). 13 pytests + 185 fwdllm mode + 115 parity green. + *Retired/superseded anchors (kept only as pointers): K-D6 (→K-D12), K-D7/K-D8/K-D10/K-D16/K-D18/K-D19/K-D20/K-D23 — landed scaffolding or corrections, folded into §G/§H; see git history for detail.* + +--- + +## §L Forward-grad JVP compute profile & retained fluxtune optimizations +*(2026-07-05, tool: `scripts/profile_jvp_opt.py` — reuses the real `create_model` + `calculate_jvp`; distilbert-base ++ AdapterHub adapters, batch 8, seq 192, A40, fp16. Absolute ms are a CLEAN single-trainer profile; the real run +multiplies by ~10× from GPU contention across the 10 concurrent trainers, but pass-counts/ratios/memory transfer.)* + +**Mechanism.** Forward-grad trains via a **central finite-difference JVP** (`fwdgrad_utils.calculate_jvp`): each +perturbation = **2 forward passes** `f(θ±hv)`, h=0.01, autocast+no_grad → `jvp=(f(θ+hv)−f(θ−hv))/2h`. **fluxtune** +SELECTS the best of `perturbation_count`(=10) perturbations by |jvp| (2P=**20 passes**); **fwdllm/sync** selects by +cos-sim (**0 forward passes**) + 1 final JVP. Only **~1.5% of params are trainable** (bottleneck adapters in all 6 +layers + head, 1.04M/67.4M); the backbone is frozen. + +| path | fwd passes | ms/batch (clean) | +|---|---|---| +| sync fwdllm (current) | 5 | 50 | +| **sync fwdllm (opt)** | 2 | **16 (−68%)** | +| fluxtune P=1 (opt) | 2 | 16 (== sync) | +| fluxtune P=5 (opt) | 10 | 80 | +| fluxtune P=10 (current) | 25 | 251 | +| **fluxtune P=10 (opt)** | 20 | **159 (−37%)** | +| backprop ref (1 fwd+1 bwd) | — | 17 | + +- **Compute vs sync:** fluxtune = `2P × per-pass` → **10× sync at P=10**, linear in P, **equals sync at P=1**. The + JVP-selection is the entire fluxtune surcharge; sync's cos-sim selection is free. +- **Memory:** forward-grad peak is **FLAT in P** (~3.2–3.4 GB = model + one held forward; **no autograd graph**). + Backprop stores activations (3.71 GB). fluxtune's extra JVP inferences cost **TIME, not memory** — same footprint + as sync (this is forward-grad's design tradeoff: many cheap forward passes, no backward, low memory). +- **Per-pass:** full-param FD 10.0 ms; trainable-only FD 7.95 ms. + +**RETAINED — bit-identical (fidelity-preserving; real↔sim parity untouched):** +1. **Trainable-only FD** — skip `p−h·0=p` on the 98.5% frozen params inside `calculate_jvp`: **1.26× + −251 MB**, + `max|Δjvp|=0`. +2. **Drop the 3 diagnostic-only forward passes** (`_train_one_batch:646-648`, loss before/after-update logging — + never feed grads/telemetry) **+ reuse the winner's cached JVP** (`:645`, fluxtune): fluxtune 25→20, sync 5→2. + +Combined: **sync −68%, fluxtune −37%**, all bit-identical → should clear the `delay_factor=1` overrun +(4.2s → ~2.6s < min cohort D 4.0s) WITHOUT touching fidelity. **LANDED (K-D32), fluxtune-only & config-gated** +(`jvp_perf_opt`, default false = byte-identical; true in both fluxtune yamls; sync untouched). Startup log +`[JVP_PERF_OPT]`; 13 new pytests + 185 fwdllm mode + 115 parity green. + +**NOT retained — change fidelity:** +- **vmap-batching the perturbations** — **2.0×** (biggest single win) and mathematically exact (fp64 seq==vmap + BIT-IDENTICAL, deterministic run-to-run → *would* be real↔sim safe) BUT differs ~5% from the current sequential in + fp16/fp32: the FD subtracts two O(1) losses (catastrophic cancellation floors precision), so any reduction-order + change re-baselines the trajectory. Excluded per the fidelity bar; available if a re-baseline is ever accepted. +- **Forward-mode AD** (exact JVP) — slower (0.5×, needs eager attention; not impl for SDPA) + different math. +- **`perturbation_count`↓** — the direct lever, but changes the baseline algorithm. diff --git a/lib/python/examples/fwdllm/trainer/fl_main.py b/lib/python/examples/fwdllm/trainer/fl_main.py index 2aab33cd0..f12cfc84c 100644 --- a/lib/python/examples/fwdllm/trainer/fl_main.py +++ b/lib/python/examples/fwdllm/trainer/fl_main.py @@ -157,6 +157,10 @@ def post_complete_message(tc_args): # P2-5: forward-pass count knob (default 10 = historical behavior). "perturbation_count": getattr( config.hyperparameters, "perturbation_count", 10), + # §L: fluxtune JVP perf-opt (bit-identical). Default False = + # byte-identical; enabled only in the fluxtune yamls. + "jvp_perf_opt": getattr( + config.hyperparameters, "jvp_perf_opt", False), } ) model_args.config["num_labels"] = num_labels diff --git a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py index a405ca607..04a4f504c 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/FedSgdTrainer.py @@ -199,6 +199,11 @@ def __init__( self._sim_send_ts = None self._sim_completion_ts = None self._sim_round_duration_s = None + # Pure modeled delay D for this round (K-D31/P2-7a): stamped in the grad + # message (MODELED_DELAY_S) so the aggregator can canonically order the + # cohort's commits by (D, trainer_id) identically in real and sim. None + # until the first round completes / when delays are disabled. + self._modeled_delay_s = None self._wall_recv_ts = None self.trainer_start_ts = time.time() @@ -607,6 +612,11 @@ def train_with_data_id(self): # sleeps max(0, delay - gpu); sim skips it. Returns the modeled budget, # the remainder actually waited, and whether the GPU overran the budget. _delay_s, _remaining_s, _overran = self._emulate_training_delay(_real_gpu_time_s) + # Stash the pure modeled delay D (deterministic from the registry) so + # _send_grads can stamp it (MODELED_DELAY_S) for the aggregator's + # canonical (D, trainer_id) commit ordering. 0.0 when delays are off -> + # stays None-equivalent (all-zero => no ordering signal, arrival order). + self._modeled_delay_s = _delay_s if _delay_s else None # post_train phase starts AFTER the modeled delay (Root B / #6): real # SLEEPS _delay_s above (the modeled-latency term, compared via diff --git a/lib/python/examples/fwdllm/trainer/forward_training/fwdgrad_utils.py b/lib/python/examples/fwdllm/trainer/forward_training/fwdgrad_utils.py index b1511388f..322c3d4c9 100644 --- a/lib/python/examples/fwdllm/trainer/forward_training/fwdgrad_utils.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/fwdgrad_utils.py @@ -70,16 +70,30 @@ def functional_get_loss( return _get_loss(y, t, num_classes) -def calculate_jvp(func, params, v): +def calculate_jvp(func, params, v, trainable_idx=None): """ - Calculations Jacobian-vector product using numerical differentiation + Calculations Jacobian-vector product using numerical differentiation. + + trainable_idx (fluxtune perf-opt, simulate_fwdllm.md §L): when given, only + those param indices are perturbed; the rest keep v=0 so `p - h*0 = p` + EXACTLY -> BIT-IDENTICAL to perturbing every param, but skips copying the + ~98.5% frozen backbone twice per perturbation (1.26x, -251MB measured). + None => legacy all-param path (byte-identical to before). """ h = 0.01 with torch.no_grad(), autocast(): - loss = func(tuple([params[i] - h * v[i] for i in range(len(params))])) - terbulence_loss = func( - tuple([params[i] + h * v[i] for i in range(len(params))]) - ) + if trainable_idx is None: + minus = tuple([params[i] - h * v[i] for i in range(len(params))]) + plus = tuple([params[i] + h * v[i] for i in range(len(params))]) + else: + minus = list(params) + plus = list(params) + for i in trainable_idx: + minus[i] = params[i] - h * v[i] + plus[i] = params[i] + h * v[i] + minus, plus = tuple(minus), tuple(plus) + loss = func(minus) + terbulence_loss = func(plus) avg_loss = (terbulence_loss + loss) / 2 jvp = (terbulence_loss - loss) / (2 * h) return avg_loss, jvp diff --git a/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py b/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py index 825268332..8b5b15c93 100755 --- a/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py +++ b/lib/python/examples/fwdllm/trainer/forward_training/tc_transformer_trainer_distribute.py @@ -217,6 +217,24 @@ def __init__( except (TypeError, ValueError): self.perturbation_count = 10 + # Fluxtune JVP perf optimizations (simulate_fwdllm.md §L) — all + # BIT-IDENTICAL to the current grads (validated by scripts/profile_jvp_opt.py): + # (a) trainable-only finite difference (skip frozen p-h*0=p); + # (b) skip the 3 diagnostic-only forward passes (loss before/after-update + # logging — never feed grads/telemetry); + # (c) reuse the selected perturbation's JVP already computed in selection. + # Config-gated, default OFF (byte-identical); enabled ONLY in the fluxtune + # yamls (`jvp_perf_opt: true`). Revertible per-config -> fluxtune can run + # the un-optimized path any time. Real and sim MUST match (they read the + # same config), so it never breaks real<->sim parity. + self.jvp_perf_opt = bool(getattr(self.args, "jvp_perf_opt", False)) + self._sel_jvp_cache = {} + logging.info( + f"[JVP_PERF_OPT] jvp_perf_opt={self.jvp_perf_opt} " + f"(trainable-only FD + skip diagnostic passes + reuse winner JVP; " + f"all bit-identical — simulate_fwdllm.md §L)" + ) + # var control TODO: It is not layer id it is param id. Distilbert for eg # has only 6 layers. if self.args.model_type == "distilbert": @@ -452,11 +470,16 @@ def _select_optimal_perturbations(device, logging_state ): jvp_all_perturbations = [] + # perf-opt: cache each perturbation's (loss, jvp) so the winner's + # JVP is reused below instead of recomputed (2 fewer passes, §L). + self._sel_jvp_cache = {} for i in range(0, self.perturbation_count): v_params = _prepare_perturbation_tensors(device, v_buffer, i) loss, jvp = _compute_forward_jvp(device, x, labels, v_params) # logging.info(f"Jvp of option: {jvp}") jvp_all_perturbations.append(jvp) + if self.jvp_perf_opt: + self._sel_jvp_cache[i] = (loss, jvp) logging.info(f"Number of pert and jvps: {len(jvp_all_perturbations)}") @@ -569,7 +592,11 @@ def _compute_forward_jvp(device, x, labels, v_params): t=labels, ) - loss, jvp = calculate_jvp(f, self.params, v_params) + # Perf-opt (fluxtune): perturb only trainable params — bit-identical + # since v=0 on frozen params (p-h*0=p). None => legacy all-param path. + _tidx = ([i for i, p in enumerate(self.params) if p.requires_grad] + if self.jvp_perf_opt else None) + loss, jvp = calculate_jvp(f, self.params, v_params, trainable_idx=_tidx) jvp = jvp.to(device) return loss, jvp @@ -642,11 +669,23 @@ def _accumulate_and_extract_grads(device, jvp, v_params): logging.debug(f"v_params hashes: {[(_calculate_hash(v), v.shape) for v in v_params if v.requires_grad]}") logging.info(f"params hashes: {[(_calculate_hash(p), p.shape) for p in self.params]}") - loss, jvp = _compute_forward_jvp(device, x, labels, v_params) - nonscaled_global_loss = _compute_loss_after_update(device, x, labels, v_params, jvp) - scaled_global_loss = _compute_loss_after_update(device, x, labels, v_params, jvp/15) - loss_before_update = _compute_loss_before_update(device, x, labels, v_params, jvp) - logging.info(f"At trainer: {self.trainer_id} - iteration: {logging_state.get('iteration')} - jvp_magnitude: {jvp} - loss before update: { loss_before_update } - loss after update (not downscaled): {nonscaled_global_loss} - loss after update (down scaled): {scaled_global_loss}") + # perf-opt: reuse the winner's JVP already computed during selection + # (same params + same v_params for best_idx -> bit-identical), saving 2 + # forward passes. Falls back to compute when the cache is absent (the + # cos-sim / sync path never ran the JVP-selection loop) or best_idx used + # the carried global-best v_params (the `best_idx == -1` branch). + if (self.jvp_perf_opt and best_idx != -1 + and best_idx in getattr(self, "_sel_jvp_cache", {})): + loss, jvp = self._sel_jvp_cache[best_idx] + else: + loss, jvp = _compute_forward_jvp(device, x, labels, v_params) + # 3 diagnostic-only passes: their losses ONLY feed the log below (never + # grads/telemetry), so skip them under perf-opt (bit-identical grads, §L). + if not self.jvp_perf_opt: + nonscaled_global_loss = _compute_loss_after_update(device, x, labels, v_params, jvp) + scaled_global_loss = _compute_loss_after_update(device, x, labels, v_params, jvp/15) + loss_before_update = _compute_loss_before_update(device, x, labels, v_params, jvp) + logging.info(f"At trainer: {self.trainer_id} - iteration: {logging_state.get('iteration')} - jvp_magnitude: {jvp} - loss before update: { loss_before_update } - loss after update (not downscaled): {nonscaled_global_loss} - loss after update (down scaled): {scaled_global_loss}") self.jvp_for_snr_check = abs(jvp) logging.info(f"JVP of the perturbation: {jvp}") diff --git a/lib/python/examples/fwdllm/trainer/main.py b/lib/python/examples/fwdllm/trainer/main.py index d85a51e76..f5e82b2af 100644 --- a/lib/python/examples/fwdllm/trainer/main.py +++ b/lib/python/examples/fwdllm/trainer/main.py @@ -121,6 +121,10 @@ def post_complete_message(tc_args): # P2-5: forward-pass count knob (default 10 = historical behavior). "perturbation_count": getattr( config.hyperparameters, "perturbation_count", 10), + # §L: fluxtune JVP perf-opt (bit-identical). Default False = + # byte-identical; enabled only in the fluxtune yamls. + "jvp_perf_opt": getattr( + config.hyperparameters, "jvp_perf_opt", False), } ) model_args.config["num_labels"] = num_labels diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py index 2396bda63..257b118a1 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py @@ -275,6 +275,11 @@ def internal_init(self) -> None: self._updates_in_queue = 0 self._updates_received = {} self._per_agg_trainer_list = [] + # end -> canonical commit-order key (modeled_delay D, str(end)) for the + # current cycle's cohort (K-D31/P2-7a). Populated per contribution in + # aggregate_weights; consumed by _canonicalize_cohort_commit_order to + # break equal-D ties by trainer_id IDENTICALLY in real and sim. + self._commit_key_by_end = {} self._model_version_unique_trainers = set() self._model_version_trainer_stats = { "train_duration": [], @@ -1279,6 +1284,18 @@ def _process_single_trainer_message(self, channel, msg, end, timestamp): self._updates_in_queue += 1 self._per_agg_trainer_list.append(end) + # Canonical commit-order key (K-D31/P2-7a): the trainer's pure modeled + # delay D (deterministic from the registry) + str(end) as the tie-break. + # Lets _canonicalize_cohort_commit_order reproduce real's D-ordered + # arrival AND break equal-D ties by trainer_id identically in both modes. + # None when delays are off / the trainer did not stamp D -> that cycle + # falls back to arrival order (byte-identical legacy behavior). + _md = msg.get(MessageType.MODELED_DELAY_S) + self._commit_key_by_end = getattr(self, "_commit_key_by_end", {}) + self._commit_key_by_end[end] = ( + (float(_md), str(end)) if _md is not None else None + ) + # Re-pick guard (async_oort triplet filter): record the (model_version, # data_id, iteration) this trainer just CONTRIBUTED at, so it is not # re-selected until the agg version advances (the commit boundary prunes @@ -1509,12 +1526,71 @@ def _build_dynamic_kc_metrics(self, channel) -> dict: "target_iter_per_data_id": target_iter, } + def _canonicalize_cohort_commit_order(self): + """Reorder THIS cycle's cohort commits to a canonical (D, trainer_id) + order — IDENTICALLY in real and sim (K-D31/P2-7a). + + Real receives updates in strict modeled-delay (D) order (device wall = + D, K-D29); sim commits in sct order (= D order). The sole residual + real↔sim divergence is the tie-break when two trainers share a D (a + realistic registry collision, e.g. trainers 3 & 9 both 13.0s → D=6.5): + real breaks it by physical arrival, sim by sct-sort. Both tied members + land in the SAME split-half so `var` is unchanged, but the EXACT-order + `cohort_sequence` rung flags the swap. Here we canonicalize: sort the + cohort by (D, str(end)) so equal-D ties break by trainer_id in BOTH + modes → identical receive order; `var`/grads stay bit-identical (the + aggregated grad is an order-independent sum; the split-half only ever + reshuffles WITHIN a half on a tie). + + Scope = this cycle only. `grad_for_var_check_list` ACCUMULATES across a + data_id's iterations (reset on commit) while `_per_agg_trainer_list` is + per-cycle, so we reorder just the TRAILING len(cohort) slice of the + grad/jvp lists — the sync barrier appends this cohort contiguously at + the end, in `_per_agg_trainer_list` order. + + No-op unless every contributor stamped a modeled delay (delays on) AND + a tie actually changes the order → delays-off / legacy runs and the + common non-tie case keep arrival order (byte-identical). + """ + ends = self._per_agg_trainer_list + n = len(ends) + if n < 2: + return + keys = [self._commit_key_by_end.get(e) for e in ends] + if any(k is None for k in keys): + return # delays off / a contributor without a stamp → arrival order + perm = sorted(range(n), key=lambda i: keys[i]) + if perm == list(range(n)): + return # already canonical (the common, non-tie path) + self._per_agg_trainer_list = [ends[i] for i in perm] + # Reorder the trailing cohort slice of the accumulating var/jvp lists in + # lockstep. Guard on length: only when the slice aligns 1:1 with this + # cohort (it does under the sync barrier; a mismatch means a non-grad + # message slipped in → leave the lists untouched rather than corrupt). + for lst in (self.grad_for_var_check_list, self.jvp_for_snr_check_list): + if len(lst) >= n: + tail = lst[-n:] + lst[-n:] = [tail[i] for i in perm] + logger.info( + f"[COMMIT_CANON] equal-D tie → reordered {n}-cohort to (D,id) order " + f"(perm={perm}); var/grads unchanged, receive order now real↔sim identical." + ) + @timer_decorator def _process_aggregation_goal_met(self, tag, channel, is_async=False): logger.info( f"Aggregation goal {self._agg_goal} reached. Performing FwdLLM aggregation." ) + # Canonicalize this cohort's commit order to (D, trainer_id) BEFORE the + # telemetry snapshot and aggregate() so the recorded receive order and + # the split-half var are computed on the SAME deterministic order in + # real and sim (K-D31/P2-7a). Gated on the presence of commit-key state + # (populated per contribution in aggregate_weights) → skipped entirely + # when delays are off / no keys were stamped (arrival order, unchanged). + if getattr(self, "_commit_key_by_end", None): + self._canonicalize_cohort_commit_order() + # Snapshot for this cycle's agg_round telemetry (emitted further down, # after self._per_agg_trainer_list is cleared and self._model_version # may have advanced -- see build_agg_round call below). @@ -1841,6 +1917,7 @@ def _process_aggregation_goal_met(self, tag, channel, is_async=False): self._updates_in_queue -= self._agg_goal self._per_agg_trainer_list = [] + self._commit_key_by_end = {} # cohort-scoped (K-D31/P2-7a) logger.info( f"====== aggregation finished for round {self._round}, " diff --git a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py index d7e195846..d0ecbc8fb 100644 --- a/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py +++ b/lib/python/flame/mode/horizontal/syncfl/fwdllm_trainer.py @@ -580,6 +580,12 @@ def _send_grads(self, tag: str) -> None: # the intrinsic (server-overhead-free) task duration from. All # None in real mode -> aggregator ignores them (arrival order). MessageType.SIM_COMPLETION_TS: self._sim_completion_ts, + # Pure modeled delay D (K-D31/P2-7a): deterministic from the + # registry (unlike SIM_COMPLETION_TS, which folds in GPU jitter), + # so the aggregator orders this cohort's commits by (D, + # trainer_id) identically in real and sim. Stamped in BOTH modes; + # None when delays are off -> aggregator keeps arrival order. + MessageType.MODELED_DELAY_S: getattr(self, "_modeled_delay_s", None), # Echo the dispatch stamp so the aggregator can reconstruct this # contribution's [dispatch, completion] interval for R1 (§L.3). MessageType.SIM_SEND_TS: self._sim_send_ts, diff --git a/lib/python/flame/mode/message.py b/lib/python/flame/mode/message.py index 89d1ac8bf..eab578faf 100644 --- a/lib/python/flame/mode/message.py +++ b/lib/python/flame/mode/message.py @@ -96,3 +96,12 @@ class MessageType(Enum): # skew, same class of bug as B2.0.3). Sim mode never sends this -- its # shared origin is the single _vclock, already exposed via SIM_SEND_TS. AGG_START_TS = 42 + + # Pure modeled per-trainer mobile delay D = training_delay_s / factor / + # speedup (K-D29), stamped by the trainer in BOTH modes. Unlike + # SIM_COMPLETION_TS / TRAINING_BUDGET_S (which fold in the measured GPU time + # and so differ run-to-run), D is deterministic from the registry, so the + # aggregator can order a cohort's commits by (D, trainer_id) IDENTICALLY in + # real and sim — breaking equal-D ties by trainer_id (K-D31, P2-7a). None + # when delays are disabled (aggregator then falls back to arrival order). + MODELED_DELAY_S = 43 diff --git a/lib/python/tests/mode/test_fwdllm_commit_canon.py b/lib/python/tests/mode/test_fwdllm_commit_canon.py new file mode 100644 index 000000000..e188ecb16 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_commit_canon.py @@ -0,0 +1,140 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""K-D31 / P2-7a: canonical (D, trainer_id) commit ordering. + +On the databin1 logical-parity run the sync baselines reached bit-identical +`var` + cadence + cohort SET, with the SOLE residual a receive-ORDER swap of +two trainers that share a registry delay (D=6.5): real breaks the sct tie by +physical arrival, sim by sct-sort. Both land in the same split-half so `var` +is unchanged, but the EXACT-order `cohort_sequence` rung flags it. + +`_canonicalize_cohort_commit_order` reorders THIS cycle's cohort by +(D, str(end)) so equal-D ties break by trainer_id IDENTICALLY in real and sim. +These tests pin: (1) two different input orders (real-physical vs sim-sct) +canonicalize to the SAME sequence; (2) the grad/jvp trailing slice reorders in +lockstep; (3) accumulated earlier-iteration entries are untouched; (4) no-op +when keys are missing (delays off) or the cohort is already canonical. +""" + +from flame.mode.horizontal.syncfl.fwdllm_aggregator import TopAggregator + + +class _CanonAgg: + """Minimal stand-in exposing only what _canonicalize_cohort_commit_order + touches. `end` values are the last-3-hex trainer tokens for readability.""" + + def __init__(self, ends, delays, grad_list=None, jvp_list=None): + self._per_agg_trainer_list = list(ends) + # end -> (D, str(end)) key, mirroring aggregate_weights' capture; a None + # delay models a contributor that did not stamp D (delays off). + self._commit_key_by_end = { + e: ((float(d), str(e)) if d is not None else None) + for e, d in zip(ends, delays) + } + self.grad_for_var_check_list = list( + grad_list if grad_list is not None else ends + ) + self.jvp_for_snr_check_list = list( + jvp_list if jvp_list is not None else ends + ) + + canon = TopAggregator._canonicalize_cohort_commit_order + + +# D-values keyed by the last-3-hex token, matching the databin1 registry +# (trainers 372 & 378 both drew training_delay_s=13.0 -> D=6.5, the tie). +_D = { + "370": 2.0, "375": 2.5, "373": 3.5, "376": 5.0, "379": 5.5, + "378": 6.5, "372": 6.5, "371": 8.0, "377": 8.5, "374": 9.0, +} +_CANON_ORDER = ["370", "375", "373", "376", "379", "372", "378", "371", "377", "374"] + + +def _mk(ends): + return _CanonAgg(ends, [_D[e] for e in ends]) + + +class TestCanonicalizesToOneOrder: + def test_real_physical_and_sim_sct_orders_converge(self): + # real broke the 6.5 tie 378-before-372; sim broke it 372-before-378. + real_order = ["370", "375", "373", "376", "379", "378", "372", "371", "377", "374"] + sim_order = ["370", "375", "373", "376", "379", "372", "378", "371", "377", "374"] + ra, sa = _mk(real_order), _mk(sim_order) + ra.canon() + sa.canon() + assert ra._per_agg_trainer_list == _CANON_ORDER + assert sa._per_agg_trainer_list == _CANON_ORDER + assert ra._per_agg_trainer_list == sa._per_agg_trainer_list + + def test_tie_breaks_by_trainer_id_not_arrival(self): + # only the two tied (D=6.5) members may move, and by id order (372<378) + a = _mk(["379", "378", "372", "371"]) + a.canon() + assert a._per_agg_trainer_list == ["379", "372", "378", "371"] + + def test_grad_and_jvp_reorder_in_lockstep(self): + a = _mk(["379", "378", "372", "371"]) + a.canon() + assert a.grad_for_var_check_list == ["379", "372", "378", "371"] + assert a.jvp_for_snr_check_list == ["379", "372", "378", "371"] + + +class TestSliceScope: + def test_only_trailing_cohort_slice_reorders(self): + # grad list ACCUMULATES: an earlier iteration's 2 entries precede this + # cycle's 4-cohort. Only the trailing 4 may move. + prev = ["p0", "p1"] + cohort = ["379", "378", "372", "371"] + a = _CanonAgg(cohort, [_D[e] for e in cohort], grad_list=prev + cohort, + jvp_list=prev + cohort) + a.canon() + assert a.grad_for_var_check_list == prev + ["379", "372", "378", "371"] + assert a._per_agg_trainer_list == ["379", "372", "378", "371"] + + def test_misaligned_grad_list_left_untouched(self): + # a shorter-than-cohort grad list (a non-grad message slipped in) must + # not be sliced/corrupted -- reorder the contributor list only. + cohort = ["379", "378", "372", "371"] + a = _CanonAgg(cohort, [_D[e] for e in cohort], grad_list=["x"], + jvp_list=["x"]) + a.canon() + assert a._per_agg_trainer_list == ["379", "372", "378", "371"] + assert a.grad_for_var_check_list == ["x"] # untouched + + +class TestNoOps: + def test_already_canonical_is_unchanged(self): + a = _mk(_CANON_ORDER) + before = list(a._per_agg_trainer_list) + a.canon() + assert a._per_agg_trainer_list == before + + def test_missing_delay_key_falls_back_to_arrival_order(self): + # a contributor without a stamped D (delays off) -> arrival order kept + ends = ["379", "378", "372", "371"] + a = _CanonAgg(ends, [5.5, None, 6.5, 8.0]) + a.canon() + assert a._per_agg_trainer_list == ends # no reorder + + def test_single_contributor_is_noop(self): + a = _mk(["370"]) + a.canon() + assert a._per_agg_trainer_list == ["370"] + + +class TestVarInvariance: + def test_tie_swap_stays_within_split_half(self): + # the databin1 tie (372/378) sits at indices 5-6 of the 10-cohort; the + # split-half boundary is n//2 = 5, so both are in the SECOND half in + # BOTH orders -> the reorder cannot move a grad across the boundary -> + # calculate_var (mean of first-half vs second-half) is invariant. + n = len(_CANON_ORDER) + real_order = ["370", "375", "373", "376", "379", "378", "372", "371", "377", "374"] + first_ids = lambda seq: set(seq[: n // 2]) + second_ids = lambda seq: set(seq[n // 2:]) + # half-MEMBERSHIP is identical before and after canonicalization + a = _mk(real_order) + pre_first, pre_second = first_ids(real_order), second_ids(real_order) + a.canon() + assert first_ids(a._per_agg_trainer_list) == pre_first + assert second_ids(a._per_agg_trainer_list) == pre_second diff --git a/lib/python/tests/mode/test_fwdllm_jvp_perf_opt.py b/lib/python/tests/mode/test_fwdllm_jvp_perf_opt.py new file mode 100644 index 000000000..f3cf47504 --- /dev/null +++ b/lib/python/tests/mode/test_fwdllm_jvp_perf_opt.py @@ -0,0 +1,66 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Fluxtune JVP perf-opt (simulate_fwdllm.md §L) — the retained optimizations +must be BIT-IDENTICAL to the un-optimized grads, so enabling `jvp_perf_opt` on +fluxtune never changes training fidelity or real↔sim parity. + +The core claim is `calculate_jvp(..., trainable_idx=T)` == `calculate_jvp(...)` +when the perturbation `v` is zero on the non-T (frozen) params: `p - h·0 = p` +exactly, so both paths feed the model identical param values. These tests pin +that (the trainer-level gating just chooses which path to call).""" + +import pytest + +torch = pytest.importorskip("torch") + +from examples.fwdllm.trainer.forward_training.fwdgrad_utils import calculate_jvp + + +def _linear_func(params): + # a deterministic scalar of the params (order-independent) so the JVP is + # well-defined and precision-stable for an exact-equality check + return sum(p.sum() for p in params) + + +def _params_and_v(n=6, frozen=(1, 3, 4), shape=(4, 4), seed=0): + g = torch.Generator().manual_seed(seed) + params = [torch.randn(shape, generator=g) for _ in range(n)] + v = [torch.randn(shape, generator=g) for _ in range(n)] + for i in frozen: # frozen params get a ZERO perturbation + v[i] = torch.zeros(shape) + trainable_idx = [i for i in range(n) if i not in frozen] + return params, v, trainable_idx + + +class TestTrainableOnlyFD: + def test_bit_identical_to_all_param_fd(self): + params, v, tidx = _params_and_v() + l0, j0 = calculate_jvp(_linear_func, params, v) # legacy + l1, j1 = calculate_jvp(_linear_func, params, v, trainable_idx=tidx) # perf-opt + assert torch.equal(j0, j1), (j0, j1) + assert torch.equal(l0, l1) + + def test_none_is_the_legacy_path(self): + params, v, _ = _params_and_v() + a = calculate_jvp(_linear_func, params, v) + b = calculate_jvp(_linear_func, params, v, trainable_idx=None) + assert torch.equal(a[1], b[1]) and torch.equal(a[0], b[0]) + + def test_frozen_params_untouched_by_perturbation(self): + # a func that reads a FROZEN param heavily — its contribution must be + # identical in both signs (v=0 there), so it cancels out of the JVP + params, v, tidx = _params_and_v(frozen=(0,)) + def f(p): + return (p[0] ** 3).sum() + p[2].sum() # p[0] frozen, nonlinear + _, j_all = calculate_jvp(f, params, v) + _, j_opt = calculate_jvp(f, params, v, trainable_idx=tidx) + assert torch.equal(j_all, j_opt) + + def test_covers_extra_zero_indices_safely(self): + # trainable_idx may include an index whose v happens to be 0 (a trainable + # param that drew a zero dir) — still p-h·0=p, so no divergence + params, v, tidx = _params_and_v(frozen=(2,)) + v[0] = torch.zeros_like(v[0]) # a "trainable" idx with zero dir + l0, j0 = calculate_jvp(_linear_func, params, v) + l1, j1 = calculate_jvp(_linear_func, params, v, trainable_idx=tidx) + assert torch.equal(j0, j1) and torch.equal(l0, l1) From fa5c0ade5399999f663b2f9474f98e9ae05cb5f4 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Mon, 6 Jul 2026 02:27:26 -0400 Subject: [PATCH 31/56] fwdllm #15: root-cause the fluxtune sim_rate=0.50 slowdown + K-D33 pinning Root cause (evidence-backed, FLUXTUNE_SIMRATE_15_HANDOFF.md): a circular wait between the drain's earlier_stuck gate and hold-to-commit residence. The gate blocks real-wall waiting for the lowest-expected in-flight trainer to preserve sct order, but that trainer is blocked in recv_wrapper (hold-to-commit, not computing) so its grad never comes -> 30s failsafe. 48% of commits stall >2s = ~1974s (81% of wall); recv_wrapper mean 30.9s sim vs 0.01s real. Proper fix: re-dispatch on return (decouple physical GPU pipelining from the virtual in-flight ledger); NOT yet implemented -- plan + repro scripts in the handoff. K-D33 pinning (landed, 120 launch tests green): - trainer [PIN] self-report of actual CUDA_VISIBLE_DEVICES/device/cpu_affinity (examples/fwdllm/trainer/main.py; fl_main.py is dead) - [LOAD_BALANCE] post-proc check in spawner.spawn_all (warns on GPU imbalance / under-provisioning) - aggregator GPU pin (aggregator_spawner gpu_id + runner picks idle/least-loaded) Docs updated to current status: sync sim_rate 2.9-3.0 (#12c resolved), K-D31/P2-7a validated, bin-7 float-nondeterminism wall, fluxtune #15 = pipelining. Co-Authored-By: Claude Opus 4.8 --- .../fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md | 178 ++++++++++++++++++ .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 25 ++- .../examples/fwdllm/fluxtune_contributions.md | 7 +- lib/python/examples/fwdllm/simulate_fwdllm.md | 168 ++++++++++------- lib/python/examples/fwdllm/trainer/main.py | 23 +++ lib/python/flame/launch/aggregator_spawner.py | 9 + lib/python/flame/launch/runner.py | 16 ++ lib/python/flame/launch/spawner.py | 52 +++++ 8 files changed, 398 insertions(+), 80 deletions(-) create mode 100644 lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md diff --git a/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md b/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md new file mode 100644 index 000000000..d095d0b5d --- /dev/null +++ b/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md @@ -0,0 +1,178 @@ +# #15 fluxtune `sim_rate` — ROOT CAUSE FOUND, resume-here handoff + +**Status (2026-07-05):** Root cause of fluxtune `sim_rate = 0.50` is IDENTIFIED and evidence-backed. The fix +(re-dispatch-on-return) is designed and de-risked but NOT yet implemented. Pick up at "IMPLEMENTATION PLAN" below. +This is a TEMP tracker for the #15 work — fold into `simulate_fwdllm.md` §G/§K and delete when the fix lands. + +> Prereqs / cross-refs: `simulate_fwdllm.md` (§A open issue #15, §H corrections, principles #13/#15/#16), +> `PARITY_LOGICAL_TASKS.md` (checkpoint). The banked evidence run is +> `experiments/run_20260705_204619_fluxtune_n10_smoke_syn_0_sim` (+ its `_real` pair `run_20260705_202448`). + +--- + +## 1. THE ROOT CAUSE (definitive, evidence-backed) + +**A circular wait between the sim drain's `earlier_stuck` gate and the hold-to-commit slot residence.** + +The drain (`_sim_recv_min_grad`, `flame/mode/horizontal/syncfl/fwdllm_aggregator.py:752`) blocks in REAL WALL to +preserve sct-ordered commits: it will not commit a buffered grad while an *in-flight* trainer has a smaller +**expected** sct (`_sim_inflight_expected[end]`). But that in-flight trainer is itself **blocked in `recv_wrapper`** +(hold-to-commit: `_release_end_on_return:1136` returns early on the residence path, so the trainer's slot only frees +when its PREVIOUS grad commits) — so it is NOT computing and cannot produce the grad the drain is waiting for. Its +previous grad won't commit because the drain is stalled. Circular; broken only by the 30s `RECV_TIMEOUT_WAIT_S` +failsafe evict. + +This is the STEADY STATE, not an edge case. The startup is fine (all 10 dispatched fresh → compute in parallel, +concurrency=10 for ~5s) and it collapses right after the first commit, when trainers start getting held. + +### Causal chain (fully closed) +``` +hold-to-commit blocks trainers in recv (recv_wrapper mean 30.9s, max 60s) + → drain earlier_stuck gate waits real-wall for these NON-COMPUTING in-flight trainers + → circular stall, broken only by the 30s failsafe evict + → ~1974s (81% of the 2425s sim wall) burned in gate-waits + → GPU concurrency 1.54x (vs real 3.37x) + → sim_rate = vclock/wall = 1191/2425 = 0.50 +``` + +## 2. THE EVIDENCE (all from the banked sim/real pair; reproduce with the scripts in §5) + +- **Per-commit:** real reaches agg-goal in **4.30s wall**; sim models **3.27s vclock** but spends **6.60s wall** → + per-commit sim_rate 0.50. +- **GPU concurrency:** real avg **3.37x** (98% busy), sim avg **1.54x** (85% busy). Same total GPU work + (~3.7-4.0k trainer-s), same ~480s 8-way pipeline floor. `max=10` at startup (proves the HW sustains 10-wide). +- **Trainer recv-block (trainer 371):** sim `recv_wrapper` mean **30.9s** / max 55s per iteration; real **0.28s** + (median **0.01s**). Compute is mode-invariant (~3-4s, delay correctly skipped `_emulate_training_delay=0.000s`). +- **The smoking-gun stall:** `[SIM_GRAD_STUCK_EVICT] round=1 end=0379 exp=24.0 bmin=27.0` — drain blocked ~30s + waiting for trainer 379 (expected sct 24) while **6 grads sat ready** (bmin 27). Trainer 379's own telemetry: + blocked in `recv_wrapper` for **60.39s** (20:47:19→20:48:19), got weights the instant AFTER the evict. +- **Steady-state:** `buf_depth` is **constantly 6** (six grads always ready, held behind the deadlocked one); + **524/1093 commits (48%) stall >2s** = **1974s** total wall (~81%). +- **Out-of-order already happens:** committed scts go 11,13,7,5.4,5.5,10,13,16… (non-monotonic) — so the strict + ordering the gate blocks for is not even being preserved; it pays full real-wall cost for nothing. +- **Eval:** once per data-bin (23 evals / 24 data_ids), NOT per iteration. But it's an 8.5s BLOCKING call on the + aggregator (~195s / 8% of wall). Secondary; overlap later. + +## 3. THE PROPER FIX + +The gate is not wrong to want sct order — the bug is it **waits for trainers that aren't running**. Fix = ensure +the trainer the gate waits for is ALWAYS actually computing → **re-dispatch on RETURN** (match real, whose recv is +0.01s), decoupling physical GPU pipelining from the virtual in-flight ledger. + +- A returned trainer immediately gets the next weights and computes → never idle-blocked in recv. +- The in-flight trainer the gate waits for then produces its grad in ~4s IN PARALLEL with 7 others → the drain + commits a BATCH per GPU-pass instead of stalling on one held trainer. `buf_depth` stays deep, gate-waits become + short and legitimate, 30s failsafe never fires. Expected: concurrency 1.54x → ~3.37x+, sim_rate 0.50 → ~1.3-2+. +- **Keep the gate + sct-order commits** (they preserve #1d cohort parity) — only remove the recv-block that starves + the pipeline. The virtual in-flight COUNT / selection eligibility (R1) stays held-to-commit; that is a SEPARATE + ledger from the trainer's physical compute. Conflating them is what caused this (K-D17b held both together). + +### The hard constraint to preserve (principle #16 — no hacks) +The re-dispatched (compute-ahead) grad must be computed against the **model version real dispatched** — fluxtune's +fedbuff down-weights by staleness `V'-V`, so a grad against the wrong version silently changes the result. Since +REAL also re-dispatches on return and cycles continuously, the versions come from the same deterministic commit +order (K-D29 remainder-wait). **This should hold but MUST be verified from telemetry, not assumed** — see the P0 +gate below. + +### Belt-and-suspenders (independently correct, do alongside) +`_sim_inflight_expected[end]` is stamped at DISPATCH (`fwdllm_aggregator.py:2819`, `_sst + _budget` where +`_sst = self._vclock.now`, `_budget = self._sim_trainer_budget.get(end, self._sim_budget_min)`) assuming the +trainer starts computing immediately. A HELD trainer hasn't started, so the expected sct is a fiction (too early) → +the gate waits for a grad that won't come. Tie the expected clock to actual compute-START, and/or bound the gate's +real-wall wait far below the 30s failsafe. + +## 4. IMPLEMENTATION PLAN (resume here tomorrow) + +**P0 — GATE (verify before coding, read-only):** Pull per-trainer `MODEL_VERSION` (the version each grad was +computed against) vs the commit sequence, real vs sim. Confirm real absorbs the staleness (grad version = dispatch +version, fedbuff down-weights) and that re-dispatch-on-return in sim yields the SAME dispatch-version sequence as +real. Anchor: `MessageType.MODEL_VERSION` on grad msgs (`fwdllm_aggregator.py:_process_single_trainer_message`, +staleness log ~line 1170); `inc_model_version_per_data_id=True`. If versions DON'T align, the design needs the +version pinned to the trainer's virtual completion (sct) rather than physical dispatch — revisit before coding. + +**P1 — CORE CHANGE (once P0 passes):** Decouple physical re-dispatch from the virtual commit-hold. +- `_release_end_on_return` (`:1128`): on the async sim residence path, currently `return`s (holds slot+guard to + commit). Change so the trainer's PHYSICAL re-dispatch (weights send → compute next grad) happens on return, + while the SELECTOR's virtual in-flight set (`selected_ends`/`all_selected`, driving R1 + `extra = c - inflight`) + stays held to commit via `_sim_hold_busy_slots`. Two ledgers: physical-dispatch (release on return) vs + virtual-inflight (release on commit). +- The re-dispatched grad enters the sct reorder buffer with its own `SIM_COMPLETION_TS`; the drain keeps committing + in sct order for the vclock. Verify the buffer/`_sim_inflight_expected`/`_sim_pending_commit` bookkeeping stays + consistent (K-D27 two-ledger discipline — do NOT reintroduce the R1 regression). +- Fix `_sim_inflight_expected` stamp to reflect actual compute-start (belt-and-suspenders above). + +**P2 — TESTS + TELEMETRY (same change, principle #11):** pytest `tests/mode -k fwdllm` (residence/R1/drain) + +`tests/mode -k parity`; assert async_cifar10 byte-identical (shared `_sim_recv_min` untouched — this is a +fwdllm_aggregator-only edit, principle #8/#9). Add a telemetry assert that `recv_wrapper` mean drops to ~0 and +concurrency rises. R1 must stay ~0 (bank it — K-D19 lesson: an R1 fix isn't done until the smoke shows R1<=2%). + +**P3 — VALIDATE:** re-run `run_sequential.sh --only fluxtune --mode both --delays on --delay-factor 1 +--max-data-id 2` (short). Confirm: `recv_wrapper`→~0, concurrency→~3.37x+, `sim_rate`→>1, `[SIM_GRAD_STUCK_EVICT]` +gone, and — critically — `cohort_sequence`/`var` parity UNCHANGED or improved (the fix must not alter which grads +commit in which cohort; if it does, the version-parity assumption (P0) was wrong). + +**P4 — DOCS:** fold the root cause into `simulate_fwdllm.md` §A (#15) + §H (the "gate waits on held trainers" root) ++ new §K deviation (K-D34, the two-ledger physical/virtual dispatch split); update `PARITY_LOGICAL_TASKS.md` +checkpoint; delete this handoff file. + +## 5. REPRO SCRIPTS (paste-ready, read-only, run from `lib/python/examples/fwdllm`) + +```bash +FS=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_sim | head -1) # banked evidence run +AGG=$(ls "$FS"/*aggregator.log | head -1) + +# (a) the stall: 48% of commits stall >2s = ~1974s wall +grep "SIM_GRAD_RECV" "$AGG" | python3 -c " +import sys,re; from datetime import datetime +prev=None;big=0;tot=0;wait=0 +for l in sys.stdin: + ts=datetime.strptime(l.split(' | ')[0],'%Y-%m-%d %H:%M:%S,%f').timestamp() + if prev is not None: + d=ts-prev; tot+=1 + if d>2: big+=1; wait+=d + prev=ts +print(f'commits={tot+1} gaps>2s={big} ({100*big/tot:.0f}%) wall_in_waits={wait:.0f}s')" + +# (b) the evict + buf_depth stuck at 6 +grep -E "SIM_GRAD_STUCK_EVICT|SIM_GRAD_RECV" "$AGG" | grep -oE "buf_depth=[0-9]+" | sort | uniq -c +grep "SIM_GRAD_STUCK_EVICT" "$AGG" + +# (c) trainer recv-block sim vs real (trainer 371) +for m in sim real; do D=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_$m|head -1) + tf=$(ls "$D"/telemetry/trainer_*371.jsonl|head -1) + python3 - "$tf" "$m" <<'PY' +import json,sys,statistics as st +f,m=sys.argv[1],sys.argv[2]; recv=[];tr=[] +for l in open(f): + e=json.loads(l) + if e.get('event')=='step_timing': + if e.get('func')=='recv_wrapper' and e.get('duration_s') is not None: recv.append(e['duration_s']) + if e.get('func')=='_train_one_batch' and e.get('duration_s') is not None: tr.append(e['duration_s']) +print(f"{m}: recv_wrapper mean={st.mean(recv):.1f}s median={st.median(recv):.2f}s | train mean={st.mean(tr):.2f}s") +PY +done +``` + +Aggregator timeline decoder (per-step wall) + GPU-concurrency scripts were run ad hoc during the investigation; +the key numbers are all captured in §2 above. The `[SIM_GRAD_RECV]` log line +(`fwdllm_aggregator.py:926`, format `end= sct= T_v= buf_depth= inflight_exp= sel_ends=`) is the primary drain +diagnostic — grep it to watch the gate. + +## 6. LANDED THIS SESSION (already committed separately / in this commit) +- **K-D33 pinning (in this branch):** trainer `[PIN]` self-report (`examples/fwdllm/trainer/main.py` — NOT the dead + `fl_main.py`); `[LOAD_BALANCE]` post-proc check (`flame/launch/spawner.py::_assert_load_balanced`); aggregator GPU + pin (`flame/launch/aggregator_spawner.py` `gpu_id` param + `flame/launch/runner.py` picks idle/least-loaded GPU). + 120 launch tests green. Corrected earlier "under-provisioned GPUs" misread — pinning IS clean (8 GPUs, balanced + round-robin, CPU sched_setaffinity 1 core/trainer). `client_idx%8` device arg is vestigial (`FedSgdTrainer:388` + overwrites `self.device=torch.device("cuda")`). +- **Docs updated:** `simulate_fwdllm.md` §A/§G/§H/§K, `PARITY_LOGICAL_TASKS.md` checkpoint, + `fluxtune_contributions.md` §5 — current status (sync sim_rate 2.9-3.0, K-D31/P2-7a validated, bin-7 float- + nondeterminism wall, fluxtune #15 = pipelining). + +## 7. RELATED OPEN ITEMS (context, not part of #15) +- **bin-7 float-nondeterminism wall (sync):** exact cadence parity unattainable past ~bin 6 (GPU fp16 grad jitter + amplified by split-half variance); relax `cohort_sequence` EXACT to `--max-bin 1` + distributional rung beyond. + Confirm with a 2-real-run P0-2 diff. (simulate_fwdllm.md open issue.) +- **#1d fluxtune cohort SET (thin-margin overrun):** `jvp_perf_opt` put GPU MEAN (3.61s) under the 4.0s budget but + the TAIL (4-5.4s) overruns on the 2 doubled GPUs. The #15 fix (keep compute near the ~2.4s uncontended floor via + pipelining) should also lift this. diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md index eca2c0e07..945cd5552 100644 --- a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -7,12 +7,27 @@ dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14 --- -## ⏸ SESSION CHECKPOINT (2026-07-05 — DATABIN1 RUN LANDED; resume here) +## ⏸ SESSION CHECKPOINT (2026-07-05 — P2-7a VALIDATED + full-run roots nailed; resume at #15) -**Code + tests landed & GREEN; the P2-7 databin1 run is DONE and CHECKED.** Full suite: 606 mode/telemetry/ -selector + 115 async_cifar10 parity green. **Outcome (see P2-7 below):** K-D29 broke the RNG-desync root — sync -baselines are `var` bit-identical + `timing_overrun=0%` on bin 1; the only sync residual is a benign delay-tie -order swap (OPEN FORK P2-7a); fluxtune still overruns 38% as predicted → next is P2-5 (`perturbation_count`↓). +**P2-7a (K-D31) VALIDATED and full runs analyzed.** Databin1 checks (`--max-bin 1`): sync `cohort_sequence` +ok=true, set/order/var/cadence=1.0 — the delay-tie is closed. Full runs (`--delay-factor 1`) then settled BOTH +open roots: +- **SYNC:** `sim_rate` 2.9–3.0 (#12c resolved). But receive-ORDER now 41/41 identical yet cadence still breaks at + **bin 7** → the "order→var→RNG-desync" root is REFUTED for the full run. Real root = **grad non-reproducibility + given matched order** (~1e-3 GPU fp16 jitter amplified by the split-half variance → gate flip at (7,2)). This + **answers P0-2**: exact cadence parity is unattainable past ~bin 6. ⇒ relax the target: `cohort_sequence` EXACT + scoped to `--max-bin 1`; add a DISTRIBUTIONAL cadence/var rung (mean-band + KS + `var_good` fraction) for the full + run. (Confirm with a 2-real-run diff before landing — strong single-run evidence already.) +- **fluxtune:** `sim_rate=0.50` is a **GPU-PIPELINING loss (#15)**, not #12c and not the commit gating (which is + correct, per-grad). Objective telemetry: sim 1.54× GPU concurrency vs real 3.37×; per-commit real 4.30s wall, sim + 6.60s wall / 3.27s vclock. The overrun (#1d, `set_match=3/272`) is a thin-margin contention tail (10 trainers / + 8 GPUs doubling + aggregator eval GPU) — `jvp_perf_opt` already put the MEAN (3.61s) under the 4.0s budget. +- **Pinning is clean** (earlier "under-provisioned" read was a misread of `gpu=4.9s` compute-time as device IDs): + 8 GPUs, balanced round-robin, CPU-pinned. Aggregator GPU pin + trainer `[PIN]` self-report + `[LOAD_BALANCE]` + check landed (K-D33, 120 launch tests green). + +**NEXT: #15 — decouple real-GPU dispatch from the sct-ordered commit drain** (keep GPUs full like real; drain orders +commits by sct for the vclock only). This is the one thing keeping fluxtune `sim_rate<1`, and it also lifts #1d. **DONE (landed + tested):** - **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both diff --git a/lib/python/examples/fwdllm/fluxtune_contributions.md b/lib/python/examples/fwdllm/fluxtune_contributions.md index 6821bb046..7051f3205 100644 --- a/lib/python/examples/fwdllm/fluxtune_contributions.md +++ b/lib/python/examples/fwdllm/fluxtune_contributions.md @@ -132,8 +132,11 @@ so training fidelity and real↔sim simulator parity are untouched: 2. **Remove redundant/diagnostic forward passes** — 3 passes that only fed a log line, plus (fluxtune) reusing the selected perturbation's already-computed JVP: **fluxtune 25→20 passes, sync 5→2.** -**Combined: sync −68%, fluxtune −37% GPU time, zero fidelity change.** This alone is expected to bring fluxtune's -per-batch compute under its modeled mobile-delay budget. +**Combined: sync −68%, fluxtune −37% GPU time, zero fidelity change.** Measured under the real 10-trainer / 8-GPU +run this brings fluxtune's **mean** per-batch compute (3.61s) under the 4.0s modeled mobile-delay budget, but the +**tail** (4.1–5.4s) still overruns on the two GPUs that carry 2 trainers each (10>8) plus the aggregator's eval +GPU — a contention effect, not the JVP cost. Clearing the tail needs the sim's GPU pipelining fix (keeps compute +near the ~2.4s uncontended floor) and/or 1-trainer-per-GPU. **Deliberately *not* adopted** (they change fidelity / are inferior here): **vmap batching** (2× but re-baselines the fp32 trajectory via §4.2 cancellation — exact only in fp64), **exact forward-mode AD** (0.5×, needs eager diff --git a/lib/python/examples/fwdllm/simulate_fwdllm.md b/lib/python/examples/fwdllm/simulate_fwdllm.md index dcfa5da9c..7cf8a4fc2 100644 --- a/lib/python/examples/fwdllm/simulate_fwdllm.md +++ b/lib/python/examples/fwdllm/simulate_fwdllm.md @@ -37,30 +37,34 @@ discipline, starvation self-termination, A6/A7/A8/K11 ground-truth rungs). ## §A Current status -**All three baselines now run end-to-end at the 2700s budget — the startup race (#14), the R1 re-dispatch churn -(#1c) and the 30s drain stall (#13) are all FIXED and validated. Two standing fronts remain: (1) LOGICAL PARITY — -prove the sim takes the same steps in the same order as real on a matched scope (first data bin); (2) `sim_rate < 1` -— the sim burns more wall than the virtual time it models, so the sim job is not faster than the real job.** - -Latest pairs (`run_sequential.sh --mode both --delays on --max-runtime-s 2700`, `run_..._041051 → 080931`): - -| baseline | real (data_id in ~2710s wall) | sim `sim_rate` | sim wall to real's data_id | R1 | verdict | -|---|---|---|---|---|---| -| **fwdllm** (sync) | 69 | **0.935** | 1345s (real 2707s → **sim ~2× faster/progress**) | 0% (sync) | runs clean; `sim_rate` just under 1 | -| **fwdllm_plus** (sync) | 18 | **0.948** | 346s (real 2719s → **sim ~8× faster/progress**) | 0% (sync) | runs clean; `sim_rate` just under 1 | -| **fluxtune** (async) | 69 | **0.303** ⛔ | 6699s (real 2704s → **sim ~2.5× slower**) | **0** ✓ (`SIM_R1_DISPATCH=0`, evict=1, 0 recv_fifo timeouts) | drain + R1 fixed; acute `sim_rate` case **+ a genuine logical divergence (#1d, async cohort order)** | - -**Why `sim_rate < 1` (the #12c root, not the drain).** With no `--delay-factor`, `training_delay_factor=None` → -the hardcoded ÷10 → modeled delay D≈0.4–1.8s. So each commit charges ~only the real GPU pass (which the sim -genuinely runs) to the vclock — there is almost no skipped transport wait to compress → `sim_rate` ceilings just -below 1 for the sync baselines, and fluxtune's async strict-sct-order straggler holds drag it to 0.30. Per -principle #13 the sim MUST deliver speedup (`sim_rate ≥ 1`); it does not yet. - -**The `sim_rate` lever — `--delay-factor` (#12c), applied within the first-data-bin scope below.** Re-run with the -full registry delay so `sct = send + gpu + D`, D≈4–18s → the vclock jumps by gpu+D per commit while wall pays only -the skipped GPU → expect `sim_rate` toward/past 1. Pair it with the #15 hard-wall-pause audit (the sim must not -sleep real seconds it should fast-forward). Sync baselines are already near-parity in rate; the same should push -them over 1. (Overall next step is first-data-bin logical parity — see STRATEGY / Next roots.) +**All three baselines run end-to-end (#14/#1c/#13 fixed). SYNC baselines are in good shape: with the full registry +delay (`--delay-factor 1`) `sim_rate` is now 2.9–3.0 (#12c RESOLVED for sync) and K-D31 makes bin-1 cohort order +BIT-EXACT (P2-7a validated). The two live fronts are now BOTH understood at the root: (1) a SYNC float-nondeterminism +wall at ~bin 7 (grads are not bit-reproducible → exact cadence parity is unattainable past bin ~6 → the parity +target beyond bin 1 must be DISTRIBUTIONAL); (2) fluxtune `sim_rate = 0.50` — a pure GPU-PIPELINING loss (the sim +runs its 8 GPUs at 1.54× concurrency vs real's 3.37×), NOT the commit gating (which is correct) and NOT GPU +under-provisioning (pinning is clean — see §H).** + +Latest FULL pairs (`run_sequential.sh --mode both --delays on --delay-factor 1 --max-runtime-s 2700`, +`run_20260705_1924 → 2046`): + +| baseline | sim `sim_rate` | vclock / wall | verdict | +|---|---|---|---| +| **fwdllm** (sync) | **2.93** ✓ | 1139s / 389s | clean; sim ~3× faster than the virtual time it models | +| **fwdllm_plus** (sync) | **3.01** ✓ | 1137s / 378s | clean; sim ~3× faster | +| **fluxtune** (async) | **0.50** ⛔ | 1191s / 2425s | `sim_rate<1` is now isolated to a GPU-pipelining loss (#15), not #12c | + +**Why fluxtune `sim_rate = 0.50` — objective real↔sim telemetry (supersedes the #12c-delay-headroom reading).** +Real and sim do the SAME GPU work (~3.7–4.0k trainer-s) with the same ~480s 8-way pipeline floor. Real packs it into +1210s wall by keeping **3.37 trainers on the GPUs at once (98% busy)**; the sim takes **2425s** at **1.54× +concurrency (85% busy)**, converting real's correctly-skipped device-delay waits (4775s) into **19807s of trainer +idle in `recv`** (vs real 380s). Per-commit: real reaches agg-goal in **4.30s wall**; the sim models **3.27s vclock** +but spends **6.60s wall** → sim_rate 0.50. ROOT: the strict-sct-order drain (`_sim_recv_min_grad`) commits one grad +per call and blocks real wall for the *earliest-sct* in-flight trainer, starving GPU pipelining. The hold-to-commit +slot residence (K-D17b) is CORRECT and per-grad (released at that grad's commit) — the loss is pure pipelining. Fix: +**#15 — decouple real-GPU dispatch from virtual commit ordering** (keep all GPUs full like real; let the drain order +commits by sct purely for the vclock). Target: 3 grads at the ~2.4s uncontended floor, pipelined ⇒ ~2.4s wall < +3.27s vclock ⇒ sim_rate ~1.35. ### Parity scoreboard — REFERENCE baseline (checker run on the pairs above; `expt_scripts/run_parity.py --yes`) *These are the numbers we hold against until the open issues resolve — it will be a while before a longer run. @@ -108,61 +112,51 @@ DETERMINISTIC in both real and sim by design (operator-confirmed) → exact matc | baseline | receive-SET (to real's max bin) | cadence | logical parity HOLDS TO | verdict | |---|---|---|---|---| -| **fwdllm** | 194/194 identical (trivial: K=10=all) | 22/194 | **data_id 7** — breaks at **8** | **BREAKS @ bin 8** — sim inserts +1 iteration at data_id 7 | -| **fwdllm_plus** | 48/48 identical | **48/48 identical** | **data_id 18 (real's max)** — no break | **PARITY ✓ to bin 18** (extend when a longer real run exists) | -| **fluxtune** | **0/17 identical** | 7/17 | **breaks at bin 0** (cohort #1) | **DIVERGES from the first aggregation** — async cohort order | - -**ROOT (2026-07-05 code+telemetry investigation — supersedes the earlier "order is benign" reading):** all three -break from ONE cause — **the sim's commit/receive ORDER ≠ real's actual arrival ORDER.** fwdllm's variance is a -**split-half statistic over the commit-ORDERED grad list** (`fwdgrad_utils.py:133-158`), so order matters even when -the cohort SET is identical; the sim commits in sct order, real in physical-arrival order → different `var` → the -`var<0.3` gate flips at a different iteration → and since each trainer's `torch.Generator` is seeded once and -**never reset**, one extra iteration desyncs every trainer's RNG → grads diverge ~1% thereafter (the fwdllm bin-8 -break). For fluxtune (agg_goal=3 **STATUS (2026-07-05 — DATABIN1 RUN LANDED, `--delay-factor 2 --max-data-id 1`, `smoke_logs/20260705_150145`).** -> The K-D29 model fix **broke the RNG-desync root** for the sync baselines: on bin 1 `cohort_sequence` now shows -> **`var` BIT-IDENTICAL** real↔sim (fwdllm `0.21480107…`==, fwdllm_plus `0.37160512…`==; banked logs had -> `var_match_frac=0.5` with real≠sim) + `set_match=1.0` + `cadence_match=1.0`, and **`timing_overrun=0%`** — grads -> are now mode-invariant on bin 1. The SOLE residual is `order_match<1.0` from **one benign delay-TIE swap**: -> trainers 3 & 9 (…372/…378) both drew `training_delay_s=13.0` (registry, → D=6.5) so their sct ties; real breaks -> it by physical arrival, sim by sct-sort — but both land in the SAME split-half, so `var`/cadence/grads are -> unchanged. `cohort_sequence` (EXACT order) still FAILs on it. **fluxtune** behaves exactly as predicted: -> `timing_overrun=38%` (GPU≈4.2s > min cohort D of 2.0/2.5/3.5s), `set_match=0.22`, `var` mean off 29% → genuine -> break, **timing-model-limited** → fix is P2-5 (`perturbation_count`↓ to put GPU **SYNC TIE FIX LANDED (K-D31, operator chose canonicalize-by-trainer_id):** the trainer stamps its pure modeled -> delay `D` (`MODELED_DELAY_S`, both modes); the aggregator sorts each cohort's commits by `(D, str(end))` before -> the telemetry snapshot + `aggregate()`, so equal-D ties break by trainer_id IDENTICALLY in real and sim → -> receive order matches exactly while `var`/grads stay bit-identical (reorder only ever moves a grad WITHIN a -> split-half). 428 mode + 9 new canon + 115 async_cifar10 parity tests green; async byte-identical. -> **PENDING: the operator's re-run** (`--only fwdllm,fwdllm_plus --delay-factor 2 --max-data-id 1`) to confirm -> `cohort_sequence` order_match→1.0. Full diagnosis + numbers: [PARITY_LOGICAL_TASKS.md](PARITY_LOGICAL_TASKS.md) P2-7/P2-7a. +| **fwdllm** | 41/41 identical (K=10=all) | 22/41 | **data_id 6** — breaks at **7** | **BREAKS @ bin 7** — but receive-ORDER now 41/41 (K-D31) → the break is NOT order | +| **fwdllm_plus** | 14/14 identical | **14/14 identical** | **real's max (~13)** — no break | **PARITY ✓** over its (shorter) real run | +| **fluxtune** | **3/272 identical** | 29/272 | **breaks at bin 0** | thin-margin overrun (§ #1d) — GPU tail > min D on the contended-doubled GPUs | + +**ROOT — CORRECTED (2026-07-05 full-run investigation; supersedes the "order → var → RNG-desync" reading for the +SYNC full run).** With K-D31, fwdllm's receive-ORDER is now **41/41 identical** on the full run — yet cadence still +breaks at data_id 7. So order is NOT the sync full-run cause. The real residual is **grad non-reproducibility given +matched order**: `|Δvar|` is ~1e-3 through bin 6 with *every* `var_good`/force decision matching, then at bin 7 +(all-10 cohort, identical order, identical RNG) grads diverge ~1e-3 (GPU fp16 non-reproducibility), which the +split-half variance — a ratio with a near-zero denominator at a bin's first iteration — **amplifies to a 0.26 var +swing**, flipping the `var<0.3` gate at (7,2). Both modes confirmed `jvp_perf_opt=False` (no config skew). This +**answers P0-2 empirically: grads are NOT bit-reproducible → exact cadence parity is unattainable past ~bin 6.** +⇒ **Parity target (operator decision): cohort SET = HARD; `var_good`/cadence = HARD to bin 1, DISTRIBUTIONAL beyond; +`var` VALUE = SOFT (tolerance/KS); receive-ORDER within a set = SOFT for sync (fedavg order-invariant + K-D31 +canonicalizes). Keep `cohort_sequence` EXACT but scoped to `--max-bin 1`; add a distributional cadence/var rung for +the full run.** For **fluxtune** the SET still genuinely diverges (#1d) — a timing/pipelining cause, not +nondeterminism. Full diagnosis: [PARITY_LOGICAL_TASKS.md](PARITY_LOGICAL_TASKS.md). + +> **STATUS (2026-07-05 — P2-7a VALIDATED + full-run bin-7 wall found).** Databin1 checks +> (`parity_fwdllm_syn_0_165256`, `_plus_165736`, `--max-bin 1`): sync `cohort_sequence` **ok=true, +> set/order/var/cadence = 1.0** — K-D31 closed the benign delay-tie; P2-7a DONE. Full runs then exposed the bin-7 +> float-nondeterminism wall above (order matches 41/41, cadence still breaks) → the doc's exact-cadence target is +> valid only ≤bin 1; beyond it must be distributional (pending a two-real-run P0-2 confirmation — strong single-run +> evidence already). **fluxtune (`--delay-factor 1` full run):** `jvp_perf_opt=True` cut GPU 7.57→3.61s MEAN (under +> 4.0s min budget) but the TAIL (4.1–5.4s) still overruns on the two doubled GPUs (10 trainers / 8 GPUs) → +> `set_match=3/272`. The overrun is now a thin-margin contention effect, not the JVP algorithm. ### Open issues (OPEN only — closed items live in §G/§H) | # | issue | baseline(s) | next step | |---|---|---|---| -| **#1d** ⭐⭐ | **fluxtune LOGICAL divergence (not timing): async cohort composition differs every aggregation (0/17 receive-sets match, bin≤1).** The sim's sct-sorted receive order does not reproduce real's actual first-`agg_goal` arrival order → var<0.3 crossing shifts → sim needs +2 aggregations to clear bin 0 (V1 genuinely off). CORRECTNESS bug — fix BEFORE fluxtune's time dimension (principle #14/#16). | fluxtune | Diff per-trainer real-GPU time vs sim-sct order — does the sct use ACTUAL measured GPU or a nominal speed? First resolve the conceptual Q: is real's receive-order deterministic or wall-jittery (§A logical-parity check)? | -| **#12c** ⭐ | `sim_rate < 1` — no delay-headroom to compress. `--delays on` uses the hardcoded ÷10 shrink (D≈0.4–1.8s) unless `--delay-factor` is passed, so the vclock charges ~only GPU. | all (acute: fluxtune) | Re-run with `--delay-factor` (sct = send + gpu + D, D≈4–18s) → `sim_rate` toward/past 1. Then re-run the checker. Config-flow otherwise correct (K-D24/K-D21). | -| **#15** ⭐ | **Hard real-wall pauses in the sim.** fluxtune sim wall 8879s ≫ vclock 2694s (3× overhead) — the vclock must advance THROUGH the modeled wait after GPU compute (fast-forward), never make the sim PROCESS sleep real seconds. Suspect: drain holds waiting real-wall for the slowest straggler's GPU + any non-parallel GPU dispatch. | all (acute: fluxtune) | Audit every place the sim blocks real-wall (drain grace, recv_fifo, GPU dispatch serialization). Confirm the 10 trainers' GPU runs in PARALLEL and the only irreducible real wall is one GPU pass; everything else must be a vclock jump. | -| **#7** | fwdllm_plus real ~4× slower/round (156 vs 41 s/round); at syn_0 real sees only ~4.9 eligible vs sim ~9.6. Not a sim bug. | fwdllm_plus | Profile per-iteration reselection + oracular-read cost from the banked per-phase log; explain the eligible-count gap at 100% avail. | -| **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits; one-grad-per-poll drain tail) — real-only. | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run (principle #8/#11c). Never change grad values/cadence. | +| **#15** ⭐⭐ | **fluxtune `sim_rate = 0.50` — GPU-PIPELINING loss, not delay-headroom (#12c) or gating.** Objective telemetry: same GPU work as real (~3.7–4.0k trainer-s), same ~480s 8-way floor, but sim runs 1.54× concurrency vs real 3.37×; per-commit real 4.30s wall, sim 6.60s wall / 3.27s vclock. The strict-sct-order drain commits one grad/call blocking real wall for the earliest-sct straggler → GPU starved. Hold-to-commit residence is CORRECT (per-grad). | fluxtune | **DECOUPLE real-GPU dispatch from virtual commit ordering**: keep all 8 GPUs full (dispatch/refill like real); `_sim_recv_min_grad` orders commits by sct purely for the vclock, never blocking a ready GPU on a slower-sct straggler. Target sim_rate ~1.35. | +| **#N (bin-7 nondeterminism)** ⭐ | **SYNC exact-cadence parity has a float-nondeterminism wall at ~bin 7.** Order matches 41/41 (K-D31) yet cadence breaks: ~1e-3 GPU fp16 grad jitter, amplified by the split-half variance ratio, flips the `var<0.3` gate at (7,2). Not a sim bug. | fwdllm (fwdllm_plus latent) | Confirm with a 2-real-run diff (P0-2). Then land the parity-target relaxation: `cohort_sequence` EXACT scoped to `--max-bin 1`; distributional cadence/var rung (mean-band + KS + `var_good` fraction) for the full run. | +| **#1d** ⭐ | **fluxtune cohort SET diverges (thin-margin overrun).** `set_match=3/272`. `jvp_perf_opt` cut GPU to 3.61s MEAN (<4.0s budget) but the tail (4.1–5.4s) still overruns on the two GPUs that carry 2 trainers each (10/8) + the aggregator's eval GPU. Order flips → wrong 3-of-K commit. | fluxtune | Aggregator-GPU pin landed (K-D33); with #15's pipelining the compute drops toward the ~2.4s uncontended floor (<4.0s). If a residual tail remains: `perturbation_count`↓ (P2-5) or 1-trainer/GPU. | +| **#7** | fwdllm_plus real ~4× slower/round; at syn_0 real sees only ~4.9 eligible vs sim ~9.6. Not a sim bug. | fwdllm_plus | Profile per-iteration reselection + oracular-read cost from the banked per-phase log; explain the eligible-count gap at 100% avail. | +| **#11** | real-mode critical-path waste (`sleep(0.1)` MQTT-settle busy-waits; one-grad-per-poll drain tail) — real-only. | fwdllm, fwdllm_plus (real) | **Deferred to a validated pass** — ZERO parity impact (sim already skips them); removing them changes the working real reference + needs a real run (principle #8/#11c). | ### Next roots — ranked (correctness before time; SHARED before per-baseline — principle #14) -0. **DONE — first-data-bin logical-parity check ran.** fwdllm + fwdllm_plus are **logically clean on bin 1** - (cadence + cohorts identical) → their whole-run cadence fails are length-confound, and they are **cleared to - chase the time dimension**. fluxtune has a **genuine logical divergence (#1d)**. -1. **#1d fluxtune async cohort/receive-order (correctness, per-baseline) — TOP.** The one real logic bug. Root-cause - the sct receive-order vs real arrival order; resolve the deterministic-vs-jittery conceptual question first. -2. **#12c `--delay-factor` + #15 hard-wall-pause audit (time, SHARED).** Give the vclock its delay-headroom AND - stop the sim burning real wall it should skip → `sim_rate ≥ 1`. These are the `throughput`/`overhead`/`per_round` - fail family; safe to pursue for the two sync baselines now, and for fluxtune once #1d lands. -3. **#7 fwdllm_plus real speed / selection divergence (per-baseline).** `eligibility`/`selection_detail` — real - completes now; explain the 4× slowness + the syn_0 eligible-count gap from telemetry. Not a sim bug. -4. Then C1/C2 convergence at matched `data_id` per baseline → gate to Phase 2 (unavailability). +1. **#15 fluxtune GPU-pipelining decouple (time, per-baseline) — TOP, IN PROGRESS.** The one thing keeping fluxtune + `sim_rate<1`. Decouple GPU dispatch from the sct-ordered commit drain. Also lifts #1d (compute → uncontended floor + < budget). Sync `sim_rate` already 2.9–3.0. +2. **bin-7 nondeterminism → relax the parity target (SHARED, correctness-of-CHECK).** Land the distributional + cadence/var rung + scope `cohort_sequence` EXACT to bin 1, after the P0-2 two-real-run confirmation. Un-reds the + sync full-run cadence fails that are float-noise, not bugs. +3. **#7 fwdllm_plus real speed / selection divergence (per-baseline).** Not a sim bug; explain from telemetry. +4. Then C1/C2 convergence (distributional target) at matched `data_id` per baseline → gate to Phase 2. ### SKIP audit (20–22 skips; ~17 legit) Legit at Phase-1 syn_0 + `random` selector: 7 availability ground-truth rungs + 4 delivery/withheld (Phase-2 @@ -306,6 +300,14 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso --- ## §G Fixes landed (what worked — ≤20-word problem + ≤20-word fix; do not redo) +- **#12c sync `sim_rate` (delay-factor).** No delay-headroom starved the vclock. Fix: `--delay-factor 1` (full + registry D) → sct = send + max(gpu,D) charges the skipped device wall → sync `sim_rate` 0.94→2.9–3.0. +- **K-D31 validated (P2-7a).** Databin1 checks: sync `cohort_sequence` ok=true, set/order/var/cadence=1.0 — the + benign delay-tie is canonicalized away; bin-1 cohort order is bit-exact. +- **Pinning visibility + aggregator GPU pin (K-D33).** Trainer emitted no actual CPU/GPU; aggregator eval defaulted + to GPU 0 (contended with trainers 1/9). Fix: trainer `[PIN]` self-report (`trainer/main.py`), `[LOAD_BALANCE]` + post-proc check (`spawner.spawn_all` — warns on imbalance / under-provisioning), aggregator `gpu_id` pin (least- + loaded/idle GPU). 120 launch tests green; async_cifar10 shares the launcher, unaffected. - **#13 drain stall (K-D28/b/c).** fwdllm's `_sim_recv_min_grad` left a stuck straggler in the expected set → re-fired the full 30s `RECV_TIMEOUT` every cycle → pipeline starved (`sim_rate` 0.06). Fix: felix-port stuck-end eviction + recv-grace 2→5s + probe-ceiling/ready-gating + `drain_ready` direct ingest → `sim_rate` @@ -340,6 +342,18 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso --- ## §H Dead-ends & corrections — do NOT retry +- **"The sync cadence break is a sim ORDER bug (order → split-half var → RNG desync) — exact cadence parity is + achievable once order matches."** CORRECT for bin ≤1, REFUTED for the full run (2026-07-05). K-D31 made + receive-ORDER 41/41 identical, yet fwdllm cadence STILL breaks at bin 7. Root is grad NON-reproducibility given + matched order: ~1e-3 GPU fp16 jitter, amplified by the split-half variance ratio, flips the `var<0.3` gate at a + sensitive bin. *Lesson:* exact `var` is only a valid target ≤bin 1; beyond it the target is DISTRIBUTIONAL. Don't + chase exact cadence past the nondeterminism wall (principle #16) — it reds on float noise, not a bug. +- **"The fluxtune runs are GPU under-provisioned (2–4 of 8 GPUs, 5 trainers/GPU) → contention is the root."** WRONG — + a misread of `gpu=4.9s` (GPU compute SECONDS in the delay log) as device IDs. The spawn table is authoritative: + **8 GPUs, balanced round-robin** (`spawner.py:305` `(tid-1)%num_gpus`), CPU `sched_setaffinity` 1 core/trainer, + threads capped. Only structural imbalance is 10 trainers > 8 GPUs (GPU 0,1 carry 2 each). Compute IS mode-invariant + at the floor (min 2.4s both modes). *Lesson:* verify a "device id" is a device id; confirm pinning from the spawn + table, not a grep of timing logs. - **"#13 step 4 (freed-slot staggered re-dispatch) closes the residual 11-12s drain holds."** NEUTRAL/REFUTED (K-D28d): `sim_rate` 0.289→0.274, holds persisted. The holds are the drain correctly waiting (strict sct order) for the earliest-sct in-flight straggler while higher-sct grads buffer — INHERENT to real-GPU + strict-order + @@ -449,6 +463,14 @@ Phase 2); D4 (eval-delay factor — confirmed ~1× train cost, K-D3). D1/D6 reso real↔sim; revertible per-config). Startup `[JVP_PERF_OPT]` log confirms 10/10 trainers active. NOT adopted: vmap (2× but fp32-diverges via FD cancellation), fwd-AD (slower). 13 pytests + 185 fwdllm mode + 115 parity green. +- **K-D33** — **pinning self-report + load-balance check + aggregator GPU pin.** Trainers now emit a `[PIN]` line + (actual `CUDA_VISIBLE_DEVICES`/cuda device+name/`cpu_affinity`) at startup (`trainer/main.py` — NOT the dead + `fl_main.py`); `spawner.spawn_all` emits `[LOAD_BALANCE]` (WARN on GPU imbalance, `num_gpusnum_gpus`, else least-loaded) GPU so its eval stops contending GPU 0. Confirmed the + `client_idx%8` device arg is vestigial (`FedSgdTrainer:388` overwrites `self.device=torch.device("cuda")`=cuda:0 of + the CVD-masked view) → the spawner's pin is authoritative. Shared launcher; 120 launch tests green. + *Retired/superseded anchors (kept only as pointers): K-D6 (→K-D12), K-D7/K-D8/K-D10/K-D16/K-D18/K-D19/K-D20/K-D23 — landed scaffolding or corrections, folded into §G/§H; see git history for detail.* diff --git a/lib/python/examples/fwdllm/trainer/main.py b/lib/python/examples/fwdllm/trainer/main.py index f5e82b2af..911d739e8 100644 --- a/lib/python/examples/fwdllm/trainer/main.py +++ b/lib/python/examples/fwdllm/trainer/main.py @@ -76,6 +76,29 @@ def post_complete_message(tc_args): config.hyperparameters.time_mode = _cli_args.time_mode set_seed(config.hyperparameters.manual_seed) + # Pinning self-report: confirm the CPU/GPU affinity the spawner INTENDED + # (spawner.py sets CUDA_VISIBLE_DEVICES + os.sched_setaffinity per trainer) + # actually took effect in THIS child. The trainer trains on torch.device + # ("cuda") == cuda:0 of the CVD-masked single-GPU view (FedSgdTrainer:388), + # so one visible device here == correct pinning. Emitted as a grep-able + # [PIN] line so a post-proc step can verify balanced trainer->(gpu,core) + # placement against the hardware, not just the parent's intended table. + try: + _cvd = os.environ.get("CUDA_VISIBLE_DEVICES", "") + _cores = sorted(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else [] + _gpu_name, _gpu_dev, _ndev = "", -1, 0 + if torch.cuda.is_available(): + _gpu_dev = torch.cuda.current_device() + _gpu_name = torch.cuda.get_device_name(_gpu_dev) + _ndev = torch.cuda.device_count() + logging.info( + f"[PIN] pid={os.getpid()} client_idx={config.hyperparameters.client_idx} " + f"CUDA_VISIBLE_DEVICES={_cvd} cuda_device={_gpu_dev} ({_gpu_name}) " + f"cuda_device_count={_ndev} cpu_affinity={_cores}" + ) + except Exception as _pin_exc: # never let self-report break a trainer + logging.warning(f"[PIN] self-report failed: {_pin_exc}") + # dataset attributes attributes = BaseDataManager.load_attributes(config.hyperparameters.data_file_path) num_labels = len(attributes["label_vocab"]) diff --git a/lib/python/flame/launch/aggregator_spawner.py b/lib/python/flame/launch/aggregator_spawner.py index badf23356..128b6d438 100644 --- a/lib/python/flame/launch/aggregator_spawner.py +++ b/lib/python/flame/launch/aggregator_spawner.py @@ -28,6 +28,7 @@ def spawn( log_to_wandb: bool = False, wandb_run_name: Optional[str] = None, cpu_cores: Optional[set] = None, + gpu_id: Optional[int] = None, ) -> subprocess.Popen: """Spawn aggregator process. @@ -88,6 +89,14 @@ def spawn( # math libs use exactly that many threads (it benefits from a few cores # for chunk reassembly / aggregation, unlike a 1-core-pinned trainer). env = os.environ.copy() + # GPU pin: give the aggregator its own device so its eval pass (a real + # forward on the model, ~8s for fwdllm) does not time-slice the GPU a + # trainer is pinned to. Without this the aggregator defaults to GPU 0 — + # exactly where trainer 1 (and, at 10 trainers/8 GPUs, trainer 9) run — + # inflating those trainers' compute over the modeled delay budget. + if gpu_id is not None: + env["CUDA_VISIBLE_DEVICES"] = str(gpu_id) + print(f" ✓ Aggregator pinned to GPU {gpu_id}") preexec_fn = None if cpu_cores: _cores = {int(c) for c in cpu_cores} diff --git a/lib/python/flame/launch/runner.py b/lib/python/flame/launch/runner.py index e625f2f62..a1ea60e28 100644 --- a/lib/python/flame/launch/runner.py +++ b/lib/python/flame/launch/runner.py @@ -202,12 +202,28 @@ def run_experiment(self, exp: ExperimentConfig) -> None: }, ) + # Dedicated aggregator GPU: prefer a physical GPU the trainer pool + # does NOT use (visible > num_gpus → the first idle one); else the + # least-loaded trainer GPU (highest index under (tid-1)%num_gpus). + _num_gpus = exp.execution.num_gpus + try: + import torch as _torch + _visible = _torch.cuda.device_count() + except Exception: + _visible = 0 + if _visible > _num_gpus: + _agg_gpu = _num_gpus # a fully idle physical GPU + elif _num_gpus > 0: + _agg_gpu = _num_gpus - 1 # least-loaded trainer GPU + else: + _agg_gpu = None self.aggregator_spawner.spawn( paths["aggregator_main"], config_json=json.dumps(agg_cfg), log_to_wandb=exp.aggregator.log_to_wandb, wandb_run_name=exp.aggregator.wandb_run_name, cpu_cores=reserved_cores, + gpu_id=_agg_gpu, ) if not self.aggregator_spawner.wait_until_ready( exp.execution.aggregator_warmup_time diff --git a/lib/python/flame/launch/spawner.py b/lib/python/flame/launch/spawner.py index de1c06f35..1151ce182 100644 --- a/lib/python/flame/launch/spawner.py +++ b/lib/python/flame/launch/spawner.py @@ -428,6 +428,58 @@ def spawn_all( print(f" {'Trainer':>8} {'GPU':>4} {'CPU core':>9} {'PID':>7}") for p in self.processes: print(f" {p['trainer_id']:>8} {p['gpu_id']:>4} {str(p.get('cpu_core', 'N/A')):>9} {p['process'].pid:>7}") + self._assert_load_balanced() + + def _assert_load_balanced(self) -> None: + """Post-spawn load-balance check RELATIVE TO available hardware. + + Verifies the round-robin placement actually spread work evenly and did + not leave hardware idle (an under-provisioned num_gpus is the classic + silent contention source: e.g. 10 trainers on num_gpus=2 packs 5/GPU + while 6 physical GPUs sit idle). Emits one grep-able [LOAD_BALANCE] + verdict; WARNs (never raises) so a run is never blocked by it. + """ + procs = self.processes + if not procs: + return + # trainers per assigned GPU + per CPU core + from collections import Counter + gpu_counts = Counter(p["gpu_id"] for p in procs) + core_counts = Counter(p.get("cpu_core") for p in procs if p.get("cpu_core") is not None) + # visible physical GPUs on the node (independent of num_gpus we chose) + try: + import torch + visible_gpus = torch.cuda.device_count() + except Exception: + visible_gpus = 0 + + issues = [] + # 1. Even GPU spread: round-robin guarantees max-min <= 1; flag otherwise. + if gpu_counts: + spread = max(gpu_counts.values()) - min(gpu_counts.values()) + if spread > 1: + issues.append(f"GPU imbalance: per-GPU trainer counts {dict(sorted(gpu_counts.items()))} (spread={spread}>1)") + # 2. Under-provisioning: physical GPUs left completely idle. + if visible_gpus and self.num_gpus < visible_gpus: + issues.append(f"under-provisioned: num_gpus={self.num_gpus} < visible={visible_gpus} " + f"→ {visible_gpus - self.num_gpus} GPU(s) idle; raise execution.num_gpus") + if visible_gpus and self.num_gpus > visible_gpus: + issues.append(f"over-subscribed: num_gpus={self.num_gpus} > visible={visible_gpus}") + # 3. Even CPU-core spread among pinned trainers. + if core_counts: + cspread = max(core_counts.values()) - min(core_counts.values()) + if cspread > 1: + issues.append(f"CPU imbalance: core reuse spread={cspread}>1") + + per_gpu = ", ".join(f"gpu{g}={n}" for g, n in sorted(gpu_counts.items())) + if issues: + print(f" ⚠ [LOAD_BALANCE] WARN ({len(procs)} trainers, num_gpus={self.num_gpus}, " + f"visible={visible_gpus}): {per_gpu}") + for it in issues: + print(f" - {it}") + else: + print(f" ✓ [LOAD_BALANCE] balanced: {len(procs)} trainers over {self.num_gpus} GPU(s) " + f"(visible={visible_gpus}): {per_gpu}; CPU cores 1/trainer") def wait_all(self, timeout_per_trainer: float = 30.0): """Wait for all trainer processes to complete. From cab01b6dcfd4af2ce1e75fd9561e8c61f831d55c Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Mon, 6 Jul 2026 02:29:48 -0400 Subject: [PATCH 32/56] fwdllm #15: fold root-cause + resume plan into PARITY_LOGICAL_TASKS.md Move the fluxtune sim_rate=0.50 root cause, evidence, fix, and P0-P4 implementation plan into the existing temp tracker (PARITY_LOGICAL_TASKS.md); delete the standalone handoff markdown per review. Co-Authored-By: Claude Opus 4.8 --- .../fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md | 178 ------------------ .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 76 ++++++++ 2 files changed, 76 insertions(+), 178 deletions(-) delete mode 100644 lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md diff --git a/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md b/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md deleted file mode 100644 index d095d0b5d..000000000 --- a/lib/python/examples/fwdllm/FLUXTUNE_SIMRATE_15_HANDOFF.md +++ /dev/null @@ -1,178 +0,0 @@ -# #15 fluxtune `sim_rate` — ROOT CAUSE FOUND, resume-here handoff - -**Status (2026-07-05):** Root cause of fluxtune `sim_rate = 0.50` is IDENTIFIED and evidence-backed. The fix -(re-dispatch-on-return) is designed and de-risked but NOT yet implemented. Pick up at "IMPLEMENTATION PLAN" below. -This is a TEMP tracker for the #15 work — fold into `simulate_fwdllm.md` §G/§K and delete when the fix lands. - -> Prereqs / cross-refs: `simulate_fwdllm.md` (§A open issue #15, §H corrections, principles #13/#15/#16), -> `PARITY_LOGICAL_TASKS.md` (checkpoint). The banked evidence run is -> `experiments/run_20260705_204619_fluxtune_n10_smoke_syn_0_sim` (+ its `_real` pair `run_20260705_202448`). - ---- - -## 1. THE ROOT CAUSE (definitive, evidence-backed) - -**A circular wait between the sim drain's `earlier_stuck` gate and the hold-to-commit slot residence.** - -The drain (`_sim_recv_min_grad`, `flame/mode/horizontal/syncfl/fwdllm_aggregator.py:752`) blocks in REAL WALL to -preserve sct-ordered commits: it will not commit a buffered grad while an *in-flight* trainer has a smaller -**expected** sct (`_sim_inflight_expected[end]`). But that in-flight trainer is itself **blocked in `recv_wrapper`** -(hold-to-commit: `_release_end_on_return:1136` returns early on the residence path, so the trainer's slot only frees -when its PREVIOUS grad commits) — so it is NOT computing and cannot produce the grad the drain is waiting for. Its -previous grad won't commit because the drain is stalled. Circular; broken only by the 30s `RECV_TIMEOUT_WAIT_S` -failsafe evict. - -This is the STEADY STATE, not an edge case. The startup is fine (all 10 dispatched fresh → compute in parallel, -concurrency=10 for ~5s) and it collapses right after the first commit, when trainers start getting held. - -### Causal chain (fully closed) -``` -hold-to-commit blocks trainers in recv (recv_wrapper mean 30.9s, max 60s) - → drain earlier_stuck gate waits real-wall for these NON-COMPUTING in-flight trainers - → circular stall, broken only by the 30s failsafe evict - → ~1974s (81% of the 2425s sim wall) burned in gate-waits - → GPU concurrency 1.54x (vs real 3.37x) - → sim_rate = vclock/wall = 1191/2425 = 0.50 -``` - -## 2. THE EVIDENCE (all from the banked sim/real pair; reproduce with the scripts in §5) - -- **Per-commit:** real reaches agg-goal in **4.30s wall**; sim models **3.27s vclock** but spends **6.60s wall** → - per-commit sim_rate 0.50. -- **GPU concurrency:** real avg **3.37x** (98% busy), sim avg **1.54x** (85% busy). Same total GPU work - (~3.7-4.0k trainer-s), same ~480s 8-way pipeline floor. `max=10` at startup (proves the HW sustains 10-wide). -- **Trainer recv-block (trainer 371):** sim `recv_wrapper` mean **30.9s** / max 55s per iteration; real **0.28s** - (median **0.01s**). Compute is mode-invariant (~3-4s, delay correctly skipped `_emulate_training_delay=0.000s`). -- **The smoking-gun stall:** `[SIM_GRAD_STUCK_EVICT] round=1 end=0379 exp=24.0 bmin=27.0` — drain blocked ~30s - waiting for trainer 379 (expected sct 24) while **6 grads sat ready** (bmin 27). Trainer 379's own telemetry: - blocked in `recv_wrapper` for **60.39s** (20:47:19→20:48:19), got weights the instant AFTER the evict. -- **Steady-state:** `buf_depth` is **constantly 6** (six grads always ready, held behind the deadlocked one); - **524/1093 commits (48%) stall >2s** = **1974s** total wall (~81%). -- **Out-of-order already happens:** committed scts go 11,13,7,5.4,5.5,10,13,16… (non-monotonic) — so the strict - ordering the gate blocks for is not even being preserved; it pays full real-wall cost for nothing. -- **Eval:** once per data-bin (23 evals / 24 data_ids), NOT per iteration. But it's an 8.5s BLOCKING call on the - aggregator (~195s / 8% of wall). Secondary; overlap later. - -## 3. THE PROPER FIX - -The gate is not wrong to want sct order — the bug is it **waits for trainers that aren't running**. Fix = ensure -the trainer the gate waits for is ALWAYS actually computing → **re-dispatch on RETURN** (match real, whose recv is -0.01s), decoupling physical GPU pipelining from the virtual in-flight ledger. - -- A returned trainer immediately gets the next weights and computes → never idle-blocked in recv. -- The in-flight trainer the gate waits for then produces its grad in ~4s IN PARALLEL with 7 others → the drain - commits a BATCH per GPU-pass instead of stalling on one held trainer. `buf_depth` stays deep, gate-waits become - short and legitimate, 30s failsafe never fires. Expected: concurrency 1.54x → ~3.37x+, sim_rate 0.50 → ~1.3-2+. -- **Keep the gate + sct-order commits** (they preserve #1d cohort parity) — only remove the recv-block that starves - the pipeline. The virtual in-flight COUNT / selection eligibility (R1) stays held-to-commit; that is a SEPARATE - ledger from the trainer's physical compute. Conflating them is what caused this (K-D17b held both together). - -### The hard constraint to preserve (principle #16 — no hacks) -The re-dispatched (compute-ahead) grad must be computed against the **model version real dispatched** — fluxtune's -fedbuff down-weights by staleness `V'-V`, so a grad against the wrong version silently changes the result. Since -REAL also re-dispatches on return and cycles continuously, the versions come from the same deterministic commit -order (K-D29 remainder-wait). **This should hold but MUST be verified from telemetry, not assumed** — see the P0 -gate below. - -### Belt-and-suspenders (independently correct, do alongside) -`_sim_inflight_expected[end]` is stamped at DISPATCH (`fwdllm_aggregator.py:2819`, `_sst + _budget` where -`_sst = self._vclock.now`, `_budget = self._sim_trainer_budget.get(end, self._sim_budget_min)`) assuming the -trainer starts computing immediately. A HELD trainer hasn't started, so the expected sct is a fiction (too early) → -the gate waits for a grad that won't come. Tie the expected clock to actual compute-START, and/or bound the gate's -real-wall wait far below the 30s failsafe. - -## 4. IMPLEMENTATION PLAN (resume here tomorrow) - -**P0 — GATE (verify before coding, read-only):** Pull per-trainer `MODEL_VERSION` (the version each grad was -computed against) vs the commit sequence, real vs sim. Confirm real absorbs the staleness (grad version = dispatch -version, fedbuff down-weights) and that re-dispatch-on-return in sim yields the SAME dispatch-version sequence as -real. Anchor: `MessageType.MODEL_VERSION` on grad msgs (`fwdllm_aggregator.py:_process_single_trainer_message`, -staleness log ~line 1170); `inc_model_version_per_data_id=True`. If versions DON'T align, the design needs the -version pinned to the trainer's virtual completion (sct) rather than physical dispatch — revisit before coding. - -**P1 — CORE CHANGE (once P0 passes):** Decouple physical re-dispatch from the virtual commit-hold. -- `_release_end_on_return` (`:1128`): on the async sim residence path, currently `return`s (holds slot+guard to - commit). Change so the trainer's PHYSICAL re-dispatch (weights send → compute next grad) happens on return, - while the SELECTOR's virtual in-flight set (`selected_ends`/`all_selected`, driving R1 + `extra = c - inflight`) - stays held to commit via `_sim_hold_busy_slots`. Two ledgers: physical-dispatch (release on return) vs - virtual-inflight (release on commit). -- The re-dispatched grad enters the sct reorder buffer with its own `SIM_COMPLETION_TS`; the drain keeps committing - in sct order for the vclock. Verify the buffer/`_sim_inflight_expected`/`_sim_pending_commit` bookkeeping stays - consistent (K-D27 two-ledger discipline — do NOT reintroduce the R1 regression). -- Fix `_sim_inflight_expected` stamp to reflect actual compute-start (belt-and-suspenders above). - -**P2 — TESTS + TELEMETRY (same change, principle #11):** pytest `tests/mode -k fwdllm` (residence/R1/drain) + -`tests/mode -k parity`; assert async_cifar10 byte-identical (shared `_sim_recv_min` untouched — this is a -fwdllm_aggregator-only edit, principle #8/#9). Add a telemetry assert that `recv_wrapper` mean drops to ~0 and -concurrency rises. R1 must stay ~0 (bank it — K-D19 lesson: an R1 fix isn't done until the smoke shows R1<=2%). - -**P3 — VALIDATE:** re-run `run_sequential.sh --only fluxtune --mode both --delays on --delay-factor 1 ---max-data-id 2` (short). Confirm: `recv_wrapper`→~0, concurrency→~3.37x+, `sim_rate`→>1, `[SIM_GRAD_STUCK_EVICT]` -gone, and — critically — `cohort_sequence`/`var` parity UNCHANGED or improved (the fix must not alter which grads -commit in which cohort; if it does, the version-parity assumption (P0) was wrong). - -**P4 — DOCS:** fold the root cause into `simulate_fwdllm.md` §A (#15) + §H (the "gate waits on held trainers" root) -+ new §K deviation (K-D34, the two-ledger physical/virtual dispatch split); update `PARITY_LOGICAL_TASKS.md` -checkpoint; delete this handoff file. - -## 5. REPRO SCRIPTS (paste-ready, read-only, run from `lib/python/examples/fwdllm`) - -```bash -FS=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_sim | head -1) # banked evidence run -AGG=$(ls "$FS"/*aggregator.log | head -1) - -# (a) the stall: 48% of commits stall >2s = ~1974s wall -grep "SIM_GRAD_RECV" "$AGG" | python3 -c " -import sys,re; from datetime import datetime -prev=None;big=0;tot=0;wait=0 -for l in sys.stdin: - ts=datetime.strptime(l.split(' | ')[0],'%Y-%m-%d %H:%M:%S,%f').timestamp() - if prev is not None: - d=ts-prev; tot+=1 - if d>2: big+=1; wait+=d - prev=ts -print(f'commits={tot+1} gaps>2s={big} ({100*big/tot:.0f}%) wall_in_waits={wait:.0f}s')" - -# (b) the evict + buf_depth stuck at 6 -grep -E "SIM_GRAD_STUCK_EVICT|SIM_GRAD_RECV" "$AGG" | grep -oE "buf_depth=[0-9]+" | sort | uniq -c -grep "SIM_GRAD_STUCK_EVICT" "$AGG" - -# (c) trainer recv-block sim vs real (trainer 371) -for m in sim real; do D=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_$m|head -1) - tf=$(ls "$D"/telemetry/trainer_*371.jsonl|head -1) - python3 - "$tf" "$m" <<'PY' -import json,sys,statistics as st -f,m=sys.argv[1],sys.argv[2]; recv=[];tr=[] -for l in open(f): - e=json.loads(l) - if e.get('event')=='step_timing': - if e.get('func')=='recv_wrapper' and e.get('duration_s') is not None: recv.append(e['duration_s']) - if e.get('func')=='_train_one_batch' and e.get('duration_s') is not None: tr.append(e['duration_s']) -print(f"{m}: recv_wrapper mean={st.mean(recv):.1f}s median={st.median(recv):.2f}s | train mean={st.mean(tr):.2f}s") -PY -done -``` - -Aggregator timeline decoder (per-step wall) + GPU-concurrency scripts were run ad hoc during the investigation; -the key numbers are all captured in §2 above. The `[SIM_GRAD_RECV]` log line -(`fwdllm_aggregator.py:926`, format `end= sct= T_v= buf_depth= inflight_exp= sel_ends=`) is the primary drain -diagnostic — grep it to watch the gate. - -## 6. LANDED THIS SESSION (already committed separately / in this commit) -- **K-D33 pinning (in this branch):** trainer `[PIN]` self-report (`examples/fwdllm/trainer/main.py` — NOT the dead - `fl_main.py`); `[LOAD_BALANCE]` post-proc check (`flame/launch/spawner.py::_assert_load_balanced`); aggregator GPU - pin (`flame/launch/aggregator_spawner.py` `gpu_id` param + `flame/launch/runner.py` picks idle/least-loaded GPU). - 120 launch tests green. Corrected earlier "under-provisioned GPUs" misread — pinning IS clean (8 GPUs, balanced - round-robin, CPU sched_setaffinity 1 core/trainer). `client_idx%8` device arg is vestigial (`FedSgdTrainer:388` - overwrites `self.device=torch.device("cuda")`). -- **Docs updated:** `simulate_fwdllm.md` §A/§G/§H/§K, `PARITY_LOGICAL_TASKS.md` checkpoint, - `fluxtune_contributions.md` §5 — current status (sync sim_rate 2.9-3.0, K-D31/P2-7a validated, bin-7 float- - nondeterminism wall, fluxtune #15 = pipelining). - -## 7. RELATED OPEN ITEMS (context, not part of #15) -- **bin-7 float-nondeterminism wall (sync):** exact cadence parity unattainable past ~bin 6 (GPU fp16 grad jitter - amplified by split-half variance); relax `cohort_sequence` EXACT to `--max-bin 1` + distributional rung beyond. - Confirm with a 2-real-run P0-2 diff. (simulate_fwdllm.md open issue.) -- **#1d fluxtune cohort SET (thin-margin overrun):** `jvp_perf_opt` put GPU MEAN (3.61s) under the 4.0s budget but - the TAIL (4-5.4s) overruns on the 2 doubled GPUs. The #15 fix (keep compute near the ~2.4s uncontended floor via - pipelining) should also lift this. diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md index 945cd5552..698db2b41 100644 --- a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -29,6 +29,82 @@ open roots: **NEXT: #15 — decouple real-GPU dispatch from the sct-ordered commit drain** (keep GPUs full like real; drain orders commits by sct for the vclock only). This is the one thing keeping fluxtune `sim_rate<1`, and it also lifts #1d. +--- + +## ⭐ #15 fluxtune `sim_rate=0.50` — ROOT CAUSE FOUND (resume implementation here) + +Banked evidence run: `experiments/run_20260705_204619_fluxtune_n10_smoke_syn_0_sim` (+ `_real` pair +`run_20260705_202448`). Primary drain diagnostic = the `[SIM_GRAD_RECV]` log line +(`fwdllm_aggregator.py:926`, format `end= sct= T_v= buf_depth= inflight_exp= sel_ends=`). + +**ROOT CAUSE (evidence-backed): a circular wait between the drain's `earlier_stuck` gate and hold-to-commit.** +The drain (`_sim_recv_min_grad`, `fwdllm_aggregator.py:752`) blocks REAL WALL to keep sct-ordered commits: it won't +commit a buffered grad while an in-flight trainer has a smaller **expected** sct (`_sim_inflight_expected[end]`). But +that trainer is itself blocked in `recv_wrapper` (hold-to-commit: `_release_end_on_return:1136` returns early on the +residence path, so its slot frees only when its PREVIOUS grad commits) → it is NOT computing → its grad never comes → +30s `RECV_TIMEOUT_WAIT_S` failsafe evict. Circular. Steady state (startup is fine — 10 dispatched fresh → parallel; +collapses at the first commit when trainers start getting held). +``` +hold-to-commit blocks trainers in recv → gate waits real-wall for NON-COMPUTING in-flight trainers + → circular stall (30s failsafe) → ~1974s (81% of 2425s wall) burned → GPU 1.54x (real 3.37x) → sim_rate 0.50 +``` + +**EVIDENCE:** per-commit real 4.30s wall vs sim 6.60s wall / 3.27s vclock (sim_rate 0.50). GPU concurrency real +3.37x (98% busy) / sim 1.54x (85%); same GPU work (~3.7-4.0k trainer-s), same ~480s 8-way floor, `max=10` at +startup (HW sustains 10-wide). Trainer 371 `recv_wrapper` mean **30.9s** sim (max 55) vs **0.01s** median real; +compute mode-invariant (~3-4s, `_emulate_training_delay=0.000s`). Smoking gun: `[SIM_GRAD_STUCK_EVICT] end=0379 +exp=24.0 bmin=27.0` — drain blocked ~30s for 379 while **6 grads sat ready**; 379's telemetry: blocked in +`recv_wrapper` **60.39s**, got weights the instant after evict. `buf_depth` **constantly 6**; **524/1093 commits +(48%) stall >2s = 1974s**. Committed scts already go out of order (11,13,7,5.4,5.5) → the strict order the gate +blocks for isn't even preserved. Eval is once/data-bin (23/24), not per-iter, but 8.5s BLOCKING (~195s, secondary). + +**PROPER FIX — re-dispatch on RETURN** (match real, recv 0.01s), decoupling physical GPU pipelining from the +virtual in-flight ledger. Returned trainer gets next weights + computes immediately → never idle in recv → the +in-flight trainer the gate waits for produces its grad in ~4s IN PARALLEL with 7 others → drain commits a BATCH per +GPU-pass. Keep the gate + sct-order commits (preserve #1d cohort parity); keep the virtual in-flight COUNT / +selection eligibility (R1) held-to-commit — that's a SEPARATE ledger from physical compute (K-D17b wrongly welded +them). **Hard constraint (principle #16):** the compute-ahead grad must use the model version real dispatched +(fedbuff staleness `V'-V`) — real also cycles on return so versions come from the same deterministic commit order +(K-D29); VERIFY from telemetry, don't assume. **Belt-and-suspenders:** `_sim_inflight_expected[end]` is stamped at +DISPATCH (`:2819`, `_sst + _budget`, `_sst=self._vclock.now`) assuming immediate compute-start; a held trainer +hasn't started → expected sct is a fiction. Tie it to actual compute-START and/or bound the wait << 30s. + +**IMPLEMENTATION PLAN (resume tomorrow):** +- **P0 (verify, read-only):** diff per-trainer `MessageType.MODEL_VERSION` (grad's dispatch version) vs the commit + sequence, real vs sim (`inc_model_version_per_data_id=True`; staleness log ~`fwdllm_aggregator.py:1170`). Confirm + re-dispatch-on-return in sim yields the SAME dispatch-version sequence as real. If not, pin version to the + trainer's virtual completion (sct), not physical dispatch — revisit before coding. +- **P1 (core):** in `_release_end_on_return:1128` async-sim-residence branch, let PHYSICAL re-dispatch (weights send + → compute next grad) happen on return, while the SELECTOR's virtual in-flight set (`selected_ends`/`all_selected`, + driving R1 + `extra = c - inflight`) stays held to commit via `_sim_hold_busy_slots`. Re-dispatched grad enters + the sct buffer with its own `SIM_COMPLETION_TS`; drain keeps committing in sct order. Keep K-D27 two-ledger + discipline (`_sim_pending_commit`) — do NOT reintroduce the R1 regression. Fix the `_sim_inflight_expected` stamp. +- **P2 (tests+telemetry, same change):** `pytest tests/mode -k fwdllm` + `-k parity`; async_cifar10 byte-identical + (fwdllm_aggregator-only edit; `_sim_recv_min` untouched, principle #8/#9). Assert `recv_wrapper`→~0, concurrency + up; R1 must stay ~0 (bank it — K-D19: not done until smoke shows R1<=2%). +- **P3 (validate):** `run_sequential.sh --only fluxtune --mode both --delays on --delay-factor 1 --max-data-id 2`. + Expect `recv_wrapper`→~0, concurrency→~3.37x+, `sim_rate`→>1, no `[SIM_GRAD_STUCK_EVICT]`, and `cohort_sequence`/ + `var` parity UNCHANGED (if it moves, the P0 version assumption was wrong). +- **P4 (docs):** fold into `simulate_fwdllm.md` §G/§K (new K-D34, the two-ledger physical/virtual dispatch split); + update this checkpoint. + +**Repro (read-only, from `lib/python/examples/fwdllm`):** +```bash +FS=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_sim | head -1); AGG=$(ls "$FS"/*aggregator.log|head -1) +grep "SIM_GRAD_RECV" "$AGG" | python3 -c "import sys;from datetime import datetime as D;p=None;b=w=t=0 +for l in sys.stdin: + s=D.strptime(l.split(' | ')[0],'%Y-%m-%d %H:%M:%S,%f').timestamp() + if p is not None: + d=s-p;t+=1 + if d>2:b+=1;w+=d + p=s +print(f'commits={t+1} gaps>2s={b} wall_in_waits={w:.0f}s')" # -> 48% / ~1974s +grep "SIM_GRAD_STUCK_EVICT" "$AGG" # the head-of-line evict +grep "SIM_GRAD_RECV" "$AGG" | grep -oE 'buf_depth=[0-9]+' | sort | uniq -c # buf_depth stuck at 6 +``` + +--- + **DONE (landed + tested):** - **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both new rungs correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus From f9912b61ac02bd6abef0ca63f6e4ad03a5b3f421 Mon Sep 17 00:00:00 2001 From: Dhruv Garg Date: Mon, 6 Jul 2026 10:57:53 -0400 Subject: [PATCH 33/56] fwdllm #15: compute-truthful commit gate (fluxtune sim_rate stall fix) + docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root-cause fluxtune sim_rate=0.50 as a COMMIT-PATH STALL, not a residence/ pipelining/GPU bug (D1/D2 confirmed on the banked pair). The _sim_recv_min_grad earlier_stuck gate blocks real wall on PHANTOM _sim_inflight_expected entries -- trainers stamped expected-at-dispatch but idle-in-recv (not computing) behind the single-threaded drain -- burning ~1974s (82% of wall) to the grace floor/30s failsafe, which (via correct hold-to-commit) starves re-dispatch: concurrent compute sim 1.65 vs real 7.69. Fix (sim_compute_truthful_gate, flag-gated, default off = byte-identical; fwdllm_aggregator-only -> async_cifar10 untouched): - stamp _sim_dispatch_wall[end] at the real channel.send - earlier_stuck gate skips any expected entry whose last dispatch is older than sim_gate_compute_cap_s (default 10s > ~5.6s max JVP compute) or never dispatched, so a stamped-but-idle phantom no longer blocks a ready commit; a genuine in-window straggler is STILL held (sct commit order preserved) - hold-to-commit, sct-ordered drain, K-D12 carried surplus, K-D27 all UNTOUCHED - [SIM_GRAD_RECV] emits phantom_skip=; enabled in fluxtune_n10_smoke_sim.yaml Tests: 4 new TestComputeTruthfulGate (phantom skipped, stale=phantom, in-window straggler still held, flag-off byte-identical) + 189 fwdllm mode tests green. Docs: PARITY_LOGICAL_TASKS.md condensed + FELIX GROUNDING (F1-F7) reference; simulate_fwdllm.md K-D34 + §H dead-ends + §B.1 row 6. P3 validation run pending (operator; needs GPU env). Co-Authored-By: Claude Opus 4.8 --- .../examples/fwdllm/PARITY_LOGICAL_TASKS.md | 490 +++++++----------- .../expt_scripts/fluxtune_n10_smoke_sim.yaml | 12 + lib/python/examples/fwdllm/simulate_fwdllm.md | 69 ++- .../horizontal/syncfl/fwdllm_aggregator.py | 46 +- .../tests/mode/test_fwdllm_sim_grad_loop.py | 74 +++ 5 files changed, 363 insertions(+), 328 deletions(-) diff --git a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md index 698db2b41..b9732afd7 100644 --- a/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md +++ b/lib/python/examples/fwdllm/PARITY_LOGICAL_TASKS.md @@ -1,336 +1,206 @@ # TEMP task tracker — logical real↔sim parity (delete when folded into simulate_fwdllm.md §G) -**Goal.** Prove the sim takes the SAME logical steps in the SAME order as real (same cohorts, same receive -order, same variance cadence, same grads) up to data bin 1 across fwdllm / fwdllm_plus / fluxtune — and make the -parity CHECKS + PYTESTS actually *exhibit* these properties (they don't today). Only then chase the time -dimension. Correctness before speed; no hacks (simulate_fwdllm.md principles #14/#16). +**Goal.** Prove the sim takes the SAME logical steps in the SAME order as real (same cohorts, receive order, variance +cadence, grads) up to data bin 1 across fwdllm / fwdllm_plus / fluxtune, and make the parity CHECKS + PYTESTS exhibit +these properties. Correctness before speed; no hacks (simulate_fwdllm.md principles #14/#16). --- -## ⏸ SESSION CHECKPOINT (2026-07-05 — P2-7a VALIDATED + full-run roots nailed; resume at #15) +## ⏸ RESUME HERE (2026-07-06) — #15 fix LANDED, P3 run is the gate -**P2-7a (K-D31) VALIDATED and full runs analyzed.** Databin1 checks (`--max-bin 1`): sync `cohort_sequence` -ok=true, set/order/var/cadence=1.0 — the delay-tie is closed. Full runs (`--delay-factor 1`) then settled BOTH -open roots: -- **SYNC:** `sim_rate` 2.9–3.0 (#12c resolved). But receive-ORDER now 41/41 identical yet cadence still breaks at - **bin 7** → the "order→var→RNG-desync" root is REFUTED for the full run. Real root = **grad non-reproducibility - given matched order** (~1e-3 GPU fp16 jitter amplified by the split-half variance → gate flip at (7,2)). This - **answers P0-2**: exact cadence parity is unattainable past ~bin 6. ⇒ relax the target: `cohort_sequence` EXACT - scoped to `--max-bin 1`; add a DISTRIBUTIONAL cadence/var rung (mean-band + KS + `var_good` fraction) for the full - run. (Confirm with a 2-real-run diff before landing — strong single-run evidence already.) -- **fluxtune:** `sim_rate=0.50` is a **GPU-PIPELINING loss (#15)**, not #12c and not the commit gating (which is - correct, per-grad). Objective telemetry: sim 1.54× GPU concurrency vs real 3.37×; per-commit real 4.30s wall, sim - 6.60s wall / 3.27s vclock. The overrun (#1d, `set_match=3/272`) is a thin-margin contention tail (10 trainers / - 8 GPUs doubling + aggregator eval GPU) — `jvp_perf_opt` already put the MEAN (3.61s) under the 4.0s budget. -- **Pinning is clean** (earlier "under-provisioned" read was a misread of `gpu=4.9s` compute-time as device IDs): - 8 GPUs, balanced round-robin, CPU-pinned. Aggregator GPU pin + trainer `[PIN]` self-report + `[LOAD_BALANCE]` - check landed (K-D33, 120 launch tests green). +**#15 fluxtune `sim_rate=0.50` fix (`sim_compute_truthful_gate`) is CODE-COMPLETE + unit-green; the P3 validation run +is the only remaining step and needs the GPU + h5py env (operator runs it).** Everything else below is either DONE or +a lower-priority open task. -**NEXT: #15 — decouple real-GPU dispatch from the sct-ordered commit drain** (keep GPUs full like real; drain orders -commits by sct for the vclock only). This is the one thing keeping fluxtune `sim_rate<1`, and it also lifts #1d. +### ▶ IMMEDIATE NEXT COMMAND (P3 — validate the #15 fix) +```bash +cd lib/python/examples/fwdllm/expt_scripts +bash run_sequential.sh --only fluxtune --mode both --delays on --delay-factor 1 --max-data-id 2 --yes +python run_parity.py --yes --max-bin 1 --baselines fluxtune +# then inspect the new sim agg log (see "Repro" at the end of the #15 section): +# FS=$(ls -td ../experiments/run_*_fluxtune_n10_smoke_syn_0_sim | head -1); AGG=$(ls "$FS"/*aggregator.log|head -1) +# grep -c SIM_GRAD_STUCK_EVICT "$AGG" # expect ~0 +# grep SIM_GRAD_RECV "$AGG" | grep -oE 'phantom_skip=[0-9]+' | tail -1 # expect > 0 (guard firing) +``` +**EXPECT (PASS):** no/near-zero `[SIM_GRAD_STUCK_EVICT]`, `phantom_skip` rising, the `gap>2s` wall (~1974s baseline) +collapsing, concurrency↑, **`sim_rate` → >1** — AND `cohort_sequence`/`var`/`staleness` parity **UNCHANGED** vs the +flag-off banked run. **If parity MOVES:** the compute cap is too tight (skipping a genuine straggler) → raise +`sim_gate_compute_cap_s` (yaml, default 10.0) and re-run; if it still moves, the guard is reordering commits → +revisit (the fix must only remove dead wall, not change WHICH grad commits WHEN). + +### Other OPEN fronts (after P3), priority order +1. **bin-7 nondeterminism → relax the parity target (SHARED, correctness-of-CHECK).** SYNC full-run cadence breaks at + bin 7 even with order 41/41 matched (K-D31): ~1e-3 GPU fp16 grad jitter, amplified by the split-half variance + ratio, flips the `var<0.3` gate at (7,2) — NOT a sim bug. **Task:** keep `cohort_sequence` EXACT but scoped to + `--max-bin 1`; ADD a DISTRIBUTIONAL cadence/var rung (mean-band + KS + `var_good` fraction) for the full run. + Confirm with a 2-real-run diff first (= **P0-2**, below). +2. **P0-2 — grad-determinism-given-order confirmation.** Force identical commit order in a sim + a real short run (or + two real runs), diff per-iteration `var` + a grad norm. Decides exact (sim-bug) vs distributional (nondeterminism) + target beyond bin 1. Also unblocks the P1-6 LIVE sim==real grad-determinism pytest (still TODO). +3. **#1d / P2-5 — fluxtune cohort SET diverges (thin-margin overrun), fluxtune only.** `set_match=3/272`. `jvp_perf_opt` + (K-D32) cut GPU mean 7.57→3.61s (<4.0s budget) but the TAIL (4.1–5.4s) overruns on the two doubled GPUs (10/8) → + order flips → wrong 3-of-K commit. **Task:** once #15's faster commits land, re-check; if a residual tail remains, + tune `perturbation_count`↓ (P2-5, a knob already wired, default 10; LOWERING changes the baseline algorithm → operator + call) so GPU < min cohort D, and/or `--delay-factor` up. +4. **#7 — fwdllm_plus real ~4× slower/round; at syn_0 real sees ~4.9 eligible vs sim ~9.6.** Not a sim bug. Profile the + per-iteration reselection + oracular-read cost from the banked per-phase log; explain the eligible-count gap. +5. **#11 — real-mode critical-path waste** (`sleep(0.1)` MQTT-settle, one-grad-per-poll drain tail), real-only, ZERO + parity impact (sim already skips). Deferred to a validated pass (removing it needs a real run; principle #8/#11c). +6. **Phase 3 — extend beyond bin 1** once bin-1 parity holds; re-check where it breaks; iterate. Then C1/C2 + convergence (distributional target) at matched `data_id` → gate to Phase 2 (unavailability). + +### D3 (optional, secondary) — N=20/C=10/K=3 run +Adds a 10-trainer idle pool; confirms real fills C from fresh idle trainers while a returner waits for its commit +(validates F5) and the sim reproduces the selection sequence. `run_sequential.sh --only fluxtune --mode both`, +`num_trainers=20`, selector `c: 10`, `agg_goal: 3`. --- -## ⭐ #15 fluxtune `sim_rate=0.50` — ROOT CAUSE FOUND (resume implementation here) - -Banked evidence run: `experiments/run_20260705_204619_fluxtune_n10_smoke_syn_0_sim` (+ `_real` pair -`run_20260705_202448`). Primary drain diagnostic = the `[SIM_GRAD_RECV]` log line -(`fwdllm_aggregator.py:926`, format `end= sct= T_v= buf_depth= inflight_exp= sel_ends=`). - -**ROOT CAUSE (evidence-backed): a circular wait between the drain's `earlier_stuck` gate and hold-to-commit.** -The drain (`_sim_recv_min_grad`, `fwdllm_aggregator.py:752`) blocks REAL WALL to keep sct-ordered commits: it won't -commit a buffered grad while an in-flight trainer has a smaller **expected** sct (`_sim_inflight_expected[end]`). But -that trainer is itself blocked in `recv_wrapper` (hold-to-commit: `_release_end_on_return:1136` returns early on the -residence path, so its slot frees only when its PREVIOUS grad commits) → it is NOT computing → its grad never comes → -30s `RECV_TIMEOUT_WAIT_S` failsafe evict. Circular. Steady state (startup is fine — 10 dispatched fresh → parallel; -collapses at the first commit when trainers start getting held). -``` -hold-to-commit blocks trainers in recv → gate waits real-wall for NON-COMPUTING in-flight trainers - → circular stall (30s failsafe) → ~1974s (81% of 2425s wall) burned → GPU 1.54x (real 3.37x) → sim_rate 0.50 -``` - -**EVIDENCE:** per-commit real 4.30s wall vs sim 6.60s wall / 3.27s vclock (sim_rate 0.50). GPU concurrency real -3.37x (98% busy) / sim 1.54x (85%); same GPU work (~3.7-4.0k trainer-s), same ~480s 8-way floor, `max=10` at -startup (HW sustains 10-wide). Trainer 371 `recv_wrapper` mean **30.9s** sim (max 55) vs **0.01s** median real; -compute mode-invariant (~3-4s, `_emulate_training_delay=0.000s`). Smoking gun: `[SIM_GRAD_STUCK_EVICT] end=0379 -exp=24.0 bmin=27.0` — drain blocked ~30s for 379 while **6 grads sat ready**; 379's telemetry: blocked in -`recv_wrapper` **60.39s**, got weights the instant after evict. `buf_depth` **constantly 6**; **524/1093 commits -(48%) stall >2s = 1974s**. Committed scts already go out of order (11,13,7,5.4,5.5) → the strict order the gate -blocks for isn't even preserved. Eval is once/data-bin (23/24), not per-iter, but 8.5s BLOCKING (~195s, secondary). +## 📚 FELIX GROUNDING (async_cifar10) — the reference model for #15 (durable; keep) + +*How async_cifar10's felix drain ACTUALLY behaves (read from code + `async_cifar10/PARITY.md`). Anchors: **asyncfl** = +`flame/mode/horizontal/asyncfl/top_aggregator.py`; **fwdllm** = `flame/mode/horizontal/syncfl/fwdllm_aggregator.py`.* + +- **F1 — Felix's expected-sct arrival gate is INERT (gate_holds=0), not load-bearing.** It DOES stamp + `_sim_inflight_expected[end]=dispatch_vclock+budget` (asyncfl:1668-71, a LOWER BOUND) and CAN block (`earlier_stuck`, + asyncfl:430-446), but over a cifar run `gate_holds=0` (asyncfl:157-162; PARITY.md:294-296/321-323): cifar GPU ≈ + **0.4s**, so every in-flight update is already arrived+buffered when the drain runs. Felix's EFFECTIVE behavior = + sort arrived updates by sct and commit. The gate is a dormant safety net (min-budget lower bound + clock-jump clamp). +- **F2 — Felix re-dispatches on COMMIT; the felix trainer idles in recv — but sub-second, only because compute≈0.** + `_sim_hold_busy_slots` (asyncfl:1453-1500) at the agg-goal boundary holds each busy trainer in + `selected_ends`/`all_selected`/`_sim_pending_commit` until its update commits (released asyncfl:618-627); the trainer + blocks in `_fetch_weights`→`channel.recv` (`syncfl/trainer.py:187`) meanwhile. So felix IS hold-to-commit — + **fwdllm's K-D17b is a FAITHFUL port.** Precondition: return→commit is sub-second in cifar → ≈0 idle wall. +- **F3 — Two ledgers already SEPARATE in felix.** PHYSICAL (GPU/MQTT, `_sim_buffer`, `cleanup_recvd_ends`) vs VIRTUAL + (selection eligibility `_sim_pending_commit`/`all_selected`/`selected_ends`, staleness gate `_sim_inflight_expected`, + vclock). "Busy ≠ UN_AVL": a busy trainer holds a SLOT (`extra = c − len(selected_ends)`), released on commit + (asyncfl:1459-62; PARITY.md:324-328). +- **F4 — Regime table (why the same mechanism transfers to sync fwdllm but breaks fluxtune):** + + | baseline | GPU | agg_goal vs c | return→commit idle | gate | outcome | + |---|---|---|---|---|---| + | cifar felix | ~0.4s | (varies) | sub-second | inert | sim_rate ≫ 1 | + | fwdllm/plus (sync) | ~1.0s | K=c (barrier) | ~1 GPU pass, ALL commit | inert | `sim_rate` 2.9-3.0 ✓ | + | **fluxtune (async)** | **~4s** | **3 ≪ 10** | **multi-sec → 30s** | **load-bearing → stall** | **`sim_rate` 0.50 ⛔** | + +- **F5 — Hold-to-commit is a CORRECTNESS CHECK, not over-restriction.** The aggregator marks a trainer busy/idle by + whether it RESPONDED to its task; a trainer is freed (re-selectable) ONLY once its returned update is + PROCESSED/COMMITTED. That enforces three guards: **(a)** never dispatch a version it already computed + (same-shard×same-version = wasted grad); **(b)** never dispatch while still computing; **(c)** never dispatch while + its returned update is uncommitted. Do NOT weaken it. Concurrency: with **N>C** a fast returner is held while fresh + idle trainers keep C busy; with **N=C** (fluxtune 10/10/3, NO idle pool) concurrency is agg-cadence-limited and + real's **3.37 is a duty cycle** (gpu/max(gpu,D) ≈ 4/12 ≈ 0.34 → N×0.34), NOT under-utilization. +- **F6 — The bug is purely the COMMIT PATH STALLING; commit RATE is the throughput lever.** Because trainers are freed + only on commit, commit rate = free rate = throughput. cifar's commit path is instant → held trainers barely idle; + fluxtune's STALLS (see #15) → held trainers idle 30s. Fix = fast/non-stalling commit path; the gate must only wait + on a genuinely-COMPUTING trainer, never a phantom. Hold-to-commit untouched. +- **F7 — Anchors.** Felix gate/HOLD asyncfl:359-448; clamp 461-476; `_sim_hold_busy_slots` 1453-1500; commit-release + 618-627; expected-sct seed 128-137/1668-1671; sct formula (trainer) `async_cifar10/trainer/pytorch/main.py:838-846`; + PARITY.md gate-inert 294-296/321-323, residence §3.resid, past-dating 537-563. -**PROPER FIX — re-dispatch on RETURN** (match real, recv 0.01s), decoupling physical GPU pipelining from the -virtual in-flight ledger. Returned trainer gets next weights + computes immediately → never idle in recv → the -in-flight trainer the gate waits for produces its grad in ~4s IN PARALLEL with 7 others → drain commits a BATCH per -GPU-pass. Keep the gate + sct-order commits (preserve #1d cohort parity); keep the virtual in-flight COUNT / -selection eligibility (R1) held-to-commit — that's a SEPARATE ledger from physical compute (K-D17b wrongly welded -them). **Hard constraint (principle #16):** the compute-ahead grad must use the model version real dispatched -(fedbuff staleness `V'-V`) — real also cycles on return so versions come from the same deterministic commit order -(K-D29); VERIFY from telemetry, don't assume. **Belt-and-suspenders:** `_sim_inflight_expected[end]` is stamped at -DISPATCH (`:2819`, `_sst + _budget`, `_sst=self._vclock.now`) assuming immediate compute-start; a held trainer -hasn't started → expected sct is a fiction. Tie it to actual compute-START and/or bound the wait << 30s. +--- -**IMPLEMENTATION PLAN (resume tomorrow):** -- **P0 (verify, read-only):** diff per-trainer `MessageType.MODEL_VERSION` (grad's dispatch version) vs the commit - sequence, real vs sim (`inc_model_version_per_data_id=True`; staleness log ~`fwdllm_aggregator.py:1170`). Confirm - re-dispatch-on-return in sim yields the SAME dispatch-version sequence as real. If not, pin version to the - trainer's virtual completion (sct), not physical dispatch — revisit before coding. -- **P1 (core):** in `_release_end_on_return:1128` async-sim-residence branch, let PHYSICAL re-dispatch (weights send - → compute next grad) happen on return, while the SELECTOR's virtual in-flight set (`selected_ends`/`all_selected`, - driving R1 + `extra = c - inflight`) stays held to commit via `_sim_hold_busy_slots`. Re-dispatched grad enters - the sct buffer with its own `SIM_COMPLETION_TS`; drain keeps committing in sct order. Keep K-D27 two-ledger - discipline (`_sim_pending_commit`) — do NOT reintroduce the R1 regression. Fix the `_sim_inflight_expected` stamp. -- **P2 (tests+telemetry, same change):** `pytest tests/mode -k fwdllm` + `-k parity`; async_cifar10 byte-identical - (fwdllm_aggregator-only edit; `_sim_recv_min` untouched, principle #8/#9). Assert `recv_wrapper`→~0, concurrency - up; R1 must stay ~0 (bank it — K-D19: not done until smoke shows R1<=2%). -- **P3 (validate):** `run_sequential.sh --only fluxtune --mode both --delays on --delay-factor 1 --max-data-id 2`. - Expect `recv_wrapper`→~0, concurrency→~3.37x+, `sim_rate`→>1, no `[SIM_GRAD_STUCK_EVICT]`, and `cohort_sequence`/ - `var` parity UNCHANGED (if it moves, the P0 version assumption was wrong). -- **P4 (docs):** fold into `simulate_fwdllm.md` §G/§K (new K-D34, the two-ledger physical/virtual dispatch split); - update this checkpoint. +## ⭐ #15 fluxtune `sim_rate=0.50` — COMMIT-PATH STALL (root confirmed; fix LANDED, P3 pending) + +**Superseded framings (do not revisit):** (1) "GPU-pipelining loss / decouple dispatch" — felix's gate is INERT (F1). +(2) "re-dispatch on return / hold-to-commit is over-restrictive" — hold-to-commit is a CORRECTNESS check (F5). The +final root is a commit-path stall. + +**ROOT (D1/D2 CONFIRMED on `run_20260705_204619` sim / `_202448` real):** hold-to-commit correctly frees a trainer +only on commit, so commit RATE sets throughput (F5/F6). fluxtune's `_sim_recv_min_grad` `earlier_stuck` gate blocks +real wall on a PHANTOM `_sim_inflight_expected` entry — a trainer stamped expected-at-DISPATCH that isn't computing +(idle-in-recv, waiting for weights the gate-blocked single-threaded aggregator can't send) → 30s failsafe → the +correctly-held trainers idle ~30s instead of ~one GPU pass → sim_rate 0.50. + +**EVIDENCE:** +- Gate holds ALREADY-ARRIVED grads: every top stall commits a grad that arrived (`[MSG_ARRIVAL]`) 20-23s before the + stall; `buf_depth=7` on 99.7% of commits (buffer pinned full). Awaited trainers provably `recv_wrapper`-IDLE + (0370 idle 13s, 0371 44.6s spanning the stall). +- Dominant tax: inter-commit gap mean 2.20s; 496 commits pay ~2s + 24 gaps ≥10s + 1×31s failsafe = **1974s = 82% of + the 2398s wall**. `sim_rate` 0.494. +- Closed self-throttle: arrival rate 0.459/s == commit rate 0.456/s. Concurrent compute **sim 1.65 vs real 7.69**. + `recv_wrapper` idle: sim mean 17.9s vs real median 0.01s. Compute mode-invariant (~3.6s). +- Mechanism = **(c1)** optimistic `exp` stamped at DISPATCH (`:2819`, contention-free lower-bound budget) + **(c3)** + `sim_staggered_redispatch=False` → `_sst=_round_now` (`:2813`) collapses fresh-cohort exp onto the round frontier so + it always looks earlier than buffered later-sct grads → `earlier_stuck` stays armed + **(c2)** single-threaded agg + BLOCKS in the `for _pass` loop (`:772`), can't run `_distribute_weights_async` → can't send weights to the trainer + it waits on → deadlock to grace/30s failsafe. `SIM_R1_DISPATCH=0` — hold-to-commit is clean, NOT implicated. +- Also: fluxtune's self-throttle is the variance-gating keep-training loop — after the initial `Sent 10 WEIGHTS`, + distributes are `0 WEIGHTS + 1–2 VAR=bad` (only 1–2 trainers told to keep training/cycle). Real frees on grad + PROCESS (recv 0.01s, non-residence path) → ~8 compute; sim defers to commit → the slow commit starves re-dispatch. + +**FIX — ✅ P1/P2 LANDED (`sim_compute_truthful_gate`, flag-gated, default off = byte-identical; fwdllm_aggregator-only +→ async_cifar10 untouched):** +- **P1:** `_sim_dispatch_wall[end]` stamped at the real `channel.send` (`fwdllm_aggregator.py:2836`). The + `earlier_stuck` gate (`:856`) skips any `_sim_inflight_expected` entry whose last dispatch is older than + `sim_gate_compute_cap_s` (default **10.0s** > ~5.6s max JVP compute) OR never dispatched → a stamped-but-idle phantom + no longer blocks a ready commit; a genuine in-window straggler is STILL held (sct order preserved). Hold-to-commit, + the sct-ordered drain, K-D12 carried surplus, K-D27 `_sim_pending_commit` — all UNTOUCHED. `[SIM_GRAD_RECV]` now + emits `phantom_skip=`. Enabled in `expt_scripts/fluxtune_n10_smoke_sim.yaml` (`sim_compute_truthful_gate: true`, + `sim_gate_compute_cap_s: 10.0`). +- **P2:** 4 new `TestComputeTruthfulGate` (phantom skipped; stale-dispatch=phantom; in-window straggler STILL held; + flag-off byte-identical) in `tests/mode/test_fwdllm_sim_grad_loop.py`; +189 fwdllm mode tests green. +- **P3 = the gate (RESUME HERE, command at top).** Preserve the committed LOGICAL trace (grads/versions/order/ + staleness = real; #1d/cohort parity) — the fix must only remove dead wall. **P4:** §G one-liner after P3 passes. **Repro (read-only, from `lib/python/examples/fwdllm`):** ```bash FS=$(ls -td experiments/run_*_fluxtune_n10_smoke_syn_0_sim | head -1); AGG=$(ls "$FS"/*aggregator.log|head -1) -grep "SIM_GRAD_RECV" "$AGG" | python3 -c "import sys;from datetime import datetime as D;p=None;b=w=t=0 +grep SIM_GRAD_RECV "$AGG" | python3 -c "import sys;from datetime import datetime as D;p=None;b=w=t=0 for l in sys.stdin: s=D.strptime(l.split(' | ')[0],'%Y-%m-%d %H:%M:%S,%f').timestamp() if p is not None: d=s-p;t+=1 if d>2:b+=1;w+=d p=s -print(f'commits={t+1} gaps>2s={b} wall_in_waits={w:.0f}s')" # -> 48% / ~1974s -grep "SIM_GRAD_STUCK_EVICT" "$AGG" # the head-of-line evict -grep "SIM_GRAD_RECV" "$AGG" | grep -oE 'buf_depth=[0-9]+' | sort | uniq -c # buf_depth stuck at 6 -``` - ---- - -**DONE (landed + tested):** -- **P1-1/P1-2/P1-3** — enforced `cohort_sequence` rung (EXACT, ungated), V2 mean-guard, `--max-bin` window. Both - new rungs correctly FAIL the banked pairs (were invisible). New enforced ref: fwdllm 41/13/21, fwdllm_plus - 36/17/21, fluxtune 35/19/19. -- **P1-5 / P1-4** — full-cohort determinism gate (un-gates fwdllm selection/aggregation_sequence/utility; - fluxtune/fwdllm_plus stay gated); `timing_overrun` DIAG signal rung added; P1-4 assessed redundant. See the - DEFERRED list below for the full rationale. +9 pytests; banked scoreboard stable (only a new SKIP for the - overrun rung, which predates the banked logs). -- **P1-6 (partial)** — 11 rung/guard pytests. *Still missing: a LIVE sim==real grad-determinism test (needs P0-2).* -- **P1-8** — banked logs re-run through the upgraded checker. -- **P2-1/P2-3/P2-5/P2-6** — remainder-wait delay model (K-D29: real sleeps `max(0,D−gpu)`, sct `max(gpu,D)`, - overrun telemetry), crc32 straggler disabled, `perturbation_count` knob (default 10), overrun watch. -- **P2-2 (config)** — factor=2 will be applied via the launch flag (below), not yet run. -- **P2-4 (partial)** — GPU profiled: fwdllm/plus ~1.0s, **fluxtune 7.57s** (JVP 20 passes). Optimization deferred. - -**NOT DONE / DEFERRED (pick up here) — in priority order:** -1. **✅ P1-4 / P1-5 RESOLVED (this session):** - - **P1-5 DONE** — data-driven **full-cohort determinism gate** (`_selection_is_deterministic`: - `num_chosen==num_candidates` in both modes, else the DETERMINISTIC_SELECTORS name rule as a legacy fallback). - Un-gates `selection`/`aggregation_sequence`/`utility` for fwdllm (syn_0, K=all → GENUINELY enforced, no - longer a trivial gated pass) while self-keeping fluxtune (agg_goal=3) + fwdllm_plus (#7 asymmetric eligible) - gated. **participation deliberately NOT un-gated** — it keys on `round` (constant for fwdllm's data_id axis) - so it degenerates to a mechanical KS=1.0; cohort_sequence is fwdllm's per-cycle enforcement. Banked scoreboard - unchanged (41/13, 36/17, 35/19) — the un-gated rungs pass genuinely; utility's fail is a pre-existing pooled - KS=0.45 (bin-8 desync), not new. - - **P1-4 ASSESSED REDUNDANT (not built)** — extending `cohort_sequence` to async_cifar10's per-commit shape - yields a GATED trivial pass (stochastic subset, no `var`, exact order unattainable by design); its set+order - are already covered there by `aggregation_sequence` (gated) + `inter_arrival_order` + `first_divergence`. - Building the adapter adds a muddying no-signal rung (principle #16). Revisit only if async_cifar10 ever needs - an exact-ordered cohort rung. - - **NEW SIGNAL — `timing_overrun` DIAG rung** (`checks.py`, wired + tested): surfaces the P2-6 - `training_overran` fraction per mode + earliest `(data_id, iter)` overrun — the K-D29 order-determinism tell - (gpu > modeled D → arrival order can flip → cohort/var break is a TIMING-MODEL limit, not a sim bug). SKIPs on - the banked logs (predate P2-6); will populate on the databin1 run and tell us WHY fluxtune breaks. -2. **✅ P2-7 DONE (databin1 run + `--max-bin 1` check).** Cascade verified — see P2-7 in PHASE 2. Sync RNG-desync - root broken (`var` bit-identical, `timing_overrun=0%`); fluxtune overruns 38% as predicted. -3. **✅ P2-7a CODE LANDED (K-D31; operator chose canonicalize-by-trainer_id) — PENDING VALIDATION RUN.** Trainer - stamps pure `D` (`MODELED_DELAY_S`, both modes); agg `_canonicalize_cohort_commit_order` sorts each cohort by - `(D, str(end))` before the telemetry snapshot + `aggregate()` (reorders `_per_agg_trainer_list` + trailing - grad/jvp slice in lockstep). No-op when delays off / already canonical. 428 mode + **9 new** - `test_fwdllm_commit_canon.py` + 115 async_cifar10 parity green; async byte-identical. **NEXT: the operator runs - the validation command below** → confirm sync `cohort_sequence` order_match→1.0 (var already bit-identical). - A dry `--only fwdllm,fwdllm_plus` attempt crashed on a **missing `h5py`** (environment, not code) → operator - will run in the correct env. -4. **✅ P2-4 GPU optimization — LANDED (K-D32, `jvp_perf_opt`, fluxtune-only, config-gated, bit-identical).** - Implemented in `calculate_jvp` (trainable_idx) + `tc_transformer_trainer_distribute._train_one_batch` - (skip 3 diagnostic passes + reuse winner JVP), threaded via main.py/fl_main.py, enabled in both fluxtune yamls - (default false in trainer_base). Startup `[JVP_PERF_OPT]` confirms 10/10 trainers True (aggregator's eval-only - trainer False, harmless). 13 pytests + 185 fwdllm mode + 115 parity green. NOT retained: vmap (fp32 FD - cancellation), fwd-AD (slower). **NEXT: the batch run** confirms fluxtune `timing_overrun`→0 + cohort set - recovers + real↔sim var parity at multi-bin. -5. **P2-5 (tune fluxtune `perturbation_count`)** — only if the bit-identical cuts above don't fully clear overrun. - Lower it so GPU < min cohort D. Changes the baseline algorithm (deferred, operator call). -5. **P0-2** — controlled grad-determinism-given-order confirmation (also unblocks the P1-6 live test). -6. **Phase 3** — extend beyond bin 1 once bin-1 parity holds. - -**IMMEDIATE NEXT COMMAND (P2-7a validation — confirm K-D31 closes the sync tie):** -``` -cd lib/python/examples/fwdllm/expt_scripts -# sync baselines only (fluxtune still overruns → deferred to P2-5): -bash run_sequential.sh --mode both --delays on --delay-factor 2 --max-data-id 1 \ - --max-runtime-s 600 --only fwdllm,fwdllm_plus --yes -python run_parity.py --yes --max-bin 1 --baselines fwdllm fwdllm_plus -# EXPECT: cohort_sequence order_match_frac → 1.0 (was 0.5 / 0.0); var still bit-identical. -# grep the agg log for [COMMIT_CANON] to see the tie reorder fire. +print(f'commits={t+1} gaps>2s={b} wall_in_waits={w:.0f}s')" # baseline sim: 48% / ~1974s -> expect near-0 +grep SIM_GRAD_STUCK_EVICT "$AGG"; grep SIM_GRAD_RECV "$AGG" | grep -oE 'phantom_skip=[0-9]+' | tail -1 ``` -(The dry attempt crashed on a missing `h5py` — environment, not code. Run in the env that has the fwdllm deps.) -Expected: fwdllm/fwdllm_plus (GPU≈1s < D/2 of 2–9s) → order deterministic → `cohort_sequence` improves; fluxtune -(GPU 7.57s > D/2) → `[TIMING_OVERRUN]` fires, order still flips → tune `perturbation_count` down next. - -**KEY LEARNINGS (durable):** -- The divergence root is **commit ORDER**, not nondeterminism: fwdllm var is a split-half stat over the - commit-ordered grad list → wrong order → wrong var → threshold flips → per-trainer RNG (seeded once, never - reset) desyncs → grads diverge ~1%. Grads ARE deterministic given matched order ⇒ exact parity is achievable. -- The order wasn't deterministic because D≈0.4s (÷10) ≪ GPU 1–1.7s → GPU-jitter-dominated. Fix = remainder-wait - `max(gpu,D)` + per-trainer D + D≫GPU (D/2) ⇒ order = D-order = deterministic. -- fwdllm is **forward-only** (FedFwd, no backprop); the GPU lever is `perturbation_count` (JVP passes), not a - backprop optimization. fluxtune's 20 passes (7.57s) is the acute cost. -- The old checker was **blind** to all this (cohort rung gated off, V2 KS-only). The new `cohort_sequence` + - V2-guard are the spec the model fix must satisfy. - ---- - -## ROOT DIAGNOSIS (from the 2026-07-05 investigation — supersedes "order is benign") - -**One root explains all three baselines: the sim's commit/receive ORDER ≠ real's actual arrival ORDER.** -- fwdllm's variance is a **split-half statistic over the commit-ORDERED grad list** (`fwdgrad_utils.py:133-158`, - appended in commit order at `fwdllm_aggregator.py:718-721`). Order matters even for an identical cohort SET. -- Sim commits in **sct order** (`_sim_recv_min_grad` / `_sync_sim_recv_first_k`); real commits in **physical - arrival order**. Different order → different split-half `var` → `var_good_enough` flips at a different - `iteration_per_data_id`. -- Each trainer's `torch.Generator` is seeded ONCE and **never reset** (`tc_transformer_trainer_distribute.py: - 222-225`); the perturbation for a given `(data_id, iter)` depends on how many prior forward passes that trainer - did. One extra iteration → every trainer's RNG desyncs from that point → all later grads differ (~1%). This is - the fwdllm bin-8 cadence break AND the fluxtune 0/17 cohort break — same cause. -- **fluxtune** (async, agg_goal=3 delay` handling. -- Sim sct adds a crc32 `_sim_straggler_offset_s` (spread 0.9s) + `_wan_s` that **real has no counterpart for** - (`FedSgdTrainer.py:550-565`, `:623-624`) → an extra sim-only reorder vs real. - -**Foundational question — likely ANSWERED by the code:** grads are deterministic GIVEN matched commit order -(batch selection is deterministic in `data_id`; seeds are mode-invariant; JVP is fixed). The ONLY divergence -sources are commit-order + retry-count, both downstream of order. ⇒ **it's a sim ORDER bug, not nondeterminism.** -Needs one controlled confirmation (P0-2). If confirmed, exact cadence parity IS achievable. - -**Checker is blind to all of this (Agent A):** `aggregation_sequence` (cohort set) is gated `ok=True` for -stochastic selectors (`DETERMINISTIC_SELECTORS=∅`, `checks.py:882`) and keys on `round` not `cycle_data_id`; -`inter_arrival_order` is WARN-only; `v2_var_trajectory` is **KS-only, no mean guard** (a 1% offset passes); no -`--max-bin`. Both divergences pass today. - ---- - -## PHASE 0 — settle decisions + the foundational question (BLOCKING) -- [x] **P0-1 Operator decisions — RESOLVED 2026-07-05** (see "Resolved decisions" below): remainder-wait delay; - D configurable, start at **D/2** (not full registry); **optimize GPU now** (prove forward < backward); - checker rungs **hard-FAIL**. New scope added: fluxtune JVP forward-pass-count knob + budget-overrun ordering - watch. -- [ ] **P0-2 Confirm grad-determinism-given-order.** Controlled check: force identical commit order in a sim and - a real short run (or two real runs) and diff per-iteration `var` + a grad norm. Expect bit-match if order - matches. Decides sim-bug (exact target) vs nondeterminism (distributional target). - -## PHASE 1 — make the CHECKS/PYTESTS exhibit the properties (TOP PRIORITY, test-driven) -*Build the failing check FIRST, confirm it catches the current divergence on banked logs, then fix the model.* -- [x] **P1-1 DONE — enforced `cohort_sequence` rung landed** (`parity/checks.py::cohort_sequence_parity`, tier - EXACT, ungated, keyed on the `_fwd_cadence_cycles` stream). Per cycle asserts cohort SET + receive-ORDER + - cadence tuple + `var` value (rel-tol 1e-3). Wired into `run_all_parity` + `CHECK_META` + the `run_parity.py` - headline. **Now FAILS all 3 banked pairs** (was invisible). -- [x] **P1-2 DONE — V2 mean-guard added** (`var_trajectory_parity`, `mean_tol_rel=0.02`). **Now FAILS all 3** - (KS-only used to pass the ~1% offset). -- [x] **P1-3 DONE — `--max-bin` window** threaded through `_fwd_cadence_cycles` → V1/V2/V3/V4/V5 + `cohort_sequence` - + `run_all_parity` + `run_parity.py` + `cli.py`. `run_parity.py --max-bin 1` verified. -- [x] **P1-4 ASSESSED REDUNDANT (not built).** Extending `cohort_sequence` to async_cifar10's per-commit shape is - a GATED trivial pass (stochastic subset, no `var`); async_cifar10's set+order are already covered by - `aggregation_sequence` + `inter_arrival_order` + `first_divergence`. No new signal → not built (principle #16). -- [x] **P1-5 DONE — data-driven full-cohort determinism gate** (`_selection_is_deterministic` / - `_full_cohort_selection` / `_has_cohort_counts`, `checks.py`). Un-gates `selection`/`aggregation_sequence`/ - `utility` when `num_chosen==num_candidates` in both modes (fwdllm syn_0) → GENUINELY enforced; fluxtune + - fwdllm_plus stay gated (subset / #7 asymmetric eligible); legacy no-count telemetry falls back to the old - selector-name rule (no regression). participation EXCLUDED (round-keyed → mechanical KS on fwdllm's data_id - axis). `decision_determinism` left DIAG (a localizer by design, not an enforce target). 6 gate pytests + - banked-log validation (scoreboard stable). NEW `timing_overrun` DIAG rung surfaces the P2-6 overrun signal. -- [~] **P1-6 PARTIAL — rung + guard pytests landed** (`TestCohortSequence` ×9, `TestVarTrajectoryMeanGuard` ×2 in - `tests/mode/test_parity_checks.py`). STILL TODO: a LIVE sim==real grad-determinism test given matched order - (needs P0-2 infra). -- [ ] **P1-7 Cross-example + cross-baseline coverage.** Confirm the rung runs on async_cifar10's per-commit shape - (P1-4). fwdllm_plus & fluxtune already covered by the banked-log run. -- [x] **P1-8 DONE — banked logs re-run through the upgraded stack.** New ENFORCED reference (full run): - **fwdllm 41/13/21, fwdllm_plus 36/17/21, fluxtune 35/19/19** — each now lists `cohort_sequence` + - `v2_var_trajectory` in FAILS. At **`--max-bin 1`**: cadence (V1) passes but `cohort_sequence` still fails — - fwdllm shows `order_match_frac=0.0`, `var_match_frac=0.5` (real `var=0.371605` vs sim `0.371067` at cycle 0), - proving the order→split-half-var→grad-desync mechanism. Regression: 337 mode/telemetry/selector +115 - async_cifar10 parity tests green. - -## PHASE 2 — fix the timing/order MODEL until the (now-failing) checks pass -- [x] **P2-1 DONE — remainder-wait delay model** (`FedSgdTrainer._emulate_training_delay(gpu_time_s)` → - `(modeled_delay, remaining, overran)`; real sleeps `max(0, delay−gpu)`, sim skips; `[TIMING_OVERRUN]` log). - sct is now `max(gpu, delay)` (not `gpu+delay`). Supersedes K-D2 → **K-D29**. Tests rewritten - (`test_fwdllm_trainer_sim_duration.py`), full suite green. -- [x] **P2-2 DONE (config) — factor=2 (D/2)** via launch flag `--delay-factor 2` (knob stays live; `training_delay_ - factor` per-config). Applied at launch below. -- [x] **P2-3 DONE — crc32 straggler offset disabled** (`sim_straggler_spread_s: 0.0` in all 3 sim yamls). The - per-trainer registry delays now supply the completion spread; the offset would re-noise the deterministic - order. `_wan_s` already 0. (Helper kept flag-gated for its unit tests.) -- [~] **P2-4 PARTIAL — GPU profiled + per-step telemetry LANDED.** Coarse (from `trainer_round`): fwdllm/ - fwdllm_plus **~1.0s** (cos-sim path), **fluxtune 7.57s mean** (JVP, 20 passes). NEW **per-step telemetry** - (`EVENT_STEP_TIMING` via `timer_decorator` — no-op unless telemetry on) + plotter - `expt_scripts/plot_step_timing.py` (fine `step_timing` funcs + coarse `trainer_round` phases; PNG + table). - **Finding from banked logs:** the dominant REAL wall is `mqtt_fetch_s` (fwdllm 4.2s/round, **fluxtune - 18.3s/round**) ≫ `gpu_compute_s` (1.1 / 3.6s) — that's the #11 real-transport cost the SIM already skips, not - a GPU lever. The FINE GPU sub-step split (perturbation-selection vs per-batch JVP) populates on the next run - (telemetry now in place). fwdllm is forward-ONLY → the GPU lever is `perturbation_count` (P2-5). -- [x] **P2-5 DONE (knob) — `perturbation_count` config knob** (default 10 = byte-identical) threaded - config→`main.py`/`fl_main.py`→`tc_transformer_trainer_distribute.py`, replacing the hardcoded `1*10` / - `range(0,10)` in all 4 sites. Lowering it cuts fluxtune's forward-pass cost. LEFT AT 10 for this run - (operator: fluxtune tuning is a next step). True per-baseline JVP cost measured (P2-4). -- [x] **P2-6 DONE — budget-overrun telemetry** (`training_overran` + `remaining_time_s` on `trainer_round`, - `[TIMING_OVERRUN]` warning). This is the "keep a tab" watch: if actual GPU > modeled sct, the update arrives - after the vclock passed its sct → out-of-order commit. Expect it to fire on fluxtune this run. -- [x] **P2-7 DONE — cascade VERIFIED on the databin1 run** (`--delay-factor 2 --max-data-id 1`, - `smoke_logs/20260705_150145` → `experiments/run_20260705_15*`; `run_parity.py --yes --max-bin 1`). - **RNG-desync root BROKEN for the sync baselines:** on bin 1 `cohort_sequence` shows `var` **bit-identical** - real↔sim (fwdllm `0.21480107…`, fwdllm_plus `0.37160512…`; banked had `var_match=0.5`, real≠sim) + - `set_match=1.0` + `cadence_match=1.0` + **`timing_overrun=0%`** ⇒ grads mode-invariant on bin 1. RESIDUAL: - `order_match` 0.5 (fwdllm) / 0.0 (fwdllm_plus) from ONE benign **delay-TIE** — trainers 3 & 9 (…372/…378) - both drew `training_delay_s=13.0` (`_metadata/trainer_registry.yaml`, → D=6.5); their sct ties, real breaks - it by physical arrival & sim by sct-sort, both in the SAME split-half ⇒ var/grads unchanged. **fluxtune:** - `timing_overrun=38%` (GPU≈4.2s > min cohort D 2.0/2.5/3.5s), `set_match=0.22`, `var` mean off 29% → genuine - break, timing-model-limited → P2-5 (`perturbation_count`↓ so GPU