FL simulator client-unavailability substrate + parity ladder (async_cifar10) + fwdllm sim design - #69
Merged
Merged
Conversation
…l spec
Rewrite UNAVAILABILITY_DESIGN.md to fold in the Jun25 design resolutions:
- v1 scope locked: aggregator reads the shared trace (oracular) for ALL
baselines, client_notify OFF. The old per-baseline oracle-vs-notify
asymmetry collapses; aware vs unaware now differ only in WHEN a stalled
slot is freed (aware: proactively at next selection boundary; unaware:
reactively at the 90s abandon deadline). True avl_* transport +
continuous mid-round scheduling deferred to Stage H.
- Withhold-then-deliver corrected to a send-time gate, not a mid-compute
interrupt: compute always completes, only the upload is gated, delivered
stale at delivery_ts=max(sct,next_avail_ts). Real-side send-gate change
documented; sim commits via the agg-side buffer.
- 90s abandon + withhold are both true = two ledgers (slot vs delivery);
the existing SEND/RECV_TIMEOUT_WAIT_S must re-clock to _vclock.now.
- Three correctness invariants (no double-count; never re-select a
still-down trainer; aware/unaware = trigger only).
- Library mixin seam: flame/availability/{trace.py,availability_mixin.py}
mixed into all four TopAggregators, consolidating the 3 duplicate
read_trainer_unavailability; spans async_cifar10 + fwdllm.
- New section 8: file-level v1 implementation spec (Stage A + C).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New files: - flame/availability/trace.py: load_trace / state_at / next_avail_after (single bisect_right resolver; lru_cache for YAML files; replaces three inlined bisect_right copies) - flame/availability/availability_mixin.py: AvailabilityMixin with _init_availability, _avail_now (vclock in sim), read_trainer_unavailability, get_curr_unavail_trainers, free_stalled_slot (dormant hook for Stage C/D/H) Deletions (consolidation): - read_trainer_unavailability: removed from main_oort_sync_agg.py, main_asyncfl_agg.py, fwdllm_aggregator.py (all three dup copies) - get_curr_unavail_trainers: removed from syncfl/top_aggregator.py body (wall-clock impl) and main_oort_sync_agg.py (wall-time-in-sim bug); mixin provides the single vclock-correct version Wiring: - AvailabilityMixin mixed into syncfl TopAggregator → all four stacks inherit - _init_availability called from internal_init; sim_unavailability=False (default) ⇒ trainer_event_dict=None ⇒ byte-identical to all existing runs - flame/config.py: sim_unavailability, availability_aware, availability_trace_dir fields - Supports both new sim_unavailability gate and legacy track_trainer_avail path 389/389 unit tests pass. Syn_0 regression smoke pending on training node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Land the reusable C.2/C.4/C.5 core (gated/dormant until the live commit loop calls it; default OFF => byte-identical): - AvailabilityMixin.compute_delivery_ts: earliest t>=sct with state AVL_* (immediate if already up at sct; inf if the trace never recovers). - free_stalled_slot activated: frees the selector slot ledger AND registers pending_withheld[end]=delivery_ts (two ledgers, Challenge 4 / invariant 1). - withheld_held_ends / ready_withheld / commit_withheld delivery-ledger helpers (ready_withheld orders by (delivery_ts, end_id) -> no past-dating). - invariant 2 wired: withheld_held_ends() unioned into the unavailable list in oort + asyncfl _distribute_weights so a held trainer stays out of the eligible pool until its delivery_ts. - asyncfl __init__: _sim_withheld_payload store added (consumed by C.2 next). 10 new unit tests (tests/availability/test_delivery_ledger.py); full suite 420/420. Detailed C.2/C.3 live-wiring resume plan added to UNAVAILABILITY_DESIGN.md (RESUME HERE subsection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y rungs
Wire the send-gate withhold / late re-commit (C.2) and the 90s vclock abandon
(C.3) into both sim commit loops, as ONE shared library-level core in
AvailabilityMixin so the asyncfl (felix/fedbuff) and oort (oort/refl) stacks ride
identical logic instead of forking it:
* _sim_withhold_if_unavail — per-update send-gate primitive (invariant-1 stash
for an already-abandoned end; else hold if state_at(sct)=UN_AVL: stash payload,
free_stalled_slot, register the delivery ledger). Pure per-update decision —
does not consult round_start, which is why oort applies its carry-over gate
first (Challenge 7).
* _sim_pop_committable / _sim_reinject_ready_withheld — asyncfl pop loop + due
re-injection at delivery_ts (ordered), slot-only ledger drop.
* _sim_abandon_stalled — vclock 90s abandon over the selector slot ledger;
_avail_free_slot_ledger / _avail_inflight_ends are robust to both selector
shapes (fedbuff dict-by-requester + all_selected; oort/refl/feddance flat set).
* _sim_take_withheld_delivering / _emit_withheld_delivery — late-stale-commit
telemetry; asyncfl tags the "withheld" past-dating bucket.
Call sites: asyncfl/_sim_recv_min pop site; oort/_sim_drain_buffer pop loop
(composed after the §4.9 carry-over gate); both _distribute_weights run the
abandon scan. All helpers are getattr-guarded ⇒ gate-off byte-identity holds for
bare aggregators that never ran _init_availability.
Run activation: trace-name normalization (avl_events_syn_20 → syn_20) in
flame/availability/trace.py — the legacy oort/refl JSON configs resolved to 0
traces / silent-OFF before. Confirmed end-to-end: 300 traces load, 34/300 UN_AVL
mid-window, compiled config carries enabled/type/trace.
Telemetry: withheld_delivery + abandon_timeout events/builders (events.py).
Parity rungs (checks.py, wired into run_all_parity + CHECK_META):
* withheld_delivery (DIAG) — structural: delivery_ts≥sct, delay_s≥0,
staleness≥0; reports delay/staleness dist + accept_frac.
* abandon_timeout (CONTROL) — fails loudly on a wall-clock leak.
* eligible_pool_reduction (DIAG) — candidates−eligible across modes.
Loaders surface the new agg events + trainer avail_change; duty_cycle reads the
real {old_state,new_state} format (A4 bug). observation_lag + A4b held (need run
data — see §7).
Tests: 436/436 lib (16 new in tests/availability/test_live_wiring.py) + 38 parity
(14 new in scripts/parity/test_availability_rungs.py).
Next: oort syn_20 smoke (sim+real, --runtime-s 1800 to clear the 600s down
window) before felix/fedbuff or Stage D — see UNAVAILABILITY_DESIGN.md NEXT ACTION.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ired C.6.4: add trainer_state_fractions_sorted.pdf (per-trainer UN_AVL sorted bar, A4dur visual companion) to analyze_run.py availability_plots. Syntax clean, crash-safe. Visual correctness needs a fresh syn_20 run. D.0: felix gate plumbing in parity yaml — sim_unavailability/availability_aware/ client_notify added to both felix entries so _init_availability activates for felix on syn_20 runs (client_notify.trace overridden by --trace flag). D.1: _sim_evict_unavail_inflight in AvailabilityMixin — proactive boundary eviction for availability_aware baselines: at each selection boundary, any in-flight trainer showing UN_AVL per the trace has its slot freed immediately (no 90s wait). Called from oort and asyncfl top_aggregators after _sim_abandon_stalled; guarded by _availability_aware so oort/refl (unaware) stay on the 90s-abandon path. build_abandon_timeout gains reason field to distinguish C.3 from D.1 events. 8 new unit tests; 444/444 total. Next: syn_0 byte-identity smoke (felix+oort) then felix syn_20 smoke to validate D.1 eviction fires and C.6.4 plots populate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…iners wait_all() waited up to 30s per straggler trainer sequentially, adding minutes of idle wait when several trainers got stuck post-aggregator-exit; now all trainers share one timeout window polled concurrently. Also add debug_run.sh --num-trainers to shrink the cohort below 300 for runs that need a real --runtime-s budget (e.g. a vclock floor for an availability trace) that the existing smoke mode's hardcoded rounds=4/runtime=240 would cut short. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… gating Lands the four implementation items from UNAVAILABILITY_DESIGN.md's Next actions: (1) real-side send-time gate (§8.3) — trainer task-start no longer skips on avl_state (compute always completes), and syncfl/trainer.py's existing send-time UN_AVL check is decoupled from client_notify["enabled"] (always False in v1) so it actually fires for real felix/oort; (2) fixes avail_composition/avl_state staying all-UNKNOWN by stamping PROP_AVL_STATE from the oracular trace onto every known end before each selection; (3) fixes withheld_delivery under-emission caused by the delivery ledger dropping a slot-only entry before its payload physically arrived; (4) D.2 AVL_TRAIN/ AVL_EVAL task-type eligibility gating in selection, wired into oort + asyncfl. 452/452 lib + 63/63 example-parity tests pass (was 444/444). Real/sim parity run validation still pending (see Next actions). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
D.2's AVL_TRAIN-exclusion-from-eval rule made eval's eligible pool permanently empty on a 2-state trace (syn_0/syn_20/syn_50 never produce AVL_EVAL), which corrupted the selector's shared in-flight tracking for the train task too (async_oort.py/fedbuff.py _handle_send_state's disconnect-cleanup loop wipes selected_ends when handed an empty pool). Found via a real felix syn_0 smoke that hung with 0 AGG_RECV_WEIGHTS despite all 48 trainers training and sending. _init_availability now computes _trace_has_avl_eval once (does this trace ever produce AVL_EVAL for any trainer); the eval-side exclusion in get_curr_task_ineligible_trainers only applies when true, matching the design doc's own "2-state collapses the split" statement. Train-side exclusion is left unguarded (no 3-state trace exists yet to exercise the symmetric risk) and documented as a residual risk in UNAVAILABILITY_DESIGN.md (Challenge 13). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
838 → 452 lines. Completed stages compressed (mechanism + where it lives + exit met). Updated status: D.2 hang fix CONFIRMED via Jun 28 syn_0 runs (felix+oort, both complete 4 FL rounds, all avail rungs PASS, abandon/withheld correctly SKIP). Removed run-unconfirmed tags now folded into "pending syn_20" which is in progress. Added D.2 empty-pool hang to §9 dead-ends. Dropped verbose per-item prose for landed stages; kept the invariants and land-mines that bite if forgotten. parity_felix.json/parity_oort.json from Jun 28 syn_0 regression runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rungs Three parity check issues found and fixed during syn_20 validation: 1. Raise NEAR_ZERO_LAG_S 0.05→0.10s (U6): carry-over burst after avail windows inflates sim mean to ~70ms — still immediate-commit semantically. 2. Add point-mass guard in trainer_phase_split: both means ≤5ms → skip KS, pass on mean (pre_train_s, weights_to_gpu_s, weights_to_ram_s near-zero). 3. Wire four new avail rungs into report.py _SECTIONS: A4dur, Aa (eligible_pool_reduction), C.3 (abandon_timeout), C.2 (withheld_delivery). Fix abandon_timeout tier CONTROL→INV so _TIER_TAG renders correctly. UNAVAILABILITY_DESIGN.md: add §9.1 (oort parity settled diagnosis — K3b run-length, T2 pre-existing, A2 KS shape artifact from avail-window bimodal distribution; none block Stage E). Reorder Next Actions: Stage E first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Collapse remaining v1 work (E feddance, F starvation, G integration) into one code batch + one long-run batch. Non-conflicting code surfaces land together, gated on cheap signals (unit tests + syn_0 byte-identity + syn_20 feddance / syn_50 smokes). Share the F-validation syn_50 run with G.2's ramp rung; overlap the oort 3600s confirmation for free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stage E (syncfl path — feddance + refl): - syncfl _distribute_weights: abandon/evict/stamp + scarcity-advance (calls _next_avail_vclock() and advances vclock instead of sleeping when no trainers selectable at round start) - syncfl _sync_sim_recv_first_k: withhold send-gate (_sim_withhold_if_unavail) + withheld bonus drain (_sim_reinject_ready_withheld + _emit_withheld_delivery) - syncfl internal_init: _sim_buffer (SimReorderBuffer) + _sim_committed init - E.2 = accept-stale (FedAvg has no staleness gate); E.3 = naturally handled (withheld updates excluded from committed, so U6 is over actual cohort) Stage F (starvation vclock-advance): - New _next_avail_vclock() mixin helper: min across all trace transitions + pending_withheld delivery timestamps; returns None when gate off - Wired into: (a) syncfl if-not-selected_ends path, (b) oort max_retries loop - Felix asyncfl (top_aggregator.py:639) gap noted; deferred to syn_50 observation G.1 (parity rungs): - starvation_advance_parity() in checks.py: DIAG rung, jump_factor=5x mean gap, gate = withheld_deliveries or abandon_timeouts or avail_composition subfield - Dispatched in Stage 2 Availability results dict - Added to report.py _SECTIONS section "2" as "Fst starvation_advance (diag)" - withheld_delivery deps chain updated to include abandon_timeout - 4 new starvation_advance unit tests; 67/67 parity tests pass (was 63) Felix master-gate (prereq): - debug_run.sh trace-override block now injects simUnavailability: True (+ availabilityAware: True if client_notify.enabled) for non-ORACULAR baselines when --trace syn_X is passed; oort/refl still use legacy path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Terminology overhaul to eliminate the dual meaning of "ORACULAR": - "ORACULAR" in YAML/code now explicitly labelled as the legacy-gate config mechanism (field name in oort/refl YAML), not a knowledge model descriptor - v1 knowledge model renamed "trace-read" (agg reads trainer_event_dict directly) — applies to ALL baselines, including feddance - "aware/unaware" split renamed "proactive/reactive-90s" for slot-free timing (proactive = felix at next boundary; reactive-90s = oort/refl/feddance at 90s vclock deadline) - Stage H transport renamed "message-transport" (avl_* msgs) vs "trace-read" - Config-gate section now defines legacy-gate (oort/refl) vs simUnavail-gate (felix/feddance) and explains they activate the same trace-read code path - §0 existing paths table updated: path A = "trace-read", path C = "message-transport" - §8.4 config surface updated with mapping table Stage E smoke rationale added: feddance-only is sufficient because refl shares the same syncfl _distribute_weights/_sync_sim_recv_first_k code and its legacy-gate was already confirmed in Stage C (oort syn_20 run). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
oort n=300 3600s syn_20 results (Jun 28 Run 2: 40/48 PASS): - T2 training_budget: CONFIRMED self-corrected (ratio=1.133 ≤ 1.15) ✅ - K3b overhead_residual: did NOT self-correct as predicted (still rel=0.116, now gated downstream by P3); prediction revised - A2 eligibility: improving (KS 0.437→0.338) but still shape artifact; need more rounds (Batch 2 long run) - P3 trainer_speed: NEW marginal root cause at n=300 (ratio=1.153 vs tol 1.15, 1.3% over); clean at n=48; likely speed-tail noise at full cohort - Fst starvation_advance: PASS "no starvation advances detected" — correct (n=300 + syn_20 always leaves ~240 trainers available, Stage F never fires) G.4 added to Stage G: consolidate config-gate paths + rename legacy function `oracular_trainer_avail_check` → `_trace_read_avail_check` + update comments/logs to use new terminology (trace-read, proactive, reactive-90s, legacy-gate, simUnavail-gate). Deferred to after Batch 2 long runs pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…in progress feddance syn_20 1800s parity check: 46/47 PASS. All availability and mechanism rungs pass (A1–A4, U3/U6/K8/U2). Sole fail = C2 loss (0.1696 vs 0.15, 2 eval pts, early-train noise at alpha=0.1/syn_20 — not a mechanism gap). Challenge 9 confirmed: no K8/U2 movement from withheld stale commits. Stage E exit met. Stage F syn_50 runs launched (feddance + oort, 45 min each, real+sim). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
F.2 replaces the broken per-aggregator retry mechanisms with a single pre-selection return-early pattern across all three aggregators: - oort: was max_retries=5 + min_required=5 (50% of agg_goal) + "proceed anyway" fallback + 2s blocking sleep. Now: threshold=desired_selection (=int(aggr_num×overcommitment)=13), unbounded, non-blocking (0.5s + return). - syncfl (feddance/refl): was post-selection `if not selected_ends:` which FedDanceSelector's partial returns (3/10 eligible) never triggered. Now: pre-selection threshold=agg_goal=10. - asyncfl (felix/fedbuff): was time.sleep(0.5) in both modes at no-recv-ends path. Now: sim advances _next_avail_vclock(); real keeps 0.5s sleep. All three use the same pattern: num_eligible < threshold → sim vclock-advance + re-stamp + return; real: 0.5s sleep + return. Outer run() loop retries non-blocking. No max_retries ceiling; no "proceed anyway" fallback. At n=10 syn_50: min_avail≈3 at t≈1200s → eligible=3 < desired_selection=13 (oort) and < agg_goal=10 (feddance) → both starvation paths now exercisable. UNAVAILABILITY_DESIGN.md: 746 → 248 lines. Completed stages compressed to 2-3 lines each. Full detail kept only for active (Stage F exit) and next (Batch 2). Known parity failures moved to a single reference table (§7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…progress syn_0 results (n=48): Fst PASS (no starvation fires), C1/C2 diff=0.0, K1/K5 PASS on both baselines. Oort 43/50 (K3b pre-existing gates K2/K3); feddance 46/48 (A2 shape artifact + gpu_compute short-run noise, K2/K3/P3 PASS). F.2 threshold check confirmed not firing spuriously at n=48 with syn_0. n=10 syn_50 starvation smoke launched; results to check in morning. Next actions updated to morning check commands. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
debug_run.sh: simUnavailability was never injected for feddance (parity config has tracking_mode:client_notify at aggregator level but no HP-level tracking block, so neither the trackTrainerAvail nor client_notify branch fired). Result: trainer_event_dict=None → vclock deadlock at 0.0 for full 5400s. Fix: add third elif branch for non-oracular baselines with no HP tracking block, injecting availability_trace + simUnavailability=True. Also add simUnavailability to the client_notify-in-HP branch (felix path). runner.py: aggregator exit code was not logged, making silent crashes (e.g. feddance real crash at 12s) undiagnosable. Log exit code after wait(); also set PYTHONFAULTHANDLER=1 in child env so C-level crashes (segfault in paho/torch) write a traceback to _aggregator.log. UNAVAILABILITY_DESIGN.md: update status (n=10 bugs root-caused), replace next-actions with n=25 1800s run command + corrected rationale. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…untime_s Real-mode oort aggregator could block indefinitely in recv_fifo when syn_50 trainers went unavailable mid-round: no timeout meant max_runtime_s check in inc_round() was never reached, causing the process to run until manually killed. Fix: add trainer_recv_wall_timeout_s HP (default 90s, YAML-configurable) as a per-message wall-clock timeout on recv_fifo in real mode. On timeout recv_fifo yields None → progressed=False → while loop breaks → aggregator commits what it has and proceeds to inc_round() where the budget check fires. Also capped at remaining wall budget (min with max_experiment_runtime_s) so the process never overshoots. Sim mode unaffected (_recv_timeout=None; _oort_sim_recv/drain_ready is non-blocking). Rename max_runtime_s → max_experiment_runtime_s across all 41 source files (YAMLs, scripts, lib code, docs). The new name makes the experiment-level scope clear; comments document the dual-clock behavior (wall in real mode, vclock in sim mode). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Background loop prints elapsed/remaining/percent every 30s while run_node is running; tracks experiments-started via new run_* dir count. Budget passed as n_exps × runtime_s (upper bound; sim faster). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…nup + oort cohort floor Three correctness fixes for sim-unavailability under scarcity, plus design doc update. 1. syncfl real-mode aggregate recv had NO timeout: recv_fifo(first_k=agg_goal) blocked forever when unavailable trainers withhold their uploads (feddance n=12 real hung ~46 min on a 30 min budget, 0 rounds, killed only by the runner watchdog). Bound it with min(trainer_recv_wall_timeout_s=90s, remaining budget), mirroring oort/asyncfl which already had this. Only the syncfl base lacked it, which is why oort completed and feddance hung. (Challenge 16 / B2.0.1) 2. Challenge 13: _handle_send_state "invalid prior selection" cleanup keyed off the availability-filtered eligible pool, so an in-flight trainer that merely went UN_AVL (or wrong task-type) was dropped from selected_ends; an empty per-task pool wiped ALL shared in-flight tracking -> hang. Added connected_ends param (full connected pool) for the cleanup membership check in async_oort/async_random/fedbuff; new-candidate selection still uses the eligible pool. +3 regression tests (TestChallenge13SendStateCleanup). 3. oort cohort-floor guardrail: clamp the starvation threshold to min(desired_selection, connected) + one-time [COHORT_FLOOR] warning, so an undersized cohort (n < desired_selection) can't silently degenerate into an all-starvation, budget-exhausting run. Tests: 449 passed; 7 pre-existing failures (test_sync_sim_ordering / test_sim_barrier fixtures missing _sim_buffer, fail identically on clean HEAD); parity ladder 24/24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ully green
The 7 long-standing failures in test_sync_sim_ordering (x6) and test_sim_barrier
(x1) were stale fixtures, not product bugs. They __new__ an aggregator (bypassing
__init__ + _init_availability), so the availability state the E/F stages added was
never set:
- _sync_sim_recv_first_k references self._sim_buffer directly (set in __init__)
- the AvailabilityMixin helpers expect trainer_event_dict / pending_withheld
- the oort real-recv timeout reads self.config.hyperparameters
Fix: give the fixtures the gate-OFF availability state
(_sim_buffer=SimReorderBuffer(), trainer_event_dict=None, pending_withheld={},
a minimal _HP with trainer_recv_wall_timeout_s / max_experiment_runtime_s).
Byte-identical to a real gate-off run, so the sim-ordering / stale-reject logic
is still exercised in isolation.
Result: 456 passed / 0 failed / 7 skipped + parity ladder 24/24. Doc updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…, 6-baseline scaffold, 79 tests
T0 (blocking): B2.0.2 starvation self-termination fixed in syncfl/oort/asyncfl.
- Starvation else-branch sets _work_done=True at trace horizon or budget.
- Budget check changed from > to >= so vclock==budget stops the run.
- Real-mode wall-budget guard added to the scarcity sleep path.
- 10 regression tests in test_starvation_termination.py.
T1: Two-axis flag split (_availability_aware → avail_select_filter + proactive_inflight_evict).
- avail_select_filter gates get_curr_task_ineligible_trainers (selection pool filter).
- proactive_inflight_evict gates _sim_evict_unavail_inflight (felix only).
- [ORACULAR] → [TRACE_READ]; oracular_trainer_avail_check → _trace_read_avail_check.
- Legacy fallback comment clarified: getattr fallback is dead (pydantic default=None
means attribute always exists; bool(None)=False is the correct safe default).
T2: Per-baseline flags set in parity YAML to match canonical baseline matrix.
felix: both ON. oort/fedbuff: both OFF. refl/feddance/oort_star: filter ON, evict OFF.
T3: oort_star + fedbuff scaffolded (baselines.yaml + parity YAML sim+real entries, 12 total).
- fedbuff lrDecay HPs added to match felix and all other async cifar10 runs.
- debug_run.sh smoke default updated to all 6 baselines.
- debug_run.sh proactiveInflightEvict auto-detect comment clarified (always dead;
Stage H future; explicit YAML values are authoritative).
T4: 41-test state-fidelity suite (test_state_fidelity.py): T-state-exact, T-eval-pool,
T-withhold-deliver, T-aware-vs-reactive, T-starvation-sync.
T5 pre-work: A5 state_timeline_agreement wired into checks.py + report.py (DIST tier,
0.95 tol). 28-test config-wiring suite (test_baseline_wiring.py): all 6 baselines,
catches oort_star-missing class of bug. oort_star added to baselines.yaml.
Tests: 536 pass / 7 skip.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…paign
debug_run.sh: LOGDIR is now env-overridable via FLAME_LOGDIR so each
per-run invocation from the suite writes its Python agg output to an
isolated directory instead of the shared /tmp/debug_run_logs.
smoke_suite.sh: new parent orchestrator that runs 5 ordered steps
(pytest → syn_0 sim → syn_20 sim → syn_20 both → syn_50 starvation)
with individual per-(baseline, mode) wall-clock timeouts:
- Each run launched via setsid so the whole process tree (launcher +
300 trainers) shares a session group; timeout kills via
SIGTERM → 20s grace → SIGKILL on the group PGID.
- Per-run grep checks on debug_run.out: stopping_run count (must be >0),
SIM_WALL_CEILING count (must be 0), SIM_STARVATION count (informational).
- Status: PASS / FAIL(no_stop) / FAIL(wall_ceil) / ERROR / TIMEOUT / SKIP.
- Final report printed to stdout and written to <output-dir>/report.txt.
- --dry-run shows every command without launching anything.
- Configurable per-step runtimes (--runtime-syn{0,20,50}-s) and
timeout buffer (--timeout-buffer-s); --steps to run a subset.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
setsid forks when the calling process is already a process group leader (which interactive terminals guarantee for every background job). This makes runner_pid point to the dead parent that setsid discards, so kill -TERM -$runner_pid was targeting a defunct PGID — the entire stuck process tree survived as orphans. Verified with a 3-level (bash → launcher → trainers) simulated stuck runner: 4 orphan sleep 9999 processes confirmed after the setsid kill. Fix: use `set -m` (job control) immediately before the background launch and `set +m` immediately after. With job control active, bash assigns PGID = runner_pid to the job unconditionally, in both interactive and non-interactive contexts. Since flame/launch/spawner.py uses plain subprocess.Popen with no start_new_session or preexec_fn=os.setsid, all 300 trainer processes inherit the same PGID and are killed by kill -TERM/-KILL -$runner_pid. Verified: 6-process tree (runner, ticker, launcher, 3 trainers) all dead after kill, PGID=runner_pid in both interactive (bash -c) and non-interactive contexts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three fixes to _run_baseline:
1. Post-kill cleanup (timeout path)
When SIGKILL fires, run_experiment_batch's finally block
(_sweep_stragglers: pkill trainers/agg + 45s GPU wait) is in the
killed process group and may not complete. Replicate it in bash:
pkill -9 -f "trainer/pytorch/main.py"
pkill -9 -f "aggregator/pytorch/main_"
sleep $KILL_SETTLE_S # default 20s GPU memory drain
so no stale trainer/aggregator processes or held GPU memory bleeds
into the next run's allocation.
2. wait() order fixed
Added wait after SIGKILL on timeout path; non-timeout path uses
conditional wait so we never double-wait the same PID.
3. Grep checks scan the correct file
AggregatorSpawner redirects the aggregator subprocess's stdout/stderr
to experiments/run_*/..._aggregator.log (NOT $FLAME_LOGDIR/debug_run.out,
which is only run_experiment.py's own print()s). Previous grep always
returned 0 for "stopping run" / SIM_WALL_CEILING / SIM_STARVATION,
causing every successful run to be classified as FAIL(no_stop).
Fix: use -newer $ts_marker to find aggregator logs created during this
run's lifetime; accumulate counts across them (handles multi-exp batches).
Paths saved to $run_dir/agg_logs.txt for easy post-mortem access.
Also adds --kill-settle-s CLI option and KILL_SETTLE_S=20 default.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…otheses 23 jobs / 22 FL experiments (steps 1,2,4,5; step 3 skipped as redundant with step 4 sim half). Expected ~4h 38min wall, fits 6h with ~1h margin. Documents per-baseline predictions, starvation-at-n300 note (SIM_STARVATION absent is correct, not a failure), and failure watch-list for oort_star/fedbuff first live runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix arithmetic crash: grep -c always prints a count even for 0 matches and exits 1, causing || echo 0 to fire and produce "0\n0"; $(( N + 0\n0 )) is a bash syntax error that aborted _run_baseline before _record, silently dropping all runs after the first baseline. Fix: || echo 0 → || true. - Add per-run in-place ticker (stderr, \r): shows elapsed/budget/pct within the current run plus kill-in countdown and overall N/total done count. Overwrites in place each 5s tick — lets you spot stuck runs without log flood. - Add _compute_total_runs + TOTAL_RUNS/COMPLETED_RUNS globals; _record now increments COMPLETED_RUNS so the ticker stays accurate across all steps. - Move COMPLETED_RUNS increment into _record so pytest (step 1) also counts. - Drop nohup from the design-doc command (tmux keeps the session alive). - Expand T5-smoke hypotheses into a 23-row per-run table with Actual column. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bash treats ${#var//x/y} as a parse error ("bad substitution"), which
is fatal regardless of set -e/-u — it killed the whole suite right at
the step 4 header, before any step 4/5 runs executed. Replace with
wc -w to count baselines.
…lientAvailability + B2.0.3 join-barrier re-anchor vclock_now was hardcoded None for real-mode selection telemetry, so A4dur's duration-weighted real/sim comparison fell back to a join-ramp-skewed origin; now stamped via ClientAvailability._avail_now() for both modes. Renamed AvailabilityMixin -> ClientAvailability (availability_mixin.py -> client_availability.py). Also fixes B2.0.3: agg_start_time_ts was stamped before the trainer join barrier, so real's trace-read clock baked in the join wait (~300s at n=300), reading the availability trace ~300s ahead of sim/ground-truth. Root-caused via feddance syn_50's A3 (avail_timebase) failure against the trace's own ground truth. Fixed via _mark_join_barrier_done(), which re-anchors agg_start_time_ts to real time once the join barrier resolves (real mode only, self-correcting to actual join duration) -- one fix point in syncfl/top_aggregator.py covers all six baselines via inheritance. Confirmed A4dur fix live on a fresh felix syn_20 real+sim pair (mean_err=0.0). B2.0.3 fix confirmation (fresh feddance syn_50 pair) still pending. 6 new regression tests in test_join_barrier_reanchor.py. Full suite green: 542 pass / 7 skip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-execs the suite via nohup+disown and returns immediately, printing the PID, nohup log, and eventual report.txt path. Lets a smoke campaign survive the launching shell/SSH session closing, so it can be fired off and checked on later rather than needing a session held open for hours. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…d, next-step command Whole-doc crisp pass per the Working Agreement: Status, T5-smoke, Challenges (#17 B2.0.2 was still marked open, now closed; new #18 for B2.0.3), and the known-parity-failures table all updated to reflect the A4dur confirmation and the new B2.0.3 root-cause + fix. Collapsed the old SCRATCH handoff section down to the still-open next steps (trackTrainerAvail cleanup, mobiperf live exercise, PR workflow) and dropped Q&A material now folded into permanent sections. Added a prominent NEXT STEP callout at the top of the doc with the exact unattended smoke_suite.sh command to confirm both fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Forwards --num-trainers through to every debug_run.sh invocation (debug_run.sh already auto-scales min_trainers_to_start accordingly). Lets a confirmation run use a smaller cohort under resource constraints without editing the parity config -- useful for mechanism-level fixes (e.g. B2.0.3's join-barrier re-anchor) that are n-independent by construction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Resource-constrained follow-up: A4dur and B2.0.3 are both mechanism-level fixes (already confirmed n-independent / self-correcting to actual join duration), so n=100 is sufficient to confirm them without reproducing the original n=300 failure magnitude. Notes the tradeoff -- not directly comparable to the existing n=300 parity_*.json snapshots for the still-open §7 checks, which stay gated on a real n=300 T5 run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tability) open() without an explicit encoding falls back to the node's locale-preferred encoding. On a non-UTF-8 locale (e.g. C/POSIX without Python's UTF-8 mode coercion) this mis-decodes non-ASCII bytes in the parity YAML (an arrow character in a comment) byte-by-byte, and yaml.safe_load then rejects the resulting control characters with a ReaderError -- reproduced exactly with PYTHONUTF8=0 LC_ALL=C + encoding="latin-1". Fixed the crashing read (debug_run.sh's parity-config generator) plus the write/re-read round trip, and hardened the other YAML config reads in the availability substrate (trace.py's synthetic/mobiperf trace files, client_availability.py's trainer registry) against the same failure mode before it bites on a node with non-UTF-8 content in those files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…3.0-T3.5) Adds absolute (vs. raw ground-truth trace) availability fidelity checks to complement the existing real-vs-sim relative checks, which is how the real-mode send-gate went dead code for the whole project without any check catching it (root-caused as T3.1a). - T3.0/T3.1a/T3.1b: shared trainer<->aggregator time origin (real mode), the debug_run.sh trace-wiring fix (trainer's own trace was never substituted), _refresh_avl_state() mode-dispatch cleanup. - T3.2: A6 trainer_trace_fidelity -- trainer's own avail_change telemetry vs. raw ground-truth trace. New scripts/parity/ground_truth.py; sim_now field added to avail_change. - T3.3: A7 agg_belief_fidelity -- aggregator's belief vs. ground truth, both selection and commit checkpoints. New agg_belief_change telemetry + ClientAvailability hooks. - T3.4: A8 send_gate_wait_fidelity -- observed vs. ground-truth-expected [SEND_GATE] wait duration. New send_gate_wait_s/send_gate_sct fields on task_send telemetry. - T3.5: K11 commit_promptness -- actual commit time vs. delivery_ts for withheld-then-delivered updates. New actual_commit_ts field on withheld_delivery telemetry. Each check ships with new telemetry, a parity rung in checks.py, a plot in analyze_run.py, and synthetic-data unit tests. Batch 3 is now code+test complete; UNAVAILABILITY_DESIGN.md's own plan calls for exactly one real run next (Phase 5) to generate live telemetry for all four new checks at once. Tests: lib/python/tests/ 574 passed / 7 skipped; examples/async_cifar10 scripts/parity/ + trainer/pytorch/ 121 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…e, A7 checker fix Phase 5/6's first real run against the new absolute fidelity checks (A6/A7/K11) surfaced three gaps; all three are root-caused, fixed, and unit-tested here: 1. felix real-mode TIMEOUT: D.1 proactive in-flight eviction (_sim_evict_unavail_inflight) was called only inside `if self.simulated:`, alongside the genuinely sim-only 90s vclock re-clock. But async_oort's selector derives `recv_ends` directly from `selected_ends`, so a stalled UN_AVL trainer never evicted in real mode never left recv_ends either, hanging the aggregator's 30s recv_fifo loop past its own budget. Un-nested D.1 from the sim-only gate in all three top_aggregator.py stacks (a no-op everywhere except felix, the only baseline with proactive_inflight_evict). 2. A6/A4dur/K6: sim-mode Trainer._sim_now() returns the frozen _sim_send_ts from its last dispatch, so a trainer withheld as UN_AVL can never observe later trace transitions while idle. Fixed in two zero-new-comms parts: (a) avail_change.sim_now now stamps the transition's own scheduled trace-time instead of _sim_now() at processing time; (b) inform_end_of_training's existing broadcast now piggybacks the aggregator's final vclock (sim mode only), and the trainer's shared _fetch_weights flushes one last _refresh_avl_state() catch-up on EOT. This also resolves the long-open K6 (`sim_send_ts==0`) mystery — same frozen-clock mechanism, not selector-starvation or a checker false positive. 3. A7 commit-checkpoint: not a system bug — the checker (_fidelity_score) extrapolated a trainer's last recorded commit-belief across the run's full span, wrongly scoring the silence after a trainer stops committing (typically because it went UN_AVL) as stale drift. Added extrapolate_tail=False for the commit-checkpoint call site, truncating the scoring window to [t_start, last observed t], symmetric with the existing start-side truncation. 12 new/updated tests (tests/ 586 pass/7 skip, scripts/parity/+trainer/pytorch/ 125 pass). Not yet confirmed on a live run — see UNAVAILABILITY_DESIGN.md's Batch 4 section for the re-confirmation command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live re-confirmation of the Batch 4 fix 1 (D.1 proactive eviction in real mode) found the fix is necessary but not sufficient: felix real still hangs at n=100, stuck at one round for 10+ minutes with only a handful of the ~20 stalled trainers ever getting evicted. Two isolated reproductions against the real AsyncOortSelector/ClientAvailability code (single stalled trainer, and 40/100 simultaneously UN_AVL) both show the eviction mechanism working correctly every cycle, so whatever is different about the live run isn't reproduced by either repro yet. Adds an [EVICT_DEBUG] summary log line to _sim_evict_unavail_inflight (inflight count, evicted count, and per-skip-reason counts: still available / buffered / already committed-or-withheld / no trace) so the next live run shows which guard is actually firing instead of inferring it from AWARE_EVICT's absence. Log-only change, no behavior change. Remove once root-caused (see UNAVAILABILITY_DESIGN.md's "Open A" for tracking). Also documents Open B: a separate, pre-existing bug found while investigating A -- trace loading falls back to the shared syn_20 "pattern" for most trainers instead of their individually-assigned per_trainer entry (hand-verified on two trainers whose own traces shouldn't transition until t=13800s/t=34200s, but which actually transitioned at t=600s/1200s matching the fallback exactly). Not yet root-caused or fixed; deliberately investigating Open A first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Explicit top-to-bottom checklist (Open A eviction-miss root cause, Open B trace-fallback root cause, cleanup, re-confirmation run, then the pre-existing Open items 1-2) so a new session can pick up exactly where this one left off without re-deriving status from chat history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Open A (felix real never self-stops): the real-mode 90s SEND_TIMEOUT_WAIT_S abandon in async_oort.py/fedbuff.py only cleared all_selected, never selected_ends. recv_ends derives from selected_ends, not all_selected, so a stalled end never left the recv_fifo wait set and max_experiment_runtime_s's self-stop check (gated on recv_ends being empty) was unreachable. Both selectors now also discard the end from selected_ends. Open B (trainers ignore their per-trainer trace): spawner.py's get_synthetic_trace never took a trainer_id, so every trainer process baked in the shared cohort-wide `pattern` instead of its own registry-assigned per_trainer trace, even though the aggregator's own belief-tracking already resolved per-trainer correctly via flame.availability.trace.load_trace. get_synthetic_trace now delegates to that same resolver when given a trainer_id. Trainer startup also logs an [AVAIL_TRACE] signature (trace_hash) so a shared hash across trainer_ids is visible directly in the logs. debug_run.sh's --trace now accepts a space-separated list (one experiment set per trace) to make cross-trace validation runs a single invocation. Tests: tests/selector/test_send_timeout_frees_selected_ends.py, tests/launch/test_config_generator.py::TestSyntheticTracePerTrainer, tests/launch/test_debug_run_trace_substitution.py::TestMultiTraceSubstitution — all verified to fail without their respective fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends 994f818's node-portability cleanup to the lines added in the last two commits (0514d42, 8d14586) that it predates: em-dash/en-dash/multiply/ approx/bullet glyphs in debug_run.sh comments and UNAVAILABILITY_DESIGN.md's NEXT STEP section, replaced with ASCII equivalents (--, -, x, ~). Scoped to those additions only, not the doc's older pre-existing prose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_FakeChannel.get_end_property() matched on the literal string
"round_duration", but _process_aggregation_goal_met() actually looks up
PROP_CLIENT_TASK_TRAIN_DURATION ("client_task_train_duration_s"). The fake
channel silently returned None instead of the seeded duration, so
trainer_speed_s/agg_observed_s always came back empty -- a fixture bug, not
a production one (traced back to a pre-existing mismatch already present on
dg-fork-main before this branch's rebase).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…se rules Condense fully-resolved sections (Batch 3 T3.0-T3.5, Phase 5/6 results, T5-smoke/A4dur/B2.0.3) to the doc's own stated format for completed stages -- mechanism + files + tests + exit, 2-3 lines -- dropping narrative/discovery play-by-play that's already preserved in commit history. Also updates the NEXT STEP run command to the actual planned first-pass confirmation run (1800s, syn_50 only, felix+fedbuff) ahead of the full 7h campaign. 1375 -> 945 lines, no content loss (full detail remains in git log for every referenced fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…syncfl top_aggregator.py Comment-only cleanup, no behavior change (verified via targeted availability/ sim-ordering/sim-barrier test runs, 141 passed): - client_availability.py: removed the TEMP DIAGNOSTIC eviction-miss counters and [EVICT_DEBUG] log line in _sim_evict_unavail_inflight -- explicitly marked for removal once Open A was root-caused, which UNAVAILABILITY_ DESIGN.md now confirms. Trimmed several multi-paragraph docstrings (get_curr_task_ineligible_trainers, _avail_stamp_end_states, _emit_withheld_delivery, _sim_reinject_ready_withheld) down to the causal fact + pointer, dropping session-narrative framing. - syncfl/top_aggregator.py: trimmed _mark_join_barrier_done's docstring and the EOT-broadcast piggyback comment the same way. First batch of a broader pass across the dg-fork-main diff; the other top_aggregator.py variants (oort/asyncfl), analyze_run.py, and the test files still have similar cleanup pending. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…n diff
Comment-only, no behavior change (full suite green: 728 passed / 7 skipped,
same as before this batch):
- oort/top_aggregator.py: trimmed the trainer_recv_wall_timeout_s explainer
and the cohort-floor guardrail comment; dropped an internal "Next actions
§2" task-tracking reference (also in asyncfl/top_aggregator.py).
- syncfl/trainer.py: trimmed the AGG_START_TS caching comment, the EOT
avl_state catch-up comment, and the [SEND_GATE] block's docstring-length
comment down to the causal fact.
- async_oort.py: reworded the Challenge 13 cleanup comment to match the
shorter version already used in fedbuff.py/async_random.py (all three
carry the identical fix; no reason for one copy to be 3x longer).
- message.py, examples/async_cifar10/trainer/pytorch/main.py,
analyze_run.py: trimmed remaining multi-paragraph docstrings/comments
down to mechanism + one-line rationale, dropping session/phase-tracking
references ("Batch 3 T3.0", "Batch 4 finding 2") in favor of plain
causal language -- that history lives in UNAVAILABILITY_DESIGN.md and
git log, not duplicated inline.
Completes the comment-cleanup pass across the highest-density files in the
diff (client_availability.py + the 3 top_aggregator.py variants + trainer.py
+ analyze_run.py). Remaining lower-density files and test-file docstrings
were left as-is -- most already follow the repo's convention of a short
per-test docstring naming the specific regression it guards against, which
is exactly the kind of comment worth keeping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher tied "how many trainers to spawn" and "which <name>_alpha<a>_n<N>.yaml split to read" to a single num_trainers knob, so a 10-trainer smoke demanded a dedicated n10 split file and KeyError-ed when only n300/n50/n48 existed. - experiment_config: add trainer.split_num_trainers (defaults to num_trainers) to pick the split partition independently of cohort size. - runner: split lookup uses split_num_trainers or num_trainers; the spawn cohort (trainer_ids) still uses num_trainers. - debug_run.sh: --num-trainers now preserves the config's native split size so a shrunk cohort reads the existing n300 split; add --alpha flag to choose dirichlet_alpha (0.1/1.0/10.0/100.0) against that n300 split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_fidelity_score held each sparse commit-checkpoint observation constant across interior gaps between commits, penalizing the absence of a mid-gap commit rather than any wrong belief. Add max_gap_s (wired to lag_tol_s at the commit call site): each observation vouches for its own state only up to max_gap_s past itself, on both sides; a transition with no covering window is excluded from both the TVD score and the missed/spurious diagnostic. Generalizes the earlier extrapolate_tail tail-truncation to interior gaps. Took felix 59/61 -> 62/62 on the live n=300 run. Tests: test_agg_belief_fidelity.py (interior-gap + wrong-belief-at-own-instant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…BILITY_DESIGN, drop generated artifacts
- PARITY.md: generalize to a multi-example parity methodology; add §F fwdllm rung
catalog (baseline matrix, modified rungs, variance-cadence / dynamic-K/C /
forward-grad layers).
- simulate_fwdllm.md: rewrite as the design-only build plan -- correct code anchors
to flame-core fwdllm_aggregator.py, resolve the §C baseline matrix from the landed
launcher configs, integrate unavailability as first-class, surface decisions D1-D4.
- UNAVAILABILITY_DESIGN.md: trim 1027 -> 283 lines to open items + next-steps +
durable reference (baseline matrix, flags, v1 decisions, land-mines, dead-ends);
collapse landed history to pointers; add the parallel-branch validation note.
- Remove generated, unreferenced parity_{felix,oort,feddance}.json (-2152 lines;
already covered by .gitignore).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adds a config-gated “client unavailability” model to FLAME’s FL simulator to improve real↔sim parity (without wall sleeps), and extends telemetry/parity tooling to validate trace-fidelity, gating, and delivery semantics. It also updates launcher robustness (timeouts / watchdogs) and refreshes parity documentation (including a design-only fwdllm simulator plan).
Changes:
- Introduces a shared availability-trace substrate (
flame/availability/*) and integrates it with selectors/aggregators, including new telemetry events and parity checks for absolute trace fidelity and send-gate behavior. - Expands/updates tests across selector, mode, launch, availability, and example parity suites to cover the new semantics (send-time gate, late delivery, proactive evict, join-barrier re-anchor, starvation termination).
- Updates runner/spawner behavior and many example configs/docs to reflect new runtime/budget keys and parity methodology.
Reviewed changes
Copilot reviewed 102 out of 102 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/analysis/plot_helpers.py | Adds a timeline plot helper for ground-truth vs observed availability state bands. |
| lib/python/tests/selector/test_send_timeout_frees_selected_ends.py | Adds regression tests ensuring SEND_TIMEOUT frees both all_selected and selected_ends. |
| lib/python/tests/selector/test_oort_selector.py | Adds tests for “connected vs eligible” cleanup semantics (Challenge 13). |
| lib/python/tests/mode/test_sync_sim_ordering.py | Updates sync sim-ordering tests for new sim buffer/HP defaults. |
| lib/python/tests/mode/test_sim_barrier.py | Updates sim barrier tests to initialize new sim buffer/availability no-op state. |
| lib/python/tests/mode/test_send_gate_wait.py | Adds tests for trainer-side SEND_GATE telemetry (real vs sim behavior). |
| lib/python/tests/mode/test_proactive_evict_call_site.py | Adds tests ensuring proactive inflight eviction is invoked in both modes. |
| lib/python/tests/mode/test_join_barrier_reanchor.py | Adds tests for real-mode join-barrier re-anchoring of agg_start_time_ts. |
| lib/python/tests/mode/test_fwdllm_agg_telemetry.py | Updates fwdllm telemetry test to use selector property constants. |
| lib/python/tests/mode/test_eot_avail_catchup.py | Adds tests for sim-mode end-of-training “catch-up” availability refresh. |
| lib/python/tests/mode/test_async_staggered_redispatch.py | Updates async staggered redispatch tests for new channel API expectations. |
| lib/python/tests/mode/test_agg_start_ts_broadcast.py | Adds tests for broadcasting/caching real-mode aggregator time origin to trainers. |
| lib/python/tests/launch/test_debug_run_trace_substitution.py | Adds regression tests verifying --trace substitution reaches trainer configs. |
| lib/python/tests/launch/test_config_generator.py | Adds regression tests for per-trainer synthetic trace resolution in configs. |
| lib/python/tests/availability/test_starvation_termination.py | Adds tests for starvation self-termination and budget equality behavior. |
| lib/python/tests/availability/test_delivery_ledger.py | Adds tests for delivery ledger, slot-free effects, and gate-off no-ops. |
| lib/python/tests/availability/test_agg_belief.py | Adds tests for aggregator belief telemetry hooks and withheld delivery emission. |
| lib/python/flame/telemetry/events.py | Adds new event builders and fields (avail_change.sim_now, task_send gate fields, delivery/abandon/belief events). |
| lib/python/flame/selector/refl_oort.py | Adds vclock_now to selection telemetry extras. |
| lib/python/flame/selector/oort.py | Adds vclock_now to selection telemetry extras in multiple selection modes. |
| lib/python/flame/selector/feddance.py | Adds vclock_now to selection telemetry extras. |
| lib/python/flame/selector/fedbuff.py | Fixes cleanup semantics via connected_ends and adds vclock_now to telemetry extras. |
| lib/python/flame/selector/async_random.py | Passes connected_ends to send-state handling and updates signature. |
| lib/python/flame/selector/async_oort.py | Passes connected_ends to send-state handling and updates signature. |
| lib/python/flame/selector/init.py | Extends per-trainer selection telemetry with avl_state. |
| lib/python/flame/plugin/init.py | Replaces deprecated logger.warn with logger.warning. |
| lib/python/flame/mode/message.py | Adds MessageType.AGG_START_TS for real-mode shared origin broadcast. |
| lib/python/flame/mode/horizontal/syncfl/trainer.py | Caches AGG_START_TS, adds EOT refresh hook, and emits send-gate telemetry fields. |
| lib/python/flame/mode/horizontal/syncfl/fwdllm_aggregator.py | Switches ORACULAR availability check to _trace_read_avail_check. |
| lib/python/flame/launch/spawner.py | Adds per-trainer synthetic trace resolution; changes trainer shutdown waiting/termination logic. |
| lib/python/flame/launch/runner.py | Adds fault handler env, updates budget key usage, and improves trainer split sizing and logs. |
| lib/python/flame/launch/experiment_config.py | Adds split_num_trainers to trainer config schema and YAML loading. |
| lib/python/flame/config.py | Adds unavailability feature flags and renames/aligns runtime cap terminology in comments. |
| lib/python/flame/availability/trace.py | Introduces shared trace resolver utilities (load/state_at/next_avail/read_unavailability). |
| lib/python/flame/availability/init.py | Exports trace helpers and ClientAvailability from availability package. |
| lib/python/examples/async_cifar10/trainer/pytorch/test_refresh_avl_state.py | Adds tests ensuring refresh logic works in real mode with shared origin. |
| lib/python/examples/async_cifar10/trainer/pytorch/test_avail_change_due_ts.py | Adds tests ensuring avail_change.sim_now uses transition due time. |
| lib/python/examples/async_cifar10/trainer/pytorch/test_agg_start_ts_sim_now.py | Adds tests for trainer real-mode _sim_now() origin selection. |
| lib/python/examples/async_cifar10/trainer/pytorch/main.py | Adds shared-origin support, refresh logic, due-ts stamping, and compute-completes send-gate semantics. |
| lib/python/examples/async_cifar10/trainer/pytorch/conftest.py | Adjusts sys.path to match script-style imports under pytest. |
| lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh | Updates hyperparameter key naming (max_experiment_runtime_s). |
| lib/python/examples/async_cifar10/scripts/parity/test_send_gate_wait_fidelity.py | Adds unit tests for A8 send-gate wait fidelity rung. |
| lib/python/examples/async_cifar10/scripts/parity/test_ladder.py | Adds unit tests for new parity behaviors (A3 pass/skip, A4 pass/fail/skip). |
| lib/python/examples/async_cifar10/scripts/parity/test_ground_truth.py | Adds unit tests for ground-truth trace resolution and A6 fidelity behavior. |
| lib/python/examples/async_cifar10/scripts/parity/test_avail_state_series.py | Adds unit tests for shared availability state series resolver. |
| lib/python/examples/async_cifar10/scripts/parity/test_agg_belief_fidelity.py | Adds unit tests for A7 agg belief fidelity behavior and gap-handling. |
| lib/python/examples/async_cifar10/scripts/parity/report.py | Extends parity report catalog/formatting for new rungs and diagnostics. |
| lib/python/examples/async_cifar10/scripts/parity/ground_truth.py | Adds ground-truth trace loading and duration-weighted state query helpers. |
| lib/python/examples/async_cifar10/scripts/parity/cli.py | Loads ground-truth traces and passes them into parity checks; updates budget help text. |
| lib/python/examples/async_cifar10/scripts/parity/avail_state_series.py | Adds shared per-trainer state series builders and timeline helpers for parity and plotting. |
| lib/python/examples/async_cifar10/scripts/gen_n50_experiment.py | Updates generated experiment configs to use max_experiment_runtime_s. |
| lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md | Updates docs to reflect max_experiment_runtime_s naming. |
| lib/python/examples/async_cifar10/PARITY.md | Generalizes parity methodology doc; adds fwdllm rung catalog section. |
| lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/oort_n300_alpha0.1_syn0_STREAMING.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_sim.yaml | Updates runtime key naming across many experiments. |
| lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node4.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node3.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node2.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node1.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1.yaml | Updates runtime key naming throughout baseline batch. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node2.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node2.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node1.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node2.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node1.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/felix_n300_alpha0.1_syn0_STREAMING.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_sim.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_real.yaml | Updates runtime key naming. |
| lib/python/examples/async_cifar10/docs/PLAN_felix_streaming_experiment.md | Updates documentation references to the runtime cap key. |
| lib/python/examples/async_cifar10/docs/IMPLEMENTATION_felix_streaming.md | Updates implementation doc references to the runtime cap key. |
| lib/python/examples/async_cifar10/docs/EXPERIMENT_felix_streaming.md | Updates experiment doc references to the runtime cap key. |
| lib/python/examples/async_cifar10/aggregator/pytorch/main_oort_sync_agg.py | Removes inlined availability-trace loading logic in favor of shared substrate. |
| lib/python/examples/async_cifar10/aggregator/pytorch/main_asyncfl_agg.py | Removes legacy stub availability-trace reader wiring. |
| lib/python/examples/_metadata/baselines.yaml | Adds/updates the oort_star baseline metadata entry. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+445
to
+451
| deadline = time.time() + timeout_per_trainer | ||
| pending = [p["process"] for p in self.processes] | ||
|
|
||
| while pending and time.time() < deadline: | ||
| pending = [p for p in pending if p.poll() is None] | ||
| if pending: | ||
| time.sleep(0.5) |
Comment on lines
+466
to
+470
| kill_deadline = time.time() + 5 | ||
| while pending and time.time() < kill_deadline: | ||
| pending = [p for p in pending if p.poll() is None] | ||
| if pending: | ||
| time.sleep(0.5) |
| sel.pacer = _boom | ||
| return sel | ||
|
|
||
| def test_stale_end_leaves_both_all_selected_and_selected_ends(self, make_ends): |
| sel.all_selected = {"stale": time.time() - 100} # past 90s SEND_TIMEOUT_WAIT_S | ||
| return sel | ||
|
|
||
| def test_stale_end_leaves_both_all_selected_and_selected_ends(self, make_ends): |
dhruvsgarg
added a commit
that referenced
this pull request
Jul 6, 2026
…ifar10) + fwdllm sim design (#69) * Sim unavailability: lock v1 scope (oracular-for-all) + file-level impl spec Rewrite UNAVAILABILITY_DESIGN.md to fold in the Jun25 design resolutions: - v1 scope locked: aggregator reads the shared trace (oracular) for ALL baselines, client_notify OFF. The old per-baseline oracle-vs-notify asymmetry collapses; aware vs unaware now differ only in WHEN a stalled slot is freed (aware: proactively at next selection boundary; unaware: reactively at the 90s abandon deadline). True avl_* transport + continuous mid-round scheduling deferred to Stage H. - Withhold-then-deliver corrected to a send-time gate, not a mid-compute interrupt: compute always completes, only the upload is gated, delivered stale at delivery_ts=max(sct,next_avail_ts). Real-side send-gate change documented; sim commits via the agg-side buffer. - 90s abandon + withhold are both true = two ledgers (slot vs delivery); the existing SEND/RECV_TIMEOUT_WAIT_S must re-clock to _vclock.now. - Three correctness invariants (no double-count; never re-select a still-down trainer; aware/unaware = trigger only). - Library mixin seam: flame/availability/{trace.py,availability_mixin.py} mixed into all four TopAggregators, consolidating the 3 duplicate read_trainer_unavailability; spans async_cifar10 + fwdllm. - New section 8: file-level v1 implementation spec (Stage A + C). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage A: trace resolver + AvailabilityMixin substrate New files: - flame/availability/trace.py: load_trace / state_at / next_avail_after (single bisect_right resolver; lru_cache for YAML files; replaces three inlined bisect_right copies) - flame/availability/availability_mixin.py: AvailabilityMixin with _init_availability, _avail_now (vclock in sim), read_trainer_unavailability, get_curr_unavail_trainers, free_stalled_slot (dormant hook for Stage C/D/H) Deletions (consolidation): - read_trainer_unavailability: removed from main_oort_sync_agg.py, main_asyncfl_agg.py, fwdllm_aggregator.py (all three dup copies) - get_curr_unavail_trainers: removed from syncfl/top_aggregator.py body (wall-clock impl) and main_oort_sync_agg.py (wall-time-in-sim bug); mixin provides the single vclock-correct version Wiring: - AvailabilityMixin mixed into syncfl TopAggregator → all four stacks inherit - _init_availability called from internal_init; sim_unavailability=False (default) ⇒ trainer_event_dict=None ⇒ byte-identical to all existing runs - flame/config.py: sim_unavailability, availability_aware, availability_trace_dir fields - Supports both new sim_unavailability gate and legacy track_trainer_avail path 389/389 unit tests pass. Syn_0 regression smoke pending on training node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * stage A tests and wip stage B * stage B complete * Sim unavailability Stage C: delivery-ledger substrate + resume plan Land the reusable C.2/C.4/C.5 core (gated/dormant until the live commit loop calls it; default OFF => byte-identical): - AvailabilityMixin.compute_delivery_ts: earliest t>=sct with state AVL_* (immediate if already up at sct; inf if the trace never recovers). - free_stalled_slot activated: frees the selector slot ledger AND registers pending_withheld[end]=delivery_ts (two ledgers, Challenge 4 / invariant 1). - withheld_held_ends / ready_withheld / commit_withheld delivery-ledger helpers (ready_withheld orders by (delivery_ts, end_id) -> no past-dating). - invariant 2 wired: withheld_held_ends() unioned into the unavailable list in oort + asyncfl _distribute_weights so a held trainer stays out of the eligible pool until its delivery_ts. - asyncfl __init__: _sim_withheld_payload store added (consumed by C.2 next). 10 new unit tests (tests/availability/test_delivery_ledger.py); full suite 420/420. Detailed C.2/C.3 live-wiring resume plan added to UNAVAILABILITY_DESIGN.md (RESUME HERE subsection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage C: live commit-loop wiring (C.2/C.3) + parity rungs Wire the send-gate withhold / late re-commit (C.2) and the 90s vclock abandon (C.3) into both sim commit loops, as ONE shared library-level core in AvailabilityMixin so the asyncfl (felix/fedbuff) and oort (oort/refl) stacks ride identical logic instead of forking it: * _sim_withhold_if_unavail — per-update send-gate primitive (invariant-1 stash for an already-abandoned end; else hold if state_at(sct)=UN_AVL: stash payload, free_stalled_slot, register the delivery ledger). Pure per-update decision — does not consult round_start, which is why oort applies its carry-over gate first (Challenge 7). * _sim_pop_committable / _sim_reinject_ready_withheld — asyncfl pop loop + due re-injection at delivery_ts (ordered), slot-only ledger drop. * _sim_abandon_stalled — vclock 90s abandon over the selector slot ledger; _avail_free_slot_ledger / _avail_inflight_ends are robust to both selector shapes (fedbuff dict-by-requester + all_selected; oort/refl/feddance flat set). * _sim_take_withheld_delivering / _emit_withheld_delivery — late-stale-commit telemetry; asyncfl tags the "withheld" past-dating bucket. Call sites: asyncfl/_sim_recv_min pop site; oort/_sim_drain_buffer pop loop (composed after the §4.9 carry-over gate); both _distribute_weights run the abandon scan. All helpers are getattr-guarded ⇒ gate-off byte-identity holds for bare aggregators that never ran _init_availability. Run activation: trace-name normalization (avl_events_syn_20 → syn_20) in flame/availability/trace.py — the legacy oort/refl JSON configs resolved to 0 traces / silent-OFF before. Confirmed end-to-end: 300 traces load, 34/300 UN_AVL mid-window, compiled config carries enabled/type/trace. Telemetry: withheld_delivery + abandon_timeout events/builders (events.py). Parity rungs (checks.py, wired into run_all_parity + CHECK_META): * withheld_delivery (DIAG) — structural: delivery_ts≥sct, delay_s≥0, staleness≥0; reports delay/staleness dist + accept_frac. * abandon_timeout (CONTROL) — fails loudly on a wall-clock leak. * eligible_pool_reduction (DIAG) — candidates−eligible across modes. Loaders surface the new agg events + trainer avail_change; duty_cycle reads the real {old_state,new_state} format (A4 bug). observation_lag + A4b held (need run data — see §7). Tests: 436/436 lib (16 new in tests/availability/test_live_wiring.py) + 38 parity (14 new in scripts/parity/test_availability_rungs.py). Next: oort syn_20 smoke (sim+real, --runtime-s 1800 to clear the 600s down window) before felix/fedbuff or Stage D — see UNAVAILABILITY_DESIGN.md NEXT ACTION. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * wip fixes, context saved in unavail_design.md * Sim unavailability Stage C.6 complete + Stage D.1 boundary eviction wired C.6.4: add trainer_state_fractions_sorted.pdf (per-trainer UN_AVL sorted bar, A4dur visual companion) to analyze_run.py availability_plots. Syntax clean, crash-safe. Visual correctness needs a fresh syn_20 run. D.0: felix gate plumbing in parity yaml — sim_unavailability/availability_aware/ client_notify added to both felix entries so _init_availability activates for felix on syn_20 runs (client_notify.trace overridden by --trace flag). D.1: _sim_evict_unavail_inflight in AvailabilityMixin — proactive boundary eviction for availability_aware baselines: at each selection boundary, any in-flight trainer showing UN_AVL per the trace has its slot freed immediately (no 90s wait). Called from oort and asyncfl top_aggregators after _sim_abandon_stalled; guarded by _availability_aware so oort/refl (unaware) stay on the 90s-abandon path. build_abandon_timeout gains reason field to distinguish C.3 from D.1 events. 8 new unit tests; 444/444 total. Next: syn_0 byte-identity smoke (felix+oort) then felix syn_20 smoke to validate D.1 eviction fires and C.6.4 plots populate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Speed up smoke-test debug runs: parallel trainer shutdown + --num-trainers wait_all() waited up to 30s per straggler trainer sequentially, adding minutes of idle wait when several trainers got stuck post-aggregator-exit; now all trainers share one timeout window polled concurrently. Also add debug_run.sh --num-trainers to shrink the cohort below 300 for runs that need a real --runtime-s budget (e.g. a vclock floor for an availability trace) that the existing smoke mode's hardcoded rounds=4/runtime=240 would cut short. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sim unavailability: real send-gate, avl_state telemetry fix, D.2 task gating Lands the four implementation items from UNAVAILABILITY_DESIGN.md's Next actions: (1) real-side send-time gate (§8.3) — trainer task-start no longer skips on avl_state (compute always completes), and syncfl/trainer.py's existing send-time UN_AVL check is decoupled from client_notify["enabled"] (always False in v1) so it actually fires for real felix/oort; (2) fixes avail_composition/avl_state staying all-UNKNOWN by stamping PROP_AVL_STATE from the oracular trace onto every known end before each selection; (3) fixes withheld_delivery under-emission caused by the delivery ledger dropping a slot-only entry before its payload physically arrived; (4) D.2 AVL_TRAIN/ AVL_EVAL task-type eligibility gating in selection, wired into oort + asyncfl. 452/452 lib + 63/63 example-parity tests pass (was 444/444). Real/sim parity run validation still pending (see Next actions). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix D.2 hang: guard eval task-type exclusion against 2-state traces D.2's AVL_TRAIN-exclusion-from-eval rule made eval's eligible pool permanently empty on a 2-state trace (syn_0/syn_20/syn_50 never produce AVL_EVAL), which corrupted the selector's shared in-flight tracking for the train task too (async_oort.py/fedbuff.py _handle_send_state's disconnect-cleanup loop wipes selected_ends when handed an empty pool). Found via a real felix syn_0 smoke that hung with 0 AGG_RECV_WEIGHTS despite all 48 trainers training and sending. _init_availability now computes _trace_has_avl_eval once (does this trace ever produce AVL_EVAL for any trainer); the eval-side exclusion in get_curr_task_ineligible_trainers only applies when true, matching the design doc's own "2-state collapses the split" statement. Train-side exclusion is left unguarded (no 3-state trace exists yet to exercise the symmetric risk) and documented as a residual risk in UNAVAILABILITY_DESIGN.md (Challenge 13). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: crisp down UNAVAILABILITY_DESIGN.md + add syn_0 confirmation 838 → 452 lines. Completed stages compressed (mechanism + where it lives + exit met). Updated status: D.2 hang fix CONFIRMED via Jun 28 syn_0 runs (felix+oort, both complete 4 FL rounds, all avail rungs PASS, abandon/withheld correctly SKIP). Removed run-unconfirmed tags now folded into "pending syn_20" which is in progress. Added D.2 empty-pool hang to §9 dead-ends. Dropped verbose per-item prose for landed stages; kept the invariants and land-mines that bite if forgotten. parity_felix.json/parity_oort.json from Jun 28 syn_0 regression runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Parity fixes + oort syn_20 analysis: NEAR_ZERO_LAG, phase guard, new rungs Three parity check issues found and fixed during syn_20 validation: 1. Raise NEAR_ZERO_LAG_S 0.05→0.10s (U6): carry-over burst after avail windows inflates sim mean to ~70ms — still immediate-commit semantically. 2. Add point-mass guard in trainer_phase_split: both means ≤5ms → skip KS, pass on mean (pre_train_s, weights_to_gpu_s, weights_to_ram_s near-zero). 3. Wire four new avail rungs into report.py _SECTIONS: A4dur, Aa (eligible_pool_reduction), C.3 (abandon_timeout), C.2 (withheld_delivery). Fix abandon_timeout tier CONTROL→INV so _TIER_TAG renders correctly. UNAVAILABILITY_DESIGN.md: add §9.1 (oort parity settled diagnosis — K3b run-length, T2 pre-existing, A2 KS shape artifact from avail-window bimodal distribution; none block Stage E). Reorder Next Actions: Stage E first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: optimized batched plan for E/F/G Collapse remaining v1 work (E feddance, F starvation, G integration) into one code batch + one long-run batch. Non-conflicting code surfaces land together, gated on cheap signals (unit tests + syn_0 byte-identity + syn_20 feddance / syn_50 smokes). Share the F-validation syn_50 run with G.2's ramp rung; overlap the oort 3600s confirmation for free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Batch 1: syncfl E.1/E.2/E.3, Stage F vclock-advance, G.1 starvation rung Stage E (syncfl path — feddance + refl): - syncfl _distribute_weights: abandon/evict/stamp + scarcity-advance (calls _next_avail_vclock() and advances vclock instead of sleeping when no trainers selectable at round start) - syncfl _sync_sim_recv_first_k: withhold send-gate (_sim_withhold_if_unavail) + withheld bonus drain (_sim_reinject_ready_withheld + _emit_withheld_delivery) - syncfl internal_init: _sim_buffer (SimReorderBuffer) + _sim_committed init - E.2 = accept-stale (FedAvg has no staleness gate); E.3 = naturally handled (withheld updates excluded from committed, so U6 is over actual cohort) Stage F (starvation vclock-advance): - New _next_avail_vclock() mixin helper: min across all trace transitions + pending_withheld delivery timestamps; returns None when gate off - Wired into: (a) syncfl if-not-selected_ends path, (b) oort max_retries loop - Felix asyncfl (top_aggregator.py:639) gap noted; deferred to syn_50 observation G.1 (parity rungs): - starvation_advance_parity() in checks.py: DIAG rung, jump_factor=5x mean gap, gate = withheld_deliveries or abandon_timeouts or avail_composition subfield - Dispatched in Stage 2 Availability results dict - Added to report.py _SECTIONS section "2" as "Fst starvation_advance (diag)" - withheld_delivery deps chain updated to include abandon_timeout - 4 new starvation_advance unit tests; 67/67 parity tests pass (was 63) Felix master-gate (prereq): - debug_run.sh trace-override block now injects simUnavailability: True (+ availabilityAware: True if client_notify.enabled) for non-ORACULAR baselines when --trace syn_X is passed; oort/refl still use legacy path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: clarify ORACULAR naming, add baseline smoke rationale Terminology overhaul to eliminate the dual meaning of "ORACULAR": - "ORACULAR" in YAML/code now explicitly labelled as the legacy-gate config mechanism (field name in oort/refl YAML), not a knowledge model descriptor - v1 knowledge model renamed "trace-read" (agg reads trainer_event_dict directly) — applies to ALL baselines, including feddance - "aware/unaware" split renamed "proactive/reactive-90s" for slot-free timing (proactive = felix at next boundary; reactive-90s = oort/refl/feddance at 90s vclock deadline) - Stage H transport renamed "message-transport" (avl_* msgs) vs "trace-read" - Config-gate section now defines legacy-gate (oort/refl) vs simUnavail-gate (felix/feddance) and explains they activate the same trace-read code path - §0 existing paths table updated: path A = "trace-read", path C = "message-transport" - §8.4 config surface updated with mapping table Stage E smoke rationale added: feddance-only is sufficient because refl shares the same syncfl _distribute_weights/_sync_sim_recv_first_k code and its legacy-gate was already confirmed in Stage C (oort syn_20 run). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: oort 3600s syn_20 analysis + G.4 terminology-in-code todo oort n=300 3600s syn_20 results (Jun 28 Run 2: 40/48 PASS): - T2 training_budget: CONFIRMED self-corrected (ratio=1.133 ≤ 1.15) ✅ - K3b overhead_residual: did NOT self-correct as predicted (still rel=0.116, now gated downstream by P3); prediction revised - A2 eligibility: improving (KS 0.437→0.338) but still shape artifact; need more rounds (Batch 2 long run) - P3 trainer_speed: NEW marginal root cause at n=300 (ratio=1.153 vs tol 1.15, 1.3% over); clean at n=48; likely speed-tail noise at full cohort - Fst starvation_advance: PASS "no starvation advances detected" — correct (n=300 + syn_20 always leaves ~240 trainers available, Stage F never fires) G.4 added to Stage G: consolidate config-gate paths + rename legacy function `oracular_trainer_avail_check` → `_trace_read_avail_check` + update comments/logs to use new terminology (trace-read, proactive, reactive-90s, legacy-gate, simUnavail-gate). Deferred to after Batch 2 long runs pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: Stage E confirmed (feddance syn_20 1800s 46/47), Stage F in progress feddance syn_20 1800s parity check: 46/47 PASS. All availability and mechanism rungs pass (A1–A4, U3/U6/K8/U2). Sole fail = C2 loss (0.1696 vs 0.15, 2 eval pts, early-train noise at alpha=0.1/syn_20 — not a mechanism gap). Challenge 9 confirmed: no K8/U2 movement from withheld stale commits. Stage E exit met. Stage F syn_50 runs launched (feddance + oort, 45 min each, real+sim). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Stage F.2: unified pre-selection starvation gate + doc overhaul F.2 replaces the broken per-aggregator retry mechanisms with a single pre-selection return-early pattern across all three aggregators: - oort: was max_retries=5 + min_required=5 (50% of agg_goal) + "proceed anyway" fallback + 2s blocking sleep. Now: threshold=desired_selection (=int(aggr_num×overcommitment)=13), unbounded, non-blocking (0.5s + return). - syncfl (feddance/refl): was post-selection `if not selected_ends:` which FedDanceSelector's partial returns (3/10 eligible) never triggered. Now: pre-selection threshold=agg_goal=10. - asyncfl (felix/fedbuff): was time.sleep(0.5) in both modes at no-recv-ends path. Now: sim advances _next_avail_vclock(); real keeps 0.5s sleep. All three use the same pattern: num_eligible < threshold → sim vclock-advance + re-stamp + return; real: 0.5s sleep + return. Outer run() loop retries non-blocking. No max_retries ceiling; no "proceed anyway" fallback. At n=10 syn_50: min_avail≈3 at t≈1200s → eligible=3 < desired_selection=13 (oort) and < agg_goal=10 (feddance) → both starvation paths now exercisable. UNAVAILABILITY_DESIGN.md: 746 → 248 lines. Completed stages compressed to 2-3 lines each. Full detail kept only for active (Stage F exit) and next (Batch 2). Known parity failures moved to a single reference table (§7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: syn_0 regression passed; syn_50 n=10 starvation smoke in progress syn_0 results (n=48): Fst PASS (no starvation fires), C1/C2 diff=0.0, K1/K5 PASS on both baselines. Oort 43/50 (K3b pre-existing gates K2/K3); feddance 46/48 (A2 shape artifact + gpu_compute short-run noise, K2/K3/P3 PASS). F.2 threshold check confirmed not firing spuriously at n=48 with syn_0. n=10 syn_50 starvation smoke launched; results to check in morning. Next actions updated to morning check commands. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix syn_50 feddance sim deadlock + improve crash diagnostics; update doc debug_run.sh: simUnavailability was never injected for feddance (parity config has tracking_mode:client_notify at aggregator level but no HP-level tracking block, so neither the trackTrainerAvail nor client_notify branch fired). Result: trainer_event_dict=None → vclock deadlock at 0.0 for full 5400s. Fix: add third elif branch for non-oracular baselines with no HP tracking block, injecting availability_trace + simUnavailability=True. Also add simUnavailability to the client_notify-in-HP branch (felix path). runner.py: aggregator exit code was not logged, making silent crashes (e.g. feddance real crash at 12s) undiagnosable. Log exit code after wait(); also set PYTHONFAULTHANDLER=1 in child env so C-level crashes (segfault in paho/torch) write a traceback to _aggregator.log. UNAVAILABILITY_DESIGN.md: update status (n=10 bugs root-caused), replace next-actions with n=25 1800s run command + corrected rationale. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix real-mode recv deadlock + rename max_runtime_s → max_experiment_runtime_s Real-mode oort aggregator could block indefinitely in recv_fifo when syn_50 trainers went unavailable mid-round: no timeout meant max_runtime_s check in inc_round() was never reached, causing the process to run until manually killed. Fix: add trainer_recv_wall_timeout_s HP (default 90s, YAML-configurable) as a per-message wall-clock timeout on recv_fifo in real mode. On timeout recv_fifo yields None → progressed=False → while loop breaks → aggregator commits what it has and proceeds to inc_round() where the budget check fires. Also capped at remaining wall budget (min with max_experiment_runtime_s) so the process never overshoots. Sim mode unaffected (_recv_timeout=None; _oort_sim_recv/drain_ready is non-blocking). Rename max_runtime_s → max_experiment_runtime_s across all 41 source files (YAMLs, scripts, lib code, docs). The new name makes the experiment-level scope clear; comments document the dual-clock behavior (wall in real mode, vclock in sim mode). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * debug_run.sh: add 30s progress ticker + experiment count display Background loop prints elapsed/remaining/percent every 30s while run_node is running; tracks experiments-started via new run_* dir count. Budget passed as n_exps × runtime_s (upper bound; sim faster). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(availability): real-mode recv-barrier timeout + Challenge 13 cleanup + oort cohort floor Three correctness fixes for sim-unavailability under scarcity, plus design doc update. 1. syncfl real-mode aggregate recv had NO timeout: recv_fifo(first_k=agg_goal) blocked forever when unavailable trainers withhold their uploads (feddance n=12 real hung ~46 min on a 30 min budget, 0 rounds, killed only by the runner watchdog). Bound it with min(trainer_recv_wall_timeout_s=90s, remaining budget), mirroring oort/asyncfl which already had this. Only the syncfl base lacked it, which is why oort completed and feddance hung. (Challenge 16 / B2.0.1) 2. Challenge 13: _handle_send_state "invalid prior selection" cleanup keyed off the availability-filtered eligible pool, so an in-flight trainer that merely went UN_AVL (or wrong task-type) was dropped from selected_ends; an empty per-task pool wiped ALL shared in-flight tracking -> hang. Added connected_ends param (full connected pool) for the cleanup membership check in async_oort/async_random/fedbuff; new-candidate selection still uses the eligible pool. +3 regression tests (TestChallenge13SendStateCleanup). 3. oort cohort-floor guardrail: clamp the starvation threshold to min(desired_selection, connected) + one-time [COHORT_FLOOR] warning, so an undersized cohort (n < desired_selection) can't silently degenerate into an all-starvation, budget-exhausting run. Tests: 449 passed; 7 pre-existing failures (test_sync_sim_ordering / test_sim_barrier fixtures missing _sim_buffer, fail identically on clean HEAD); parity ladder 24/24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(availability): fix 7 stale sync-aggregator fixtures; suite now fully green The 7 long-standing failures in test_sync_sim_ordering (x6) and test_sim_barrier (x1) were stale fixtures, not product bugs. They __new__ an aggregator (bypassing __init__ + _init_availability), so the availability state the E/F stages added was never set: - _sync_sim_recv_first_k references self._sim_buffer directly (set in __init__) - the AvailabilityMixin helpers expect trainer_event_dict / pending_withheld - the oort real-recv timeout reads self.config.hyperparameters Fix: give the fixtures the gate-OFF availability state (_sim_buffer=SimReorderBuffer(), trainer_event_dict=None, pending_withheld={}, a minimal _HP with trainer_recv_wall_timeout_s / max_experiment_runtime_s). Byte-identical to a real gate-off run, so the sim-ordering / stale-reject logic is still exercised in isolation. Result: 456 passed / 0 failed / 7 skipped + parity ladder 24/24. Doc updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * latest unavail markdown file * feat(availability): T0-T5 pre-flight — starvation fix, two-axis flags, 6-baseline scaffold, 79 tests T0 (blocking): B2.0.2 starvation self-termination fixed in syncfl/oort/asyncfl. - Starvation else-branch sets _work_done=True at trace horizon or budget. - Budget check changed from > to >= so vclock==budget stops the run. - Real-mode wall-budget guard added to the scarcity sleep path. - 10 regression tests in test_starvation_termination.py. T1: Two-axis flag split (_availability_aware → avail_select_filter + proactive_inflight_evict). - avail_select_filter gates get_curr_task_ineligible_trainers (selection pool filter). - proactive_inflight_evict gates _sim_evict_unavail_inflight (felix only). - [ORACULAR] → [TRACE_READ]; oracular_trainer_avail_check → _trace_read_avail_check. - Legacy fallback comment clarified: getattr fallback is dead (pydantic default=None means attribute always exists; bool(None)=False is the correct safe default). T2: Per-baseline flags set in parity YAML to match canonical baseline matrix. felix: both ON. oort/fedbuff: both OFF. refl/feddance/oort_star: filter ON, evict OFF. T3: oort_star + fedbuff scaffolded (baselines.yaml + parity YAML sim+real entries, 12 total). - fedbuff lrDecay HPs added to match felix and all other async cifar10 runs. - debug_run.sh smoke default updated to all 6 baselines. - debug_run.sh proactiveInflightEvict auto-detect comment clarified (always dead; Stage H future; explicit YAML values are authoritative). T4: 41-test state-fidelity suite (test_state_fidelity.py): T-state-exact, T-eval-pool, T-withhold-deliver, T-aware-vs-reactive, T-starvation-sync. T5 pre-work: A5 state_timeline_agreement wired into checks.py + report.py (DIST tier, 0.95 tol). 28-test config-wiring suite (test_baseline_wiring.py): all 6 baselines, catches oort_star-missing class of bug. oort_star added to baselines.yaml. Tests: 536 pass / 7 skip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(smoke): add smoke_suite.sh — timeout-guarded overnight smoke campaign debug_run.sh: LOGDIR is now env-overridable via FLAME_LOGDIR so each per-run invocation from the suite writes its Python agg output to an isolated directory instead of the shared /tmp/debug_run_logs. smoke_suite.sh: new parent orchestrator that runs 5 ordered steps (pytest → syn_0 sim → syn_20 sim → syn_20 both → syn_50 starvation) with individual per-(baseline, mode) wall-clock timeouts: - Each run launched via setsid so the whole process tree (launcher + 300 trainers) shares a session group; timeout kills via SIGTERM → 20s grace → SIGKILL on the group PGID. - Per-run grep checks on debug_run.out: stopping_run count (must be >0), SIM_WALL_CEILING count (must be 0), SIM_STARVATION count (informational). - Status: PASS / FAIL(no_stop) / FAIL(wall_ceil) / ERROR / TIMEOUT / SKIP. - Final report printed to stdout and written to <output-dir>/report.txt. - --dry-run shows every command without launching anything. - Configurable per-step runtimes (--runtime-syn{0,20,50}-s) and timeout buffer (--timeout-buffer-s); --steps to run a subset. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): replace setsid with set -m for reliable process-tree kill setsid forks when the calling process is already a process group leader (which interactive terminals guarantee for every background job). This makes runner_pid point to the dead parent that setsid discards, so kill -TERM -$runner_pid was targeting a defunct PGID — the entire stuck process tree survived as orphans. Verified with a 3-level (bash → launcher → trainers) simulated stuck runner: 4 orphan sleep 9999 processes confirmed after the setsid kill. Fix: use `set -m` (job control) immediately before the background launch and `set +m` immediately after. With job control active, bash assigns PGID = runner_pid to the job unconditionally, in both interactive and non-interactive contexts. Since flame/launch/spawner.py uses plain subprocess.Popen with no start_new_session or preexec_fn=os.setsid, all 300 trainer processes inherit the same PGID and are killed by kill -TERM/-KILL -$runner_pid. Verified: 6-process tree (runner, ticker, launcher, 3 trainers) all dead after kill, PGID=runner_pid in both interactive (bash -c) and non-interactive contexts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): clean termination + correct aggregator log grep Three fixes to _run_baseline: 1. Post-kill cleanup (timeout path) When SIGKILL fires, run_experiment_batch's finally block (_sweep_stragglers: pkill trainers/agg + 45s GPU wait) is in the killed process group and may not complete. Replicate it in bash: pkill -9 -f "trainer/pytorch/main.py" pkill -9 -f "aggregator/pytorch/main_" sleep $KILL_SETTLE_S # default 20s GPU memory drain so no stale trainer/aggregator processes or held GPU memory bleeds into the next run's allocation. 2. wait() order fixed Added wait after SIGKILL on timeout path; non-timeout path uses conditional wait so we never double-wait the same PID. 3. Grep checks scan the correct file AggregatorSpawner redirects the aggregator subprocess's stdout/stderr to experiments/run_*/..._aggregator.log (NOT $FLAME_LOGDIR/debug_run.out, which is only run_experiment.py's own print()s). Previous grep always returned 0 for "stopping run" / SIM_WALL_CEILING / SIM_STARVATION, causing every successful run to be classified as FAIL(no_stop). Fix: use -newer $ts_marker to find aggregator logs created during this run's lifetime; accumulate counts across them (handles multi-exp batches). Paths saved to $run_dir/agg_logs.txt for easy post-mortem access. Also adds --kill-settle-s CLI option and KILL_SETTLE_S=20 default. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(unavail): add T5-smoke section with 6h command + run count + hypotheses 23 jobs / 22 FL experiments (steps 1,2,4,5; step 3 skipped as redundant with step 4 sim half). Expected ~4h 38min wall, fits 6h with ~1h margin. Documents per-baseline predictions, starvation-at-n300 note (SIM_STARVATION absent is correct, not a failure), and failure watch-list for oort_star/fedbuff first live runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): grep-c double-output crash + in-place progress ticker - Fix arithmetic crash: grep -c always prints a count even for 0 matches and exits 1, causing || echo 0 to fire and produce "0\n0"; $(( N + 0\n0 )) is a bash syntax error that aborted _run_baseline before _record, silently dropping all runs after the first baseline. Fix: || echo 0 → || true. - Add per-run in-place ticker (stderr, \r): shows elapsed/budget/pct within the current run plus kill-in countdown and overall N/total done count. Overwrites in place each 5s tick — lets you spot stuck runs without log flood. - Add _compute_total_runs + TOTAL_RUNS/COMPLETED_RUNS globals; _record now increments COMPLETED_RUNS so the ticker stays accurate across all steps. - Move COMPLETED_RUNS increment into _record so pytest (step 1) also counts. - Drop nohup from the design-doc command (tmux keeps the session alive). - Expand T5-smoke hypotheses into a 23-row per-run table with Actual column. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): step4 ETA log used invalid ${#var//pat/} substitution Bash treats ${#var//x/y} as a parse error ("bad substitution"), which is fatal regardless of set -e/-u — it killed the whole suite right at the step 4 header, before any step 4/5 runs executed. Replace with wc -w to count baselines. * feat(availability): stamp real-mode vclock_now (A4dur fix) + rename ClientAvailability + B2.0.3 join-barrier re-anchor vclock_now was hardcoded None for real-mode selection telemetry, so A4dur's duration-weighted real/sim comparison fell back to a join-ramp-skewed origin; now stamped via ClientAvailability._avail_now() for both modes. Renamed AvailabilityMixin -> ClientAvailability (availability_mixin.py -> client_availability.py). Also fixes B2.0.3: agg_start_time_ts was stamped before the trainer join barrier, so real's trace-read clock baked in the join wait (~300s at n=300), reading the availability trace ~300s ahead of sim/ground-truth. Root-caused via feddance syn_50's A3 (avail_timebase) failure against the trace's own ground truth. Fixed via _mark_join_barrier_done(), which re-anchors agg_start_time_ts to real time once the join barrier resolves (real mode only, self-correcting to actual join duration) -- one fix point in syncfl/top_aggregator.py covers all six baselines via inheritance. Confirmed A4dur fix live on a fresh felix syn_20 real+sim pair (mean_err=0.0). B2.0.3 fix confirmation (fresh feddance syn_50 pair) still pending. 6 new regression tests in test_join_barrier_reanchor.py. Full suite green: 542 pass / 7 skip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --background flag for unattended overnight runs Re-execs the suite via nohup+disown and returns immediately, printing the PID, nohup log, and eventual report.txt path. Lets a smoke campaign survive the launching shell/SSH session closing, so it can be fired off and checked on later rather than needing a session held open for hours. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass -- A4dur confirmed, B2.0.3 root-caused+fixed, next-step command Whole-doc crisp pass per the Working Agreement: Status, T5-smoke, Challenges (#17 B2.0.2 was still marked open, now closed; new #18 for B2.0.3), and the known-parity-failures table all updated to reflect the A4dur confirmation and the new B2.0.3 root-cause + fix. Collapsed the old SCRATCH handoff section down to the still-open next steps (trackTrainerAvail cleanup, mobiperf live exercise, PR workflow) and dropped Q&A material now folded into permanent sections. Added a prominent NEXT STEP callout at the top of the doc with the exact unattended smoke_suite.sh command to confirm both fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --num-trainers to shrink the cohort below 300 Forwards --num-trainers through to every debug_run.sh invocation (debug_run.sh already auto-scales min_trainers_to_start accordingly). Lets a confirmation run use a smaller cohort under resource constraints without editing the parity config -- useful for mechanism-level fixes (e.g. B2.0.3's join-barrier re-anchor) that are n-independent by construction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): scope next-step confirmation run to n=100 Resource-constrained follow-up: A4dur and B2.0.3 are both mechanism-level fixes (already confirmed n-independent / self-correcting to actual join duration), so n=100 is sufficient to confirm them without reproducing the original n=300 failure magnitude. Notes the tradeoff -- not directly comparable to the existing n=300 parity_*.json snapshots for the still-open §7 checks, which stay gated on a real n=300 T5 run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(config): explicit encoding="utf-8" on YAML config reads (node-portability) open() without an explicit encoding falls back to the node's locale-preferred encoding. On a non-UTF-8 locale (e.g. C/POSIX without Python's UTF-8 mode coercion) this mis-decodes non-ASCII bytes in the parity YAML (an arrow character in a comment) byte-by-byte, and yaml.safe_load then rejects the resulting control characters with a ReaderError -- reproduced exactly with PYTHONUTF8=0 LC_ALL=C + encoding="latin-1". Fixed the crashing read (debug_run.sh's parity-config generator) plus the write/re-read round trip, and hardened the other YAML config reads in the availability substrate (trace.py's synthetic/mobiperf trace files, client_availability.py's trainer registry) against the same failure mode before it bites on a node with non-UTF-8 content in those files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(unavail): Batch 3 trace-fidelity & delay-enforcement overhaul (T3.0-T3.5) Adds absolute (vs. raw ground-truth trace) availability fidelity checks to complement the existing real-vs-sim relative checks, which is how the real-mode send-gate went dead code for the whole project without any check catching it (root-caused as T3.1a). - T3.0/T3.1a/T3.1b: shared trainer<->aggregator time origin (real mode), the debug_run.sh trace-wiring fix (trainer's own trace was never substituted), _refresh_avl_state() mode-dispatch cleanup. - T3.2: A6 trainer_trace_fidelity -- trainer's own avail_change telemetry vs. raw ground-truth trace. New scripts/parity/ground_truth.py; sim_now field added to avail_change. - T3.3: A7 agg_belief_fidelity -- aggregator's belief vs. ground truth, both selection and commit checkpoints. New agg_belief_change telemetry + ClientAvailability hooks. - T3.4: A8 send_gate_wait_fidelity -- observed vs. ground-truth-expected [SEND_GATE] wait duration. New send_gate_wait_s/send_gate_sct fields on task_send telemetry. - T3.5: K11 commit_promptness -- actual commit time vs. delivery_ts for withheld-then-delivered updates. New actual_commit_ts field on withheld_delivery telemetry. Each check ships with new telemetry, a parity rung in checks.py, a plot in analyze_run.py, and synthetic-data unit tests. Batch 3 is now code+test complete; UNAVAILABILITY_DESIGN.md's own plan calls for exactly one real run next (Phase 5) to generate live telemetry for all four new checks at once. Tests: lib/python/tests/ 574 passed / 7 skipped; examples/async_cifar10 scripts/parity/ + trainer/pytorch/ 121 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Batch 4 — felix real hang, sim-mode trainer clock freeze, A7 checker fix Phase 5/6's first real run against the new absolute fidelity checks (A6/A7/K11) surfaced three gaps; all three are root-caused, fixed, and unit-tested here: 1. felix real-mode TIMEOUT: D.1 proactive in-flight eviction (_sim_evict_unavail_inflight) was called only inside `if self.simulated:`, alongside the genuinely sim-only 90s vclock re-clock. But async_oort's selector derives `recv_ends` directly from `selected_ends`, so a stalled UN_AVL trainer never evicted in real mode never left recv_ends either, hanging the aggregator's 30s recv_fifo loop past its own budget. Un-nested D.1 from the sim-only gate in all three top_aggregator.py stacks (a no-op everywhere except felix, the only baseline with proactive_inflight_evict). 2. A6/A4dur/K6: sim-mode Trainer._sim_now() returns the frozen _sim_send_ts from its last dispatch, so a trainer withheld as UN_AVL can never observe later trace transitions while idle. Fixed in two zero-new-comms parts: (a) avail_change.sim_now now stamps the transition's own scheduled trace-time instead of _sim_now() at processing time; (b) inform_end_of_training's existing broadcast now piggybacks the aggregator's final vclock (sim mode only), and the trainer's shared _fetch_weights flushes one last _refresh_avl_state() catch-up on EOT. This also resolves the long-open K6 (`sim_send_ts==0`) mystery — same frozen-clock mechanism, not selector-starvation or a checker false positive. 3. A7 commit-checkpoint: not a system bug — the checker (_fidelity_score) extrapolated a trainer's last recorded commit-belief across the run's full span, wrongly scoring the silence after a trainer stops committing (typically because it went UN_AVL) as stale drift. Added extrapolate_tail=False for the commit-checkpoint call site, truncating the scoring window to [t_start, last observed t], symmetric with the existing start-side truncation. 12 new/updated tests (tests/ 586 pass/7 skip, scripts/parity/+trainer/pytorch/ 125 pass). Not yet confirmed on a live run — see UNAVAILABILITY_DESIGN.md's Batch 4 section for the re-confirmation command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): ASCII-only comments in Batch 4 source files (node portability) * debug(unavail): temp diagnostic logging for D.1 eviction-miss mystery Live re-confirmation of the Batch 4 fix 1 (D.1 proactive eviction in real mode) found the fix is necessary but not sufficient: felix real still hangs at n=100, stuck at one round for 10+ minutes with only a handful of the ~20 stalled trainers ever getting evicted. Two isolated reproductions against the real AsyncOortSelector/ClientAvailability code (single stalled trainer, and 40/100 simultaneously UN_AVL) both show the eviction mechanism working correctly every cycle, so whatever is different about the live run isn't reproduced by either repro yet. Adds an [EVICT_DEBUG] summary log line to _sim_evict_unavail_inflight (inflight count, evicted count, and per-skip-reason counts: still available / buffered / already committed-or-withheld / no trace) so the next live run shows which guard is actually firing instead of inferring it from AWARE_EVICT's absence. Log-only change, no behavior change. Remove once root-caused (see UNAVAILABILITY_DESIGN.md's "Open A" for tracking). Also documents Open B: a separate, pre-existing bug found while investigating A -- trace loading falls back to the shared syn_20 "pattern" for most trainers instead of their individually-assigned per_trainer entry (hand-verified on two trainers whose own traces shouldn't transition until t=13800s/t=34200s, but which actually transitioned at t=600s/1200s matching the fallback exactly). Not yet root-caused or fixed; deliberately investigating Open A first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): add PR-readiness checklist for fresh-context handoff Explicit top-to-bottom checklist (Open A eviction-miss root cause, Open B trace-fallback root cause, cleanup, re-confirmation run, then the pre-existing Open items 1-2) so a new session can pick up exactly where this one left off without re-deriving status from chat history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Open A/B — selector abandon leak, trainer trace mismatch Open A (felix real never self-stops): the real-mode 90s SEND_TIMEOUT_WAIT_S abandon in async_oort.py/fedbuff.py only cleared all_selected, never selected_ends. recv_ends derives from selected_ends, not all_selected, so a stalled end never left the recv_fifo wait set and max_experiment_runtime_s's self-stop check (gated on recv_ends being empty) was unreachable. Both selectors now also discard the end from selected_ends. Open B (trainers ignore their per-trainer trace): spawner.py's get_synthetic_trace never took a trainer_id, so every trainer process baked in the shared cohort-wide `pattern` instead of its own registry-assigned per_trainer trace, even though the aggregator's own belief-tracking already resolved per-trainer correctly via flame.availability.trace.load_trace. get_synthetic_trace now delegates to that same resolver when given a trainer_id. Trainer startup also logs an [AVAIL_TRACE] signature (trace_hash) so a shared hash across trainer_ids is visible directly in the logs. debug_run.sh's --trace now accepts a space-separated list (one experiment set per trace) to make cross-trace validation runs a single invocation. Tests: tests/selector/test_send_timeout_frees_selected_ends.py, tests/launch/test_config_generator.py::TestSyntheticTracePerTrainer, tests/launch/test_debug_run_trace_substitution.py::TestMultiTraceSubstitution — all verified to fail without their respective fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): strip non-ASCII chars from this session's additions Extends 994f8187's node-portability cleanup to the lines added in the last two commits (0514d427, 8d145865) that it predates: em-dash/en-dash/multiply/ approx/bullet glyphs in debug_run.sh comments and UNAVAILABILITY_DESIGN.md's NEXT STEP section, replaced with ASCII equivalents (--, -, x, ~). Scoped to those additions only, not the doc's older pre-existing prose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): fwdllm agg telemetry test used stale property key _FakeChannel.get_end_property() matched on the literal string "round_duration", but _process_aggregation_goal_met() actually looks up PROP_CLIENT_TASK_TRAIN_DURATION ("client_task_train_duration_s"). The fake channel silently returned None instead of the seeded duration, so trainer_speed_s/agg_observed_s always came back empty -- a fixture bug, not a production one (traced back to a pre-existing mismatch already present on dg-fork-main before this branch's rebase). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass on UNAVAILABILITY_DESIGN.md per its own house rules Condense fully-resolved sections (Batch 3 T3.0-T3.5, Phase 5/6 results, T5-smoke/A4dur/B2.0.3) to the doc's own stated format for completed stages -- mechanism + files + tests + exit, 2-3 lines -- dropping narrative/discovery play-by-play that's already preserved in commit history. Also updates the NEXT STEP run command to the actual planned first-pass confirmation run (1800s, syn_50 only, felix+fedbuff) ahead of the full 7h campaign. 1375 -> 945 lines, no content loss (full detail remains in git log for every referenced fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): trim verbose comments in client_availability.py + syncfl top_aggregator.py Comment-only cleanup, no behavior change (verified via targeted availability/ sim-ordering/sim-barrier test runs, 141 passed): - client_availability.py: removed the TEMP DIAGNOSTIC eviction-miss counters and [EVICT_DEBUG] log line in _sim_evict_unavail_inflight -- explicitly marked for removal once Open A was root-caused, which UNAVAILABILITY_ DESIGN.md now confirms. Trimmed several multi-paragraph docstrings (get_curr_task_ineligible_trainers, _avail_stamp_end_states, _emit_withheld_delivery, _sim_reinject_ready_withheld) down to the causal fact + pointer, dropping session-narrative framing. - syncfl/top_aggregator.py: trimmed _mark_join_barrier_done's docstring and the EOT-broadcast piggyback comment the same way. First batch of a broader pass across the dg-fork-main diff; the other top_aggregator.py variants (oort/asyncfl), analyze_run.py, and the test files still have similar cleanup pending. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): finish comment-cleanup pass across the dg-fork-main diff Comment-only, no behavior change (full suite green: 728 passed / 7 skipped, same as before this batch): - oort/top_aggregator.py: trimmed the trainer_recv_wall_timeout_s explainer and the cohort-floor guardrail comment; dropped an internal "Next actions §2" task-tracking reference (also in asyncfl/top_aggregator.py). - syncfl/trainer.py: trimmed the AGG_START_TS caching comment, the EOT avl_state catch-up comment, and the [SEND_GATE] block's docstring-length comment down to the causal fact. - async_oort.py: reworded the Challenge 13 cleanup comment to match the shorter version already used in fedbuff.py/async_random.py (all three carry the identical fix; no reason for one copy to be 3x longer). - message.py, examples/async_cifar10/trainer/pytorch/main.py, analyze_run.py: trimmed remaining multi-paragraph docstrings/comments down to mechanism + one-line rationale, dropping session/phase-tracking references ("Batch 3 T3.0", "Batch 4 finding 2") in favor of plain causal language -- that history lives in UNAVAILABILITY_DESIGN.md and git log, not duplicated inline. Completes the comment-cleanup pass across the highest-density files in the diff (client_availability.py + the 3 top_aggregator.py variants + trainer.py + analyze_run.py). Remaining lower-density files and test-file docstrings were left as-is -- most already follow the repo's convention of a short per-test docstring naming the specific regression it guards against, which is exactly the kind of comment worth keeping. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(launch): decouple spawn cohort size from dataset-split selector The launcher tied "how many trainers to spawn" and "which <name>_alpha<a>_n<N>.yaml split to read" to a single num_trainers knob, so a 10-trainer smoke demanded a dedicated n10 split file and KeyError-ed when only n300/n50/n48 existed. - experiment_config: add trainer.split_num_trainers (defaults to num_trainers) to pick the split partition independently of cohort size. - runner: split lookup uses split_num_trainers or num_trainers; the spawn cohort (trainer_ids) still uses num_trainers. - debug_run.sh: --num-trainers now preserves the config's native split size so a shrunk cohort reads the existing n300 split; add --alpha flag to choose dirichlet_alpha (0.1/1.0/10.0/100.0) against that n300 split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): bound A7 commit-checkpoint scoring by max_gap_s _fidelity_score held each sparse commit-checkpoint observation constant across interior gaps between commits, penalizing the absence of a mid-gap commit rather than any wrong belief. Add max_gap_s (wired to lag_tol_s at the commit call site): each observation vouches for its own state only up to max_gap_s past itself, on both sides; a transition with no covering window is excluded from both the TVD score and the missed/spurious diagnostic. Generalizes the earlier extrapolate_tail tail-truncation to interior gaps. Took felix 59/61 -> 62/62 on the live n=300 run. Tests: test_agg_belief_fidelity.py (interior-gap + wrong-belief-at-own-instant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(sim-unavail): fwdllm sim design, extend PARITY.md, trim UNAVAILABILITY_DESIGN, drop generated artifacts - PARITY.md: generalize to a multi-example parity methodology; add §F fwdllm rung catalog (baseline matrix, modified rungs, variance-cadence / dynamic-K/C / forward-grad layers). - simulate_fwdllm.md: rewrite as the design-only build plan -- correct code anchors to flame-core fwdllm_aggregator.py, resolve the §C baseline matrix from the landed launcher configs, integrate unavailability as first-class, surface decisions D1-D4. - UNAVAILABILITY_DESIGN.md: trim 1027 -> 283 lines to open items + next-steps + durable reference (baseline matrix, flags, v1 decisions, land-mines, dead-ends); collapse landed history to pointers; add the parallel-branch validation note. - Remove generated, unreferenced parity_{felix,oort,feddance}.json (-2152 lines; already covered by .gitignore). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds client unavailability modeling to the FLAME FL simulator so a fast simulated run (virtual
clock, no real sleeps) reaches real<->sim parity with a wall-clock run, config-gated and default-OFF
(byte-identical when off). Built and landed across all six async_cifar10 baselines;
felix is confirmed at full parity (62/62 enforced checks, live n=300 syn_50) and self-stops cleanly.
Also ships the design-only plan for porting the same high-fidelity simulator + unavailability to the
fwdllm example (fluxtune / fwdllm / fwdllm++); no fwdllm feature code is in this PR.
What's in this PR
Mechanism (code + tests, all baselines):
flame/availability/{trace.py, client_availability.py}(
ClientAvailabilitymixin) inherited by the syncfl / asyncfl / oort aggregator stacks.(felix only); starvation vclock-advance with self-termination.
scripts/parity/ground_truth.pyand a canonical trainer<->aggregator time origin, alongside the existing relative rungs.
Docs:
examples/async_cifar10/PARITY.mdextended into a multi-example methodology doc: new §F fwdllmrung catalog (modified rungs + the variance-cadence / dynamic-K/C / forward-grad layers).
examples/fwdllm/simulate_fwdllm.mdrewritten as the fwdllm simulator build plan (design only):corrected code anchors, resolved baseline->config matrix (§C), unavailability integrated as first-class,
4 open design decisions (D1-D4) surfaced. References PARITY.md §F for rung definitions.
examples/async_cifar10/UNAVAILABILITY_DESIGN.mdtrimmed (1027 -> ~283 lines) to open items + next-stepshistory collapsed to pointers.
Diff hygiene:
parity_{felix,oort,feddance}.json(-2,152 lines; already covered by
.gitignore).Not in this PR (follow-up)
sim+real, n=300) -- parallel branch; fixes merge back to
dg-fork-mainas they land.(shared-resource investigation + isolated re-run pending; see UNAVAILABILITY_DESIGN.md Status).
Test / verification
cd lib/python && conda run -n dg_flame python -m pytest tests/cd lib/python/examples/async_cifar10 && conda run -n dg_flame python -m pytest scripts/parity/ trainer/pytorch/python -m scripts.parity.cli --batch --experiments-dir experiments --baselines felix --agg-goal 10(62/62).