Add satellite RTT + A40 compute time delay model - #66
Open
avasadasivan wants to merge 20 commits into
Open
Conversation
dhruvsgarg
reviewed
Jun 27, 2026
dhruvsgarg
left a comment
dhruvsgarg
left a comment
Owner
There was a problem hiding this comment.
Thanks for the PR, please review these comments and address them in code directly (the ones you agree with). you can ping me about questions/ concerns about the ones you dont.
| @@ -0,0 +1,17 @@ | |||
| import yaml | |||
Owner
There was a problem hiding this comment.
It'll be cleaner to move these python scripts that generate metadata into a separate folder. say _metadata/scripts/
| speed_class: fast | ||
| mobiperf_device_id: device_001 | ||
| computation_time_ms: '2800.0' |
Owner
There was a problem hiding this comment.
The computation_time_ms and training_delay_s seem to be required to communicate the same thing (training time at the client/satellite) but are in 2 different variables. we can rename training_delay_s to computation_time_ms itself and retain the values (for mobile device and gpu_a40) currently held by training_delay_s?
| speed_class: fast | ||
| mobiperf_device_id: device_001 | ||
| computation_time_ms: '2800.0' | ||
| satellite_index: 0 |
Owner
There was a problem hiding this comment.
The satellite_index might be used for it to rightly pick up the latency distribution from the .npy tracefile that you have? if so, we should have the path for that npy file in the base_trainer.yaml so that we dont hard-code the path in the main.py as you currently have? we want to make the code runnable for anyone who uses the system, so we should avoid any path hardcoding. If I have mistakenly done some hardcoding in a few parts, we should remove that too as a separate fix
| dirichlet_alpha: 100.0 | ||
| availability: | ||
| mode: syn_20 | ||
| mode: syn_0 |
Owner
There was a problem hiding this comment.
For some small changes like this, you need not push. But in case you do change and push, please ensure that the file name remains consistent with the metadata. In this case, it would be confusing to someone later since the file name says syn_20 but within the file the availability mode has been changed to syn_0
| self.computation_time_ms = float(self.config.hyperparameters.computation_time_ms) | ||
| self.satellite_index = int(self.config.hyperparameters.satellite_index) | ||
| self.satellite_latencies = np.load( | ||
| "/home/dgarg39/ava/flame/lib/python/examples/_metadata/satellite_latencies.npy" |
Owner
There was a problem hiding this comment.
this hardcoding can be removed based on the previous comment. also, to keep the variable name and what it remains unambiguous (since we have compute and communication latencies), should we rename the variable to satellite_rtt_latencies_ms?
| elapsed = time.time() - self.start_time | ||
| timestep = min(int(elapsed), self.satellite_latencies.shape[0] - 1) | ||
| current_rtt = float(self.satellite_latencies[timestep, self.satellite_index]) * 2 | ||
| _modeled_delay_s = (self.computation_time_ms + current_rtt) / 1000 if self.training_delay_enabled else 0.0 |
Owner
There was a problem hiding this comment.
There is likely a bug here due to the duplication of the variables. the training_delay_s which has the mobile/a40 values is not used here and instead you are using the fixed but other value of computation_time_ms from the same yaml? I think the current value of computation_time_ms should be updated as specified in the previous comment.
Owner
There was a problem hiding this comment.
dont need to commit this, can you please add it to the root git ignore to not commit any .py.save files? they are temporary editor files
| config["hyperparameters"]["training_delay_s"] = trainer_meta["training_delay_s"] | ||
|
|
||
| # Set training_delay_enabled from overrides (default True) | ||
| config["hyperparameters"]["computation_time_ms"] = trainer_meta["computation_time_ms"] |
Owner
There was a problem hiding this comment.
Please make appropriate changes to spawner.py and config.py based on the final variable for training/ compute time as discussed above. since we might be renaming/ removing training_delay_s put a one-liner note saying that it was deprecated in favour of separating out compute latencies on different devices.
On that note, should we also separate out the communication latencies into rtt_ms and bandwidth?
Essentially, total_modeled_training_delay_ms = compute_on_device_ms + communication_latency_ms
the compute directly depends on the device type. the communication latency depends on three things: rtt time, bandwidth and size of data moved. youve incorporated rtt already, for now take dummy values of bandwidth in mega-bits-per-second and data-moved in megabits (that evaluates to say a fixed 10ms time cost). you should be able to see the trainer calculate and delay the response accordingly. Please ping if you have questions.
…t_amplitude, rtt_period_s)
…s); deprecate training_delay_s
…te_latencies.npy to _metadata/leo/, add README, remove obsolete scripts
…to config and load dynamically in main.py
… (computation_time_ms + rtt_ms) / 1000
…to match filename
* update launcher script with latest changes of async cifar
* fwdllm migration to launcher plan
* checkpooint1
* checkpooint2
* checkpooint3
* checkpooint4
* smoketestA
* checkpooint5
* checkpooint6
* plan updates and some metadata updates from before
* checkpooint7
* checkpooint8 and smoke test B
* checkpoiioint9
* checkpoint10
* smoketestC
* checkpoint11
* checkpoint12
* checkpoint13
* smoketestD
* redesign to clean up baselines across works
* Implement fwdllm baseline taxonomy (fwdllm/fwdllm_plus/fluxtune/fluxtune_dynkc)
Phase 7 of the launcher migration: makes _validate_stack selector-driven for
the fwdllm stack (and fixes a latent bug where its regex never matched the
real main_fedfwd_agg.py entrypoint), adds an explicit learning_rate kwarg to
FedBuff, re-sources oracular availability from _metadata instead of legacy
json_scripts/, and adds a reselect_each_iteration selection-granularity gate.
Replaces the fedfwd_async_random_dynkc/fedfwd_oracular baselines with the
owner's three-way taxonomy (fwdllm, fwdllm_plus, fluxtune) plus a preserved
fluxtune_dynkc parity variant, with fluxtune's K/C policy made config-driven
(dynamic_kc enabled/disabled + policy) rather than hardcoded fixed K/C.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add fwdllm_plus/fluxtune smoke YAMLs and restructure migration doc chronologically
Adds the two missing per-baseline smoke-test experiment YAMLs (fwdllm_plus,
fluxtune) alongside the existing fwdllm one, so all three owner-spec
baselines get real/sim parity coverage going forward, with a regression test
(TestFwdllmSmokeYamlsResolve) that resolves and validates each against the
real aggregator entrypoint.
Rewrites MIGRATION_TO_LAUNCHER_FWDLLM.md into strict chronological order
(Context/Phase1-5/Smoke A-D/Phase6/Phase7, previously Phase6-7 were
prepended ahead of older phases) with a single "Status & next step" section
at the top documenting Phase 7 step P8 (live smoke tests) as the sole
remaining item, including the full pinned-ML-stack (adapter-transformers /
tokenizers Rust-toolchain) blocker writeup and exact run commands, so a
future session can resume from this file alone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Reconcile a single dev env for fwdllm + vision/speech examples
Migrate fwdllm onto a modern, single dependency stack so one conda env runs
async_cifar10, async_google_speech, and fwdllm — replacing the un-buildable
adapter-transformers fork.
setup.py ([examples] extra is now the single source of truth):
- Add the NLP forward-mode stack (transformers>=4.57<4.58, adapters>=1.3<1.4,
h5py, pandas, scikit-learn, setproctitle) and torchaudio (google_speech),
so `pip install -e lib/python[examples,dev]` provisions one env for all.
Code (modern `adapters` add-on; keeps peft_method: adapter on DistilBERT):
- tc_transformer_trainer_distribute.py: import AdamW from torch.optim (removed
from transformers top-level in v4.x).
- expts/initializer.py: import adapters/BnConfig/LoRAConfig from `adapters`;
adapters.init(model) in the adapter/lora branches; build the bottleneck
adapter via BnConfig(**adapter_config) (legacy keys map 1:1).
Verified on py3.11/torch2.12/numpy2.4: stack installs from prebuilt wheels
(no Rust/tokenizers compile), both launcher entrypoints import past the old
wall, the adapter path freezes the base (~0.27% trainable), and the
launch/mode/optimizer/selector suites stay green (315 passed, 7 skipped).
Docs:
- New single setup+run guide at lib/python/README.md (covers all 3 examples).
- Updated prerequisites/quickstarts/example READMEs to note the extra now
covers the NLP/speech examples.
- fwdllm/MIGRATION_TO_LAUNCHER_FWDLLM.md: Phase 8 (env reconciliation) DONE;
P8 live smoke is the next step.
- DEPRECATED_FILES.md manifest + req.txt deprecation banner (cleanup deferred
to a dedicated commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix fwdllm smoke-test crashes + CUDA 12.9 env setup
- FedSgdTrainer: replace ast.literal_eval on avl_events_* with
_parse_avl_events() -- YAML delivers lists, not strings
- Aggregator: add manual_seed to aggregator.config_overrides in all
three smoke YAMLs so main_fedfwd_agg.py can read it directly
- setup_env.sh: force-reinstall torch/torchvision from cu126 index
(required for CUDA driver >= 12.9)
- setup.py: note cu126 requirement in examples extra comment
- .gitignore: ignore lib/python/examples/fwdllm/experiments/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix remaining fwdllm smoke crashes + establish configs/ design invariant
Per-example config templates belong in configs/, not _metadata/:
- configs/aggregator_base.json: new fwdllm aggregator template with all 25
NLP/model fields (model_name/type, peft_method, fp16, manual_seed, etc.)
that main_fedfwd_agg.py reads; shared _metadata/aggregator_base.json has
none of these (generic FL fields only)
- smoke YAMLs: switch config_template to ../configs/aggregator_base.json;
drop manual_seed override (now in the template default)
- configs/trainer_base.yaml: add avl_events_syn_train_* defaults ([[0,
"AVL_TRAIN"]] = always-available) for three legacy traces that
FedSgdTrainer.__init__ reads unconditionally but spawner never injects
- runner.py: accept .yaml/.yml config templates (extension-sniffed) alongside
.json, enabling future migration of aggregator_base.json to YAML
MIGRATION_TO_LAUNCHER_FWDLLM.md: document design invariant as Phase 9 +
record all four P8 live-run bug fixes for the cross-example rule-set doc.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix config_template path: configs/ not ../configs/
config_template is resolved as example_dir/path, so the correct path
is configs/aggregator_base.json (relative to fwdllm/), not ../configs/
which would look for examples/configs/ (doesn't exist).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix two more fwdllm smoke crashes
- aggregator_base.json: add data_loader_num_workers, client_num_in_total,
warmup_ratio (full coverage -- verified against all config.hyperparameters.*
reads in main_fedfwd_agg.py)
- FedSgdTrainer: replace hardcoded ../../.. path in _write_client_data_to_file
with self.args.output_dir/client_data_files (/tmp/ by default)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Gate minInitialTrainers assertion on is_async in fwdllm_aggregator
sync/random baselines (fwdllm, fwdllm_plus) have no minInitialTrainers in
selector kwargs; only fluxtune (async_oort) does. The unconditional assert
blocked all sync smoke runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* wip changes, context saved in launcher markdown files
* ready for next smoke tests, misc: data partition files for cifar
* updates as of 7.30pm est
* smoke tests and stopping criteria
* fix fwdllm bug
* all 3 working
* bug fix
* trim comments
* PR comment adressed: Remove files that we not longer needed.
* Updated .md files with the learnings from actual code changes
* recover required cifar files
* Fix F+/FT smoke-test crashes: var-check crash guard, per-baseline staleness policy, GPU undersubscription
- fwdgrad_utils.calculate_var/calculate_real_var: guard n<2 (matches the
n<2 guard pattern already used by calculate_snr/calculate_cv) instead of
crashing on torch.stack([]). Root-caused to a stale trainer update
racing into a freshly-cleared grad_for_var_check_list at a data_id
transition, which crashed the fwdllm_plus aggregator ~30min into the
09-29 21:03 smoke run.
- Add Hyperparameters.staleness_policy (flame/config.py) with three modes
-- "exact" (must match the aggregator's current round/data_id/iteration),
"round_data_id" (must match round/data_id, any iteration), "none" (no
gate) -- superseding the old reject_stale_updates boolean for FedFwd.
Wired into fwdllm_aggregator._process_single_trainer_message and set per
baseline: fwdllm=exact, fwdllm_plus=round_data_id, fluxtune=none. The
round_data_id policy would have rejected the stale update that caused
the crash above instead of silently mixing it into a newer cycle.
"exact" needed ITERATION_PER_DATA_ID echoed back in the trainer's
GRADIENTS message (fwdllm_trainer.py), which the wire protocol didn't
carry before. Removed the now-redundant/misleading
rejectStaleUpdates: false from the three FedFwd baselines that set
stalenessPolicy explicitly; left it as-is on fluxtune_dynkc and other
non-FedFwd examples that still rely on the plain boolean.
- expt_scripts/*.yaml: default execution.num_gpus to 8 instead of 1 --
each smoke YAML's default is sized for its own 10-trainer count, and
run_sequential.sh's --num-trainers scale-up had no matching GPU
override, so 100 trainers crammed onto 1 GPU OOM'd almost immediately
(the fluxtune_n10_smoke 09-29 22:11 run). Also add --num-gpus and --only
flags to run_sequential.sh, and have it rewrite the _nNN_ token in
exp.name when --num-trainers overrides the count, so the resulting
experiments/run_<ts>_<name> directory reflects the actual trainer count
instead of staying named after the YAML's checked-in default.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* run_sequential.sh: use plain baseline names as run identifiers, not yaml's n10-suffixed name
--only/RUNS keys are now "fwdllm"/"fwdllm_plus"/"fluxtune" instead of
"fwdllm_n10_smoke"/etc. -- the n10 was each source YAML's own checked-in
default trainer count, not the run's identity, so a 100-trainer
--num-trainers override still left the command itself saying "n10".
The exp["name"] rewrite (which feeds the run_<ts>_<name> experiment
directory) is now derived directly from this clean run key + the actual
--num-trainers value, rather than regex-substituting the nNN token out of
the source YAML's own name field -- more correct (independent of whatever
naming convention a given source YAML happens to use) and simpler.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* run_sequential.sh: keep job.id in sync with the renamed exp.name
Verifying the previous commit's generated YAML end-to-end (via
ExperimentBatch.from_yaml, the launcher's own loader) surfaced a field
patch_yaml missed: aggregator.config_overrides.job.id, which every
checked-in source YAML keeps equal to exp["name"] but which the n10->n100
rename wasn't touching, leaving it stuck on the stale "..._n10_smoke" id
even after exp["name"] was correctly rewritten.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fwdllm aggregator: log before the CUDA availability check, not after
torch.cuda.is_available() can block indefinitely if the GPU driver is
wedged, and it ran before logging.basicConfig() was even called -- so a
hung aggregator produced a silently empty log file instead of a log that
visibly stalls right after "checking CUDA availability...".
* fwdllm PR cleanup plan: add node-testing runbook + JSON->YAML migration audit
Adds the concrete commands for the next session (on a healthy GPU node --
jayne is still wedged) to run the pytest suite, smoke-test all three
launcher baselines at minimal length, and verify the migrated trace/config
data is wired correctly at runtime. Also records this session's static
JSON-vs-YAML field comparison (trainer_base.yaml, mobiperf_traces.yaml,
fluxtune_dynkc vs the legacy aggregator.json), five open questions it
surfaced, and a first-pass legacy-launch-path deletion candidate list for a
follow-up PR. Workstream A/B code changes remain uncommitted pending the
deferred test run.
* fwdllm: port missing syn_train_* availability traces into _metadata
Two of the three avl_events_syn_train_* traces (90_eval_10_unavail_0,
50_eval_30_unavail_20) had real per-trainer data sitting unused in the
legacy expts/run_tc_expts/json_scripts/trainer_*.json (150 trainers) that
was never ported into _metadata/availability_traces/synthetic_traces.yaml.
Ports all three (the third, 100_eval_0_unavail_0, is degenerate/always-
available like syn_0) as per_trainer.n300 entries, wrapping trainers 151-300
onto the 150 source trainers' traces to preserve the named distribution.
Verified against the JSON source with a temporary migrate/verify script
pair (run, confirmed pass, deleted). Spawner auto-injection for these keys
isn't wired up yet -- data correctness only, wiring is future work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: consolidate migration notes, trim changelog-style comments
MIGRATING_TO_LAUNCHER.md's fwdllm section (§9) predated this branch's most
recent commits, so it was missing several real lessons -- adds a subsection
covering the CUDA-before-logging ordering fix, the var-check n<2 guard,
staleness_policy's three modes, run_sequential.sh's num-gpus/naming
conventions, and a pointer to the now-available syn_train_* traces.
MIGRATION_TO_LAUNCHER_FWDLLM.md (1249 lines, a dated phase-numbered dev
journal) is now a short pointer stub at §9 + git log, matching this repo's
DEPRECATED.md convention instead of leaving two overlapping docs.
Trims verbose changelog-style comment blocks (logic unchanged) in
run_sequential.sh and fwdllm_aggregator.py, and fixes the repo-wide
references to MIGRATION_TO_LAUNCHER_FWDLLM.md's now-removed Phase
N/decision DN numbering that the doc consolidation above left dangling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: fix stale flame import, drop hardcoded conda env
PYTHONPATH now prepends this checkout's lib/python so a stale
`pip install -e` editable install pointing at a different flame clone
can no longer shadow flame.launch. ENVNAME now comes from
FLAME_CONDA_ENV or the shell's already-active CONDA_DEFAULT_ENV
instead of a hardcoded default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: broadcast EOT in sync-mode compose too
The sync-mode compose() branch built its tasklet chain from scratch
and dropped the inform_end_of_training tasklet (only present as a
stale comment), unlike the async branch which keeps it. When a
sync-mode aggregator's loop ended (e.g. max_data_id_progress /
max_runtime_s early-stop), it exited without ever broadcasting EOT,
leaving trainers blocked in await_join with no way to self-terminate
-- the launcher's wait_all() then had to force-kill each one after a
30s timeout, turning a ~90s aggregator finish into several extra
minutes before the batch could move to the next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* spawner: wait_all() polls trainers concurrently, not sequentially
wait_all() waited on each trainer one at a time (proc.wait(timeout=30)
in a for loop), so when every trainer misses the aggregator's one-shot
EOT broadcast (a real race: await_join() only catches peers already
joined at broadcast time, and has no timeout), the batch paid
30s * num_trainers before moving to the next sequential run --
10 trainers turned a ~90s aggregator finish into ~390s wall time.
Now all trainers share a single timeout_per_trainer window, polled
concurrently, so the worst case is one 30s wait regardless of trainer
count.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: add --partition-method override
Needed to run a heterogeneous (non-IID) data split for the longer
convergence soak run, instead of the smoke-test default of "uniform"
(IID, chosen for smoke tests specifically to isolate launcher-mechanics
validation from data-skew effects). Patches
hyperparameters.partition_method on both the trainer and aggregator
config_overrides, same pattern as the existing --num-trainers/--c/--k
overrides.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: clean up dead/confusing fields in aggregator_base.json
Removed the unused top-level "dependencies" field (Optional in the
generic ExperimentConfig schema, safe to drop) and hyperparameters
"batchSize"/"learningRate" (alias into Hyperparameters.batch_size/
learning_rate, but no fwdllm code path reads those -- only
scaffold/trainer.py and datasampler/fedbalancer.py do, neither used by
fwdllm). The top-level "dataset" field can't be removed outright --
it's required (non-Optional) on the schema -- so its value was changed
from a misleading MNIST URL to an explicit "unused" placeholder.
Verified by round-tripping the edited JSON through the real
flame.config.Config parser.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm PR cleanup plan: resolve open questions, split off deletion tracking
All 4 open questions from the static JSON->YAML comparison pass are
resolved (rounds=1000 kept, dead config fields cleaned up, 3 legacy
async json variants confirmed droppable, fl_main.py x2 confirmed dead).
Moves the deletion candidate list into its own DELETION_CANDIDATES.md
so it survives PR_CLEANUP_PLAN.md's deletion once this PR merges.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector: fix SEND_TIMEOUT_WAIT_S reclaim so stuck ends actually free their slot
async_oort's reclaim ran after the extra==0 early-return, so once concurrency
saturated with stuck trainers the reclaim could never execute -- a full
deadlock at scale (reproduced at n30/~90min, not at n10/10min). It also only
deleted from all_selected, never from selected_ends, which is what the
concurrency accounting actually counts. Moved the reclaim to the top of
_handle_send_state and free both dicts together, mirroring random.py's
already-correct pattern. Same two-part bug existed in async_random.py;
fedbuff.py selector only had the incompleteness half (no early-return gates
it).
* asyncfl top_aggregator: add per-cycle duplicate-contribution guard
The selector reclaim fix makes a stuck trainer's slot reopen while its
original response may still be in flight -- previously not reachable since
the slot never actually reopened. The generic asyncfl aggregator (used by
async_cifar10's fedbuff/async_random baselines) had no per-cycle dedup,
unlike fwdllm's _per_agg_trainer_list guard. Add _agg_cycle_contributed_ends,
reset once per aggregation cycle in _reset_agg_goal_variables, checked in
_aggregate_weights before counting a contribution.
* fwdllm_aggregator: prune departed ends from per-round reselection cache
reselect_each_iteration=False accumulates picks into _round_selected_ends
and reuses that exact list for the rest of the round, bypassing select()
(and its SEND_TIMEOUT_WAIT_S reclaim) once the cache is full. A trainer that
disconnects or reports UN_AVL mid-round was never pruned from this
aggregator-level cache, so the round would stall forever waiting on a
contribution that can never arrive. Add _prune_departed_from_round_cache,
called before the cache-size check, so a departed end shrinks the cache
below target and the existing accumulate path backfills it.
* run_sequential.sh: add --agg-goal/--c-async/--min-initial-trainers/--avail-trace
Lets one invocation run a genuinely controlled comparison across all three
fwdllm baselines: --agg-goal decouples aggregator.agg_goal from concurrency
(previously --c conflated them), --c-async overrides concurrency for the
async baseline (fluxtune) only, --min-initial-trainers overrides
minInitialTrainers directly, and --avail-trace overrides the availability
trace across all three baselines' real trace-consuming fields so selection/
aggregation bugs can be isolated from trace-driven scarcity (e.g. syn_0).
Also gitignore smoke-test/parity-report artifacts left over from this
session's runs, and record the round-2 baseline investigation (deadlock
root cause + fixes, invariant re-check, stale-cache gap) in
MIGRATION_TO_LAUNCHER_FWDLLM.md.
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record commit hashes, correct next-step order
The doc was written before this session's fixes were actually committed;
update the commit-status lines and clarify the next step is verifying
run_sequential.sh configs for fluxtune/fwdllm_plus/fwdllm (not feddance,
which is an async_cifar10 baseline, not one of fwdllm's three).
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record push/config verification, plan Part 5 (telemetry/analysis parity)
Updates stale status (push done, 1.5h n=100 run verified and launched,
catching a real --max-data-id early-termination bug in the process) and
adds Part 5: a cross-example telemetry/analysis parity plan. scripts/
analysis/analyze_run.py was written for async_cifar10 and only informally
generalized -- it hardcodes that example's model size and assumes round is
the sole, fine-grained progress unit. Neither holds for fwdllm (round
advances once per 150 data bins; real progress lives in data_id/
iteration_per_data_id), and fwdllm's aggregator emits no telemetry at all
today, so performance/insights plots can't exist regardless of plotting
fixes. Also flags an unconfirmed telemetry cross-contamination anomaly
(foreign selector name + test-like fingerprints in real run telemetry)
that must be root-caused before trusting analysis built on existing data.
Part 5 tracks this as subtasks P5.1-P5.7, not yet started.
* fwdllm_aggregator: emit agg_eval/agg_round telemetry per aggregation cycle
The aggregator previously emitted zero telemetry of any kind, making
plots/performance/ and plots/insights/system/ structurally impossible
regardless of any analyzer fix. Both events are wired into
_process_aggregation_goal_met (the single per-cycle entrypoint shared by
fwdllm/fwdllm_plus/fluxtune), snapshotting contributor/staleness/utility/
speed stats before self._per_agg_trainer_list is reset and before
self._model_version potentially advances later in the same call, so
staleness is computed against the model version contributors actually
trained against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector/random: emit selection telemetry from the real select() decision
RandomSelector (the actual aggregator-side FL-selection algorithm fwdllm/
fwdllm_plus use) never called emit_selection(), unlike every sibling
selector (oort/refl_oort/async_oort/feddance/fedbuff). Every "selection"
event that existed for those two baselines came exclusively from the
trainer's own trivial 1-candidate channel selector, never from the real
"which trainers got picked" decision. Adds the emit_selection(...) call in
select()'s SEND branch, right after selected_candidates is finalized.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* analyze_run: generalize progress axis + comm-cost via per-example manifest
analyze_run.py was written for async_cifar10 and generalized only
informally: it hardcoded that example's model param count and assumed
`round` is the only, fine-grained progress unit. Neither holds for
fwdllm-family examples, where `round` only advances once all
total_data_bins sub-units finish, collapsing hundreds of events onto one
x-value.
Adds an optional per-example telemetry_manifest.yaml (model_param_count,
progress_hierarchy, event_categories), loaded via
_find_manifest_path()/load_manifest()/configure_from_manifest() and applied
to progress_key() (generalized from a single hardcoded data_id fold to a
mixed-radix fold over the declared hierarchy) and MODEL_MB. An example with
no manifest (async_cifar10) is unaffected -- confirmed via a real-run
regression check. write_summary() cross-checks the manifest's declared
event_categories against which plots/<category>/ dirs actually got files,
flagging MISMATCH/DRIFT.
Wires in fwdllm's own manifest: 450,340 trainable params (not the full
66.8M-param model -- FedFwd's forward-mode/JVP scheme only ever puts the
trainable subset on the wire), data_id(150)/iteration_per_data_id(15)
hierarchy, and an evidence-based event_categories audit. Verified against
real GPU telemetry from the n=100 3-baseline experiment: manifest correctly
resolved, all 8 declared categories read "ok", zero MISMATCH/DRIFT.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* scripts/analysis: add README with onboarding checklist for new examples
Documents the 8 plot categories and what telemetry each needs, the
per-example telemetry_manifest.yaml schema, a 7-step checklist for wiring
telemetry into a new flame.launch example, and gotchas found while
onboarding fwdllm (the trainer-side selection.selector channel-
implementation artifact, the stale-reselection-cache bug class).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 5 completion, GPU experiment results
The 3-baseline (fluxtune/fwdllm_plus/fwdllm) n=100 GPU experiment finished:
no deadlock recurred (fluxtune, 3x the scale it stalled at), and the
fwdllm_plus round-count throttle did not reproduce. Manifest wiring
verified against the run's real telemetry (clean event_categories check,
zero MISMATCH/DRIFT). All of Part 5 (P5.1-P5.7) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector/random: attach data_id/iteration_per_data_id to selection events
fwdllm_aggregator.py already threads (model_version, data_id, iteration_id)
through channel.ends(agg_version_state=...) -> select()'s kwargs, but
nothing consumed it. select()'s SEND branch now reads
kwargs["agg_version_state"] and attaches data_id/iteration_per_data_id to
the emitted selection event when present, letting analyze_run.py's
progress_key() place selection-derived plots on the same fine-grained axis
as trainer_round/agg_round/agg_eval instead of collapsing onto fwdllm's
coarse round. No-op for callers that don't pass agg_version_state (e.g.
async_cifar10's fedavg baseline also uses this selector).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: emit utility_belief; agg_round carries agg_observed_s
fwdllm_aggregator.py never emitted utility_belief (only
asyncfl/top_aggregator.py did), so selected_utility_believed_vs_actual*/
selected_utility_belief_gap* were structurally impossible for fwdllm-family
baselines regardless of selector. _process_single_trainer_message's
STAT_UTILITY branch now emits it before overwriting PROP_STAT_UTILITY
(believed = the prior value, actual = this message's fresh value), with
staleness computed against self._model_version (not self._round, which can
sit at 1 for an entire run) to match agg_round's own staleness convention.
_process_aggregation_goal_met now also builds agg_observed_s (an
{end_id: seconds} dict) from the PROP_ROUND_DURATION values already read in
the existing per-contributor loop, unlocking runtime_agg_vs_trainer/
runtime_overhead_hist/runtime_overhead_cdf -- a genuine wall-clock
measurement already computed, not an invented one.
build_utility_belief gained an extra param (mirroring build_trainer_round)
so fwdllm can attach data_id/iteration_per_data_id for progress_key().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* FedSgdTrainer: report sim_round_duration_s (gpu time + emulated delay)
fwdllm's trainer reported only real_gpu_time_s, unlike async_cifar10's
trainer which reports total round wall time. _emulate_training_delay() now
returns the seconds actually slept (0.0 if delay emulation is disabled);
train_with_data_id adds this to real_gpu_time_s and reports it as
sim_round_duration_s.
Deliberately not added: training_budget_s/overran/remaining_time_s.
fwdllm's delay is a flat additive sleep, not async_cifar10's budget-minus-
actual sleep-to-fill model, so there's no faithful "overrun" concept to
report -- fabricating one would be misleading rather than merely
incomplete. The budget/overrun-family plots stay not_populated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* analyze_run: re-key accuracy/loss/comm/selection plots to progress_key()
accuracy_by_round/loss_by_round/cumulative_comm_by_round previously stayed
on plain round deliberately (P5.2), since folding data_id into only one
side of comm_vs_accuracy_series's cumulative join would have silently
broken it -- selection events didn't carry data_id at the time. Now that
selector/random.py attaches it, all three (and the selection-derived
selection_funnel_over_rounds/selection_count_consistency/
eval_vs_train_selections/selected_speed_utility_*/
selected_utility_belief_gap_over_rounds) are consistently keyed by
progress_key(), so the join needed no code change -- just consistent
inputs on both sides.
Iteration-level granularity (not just data_id-level) chosen after checking
real n=100 telemetry: iteration_per_data_id varies 0..3 in practice, giving
73 distinct buckets vs. 31 at data_id-only resolution, ~11 events/bucket on
average. New PROGRESS_AXIS_LABEL constant applied everywhere a plot uses
this axis, including 3 pre-existing call sites whose label was never
updated when they were wired into progress_key() in an earlier session.
Verified end-to-end against synthetic fwdllm_plus-shaped telemetry (all
targeted plots populate with no errors) and a real async_cifar10 run (no
regression, progress_key() still a no-op there).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 6 (plot-coverage gap, fixed)
Root-caused why fwdllm's real n=100 plots were much thinner than a real
async_cifar10 run's despite Part 5's telemetry fixes: fwdllm's cache-reuse
selector fires once per round (not continuously), the trainer reports far
fewer timing fields, accuracy/loss stayed on plain round, and utility_belief
was never emitted. All four fixed and verified end-to-end (full suite: 470
passed, 7 skipped, 0 failed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: bound recv_fifo waits so max_runtime_s can't be starved
_aggregate_grads_async (fluxtune/async) and sync_collect_and_accumulate_
grads (fwdllm/fwdllm_plus/sync) both called channel.recv_fifo() with no
timeout, which defaults to blocking forever. If a selected/in-flight
trainer went quiet, the call never returned -- confirmed via a real
fluxtune run (run_20260701_225428_fluxtune_n10_smoke) that logged nothing
for 11+ minutes then had to be manually killed, despite --max-runtime-s 600.
Root cause: _check_early_stop_conditions() (which enforces max_runtime_s/
max_data_id_progress) only runs from _distribute_weights, on the other side
of the composer's put >> aggregate loop. While blocked inside the aggregate
call, the loop never cycles back to re-check it.
Both call sites now pass timeout=RECV_TIMEOUT_WAIT_S (30s), matching the
pattern asyncfl/top_aggregator.py's _aggregate_weights already uses for the
identical reason. A timed-out recv already degrades gracefully in both
methods (yields (None, ...), treated as "no data this tick") -- no other
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 7 (max_runtime_s starvation fix)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* PR_CLEANUP_PLAN.md/MIGRATION_TO_LAUNCHER_FWDLLM.md: reconcile PR-readiness state
PR_CLEANUP_PLAN.md predated the post-migration investigation (Parts 2-7 of
MIGRATION_TO_LAUNCHER_FWDLLM.md) and no longer reflected reality: 3 more
real bugs found and fixed since its last update (fluxtune deadlock, asyncfl
duplicate-contribution gap, max_runtime_s starvation), the planned soak run
never actually happened (no niid_label_clients run exists), and the
originally-planned diff base (dg-fork-main) turns out to include an
unrelated ~20-commit async_cifar10 launcher initiative this branch was
built on top of, not this PR's own content. Rewrote "what's left before
merge" to reflect actual current state, flagged a scope decision (does the
telemetry/analysis tooling work belong in this PR), and corrected the
recommended diff base (35f8d65, not dg-fork-main). Also noted an exposed
GitHub PAT in the origin remote URL, unrelated to this PR but surfaced
while checking git state.
MIGRATION_TO_LAUNCHER_FWDLLM.md: updated the top summary through Part 7,
added a cross-reference to PR_CLEANUP_PLAN.md, corrected the "100%
availability" premise question with real trace data, and recorded the
Part 7 GPU re-validation run as in-progress.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: retire PR_CLEANUP_PLAN.md, confirm Part 7 fix on GPU, explain n=100 gap
Part 7's max_runtime_s starvation fix is GPU-confirmed: the fluxtune re-run
(run_20260701_234833_fluxtune_n10_smoke, same mobiperf_3st_50 trace that
exposed the bug) self-terminated within budget with no manual kill needed.
Part 8: investigated why fwdllm's own n=100 run (Part 4 experiment) reached
only ~46% accuracy/data_id 29 while fwdllm_plus/fluxtune both passed 80%+
accuracy in the same 1.5h. Root-caused via direct log evidence (not
assumed): the aggregator's live candidate pool never grew past 30 of 100
trainers for the entire run ("Total ends: 30" on every cycle), and
"hasn't received weights" fired 9,680 times. Traces to --min-initial-
trainers 95 gating startup at n=100, combined with fwdllm's round-cached
reselection (Part 3) having no mechanism to replace a trainer that's
unresponsive but not formally departed -- unlike fwdllm_plus's per-
iteration reselect or fluxtune's continuous async reselection. A real,
distinct, scale-dependent gap in fwdllm's own selection design, not one of
bugs #4-7 -- documented as a non-blocking follow-up per user direction, the
convergence soak run is bypassed in favor of this existing n=100 evidence.
PR_CLEANUP_PLAN.md is removed from the branch (not just deferred to post-
merge as it originally said) -- its still-relevant content (bug list,
known limitations, scope decisions) is folded into a new "Merge readiness"
section in this doc. DELETION_CANDIDATES.md updated to match (it's the one
thing from that cleanup that survives, per the original plan).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: fold fwdllm's migration lessons into MIGRATING_TO_LAUNCHER.md, trim fwdllm doc to pending items only
MIGRATION_TO_LAUNCHER_FWDLLM.md had grown to ~1500 lines of investigation
narrative (evidence trails, exact log lines, commit-by-commit history) --
too big to read or maintain, and most of it was already resolved. Split:
- Durable, generalizable lessons (positive and negative) folded into
MIGRATING_TO_LAUNCHER.md's core sections: §2 gains aggregator gotchas
(unbounded recv_fifo blocking, selector reclaim ordering/completeness,
per-cycle dedup guards, cache-vs-selector state drift); §5 gains the
per-example telemetry manifest mechanism (progress_key()/
PROGRESS_AXIS_LABEL, model_param_count, event_categories) plus telemetry
gotchas (trainer-side selector artifact, subclasses must wire their own
telemetry.emit(), NO DATA placeholders looking populated at a glance);
§8 gains launcher/spawner gotchas (PYTHONPATH control, concurrent
trainer-wait polling, the await_join() race, sync/async compose()-branch
drift). §9 (fwdllm-specific) gains fwdllm's own specifics: run_sequential.sh's
controlled-comparison flags, the round-cached-vs-per-iteration reselection
design, the recv_fifo fix's concrete instance, and the telemetry manifest's
actual values.
- MIGRATION_TO_LAUNCHER_FWDLLM.md rewritten from scratch as a short pending-
items tracker (2 correctness/design follow-ups, 3 telemetry/analysis
improvements, 3 sanity checks) -- nothing blocking merge. Full history
stays in `git log` for that file, as the doc itself already pointed to.
Swept ~15 code comments across flame/selector/, flame/mode/, scripts/analysis/,
and various tests that referenced now-removed "Part N"/"P5.x" sections,
redirecting them to where the content actually landed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: evict a round-cache member that's stuck, not just departed
The per-round reselection cache (_round_selected_ends, reselect_each_
iteration=False) only ever pruned an explicitly departed trainer
(disconnected or UN_AVL). A trainer that's still formally connected but
silently stuck -- e.g. never finished receiving its initial weights --
occupied a cache slot indefinitely, since nothing else in this design
could tell it apart from a trainer that's just slow.
Confirmed as the root cause of a real n=100 run's stall: the aggregator's
working set stayed capped at 30 of 100 trainers for the entire 90-minute
run ("hasn't received weights" fired ~9,700 times), ending at only ~46%
accuracy vs. fwdllm_plus/fluxtune's 80%+ in the same window -- those two
don't share this gap since they reselect continuously instead of caching
per round.
Fix: track _round_cache_activity_ts per cached end (stamped on cache entry,
reset on every real accepted contribution in
_process_single_trainer_message); _prune_departed_from_round_cache now also
evicts a member that's gone ROUND_CACHE_STUCK_TIMEOUT_S (5 min) without one,
same timeout-based reclaim pattern as SEND_TIMEOUT_WAIT_S/RECV_TIMEOUT_WAIT_S
used elsewhere in this codebase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: add --avail-traces for unattended multi-trace overnight runs
--avail-trace only applies one trace to the whole --only baseline sequence
per invocation. Add --avail-traces (comma-separated) to loop the entire
sequence once per trace, all within one invocation/process -- for an
unattended overnight comparison (e.g. syn_0 then syn_20) without needing to
babysit and launch the next trace by hand.
Each (baseline, trace) run's name/job-id/log file are disambiguated by
trace when looping multiple (e.g. fwdllm_n100_syn_0_smoke vs.
fwdllm_n100_syn_20_smoke) so results never collide; single-trace/no-trace
invocations keep today's exact naming. --stop-on-fail now aborts the
remaining runs across all traces, not just the current one.
Dry-run verified against the real checked-in YAMLs (patch_yaml's python
heredoc extracted and run standalone, not an actual launch).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_trainer: fix uncaught KeyError on end-of-training broadcast
_fetch_weights logged msg[MessageType.DATA_ID]/msg[MessageType.ITERATION_PER_DATA_ID]
unconditionally, a few lines above the code's own existence guard for those
same keys. The aggregator's end-of-training broadcast only carries
{MessageType.EOT: ...}, so any trainer still waiting in _fetch_weights when
the aggregator wrapped up hit an uncaught KeyError and crashed instead of
reaching the EOT handling that sets self._work_done and exits cleanly.
Confirmed across three n=100 overnight runs (2026-07-02): every
idle/unselected trainer (82-90 per run) crashed with this exact KeyError at
the run's stop timestamp. Harmless to run correctness (those trainers were
already about to be swept by the launcher), but every run ended with dozens
of uncaught-exception tracebacks instead of clean shutdowns.
Fix: switch the log line to msg.get(...), matching the guarded-access
pattern already used a few lines below for the analogous WEIGHTS log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: confirm fwdllm merge-readiness via 2026-07-02 n=100 runs, trim fwdllm doc to a redirect
Three fresh 2h n=100 runs (fwdllm, fwdllm_plus, fluxtune, all syn_0) confirm
the round-cache-stuck fix holds under real GPU load: 0 "hasn't received
weights" messages (vs ~9,700 before), steady progress to data_id 135/150,
accuracy climbing 30%->67%, using the full max_runtime_s budget instead of
stalling. Telemetry manifests clean on all three; no unhandled exceptions
after the fwdllm_trainer.py KeyError fix in the previous commit.
MIGRATING_TO_LAUNCHER.md: record the GPU-validation result on the
round-cache fix, fold in the new KeyError bug+fix, fold in the two
still-open (non-blocking) telemetry-improvement ideas that used to live in
the fwdllm-only doc, update the status table.
MIGRATION_TO_LAUNCHER_FWDLLM.md: no fwdllm-specific blockers remain, so
trim it to a short status note + redirect to MIGRATING_TO_LAUNCHER.md,
per the established fold-in-then-trim pattern (see d10bb53). Full
investigation history stays recoverable via git log on this path.
DELETION_CANDIDATES.md: fix a stale cross-reference to a "Merge readiness"
section that no longer exists in the fwdllm doc.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Dhruv Garg <dhruv.s.garg@hotmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu>
Co-authored-by: Gaurav Dadlaney <gaurav18n@gmail.com>
…ifar10) + fwdllm sim design (#69) * Sim unavailability: lock v1 scope (oracular-for-all) + file-level impl spec Rewrite UNAVAILABILITY_DESIGN.md to fold in the Jun25 design resolutions: - v1 scope locked: aggregator reads the shared trace (oracular) for ALL baselines, client_notify OFF. The old per-baseline oracle-vs-notify asymmetry collapses; aware vs unaware now differ only in WHEN a stalled slot is freed (aware: proactively at next selection boundary; unaware: reactively at the 90s abandon deadline). True avl_* transport + continuous mid-round scheduling deferred to Stage H. - Withhold-then-deliver corrected to a send-time gate, not a mid-compute interrupt: compute always completes, only the upload is gated, delivered stale at delivery_ts=max(sct,next_avail_ts). Real-side send-gate change documented; sim commits via the agg-side buffer. - 90s abandon + withhold are both true = two ledgers (slot vs delivery); the existing SEND/RECV_TIMEOUT_WAIT_S must re-clock to _vclock.now. - Three correctness invariants (no double-count; never re-select a still-down trainer; aware/unaware = trigger only). - Library mixin seam: flame/availability/{trace.py,availability_mixin.py} mixed into all four TopAggregators, consolidating the 3 duplicate read_trainer_unavailability; spans async_cifar10 + fwdllm. - New section 8: file-level v1 implementation spec (Stage A + C). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage A: trace resolver + AvailabilityMixin substrate New files: - flame/availability/trace.py: load_trace / state_at / next_avail_after (single bisect_right resolver; lru_cache for YAML files; replaces three inlined bisect_right copies) - flame/availability/availability_mixin.py: AvailabilityMixin with _init_availability, _avail_now (vclock in sim), read_trainer_unavailability, get_curr_unavail_trainers, free_stalled_slot (dormant hook for Stage C/D/H) Deletions (consolidation): - read_trainer_unavailability: removed from main_oort_sync_agg.py, main_asyncfl_agg.py, fwdllm_aggregator.py (all three dup copies) - get_curr_unavail_trainers: removed from syncfl/top_aggregator.py body (wall-clock impl) and main_oort_sync_agg.py (wall-time-in-sim bug); mixin provides the single vclock-correct version Wiring: - AvailabilityMixin mixed into syncfl TopAggregator → all four stacks inherit - _init_availability called from internal_init; sim_unavailability=False (default) ⇒ trainer_event_dict=None ⇒ byte-identical to all existing runs - flame/config.py: sim_unavailability, availability_aware, availability_trace_dir fields - Supports both new sim_unavailability gate and legacy track_trainer_avail path 389/389 unit tests pass. Syn_0 regression smoke pending on training node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * stage A tests and wip stage B * stage B complete * Sim unavailability Stage C: delivery-ledger substrate + resume plan Land the reusable C.2/C.4/C.5 core (gated/dormant until the live commit loop calls it; default OFF => byte-identical): - AvailabilityMixin.compute_delivery_ts: earliest t>=sct with state AVL_* (immediate if already up at sct; inf if the trace never recovers). - free_stalled_slot activated: frees the selector slot ledger AND registers pending_withheld[end]=delivery_ts (two ledgers, Challenge 4 / invariant 1). - withheld_held_ends / ready_withheld / commit_withheld delivery-ledger helpers (ready_withheld orders by (delivery_ts, end_id) -> no past-dating). - invariant 2 wired: withheld_held_ends() unioned into the unavailable list in oort + asyncfl _distribute_weights so a held trainer stays out of the eligible pool until its delivery_ts. - asyncfl __init__: _sim_withheld_payload store added (consumed by C.2 next). 10 new unit tests (tests/availability/test_delivery_ledger.py); full suite 420/420. Detailed C.2/C.3 live-wiring resume plan added to UNAVAILABILITY_DESIGN.md (RESUME HERE subsection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage C: live commit-loop wiring (C.2/C.3) + parity rungs Wire the send-gate withhold / late re-commit (C.2) and the 90s vclock abandon (C.3) into both sim commit loops, as ONE shared library-level core in AvailabilityMixin so the asyncfl (felix/fedbuff) and oort (oort/refl) stacks ride identical logic instead of forking it: * _sim_withhold_if_unavail — per-update send-gate primitive (invariant-1 stash for an already-abandoned end; else hold if state_at(sct)=UN_AVL: stash payload, free_stalled_slot, register the delivery ledger). Pure per-update decision — does not consult round_start, which is why oort applies its carry-over gate first (Challenge 7). * _sim_pop_committable / _sim_reinject_ready_withheld — asyncfl pop loop + due re-injection at delivery_ts (ordered), slot-only ledger drop. * _sim_abandon_stalled — vclock 90s abandon over the selector slot ledger; _avail_free_slot_ledger / _avail_inflight_ends are robust to both selector shapes (fedbuff dict-by-requester + all_selected; oort/refl/feddance flat set). * _sim_take_withheld_delivering / _emit_withheld_delivery — late-stale-commit telemetry; asyncfl tags the "withheld" past-dating bucket. Call sites: asyncfl/_sim_recv_min pop site; oort/_sim_drain_buffer pop loop (composed after the §4.9 carry-over gate); both _distribute_weights run the abandon scan. All helpers are getattr-guarded ⇒ gate-off byte-identity holds for bare aggregators that never ran _init_availability. Run activation: trace-name normalization (avl_events_syn_20 → syn_20) in flame/availability/trace.py — the legacy oort/refl JSON configs resolved to 0 traces / silent-OFF before. Confirmed end-to-end: 300 traces load, 34/300 UN_AVL mid-window, compiled config carries enabled/type/trace. Telemetry: withheld_delivery + abandon_timeout events/builders (events.py). Parity rungs (checks.py, wired into run_all_parity + CHECK_META): * withheld_delivery (DIAG) — structural: delivery_ts≥sct, delay_s≥0, staleness≥0; reports delay/staleness dist + accept_frac. * abandon_timeout (CONTROL) — fails loudly on a wall-clock leak. * eligible_pool_reduction (DIAG) — candidates−eligible across modes. Loaders surface the new agg events + trainer avail_change; duty_cycle reads the real {old_state,new_state} format (A4 bug). observation_lag + A4b held (need run data — see §7). Tests: 436/436 lib (16 new in tests/availability/test_live_wiring.py) + 38 parity (14 new in scripts/parity/test_availability_rungs.py). Next: oort syn_20 smoke (sim+real, --runtime-s 1800 to clear the 600s down window) before felix/fedbuff or Stage D — see UNAVAILABILITY_DESIGN.md NEXT ACTION. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * wip fixes, context saved in unavail_design.md * Sim unavailability Stage C.6 complete + Stage D.1 boundary eviction wired C.6.4: add trainer_state_fractions_sorted.pdf (per-trainer UN_AVL sorted bar, A4dur visual companion) to analyze_run.py availability_plots. Syntax clean, crash-safe. Visual correctness needs a fresh syn_20 run. D.0: felix gate plumbing in parity yaml — sim_unavailability/availability_aware/ client_notify added to both felix entries so _init_availability activates for felix on syn_20 runs (client_notify.trace overridden by --trace flag). D.1: _sim_evict_unavail_inflight in AvailabilityMixin — proactive boundary eviction for availability_aware baselines: at each selection boundary, any in-flight trainer showing UN_AVL per the trace has its slot freed immediately (no 90s wait). Called from oort and asyncfl top_aggregators after _sim_abandon_stalled; guarded by _availability_aware so oort/refl (unaware) stay on the 90s-abandon path. build_abandon_timeout gains reason field to distinguish C.3 from D.1 events. 8 new unit tests; 444/444 total. Next: syn_0 byte-identity smoke (felix+oort) then felix syn_20 smoke to validate D.1 eviction fires and C.6.4 plots populate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Speed up smoke-test debug runs: parallel trainer shutdown + --num-trainers wait_all() waited up to 30s per straggler trainer sequentially, adding minutes of idle wait when several trainers got stuck post-aggregator-exit; now all trainers share one timeout window polled concurrently. Also add debug_run.sh --num-trainers to shrink the cohort below 300 for runs that need a real --runtime-s budget (e.g. a vclock floor for an availability trace) that the existing smoke mode's hardcoded rounds=4/runtime=240 would cut short. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sim unavailability: real send-gate, avl_state telemetry fix, D.2 task gating Lands the four implementation items from UNAVAILABILITY_DESIGN.md's Next actions: (1) real-side send-time gate (§8.3) — trainer task-start no longer skips on avl_state (compute always completes), and syncfl/trainer.py's existing send-time UN_AVL check is decoupled from client_notify["enabled"] (always False in v1) so it actually fires for real felix/oort; (2) fixes avail_composition/avl_state staying all-UNKNOWN by stamping PROP_AVL_STATE from the oracular trace onto every known end before each selection; (3) fixes withheld_delivery under-emission caused by the delivery ledger dropping a slot-only entry before its payload physically arrived; (4) D.2 AVL_TRAIN/ AVL_EVAL task-type eligibility gating in selection, wired into oort + asyncfl. 452/452 lib + 63/63 example-parity tests pass (was 444/444). Real/sim parity run validation still pending (see Next actions). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix D.2 hang: guard eval task-type exclusion against 2-state traces D.2's AVL_TRAIN-exclusion-from-eval rule made eval's eligible pool permanently empty on a 2-state trace (syn_0/syn_20/syn_50 never produce AVL_EVAL), which corrupted the selector's shared in-flight tracking for the train task too (async_oort.py/fedbuff.py _handle_send_state's disconnect-cleanup loop wipes selected_ends when handed an empty pool). Found via a real felix syn_0 smoke that hung with 0 AGG_RECV_WEIGHTS despite all 48 trainers training and sending. _init_availability now computes _trace_has_avl_eval once (does this trace ever produce AVL_EVAL for any trainer); the eval-side exclusion in get_curr_task_ineligible_trainers only applies when true, matching the design doc's own "2-state collapses the split" statement. Train-side exclusion is left unguarded (no 3-state trace exists yet to exercise the symmetric risk) and documented as a residual risk in UNAVAILABILITY_DESIGN.md (Challenge 13). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: crisp down UNAVAILABILITY_DESIGN.md + add syn_0 confirmation 838 → 452 lines. Completed stages compressed (mechanism + where it lives + exit met). Updated status: D.2 hang fix CONFIRMED via Jun 28 syn_0 runs (felix+oort, both complete 4 FL rounds, all avail rungs PASS, abandon/withheld correctly SKIP). Removed run-unconfirmed tags now folded into "pending syn_20" which is in progress. Added D.2 empty-pool hang to §9 dead-ends. Dropped verbose per-item prose for landed stages; kept the invariants and land-mines that bite if forgotten. parity_felix.json/parity_oort.json from Jun 28 syn_0 regression runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Parity fixes + oort syn_20 analysis: NEAR_ZERO_LAG, phase guard, new rungs Three parity check issues found and fixed during syn_20 validation: 1. Raise NEAR_ZERO_LAG_S 0.05→0.10s (U6): carry-over burst after avail windows inflates sim mean to ~70ms — still immediate-commit semantically. 2. Add point-mass guard in trainer_phase_split: both means ≤5ms → skip KS, pass on mean (pre_train_s, weights_to_gpu_s, weights_to_ram_s near-zero). 3. Wire four new avail rungs into report.py _SECTIONS: A4dur, Aa (eligible_pool_reduction), C.3 (abandon_timeout), C.2 (withheld_delivery). Fix abandon_timeout tier CONTROL→INV so _TIER_TAG renders correctly. UNAVAILABILITY_DESIGN.md: add §9.1 (oort parity settled diagnosis — K3b run-length, T2 pre-existing, A2 KS shape artifact from avail-window bimodal distribution; none block Stage E). Reorder Next Actions: Stage E first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: optimized batched plan for E/F/G Collapse remaining v1 work (E feddance, F starvation, G integration) into one code batch + one long-run batch. Non-conflicting code surfaces land together, gated on cheap signals (unit tests + syn_0 byte-identity + syn_20 feddance / syn_50 smokes). Share the F-validation syn_50 run with G.2's ramp rung; overlap the oort 3600s confirmation for free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Batch 1: syncfl E.1/E.2/E.3, Stage F vclock-advance, G.1 starvation rung Stage E (syncfl path — feddance + refl): - syncfl _distribute_weights: abandon/evict/stamp + scarcity-advance (calls _next_avail_vclock() and advances vclock instead of sleeping when no trainers selectable at round start) - syncfl _sync_sim_recv_first_k: withhold send-gate (_sim_withhold_if_unavail) + withheld bonus drain (_sim_reinject_ready_withheld + _emit_withheld_delivery) - syncfl internal_init: _sim_buffer (SimReorderBuffer) + _sim_committed init - E.2 = accept-stale (FedAvg has no staleness gate); E.3 = naturally handled (withheld updates excluded from committed, so U6 is over actual cohort) Stage F (starvation vclock-advance): - New _next_avail_vclock() mixin helper: min across all trace transitions + pending_withheld delivery timestamps; returns None when gate off - Wired into: (a) syncfl if-not-selected_ends path, (b) oort max_retries loop - Felix asyncfl (top_aggregator.py:639) gap noted; deferred to syn_50 observation G.1 (parity rungs): - starvation_advance_parity() in checks.py: DIAG rung, jump_factor=5x mean gap, gate = withheld_deliveries or abandon_timeouts or avail_composition subfield - Dispatched in Stage 2 Availability results dict - Added to report.py _SECTIONS section "2" as "Fst starvation_advance (diag)" - withheld_delivery deps chain updated to include abandon_timeout - 4 new starvation_advance unit tests; 67/67 parity tests pass (was 63) Felix master-gate (prereq): - debug_run.sh trace-override block now injects simUnavailability: True (+ availabilityAware: True if client_notify.enabled) for non-ORACULAR baselines when --trace syn_X is passed; oort/refl still use legacy path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: clarify ORACULAR naming, add baseline smoke rationale Terminology overhaul to eliminate the dual meaning of "ORACULAR": - "ORACULAR" in YAML/code now explicitly labelled as the legacy-gate config mechanism (field name in oort/refl YAML), not a knowledge model descriptor - v1 knowledge model renamed "trace-read" (agg reads trainer_event_dict directly) — applies to ALL baselines, including feddance - "aware/unaware" split renamed "proactive/reactive-90s" for slot-free timing (proactive = felix at next boundary; reactive-90s = oort/refl/feddance at 90s vclock deadline) - Stage H transport renamed "message-transport" (avl_* msgs) vs "trace-read" - Config-gate section now defines legacy-gate (oort/refl) vs simUnavail-gate (felix/feddance) and explains they activate the same trace-read code path - §0 existing paths table updated: path A = "trace-read", path C = "message-transport" - §8.4 config surface updated with mapping table Stage E smoke rationale added: feddance-only is sufficient because refl shares the same syncfl _distribute_weights/_sync_sim_recv_first_k code and its legacy-gate was already confirmed in Stage C (oort syn_20 run). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: oort 3600s syn_20 analysis + G.4 terminology-in-code todo oort n=300 3600s syn_20 results (Jun 28 Run 2: 40/48 PASS): - T2 training_budget: CONFIRMED self-corrected (ratio=1.133 ≤ 1.15) ✅ - K3b overhead_residual: did NOT self-correct as predicted (still rel=0.116, now gated downstream by P3); prediction revised - A2 eligibility: improving (KS 0.437→0.338) but still shape artifact; need more rounds (Batch 2 long run) - P3 trainer_speed: NEW marginal root cause at n=300 (ratio=1.153 vs tol 1.15, 1.3% over); clean at n=48; likely speed-tail noise at full cohort - Fst starvation_advance: PASS "no starvation advances detected" — correct (n=300 + syn_20 always leaves ~240 trainers available, Stage F never fires) G.4 added to Stage G: consolidate config-gate paths + rename legacy function `oracular_trainer_avail_check` → `_trace_read_avail_check` + update comments/logs to use new terminology (trace-read, proactive, reactive-90s, legacy-gate, simUnavail-gate). Deferred to after Batch 2 long runs pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: Stage E confirmed (feddance syn_20 1800s 46/47), Stage F in progress feddance syn_20 1800s parity check: 46/47 PASS. All availability and mechanism rungs pass (A1–A4, U3/U6/K8/U2). Sole fail = C2 loss (0.1696 vs 0.15, 2 eval pts, early-train noise at alpha=0.1/syn_20 — not a mechanism gap). Challenge 9 confirmed: no K8/U2 movement from withheld stale commits. Stage E exit met. Stage F syn_50 runs launched (feddance + oort, 45 min each, real+sim). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Stage F.2: unified pre-selection starvation gate + doc overhaul F.2 replaces the broken per-aggregator retry mechanisms with a single pre-selection return-early pattern across all three aggregators: - oort: was max_retries=5 + min_required=5 (50% of agg_goal) + "proceed anyway" fallback + 2s blocking sleep. Now: threshold=desired_selection (=int(aggr_num×overcommitment)=13), unbounded, non-blocking (0.5s + return). - syncfl (feddance/refl): was post-selection `if not selected_ends:` which FedDanceSelector's partial returns (3/10 eligible) never triggered. Now: pre-selection threshold=agg_goal=10. - asyncfl (felix/fedbuff): was time.sleep(0.5) in both modes at no-recv-ends path. Now: sim advances _next_avail_vclock(); real keeps 0.5s sleep. All three use the same pattern: num_eligible < threshold → sim vclock-advance + re-stamp + return; real: 0.5s sleep + return. Outer run() loop retries non-blocking. No max_retries ceiling; no "proceed anyway" fallback. At n=10 syn_50: min_avail≈3 at t≈1200s → eligible=3 < desired_selection=13 (oort) and < agg_goal=10 (feddance) → both starvation paths now exercisable. UNAVAILABILITY_DESIGN.md: 746 → 248 lines. Completed stages compressed to 2-3 lines each. Full detail kept only for active (Stage F exit) and next (Batch 2). Known parity failures moved to a single reference table (§7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: syn_0 regression passed; syn_50 n=10 starvation smoke in progress syn_0 results (n=48): Fst PASS (no starvation fires), C1/C2 diff=0.0, K1/K5 PASS on both baselines. Oort 43/50 (K3b pre-existing gates K2/K3); feddance 46/48 (A2 shape artifact + gpu_compute short-run noise, K2/K3/P3 PASS). F.2 threshold check confirmed not firing spuriously at n=48 with syn_0. n=10 syn_50 starvation smoke launched; results to check in morning. Next actions updated to morning check commands. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix syn_50 feddance sim deadlock + improve crash diagnostics; update doc debug_run.sh: simUnavailability was never injected for feddance (parity config has tracking_mode:client_notify at aggregator level but no HP-level tracking block, so neither the trackTrainerAvail nor client_notify branch fired). Result: trainer_event_dict=None → vclock deadlock at 0.0 for full 5400s. Fix: add third elif branch for non-oracular baselines with no HP tracking block, injecting availability_trace + simUnavailability=True. Also add simUnavailability to the client_notify-in-HP branch (felix path). runner.py: aggregator exit code was not logged, making silent crashes (e.g. feddance real crash at 12s) undiagnosable. Log exit code after wait(); also set PYTHONFAULTHANDLER=1 in child env so C-level crashes (segfault in paho/torch) write a traceback to _aggregator.log. UNAVAILABILITY_DESIGN.md: update status (n=10 bugs root-caused), replace next-actions with n=25 1800s run command + corrected rationale. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix real-mode recv deadlock + rename max_runtime_s → max_experiment_runtime_s Real-mode oort aggregator could block indefinitely in recv_fifo when syn_50 trainers went unavailable mid-round: no timeout meant max_runtime_s check in inc_round() was never reached, causing the process to run until manually killed. Fix: add trainer_recv_wall_timeout_s HP (default 90s, YAML-configurable) as a per-message wall-clock timeout on recv_fifo in real mode. On timeout recv_fifo yields None → progressed=False → while loop breaks → aggregator commits what it has and proceeds to inc_round() where the budget check fires. Also capped at remaining wall budget (min with max_experiment_runtime_s) so the process never overshoots. Sim mode unaffected (_recv_timeout=None; _oort_sim_recv/drain_ready is non-blocking). Rename max_runtime_s → max_experiment_runtime_s across all 41 source files (YAMLs, scripts, lib code, docs). The new name makes the experiment-level scope clear; comments document the dual-clock behavior (wall in real mode, vclock in sim mode). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * debug_run.sh: add 30s progress ticker + experiment count display Background loop prints elapsed/remaining/percent every 30s while run_node is running; tracks experiments-started via new run_* dir count. Budget passed as n_exps × runtime_s (upper bound; sim faster). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(availability): real-mode recv-barrier timeout + Challenge 13 cleanup + oort cohort floor Three correctness fixes for sim-unavailability under scarcity, plus design doc update. 1. syncfl real-mode aggregate recv had NO timeout: recv_fifo(first_k=agg_goal) blocked forever when unavailable trainers withhold their uploads (feddance n=12 real hung ~46 min on a 30 min budget, 0 rounds, killed only by the runner watchdog). Bound it with min(trainer_recv_wall_timeout_s=90s, remaining budget), mirroring oort/asyncfl which already had this. Only the syncfl base lacked it, which is why oort completed and feddance hung. (Challenge 16 / B2.0.1) 2. Challenge 13: _handle_send_state "invalid prior selection" cleanup keyed off the availability-filtered eligible pool, so an in-flight trainer that merely went UN_AVL (or wrong task-type) was dropped from selected_ends; an empty per-task pool wiped ALL shared in-flight tracking -> hang. Added connected_ends param (full connected pool) for the cleanup membership check in async_oort/async_random/fedbuff; new-candidate selection still uses the eligible pool. +3 regression tests (TestChallenge13SendStateCleanup). 3. oort cohort-floor guardrail: clamp the starvation threshold to min(desired_selection, connected) + one-time [COHORT_FLOOR] warning, so an undersized cohort (n < desired_selection) can't silently degenerate into an all-starvation, budget-exhausting run. Tests: 449 passed; 7 pre-existing failures (test_sync_sim_ordering / test_sim_barrier fixtures missing _sim_buffer, fail identically on clean HEAD); parity ladder 24/24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(availability): fix 7 stale sync-aggregator fixtures; suite now fully green The 7 long-standing failures in test_sync_sim_ordering (x6) and test_sim_barrier (x1) were stale fixtures, not product bugs. They __new__ an aggregator (bypassing __init__ + _init_availability), so the availability state the E/F stages added was never set: - _sync_sim_recv_first_k references self._sim_buffer directly (set in __init__) - the AvailabilityMixin helpers expect trainer_event_dict / pending_withheld - the oort real-recv timeout reads self.config.hyperparameters Fix: give the fixtures the gate-OFF availability state (_sim_buffer=SimReorderBuffer(), trainer_event_dict=None, pending_withheld={}, a minimal _HP with trainer_recv_wall_timeout_s / max_experiment_runtime_s). Byte-identical to a real gate-off run, so the sim-ordering / stale-reject logic is still exercised in isolation. Result: 456 passed / 0 failed / 7 skipped + parity ladder 24/24. Doc updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * latest unavail markdown file * feat(availability): T0-T5 pre-flight — starvation fix, two-axis flags, 6-baseline scaffold, 79 tests T0 (blocking): B2.0.2 starvation self-termination fixed in syncfl/oort/asyncfl. - Starvation else-branch sets _work_done=True at trace horizon or budget. - Budget check changed from > to >= so vclock==budget stops the run. - Real-mode wall-budget guard added to the scarcity sleep path. - 10 regression tests in test_starvation_termination.py. T1: Two-axis flag split (_availability_aware → avail_select_filter + proactive_inflight_evict). - avail_select_filter gates get_curr_task_ineligible_trainers (selection pool filter). - proactive_inflight_evict gates _sim_evict_unavail_inflight (felix only). - [ORACULAR] → [TRACE_READ]; oracular_trainer_avail_check → _trace_read_avail_check. - Legacy fallback comment clarified: getattr fallback is dead (pydantic default=None means attribute always exists; bool(None)=False is the correct safe default). T2: Per-baseline flags set in parity YAML to match canonical baseline matrix. felix: both ON. oort/fedbuff: both OFF. refl/feddance/oort_star: filter ON, evict OFF. T3: oort_star + fedbuff scaffolded (baselines.yaml + parity YAML sim+real entries, 12 total). - fedbuff lrDecay HPs added to match felix and all other async cifar10 runs. - debug_run.sh smoke default updated to all 6 baselines. - debug_run.sh proactiveInflightEvict auto-detect comment clarified (always dead; Stage H future; explicit YAML values are authoritative). T4: 41-test state-fidelity suite (test_state_fidelity.py): T-state-exact, T-eval-pool, T-withhold-deliver, T-aware-vs-reactive, T-starvation-sync. T5 pre-work: A5 state_timeline_agreement wired into checks.py + report.py (DIST tier, 0.95 tol). 28-test config-wiring suite (test_baseline_wiring.py): all 6 baselines, catches oort_star-missing class of bug. oort_star added to baselines.yaml. Tests: 536 pass / 7 skip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(smoke): add smoke_suite.sh — timeout-guarded overnight smoke campaign debug_run.sh: LOGDIR is now env-overridable via FLAME_LOGDIR so each per-run invocation from the suite writes its Python agg output to an isolated directory instead of the shared /tmp/debug_run_logs. smoke_suite.sh: new parent orchestrator that runs 5 ordered steps (pytest → syn_0 sim → syn_20 sim → syn_20 both → syn_50 starvation) with individual per-(baseline, mode) wall-clock timeouts: - Each run launched via setsid so the whole process tree (launcher + 300 trainers) shares a session group; timeout kills via SIGTERM → 20s grace → SIGKILL on the group PGID. - Per-run grep checks on debug_run.out: stopping_run count (must be >0), SIM_WALL_CEILING count (must be 0), SIM_STARVATION count (informational). - Status: PASS / FAIL(no_stop) / FAIL(wall_ceil) / ERROR / TIMEOUT / SKIP. - Final report printed to stdout and written to <output-dir>/report.txt. - --dry-run shows every command without launching anything. - Configurable per-step runtimes (--runtime-syn{0,20,50}-s) and timeout buffer (--timeout-buffer-s); --steps to run a subset. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): replace setsid with set -m for reliable process-tree kill setsid forks when the calling process is already a process group leader (which interactive terminals guarantee for every background job). This makes runner_pid point to the dead parent that setsid discards, so kill -TERM -$runner_pid was targeting a defunct PGID — the entire stuck process tree survived as orphans. Verified with a 3-level (bash → launcher → trainers) simulated stuck runner: 4 orphan sleep 9999 processes confirmed after the setsid kill. Fix: use `set -m` (job control) immediately before the background launch and `set +m` immediately after. With job control active, bash assigns PGID = runner_pid to the job unconditionally, in both interactive and non-interactive contexts. Since flame/launch/spawner.py uses plain subprocess.Popen with no start_new_session or preexec_fn=os.setsid, all 300 trainer processes inherit the same PGID and are killed by kill -TERM/-KILL -$runner_pid. Verified: 6-process tree (runner, ticker, launcher, 3 trainers) all dead after kill, PGID=runner_pid in both interactive (bash -c) and non-interactive contexts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): clean termination + correct aggregator log grep Three fixes to _run_baseline: 1. Post-kill cleanup (timeout path) When SIGKILL fires, run_experiment_batch's finally block (_sweep_stragglers: pkill trainers/agg + 45s GPU wait) is in the killed process group and may not complete. Replicate it in bash: pkill -9 -f "trainer/pytorch/main.py" pkill -9 -f "aggregator/pytorch/main_" sleep $KILL_SETTLE_S # default 20s GPU memory drain so no stale trainer/aggregator processes or held GPU memory bleeds into the next run's allocation. 2. wait() order fixed Added wait after SIGKILL on timeout path; non-timeout path uses conditional wait so we never double-wait the same PID. 3. Grep checks scan the correct file AggregatorSpawner redirects the aggregator subprocess's stdout/stderr to experiments/run_*/..._aggregator.log (NOT $FLAME_LOGDIR/debug_run.out, which is only run_experiment.py's own print()s). Previous grep always returned 0 for "stopping run" / SIM_WALL_CEILING / SIM_STARVATION, causing every successful run to be classified as FAIL(no_stop). Fix: use -newer $ts_marker to find aggregator logs created during this run's lifetime; accumulate counts across them (handles multi-exp batches). Paths saved to $run_dir/agg_logs.txt for easy post-mortem access. Also adds --kill-settle-s CLI option and KILL_SETTLE_S=20 default. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(unavail): add T5-smoke section with 6h command + run count + hypotheses 23 jobs / 22 FL experiments (steps 1,2,4,5; step 3 skipped as redundant with step 4 sim half). Expected ~4h 38min wall, fits 6h with ~1h margin. Documents per-baseline predictions, starvation-at-n300 note (SIM_STARVATION absent is correct, not a failure), and failure watch-list for oort_star/fedbuff first live runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): grep-c double-output crash + in-place progress ticker - Fix arithmetic crash: grep -c always prints a count even for 0 matches and exits 1, causing || echo 0 to fire and produce "0\n0"; $(( N + 0\n0 )) is a bash syntax error that aborted _run_baseline before _record, silently dropping all runs after the first baseline. Fix: || echo 0 → || true. - Add per-run in-place ticker (stderr, \r): shows elapsed/budget/pct within the current run plus kill-in countdown and overall N/total done count. Overwrites in place each 5s tick — lets you spot stuck runs without log flood. - Add _compute_total_runs + TOTAL_RUNS/COMPLETED_RUNS globals; _record now increments COMPLETED_RUNS so the ticker stays accurate across all steps. - Move COMPLETED_RUNS increment into _record so pytest (step 1) also counts. - Drop nohup from the design-doc command (tmux keeps the session alive). - Expand T5-smoke hypotheses into a 23-row per-run table with Actual column. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): step4 ETA log used invalid ${#var//pat/} substitution Bash treats ${#var//x/y} as a parse error ("bad substitution"), which is fatal regardless of set -e/-u — it killed the whole suite right at the step 4 header, before any step 4/5 runs executed. Replace with wc -w to count baselines. * feat(availability): stamp real-mode vclock_now (A4dur fix) + rename ClientAvailability + B2.0.3 join-barrier re-anchor vclock_now was hardcoded None for real-mode selection telemetry, so A4dur's duration-weighted real/sim comparison fell back to a join-ramp-skewed origin; now stamped via ClientAvailability._avail_now() for both modes. Renamed AvailabilityMixin -> ClientAvailability (availability_mixin.py -> client_availability.py). Also fixes B2.0.3: agg_start_time_ts was stamped before the trainer join barrier, so real's trace-read clock baked in the join wait (~300s at n=300), reading the availability trace ~300s ahead of sim/ground-truth. Root-caused via feddance syn_50's A3 (avail_timebase) failure against the trace's own ground truth. Fixed via _mark_join_barrier_done(), which re-anchors agg_start_time_ts to real time once the join barrier resolves (real mode only, self-correcting to actual join duration) -- one fix point in syncfl/top_aggregator.py covers all six baselines via inheritance. Confirmed A4dur fix live on a fresh felix syn_20 real+sim pair (mean_err=0.0). B2.0.3 fix confirmation (fresh feddance syn_50 pair) still pending. 6 new regression tests in test_join_barrier_reanchor.py. Full suite green: 542 pass / 7 skip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --background flag for unattended overnight runs Re-execs the suite via nohup+disown and returns immediately, printing the PID, nohup log, and eventual report.txt path. Lets a smoke campaign survive the launching shell/SSH session closing, so it can be fired off and checked on later rather than needing a session held open for hours. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass -- A4dur confirmed, B2.0.3 root-caused+fixed, next-step command Whole-doc crisp pass per the Working Agreement: Status, T5-smoke, Challenges (#17 B2.0.2 was still marked open, now closed; new #18 for B2.0.3), and the known-parity-failures table all updated to reflect the A4dur confirmation and the new B2.0.3 root-cause + fix. Collapsed the old SCRATCH handoff section down to the still-open next steps (trackTrainerAvail cleanup, mobiperf live exercise, PR workflow) and dropped Q&A material now folded into permanent sections. Added a prominent NEXT STEP callout at the top of the doc with the exact unattended smoke_suite.sh command to confirm both fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --num-trainers to shrink the cohort below 300 Forwards --num-trainers through to every debug_run.sh invocation (debug_run.sh already auto-scales min_trainers_to_start accordingly). Lets a confirmation run use a smaller cohort under resource constraints without editing the parity config -- useful for mechanism-level fixes (e.g. B2.0.3's join-barrier re-anchor) that are n-independent by construction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): scope next-step confirmation run to n=100 Resource-constrained follow-up: A4dur and B2.0.3 are both mechanism-level fixes (already confirmed n-independent / self-correcting to actual join duration), so n=100 is sufficient to confirm them without reproducing the original n=300 failure magnitude. Notes the tradeoff -- not directly comparable to the existing n=300 parity_*.json snapshots for the still-open §7 checks, which stay gated on a real n=300 T5 run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(config): explicit encoding="utf-8" on YAML config reads (node-portability) open() without an explicit encoding falls back to the node's locale-preferred encoding. On a non-UTF-8 locale (e.g. C/POSIX without Python's UTF-8 mode coercion) this mis-decodes non-ASCII bytes in the parity YAML (an arrow character in a comment) byte-by-byte, and yaml.safe_load then rejects the resulting control characters with a ReaderError -- reproduced exactly with PYTHONUTF8=0 LC_ALL=C + encoding="latin-1". Fixed the crashing read (debug_run.sh's parity-config generator) plus the write/re-read round trip, and hardened the other YAML config reads in the availability substrate (trace.py's synthetic/mobiperf trace files, client_availability.py's trainer registry) against the same failure mode before it bites on a node with non-UTF-8 content in those files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(unavail): Batch 3 trace-fidelity & delay-enforcement overhaul (T3.0-T3.5) Adds absolute (vs. raw ground-truth trace) availability fidelity checks to complement the existing real-vs-sim relative checks, which is how the real-mode send-gate went dead code for the whole project without any check catching it (root-caused as T3.1a). - T3.0/T3.1a/T3.1b: shared trainer<->aggregator time origin (real mode), the debug_run.sh trace-wiring fix (trainer's own trace was never substituted), _refresh_avl_state() mode-dispatch cleanup. - T3.2: A6 trainer_trace_fidelity -- trainer's own avail_change telemetry vs. raw ground-truth trace. New scripts/parity/ground_truth.py; sim_now field added to avail_change. - T3.3: A7 agg_belief_fidelity -- aggregator's belief vs. ground truth, both selection and commit checkpoints. New agg_belief_change telemetry + ClientAvailability hooks. - T3.4: A8 send_gate_wait_fidelity -- observed vs. ground-truth-expected [SEND_GATE] wait duration. New send_gate_wait_s/send_gate_sct fields on task_send telemetry. - T3.5: K11 commit_promptness -- actual commit time vs. delivery_ts for withheld-then-delivered updates. New actual_commit_ts field on withheld_delivery telemetry. Each check ships with new telemetry, a parity rung in checks.py, a plot in analyze_run.py, and synthetic-data unit tests. Batch 3 is now code+test complete; UNAVAILABILITY_DESIGN.md's own plan calls for exactly one real run next (Phase 5) to generate live telemetry for all four new checks at once. Tests: lib/python/tests/ 574 passed / 7 skipped; examples/async_cifar10 scripts/parity/ + trainer/pytorch/ 121 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Batch 4 — felix real hang, sim-mode trainer clock freeze, A7 checker fix Phase 5/6's first real run against the new absolute fidelity checks (A6/A7/K11) surfaced three gaps; all three are root-caused, fixed, and unit-tested here: 1. felix real-mode TIMEOUT: D.1 proactive in-flight eviction (_sim_evict_unavail_inflight) was called only inside `if self.simulated:`, alongside the genuinely sim-only 90s vclock re-clock. But async_oort's selector derives `recv_ends` directly from `selected_ends`, so a stalled UN_AVL trainer never evicted in real mode never left recv_ends either, hanging the aggregator's 30s recv_fifo loop past its own budget. Un-nested D.1 from the sim-only gate in all three top_aggregator.py stacks (a no-op everywhere except felix, the only baseline with proactive_inflight_evict). 2. A6/A4dur/K6: sim-mode Trainer._sim_now() returns the frozen _sim_send_ts from its last dispatch, so a trainer withheld as UN_AVL can never observe later trace transitions while idle. Fixed in two zero-new-comms parts: (a) avail_change.sim_now now stamps the transition's own scheduled trace-time instead of _sim_now() at processing time; (b) inform_end_of_training's existing broadcast now piggybacks the aggregator's final vclock (sim mode only), and the trainer's shared _fetch_weights flushes one last _refresh_avl_state() catch-up on EOT. This also resolves the long-open K6 (`sim_send_ts==0`) mystery — same frozen-clock mechanism, not selector-starvation or a checker false positive. 3. A7 commit-checkpoint: not a system bug — the checker (_fidelity_score) extrapolated a trainer's last recorded commit-belief across the run's full span, wrongly scoring the silence after a trainer stops committing (typically because it went UN_AVL) as stale drift. Added extrapolate_tail=False for the commit-checkpoint call site, truncating the scoring window to [t_start, last observed t], symmetric with the existing start-side truncation. 12 new/updated tests (tests/ 586 pass/7 skip, scripts/parity/+trainer/pytorch/ 125 pass). Not yet confirmed on a live run — see UNAVAILABILITY_DESIGN.md's Batch 4 section for the re-confirmation command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): ASCII-only comments in Batch 4 source files (node portability) * debug(unavail): temp diagnostic logging for D.1 eviction-miss mystery Live re-confirmation of the Batch 4 fix 1 (D.1 proactive eviction in real mode) found the fix is necessary but not sufficient: felix real still hangs at n=100, stuck at one round for 10+ minutes with only a handful of the ~20 stalled trainers ever getting evicted. Two isolated reproductions against the real AsyncOortSelector/ClientAvailability code (single stalled trainer, and 40/100 simultaneously UN_AVL) both show the eviction mechanism working correctly every cycle, so whatever is different about the live run isn't reproduced by either repro yet. Adds an [EVICT_DEBUG] summary log line to _sim_evict_unavail_inflight (inflight count, evicted count, and per-skip-reason counts: still available / buffered / already committed-or-withheld / no trace) so the next live run shows which guard is actually firing instead of inferring it from AWARE_EVICT's absence. Log-only change, no behavior change. Remove once root-caused (see UNAVAILABILITY_DESIGN.md's "Open A" for tracking). Also documents Open B: a separate, pre-existing bug found while investigating A -- trace loading falls back to the shared syn_20 "pattern" for most trainers instead of their individually-assigned per_trainer entry (hand-verified on two trainers whose own traces shouldn't transition until t=13800s/t=34200s, but which actually transitioned at t=600s/1200s matching the fallback exactly). Not yet root-caused or fixed; deliberately investigating Open A first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): add PR-readiness checklist for fresh-context handoff Explicit top-to-bottom checklist (Open A eviction-miss root cause, Open B trace-fallback root cause, cleanup, re-confirmation run, then the pre-existing Open items 1-2) so a new session can pick up exactly where this one left off without re-deriving status from chat history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Open A/B — selector abandon leak, trainer trace mismatch Open A (felix real never self-stops): the real-mode 90s SEND_TIMEOUT_WAIT_S abandon in async_oort.py/fedbuff.py only cleared all_selected, never selected_ends. recv_ends derives from selected_ends, not all_selected, so a stalled end never left the recv_fifo wait set and max_experiment_runtime_s's self-stop check (gated on recv_ends being empty) was unreachable. Both selectors now also discard the end from selected_ends. Open B (trainers ignore their per-trainer trace): spawner.py's get_synthetic_trace never took a trainer_id, so every trainer process baked in the shared cohort-wide `pattern` instead of its own registry-assigned per_trainer trace, even though the aggregator's own belief-tracking already resolved per-trainer correctly via flame.availability.trace.load_trace. get_synthetic_trace now delegates to that same resolver when given a trainer_id. Trainer startup also logs an [AVAIL_TRACE] signature (trace_hash) so a shared hash across trainer_ids is visible directly in the logs. debug_run.sh's --trace now accepts a space-separated list (one experiment set per trace) to make cross-trace validation runs a single invocation. Tests: tests/selector/test_send_timeout_frees_selected_ends.py, tests/launch/test_config_generator.py::TestSyntheticTracePerTrainer, tests/launch/test_debug_run_trace_substitution.py::TestMultiTraceSubstitution — all verified to fail without their respective fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): strip non-ASCII chars from this session's additions Extends 994f8187's node-portability cleanup to the lines added in the last two commits (0514d427, 8d145865) that it predates: em-dash/en-dash/multiply/ approx/bullet glyphs in debug_run.sh comments and UNAVAILABILITY_DESIGN.md's NEXT STEP section, replaced with ASCII equivalents (--, -, x, ~). Scoped to those additions only, not the doc's older pre-existing prose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): fwdllm agg telemetry test used stale property key _FakeChannel.get_end_property() matched on the literal string "round_duration", but _process_aggregation_goal_met() actually looks up PROP_CLIENT_TASK_TRAIN_DURATION ("client_task_train_duration_s"). The fake channel silently returned None instead of the seeded duration, so trainer_speed_s/agg_observed_s always came back empty -- a fixture bug, not a production one (traced back to a pre-existing mismatch already present on dg-fork-main before this branch's rebase). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass on UNAVAILABILITY_DESIGN.md per its own house rules Condense fully-resolved sections (Batch 3 T3.0-T3.5, Phase 5/6 results, T5-smoke/A4dur/B2.0.3) to the doc's own stated format for completed stages -- mechanism + files + tests + exit, 2-3 lines -- dropping narrative/discovery play-by-play that's already preserved in commit history. Also updates the NEXT STEP run command to the actual planned first-pass confirmation run (1800s, syn_50 only, felix+fedbuff) ahead of the full 7h campaign. 1375 -> 945 lines, no content loss (full detail remains in git log for every referenced fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): trim verbose comments in client_availability.py + syncfl top_aggregator.py Comment-only cleanup, no behavior change (verified via targeted availability/ sim-ordering/sim-barrier test runs, 141 passed): - client_availability.py: removed the TEMP DIAGNOSTIC eviction-miss counters and [EVICT_DEBUG] log line in _sim_evict_unavail_inflight -- explicitly marked for removal once Open A was root-caused, which UNAVAILABILITY_ DESIGN.md now confirms. Trimmed several multi-paragraph docstrings (get_curr_task_ineligible_trainers, _avail_stamp_end_states, _emit_withheld_delivery, _sim_reinject_ready_withheld) down to the causal fact + pointer, dropping session-narrative framing. - syncfl/top_aggregator.py: trimmed _mark_join_barrier_done's docstring and the EOT-broadcast piggyback comment the same way. First batch of a broader pass across the dg-fork-main diff; the other top_aggregator.py variants (oort/asyncfl), analyze_run.py, and the test files still have similar cleanup pending. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): finish comment-cleanup pass across the dg-fork-main diff Comment-only, no behavior change (full suite green: 728 passed / 7 skipped, same as before this batch): - oort/top_aggregator.py: trimmed the trainer_recv_wall_timeout_s explainer and the cohort-floor guardrail comment; dropped an internal "Next actions §2" task-tracking reference (also in asyncfl/top_aggregator.py). - syncfl/trainer.py: trimmed the AGG_START_TS caching comment, the EOT avl_state catch-up comment, and the [SEND_GATE] block's docstring-length comment down to the causal fact. - async_oort.py: reworded the Challenge 13 cleanup comment to match the shorter version already used in fedbuff.py/async_random.py (all three carry the identical fix; no reason for one copy to be 3x longer). - message.py, examples/async_cifar10/trainer/pytorch/main.py, analyze_run.py: trimmed remaining multi-paragraph docstrings/comments down to mechanism + one-line rationale, dropping session/phase-tracking references ("Batch 3 T3.0", "Batch 4 finding 2") in favor of plain causal language -- that history lives in UNAVAILABILITY_DESIGN.md and git log, not duplicated inline. Completes the comment-cleanup pass across the highest-density files in the diff (client_availability.py + the 3 top_aggregator.py variants + trainer.py + analyze_run.py). Remaining lower-density files and test-file docstrings were left as-is -- most already follow the repo's convention of a short per-test docstring naming the specific regression it guards against, which is exactly the kind of comment worth keeping. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(launch): decouple spawn cohort size from dataset-split selector The launcher tied "how many trainers to spawn" and "which <name>_alpha<a>_n<N>.yaml split to read" to a single num_trainers knob, so a 10-trainer smoke demanded a dedicated n10 split file and KeyError-ed when only n300/n50/n48 existed. - experiment_config: add trainer.split_num_trainers (defaults to num_trainers) to pick the split partition independently of cohort size. - runner: split lookup uses split_num_trainers or num_trainers; the spawn cohort (trainer_ids) still uses num_trainers. - debug_run.sh: --num-trainers now preserves the config's native split size so a shrunk cohort reads the existing n300 split; add --alpha flag to choose dirichlet_alpha (0.1/1.0/10.0/100.0) against that n300 split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): bound A7 commit-checkpoint scoring by max_gap_s _fidelity_score held each sparse commit-checkpoint observation constant across interior gaps between commits, penalizing the absence of a mid-gap commit rather than any wrong belief. Add max_gap_s (wired to lag_tol_s at the commit call site): each observation vouches for its own state only up to max_gap_s past itself, on both sides; a transition with no covering window is excluded from both the TVD score and the missed/spurious diagnostic. Generalizes the earlier extrapolate_tail tail-truncation to interior gaps. Took felix 59/61 -> 62/62 on the live n=300 run. Tests: test_agg_belief_fidelity.py (interior-gap + wrong-belief-at-own-instant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(sim-unavail): fwdllm sim design, extend PARITY.md, trim UNAVAILABILITY_DESIGN, drop generated artifacts - PARITY.md: generalize to a multi-example parity methodology; add §F fwdllm rung catalog (baseline matrix, modified rungs, variance-cadence / dynamic-K/C / forward-grad layers). - simulate_fwdllm.md: rewrite as the design-only build plan -- correct code anchors to flame-core fwdllm_aggregator.py, resolve the §C baseline matrix from the landed launcher configs, integrate unavailability as first-class, surface decisions D1-D4. - UNAVAILABILITY_DESIGN.md: trim 1027 -> 283 lines to open items + next-steps + durable reference (baseline matrix, flags, v1 decisions, land-mines, dead-ends); collapse landed history to pointers; add the parallel-branch validation note. - Remove generated, unreferenced parity_{felix,oort,feddance}.json (-2152 lines; already covered by .gitignore). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…s); deprecate training_delay_s
…e_ms, rtt_amplitude, rtt_period_s), superseded by satellite_latencies.npy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
This PR adds the satellite-latency-based RTT timing model (the core mechanism: satellite_latencies.npy + per-trainer satellite_index lookup).
It also includes computation_time_ms wiring, since the formula combines compute + RTT into one budget -- but the value currently in the registry is a placeholder, not real A40 data. Still resolving the field structure for that with Dhruv separately; will need a follow-up commit.
Jun 22 update: