Trainer delay sine - #670
Open
TummalaSeshu wants to merge 852 commits into
Open
Conversation
…g even in none state
* First attempt at creating script to stop the run after a certain acc has been reached N consecutive times + change config for Async+Fwdllm run + minor bug fix in avl config checker * Move some heavy hitter logs to DEBUG to try to reduce Agg log file size --------- Co-authored-by: Gaurav Dadlaney <gaurav18@gmail.com>
…nsecutive times (#49) * First attempt at creating script to stop the run after a certain acc has been reached N consecutive times + change config for Async+Fwdllm run + minor bug fix in avl config checker * Enhancement to add monitor logs in another file instead of polluting stdout * Add full log file name * Make sure the script run in /bin/bash & not /bin/sh to support += & echo -e syntax * Not waiting for an eval to create the monitor log file * Minor log tweaks --------- Co-authored-by: Gaurav Dadlaney <gaurav18@gmail.com>
* Fix the hardcode on file names used by Oracular knowledge * Move to relative paths --------- Co-authored-by: Gaurav Dadlaney <gaurav18@gmail.com>
* Attempt 1 to populate stat_utility & round_duration in the last_N windows used by async_oort.py * Remove unnecessary file
* Fix OORT selection logic & verified it by logging at the aactual value in the logs * Introducing a change to always avoid exploring in the common case of Async (We just need to select 1 end) * Add log to calculate stats across different selectors. Also, remove hack to random choose between exploration & exploitation in AsyncOORT when choosing just one end (It actually move completely to exploitation pretty quickly) * Add log to view trainer involvement & trainer distribution * Add 90% avail config * Add p1 & p5 to the logs * Add log parser config & throwaway script to plot the p1 to p99 values as a timeline shaded region plot * change plot presentation (#54) * change plot presentation * small fix * Add smoothening & other features required for NSDI poster submission * Fix async_oort, where we don't rely on feasible_extras anymore. This helps in the common case where feasible_extras = 1 & we reliably select the # of ends to explore & exploit based on a fixed percentage of the availabilty trainers (in sorted_utility_list) * Review comments resolved * Removing the unnecessary clipping code that had no effect as the variable was being overriden --------- Co-authored-by: Gaurav Dadlaney <gaurav18@gmail.com> Co-authored-by: AishwwaryaM <58990857+AishwwaryaM@users.noreply.github.com>
…ection (#55) * Fix major bug in _distribute_weights() to send weights to trainer that has not contributed to the current model version when var = bad. * Fix the problem of GRAD_POOL going as None for clients that had stale updates & needed weights & the updated GRAD_POOL to train * variable rename
…or distributing weights/var messages
[Fluxtune] Add dynamic KC policy impl, init experiments
* fix bug to know if weights are distributed to enough trainers for full aggregation * setup for jvp and dynamic k c expts * All changes for basic greedy jvp based perturbation selection * greediest jvp selection - disabled by if false * logs for snr and differnet stopping criteria * defualt cos sim selection * logging * tmp stash * resolve comments * rename jvp_perturbation_selection to select_perturbation_using_jvp --------- Co-authored-by: Dhruv Garg <dgarg39@jayne.cc.gatech.edu>
* Add config for mobiPerf traces * Add missing code under _force_commit --------- Co-authored-by: Gaurav Dadlaney <gaurav18@gmail.com>
* FedDance baseline + selector cleanup
Add FedDance participant selector (V_m · I_m · A_m with MAB exploration),
Poisson availability predictor, and CIFAR-10 example. Formalize
AbstractSelector lifecycle hooks (on_update_received, on_round_completed)
and migrate top_aggregator off the brittle private-attribute coupling
into selectors. Strip ~360 lines of debug-trainer-ID blocks and hasattr
guards from oort.py/refl_oort.py/top_aggregator.py. Fix latent IndexError
in OortSelector when utility_list is empty but in-flight set is not.
Start selector test suite: 63 new tests, all passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Selector cleanup: consolidate PROP_* + strip cruft + drop slow_main
Pull every PROP_* string-key duplicate into flame.selector.properties,
import from there in oort/refl_oort/feddance/fedbuff/random/async_oort/
async_random/top_aggregator. Strip stale TODOs and verbose docstrings
from oort.py without changing behavior. Remove unused slow_main.py
shadow files from examples/cifar10. All 63 selector tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Generic YAML launcher: flame.launch + shared examples/_metadata
Promote async_cifar10's launcher into flame.launch so any example
(cifar10, feddance_cifar10, future google_speech, ...) can drive
experiments from a YAML descriptor. Key changes:
- flame.launch package: ExperimentConfig, MetadataLoader, ConfigGenerator,
TrainerSpawner, AggregatorSpawner, ExperimentRunner, run_experiment CLI.
- ExampleConfig + MetadataPaths sections in the experiment YAML let each
example point at its own trainer/aggregator entry points and pick a
shared metadata bundle.
- ConfigGenerator.get_dataset_split now takes dataset_name + num_trainers
so split keys are no longer hardcoded to cifar10_alpha*_n300.
- MetadataLoader is lenient about missing trace files.
- flame.launch.cli.load_config_from_argv lets example main.py files
accept both --config <path> and --config-json <str>.
- examples/_metadata/ is the shared metadata directory (registry, traces,
dataset splits). async_cifar10/metadata becomes a symlink for backward
compat with existing experiment snapshots.
- async_cifar10/launch/ is now a thin compat shim importing from
flame.launch (preserves `from launch.X import Y` import surface).
- cifar10 and feddance_cifar10 get configs/trainer_base.yaml +
experiments/configs/smoke_10trainer.yaml + --config-json support.
13 new launcher tests, all passing. 63 selector tests still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Migrate async_cifar10 6d_3state_oort configs into shared _metadata
Add lib/python/scripts/migrate_async_cifar10.py: a one-shot migration tool
that folds the 1,200 trainer JSONs under config_dir{0.1,1,10,100}_num300_
traceFail_6d_3state_oort/ into the hierarchical shared metadata bundle.
Policy (per user decisions on 2026-05-20):
- alpha=0.1 is the ground truth for all availability state data.
- alpha=100's distinct 2_state realizations are dropped (legacy).
- syn_40 (only present in alpha=1/10/100) is dropped (legacy).
- Verification: strict byte-equivalence for alpha=0.1 (300 trainers);
indices-only verification for alpha=1/10/100 against the existing
cifar10_alpha{1,10,100}_n300.yaml splits (900 trainers).
- All 1,200 files verified before any deletion.
Metadata patches applied:
- trainer_registry.yaml: training_delay_s float -> str (matches JSONs).
- synthetic_traces.yaml: extend schema with per_trainer.n300.<tk> entries
for syn_20 and syn_50 (each per-trainer in the source data).
syn_0 stays as a uniform `pattern` field.
- migration_plan_async_cifar10.yaml: full provenance record.
Also:
- Delete the 1,200 source JSON files (and stale exec_*.sh shell scripts).
- Delete cifar10/trainer/config{1..15}/ (separate cifar10 warmup migration).
- lib/python/scripts/migrate_trainer_configs.py + verify_migration.py:
generic tool from the cifar10 warmup; kept for future use.
- examples/README.md: add a launcher + metadata-hierarchy section.
- tests/scripts/: 4 round-trip tests + 17 logic tests for migration tools.
Full test suite: 97/97 pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Clean up async_cifar10/trainer/: drop legacy config dirs + hardcoded path
Delete everything under async_cifar10/trainer/ except __init__.py and
pytorch/ (the runtime code):
- 22 config_dir* directories (~4,587 trainer JSONs + 85 exec_*.sh scripts):
* config_dir<a>_num{100,300}_traceFailure_{1.5h,12h} (sub-6d traces; user
directive to ignore < 6 days)
* config_dir<a>_num300_traceFail_48h{,_oort} (sub-6d traces)
* config_dir<a>_num100_noFail{,_fedavg,_oort} (n=100 population, not in
hierarchical metadata which targets n=300)
* config_dir<a>_num300_traceFail_6d_3state (non-oort 6d variants;
superseded by the _oort versions already migrated)
- Loose files: config.json, config2.json, config_2_test.json,
copy_trainer_indices_list_to_new_configs.py, trainer_logs.{csv,txt},
Nov26_fedbuff/ (historical run artifacts)
Patch main_oort_agg.py:
- read_trainer_unavailability() previously hardcoded the path to the now-
deleted 6d_3state_oort dir and read each trainer's JSON for avl_events_*.
- Replaced with a metadata-driven lookup: read trainer_registry.yaml for
task_ids, then mobiperf_traces.yaml or synthetic_traces.yaml for the
requested trace (matching the per_trainer.n300 fallback-to-pattern logic
used by the migration).
- Removed dead imports (ast, glob, json, os) and the legacy "find JSON files
by glob" code path entirely.
Full test suite: 97/97 still pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Baseline catalog + experiment imports + provenance logging
Add a master baselines.yaml catalog so experiment YAMLs can declare
`baseline: felix` (or refl / feddance / oort) and inherit the canonical
selector + optimizer + tracking config. Per-experiment values like c,
aggGoal, batchSize, rounds are layered on top via `config_overrides`
dicts (deep-merged).
Schema:
experiments[].baseline -> key into _metadata/baselines.yaml
experiments[].aggregator.config_overrides (dict, deep-merged last)
experiments[].trainer.config_overrides (dict, deep-merged last)
Layers:
aggregator: config_template + baseline.aggregator + experiment.config_overrides
trainer: trainer_base.yaml + baseline.trainer + experiment.config_overrides
+ per-trainer metadata + dotted-key kwargs
Runtime: the runner prints provenance per leaf path
("hp.client_notify.enabled <- baseline:felix"), so it's obvious what came
from each layer.
Felix-specific changes (per user-corrected canonical config 2026-05-20):
- selector: async_oort with evalGoalFactor=1.0 (NOT 0.0)
- optimizer: fedbuff with use_oort_lr=True, agg_rate_conf.type=new (NOT old)
- tracking: client_notify enabled on trainer side (NOT oracular)
Other changes:
- flame.launch.baselines: load_baselines, deep_merge, merge_with_provenance,
format_provenance.
- flame.launch.aggregator_spawner: spawn(config_json=...) alternative to
spawn(config_path=...). The runner now always emits the merged dict via
--config-json (and writes a snapshot to <exp_dir>/aggregator_config.json
for reproducibility).
- flame.launch.spawner.ConfigGenerator.set_baseline_overrides: applies the
merged baseline+experiment trainer dict before per-trainer metadata.
The legacy hardcoded client_notify=syn_0 default becomes a fallback only
when no upstream layer sets it.
- examples/_metadata/aggregator_base.json: generic boilerplate (channels,
brokers, registry, baseModel, role) for baseline-driven experiments.
- main_oort_agg.py: argparse now uses flame.launch.cli.load_config_from_argv
so --config-json works.
Two example YAMLs land in expt_scripts_2026/:
- felix_n300_alpha100_syn20.yaml: production target (n=300, homogeneous,
20% synthetic unavailability).
- felix_n10_alpha100_syn20_smoke.yaml: 10-trainer smoke test for the
launch pipeline.
16 new baseline tests + 113/113 full-suite passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Simplify dependency setup: Python 3.11, single install path
Move all dependency declarations into lib/python/setup.py:
- install_requires: core flame package only (aiostream, boto3, paho-mqtt,
pydantic, mlflow, grpcio, protobuf, ...). Drop the legacy hard pins on
mlflow==2.12.1, protobuf==3.20.3, grpcio==1.53.2, pydantic<2.0; let
pip resolve current-compatible versions.
- [examples]: ML-stack deps (torch, torchvision, sortedcontainers, wandb).
- [dev]: pytest, pre-commit, black, flake8, bandit, mypy, isort.
- python_requires=">=3.10".
Add scripts/setup_env.sh: one-shot conda env creator (Python 3.11) that
installs flame + [examples,dev] from setup.py. Replaces the async_cifar10-
specific setup_env.sh (deleted) and the legacy requirements_*_no_versions.txt
duplicates (deleted).
Update docs/prerequisites.md, docs/quickstart-{ubuntu,mac}.md, and
async_cifar10/README.md to reflect:
- python 3.10+ (3.11 recommended)
- single command: bash scripts/setup_env.sh <env_name>
Onboarding flow is now:
bash scripts/setup_env.sh flame
conda activate flame
python -m flame.launch.run_experiment <yaml>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Drop requirements.txt; fix quickstarts to use scripts/setup_env.sh + launcher
- Delete the legacy requirements.txt freeze; deps now live solely in
lib/python/setup.py (install_requires + [examples] + [dev] extras).
- quickstart-{ubuntu,mac}.md: rewrite Environment Setup to one command
(bash scripts/setup_env.sh flame). Rewrite Running an Example to use
the YAML launcher (python -m flame.launch.run_experiment <yaml>)
instead of the stale config_dir0.1_num300_traceFail_6d_3state_oort/
path that no longer exists.
- Point users at examples/README.md for the per-example catalog.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix snapshot self-copy when aggregator_config.json is already in exp dir
The runner now writes the merged aggregator config directly into the
experiment directory, and then calls snapshot.create_snapshot() passing
that same path. ExperimentSnapshot was unconditionally shutil.copy()ing
the source to <exp_dir>/aggregator_config.json — which is the same file
in this new flow — raising shutil.SameFileError.
Guard the copy: only do it when the source path is genuinely outside the
experiment directory.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Add zstandard + requests to install_requires
- zstandard: imported by flame.channel for message compression; ALL
examples need it. Missing it bricks the aggregator on import.
- requests: imported by flame.common.util for dataset downloads.
Both were previously available because the legacy requirements.txt
included them. After dropping requirements.txt, they need explicit entries
in setup.py.
Other lazy-imported deps that DON'T need declaring at base level:
- tensorflow (only inside MLFramework.TENSORFLOW branches)
- pynvml (lazy, inside try/except in resource_monitor)
- sklearn / functorch (only inside fwdllm aggregator)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* flame.config: explicit default=None on Optional fields for pydantic v2
Pydantic 1 implicitly treated \`Optional[X]\` (with or without Field(...))
as having a None default. Pydantic 2 does NOT — every Optional field
must declare \`default=None\` explicitly, otherwise it's REQUIRED and
fails validation.
Symptom from the smoke test:
ValidationError: 2 validation errors for Config
hyperparameters.weightDecay: Field required
groups: Field required
Adds default=None to:
- Hyperparameters: batch_size, learning_rate, weight_decay
- Channel: description, backend, broker_host
- Config: realm, hyperparameters, registry, selector, base_model,
groups, dependencies, func_tag_map
The remaining \`PydanticDeprecatedSince20\` warning about
\`class Hyperparameters(FlameSchema, extra=Extra.allow)\` still works in
pydantic 2's compat layer; can be migrated to \`model_config = ConfigDict(extra='allow')\`
in a follow-up if we want to drop the warning.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Config: populate_by_name, fix wrong dict-field defaults, str typing
- FlameSchema.model_config = ConfigDict(populate_by_name=True) so JSON
keys are accepted via either field name or alias. Without this,
pydantic v2 only honors the alias and our `client_notify` JSON key
silently fell through to the default.
- client_notify default: False -> None (it's typed Optional[dict]).
- training_delay_s / training_delay_factor defaults: False -> None.
- use_oort_loss_fn type: Optional[float] -> Optional[str] (the value
is the string "True"/"False" in every config and trainer code
compares it as a string).
- main_oort_agg.py: track_trainer_avail debug print uses .get(...)
with defaults instead of crashing when 'trace' isn't set (e.g.
Felix, which uses client_notify, not oracular tracking).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Pass VAL_CH_STATE_SEND to channel.ends() in oort distribute path
async_oort's select() requires KEY_CH_STATE in channel_props, which
Channel only populates when ends() is called with a state arg. The
oort top_aggregator's _distribute_weights was calling channel.ends()
with no arg, raising KeyError('state'). Matches the pattern used by
flame.mode.horizontal.asyncfl.top_aggregator line 825.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Felix on asyncfl stack; baseline owns aggregator main + stack validation
Felix uses async_oort + fedbuff, which require the asyncfl top aggregator,
not the sync oort one. The smoke/n300 experiments were pointing at the sync
main, causing the aggregator to mistrack in-flight ends and hang.
- Rename aggregator entrypoints for clarity:
main.py -> main_asyncfl_agg.py, main_oort_agg.py -> main_oort_sync_agg.py
- baselines.yaml: each baseline declares example.aggregator_main (the stack)
- Launcher: baseline owns aggregator_main (experiment override errors out);
fail-fast validation that async selectors run only on the asyncfl stack
- main_asyncfl_agg.py: --config-json support + guard missing trackTrainerAvail trace
- Mark legacy direct-invocation shell scripts deprecated; use the YAML launcher
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Add fedbuff/fedavg baselines + smoke YAMLs for launcher
- baselines.yaml: fedbuff (async stack) and fedavg (sync stack) entries
- fedbuff/fedavg n10 alpha100 smoke experiments
- main_fedavg_agg.py: gate wandb behind --log_to_wandb (was a module-level
wandb.init that ran on import) and accept --config-json via the launcher
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Add oort/refl smoke YAMLs (oracular sync stack)
Both run on main_oort_sync_agg.py with oracular trackTrainerAvail (trace from
the experiment). Validated baseline merge + stack compatibility.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Docs: baseline smoke-test guide, FedDance TODO, first-task writeup
- README: replace deprecated run instructions with launcher-based baselines
table (felix/fedbuff/fedavg/oort/refl smoke tests); fix renamed entrypoint
refs; point to first task
- FEDDANCE_TODO.md: why feddance isn't a runnable async_cifar10 smoke yet
- ava_first_task.md: guided task (split delay into compute+RTT, then make RTT
time-varying) with file/function pointers, no code
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Fix --config-json parsing in aggregator mains
The mains declared a positional 'config' but not --config-json, so argparse
assigned the JSON string to the positional and tried to open it as a file
(OSError: File name too long). Move all config parsing (positional + --config +
--config-json) into load_config_from_argv; mains only parse wandb flags.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Add launcher migration guide; drop legacy positional-config parsing
- examples/MIGRATING_TO_LAUNCHER.md: reference playbook for moving an example
(e.g. async_google_speech, fwdllm) off per-trainer JSON + shell scripts onto
the YAML launcher + shared _metadata. Covers trainer/aggregator changes,
adding dataset data to metadata, baseline catalog + stack validation, a
migration checklist, and legacy decommission policy.
- examples/README.md: link to the guide.
- cli.py: drop the legacy positional config arg; supported intake is now
--config-json (launcher) and --config <file> (manual).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Drop legacy async_cifar10 aggregator configs + slow_main entrypoint
Removed 46 unused aggregator/*.json (superseded by _metadata/aggregator_base.json
+ baselines.yaml) and aggregator/pytorch/slow_main.py (no running stack). The
launcher path references none of them. Deprecated *_expts/ shell-script dirs are
kept as historical records (see their DEPRECATED.md).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Complete fedbuff agg_rate_conf in felix/fedbuff baselines
fedbuff optimizer requires use_oort_lr, dataset_name, and a full agg_rate_conf.
- felix: add scale/a_exp/b_exp to agg_rate_conf.type=new (KeyError: 'scale')
- fedbuff: kwargs were empty; set use_oort_lr=False, dataset_name, agg_rate_conf.type=old
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* deleting old files for expt runs
---------
Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* async_cifar10: add streaming-data trainer mode; drop dead oort trainer Remove the orphaned main_oort_trainer.py: the launcher always spawns trainer/pytorch/main.py for every baseline, and main.py's base class already provides the oort methods, so it serves oort/refl too. Add an opt-in data-streaming mode to main.py: instead of exposing the full partition at init, retain the full pool plus a one-time shuffle (seeded by trainer_id) and reveal a growing prefix over time. Controlled by a data_streaming config block (enabled, full_data_available_after_s); unlock is linear in sim time (scaled by speedup_factor), floored at 1 sample, and stays at 100% after X. The loader is rebuilt per selection in train()/evaluate(); disabled by default in trainer_base.yaml. Document the feature for future example migrations in MIGRATING_TO_LAUNCHER.md (§3 + §5) instead of a standalone doc. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * async_cifar10: structured telemetry + post-run analysis; fix speedup wiring and async availability robustness Telemetry (generic, reused via base classes; no-op unless FLAME_TELEMETRY_DIR set): - flame/telemetry: JSONL TelemetryWriter + typed event schema (selection, agg_eval, agg_round, trainer_round, util_disparity, avail_change) - selector AbstractSelector.emit_selection() centralizes the selection schema (availability composition, utility/speed, chosen set) for cross-selector comparison; wired into oort/async_oort/fedbuff/feddance - aggregator loss/acc via base update_metrics; staleness/agg-goal/in-flight in sync+async _aggregate_weights - trainer per-round timing/availability + opt-in streamed-vs-full counterfactual statistical-utility (util_counterfactual config) - scripts/analysis/analyze_run.py + plot_helpers.py: post-run plot bundle with a --compare mode; launcher auto-runs it and exports FLAME_TELEMETRY_DIR per process - tests/telemetry: writer + schema + streaming-disparity tests Launcher / async fixes (required to run speedup/availability experiments): - spawner now passes --speedup_factor/--battery_threshold to the trainer (were ignored, so speedup_factor always fell back to 1.0) - recv_fifo gains an optional timeout; async aggregate receive is now bounded (RECV_TIMEOUT_WAIT_S) so a quiet/ghost in-flight trainer can't hang the loop; skip cycle when no live ends remain - async_oort drops departed ends from selected_ends to free concurrency - UTF-8 launcher stdio so status glyphs don't crash on latin-1 locales See examples/async_cifar10/TELEMETRY_AND_SPEEDUP_PLAN.md for the Task 2 follow-ups (virtual-clock decoupling, in-flight reconciliation, decorrelated availability). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * async_cifar10: simulated-mode correctness + speedup + parity tooling Core simulated-mode bug fixes (aggregator): - run_async: cancel future on timeout so stale _get_message_inner coroutines don't steal messages from _rx_queue (fix #1) - _sim_recv_min: buffer messages keyed by actual sender (metadata[0]) not the probed end, preventing wrong-end commits (fix #2) - _aggregate_weights: bypass recv_ends==None guard when _sim_buffer has entries; probing marks ends RECVD so all 10 end up RECVD after 3 calls, stranding buffered messages forever (fix #3) - Guard PROP_ROUND_DURATION.total_seconds() against None crash Simulator fidelity fix (trainer): - sim_round_duration = real_gpu_time_s + training_delay_s (was just D). Without GPU time, all trainers report identical D, making sim_completion_ts equal for every trainer selected in the same asyncfl-loop iteration → deterministic OORT over-selection of the same top-5 every round. Adding GPU jitter gives OORT real speed heterogeneity, replicating real-mode arrival diversity. Speedup improvements: - SIM_RECV_FILL_TIMEOUT_S: 2.0s → 0.5s (probe granularity, not training wait; shorter = less wasted time per still-computing trainer) - Inter-send sleep: 0.5s → 0.2s in simulated mode (broker still protected; trainers cycle faster in sim so lower interval is safe) - evalEveryNRounds config (default 10): skip aggregator evaluation on non-eval rounds, saves ~2s/round Experiment runner fix: - runner.py: wait for aggregator.process.wait() before calling wait_all(). Previously wait_all fired immediately after spawn, killing each trainer after 30s regardless of training progress (caused runs to stall after ~44 rounds with 10 trainers) New tooling: - scripts/compare_parity.py: 6-dimensional real-vs-sim comparator (selection Jaccard, utility KS-stat, aggregation sequence, staleness, participation counts, convergence curves) - scripts/analyze_send_recv_lag.py: post-run SEND_RECV_LAG parser with per-trainer p50/p95/max and threshold warnings - expt_scripts_2026/felix_n10_parity_real_vs_sim.yaml: sequential real+sim parity experiment (same config, only time_mode differs) - SEND_RECV_LAG instrumentation in _aggregate_weights for wall-clock send→receive lag logging per trainer per round Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * async_cifar10: fix OORT trainer deadlock in simulated mode Root cause: in simulated mode, all N trainers respond immediately so all N messages are delivered in the first asyncfl-loop iteration of each round. Only agg_goal (5) are committed; the remaining N-agg_goal trainers have their messages buffered in _sim_buffer but are never freed from OORT's all_selected. The deadlock: a trainer can only leave all_selected by being committed (cleanup_recvd_ends processes ordered_updates_recv_ends). But it can only be committed if OORT selects it. But OORT excludes all_selected from filtered_ends. So: locked in all_selected → never selectable → never committed → never freed. Permanent. Traced via telemetry: from round 3 onward, filtered_ends was always exactly {370,373,375,376,379} — the same 5 trainers that happened to commit first in rounds 1-2. Trainers {371,372,374,377,378} (including trainer 378 with the HIGHEST stat-utility) were completely locked out for 98 of 100 rounds. Fix: after cleanup_recvd_ends() at round end, release all trainers remaining in all_selected (selected but not committed). Their buffered messages remain valid in _sim_buffer as legitimate stale updates for future rounds. State is reset to NONE so OORT can re-probe them. Effect: with all trainers eligible each round, OORT's temporal uncertainty term (sqrt(0.1*log(V)/last_selected)) accumulates for unselected trainers and rotates selection correctly. At round 50, a trainer not selected since round 1 has temporal bonus ~0.625 which outweighs the ~0.1 bonus of a trainer selected last round — matching the rotation behavior seen in real-mode runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * deleting old markdowns, latest fixes to sim, lots of expt configs including old refl yamls --------- Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
… detection (#62) (#62) Unified timing model (trainer, aggregator, OORT): - sim_round_duration = max(gpu_time, D) in both modes, mirroring real recv_ts-sent_ts. No contention: max(~0.5s,D)=D; overrun: gpu>D so OORT correctly deprioritizes contended trainers. - Real mode sleep = max(0, D-gpu); sim mode virtual advance = max(gpu,D). - Adds MessageType.TRAINING_BUDGET_S (37) to carry D to the aggregator. GPU contention detection: - Trainer logs [TIMING_OVERRUN] when gpu > D with advice to reduce trainers-per-GPU or add GPUs. - Aggregator logs [TIMING_OVERRUN_AGG] from budget vs elapsed in both modes. OORT crash fix (sample_by_util ValueError): - Root cause: timedelta(0) round duration -> system_utility=0 for all trainers -> np.random.choice(replace=False, p=[...]) fails. Fixed at source (max(gpu,D) is always >0) plus a defensive zero-prob filter in sample_by_util. compare_parity.py enhancements: - Check 9: GPU contention analysis (overrun fraction; FAIL>25%, WARN>10%). - --plot-out PATH: 2-panel timing sanity plot (mean GPU vs budget bars, round-by-round deviation). Graceful on old runs without training_budget_s. Docs + first-task guides: - TELEMETRY_AND_SPEEDUP_PLAN.md: new 2026-06-01 update section; Q2 corrected to reflect max(gpu,D) not just D. - david_first_task.md: add per-trainer lat/long mobility trace, emit as telemetry each round (real mode). - seshu_first_task.md: sinusoidal training_delay_s variation ±20% of mean, 0.5s floor, config-gated (real mode). - README.md: next-steps table with all three first-task files. Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
#63) * Streaming opportunity-gap experiment: mis-selection telemetry + oracle + harness Adds the machinery to compare Felix vs OORT vs REFL under streaming data and show that OORT/REFL mis-select clients (stale statistical utility) while Felix tracks true utility via its eval-selector. Telemetry / ground truth: - Periodic global-model checkpoints (save_round_checkpoint on the shared aggregator base; wired into sync + async loops; gated by checkpoint.*). - New trainer_round fields: delta_weight_l2 (||update||), grad_norm_epoch1, task_to_perform -> relate update magnitude to amount of unlocked data. - scripts/analysis/oracle_misselection.py: offline oracle reconstructing each trainer's TRUE current utility (deterministic split+seed+visible prefix) and joining with logged believed utility to emit per-round mis-selection rate, utility regret, believed-minus-true gap, and selected-set true utility. Self-calibrates the streaming horizon from util_disparity telemetry. - analyze_run.py: comm-cost (model-equivalents, charging eval probes), new single-run plots (update/loss/utility vs visible fraction, mis-selection, comm-vs-accuracy), and compare_streaming() cross-baseline bundle incl. time-to-target. Experiment configs (n300, alpha=0.1, syn_0, agg_goal=10, 8 GPUs, horizon 10800s): canonical 6-arm YAML + per-node split (streaming/control) + 300-round pilot variants + n10 smoke. Pre-existing OORT selector fixes (blocked all sync OORT/REFL runs): - save_exploited_utility_history: len()==0 instead of truthiness on np array. - sample_by_util / sample_by_speed: cast np.str_ ids to str (dict-key safe). - increment_selected_count / stats loop: skip in-flight ids absent from ends. Sync aggregator evaluate() now honors evalEveryNRounds (was every round; the full test pass dominates per-round cost at n300 and made sync runs intractable). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Default experiment configs to time_mode=real Switch all 6-arm / node-split / pilot configs from simulated to real time_mode while simulated-mode issues are being resolved. In real mode the streaming clock is wall-clock (full_data_available_after_s is real seconds) and trainers sleep their modeled delays; the horizon may need re-calibration against the pilot's real per-round wall-clock, and the mis-selection oracle's visible-count reconstruction becomes approximate (documented in the YAML header). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Wire FedDance into async_cifar10 + fix base syncfl stack for fedavg/feddance FedDance is now a runnable async_cifar10 baseline (felix/oort/refl/feddance comparison). Resolves examples/async_cifar10/FEDDANCE_TODO.md. - baselines.yaml feddance: add aggr_num (default 10), switch tracking from ORACULAR to disabled (FedDance uses its own check-in predictor). Stack stays main_fedavg_agg.py (base syncfl), which drives on_update_received / on_round_completed (the Oort stack overrides _aggregate_weights and does not). - trainer/main.py: accumulate per-round local training accuracy (reset_local_accuracy + update_local_accuracy) so FedDance's A_m signal is real (harmless for other selectors). - main_fedavg_agg.py evaluate(): honor evalEveryNRounds (was every round). - syncfl/top_aggregator base fixes that unblock fedavg + feddance on the base stack: default FwdLLM-only data_id/iteration_per_data_id via getattr; guard PROP_ROUND_DURATION.total_seconds() against None; iterate diskcache Cache with list(self.cache) instead of .keys(). - Add 4-way quick pilot config (felix/oort/refl/feddance, 32 trainers, real mode, streaming) for ~30-min end-to-end validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix: REFL arms need aggr_num (OortSelector requires it) REFLOortSelector extends OortSelector, whose __init__ requires kwargs["aggr_num"]; the refl baseline and refl experiment arms never set it, so every refl run raised KeyError at selector init (caught by the 4-way pilot). Add aggr_num=10 default to the refl baseline and explicit aggr_num per refl arm (= agg_goal: 10 full / 5 pilot). Regenerated node-split files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * REFL: emit selection telemetry so the mis-selection oracle can score it REFLOortSelector.select() never called emit_selection, so EVENT_SELECTION was absent for REFL runs and the oracle/comparison had no REFL believed-utility / selected-set data. Add the same emit_selection call as oort/feddance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add FedDance arms (8-arm experiment) + 2-node, 2-phase run scripts - Canonical experiment now has 8 arms = {felix,oort,refl,feddance} x {streaming,control}; renamed felix_oort_refl_streaming_alpha0.1.yaml -> felix_oort_refl_feddance_alpha0.1.yaml. - Replace the old node-split (node=streaming-only / node=control-only) with a 2-phase scheme: STREAMING_node1/2 then CONTROL_node1/2, each node running 2 arms (node1=felix+oort, node2=refl+feddance) so both nodes do streaming first, then both do control. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Selector staleness audit: design doc + per-factor I_m staleness (step 1) - SELECTOR_STALENESS_AUDIT.md: documents each selector's input factors (OORT, REFL, FedDance, Felix) with believed-value source + staleness, true-value computation, and which are real staleness dimensions vs exact bookkeeping; lays out the self-relative counterfactual-replay framework. - analyze_run.py: im_staleness_by_round + compare overlays (compare_Im_rankcorr, compare_Im_gap) — per-baseline rank-correlation and normalized gap between BELIEVED and TRUE statistical utility over time, computed from existing oracle_utility.csv (no re-run needed). First staleness dimension; speed + A_m + the no-drift counterfactual replay are the next steps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Staleness audit (step 2, live): log believed scoring factors per candidate So the offline audit can compare believed-vs-true per factor (not just I_m): - FedDance: log V_m, I_m, A_m, U_m and last_engaged_round per candidate in EVENT_SELECTION (it already computes PROP_V/I/A/U on ends). - Felix (async_oort): add last_train_round (PROP_LAST_SELECTED_ROUND) alongside last_eval_round so the audit can show whether believed I_m was refreshed by eval vs train -- the freshness mechanism under test. These are additive logging fields; no selection-behavior change. Must land before the full runs so their telemetry captures the factor bundle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Staleness audit (step 3a): shared pure scoring fns + per-candidate score logging - New flame/selector/scoring.py: pure, side-effect-free scoring formulas (oort (util+temporal)*system_util; feddance V*I*A*MAB; topk) shared by the live selectors and the offline counterfactual replay so they never drift. - OORT + AsyncOort(Felix): route the score combination through scoring.* and stash per-candidate components (believed_I, temporal, system_util), logged in EVENT_SELECTION.per_trainer for the audit. (Removed OORT's dead 95th-pct clip — it was overwritten before use; behavior unchanged.) - REFL inherits OORT's scorer; emits the stashed components too. All 127 selector tests pass (behavior-preserving). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Staleness audit (step 3b): oracle counterfactual replay + true A_m - oort_utility_acc: also returns local top-1 accuracy from the forward pass; per-checkpoint loop stores it -> true A_m = accuracy slope across checkpoints (FedDance's accuracy-increment, true value). - Counterfactual replay per selection: re-rank candidates by each selector's own score with believed factors vs true factors substituted (I_m for all; A_m too for FedDance; speed ~identity here), using the logged score components + shared scoring fns. Emits self-relative cf_misselection_rate + cf_utility_regret (oracle_counterfactual.csv) and run-level means in oracle_summary.json. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Staleness audit (step 3c): self-relative regret in own-score units + cf reporting - oracle: counterfactual regret now measured in each selector's OWN true-score units (true_topk maximizes true_score => regret >= 0), fixing negative values for multi-factor selectors (FedDance V*I*A). mis-selection rate unchanged. - analyze_run compare_streaming: add compare_cf_misselection / compare_cf_regret overlays (self-relative staleness penalty per baseline) from oracle_counterfactual.csv. Validated end-to-end on a 4-way n10 audit smoke: counterfactual populates for felix/oort/refl/feddance via the shared no-drift scorers + logged components + true I_m/A_m. (Smoke numbers are noise at n10/near-random; real signal needs the full runs.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Per-run plot suite: PDF, config-stamped, grouped (performance/sanity/selection/insights/system) plot_helpers: paper-style rcParams (fonttype 42), fixed per-selector colors, PDF output, config_stamp() from snapshot, and new figure types (scatter-with-diagonal, banded line, histogram, signed bar, heatmap, dual-axis); quiet font/mpl logging. analyze_run: reorganized into 5 per-run plot classes emitting PDFs: - performance: accuracy (vs round & sim-time), Δacc/eval, loss, accuracy-vs-data- unlocked, global weight-change norm (from checkpoints). - sanity: trainer runtime expected-vs-actual (+residual hist, early/late counts, overrun rate), utility believed-vs-true (+discrepancy over rounds), data-unlock band, selection/aggregation count consistency, stale-rejection (when logged). - selection: selected-vs-pool true utility, coverage+Gini, frequency, exploration factor, eval-vs-train, participation heatmap, availability composition. - insights: I_m staleness, mis-selection, counterfactual mis-selection/regret, data-unlock effects, streamed-vs-full utility. - system: comm-vs-accuracy, per-round train/eval comm, time breakdown, queue depth. telemetry: OORT stack now emits agg_round (it overrode _aggregate_weights and emitted none) so oort/refl get staleness/contributing/queue visibility; trainer logs current lr. Validated end-to-end on a felix n10 smoke (32 PDFs). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * config_stamp: infer streaming from telemetry (snapshots omit config_overrides) Snapshots don't serialize trainer.config_overrides, so the stamp showed stream:off even when streaming was on. Fall back to inferring it from telemetry (a trainer_round with visible_samples < total_samples) so the config stamp is accurate on the real runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Plot/telemetry refinements: labels, hist %, top config title, selection & runtime sanity Per review feedback: - (i) trainer labels use last-3 digits (per-trainer bars / heatmap / breakdown); these plots already list only trainers (no aggregator). - (ii) histograms annotate each bar with its % of total. - (iii) smaller/crisper title & axis-label sizes (rcParams) so they fit the box. - (iv) config stamp moved to the TOP of each figure; also inferred from telemetry (visible<total) since snapshots omit config_overrides (was showing stream:off). - (iv-b) new selection plot: per-round avg speed & avg believed utility of the clients actually picked. - (v) aggregator-observed vs trainer-reported response time: new agg_observed_s field on agg_round (send->recv wall per contributing trainer) emitted by oort/asyncfl/syncfl stacks; new sanity plots runtime_agg_vs_trainer (scatter w/ diagonal) + runtime_overhead_hist (observed - reported) to catch aggregator processing/network overhead (the "trainer says 10s, aggregator sees 14s" case). Fix: OORT agg_round now emitted BEFORE optimizer.do (which consumes the cache) and reads cached TrainResult fields, so contributing/staleness/agg_observed populate. Analyzer joins per trainer by nearest round (robust to staleness). Validated on felix+oort n10 smokes (all plots render; OORT overhead plots populate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add max_runtime_s wall-clock cap (3.5h) to all experiment arms New optional aggregator hyperparameter max_runtime_s: when set, the aggregator stops the run once that many wall-seconds have elapsed since it started (checked in increment_round, so it applies to both sync and async stacks). Set to 12600 (3.5h) on all 8 arms + node files: streaming data fully unlocks at 3h, so this caps each arm shortly after without burning compute to the 1000-round budget. Validated: a run with max_runtime_s=60 stopped at round 3 (not 1000). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix streaming recv_fifo active-task leak that stalled async runs Channel._streamer_for_recv_fifo tracks one "active task" per end while awaiting that end's message. The streamer is fire-and-forget and outlives the recv_fifo caller, and the per-end End.get() had no timeout while the active-set entry was only removed on successful delivery. So an end whose trainer never sends (slow / dropped / unavailable) blocked forever and stayed permanently in _active_recv_fifo_tasks. That end was then skipped ("already has active task") on every future receive, so its updates were never consumed -> the aggregator stalled with a monotonically growing active_tasks count (observed: 1 -> ~30, then no progress). Fix: - Thread the caller's timeout into _streamer_for_recv_fifo and wrap each per-end get() in asyncio.wait_for, so a quiet end releases instead of blocking forever. timeout=None preserves legacy blocking for sync callers. - Move active-set cleanup into a finally (discard) so an end is always released on delivery, timeout, error, or cancellation. - Don't enqueue non-messages (timed-out ends) so they can't consume a first_k slot ahead of a real update. Simulator mode (_sim_recv_min, which probes recv_fifo per-end with a short timeout) benefits from the same fix and is verified safe: CPython's wait_for + Queue.get is cancellation-safe, so a late message is buffered on the end and delivered on the next probe rather than lost. Tests: - tests/test_channel_recv_fifo.py: direct streamer-cleanup tests plus full recv_fifo tests over a real loop mirroring the simulator probe pattern (incl. late-message-not-lost). All pass on the fix; 8/9 fail on the pre-fix code. - tests/mode/test_async_sim_ordering.py: initialize _sim_pending_commit in the fake aggregator (pre-existing test bug; suite was red). Also: debug_stuck_run.py gains a streaming active-task leak analyzer and a UTF-8 stdout fix so it no longer crashes when piped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Update stale launch tests for baseline-owned aggregator_main ExampleConfig.aggregator_main now defaults to None (baseline-owned, with a runner-side fallback to "aggregator/pytorch/main.py"), and aggregator_main is resolved via ExperimentRunner._resolve_aggregator_main rather than being part of _resolve_example_paths. The tests still asserted the old defaults/shape. - test_experiment_config: aggregator_main default is now None. - test_runner_paths: assert aggregator_main is not in _resolve_example_paths, and exercise _resolve_aggregator_main (default, example override, baseline ownership, and the baseline+override conflict that must raise). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Split node1 STREAMING batch into per-baseline YAMLs (felix, oort) felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml runs felix then oort sequentially. Split into two standalone single-baseline YAMLs (execution config inlined, no shared anchor) so felix and oort can run on separate nodes in parallel instead of back-to-back on one node. Settings copied verbatim; num_trainers unchanged at 300. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Eliminate per-round trainer overhead that inflated "compute" time At high trainer-per-host concurrency (e.g. 300 trainers / 8 GPUs), the async runs showed large excess in the aggregator's send->recv lag beyond each trainer's modeled budget D. Decomposing the lag (new analyzer below) showed it is entirely trainer-side and NOT actual compute, the aggregator, or the network: ~2.3 s/sample of "GPU compute" on 1-sample rounds. The cost was defensive memory-hygiene + profiling that backfires under co-location. Trainer (examples/async_cifar10/trainer/pytorch/main.py): - MemoryProfiler is now gated off by default (memory_profiling_enabled hyperparameter). Its per-round heap walks (gc.collect + 3x gc.get_objects() with a per-object torch.is_tensor() check, x4 calls/round, two inside the GPU-timing window) dominated per-round wall time. - Removed per-round / per-epoch / per-50-batch torch.cuda.empty_cache() and the redundant gc.collect()s: under co-location empty_cache forces a CUDA sync and frees the allocator's blocks, so the next round re-allocates from the driver (serialized across processes) -- it hurts rather than helps. - grad-norm (epoch 1) and delta_weight_l2 are now telemetry-gated and use one fused on-device reduction + a single .item(), instead of a .item() per parameter (which forced ~12 GPU->CPU syncs each). - Added first-class per-phase timing (pre_train_s / post_train_s) to [TRAIN_CYCLE] logs and the trainer_round telemetry. Library (flame/mode/horizontal/syncfl/trainer.py): - update_local_accuracy() accumulated correct-count via .item() EVERY batch (a GPU sync per batch). Accumulate on-device; sync once in finalize_local_accuracy(). Analysis (examples/async_cifar10/scripts/analyze_timing_overrun.py): - New diagnostic: histogram of late responses by lateness bucket + CDF of deviation extent, each split into trainer-attributable vs aggregator-observed (plus transport), so future runs show where the excess originates. Also surfaces TIMING_OVERRUN_AGG / SEND_RECV_LAG_HIGH counts. Validated A/B (48 trainers / 2 GPUs, felix/async_oort): per-round trainer compute dropped 0.51s->0.03s (real) and 0.46s->0.01s (sim); aggregator-observed lag 0.99s->0.09s (real). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Task 3: real/sim parity hardening — seeding, determinism fix, checks, tests Make simulated mode verifiably decision-equivalent to real mode (not just faster). Adds the seeding prerequisite, fixes a parity-breaking RNG bug, extracts a shared parity-check module, and lands tests at every level. Code: - Aggregator (syncfl/top_aggregator.py): optional deterministic seeding of the process-global np.random/random (which the selector draws from) + torch, from a `seed` hyperparameter, at internal_init. seed=None preserves legacy behavior. Without this, selection can never match across runs/modes even with identical state. - FedBuff (selector/fedbuff.py): REMOVE the per-call `random.seed(time.time())` in _handle_send_state. It reseeded the global RNG with wall-clock time on every selection, making selection non-reproducible by construction, clobbering global random state for all other consumers, and defeating any deterministic seed. This was a primary source of real/sim selection divergence. Shared checks: - New scripts/parity_checks.py: canonical, dependency-free loaders/helpers + parity/invariant functions (selection Jaccard, aggregation-sequence, staleness, participation, sim_send_ts, GPU-budget, virtual-clock monotonicity, agg_goal cycles). compare_parity.py now imports the shared loaders/helpers (single source of truth for CLI + tests). Tests: - tests/sim/test_virtual_clock.py: buffer overwrite/clear, equal-ts advance. - tests/mode/test_async_sim_ordering.py: real-path vs sim-path commit + staleness equivalence under sorted arrival; sim recovers completion order and yields a canonical staleness sequence under any arrival permutation. - tests/selector/test_selection_determinism.py: FedBuff + Oort selection is a pure function of (state, seed) — same seed reproduces, distinct seeds vary. - tests/mode/test_parity_checks.py: unit tests for the parity functions on synthetic telemetry (default suite, no MQTT/GPU). - tests/mode/test_real_sim_e2e_parity.py: opt-in (env-gated) e2e parity assertion over two run dirs, reusing parity_checks. Configs: seeded parity pair (felix_n48_parity_seeded_{real,sim}) for the faithful regime + a stress pair (felix_n48_stress_{real,sim}, 48/1-GPU, higher c/agg_goal) to map where sim ordering degrades; n48 dataset split. Determinism analysis + phased plan documented as Task 3 in TELEMETRY_AND_SPEEDUP_PLAN.md. 252 passed, 7 e2e skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sync sim-ordering for real/sim parity across sync baselines + doc Sync aggregator previously ignored time_mode in aggregation: it committed the first_k updates by PHYSICAL ARRIVAL even in simulated mode, so the wrong k were aggregated (arrival != sim-completion order without sleeps) and round duration was wrong. This broke parity for every sync baseline (fedavg/oort/refl/feddance). - syncfl/top_aggregator.py: new _sync_sim_recv_first_k. In simulated mode buffer the selected ends' updates and commit the first_k with the SMALLEST sim_completion_ts (the k that would finish first in real), advance the virtual clock to the k-th smallest, and source PROP_ROUND_DURATION from SIM_ROUND_DURATION (so OORT/REFL see the simulated speed). Real mode keeps the arrival-ordered first_k path. Aggregation is order-independent (weighted average), so parity needs only the committed set + round duration. - tests/mode/test_sync_sim_ordering.py: commit-set == k-smallest-sct independent of arrival permutation; vclock advances to k-th smallest; round-duration from SIM_ROUND_DURATION; graceful when fewer than k respond. - fedavg_n48_parity_seeded_{real,sim}.yaml: seeded sync parity pair for live validation. - Doc: cross-baseline coverage + a characterized OPEN issue — async (felix) sim over-selects (in-flight grows toward N under no-sleep; staleness ~8.6 vs ~1.2 real). A targeted slot-accounting fix did not resolve it (buffer fills faster than it drains); reverted rather than shipped unverified. Needs a focused pass to bound the sim in-flight buffer to c. Sync path is unaffected. 257 passed, 7 e2e skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * random selector: work on sync FL too; + logical commit-sequence parity tool random selector fixes (unblocks fedavg on the sync stack): - The active select() crashed with KeyError 'state' under sync FL: it read channel_props[KEY_CH_STATE] unconditionally, but the SEND/RECV channel state is the *buffered* concurrency pattern (FwdLLM / async stack) — stateless sync FL never sets it. Default the state to SEND when absent, with a comment on why it exists. fedavg/oort/refl/feddance(sync) + FwdLLM(buffered) both work now. - Removed a fully-duplicated dead select() method (silently overridden by the second definition) — the source of the confusion and ~95 lines of bloat. parity_checks: commit_sequence() + first_divergence() — a mode-agnostic LOGICAL ordering of committed updates (by round, agg_goal_count; no wall-clock) so real vs sim can be diffed for control/ordering bugs independent of timing. On the felix runs it localizes the first divergence to commit #1 (selection-level RNG desync), before staleness blows up. 260 passed, 7 e2e skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * plots: semantic participation heatmap, response-lateness CDF, aggregate time-split, acc-gain dual-axis Addresses several readability/clarity items in the run analyzer (works for all async_cifar10 baselines; reads existing telemetry): - participation heatmap (#12): discrete semantic colours instead of viridis — grey=not selected, light-blue=eval, light-green=train, light-red=unavailable, orange=selected-but-unavailable (availability forward-filled from avail_change), with a legend. New `heatmap(discrete=...)` path in plot_helpers. - response-lateness CDF (#10): the existing residual compares GPU time to budget ("early" just means GPU finished early — the trainer still SLEEPS to fill the budget). New CDF of lateness = max(gpu,D) - budget, which is 0 (on-time) or >0 (overrun), never negative — clarifies that trainers never respond early. - aggregate round-time split (#19/#5): stacked-area of mean pre/gpu/post/sleep per round across trainers — the system-level view of where round time goes (uses pre_train_s/post_train_s; visually confirms the per-round overhead fix). - accuracy-gain per eval (#4): now overlays overall accuracy on a secondary axis (new `signed_bar_line` helper) so delta and absolute trajectory show together. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * sync aggregator: guard channel.ends() returning None in sim mode In simulated mode the sync aggregator loop spins without sleeps, so channel.ends() can transiently return None before trainers join + get selected — crashing on len(None) / enumerate(None). Real mode's pacing masked it. Guard all call sites in _aggregate_weights and _distribute_weights (skip+retry on None/empty; len(... or [])). Fixes the immediate sim crash for sync baselines (fedavg validated: distribute now works, no NoneType). NOTE: a separate open issue remains — the sync sim aggregation does not yet commit (SYNC_SIM_RECV=0); see TELEMETRY_AND_SPEEDUP_PLAN / handoff. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add handoff doc: trainer overhead, real/sim parity, plots (state + next steps) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * real/sim parity across all 6 baselines + n300 overnight sim infra + plots Aggregator parity fixes (sim mode now tracks real per baseline): - syncfl: distribute->SEND / aggregate->RECV channel state so the buffered (random) selector returns the in-flight set instead of stalling (fedavg/feddance). - asyncfl: bound sim in-flight to c (keep pending in selected_ends, reset buffered RECVD->NONE) fixing felix over-selection/staleness; guard async_oort-only trainer_eval_recv_ends so fedbuff's eval path doesn't crash. - oort stack: add sim-ordering (sim_send_ts stamp + sct-ordered commit + vclock) and cross-round straggler carry (persistent buffer) for oort/refl; oracular availability now uses the virtual clock in sim (was wall-clock). - Join barrier (min_trainers_to_start) so sim/real select from the same pool; configurable timeout for n300. - Baseline staleness semantics (oort reject / refl accept <=5) preserved in sim. Parity tooling: selection-parity check gated for stochastic selectors (DETERMINISTIC_SELECTORS) with participation-frequency parity enforced instead. All 6 baselines pass the e2e parity suite; seeded real/sim config pairs added. n300 overnight: OVERNIGHT/SIMULATED node configs (sim+real per baseline, both with the barrier); non-interactive batch runner + _sweep_stragglers cleanup between runs; overnight_run.sh (smoke+run) and compare_overnight.sh. Plots: comm in MB split by task+direction (train down/up, eval down) + message accounting; Lorenz/Gini fairness; late-fraction hist; P99; overhead CDF; compute time by task; speed/utility CDFs (expected vs actual); whole-run time-split bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * overnight scripts: portable conda activation + derive repo paths from script location Auto-detect conda base via `conda info --base` / CONDA_EXE with fallbacks (incl. /coc/scratch/$USER), instead of hardcoding ~/miniconda3; fail loudly if not found. Derive the example/repo dirs from ${BASH_SOURCE} so the scripts work regardless of where the repo or conda live on each node. FLAME_CONDA_ENV overrides the env name (default dg_flame). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * overrun debug: WALL_SEND_TS + MQTT lag decomposition + debug_run.sh Add WALL_SEND_TS (MessageType 38) stamped by the trainer at channel.send() so aggregators can split wall_lag_s into training_elapsed (GPU+setup+sleep) vs mqtt_lag (broker delivery). Both async and sync aggregators now emit [MQTT_DELIVERY_LAG] lines with the full decomposition for real-mode runs. New analysis plots: sanity/pre_train_s_cdf.pdf — GPU queue wait (large = contention) sanity/gpu_compute_cdf.pdf — on-device GPU time for train rounds system/mqtt_delivery_lag_cdf.pdf — pure broker delivery time CDF system/wall_lag_decomposition_cdf.pdf — wall/train_elapsed/mqtt together New debug_run.sh: configurable script to run just felix (node1) or refl (node2) at a specified wall-clock budget (default 1h, max_runtime_s applies as virtual-clock seconds in sim mode via the vclock fix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * scripts: move run logs from /tmp to repo run_logs/ dir /tmp can be small on shared nodes; stdout/stderr from 300-trainer runs can reach hundreds of MB. Both overnight_run.sh and debug_run.sh now write to <example_dir>/run_logs/{overnight,debug}/ which lives on the same filesystem as experiments/. The run_logs/ tree is gitignored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * scripts: revert log dir back to /tmp run_experiment stores all trainer/agg logs in experiments/run_*/ already; LOGDIR only holds the launcher stdout and generated YAML configs (both small), so /tmp is fine. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * timing: 6-component LAG_DECOMP + TIMING_OVERRUN_AGG + wall-clock failsafe - Replace [MQTT_DELIVERY_LAG] with [LAG_DECOMP] across asyncfl, syncfl, oort aggregators: 6 decomposition components (agg→trainer delivery, compute, post-compute wait, MQTT transit, queue/buffer wait, processing) - Add WALL_RECV_TS (39) and ROUND_COMPUTE_S (40) MessageTypes; trainer stamps both unconditionally in _send_weights (real and sim) - Remove [SEND_RECV_LAG_HIGH] fixed-threshold alerts; replace with [TIMING_OVERRUN_AGG] using budget_s + _NETWORK_SLACK_S (2s) threshold - Fix [TIMING_OVERRUN_AGG] sim check: was _virt_elapsed <= 0 (wrong), now _virt_elapsed > budget_s - Add _oort_sent_version_ts per-version send timestamps in oort for correct lag measurement on REFL stale updates (staleness ≤ 5) - Add wall-clock failsafe to increment_round(): sim runs hard-stop at max_runtime_s wall seconds even if vclock stalls (fixes REFL sim hang) - Update analyze_run.py: _parse_lag_decomp replaces _parse_mqtt_delivery_lags - debug_run.sh: default RUNTIME_S 3600 → 1800 (30-min runs) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * timing+cpupin+vclock: per-phase trainer timing, CPU pinning, sim clock fix Task 1 — per-phase trainer timing (CPU vs GPU split): - syncfl/trainer.py: add _phase_times dict + _phase() context manager; instrument mqtt_fetch_s, weights_to_ram_s, weights_to_gpu_s (fetch side) and weights_from_gpu_s, post_cpu_s, mqtt_send_s (send side); CUDA sync on GPU-bound phases for accurate timing. - trainer/pytorch/main.py: extend trainer_round telemetry extra with gpu_compute_s, sleep_s, and all _phase_times from the base trainer. - scripts/analyze_trainer_phases.py: new analyzer — reads trainer_round JSONL, averages CPU/GPU phases per round, emits stacked-bar plots (A/B) and per-trainer heatmap (C); --compare overlay for oort vs refl. Task 2 — automatic CPU core pinning per trainer: - spawner.py: TrainerSpawner discovers usable cores via sched_getaffinity(0); round-robin assigns one core per trainer; applies via preexec_fn sched_setaffinity; sets OMP/MKL/OPENBLAS/NUMEXPR_NUM_THREADS=1 to prevent thread oversubscription; --cpu_pinning on|off toggle (default on); logs trainer→(gpu,core) table after spawn_all. - trainer/pytorch/main.py: initialize() calls torch.set_num_threads(1) when OMP_NUM_THREADS=1 is inherited from spawner. Task 3 — wall-clock vs virtual-clock termination correctness: - syncfl/top_aggregator.py: increment_round reads new max_wall_runtime_s config; sim failsafe uses it (default 4×max_runtime_s) instead of reusing the virtual budget, so a sim run is not cut before vclock≈T; periodic [VCLOCK_PROGRESS] log every 30s shows vclock/wall/speedup. - scripts/debug_run.sh: --wall-runtime-s flag; make_debug_yaml writes max_wall_runtime_s into YAML (default 4×runtime_s). - scripts/compare_clock_parity.py: new script — per-round virtual-time deviation, sim speedup, failsafe-triggered check, and 2-panel plot. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * analysis: round-2 fixes — sim_rate, sim_wall_ceiling, pinning+dynamics analysis Framework changes: - syncfl/top_aggregator: rename 'speedup' -> 'sim_rate' in [VCLOCK_PROGRESS]; replace loose 4x failsafe with sim_wall_ceiling_s (default = max_runtime_s, 1x) - trainer/main.py: add [PLACEMENT] log (gpu, cpu_cores) at init - debug_run.sh: --sim-wall-ceiling-s flag; make_debug_yaml writes sim_wall_ceiling_s=runtime_s New analysis scripts: - plotters/_annot.py: shared annotate_percentiles(P50/P90/P99) + flush_percentile_table - analyze_pinning.py: CPU-core and GPU assignment balance from [PLACEMENT] logs - sanity_check_real_sim.py: T1-T7 battery (vclock monotone, accounting, sim_rate, wall_speedup, failsafe, selection balance, phase consistency); validated on 0606 runs - analyze_dynamics_timeline.py: AVL_TRAIN/UN_AVL/in_flight/idle per round + idle CDF Updated scripts: - analyze_trainer_phases, analyze_timing_overrun, analyze_send_recv_lag, compare_clock_parity: all now use shared annotate_percentiles; compare_clock_parity rewritten to clearly distinguish sim_rate (vclock/wall) vs wall_speedup (real/sim) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * longer 3h debug run * parity: new scripts/parity/ package + K2/K3/K4/K8/K10 clock checks Implements the full §3 battery from real-sim_parity_checker_plan.md: §3.H Clock & throughput (the new enforced core): - K10 vclock_telemetry_present: FAIL-LOUD when sync sim omits vclock_now - K2 throughput_parity: rounds/virtual-s parity — catches 410 vs 673 rounds - K3 per_round_advance_parity: KS + mean on per-round Δvclock vs Δwall - K4 overlap_factor: localizes sim's missing inter-round overlap model - K8 terminal_state_parity: rounds + trainers at matched virtual budget V - U2 total_commits_parity: commit count at matched V - P3 trainer_speed_parity: control check isolating divergence to clock advance Also adds utility, convergence (C3 self-compare bug fixed), trainer_speed, inter_arrival_order, budget_not_cap, failsafe_ok checks. Structure: scripts/parity/checks.py — pure functions, stdlib only scripts/parity/report.py — section-grouped stdout + JSON + PNG scripts/parity/cli.py — argparse; single-pair and --batch mode scripts/parity_checks.py — re-export shim (backward compat for pytest) scripts/parity_check.py — stable CLI entry point All 34 unit tests pass; existing e2e tests unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * cleanup: remove stale artifacts, supersede old parity scripts, trim comments Deletions: - *_TODO.md, *_PLAN.md, *_AUDIT.md, *_HANDOFF.md planning docs (captured in git history) - aggregator/agg_logs.{csv,txt} and *.pdf plots (generated artifacts) - ava/david/seshu_first_task.md onboarding docs - scripts/compare_parity.py, sanity_check_real_sim.py, compare_clock_parity.py (superseded by scripts/parity/ package + parity_check.py entry point) Updates: - scripts/compare_overnight.sh: call parity_check.py --real/--sim DIR instead of the deleted compare_parity.py with file-glob args; add --json-out - aggregator/pytorch/main_{asyncfl,oort_sync}_agg.py: remove commented-out config entries, stub-method comments, and narrating inline comments - trainer/pytorch/main.py: remove per-phase taxonomy block comment headers (field names self-descriptive); collapse initiate_heartbeat multi-comment block to one line preserving the non-obvious dup_check_and_sleep reason Added: - experiments/parity_diagnosis_20260607.md: June 7 run parity findings Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* async_cifar10: sim-fidelity fixes, causal parity ladder, and plot overhaul
Simulator fixes (PARITY.md §5 roots):
- Add sim_commit_overhead_s hyperparameter (alias simCommitOverheadSeconds,
default 0.0): per-commit MQTT/dispatch overhead the max(gpu,D) model omits,
charged on the virtual clock via shared TopAggregator._advance_sim_clock
(syncfl base; used by asyncfl/oort/syncfl commit paths). Felix=0.58,
refl=0.115 set in baselines.yaml. Fixes CRITICAL-1/MEDIUM-1 (K3b residual).
- Stamp vclock_now on agg_round telemetry in oort + syncfl (CRITICAL-2): the
sync paths never emitted it, blocking all REFL clock/throughput checks.
Parity checker -> causal ladder (checks.py/report.py/cli.py):
- Reorganize 40+ checks into a 0-9 stage pipeline with a CHECK_META registry
(stage/role/deps). overall_verdict now returns root-cause vs downstream:
the lowest broken rung whose upstreams pass is flagged ROOT; consequences
are demoted. New checks: TC1 coverage, K3a/K3b clock decomposition, A3
trace time-base, A4 duty-cycle, T2 budget control, per-phase split, C2 loss.
- Stage-grouped report with [TIER|ROLE] tags + root-cause banner.
- Fault-injection test suite (test_ladder.py) asserting localization.
Plot overhaul (scripts/analysis/analyze_run.py, plot_helpers.py):
- Root cause of "starts very high" plots: round-0 warmup (19k-62k selection
retries before training) dominated per-round aggregations. Excluded from
selection_count / comm breakdown / cumulative comm.
- CDFs: canonical red + P50/P90/P99 (3dp or first-sig-fig); multi-CDFs get a
per-series colored percentile table.
- send_recv_lag_over_rounds: parse round from version= (was round-0 only).
- Lorenz padded to full population (was n=ever-selected); coverage shows %.
- New per-trainer/aggregate/across-trainer state-fraction plots.
- Smoothed selected-speed-utility + CDF companions for trainer-time breakdowns.
- Resource plots in seconds + percentage; comm auto-scales to GB; heatmap
light-grey/darker green-blue; accuracy-gain darker colors; data-unlocked
sorted by x; trainer-time-split annotated.
Tests: 34 flame (overhead/ordering/baselines) + 5 ladder, all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim speedup: completion-barrier recv (decouple sim-time ordering from wall)
Sim mode had ~zero wall speedup (felix sim 10,494s ≈ real 10,486s). Root cause
(LAG_DECOMP): trainers respond in 0.16s (no sleep) but each update sat ~86s in
the reorder buffer — the aggregator metered commits at ~0.79s each via per-end
0.5s recv polling + a 0.5s empty-spin. The data needed to order correctly was
already present; we were just draining it slowly.
Fix (PARITY.md §6), across the 3 sim recv helpers (asyncfl/oort/syncfl):
- Replace per-end fixed-timeout polling with ONE event-driven set-drain
(recv_fifo(set, first_k=len, timeout=grace)) — a completion barrier that
releases as soon as the in-flight set is buffered, then commits by
sim_completion_ts. Since sim trainers don't sleep, this waits ~real
compute+mqtt latency (adaptive), never a fixed clock.
- Shared helpers in the syncfl base: _sim_recv_grace_s (dead-end ceiling only,
max(floor, factor x EMA of observed drain) — never paces) and _note_sim_fill.
- Remove the 0.5s empty-spin and unused *_FILL_TIMEOUT_S / *_DEADLINE_S consts.
- [SIM_BARRIER] per-commit log: barrier_wait_s should track wall_lag_s (~0.16s).
Correctness preserved (only wall-pacing changes): commit sequence is a pure
function of {sim_send_ts, sim_completion_ts, in-flight set}; the barrier makes
"is the min final?" exact (wait for the set) vs the old approximate 0.5s window,
so it is strictly more correct under MQTT jitter, not less.
Tests:
- test_async/sync_sim_ordering pass unchanged (commit-order invariant); async
fake recv_fifo updated to model the real set-wide multi-yield.
- new test_sim_barrier.py: asserts the single set-drain across all 3 stacks +
adaptive grace (regression guard against per-end polling returning).
- fix test_parity_checks.py for the overall_verdict 4-tuple (root/downstream).
All tests/mode + tests/launch green (87 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* plots: sim speedup-factor, barrier-wait, vclock-advance & staleness debug plots
Adds simulator-focused debug plots to analyze_run.py (real runs get explicit
NO-DATA placeholders), to validate the §6 speedup fix and debug clock-tier parity:
- sim_speedup_factor_over_time: vclock/wall (1.0 = real-equivalent; >1 faster).
On the pre-fix felix sim it correctly reads ~0.27 (3.6x slower than real);
a post-fix run should rise above 1.0.
- sim_barrier_wait_cdf / _over_rounds: parsed from the new [SIM_BARRIER] log
(now carries round=); should track wall_lag (~0.16s), not the old ~0.79s/commit.
- sim_vclock_advance_decomp_over_rounds: per-round Δvclock vs modeled compute
(max committed speed) — gap = per-commit overhead + overlap (debugs K3b).
- staleness_cdf / staleness_over_rounds (both modes): debugs U3 staleness parity.
Also: [SIM_BARRIER] log lines gain round= (defensive getattr so __new__-based
test fixtures don't break); _parse_sim_barrier added.
tests/mode + tests/launch green (87 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim speedup #2: remove distribute send-stagger; retune overheads; drop plot
The barrier fix landed (recv 0.019s/commit) but sim only sped up ~5x because the
bottleneck moved to _distribute_weights, which slept a fixed time.sleep between
every weight send (async 0.2s, sync 0.5s — not shortened for sim). On felix that
was 949s = 45% of sim wall. Same wall-pacing anti-pattern as the old recv poll;
it does not affect sim-time ordering (sim_send_ts is vclock-stamped, commits by
sim_completion_ts).
- New hyperparameter sim_send_stagger_s (alias simSendStaggerSeconds, default
0.0): sim removes the stagger; real keeps its hardcoded 0.5s. Read in the
syncfl base; used by both async and sync distribute.
- Overhead retune from Jun8 parity: felix 0.58 -> 0.50 (was slightly over-
charging); refl 0.115 -> 0.24 (still under-charging: advance sim 1.64 < real
2.91; residual +0.127/commit -> sim ran 1380 rounds vs real 824, driving the
REFL selection drift / trainer_speed / participation cascade).
- Drop guard: analyze_run mqtt_delivery_plots (dispatched vs received; growing
positive gap = broker drops at stagger=0). Both modes.
- [DISTRIBUTE_TIMING] log (send-loop wall excl stagger) to localize the next
bottleneck: MQTT publish of the 2 MB model re-serialized per send (~15/round).
tests/mode + tests/launch green (87 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim speedup #3: unify send-stagger (both modes, default 0), hoist model
conversion out of the send loop, gate settle sleep to real; trim comments
- send_stagger_s replaces sim_send_stagger_s: one knob, both modes, default 0.
The stagger is broker pacing only and is hidden behind trainer sleeps in real,
so it never needed the real 0.5s either; mqtt-drop plot guards both.
- Hoist weights_to_device(self.weights, CPU) out of the per-send loop (async +
sync): the global model was being re-converted ~15x/round; now once/round.
- Gate the 0.1s distribute settle sleep to real mode.
- Trim verbose comments across the touched aggregators/config.
tests/mode + tests/launch green (87 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* parity.md: wall-time accounting (~51% explained) + ranked next-opt tasks
Document that distribute stagger (45%) + settle + barrier are fixed/accounted,
~49% remains pending the [DISTRIBUTE_TIMING] split on re-run, and list ranked
candidate optimizations (serialize-once, avoid per-commit deepcopy, etc.) with
status so experiments can proceed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* parity.md: Jun8-1800 analysis - speedup landed, real-baseline shift, diskcache bottleneck
Records: sim_rate 2.5x/1.7x; removing real stagger moved the real reference so
the fitted overhead must be re-tuned last; next shared bottleneck is the
disk-backed update cache (2MB/commit disk IO) + per-round eval/checkpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim speedup #4: in-memory cache, serialize-once sends, eval/checkpoint=50,
off-path checkpoint write
Exhaust the shared, fidelity-neutral aggregator optimizations (help real+sim,
tuned for sim's exposed critical path):
- self.cache diskcache -> MemCache (in-memory dict w/ iterkeys/pop/reset):
removes per-commit 2MB disk IO; optimizers pop after consume so memory bounded.
- channel.dumps()/send_payload(): serialize the per-round model once and reuse
the payload across recipients (skips re-pickling 2MB per send; big for sync).
- eval/checkpoint/util_counterfactual default 10 -> 50 (config + OVERNIGHT yamls).
- checkpoint: snapshot to CPU on main thread, torch.save in a daemon thread.
Overhead re-tune deferred to last (real baseline still moving). tests green
(87 passed, 7 skipped); MemCache verified against optimizer iterkeys/pop API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* tests: baseline readiness guards + PARITY.md comparator command & status
test_baseline_readiness.py (fast, no cluster): every baseline's selector+optimizer
is registered, MemCache is a diskcache drop-in + a real fedavg aggregation runs/
frees on it, and the model message survives serialize-once. Catches wiring and
sim-speedup-change regressions in CI.
PARITY.md: prominent real/sim comparator command (single + --batch), readiness
test list, and a crisp Jun8-EOD status + deferred fine-tune.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* debug_run: smoke honors --baselines (default all 4) so oort/feddance can be smoke-validated
Smoke mode previously ignored --baselines and only ran felix+refl. Now it parses
--baselines and defaults to "felix oort refl feddance".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* oort/refl distribute: serialize-once + hoist (was missing — oort has its own
_distribute_weights)
The smoke runs showed oort with DISTRIBUTE_TIMING=0: oort/top_aggregator defines
its own _distribute_weights (used by BOTH oort and refl), which still re-converted
weights_to_device + re-pickled the 2 MB model per send. Brought it in line with
the syncfl/asyncfl paths: build+dumps once, send_payload per end, + DISTRIBUTE_TIMING.
MemCache and the recv barrier already applied to these stacks; this closes the
last per-send serialization gap so all 6 baselines share the optimization.
tests green (96 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* instrument [AGG_COMMIT_TIMING]: per-commit cache-store vs optimizer cost
Direct measurement of the ~230ms/commit aggregate cost, split into cache_store_s
(was disk IO, now in-memory MemCache) and optimizer_s (deserialize+aggregate
floor), in all three aggregate paths (async per-commit; sync/oort per-round).
analyze_run parses it into system/agg_commit_timing_cdf.pdf. Lets the overnight
run confirm the MemCache win and reveal the next optimization target.
tests green (96 passed, 7 skipped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* parity_batch.py: per-baseline real/sim comparator with --since floor + stamped JSONs
Discovers the latest real+sim pair per baseline at/after --since (excludes smoke),
runs the parity comparator, prints the report, writes parity_<baseline>_<now>.json.
python scripts/parity_batch.py --since 20260608_230000 \
--baselines felix oort refl feddance --agg-goal 10 --budget-s 12600
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* parity.md: trim to current state (629->272) + overnight rounds/accuracy parity + next tasks
Removed done history (impl plan, Jun7 issues, speedup blow-by-blow). Kept the
ladder methodology + check reference. Refreshed status with the overnight
per-baseline rounds/accuracy parity table and the key read (accuracy good
everywhere; rounds off only for the fitted-overhead baselines felix/refl, which
now over-charge -> re-tune). Next tasks: overhead re-tune, lazy weight-deserialize
on the sync recv barrier, DIST-tier parity.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim fidelity: fix overhead-driven buffer backup (felix staleness+rounds) + eval off-path
Root cause (felix overnight trace): _advance_sim_clock adds sim_commit_overhead_s
per commit; in async the sct frontier advances only ~0.31s/commit (overlap), so
overhead 0.50 > 0.31 backed up the reorder buffer -> updates commit long after
their completion ts (T_v-sct median 118s, 100% late) -> staleness 9.3 vs real 2.8
AND vclock over-charge -> 1557 rounds vs real 2595. Unifies the rounds + staleness
gaps; oort/feddance (overhead 0) were unaffected.
- Re-tune overhead below the sct rate: felix 0.50->0.16, refl 0.24->0.10.
Smoke confirms commit_gap_s collapses 548s -> ~0.24s (buffer no longer backs up).
- Staleness telemetry: async agg_round now emits commit_gap_s (vclock-sct, >0 =
backup), buf_depth, residence_rounds, inflight.
- Eval off the critical path: evaluate() snapshots weights + runs the test-set
forward pass in a daemon thread (base _eval_snapshot_model/_eval_emit + 3 mains),
so async isn't penalised by per-eval pauses. Re-entrancy-guarded; telemetry.emit
is lock-safe.
Tests: 300 passed, 7 skipped. Smoke felix real+sim clean. Eval thread validated
on CUDA (snapshot, threaded forward pass, agg_eval emit, guard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* sim fidelity: retune felix overhead 0.16->0.315, update PARITY.md §3b+§4
overhead_residual from 201709 run: residual=1.55s (0.155/commit missing).
Fix: 0.16 + 1.55/K=10 = 0.315. Expected: advance 2.98->4.52s, rounds 4230->~2788
(match real 2719), staleness 12.97->~2.7 (match real 2.79), sim_rate ~3.4x.
Also: document §3b (under-charge root cause + queue_wait_s artifact explanation),
update §4 next tasks with overnight validation checklist, add cross-baseline
scorecard table, fix report.py UnicodeEncodeError on em-dash output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* sim: lazy barrier deserialize, feddance trainer_speed, log level cleanup
Lazy sync barrier deserialize (WEIGHTS_BYTES=41):
- syncfl/trainer.py (sim mode): pre-serialize weights to cloudpickle bytes in
WEIGHTS_BYTES; outer message carries bytes not live tensor so all-N barrier
drain is cheap (byte-copy vs tensor-reconstruct). N-K non-committed updates
never pay full cloudpickle.loads cost.
- syncfl/top_aggregator.py: pop K minimums -> loads(WEIGHTS_BYTES) only for
committed K. Expected: refl barrier 688ms -> ~200ms; speedup ~1.18x -> ~1.8x.
Feddance trainer_speed_s fix:
- syncfl/top_aggregator.py: set PROP_ROUND_DURATION from wall_lag_s in real mode
when not already set by oort overlay. Fixes feddance real trainer_speed=0.0.
Log level cleanup (INFO -> DEBUG):
- channel.py recv_fifo: all trace logs (first_k, payload routing, msg type)
- asyncfl: [AGG_START], [AGG_RECV_WEIGHTS], [AGG_RECV_EVAL], "proceeding to agg
weights", agg_goal reached, aggregation finished, _agg_training_stats/5,
per-trainer staleness/100, Avg training time, [SIM_PENDING], sending weights
- syncfl: Agg weights entry, Waiting for first_k, received data from end,
_agg_training_stats/5, sending weights to end
Kept at INFO: [AGG_ROUND], [LAG_DECOMP], [SEND_RECV_LAG], [SIM_BARRIER],
[SYNC_SIM_RECV], [AGG_COMMIT_TIMING], [DISTRIBUTE_TIMING], [TRAIN_CYCLE]
PARITY.md: mark tasks 3/4/6 DONE, add validation TODOs for overnight run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* analysis: single-pass agg-log parser, binned_line, plotting overhaul
Phase 1: parse_agg_log() reads the multi-GB aggregator log ONCE (was 6 full
scans, one per [TAG] parser); old parsers are thin accessors. New binned_line
helper applied to dense/noisy/slow plots; CDF companions.
Phase 2: fairness=train+eval; trainer_state_fraction_*->trainer_time_allocation_*
(idle split train/eval); availability_composition->lines; eval/explore rate
lines; weight-change accuracy overlay+conv marker; mqtt drops callout;
believed-actual signed gap hist; util_disparity vs mis-selection; cumulative comm.
Phase 3: new deep-dive subdirs plots/{availability,aggregation,selection/why}/,
gated to skip cleanly (async-only plots gate out for sync baselines).
Plan + conceptual answers in async_cifar10/PLOTTING.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* launch: deadlock watchdog on aggregator wait + pre-batch straggler sweep
Two fixes for batch runs hanging / starting dirty:
1. Watchdog on the aggregator wait (runner.run_experiment). The bare
aggregator_spawner.process.wait() had no timeout, so a deadlocked
aggregator (MQTT/barrier) blocked the whole batch forever instead of
self-stopping at max_runtime_s/sim_wall_ceiling_s. Now bound the wait at
the run's own budget + 1200s grace and hard-kill on timeout, so the batch
proceeds to cleanup/sweep. New AggregatorSpawner.wait(timeout) returns
False on timeout.
2. Pre-batch straggler sweep (runner.run_experiment_batch). _sweep_stragglers
previously ran only BETWEEN experiments, so orphans from a prior crashed
invocation poisoned the first run. Sweep once up front too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): unify lazy weight-deserialize across all aggregator paths
The lazy-deserialize change (7ec1a308) made sim trainers ship WEIGHTS_BYTES
instead of WEIGHTS, but only syncfl's sync barrier reconstructed it. The oort
and asyncfl recv/handle paths still read MessageType.WEIGHTS, so:
- refl (oort sync): UnboundLocalError on the first round-1 update -> died.
- felix (async): train/eval router keyed off `WEIGHTS in msg` -> a
bytes-carrying train update misrouted to eval; updates never counted ->
0 aggregations, no accuracy, runaway spin telemetry.
Real mode was unaffected (bytes swap was `if simulated`).
Option B (unify): trainers always send WEIGHTS_BYTES (both real+sim); every
up-path aggregator read goes through common.util.materialize_weights(msg)
(bytes->WEIGHTS in place, idempotent, backward compatible, None for eval-only)
+ a defensive `weights = None` default; the async classifier accepts WEIGHTS or
WEIGHTS_BYTES. Guarded by tests/mode/test_weights_bytes_roundtrip.py.
35 tests pass (4 new regression + 31 readiness incl. sim-barrier/ordering).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(debug_run): clear stale per-node smoke/run config before regenerating
make_debug_yaml exits WITHOUT writing the cfg when the requested baselines
don't match a node. A leftover cfg from a previous invocation was then re-run,
so e.g. `--baselines refl` re-ran a stale node1 felix config (and could time
out before ever reaching the refl node). rm -f the cfg first so a no-match
node is skipped. Applies to both the smoke loop and the normal run path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* parity(jun10): root-cause felix staleness (buffer residence §3c) + refl overhead 0.074
Jun-10 post-fix 45-min parity run (150131):
- felix: sim_rate 2.28->3.59x; rounds (+0.5%), overhead, convergence (0.073->0.024)
now PASS. Remaining gap is staleness 7.19 vs 2.79 -> root-caused in §3c to
reorder-buffer RESIDENCE (not the clock): commit_gap_s mean 25s ~= 5 rounds,
buf_depth ~28, residence ~10 rounds; concurrency=30 identical in sim+real. Sim
drains all C in-flight into SimReorderBuffer for strict min-SCT order and pops one
per commit; real commits on arrival. Fix direction: bound buffer to agg_goal K /
commit K-min as a batch, or relax to arrival-order window. New plots/aggregation/*
are the instruments.
- refl: sim_rate 1.18->2.31x (lazy-deserialize speedup). The faster barrier made
overhead=0.10 over-charge (advance 1.74 vs 1.48, residual -0.26) -> retuned to
0.074 in baselines.yaml. staleness/avail_timebase/convergence pass; eligibility KS
0.48->0.27.
Rewrote PARITY.md status/scorecard/§3c/§4 to match. No code behavior change here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): ordering gate so async vclock waits for in-flight completions
Felix sim staleness (7.19 vs real 2.79) root-caused NOT to buffer residence but
to late physical arrival vs a decoupled virtual clock: the sim trainer doesn't
sleep its budget, it stamps a future sim_completion_ts; under GPU contention a
message arrives after the fast vclock already advanced past its sct, so it
commits out of completion order and current_version - trained_version inflates
(high commit_gap, residence~0; corr(gap,residence)~=-0.06).
Real waits implicitly (the trainer actually takes its budget). The gate makes the
sim wait for the modeled completion: _sim_recv_min holds the commit of the
buffered earliest update while an un-arrived in-flight trainer is EXPECTED to
complete earlier (PROP_SIM_SEND_TS + PROP_ROUND_DURATION via
_min_outstanding_expected_sct / _safe_to_commit, slack 2.0s), bounded by a
RECV_TIMEOUT_WAIT_S failsafe. Keeps commits in true completion order so
commit_gap -> ~0 and staleness -> budget/advance, with NO version re-labeling
(an earlier trained_version band-aid was reverted).
Smoke (felix sim, 4 rounds): commit_gap_s mean 0.7, 0 errors. 40 tests pass
(5 new gate tests + 35 prior). PARITY.md §3c/§4 corrected. Validate staleness
KS + throughput on the next 45-min run (expect a slower sim_rate from waiting).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* debug_run: add --mode sim|real|both to select time_mode variant per baseline
`--baselines` filters by baseline name only, so a node's config ran BOTH the sim
and real variant of each baseline sequentially. --mode (default both) filters by
each experiment's time_mode (field, or _sim/_real name suffix), so you can run
e.g. felix-sim on one node and felix-real on another to parallelize a run.
Validated by dry-run config generation for node1/felix at sim|real|both.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* diag(sim): instrument ordering gate to find why it never waits
The 45-min felix run showed the gate is a no-op (barrier_wait p50=0, commit_gap
unchanged at 27, staleness 7.66) — _min_outstanding_expected_sct is coming back
None. Add diagnostics to localize why:
- [SIM_GATE_DIAG] every 500 commits: frac_min_expected_None, frac_no_outstanding,
mean_n_outstanding, mean_n_with_sim_send_ts, mean_n_with_round_dur,
frac_committed_out_of_order.
- [SIM_GATE_LATE] per late commit (commit_gap>20s): n_outstanding,
n_with_sim_send_ts, n_with_round_dur, min_expected_sct, barrier_wait_s — so we
see whether an earlier-expected trainer was outstanding and the gate missed it,
or the gate was blind (no outstanding / missing PROP_SIM_SEND_TS /
PROP_ROUND_DURATION).
No behavior change; INFO-level, sampled (~13 DIAG + ~1.3k LATE lines / 45-min run).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* diag(sim): instrument the monotonic commit_gap/staleness DRIFT
Plots show commit_gap_s and staleness grow over rounds (5->10->20s) with flat
buf_depth ~28 — an accumulation, not a steady offset. Add instrumentation to
localize the driver:
- _advance_sim_clock (syncfl) now splits each advance into its sct-frontier vs
per-commit-overhead components: _sim_overhead_cum / _sim_sct_adv_cum.
- [SIM_GATE_DIAG] (every 500 commits) now also logs vclock, buf_min_sct,
vclock_lead_over_buf, overhead_cum, sct_adv_cum, dup_buffer_adds — so we can
see whether the growing lead is accumulated overhead (which the sct timeline
never receives) or the buffer's earliest sct falling progressively behind.
- dup_buffer_adds counts re-adding an end already buffered (a once-per-round
violation, re: observation #3 on long residence).
No behavior change; pure instrumentation for the next 45-min felix run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): take per-commit overhead off the virtual clock (felix staleness)
Diagnostics proved the root cause: _advance_sim_clock did
vclock = max(vclock, sct) + overhead, and at 0.315 the overhead took over the
clock entirely (end-of-run vclock 3189s = overhead_cum 3150 + sct_adv_cum 39;
98.8% overhead, sct_adv frozen after round 43). The clock decoupled from trainer
completions -> staleness inflated AND drifted over rounds (3.7->9.2). The earlier
ordering gate was a no-op (mean_n_outstanding=0 always) and is removed; the
version_at band-aid stays reverted.
Fix: felix simCommitOverheadSeconds = 0 so the clock tracks completions
(vclock = max(vclock, sct)). Smoke (felix sim): commit_gap_s = 0.000, 0 errors,
aggregates. Replaced gate diagnostics with [SIM_CLOCK_DIAG] (overhead_cum vs
sct_adv_cum, buf_past vs buf_future). refl left at 0.074 (sync barrier; its
staleness already matches). 35 readiness/ordering tests pass. PARITY.md updated.
Known: throughput (advance ~3.0 vs real 4.5) will now mismatch -- the overhead was
masking a real completion-spacing/dispatch gap (mqtt_fetch 60s vs 23s), to be
modeled in the trainer sct next (task 1b), not re-faked on the clock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(sim): pin the aggregator to reserved cores (it's the bottleneck)
GPU util is 0% the whole run -- the felix sim is CPU/message-processing bound on
the single aggregator process (300 GIL-contending chunk-reassembly threads + one
asyncio recv loop + serial commit), which was NOT CPU-pinned while 300 trainer
processes were pinned to every core, time-slicing it. That backs up the recv
queue (queue_wait p90 ~28s), which surfaces updates in the reorder buffer late
(sct << vclock stragglers) and decouples commit rate from virtual time.
Fix: runner reserves a few cores (min(8, max(2, ncores//8))) for the aggregator;
AggregatorSpawner pins to them (sched_setaffinity preexec + OMP/MKL threads =
ncores) and TrainerSpawner excludes them from the trainer pool. Smoke: machine
96 cores -> 8 reserved for aggregator, 88 for trainers; pinning engages, 0 errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* perf(sim): eager rxq drain in _sim_recv_min (bounded re-sweep)
Drain every currently-ready update into the reorder buffer, then re-probe and
sweep up any that reassembled during the pass (up to _SIM_DRAIN_MAX_PASSES, tiny
_SIM_DRAIN_SWEEP_S timeout on sweep passes), so the buffer holds the true global
min-sct before committing -- a ready-but-undrained update can't surface late and
out of completion order. Bounded so it never spins. Complements the aggregator
pinning; the combined effect on queue_wait/commit_gap/staleness is measured on
the next 45-min run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): virtual-completion gate so the clock can't race past stuck updates
Pinning+eager-drain (prev commits) halved the recv backlog (queue_wait 11->5.7s)
but staleness still drifted: the residual backlog still let the virtual clock
advance past updates that had virtually completed but were stuck in the rxq /
chunk-reassembly layer (sct << vclock stragglers; confirmed NOT a stale send_ts
-- their sct matches a genuine many-round-ago dispatch).
Gate (the correctness fix): the aggregator now tracks each in-flight trainer's
EXPECTED completion in its own bookkeeping (_sim_inflight_expected = dispatch
vclock + last-observed budget, running-mean default for unseen trainers), set at
dispatch and cleared at commit. _sim_recv_min holds the commit while an
un-drained in-flight trainer is expected to complete earlier than the buffered
minimum (slack 2s) -- draining/waiting up to a RECV_TIMEOUT_WAIT_S failsafe
(which pops the stuck end so it can't block future commits). This was the piece
the earlier gate got wrong: it read recv_ends/channel props, which don't contain
a trainer whose reply is still reassembling -> it was blind (n_outstanding=0).
Effect: the clock no longer advances past a stuck-but-expected-earlier update, so
commit_gap -> ~0 (incl the tail), staleness -> budget/advance, no drift. Tradeoff:
the sim WAITS (real time) for stuck messages, so it runs slower -- cheaper now
that the backlog is halved. [SIM_CLOCK_DIAG] gains barrier_wait_s /
inflight_tracked / gate_failsafe to watch it. Smoke: rc=0 (no deadlock), 0 errors,
commit_gap=0. 35 readiness/ordering tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(parity): felix staleness — §3c overhead fix held but drift persists (§3d)
Run 012226 (felix sim, overhead=0 + virtual-completion gate) verdict:
- §3c overhead fix held (overhead_cum=0, sct_adv_cum=vclock) but staleness
still drifts monotonically (decile mean 3.65 -> 12.26; overall 9.2 vs real
2.79). Overhead was a contributing inflator, not the whole mechanism.
- The re-added gate (993ff450) is inert (barrier_wait_s~0, gate_failsafe=0):
its per-trainer duration prediction never flags deep-past stragglers, so
buf_past updates keep committing (commit_gap_s spikes to 2747s).
- Refined root cause: out-of-order commit of a past-dated update bumps
version/round while vclock=max(vclock,sct) is a no-op -> version races ahead
of the clock -> staleness inflates and compounds. Fix must couple version
and clock (clock-on-commit, or true commit-at-sct reorder), not predict
durations. PAUSED by user; §4.1/1b updated with two honest candidates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(felix-stream): n=50 streaming-misprioritization experiment + per-baseline online oracle
Adds the Felix "data streaming makes client utility non-stationary" experiment on
the async_cifar10 example (n=50, alpha=0.1, syn_0, aggGoal=10, sim mode), plus the
machinery it needs:
- Target-accuracy stop rule: stop after N consecutive evals >= target (config
targetAccuracy/stableEvalsAboveTarget; _check_target_stop in syncfl aggregator,
inherited by the async stack).
- Staggered per-client streaming (trainer _stagger_params + generalized
_visible_sample_count), mirrored byte-for-byte in oracle_misselection.py.
- Per-baseline ONLINE oracle: aggregator-side true-utility injection
(aggregator/pytorch/oracle_utility.py + OracleInjectMixin), hooked before
selection in the sync/async/oort _distribute_weights. Each baseline B is paired
with B_oracle (same selector, fed true utilities each round) -> the B vs B_oracle
time-to-acc gap is that baseline's staleness tax. No selector changes. (A global
greedy OracleSelector is also added as an optional, unused ceiling.)
- n=50 Dirichlet split generator + cifar10_alpha0.1_n50.yaml; 4-node uniform
experiment YAMLs (one B + B_oracle pair per node) via gen_n50_experiment.py.
- Cross-arm figure driver (felix_streaming_figures.py) and a 4-node orchestration
script (run_felix_streaming.sh: smoke | node <i> | run | analyze).
- Docs: EXPERIMENT / IMPLEMENTATION / PLAN under async_cifar10/docs (the
implementation doc opens with a RESUME-HERE runbook).
Statically validated (compiles, YAMLs parse, stagger-formula parity). NOT yet run
end-to-end: pilot the online injector via `run_felix_streaming.sh smoke` before the
production sweep; the offline counterfactual is the validated fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix staleness — gate predictor learns modeled budget, not contention-inflated dur
The virtual-completion gate (§3d) was inert because its per-trainer
expected-completion predictor learned `budget` from SIM_ROUND_DURATION
(= max(gpu, modeled_budget)), inflated by GPU contention. Under
contention a trainer's observed budget spikes, pushing its next expected
completion into the future, so the gate's `min_stuck < buffered_min` test
never trips → the clock laps in-flight stragglers → their updates commit
past-dated (sct << vclock) → _round increments while _advance_sim_clock
is a no-op → version drifts ahead of the clock → staleness inflates and
compounds (the §3d drift, decile mean 3.65→12.26).
Fix: learn the predictor from TRAINING_BUDGET_S (the stable, contention-
free modeled training_delay_s, already on the wire), which is a true
lower bound on real sct (= send_ts + max(gpu, budget) ≥ send_ts +
budget). This makes the existing gate machinery fire on genuine
stragglers instead of being silenced by a GPU spike — no new ordering
logic, just the correct input.
Instrumentation to confirm/refute from the next run's [SIM_CLOCK_DIAG]:
gate_holds (should now fire, was ~0), pastdated_commits / pastdated_gap_cum
/ pastdated_gap_max (should collapse toward 0), budget_mean (sanity:
≈ real mean training_budget_s, not the inflated SIM_ROUND_DURATION mean).
35 readiness/ordering tests pass. See PARITY.md §3e.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix staleness — measure in the completion frame (version_at(sct))
Run 221402 showed the §3e predictor fix made the gate fire (gate_holds
0->391k, budget_mean=11.8) but did NOT fix the drift (staleness decile
3.53->11.74, mean 8.75 vs real 2.79). New diagnostics relocated the root
cause:
- Completions are CLEAN: 0% overran, real_gpu_time~0.1s, sim_round_duration
== training_budget exactly. So sct = send_ts + modeled_budget with no
contention inflation — the "late physical arrival under GPU contention"
story doesn't apply to these runs.
- commit_gap (vclock - committed sct) grows 30->230s over the run, 59% of
18239 commits past-dated, while buf_depth ~= concurrency (30) stays flat.
Round-36 smoking gun: T_v=139 while updates with sct=3..36 commit in one
sweep (probed=12) — completed updates enter the reorder buffer long after
the clock passed their sct, so they commit massively past-dated.
So the inflation is a receive/reorder-pipeline artifact: staleness =
self._round - trained_version counts versions up to the *physical commit*,
which the pipeline delays well past the update's true completion, and it
compounds. (staleness_factor=0.0 in optimizer.do -> staleness is a
measurement, not a model input; trajectory unaffected.)
Principled fix: an update's staleness is the number of versions published
during its actual training interval [send_ts, sct] — charge it against the
version current when it truly completed (vclock=sct), not the
pipeline-inflated version at physical commit:
staleness = version_at(sct) - trained_version.
Identical to raw in real mode (in-order commits) and when commit_gap==0;
strictly removes the reorder inflation otherwise. Not the §3c reverted
band-aid (which relabeled trained_version, a fact) — here trained_version
is untouched and the incorporation reference moves to completion time.
Impl: monotone _version_vclock_log of (vclock, version) appended on each
version bump (_record_version_vclock); _version_at(t) bisects it. Raw value
kept as agg_round.staleness_raw so the next run quantifies the artifact vs
the genuine budget/advance baseline.
Does NOT fix the advance gap (2.64 vs real 4.52) — that residual baseline
(budget/advance ~= 4.4 vs real 2.65, the 1.7x advance ratio) is §4.1b, the
next target once §3f confirms the drift is flat.
Guarded by TestCompletionFrameStaleness (3 cases). 73 mode tests pass.
See PARITY.md §3f.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Revert "fix(sim): felix staleness — measure in the completion frame (version_at(sct))"
This reverts commit 61edc6de8d680ddd1a46cb95c2b72dc2e07d1936.
* docs(parity): §3f — staleness feeds fedbuff alpha; relabel reverted; root cause is commit order (run 150131 proves it's not advance/selection)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix staleness — gate probes the LIVE in-flight set, not the stale recv snapshot
Root cause of the past-dated commits that drift felix staleness (§3g).
The gate in _sim_recv_min holds the clock for the earliest-expected
in-flight straggler (`min_stuck`, computed from the live
_sim_inflight_expected), but `to_probe` was built only from `recv_ends` —
a snapshot taken once at the top of _aggregate_weights. So the straggler
the gate was holding for was routinely absent from to_probe, recv_fifo
was never called on it (its timeout is a real block, but only for probed
ends), barrier_wait stayed ~0, the gate spun to the pass cap, and the
clock committed past the straggler. That past-dated commit bumps the
version without the clock, so version - tres.version inflates — and that
value is what fedbuff (agg_rate type="new", scale=0.4) uses for the
update weight alpha AND what we report as staleness. One value, wrong.
Fix: build the probe set from the live in-flight set — recv_ends plus any
in-flight end the channel has() whose modeled completion <= buffered-min
+ slack. Bounding by the buffered minimum blocks only for trainers that
should complete before what we're about to commit, never for
legitimately-future ones. recv_fifo's timeout now actually waits for the
trainer the gate is holding for, so updates commit in completion order,
self._round advances in lockstep with the clock, and the raw staleness is
naturally correct (no relabel). Messages arrive in ~0.1s physically so
the wait is short; never-arrivers hit the existing RECV_TIMEOUT failsafe.
This supersedes the reverted §3f telemetry relabel: staleness feeds the
model via fedbuff alpha, so the fix must correct the actual commit order,
not the reported number.
Guarded by TestGateProbesLiveInflight (2 cases). 72 mode tests pass.
See PARITY.md §3g.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix advance — model real post-compute cycle leg in trainer sct (§3i)
Sim per-trainer cycle was ~2s shorter than real (sim 11.3s vs real 13.3s;
compute 11.7s matches both), so by Little's law (advance = cycle*aggGoal/inflight)
the virtual clock under-charged advance 3.68 vs real 4.32 -> +14% rounds/terminal/
commits. The gap is real's post-compute leg (buffer-residence queue_wait ~0.6s +
re-dispatch latency ~1.0s) that the sim sct = send_ts + compute omitted.
The §3h mqtt_fetch->sct idea was checked against data and rejected: real
sim_round_duration_s is compute-only and staleness 2.81 = budget/advance, so
mqtt_fetch is NOT version-relevant; adding it would have inflated staleness ~6x.
- New trainer hyperparameter simCompletionLegSeconds (alias -> sim_completion_leg_s,
default 0). Added ONLY to _sim_completion_ts, not _sim_round_duration/budget/speed,
so trainer_speed_s/OORT/gate-predictor/P3/T2 stay pure compute; staleness baseline
(= cycle/advance) is invariant. Felix = 1.6s (= measured real W_cycle - compute).
- asyncfl [TIMING_OVERRUN_AGG] now reads SIM_ROUND_DURATION (pure compute) instead of
sct - send (which now carries the leg) so it still flags only GPU overrun.
- A2 eligibility_parity: point-mass guard — rescue a KS fail only when means match
within 2% AND a side is degenerate (CV < 0.01), e.g. real num_eligible const 300 vs
sim 298.9. A real eligible-set divergence still fails.
- Tests: A2 point-mass pass + real-divergence-still-fails, leg alias mapping, felix
carries leg>0 while overhead=0. 58 mode/ladder/launch tests pass.
- PARITY.md: §3i (corrected root cause + fix), §3h superseded banner, §4 updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix staleness — drain in-flight updates by rxq readiness, not predicted completion (§3j)
The past-dated staleness tail (sim 4.09 vs real 2.81) is driven by ~8% of
commits — the SLOW trainers (budget 38-56s). Tracing them: their commits have
residence~0 but commit_gap~45 (corr -0.27), i.e. they do NOT sit in the buffer;
they are drained in LATE, after the virtual clock already passed their sct, then
committed past-dated with staleness ~33 vs real ~9 for the same budget.
Root cause: §3g's drain admitted in-flight ends only within `exp <= buffered_min
+ slack`. A slow trainer's modeled completion (send_ts + budget) is far in the
future, so it was excluded from draining even though its message had physically
arrived (wall_lag ~0.1s). It sat undrained in the rxq while the clock advanced,
then was drained late and committed past-dated.
Fix: drain any in-flight end whose rx queue is non-empty (physically arrived),
regardless of its modeled completion — new `_sim_end_has_ready_msg` admits it
alongside the existing ceiling. Slow trainers then buffer as proper futures and
commit in sct order, giving staleness ~= budget/advance ~= real. recv_fifo on a
ready end returns immediately, so this adds no blocking.
Ruled out first (do not re-chase): GPU contention (T3 overrun=0), re-dispatch
invariant violation (all_selected excludes in-flight; SEND_TIMEOUT fires 0x),
and commit_gap as a staleness proxy.
Guarded by test_drains_ready_inflight_above_ceiling (fails without the fix).
90 mode/launch tests pass; smoke run clean. Awaiting the 3000s parity run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix staleness — split the leg into pre-commit holding + post-commit re-dispatch gap (§3k)
§3j fixed the staleness FAIL (sim 161419: 4.09→3.451 PASS, tail collapsed) but
the mean still sat 0.6 high (3.45 vs real 2.81) and the p99 tail wider (19 vs 9).
Root cause (from the real LAG_DECOMP, not tuned): staleness = holding/advance
(real 11.79/4.32 = 2.73 ≈ 2.81 ✓). The §3i 1.6s completion leg put the WHOLE
real post-compute cycle BEFORE the commit (in sct), but real splits into ~0.6s
pre-commit holding (queue_wait+delivery — counts toward staleness) + ~1.0s
post-commit re-dispatch (re-selection+model push — does NOT count). Folding it
all pre-commit inflated holding 1.6→ over-charged staleness.
Fix: split the leg, cycle-preserving so advance/throughput stay matched.
- simCompletionLegSeconds 1.6→0.6 (pre-commit holding, in sct).
- new simRedispatchGapSeconds 1.0 (aggregator): a just-committed end records
_sim_cooldown_until[end]=sct+gap and is held out of selection (via
set_curr_unavailable_trainers) until vclock≥sct+gap, returning with a fresher
model_version. Selector refills to keep c=30 in-flight → the gap spaces
completions (raises advance, de-bunches overlap) without counting toward the
committed update's staleness. Predicted staleness → ~2.84 vs real 2.81.
Not a relabel (§3f dead-end): the update genuinely commits earlier (smaller sct),
so the real number fedbuff consumes for alpha drops.
Also folds in this session's checker category-error fixes (§5): P1
aggregation_sequence + F1-3 utility gated/pooled for stochastic selectors,
phase_mqtt_fetch → DIAG (wall-time I/O the sim doesn't model).
Guards: TestRedispatchGap (3 tests) + existing 83 mode/ladder tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sim): felix advance — cooling trainers hold a concurrency slot, not just availability (§3L)
§3k (run 175309) tightened staleness (3.45→3.16) but FAILED per_round_advance
(KS 0.172→0.208, mean 4.24→4.03 vs real 4.32). Root cause: the redispatch gap
was inert on throughput. It only marked a just-committed trainer *unavailable*;
with a 300-trainer pool the selector refilled the freed slot from idle trainers
(inflight_tracked≈30), so the gap never extended the effective cycle. Cycle
collapsed to holding → advance low, staleness ≈ F/g.
Real keeps a fixed 30-slot concurrency that includes ~3 trainers in ~1s
re-dispatch limbo → only ~27 computing → that is what yields advance 4.32 AND
staleness 2.81 together.
§3L: a cooling trainer now holds its concurrency slot. async_oort
_handle_send_state computes extra = max(0, c - in_flight - cooling_count), fed by
channel.properties["sim_cooling_count"] stamped in asyncfl _distribute_weights.
Predicted: computing≈27.7 → advance≈4.4, staleness≈3.0·(12.3/13.3)=2.77.
Guarded by TestCoolingHoldsConcurrency (verified failing without the fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* analysis(mqtt): plot per-round MQTT drops (flat 0 = healthy), fix broken sends source
The mqtt_delivery_plots sanity check was silently dead: the per-send dispatch line
moved to DEBUG, so _SEND_RE matched nothing → sends_by_round empty → early return.
And the metric (cumulative dispatched − received) baselined at the in-flight count
(~30), so true drops were invisible against that floor.
Rewrite as a drops view that sits at 0:
- dispatched count now from the INFO [DISTRIBUTE_TIMING] n_sends roll-up (robust),
- plot per-round (dispatched − received) — differences away the in-flight offset,
so it hovers at 0; a broker drop shows as a sustained positive run,
- add a drops CDF (mass at 0 = none) and a SEND_TIMEOUT-based verdict callout.
Verified on real run 100106: median delta 0, 84% of rounds exactly 0, max +2,
SEND_TIMEOUTs=0 → no drops. Confirms the ~1s post-compute leg is serial-aggregator
queue_wait + re-selection, not MQTT loss (round trip <0.1s). Documented in PARITY §3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(real): remove the artificial settle-sleep brake so real holds ~c computing (§3m)
The ~1s real post-compute leg that drops effective concurrency below c=30 is mostly
an artificial brake, not a fundamental property: a real-only time.sleep(0.1) before
selection in _distribute_weights fires twice per commit (put_train + put_eval) →
~0.2s of the ~0.43s/commit budget (~46%). It inflates queue_wait (updates pile up in
the serial recv queue while the loop sleeps) and the commit→re-dispatch gap.
New knob realDistributeSettleSeconds (config; default 0.1 preserves legacy behavior),
set to 0 in the felix baseline. Expected real shift: advance 4.32→~3.97, computing
~27.6→~30, staleness 2.81→~3.0 (the post-commit gap that kept L/C<1 disappears).
Sim then matches at c=30 with simRedispatchGapSeconds=0, making the §3L slot-hold
unnecessary (kept, inert at gap 0).
Aggregator CPU pinning: already confined to a reserved core set by runner.py; NOT
changed. Documented why 1-core pinning would hurt (MQTT thread shares CPU with the
FL loop) and that the loop is latency-bound, so pinning is second-order. See PARITY §3m.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* config(felix): set simRedispatchGapSeconds 1.0->0 to run sim at c=30 (§3m)
With the real settle-sleep removed (realDistributeSettleSeconds=0), real is now
compute-bound at ~c=30. Match it in sim: gap=0 lets the idle pool refill a freed
slot, so the §3L slot-hold goes inert and sim runs at c=30 too. At gap=0 the
per-trainer cycle = holding, so sim staleness -> c/g = 3.0 (matches real's new ~3.0
regardless of the leg). simCompletionLegSeconds stays 0.6 as a placeholder — at
gap=0 it only sets advance; retune against the new real [LAG_DECOMP] if advance/KS
misses (expected ~0.1-0.3 once queue_wait drops).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(sync): stamp trainer-trained version on TrainResult so feddance/fedavg staleness is real
The base syncfl aggregator built TrainResult(weights, count) without a version, so
TrainResult.version defaulted to 0 and update_staleness_val = self._round - 0 =
self._round. Feddance/fedavg therefore reported the *round number* as "staleness"
(mean ~= run_length/2, the nonsensical ~183), not real staleness. The sync trainer
already sends MODEL_VERSION = its dispatch round (syncfl/trainer.py:403), so just pass
it: staleness = self._round - trained_version (0 for a strict sync barrier, >0 only
for genuinely late/straggler updates).
Only affects feddance/fedavg (base flame.mode.horizontal.top_aggregator); oort/refl
use the oort sync aggregator, which already stamped version (staleness was sensible).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: tighten code comments and restructure PARITY.md (Done / To-evaluate / pending-overnight)
Condense the verbose multi-line comments added this cycle (sim/real knob docs, §3L
slot-hold, MQTT drop plot, feddance staleness fix) to crisp one/two-liners that point
to PARITY §3 for the detail. Restructure PARITY.md's status section into Done /
To-evaluate after the overnight runs, with a banner that it gets rewritten once the
4h felix/oort/refl/feddance real+sim logs land. No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: remove stale parity diagnostic artifacts
Old per-run parity JSONs (08jun, the superseded felix 175309) and the 0607 diagnosis
note — superseded by the upcoming 4h overnight runs and PARITY.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(parity): sound real-validator via task_send + felix redispatch gap
Real-side instrumentation + a sound §4.0 real-correctness validator, plus
the staged felix sim fix, ahead of the next real+sim run set.
- task_send trainer event ([wall_recv_ts, wall_send_ts]) stamped post-sleep in
_send_weights, bracketing the trainer's true in-flight window. trainer_round
(emitted pre-sleep) and the aggregator's in_flight (== num_chosen) cannot.
- validate_real rewritten on that basis: concurrency = interval overlap;
re-dispatch breach = overlapping intervals for one trainer. Removes the old
contributing_trainers bookkeeping that miscounted overcommit-discarded
completions (the bogus oort 4549 / refl 26988). Confirmed: 0 real breaches.
- felix simRedispatchGapSeconds 0 -> 1.0 (§4.1): restores the post-commit
slot-hold so sim stops over-overlapping (advance 3.68 -> ~4.1, staleness
4.75 -> ~2.8 hypothesized).
- retire inert sendStaggerSeconds knob (off in every baseline; MQTT drops
ruled out) + its dead branches in both top_aggregators.
- PARITY.md: verification tracker (current metric -> post-fix hypothesis) for
every failing row, so the next run flags hits vs regressions.
- 8 new tests (test_validate_real.py); tests/mode (40) + launch + parity green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): speedup/headroom readout + drop validator fallback
- validate_real: add speedup_ceiling section (intrinsic_speedup = virtual/wall,
compute_floor = GPU work / mean concurrency, pct_of_floor, ceiling_speedup).
oort sim shows 14.2x intrinsic and is overhead-bound, not GPU-bound — the
floor (needs task_send concurrency) lands next run. Informational; does not
gate admissibility.
- drop the graceful-fallback / task_send_present gating — assumes fresh runs
carry task_send (every run from this code onward does).
- PARITY §4.4: log-overhead finding (aggregator .log ~220k lines/run, 105k of
it [RECV_FIFO]; none read by the parity checker which uses JSONL only) + the
two-phase plan (correctness with logging as-is, then demote hot-path traces
for a clean before/after). §4.2 confirmed localizable from existing per_trainer
selector scores — no instrumentation gap.
- 3 new tests (gpu-work, speedup arithmetic, real-run unit speedup); 113 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(parity): add 'verify this run' block for task_send/sendStagger/speedup/felix-gap
Captures the non-parity-table expected behaviours + regression signals to look
for in the next 45-min real+sim run set, alongside the per-row tracker.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(parity): throughput-family 5% bar, matched-round S2, A2b pool check, oort residence instrumentation
Checker corrections (principled, verified against the Jun13 2700s telemetry; not
relaxations — each removes a double-count or metric artifact):
- Throughput family K2/K8/U2 share one 5% bar. U2==K8==K2 on the same matched-V
quantity (rel diff identical to 3 decimals); U2's old 2% double-charged the
throughput delta K2/K8 already own. Reconciled + tightened 10%->5%.
- S2 participation: share-KS -> matched-round count-KS. Share (count/total) folds in
the throughput delta (modes run different #rounds in the window), re-measuring it as
a shape divergence. Counting over the matched round window (like K8/U2) isolates
shape: feddance .427->.033, oort .368->.127 (were artifacts); refl .530 stays
(genuine). feddance now fully green.
- A2b eligible_speed (Stage 2): A2 checks eligible-set size; A2b checks its speed
composition (KS over per_trainer.speed_s). Localizes refl's pool divergence
(sim 12.1s vs real 6.8s, .363) that A2 misses; participation now DOWN-suppressed
beneath it.
Sim dynamics:
- felix simRedispatchGapSeconds 1.0->0.6: the 2700s run over-advanced (sim 4.35 vs
real 3.93); slope ~0.67s/s -> 0.6 tracks real's measured leg. Needs a re-run.
- oort in-flight RESIDENCE instrumentation: new inflight_residence telemetry event
emitted per round from the oort sync aggregator (entry-round tracking ->
residence_rounds + carried_over_ages). Lets us compare sim vs real straggler
residence to calibrate the eviction-timing fix (sim drains to 13.4 vs real 15.6).
oort/refl share this "sim frees busy clients early" root; the dynamics fix is
deferred to the next run's residence data.
Also: validate_real.py made utf-8-robust (latin-1 stdout crash). +4 guard tests
(A2b matched/diverged/skip, residence builder). 100 parity/mode tests pass.
PARITY.md updated with the corrected localization (overturns the §4.2
"real picks slow / sim fast" story) and the Jun-14 session log.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): refl sct-gated pool exclusion + oort selector-score localization
Sim-only, real mode byte-identical (existing 4h real runs stay valid references).
refl (§4.5, IMPLEMENTED, config-gated simInflightResidence): in sim, hold a
dispatched trainer out of selection until vclock>=sct so the eligible pool stops
carrying the slow tail (A2b pool 12.41->~6.5). Via the trainer_unavail_list (NOT
selected_ends, which would re-dispatch). Files: config.py flag, SimReorderBuffer
.pending_after, oort/top_aggregator._distribute_weights, baselines.yaml (refl only).
oort (reclassified, NOT §4.5): pool already matches real (12.12/11.35); the
divergence is the selector picking ~average vs real fast -> selector scoring,
localized to system_util (Sx KS .132). Instrumented round_preferred_duration_s +
alpha on the selection event so a run pinpoints the system_util input (modeled
sct-leg vs wall round_duration; dynamic preferred target).
New standing parity localizers (work on existing telemetry):
- A2c selection_speed_bias_parity: selected-vs-pool speed; pool-match+bias-diverge
=> selector (oort), pool-diverge => composition (refl).
- Sx selector_score_parity: per-score-term KS (oort believed_I/temporal/system_util,
feddance V/I/A/U); pinpoints the diverging term.
Wired into report.py + the causal registry; guarded in test_ladder.py.
felix held (staleness gap-coupling, §4.6); feddance localized to feddance_I (§4.8).
PARITY.md trimmed 788->~540 lines and replanned per-baseline. Guards: 87 parity/
sim/mode/selector tests PASS under dg_flame.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): base-algorithm fidelity pass for oort/refl/felix selectors
Make the Oort-family selectors true to their base implementations'
hyperparameters, config-driven and per-baseline.
- scoring.OORT_PAPER_DEFAULTS: canonical standalone-Oort paper values; both
OortSelector and AsyncOortSelector read every knob via kwargs.get(.., paper).
oort/felix -> paper (round_threshold 10, clip_bound 0.98, cut_off_util 0.7,
exploration decay 0.95 / min 0.2); refl overrides to the REFL fork
(30/0.9/0.05/0.98/0.3) via selector.kwargs in OVERNIGHT_node2.yaml.
- D1 sort fix for round_preferred_duration (oort + async).
- D2 reward normalization+clipping (scoring.oort_norm_stats/oort_normalize_reward)
so the temporal/UCB term is meaningful (was inert under the raw ~70 reward).
- cutoff_util: threshold the exploitLen-th-highest score * cut_off_util (ref
oort.py:329); the factor was previously inert (indexed near the bottom).
- refl exploitation rewritten to the REFL fork's cut_off_util-augmented,
utility-weighted np.random.choice (was deterministic top-k).
- New parity check `preferred_duration` (Sd): per-round speed-penalty binding
frequency + reconstructed pref; guard for the D1 sort bug.
- refl simInflightResidence (PARITY 4.5) persisted in the sim refl block.
- D5 (temporal time-base last_selected vs last_updated) deferred; marked in code.
- Tests: TestAlgorithmHyperparams, TestRewardNormalization,
TestRoundPreferredDuration (+async), preferred_duration ladder case.
246 selector/mode/sim + 22 parity pass. UNRUN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): deterministic per-selector RNG + decision-fingerprint telemetry; refl P3/A2b checker fixes
Make selection a pure function of (state, seed) so real and sim are comparable
instead of two independent stochastic paths (the root of the residual
participation / believed_I divergence — the Jun-17 runs were unseeded).
Seeding (all selectors):
- AbstractSelector builds dedicated self._rng (numpy RandomState) + self._pyrng
(random.Random) seeded from config.hyperparameters.seed, threaded as the
reserved _seed kwarg by channel_manager. Every draw uses these, never the
process-global RNG, so selection is insulated from other np.random consumers
— a residual real/sim divergence under a shared seed is genuine input
divergence, not RNG desync. Replaced all global np.random/random calls in
oort/async_oort/refl_oort/fedbuff/random/async_random (incl. a missed
random.randrange); sorted() set->list draws for cross-process order. Import
RNG classes directly to dodge the flame/selector/random.py submodule shadowing.
- config.Hyperparameters.seed; debug_run.sh sets the SAME seed (default 1234,
SEED=<n>/none) on every experiment so real+sim of each baseline match.
Determinism telemetry + check:
- selection event carries seed + eligible_fingerprint + decision_fingerprint.
- Sdet decision_determinism parity check (Stage-3 DIAG) reports
eligible/decision/chosen_match_frac and a verdict splitting: seeding worked
(residual is non-stochastic) / RNG desync / input divergence.
refl checker corrections (Jun-17 runs, data-backed, guarded):
- P3 trainer_speed: compare at the modeled integer-second grid (sim exact-int vs
real int+wall jitter); raw KS .194 -> grid KS .021, with a mean_overhead guard.
- A2b eligible_speed: reconstruct pool speed from static training_delay_s
metadata (real leaves PROP_ROUND_DURATION None for ~158/300 non-completers);
pool is in fact identical (KS .000). _trainer_delay_map() loader.
Tests: 271 selector/mode/sim + 23 parity pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(parity): trim seeding comments; add base-class RNG contract tests
- Tighten the verbose inline comments added with the seeding change
(selector __init__, channel_manager, config.seed, refl_oort).
- Add TestDedicatedRngContract: same-seed→same sequence (numpy+python),
distinct seeds differ, None=unseeded+independent, construction doesn't
perturb the global RNG, seed recorded (parametrized None/0/1234).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(parity): run-length guidance (1h fast loop vs convergence sign-off)
1h reaches steady state for determinism/clock/staleness/P3/eligible_speed/
selection and — post-seeding — participation (no longer horizon-gated). Only
convergence C1/C2 weakens (one-sided: a 1h pass is inconclusive, a fail is real).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): low-confidence guard for convergence on sub-2h runs
C1/C2 convergence PASS is downgraded to LOWC ([??]) when budget_s < 2h — the
real/sim acc/loss gap grows with training, so a short-run pass is inconclusive.
A genuine FAIL still surfaces (a short-run divergence is real); >=2h pass stays
a confident [OK]. Supports the 1h fast-iteration loop without misreading a weak
convergence pass. Guard: test_convergence_low_confidence_on_short_runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(parity): believed-vs-actual utility telemetry+plots; §4.9 inflight carry-over; A2c/Sr checker fixes
Completes the Jun-17c fidelity-pass work for the next seeded oort run.
- §4.9 sim_inflight_carryover (config.py + oort/top_aggregator _oort_sim_recv):
hold prior-round overcommit stragglers (sct>vclock_round_start) in-flight
across rounds instead of draining at instant physical arrival, so sim
in_flight (~3 carry) matches real. Default off; ENABLED for oort sim.
- Sr inflight_residence_parity rung (checks.py/report.py) promotes residence
telemetry to a first-class root-cause check; SKIPs on async stack.
- A2c selection_speed_bias_parity now uses training_delay_s metadata pool
(mirrors A2b), inverting the observed-pool artifact.
- believed-vs-actual utility telemetry: EVENT_UTILITY_BELIEF/build_utility_belief
(events.py) emitted at the stat-utility return-overwrite in oort/asyncfl/syncfl
top_aggregators; 4 live believed-vs-actual plots in analyze_run.py (validated).
Tests: 146 pass / 7 skip (parity + mode + selector + sim).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(debug_run): node-agnostic runner + single parity config
Drop the node1/node2 split: debug_run.sh now filters a single merged config
(felix_oort_refl_feddance_alpha0.1_parity.yaml — every baseline × sim/real) by
--baselines/--mode/--runtime-s, so any subset runs on any machine. --node is
accepted but deprecated (warned + ignored). Merges the two per-node OVERNIGHT
templates into the parity config and removes the now-obsolete overnight_run.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(parity): restructure PARITY.md around status/done/hypothesis/next-steps
Collapse the stacked dated status/handoff blocks and the blow-by-blow
changelog/plan narration (840 -> 430 lines). Reorganize around Status ->
What's landed -> Next-run hypothesis -> Next-steps (branched on outcome) ->
Dead-ends; keep the ladder spec (§1/§2) verbatim and compress the root-cause
logs into a §3 mechanism reference + discrepancy ledger. Update launch docs to
the node-agnostic debug_run.sh + parity config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(parity): drop generated parity report JSONs; flag oracle/n50 as paused
- Remove experiments/parity_*.json (parity-checker run outputs, not source) and
gitignore them alongside the existing run_*/ rule.
- Add a STATUS banner to the felix-streaming experiment doc noting the oracle/n50
path was developed then paused, is shipped for continuity, and may need
updating/fixing — the fidelity work does not depend on it (guarded no-op).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(parity): trim verbose code comments — keep the why, cut the history
Aggressively simplify the parity/fidelity code comments ahead of the PR: drop
PARITY.md section cross-refs (§3x/§4.x), dated run references (Jun-1x, run IDs),
and retry narration from inline comments and docstrings across the core flame
library (config, telemetry, selectors, async/oort/sync aggregators, virtual
clock) and the parity scripts (checks/cli/validate_real, analyze_run). Mechanism
rationale is preserved; only the historical bookkeeping is removed. Comment-only;
161 pass / 7 skip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…gn-doc condense (#67) * fix(parity): one root-fix per baseline from the seeded 45-min runs Seeded run (seed=1234) localized one root per baseline; land the highest- leverage fix for each + document in PARITY.md. refl now PASSes and felix/ feddance P3 clears on the existing data; oort/felix sim+config changes are validated by the upcoming 3 h runs. - oort: carry-over re-buffer moved into try/finally. Held stragglers were re-buffered only after the yield-loop, but the caller abandons the recv generator once agg_goal updates are accepted (GeneratorExit at the yield), so held_over was dropped every round -> in_flight drained to 0.15 vs real 4.58. The finally re-buffers on early close. (§4.9 under-fire was a bug.) - felix: revert async knobs round_threshold 10->30, exploration_decay .95->.98 to the pre-paper parity values. The Oort-paper knobs shifted the selected mix and the async overlap model collapsed (overlap 10.9x vs real 6.6x, staleness 8.4 vs 2.8). No async Oort reference exists; anchor to real. - refl: gate Sd to WARN when the speed penalty is inactive in real (no binding, no reconstructable pref) -- the PROP_ROUND_DURATION None-density artifact (A2b/A2c class). oort (real_frac>0) stays fully enforced, D1 guard intact. - feddance/felix: widen P3 mean_overhead bar 0.5->1.5s. Observed trainer_speed carries a ~1s mode-dependent, opposite-sign capture overhead the virtual clock excludes by design (training_delay_s metadata matches exactly); grid_KS stays the backstop so oort's genuine 56s tail still FAILs. Tests: 146 passed / 7 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(parity): felix async-relaxed selector knobs (rt=70, decay=0.999) The sync Oort paper defaults don't apply to the async selector — there is no async Oort reference. Set felix to async-appropriate values and document the general async≠sync theme in PARITY.md §3.async. - round_threshold 70: the system-utility speed penalty exists to protect a SYNC barrier (round=max-of-K). Async/fedbuff has no barrier (stragglers commit stale later) so the penalty should be largely inert; 70 keeps a broad mix. - exploration_decay 0.999: the decay is applied per-round and async runs ~2-3x more rounds than sync, so a sync-tuned 0.95 floors exploration in ~29 rounds. 0.999 still reaches an exploitation phase across ~1150 rounds (0.9999 would be ~permanent exploration — rejected). - PARITY.md §3.async: documents the per-round-parameterization theme (round_ threshold, exploration_decay, temporal UCB log(round), pacer cadence, staleness, D5) so async knobs aren't re-derived; flags 70/0.999 as a NEW operating point validated by the first 3 h run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(parity): pass/total scoreboard, feddance green, sim-fix hypotheses Checker (no rerun, validated against stored run_20260616 dirs): - verdict_summary(): enforced pass/total tally -> JSON summary key, report footer (SCORE:), and a pass/tot column in the --batch roll-up. - P3 trainer_speed / T2 training_budget: enforce support containment (sim must not produce speeds beyond real's range) instead of pooled grid/mean KS. The feddance speed FAIL was selection-mix (A2b pool KS=0), not a model bug, and is now owned by A2c. Bonus: the support guard caught an oort sim slow tail the old grid-KS missed. feddance -> 43/43, fully green, no rerun. - K3 per_round_advance: enforce KS at the integer grid (raw .715 was pure sim-quantization-vs-wall-jitter; mean-diff guard still catches felix). Sim-mechanism hypotheses for the next batched rerun (config-gated, reversible): - refl: simCommitOverheadSeconds 0.011 (the K3b per-commit overhead -> K2/U2). - felix: simRedispatchGapSeconds 0.6 (space future-dated commits -> lower overlap). - oort: carry-over stays on; behavioral fix (block-for-K-fresh) deferred to avoid regressing the shared sync aggregator refl also uses; rerun gathers [SIM_BARRIER]. Guards: +3 tests (149 pass / 7 skip). PARITY.md + memory document the scoreboard, the resolved feddance decision, and per-baseline hypotheses with confidence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * parity status updates as on 16th afternoon * fix(parity): D5 — stamp last-selected at selection, not commit (oort+refl) PROP_LAST_SELECTED_ROUND feeds the selector's UCB temporal term. The bug was write TIMING, not value: the aggregator wrote it at commit, so a candidate's visible value rode commit ordering (sim sct-regular vs real FIFO-jittery), skewing which trainers win the weighted draw even with pool, pool-speed and chosen-count matched. The value (selection round == the MODEL_VERSION trained on) was always correct and matches both references, where engagement-round == selection-round in sync FL. Move the stamp to selection time in OortSelector (_record_last_selected_round, all 3 select paths); remove the commit-time write in the oort aggregator. Value unchanged, timing now aligned across modes. Scope: oort + refl (shared OortSelector). felix (AsyncOortSelector) has the same latent commit-write and is deferred. Naming (context-free, oort+refl): self.round -> self._last_selection_round; _stamp_last_selected_round -> _record_last_selected_round (a round int, not a timestamp); end_last_round -> end_last_eval_round. Add a naming-discipline instruction to PARITY.md; defer base self._round -> _agg_round (all-baseline). Guard: tests/selector/test_oort_selector.py::TestLastSelectedRoundStamp. 152 pass / 7 skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(refl): propagate D5 self.round→_last_selection_round rename to REFLOortSelector The D5 commit renamed self.round→self._last_selection_round in OortSelector but left REFLOortSelector with 4 dangling self.round references (select() guard, select() assignment, pacer condition, pacer close), causing AttributeError on the first select() call in the 1h rerun. Fix: rename all 4 sites in refl_oort.py. No init needed — parent sets self._last_selection_round=0 in OortSelector.__init__. Also: PARITY.md status updated with oort 1h rerun results (36/44): committed_fresh confirmed fixed (sim=10=real); residence/carry-over still open (inflight_after sim=0.82 vs real=4.24); refl crash documented. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(parity): oort carry-over creep + felix past-dating seed (sim-only) oort: pin the §4.9 carry-over threshold to the round-start vclock (captured once in _aggregate_weights), so the block-for-K-fresh retry loop can't creep it forward mid-round and prematurely commit cross-round stragglers. Mined the existing sim run: in_flight_after decayed 4.3->0 across the run; early rounds (few retries) matched real. Gated behind simInflightCarryover -> refl neutral. felix: the gate's unseen-trainer expected-completion default was the running mean (~12s), which overshoots fast trainers, lets the clock lap them, and seeds a past-dated backlog that makes the gate (min_stuck<buffered_min) inert (gate_holds=0, 74% commits past-dated). Default -> running minimum (a true lower bound) so the gate never laps an unseen trainer. Both changes are sim-only (reuse stored reals). Suite 155 pass / 7 skip with new guards: TestSimInflightCarryover::test_pinned_threshold_holds_straggler_across_retry, TestExpectedCompletionLowerBound. PARITY.md: refl D5 validated, oort/felix roots + fixes, and a new "minimize time & runs" workflow policy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(parity): felix pastdated_by_source instrumentation; oort 3h decision Felix: attribute each past-dated sim commit (sct<vclock) to its seed — fresh / redispatch / straggler / round1, count + cumulative gap each — surfaced in [SIM_CLOCK_DIAG]. Telemetry-only, sim-path only; reuses stored real. Confirms gate_holds=0 is the cross-round lap (clock advances onto a future-dated slow sct then laps a newly-dispatched fast trainer) and reveals which seed dominates before designing buffer-aging pacing. Oort: no code change — carry-over decay pinned to the system_util penalty asymmetry at oort.py:548-551 (None duration => no penalty); test via a 3h run (run-length hypothesis) before touching the refl-shared selector. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(parity): update_visibility_lag_s commit-timeliness instrument (all baselines) One metric, both modes, every baseline: committed_ts - ready_ts in the aggregator's own clock (sim: vclock-sct; real: wall commit - wall arrival). Async target ~0; sync = barrier wait, matching in both. Emitted in asyncfl/syncfl/oort aggregators; carried on TrainResult for oort/refl. New checker rung U6 commit_visibility (Stage 6, MECHANISM/DIST, KS+mean-gap), declared upstream of staleness; self-SKIPs when the field is absent so it's inert on pre-instrument runs. Not added to field_coverage (INV) to avoid wrongly root-causing every old run. Plot (CDF + P50-over-rounds) in analyze_run.py; guard test_ladder.py::test_commit_visibility_parity. PARITY.md: instrument-first path; corrected oort root (recency guard dropped as a value-fudge under intrinsic latency -> task-type-keyed latency) and felix root (gate inert/gate_holds=0 -> clock-jump, not dispatch-ts pacing); new run-length budget table. No behavioral fix this commit; next runs are measurement runs to populate the metric. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): felix clock-jump clamp + U6 commit-visibility point-mass guard felix: re-base the inert arrival-gate onto modeled completion. The 2.5h run showed gate_holds=0 (every in-flight trainer already buffered) with 72% of commits past-dated (fresh source avg ~807s gap) and U6 sim lag 14.8s vs real 0.017s. _sim_recv_min now caps each commit's clock advance at the earliest in-flight FUTURE modeled completion (exp>vclock)+slack, so a far-future straggler can't lap the fresh fast cohort; never backwards; an already-due/ abandoned end can't pin the clock. Config simClockJumpClamp (sim-only, default on; pinned in the felix sim yaml). Guard: TestClockJumpClamp. U6: both-modes mean lag <= NEAR_ZERO_LAG_S (50ms) => pass on mean (KS uninformative on a near-zero point mass, the A2 num_candidates precedent). Restores oort 40->41/45 (sim 0.001s/real 0.004s false-positive); felix's genuine 14.8s gap still FAILs. Guard: test_commit_visibility_parity case (5). oort task-type-keyed latency DROPPED: premise falsified (sync oort dispatches 0 eval tasks; eval bundled into the train commit). Documented in PARITY.md dead-ends. Tests: 159 pass / 7 skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): felix eval ships its own sct (kill stale-train past-dating) Root cause of felix "fresh" past-dating: evaluate() never recomputed _sim_completion_ts, so the send path stamped every eval message with the trainer's LAST TRAIN completion ts. Eval then committed long after the vclock passed that stale sct (gaps to 5234s), mislabeled "fresh" (classifier keys on MODEL_VERSION), and poisoned the reorder-buffer minimum (peek_min_ts) used by the gate/probe ceiling. The clock-jump clamp was inert by construction: it caps forward advance, not a past sct on the wire. Confirmed: trainer ...0544 trained twice (sct 24.2, 165.6) then evaluated 65x, all shipping sct=165.6. Fix (sim-only): evaluate() stamps sct = send_ts + max(real_eval_gpu, D/20). Eval commits now emit task-tagged commit_gap_s / update_visibility_lag_s; analyze_run splits the commit-gap & visibility-lag plots train-vs-eval (eval series absent for no-eval baselines like sync oort). New checker rung eval_commit_timeliness (U6e, stage 6) flags eval systematically past-dated vs train. PARITY.md Status + dead-ends updated; tests 160 pass / 7 skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(parity): felix event-driven (staggered) re-dispatch — recover completion stagger Root (confirmed from logs): the felix asyncfl round boundary re-dispatched the whole freed cohort at one frozen round-start vclock, collapsing the per-trainer completion stagger real keeps (median committed-sct span 52.6s sim vs real ~3.85s) -> small per-round advance, ~2x rounds, U6 past-dating, U3 staleness 15 vs 3, K4 14x vs 6.7x. Fix (config-gated simStaggeredRedispatch, default off => byte-identical): - _sim_recv_min: on each TRAIN commit push the just-advanced vclock to a freed-slot FIFO (_sim_free_slot_ts) and record _sim_last_commit_sct. - _distribute_weights: pop the FIFO (oldest first) to stamp each re-dispatched trainer's sim_send_ts instead of one shared round-start frontier, so sct = sim_send_ts + compute regains the stagger. Backdating bounded by one round's advance (~4s) << min compute (~12s) => no commit past-dated at dispatch; MODEL_VERSION stays self._round. Eval/real/flag-off keep the single shared stamp + serialize-once. Sync (oort/syncfl) untouched (barrier cohort). - build_dispatch telemetry (event=dispatch: redispatch_stagger_s, held_s) for per-dispatch validation. Enabled in the felix sim block of the parity yaml (simRedispatchGapSeconds zeroed — now redundant). Guards: tests/mode/test_async_staggered_redispatch.py (12 cases). Full suite green (173 pass / 7 skip). Also commits the pending eval-commit partition checker sync (checks.py: eval agg_round records partitioned into eval_commits so K1/U3/U5 train-checks aren't contaminated; test_ladder guard) and the PARITY.md doc update. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): felix sct-ordered direct drain — commit-side ingestion root The async sim clock under-advanced (1.4 s/rd vs real 3.85 → 2.8x rounds, staleness 15 vs 2.8) not because of a dispatch-stagger collapse but because of COMMIT-side ingestion. In sim trainers don't sleep, so every in-flight update arrives ~instantly (wall_lag 0.29s), yet they were ingested through the recv_fifo streamer (background task + shared _rx_queue + per-end dedup + grace timeout + is_rxq_empty probe), which strands a delivered update out of the reorder buffer's view. The clock then advances off the incomplete buffer and laps the stranded lower-sct updates (queue_wait 26s mean, 90.6% >5s; 52% of commits past-dated). Only ~1.6 of 10 commits/round advanced the clock. Fix (config-gated simSctOrderedDrain, async only, default off => byte-identical): - End.get_ready_nowait() (non-blocking, peek-aware) + peek-aware is_rxq_empty(). - Channel.drain_ready(): pull raw payloads on the backend loop (queue ops only), decode off-loop so cloudpickle.loads can't stall the message pump; no streamer, no shared queue, no dedup. Per-message bookkeeping factored into _apply_recv_payload, shared with recv_fifo. - _sim_recv_min drains the live in-flight set (recv_ends | _sim_inflight_expected) via drain_ready; the min-sct gate + clock-jump clamp are unchanged (correct given a complete buffer). Sync untouched (its recv_fifo barrier already waits for the whole cohort). - simStaggeredRedispatch superseded/falsified (advance 1.93->1.38) -> off in yaml. Guards: tests/mode/test_async_sct_ordered_drain.py (End primitives; real channel.drain_ready over a threaded loop incl peek_buf; flag-on sct-order/gap0 under any arrival; flag-on == flag-off on a reliable channel). Readiness suite 182 pass / 7 skip. PARITY.md Status replaced with the corrected root. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): felix one-in-flight-per-trainer — simInflightResidence (K3b/U3/U6 tail) Post-drain, felix sim's bulk past-dating was gone (commit_gap median 0, staleness 15.2->5.1, advance 1.4->3.04) but a ~14% tail still past-dated up to 119s (staleness 34), holding K3b/K2/U3/U6/K8/U2 open. Root: felix async re-dispatched a fast trainer (freed instantly in sim — no train sleep) while its prior update was still in flight. _sim_inflight_expected is a per-end dict; the overlapping re-dispatch overwrote the entry and the earlier update became untracked, invisible to the sct gate (gate_holds=0 all run) -> lapped -> past-dated. Measured via overlapping dispatch->commit intervals: real 0.0% overlap vs sim 13.9% (83 trainers). Real never does this — the channel holds an in-flight trainer out of VAL_CH_STATE_SEND until its update returns AND is processed. Fix (sim-only, default off): the felix analog of §4.5. In _distribute_weights, when sim_inflight_residence is on, add the outstanding set (set(_sim_inflight_expected), the dispatched-but-not-committed keys) to curr_unavail_trainer_list before set_curr_unavailable_trainers — the unavailable path, NOT selected_ends. AsyncOortSelector already filters trainer_unavail_list. Released on commit (key pop). Keeps the per-trainer in-flight accounting single-valued so the sct gate the drain feeds is never blinded by an overwrite. Enabled in the felix sim parity yaml. Guard: tests/mode/test_async_inflight_residence.py (5 tests). Readiness suite 157 pass / 7 skip; async stack 26 pass; parity subsuite 30 pass. PARITY.md: Status rewritten in place; new §3.resid; core invariant recorded; durable lessons (gate_holds=0 as a tell, partition-the-tail, interval-overlap vs warning counters) + dead-end (don't tune the gate predictor). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): felix one-in-flight via slot-holding, not UN_AVL (over-selection fix) simInflightResidence v1 marked busy (compute-outstanding) trainers UN_AVL. But UN_AVL = cannot participate at all; a busy trainer is AVL_TRAIN/AVL_EVAL, just occupied. Routing them through curr_unavail_trainer_list filtered them from eligible_ends, so _handle_send_state evicted them from selected_ends and freed their concurrency slot. With extra = c - len(selected_ends), the selector refilled c slots with NEW trainers every round -> in-flight ramped to N~300, the reorder buffer became a 300-deep dense sct cluster, and the vclock crawled at 0.27 s/rd (slower than wall; run_20260621_011547 stopped early). Fix: hold a busy trainer in its selected_ends SLOT until commit (mirrors real, where the channel keeps it out of VAL_CH_STATE_SEND until aggregated). Extract _sim_hold_busy_slots (called from _aggregate_weights): flag-off holds the buffered set (unchanged), flag-on widens to pending_ends() | _sim_inflight_expected (train+eval), released on commit in _sim_recv_min. Holding in selected_ends both bounds concurrency and excludes the trainer from the pool (all_selected) -> no over-selection, no overlap. Removed the unavail-list block in _distribute_weights. Sync oort §4.5 unavail-list use is untouched (barrier re-selects the cohort). Tests rewritten: busy is slot-held and never unavail-marked; flag-off holds only buffered; eval held; committed releases its slot. 153 pass / 7 skip (mode+parity). Cluster validation pending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(parity): felix full mechanism parity (46/46); perf-pass roadmap felix sim run_20260621_154600 vs stored real …002022 (90min/5400s): 46/46 enforced pass. _sim_hold_busy_slots one-in-flight closed the overlapping-re-dispatch tail — K3b 0.82→-0.08, advance 3.04→3.93, staleness 5.11→2.83, U6 mean_diff 3.91→0.016s. K3b downstream cluster (K2/U3/U6/K8/U2) cleared together (single root). Next steps noted: (1) ≥7200s run for C1/C2 convergence sign-off; (2) close oort/refl/feddance A2c speed-model family to 46/46 BEFORE the sim perf pass (don't strip diag logging while it's still needed for root-causing); (3) flag-gate diagnostics, trim hot-path JSONL, re-run all-baseline parity as the perf guard rail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): oort/refl — record speed+utility for stale-but-returned updates Oort marks a trainer unexplored when PROP_STAT_UTILITY is None and re-selects it. Real trainers sleep their heterogeneity budget, so a slow trainer's update arrives stale; the stale-reject branches `continue` before _handle_weights_msg, recording neither speed nor utility — so real re-explored the same slow trainers forever (broad mix, carry-over ~3.9) while sim recorded them (tight mix, carry-over ~0). Sim was correct; real was the information-loss bug. Fix (default behavior, no flag; oort + refl share oort/top_aggregator): _record_returned_trainer_props() records PROP_STAT_UTILITY + PROP_ROUND_DURATION for any returned update (real: recv − per-version send; sim: SIM_ROUND_DURATION), called in both stale-reject loops before `continue`. Aggregation unchanged — stale updates still never enter the model. Also completes sim (adds the missing utility). Needs real+sim reruns for oort/refl (real path changed). Guards: TestStaleTrainerPropsRecorded. PARITY.md durable lesson added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(parity): condense felix (CLOSED), pivot Status to oort→refl→feddance Felix is 46/46, so its validation prose + redundant "prior read" detail (covered by §3.drain/§3.resid) is condensed to a scoreboard + key transferable learnings. Status/next-steps now lead with oort (stale-property fix landed, reruns pending), refl (shared fix, dirs gone), feddance (separate selection_bias root, dirs gone). Corrects the superseded "one A2c speed-model root" framing in the roadmap and the settled-roots table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(parity): integration regression for oort/refl stale-prop recording Drives the real OortTopAggregator._aggregate_weights stale-reject path end to end and asserts the rejected trainer's speed + utility ARE recorded (and the stale update is NOT aggregated). Guards the call-site wiring, not just the helper: verified to FAIL on pre-fix code (both call sites removed) and PASS on the fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): oort — record client task-train duration excluding agg read-wait Root (oort K3b, REVISED): the residual sim/real selection gap is the SCORER, not residence. Discriminator (scripts/oort_residence_discriminator.py) decomposes sel_rate = eligible_fraction x P(sel|elig): the eligible menu is identical sim/real (slow-trainer in-flight residence dead-equal 0.00444 vs 0.00438), so the whole gap is P(sel|elig) — sim picks slow 1.23x. Cause: real's stale-path duration was recv_ts - dispatch, which for a stale straggler bundles in AGGREGATOR READ-WAIT (a finished update sits unread until a later round drains the reorder buffer; up to 1.65xD, real_med 39.9s for a D=24s client) — a server artifact, not client speed, inflating slow trainers so real over-penalized/avoided them while sim (clean D) kept picking. The "speed-tail" framing was wrong. Fix (Option A — real is the buggy side): real records the CLIENT task-train duration = WALL_SEND_TS - dispatch (read-wait excluded; trainer telemetry confirms wall_send-wall_recv == compute to +0.01s). Single-sourced in _real_client_task_train_duration, applied to both fresh and stale paths. Sim unchanged (already records modeled D) => real rerun only to validate. Terminology rename (clients do TASKS not rounds; train-typed, future-proofs eval): PROP_ROUND_DURATION -> PROP_CLIENT_TASK_TRAIN_DURATION ("client_task_train_duration_s") SIM_ROUND_DURATION -> SIM_CLIENT_TASK_TRAIN_DURATION_S ROUND_COMPUTE_S -> CLIENT_TASK_TRAIN_COMPUTE_S across selectors + aggregators + message enums (enum int values unchanged -> no wire break) + checker/doc references. Tests (test_sync_sim_ordering.py) rewritten to uphold the concept: real asserts client-duration excluding read-wait (12 not 18), a dedicated test covers the no-WALL_SEND_TS fallback, the integration test exercises the production path. 300 mode+selector+telemetry tests pass. PARITY.md condensed 915->476 lines (preserving all settled roots, durable lessons, dead-ends, mechanism fixes, discrepancy ledger). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * updated status after oort run * fix(parity): feddance U6 real barrier-anchor + refl residence-class telemetry feddance U6 commit_visibility was a REAL-telemetry flaw, not a sim bug: real `_update_visibility_lag` took `committed=datetime.now()` per-message in the recv loop (measured arrival->ingestion ~0.02s), but a strict sync barrier applies all K at one post-loop instant. Real now anchors on the single round barrier: lag_i = max_dur - dur_i, dur = WALL_SEND_TS - dispatch (sim unchanged: vclock-sct is already the barrier). Validated vs stored real logs (mean 15.65 ~= sim 15.48), no rerun needed to confirm the formula. Streaming aggregators (oort/refl) keep the per-message path (each commits at its own sct -> lag ~0) so only syncfl is touched. refl A2 num_eligible decomposed to selected_ends residence SHAPE (real mode at residence=3, sim flatter; tail identical). Added telemetry-only instrumentation: inflight_residence now emits residence_staleness + residence_was_fresh paired 1:1 with residence_rounds, to split the residence gap by commit class. No refl fix yet (diagnostic for the 3h rerun); the pending_after->pending_ends lever was falsified. Guards: test_sync_sim_ordering (barrier-anchored lags), test_parity_checks (paired commit class). Full suite 199 pass / 7 skip. PARITY.md updated (§6.u6, settled roots, 3h-rerun expectations, dead-ends). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): refl/oort UCB temporal term — faithful last-receipt time_stamp + selector telemetry refl's UCB temporal-uncertainty term was DEAD (refl_oort.select() never let it fire → time_stamp None → bonus 0; 0/7513 nonzero), so refl scored on stat_utility alone and selected speed-uniformly while the real run leaned toward slower trainers — driving the K3b advance gap (sim 3911 vs real 3332 rounds at 3h). Both third_party/Oort and third_party/REFL key the bonus on the agg round of the last RECEIVED update (time_stamp = self.epoch), registration-initialized so it is never None. - Stamp PROP_LAST_RETURNED_ROUND at every receipt (fresh + stale) in oort/top_aggregator - OortSelector.calculate_temporal_uncertainty_of_trainer reads it with registration-init lazy to current round; enable_temporal kwarg (default True; False = ablation only) - Remove superseded D5 last-selected machinery (_record_last_selected_round); no baseline used the "selected" source. felix AsyncOortSelector is a separate class — untouched - Add per-candidate selector telemetry: score / selection_prob / in_exploit_pool / exploit_cutoff (stored logs lacked the final draw weight) - Guards: TestREFLTemporalFidelity, TestTemporalUncertaintyFidelity (348 pass / 7 skip) oort now uses the faithful term too → needs a re-validation run (parity is within-baseline; both modes get it symmetrically). Caveat: temporal was dead in BOTH refl modes, so this is a fidelity fix; the rerun measures whether it closes K3b. PARITY.md D7 + §S.temporal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): selector duration = intrinsic client stamps (WALL_SEND-WALL_RECV), all baselines K2 throughput root (oort+refl): sim selected a ~0.77s-faster client mix because the selector's observed-duration INPUT diverged. Real recorded WALL_SEND-dispatch, which still carried the dispatch->recv DELIVERY LAG (WALL_RECV-dispatch, +8s for slow one-in-flight stragglers), inflating round_preferred_duration (real 10.7s vs sim 7.0s) so real under-penalized slow clients -> committed a slower mix -> +0.5s/rd advance -> more rounds -> K2 fails. Proven server-side, not speed: the trainer's own stamps give WALL_SEND-WALL_RECV = D exactly (real GPU ~0.01s, device sleeps to D); obs==static D for committed but static+8 for the candidate-pool slow tail. The earlier read-wait fix anchored only the recv side; this anchors BOTH. Fix, single-sourced in flame/mode/horizontal/client_duration.py and applied to every horizontal aggregator that tracks client duration: - oort/refl (the K2 root): _real_client_task_train_duration now wraps the helper. - asyncfl/felix: was recv_agg-dispatch (read-wait + delivery lag, the crudest); latent at 1.5h (mostly-fresh commits) but the same disparity at length. - syncfl/feddance+fedavg, fwdllm: were recv-dispatch. Each falls back to its prior agg-anchored measure only when a client stamp is absent (byte-identical where stamps missing). sim paths unchanged (SIM_CLIENT_TASK_TRAIN_DURATION_S = max(gpu,D)). syncfl's _real_task_dur (WALL_SEND-dispatch) U6 barrier anchor (s6.u6) is a separate quantity, untouched. Telemetry [CLIENT_DUR_STRIP] (intrinsic vs delivery_lag per stale record). Guards: tests/mode/test_client_duration.py (helper contract) + test_sync_sim_ordering.py intrinsic/fallback. Suite 346 pass / 7 skip. Real-path only -> validate via 3h real reruns of oort+refl vs stored sim; re-validate felix at a longer horizon. See PARITY.md s.dur. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): faithful Oort pacer port (oort+refl+felix) + S2 speed-class participation §S.pacer — flame's pacer() was an unfaithful port of reference Oort (third_party/Oort/oort/oort.py:184-199; byte-identical in third_party/REFL). The reference makes two symmetric moves on the exploited-utility trend over two pacer_step windows: FLAT |Δ|<=0.1·last -> round_threshold += pacer_delta; SHARP |Δ|>=5·last -> -= pacer_delta (floored). flame raised on ANY dip (last > curr) with NO decrease branch -> a monotonic ratchet to 100, hypersensitive to per-round utility noise. Since sim/real have different stochastic utility trajectories, round_threshold drifted to different levels and, never recovering, the gap compounded in the back half -> sim under-bound the speed penalty (73.6% vs real 94.1%) -> ~1.5s-slower max-of-K (10.5 vs 8.99) -> K3b/K2 fail (the oort Jun-24 3h regression). Confirmed it's the pacer not the §S.dur input: pool static speed + selected mean (A2c) match; only the per-round pref percentile diverged. Fix: - OortSelector.pacer(round): faithful two-branch port, keyed on the current round (= reference training_round), pacer_step>0 guard. round_threshold added to the selection telemetry extra (was only inferable from pref). - REFLOortSelector.pacer override removed — it was already faithful; oort+refl now share the one reference-matching base. - AsyncOortSelector.pacer (felix, separate class): same bug, ALSO fixed in-class, AND train-gated the call — felix runs one selector for train+eval but self.round / exploitation_util_history advance only on train, so the pacer was firing on the eval hand off a stale round. NOTE: felix's penalty was inert *because* of the old ratchet; it is now active, so felix's 46/46 must be re-validated next run. S2 participation (refl) — matched_count_ks 0.464 was pure stochastic core-IDENTITY, not a mix bias: both modes build ~120-trainer cores that are speed-matched (real-only D̄ 9.8 / sim-only 9.2; participation-weighted 8.20/8.26; A2c/K8 pass) but share only 63 individuals (weighted-exploit rich-get-richer). At speed_class granularity participation matches (TVD 0.026) while per-second buckets re-expose the identity noise (TVD 0.187). participation_parity now enforces participation by speed class for stochastic selectors (new _trainer_speed_class_map), per-trainer identity demoted to diagnostic; DETERMINISTIC_SELECTORS still exact. refl 43->44. Jun-24 3h scoreboard: felix 46/46 (re-validate after pacer), refl 44/46, oort 40/46 (pacer fix validates next run), feddance 43/44. Discrepancy ledger D8. Guards: TestPacerFidelity (oort + async). 352 selector/mode/sim/parity pass / 7 skip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(parity): condense PARITY + unavailability/fwdllm design docs for pickup Compress PARITY.md's dated Status narration to a scoreboard + next-steps (durable ladder/roots/lessons/dead-ends/mechanism reference kept intact). De-duplicate UNAVAILABILITY_DESIGN.md and simulate_fwdllm.md into pickup-ready, independently actionable specs: prerequisites header -> PARITY.md, consolidated decisions, and open-question ledgers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(parity): crisp code comments to the why (no behavior change) Trim journey-narration (dated/"was X before"/§-history) in the selector + mode/horizontal mechanism files down to the mechanism + design reason that prevents regressions. Keep §-references into PARITY.md. Comment/docstring-only — no code lines changed; full selector/mode/sim/parity suite green (211 pass / 7 skip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Dhruv Garg <dgarg39@markov.cc.gatech.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu>
* update launcher script with latest changes of async cifar
* fwdllm migration to launcher plan
* checkpooint1
* checkpooint2
* checkpooint3
* checkpooint4
* smoketestA
* checkpooint5
* checkpooint6
* plan updates and some metadata updates from before
* checkpooint7
* checkpooint8 and smoke test B
* checkpoiioint9
* checkpoint10
* smoketestC
* checkpoint11
* checkpoint12
* checkpoint13
* smoketestD
* redesign to clean up baselines across works
* Implement fwdllm baseline taxonomy (fwdllm/fwdllm_plus/fluxtune/fluxtune_dynkc)
Phase 7 of the launcher migration: makes _validate_stack selector-driven for
the fwdllm stack (and fixes a latent bug where its regex never matched the
real main_fedfwd_agg.py entrypoint), adds an explicit learning_rate kwarg to
FedBuff, re-sources oracular availability from _metadata instead of legacy
json_scripts/, and adds a reselect_each_iteration selection-granularity gate.
Replaces the fedfwd_async_random_dynkc/fedfwd_oracular baselines with the
owner's three-way taxonomy (fwdllm, fwdllm_plus, fluxtune) plus a preserved
fluxtune_dynkc parity variant, with fluxtune's K/C policy made config-driven
(dynamic_kc enabled/disabled + policy) rather than hardcoded fixed K/C.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Add fwdllm_plus/fluxtune smoke YAMLs and restructure migration doc chronologically
Adds the two missing per-baseline smoke-test experiment YAMLs (fwdllm_plus,
fluxtune) alongside the existing fwdllm one, so all three owner-spec
baselines get real/sim parity coverage going forward, with a regression test
(TestFwdllmSmokeYamlsResolve) that resolves and validates each against the
real aggregator entrypoint.
Rewrites MIGRATION_TO_LAUNCHER_FWDLLM.md into strict chronological order
(Context/Phase1-5/Smoke A-D/Phase6/Phase7, previously Phase6-7 were
prepended ahead of older phases) with a single "Status & next step" section
at the top documenting Phase 7 step P8 (live smoke tests) as the sole
remaining item, including the full pinned-ML-stack (adapter-transformers /
tokenizers Rust-toolchain) blocker writeup and exact run commands, so a
future session can resume from this file alone.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Reconcile a single dev env for fwdllm + vision/speech examples
Migrate fwdllm onto a modern, single dependency stack so one conda env runs
async_cifar10, async_google_speech, and fwdllm — replacing the un-buildable
adapter-transformers fork.
setup.py ([examples] extra is now the single source of truth):
- Add the NLP forward-mode stack (transformers>=4.57<4.58, adapters>=1.3<1.4,
h5py, pandas, scikit-learn, setproctitle) and torchaudio (google_speech),
so `pip install -e lib/python[examples,dev]` provisions one env for all.
Code (modern `adapters` add-on; keeps peft_method: adapter on DistilBERT):
- tc_transformer_trainer_distribute.py: import AdamW from torch.optim (removed
from transformers top-level in v4.x).
- expts/initializer.py: import adapters/BnConfig/LoRAConfig from `adapters`;
adapters.init(model) in the adapter/lora branches; build the bottleneck
adapter via BnConfig(**adapter_config) (legacy keys map 1:1).
Verified on py3.11/torch2.12/numpy2.4: stack installs from prebuilt wheels
(no Rust/tokenizers compile), both launcher entrypoints import past the old
wall, the adapter path freezes the base (~0.27% trainable), and the
launch/mode/optimizer/selector suites stay green (315 passed, 7 skipped).
Docs:
- New single setup+run guide at lib/python/README.md (covers all 3 examples).
- Updated prerequisites/quickstarts/example READMEs to note the extra now
covers the NLP/speech examples.
- fwdllm/MIGRATION_TO_LAUNCHER_FWDLLM.md: Phase 8 (env reconciliation) DONE;
P8 live smoke is the next step.
- DEPRECATED_FILES.md manifest + req.txt deprecation banner (cleanup deferred
to a dedicated commit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix fwdllm smoke-test crashes + CUDA 12.9 env setup
- FedSgdTrainer: replace ast.literal_eval on avl_events_* with
_parse_avl_events() -- YAML delivers lists, not strings
- Aggregator: add manual_seed to aggregator.config_overrides in all
three smoke YAMLs so main_fedfwd_agg.py can read it directly
- setup_env.sh: force-reinstall torch/torchvision from cu126 index
(required for CUDA driver >= 12.9)
- setup.py: note cu126 requirement in examples extra comment
- .gitignore: ignore lib/python/examples/fwdllm/experiments/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix remaining fwdllm smoke crashes + establish configs/ design invariant
Per-example config templates belong in configs/, not _metadata/:
- configs/aggregator_base.json: new fwdllm aggregator template with all 25
NLP/model fields (model_name/type, peft_method, fp16, manual_seed, etc.)
that main_fedfwd_agg.py reads; shared _metadata/aggregator_base.json has
none of these (generic FL fields only)
- smoke YAMLs: switch config_template to ../configs/aggregator_base.json;
drop manual_seed override (now in the template default)
- configs/trainer_base.yaml: add avl_events_syn_train_* defaults ([[0,
"AVL_TRAIN"]] = always-available) for three legacy traces that
FedSgdTrainer.__init__ reads unconditionally but spawner never injects
- runner.py: accept .yaml/.yml config templates (extension-sniffed) alongside
.json, enabling future migration of aggregator_base.json to YAML
MIGRATION_TO_LAUNCHER_FWDLLM.md: document design invariant as Phase 9 +
record all four P8 live-run bug fixes for the cross-example rule-set doc.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix config_template path: configs/ not ../configs/
config_template is resolved as example_dir/path, so the correct path
is configs/aggregator_base.json (relative to fwdllm/), not ../configs/
which would look for examples/configs/ (doesn't exist).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix two more fwdllm smoke crashes
- aggregator_base.json: add data_loader_num_workers, client_num_in_total,
warmup_ratio (full coverage -- verified against all config.hyperparameters.*
reads in main_fedfwd_agg.py)
- FedSgdTrainer: replace hardcoded ../../.. path in _write_client_data_to_file
with self.args.output_dir/client_data_files (/tmp/ by default)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Gate minInitialTrainers assertion on is_async in fwdllm_aggregator
sync/random baselines (fwdllm, fwdllm_plus) have no minInitialTrainers in
selector kwargs; only fluxtune (async_oort) does. The unconditional assert
blocked all sync smoke runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* wip changes, context saved in launcher markdown files
* ready for next smoke tests, misc: data partition files for cifar
* updates as of 7.30pm est
* smoke tests and stopping criteria
* fix fwdllm bug
* all 3 working
* bug fix
* trim comments
* PR comment adressed: Remove files that we not longer needed.
* Updated .md files with the learnings from actual code changes
* recover required cifar files
* Fix F+/FT smoke-test crashes: var-check crash guard, per-baseline staleness policy, GPU undersubscription
- fwdgrad_utils.calculate_var/calculate_real_var: guard n<2 (matches the
n<2 guard pattern already used by calculate_snr/calculate_cv) instead of
crashing on torch.stack([]). Root-caused to a stale trainer update
racing into a freshly-cleared grad_for_var_check_list at a data_id
transition, which crashed the fwdllm_plus aggregator ~30min into the
09-29 21:03 smoke run.
- Add Hyperparameters.staleness_policy (flame/config.py) with three modes
-- "exact" (must match the aggregator's current round/data_id/iteration),
"round_data_id" (must match round/data_id, any iteration), "none" (no
gate) -- superseding the old reject_stale_updates boolean for FedFwd.
Wired into fwdllm_aggregator._process_single_trainer_message and set per
baseline: fwdllm=exact, fwdllm_plus=round_data_id, fluxtune=none. The
round_data_id policy would have rejected the stale update that caused
the crash above instead of silently mixing it into a newer cycle.
"exact" needed ITERATION_PER_DATA_ID echoed back in the trainer's
GRADIENTS message (fwdllm_trainer.py), which the wire protocol didn't
carry before. Removed the now-redundant/misleading
rejectStaleUpdates: false from the three FedFwd baselines that set
stalenessPolicy explicitly; left it as-is on fluxtune_dynkc and other
non-FedFwd examples that still rely on the plain boolean.
- expt_scripts/*.yaml: default execution.num_gpus to 8 instead of 1 --
each smoke YAML's default is sized for its own 10-trainer count, and
run_sequential.sh's --num-trainers scale-up had no matching GPU
override, so 100 trainers crammed onto 1 GPU OOM'd almost immediately
(the fluxtune_n10_smoke 09-29 22:11 run). Also add --num-gpus and --only
flags to run_sequential.sh, and have it rewrite the _nNN_ token in
exp.name when --num-trainers overrides the count, so the resulting
experiments/run_<ts>_<name> directory reflects the actual trainer count
instead of staying named after the YAML's checked-in default.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* run_sequential.sh: use plain baseline names as run identifiers, not yaml's n10-suffixed name
--only/RUNS keys are now "fwdllm"/"fwdllm_plus"/"fluxtune" instead of
"fwdllm_n10_smoke"/etc. -- the n10 was each source YAML's own checked-in
default trainer count, not the run's identity, so a 100-trainer
--num-trainers override still left the command itself saying "n10".
The exp["name"] rewrite (which feeds the run_<ts>_<name> experiment
directory) is now derived directly from this clean run key + the actual
--num-trainers value, rather than regex-substituting the nNN token out of
the source YAML's own name field -- more correct (independent of whatever
naming convention a given source YAML happens to use) and simpler.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* run_sequential.sh: keep job.id in sync with the renamed exp.name
Verifying the previous commit's generated YAML end-to-end (via
ExperimentBatch.from_yaml, the launcher's own loader) surfaced a field
patch_yaml missed: aggregator.config_overrides.job.id, which every
checked-in source YAML keeps equal to exp["name"] but which the n10->n100
rename wasn't touching, leaving it stuck on the stale "..._n10_smoke" id
even after exp["name"] was correctly rewritten.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fwdllm aggregator: log before the CUDA availability check, not after
torch.cuda.is_available() can block indefinitely if the GPU driver is
wedged, and it ran before logging.basicConfig() was even called -- so a
hung aggregator produced a silently empty log file instead of a log that
visibly stalls right after "checking CUDA availability...".
* fwdllm PR cleanup plan: add node-testing runbook + JSON->YAML migration audit
Adds the concrete commands for the next session (on a healthy GPU node --
jayne is still wedged) to run the pytest suite, smoke-test all three
launcher baselines at minimal length, and verify the migrated trace/config
data is wired correctly at runtime. Also records this session's static
JSON-vs-YAML field comparison (trainer_base.yaml, mobiperf_traces.yaml,
fluxtune_dynkc vs the legacy aggregator.json), five open questions it
surfaced, and a first-pass legacy-launch-path deletion candidate list for a
follow-up PR. Workstream A/B code changes remain uncommitted pending the
deferred test run.
* fwdllm: port missing syn_train_* availability traces into _metadata
Two of the three avl_events_syn_train_* traces (90_eval_10_unavail_0,
50_eval_30_unavail_20) had real per-trainer data sitting unused in the
legacy expts/run_tc_expts/json_scripts/trainer_*.json (150 trainers) that
was never ported into _metadata/availability_traces/synthetic_traces.yaml.
Ports all three (the third, 100_eval_0_unavail_0, is degenerate/always-
available like syn_0) as per_trainer.n300 entries, wrapping trainers 151-300
onto the 150 source trainers' traces to preserve the named distribution.
Verified against the JSON source with a temporary migrate/verify script
pair (run, confirmed pass, deleted). Spawner auto-injection for these keys
isn't wired up yet -- data correctness only, wiring is future work.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm docs: consolidate migration notes, trim changelog-style comments
MIGRATING_TO_LAUNCHER.md's fwdllm section (§9) predated this branch's most
recent commits, so it was missing several real lessons -- adds a subsection
covering the CUDA-before-logging ordering fix, the var-check n<2 guard,
staleness_policy's three modes, run_sequential.sh's num-gpus/naming
conventions, and a pointer to the now-available syn_train_* traces.
MIGRATION_TO_LAUNCHER_FWDLLM.md (1249 lines, a dated phase-numbered dev
journal) is now a short pointer stub at §9 + git log, matching this repo's
DEPRECATED.md convention instead of leaving two overlapping docs.
Trims verbose changelog-style comment blocks (logic unchanged) in
run_sequential.sh and fwdllm_aggregator.py, and fixes the repo-wide
references to MIGRATION_TO_LAUNCHER_FWDLLM.md's now-removed Phase
N/decision DN numbering that the doc consolidation above left dangling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: fix stale flame import, drop hardcoded conda env
PYTHONPATH now prepends this checkout's lib/python so a stale
`pip install -e` editable install pointing at a different flame clone
can no longer shadow flame.launch. ENVNAME now comes from
FLAME_CONDA_ENV or the shell's already-active CONDA_DEFAULT_ENV
instead of a hardcoded default.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: broadcast EOT in sync-mode compose too
The sync-mode compose() branch built its tasklet chain from scratch
and dropped the inform_end_of_training tasklet (only present as a
stale comment), unlike the async branch which keeps it. When a
sync-mode aggregator's loop ended (e.g. max_data_id_progress /
max_runtime_s early-stop), it exited without ever broadcasting EOT,
leaving trainers blocked in await_join with no way to self-terminate
-- the launcher's wait_all() then had to force-kill each one after a
30s timeout, turning a ~90s aggregator finish into several extra
minutes before the batch could move to the next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* spawner: wait_all() polls trainers concurrently, not sequentially
wait_all() waited on each trainer one at a time (proc.wait(timeout=30)
in a for loop), so when every trainer misses the aggregator's one-shot
EOT broadcast (a real race: await_join() only catches peers already
joined at broadcast time, and has no timeout), the batch paid
30s * num_trainers before moving to the next sequential run --
10 trainers turned a ~90s aggregator finish into ~390s wall time.
Now all trainers share a single timeout_per_trainer window, polled
concurrently, so the worst case is one 30s wait regardless of trainer
count.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: add --partition-method override
Needed to run a heterogeneous (non-IID) data split for the longer
convergence soak run, instead of the smoke-test default of "uniform"
(IID, chosen for smoke tests specifically to isolate launcher-mechanics
validation from data-skew effects). Patches
hyperparameters.partition_method on both the trainer and aggregator
config_overrides, same pattern as the existing --num-trainers/--c/--k
overrides.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: clean up dead/confusing fields in aggregator_base.json
Removed the unused top-level "dependencies" field (Optional in the
generic ExperimentConfig schema, safe to drop) and hyperparameters
"batchSize"/"learningRate" (alias into Hyperparameters.batch_size/
learning_rate, but no fwdllm code path reads those -- only
scaffold/trainer.py and datasampler/fedbalancer.py do, neither used by
fwdllm). The top-level "dataset" field can't be removed outright --
it's required (non-Optional) on the schema -- so its value was changed
from a misleading MNIST URL to an explicit "unused" placeholder.
Verified by round-tripping the edited JSON through the real
flame.config.Config parser.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm PR cleanup plan: resolve open questions, split off deletion tracking
All 4 open questions from the static JSON->YAML comparison pass are
resolved (rounds=1000 kept, dead config fields cleaned up, 3 legacy
async json variants confirmed droppable, fl_main.py x2 confirmed dead).
Moves the deletion candidate list into its own DELETION_CANDIDATES.md
so it survives PR_CLEANUP_PLAN.md's deletion once this PR merges.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector: fix SEND_TIMEOUT_WAIT_S reclaim so stuck ends actually free their slot
async_oort's reclaim ran after the extra==0 early-return, so once concurrency
saturated with stuck trainers the reclaim could never execute -- a full
deadlock at scale (reproduced at n30/~90min, not at n10/10min). It also only
deleted from all_selected, never from selected_ends, which is what the
concurrency accounting actually counts. Moved the reclaim to the top of
_handle_send_state and free both dicts together, mirroring random.py's
already-correct pattern. Same two-part bug existed in async_random.py;
fedbuff.py selector only had the incompleteness half (no early-return gates
it).
* asyncfl top_aggregator: add per-cycle duplicate-contribution guard
The selector reclaim fix makes a stuck trainer's slot reopen while its
original response may still be in flight -- previously not reachable since
the slot never actually reopened. The generic asyncfl aggregator (used by
async_cifar10's fedbuff/async_random baselines) had no per-cycle dedup,
unlike fwdllm's _per_agg_trainer_list guard. Add _agg_cycle_contributed_ends,
reset once per aggregation cycle in _reset_agg_goal_variables, checked in
_aggregate_weights before counting a contribution.
* fwdllm_aggregator: prune departed ends from per-round reselection cache
reselect_each_iteration=False accumulates picks into _round_selected_ends
and reuses that exact list for the rest of the round, bypassing select()
(and its SEND_TIMEOUT_WAIT_S reclaim) once the cache is full. A trainer that
disconnects or reports UN_AVL mid-round was never pruned from this
aggregator-level cache, so the round would stall forever waiting on a
contribution that can never arrive. Add _prune_departed_from_round_cache,
called before the cache-size check, so a departed end shrinks the cache
below target and the existing accumulate path backfills it.
* run_sequential.sh: add --agg-goal/--c-async/--min-initial-trainers/--avail-trace
Lets one invocation run a genuinely controlled comparison across all three
fwdllm baselines: --agg-goal decouples aggregator.agg_goal from concurrency
(previously --c conflated them), --c-async overrides concurrency for the
async baseline (fluxtune) only, --min-initial-trainers overrides
minInitialTrainers directly, and --avail-trace overrides the availability
trace across all three baselines' real trace-consuming fields so selection/
aggregation bugs can be isolated from trace-driven scarcity (e.g. syn_0).
Also gitignore smoke-test/parity-report artifacts left over from this
session's runs, and record the round-2 baseline investigation (deadlock
root cause + fixes, invariant re-check, stale-cache gap) in
MIGRATION_TO_LAUNCHER_FWDLLM.md.
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record commit hashes, correct next-step order
The doc was written before this session's fixes were actually committed;
update the commit-status lines and clarify the next step is verifying
run_sequential.sh configs for fluxtune/fwdllm_plus/fwdllm (not feddance,
which is an async_cifar10 baseline, not one of fwdllm's three).
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record push/config verification, plan Part 5 (telemetry/analysis parity)
Updates stale status (push done, 1.5h n=100 run verified and launched,
catching a real --max-data-id early-termination bug in the process) and
adds Part 5: a cross-example telemetry/analysis parity plan. scripts/
analysis/analyze_run.py was written for async_cifar10 and only informally
generalized -- it hardcodes that example's model size and assumes round is
the sole, fine-grained progress unit. Neither holds for fwdllm (round
advances once per 150 data bins; real progress lives in data_id/
iteration_per_data_id), and fwdllm's aggregator emits no telemetry at all
today, so performance/insights plots can't exist regardless of plotting
fixes. Also flags an unconfirmed telemetry cross-contamination anomaly
(foreign selector name + test-like fingerprints in real run telemetry)
that must be root-caused before trusting analysis built on existing data.
Part 5 tracks this as subtasks P5.1-P5.7, not yet started.
* fwdllm_aggregator: emit agg_eval/agg_round telemetry per aggregation cycle
The aggregator previously emitted zero telemetry of any kind, making
plots/performance/ and plots/insights/system/ structurally impossible
regardless of any analyzer fix. Both events are wired into
_process_aggregation_goal_met (the single per-cycle entrypoint shared by
fwdllm/fwdllm_plus/fluxtune), snapshotting contributor/staleness/utility/
speed stats before self._per_agg_trainer_list is reset and before
self._model_version potentially advances later in the same call, so
staleness is computed against the model version contributors actually
trained against.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector/random: emit selection telemetry from the real select() decision
RandomSelector (the actual aggregator-side FL-selection algorithm fwdllm/
fwdllm_plus use) never called emit_selection(), unlike every sibling
selector (oort/refl_oort/async_oort/feddance/fedbuff). Every "selection"
event that existed for those two baselines came exclusively from the
trainer's own trivial 1-candidate channel selector, never from the real
"which trainers got picked" decision. Adds the emit_selection(...) call in
select()'s SEND branch, right after selected_candidates is finalized.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* analyze_run: generalize progress axis + comm-cost via per-example manifest
analyze_run.py was written for async_cifar10 and generalized only
informally: it hardcoded that example's model param count and assumed
`round` is the only, fine-grained progress unit. Neither holds for
fwdllm-family examples, where `round` only advances once all
total_data_bins sub-units finish, collapsing hundreds of events onto one
x-value.
Adds an optional per-example telemetry_manifest.yaml (model_param_count,
progress_hierarchy, event_categories), loaded via
_find_manifest_path()/load_manifest()/configure_from_manifest() and applied
to progress_key() (generalized from a single hardcoded data_id fold to a
mixed-radix fold over the declared hierarchy) and MODEL_MB. An example with
no manifest (async_cifar10) is unaffected -- confirmed via a real-run
regression check. write_summary() cross-checks the manifest's declared
event_categories against which plots/<category>/ dirs actually got files,
flagging MISMATCH/DRIFT.
Wires in fwdllm's own manifest: 450,340 trainable params (not the full
66.8M-param model -- FedFwd's forward-mode/JVP scheme only ever puts the
trainable subset on the wire), data_id(150)/iteration_per_data_id(15)
hierarchy, and an evidence-based event_categories audit. Verified against
real GPU telemetry from the n=100 3-baseline experiment: manifest correctly
resolved, all 8 declared categories read "ok", zero MISMATCH/DRIFT.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* scripts/analysis: add README with onboarding checklist for new examples
Documents the 8 plot categories and what telemetry each needs, the
per-example telemetry_manifest.yaml schema, a 7-step checklist for wiring
telemetry into a new flame.launch example, and gotchas found while
onboarding fwdllm (the trainer-side selection.selector channel-
implementation artifact, the stale-reselection-cache bug class).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 5 completion, GPU experiment results
The 3-baseline (fluxtune/fwdllm_plus/fwdllm) n=100 GPU experiment finished:
no deadlock recurred (fluxtune, 3x the scale it stalled at), and the
fwdllm_plus round-count throttle did not reproduce. Manifest wiring
verified against the run's real telemetry (clean event_categories check,
zero MISMATCH/DRIFT). All of Part 5 (P5.1-P5.7) is now complete.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* selector/random: attach data_id/iteration_per_data_id to selection events
fwdllm_aggregator.py already threads (model_version, data_id, iteration_id)
through channel.ends(agg_version_state=...) -> select()'s kwargs, but
nothing consumed it. select()'s SEND branch now reads
kwargs["agg_version_state"] and attaches data_id/iteration_per_data_id to
the emitted selection event when present, letting analyze_run.py's
progress_key() place selection-derived plots on the same fine-grained axis
as trainer_round/agg_round/agg_eval instead of collapsing onto fwdllm's
coarse round. No-op for callers that don't pass agg_version_state (e.g.
async_cifar10's fedavg baseline also uses this selector).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: emit utility_belief; agg_round carries agg_observed_s
fwdllm_aggregator.py never emitted utility_belief (only
asyncfl/top_aggregator.py did), so selected_utility_believed_vs_actual*/
selected_utility_belief_gap* were structurally impossible for fwdllm-family
baselines regardless of selector. _process_single_trainer_message's
STAT_UTILITY branch now emits it before overwriting PROP_STAT_UTILITY
(believed = the prior value, actual = this message's fresh value), with
staleness computed against self._model_version (not self._round, which can
sit at 1 for an entire run) to match agg_round's own staleness convention.
_process_aggregation_goal_met now also builds agg_observed_s (an
{end_id: seconds} dict) from the PROP_ROUND_DURATION values already read in
the existing per-contributor loop, unlocking runtime_agg_vs_trainer/
runtime_overhead_hist/runtime_overhead_cdf -- a genuine wall-clock
measurement already computed, not an invented one.
build_utility_belief gained an extra param (mirroring build_trainer_round)
so fwdllm can attach data_id/iteration_per_data_id for progress_key().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* FedSgdTrainer: report sim_round_duration_s (gpu time + emulated delay)
fwdllm's trainer reported only real_gpu_time_s, unlike async_cifar10's
trainer which reports total round wall time. _emulate_training_delay() now
returns the seconds actually slept (0.0 if delay emulation is disabled);
train_with_data_id adds this to real_gpu_time_s and reports it as
sim_round_duration_s.
Deliberately not added: training_budget_s/overran/remaining_time_s.
fwdllm's delay is a flat additive sleep, not async_cifar10's budget-minus-
actual sleep-to-fill model, so there's no faithful "overrun" concept to
report -- fabricating one would be misleading rather than merely
incomplete. The budget/overrun-family plots stay not_populated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* analyze_run: re-key accuracy/loss/comm/selection plots to progress_key()
accuracy_by_round/loss_by_round/cumulative_comm_by_round previously stayed
on plain round deliberately (P5.2), since folding data_id into only one
side of comm_vs_accuracy_series's cumulative join would have silently
broken it -- selection events didn't carry data_id at the time. Now that
selector/random.py attaches it, all three (and the selection-derived
selection_funnel_over_rounds/selection_count_consistency/
eval_vs_train_selections/selected_speed_utility_*/
selected_utility_belief_gap_over_rounds) are consistently keyed by
progress_key(), so the join needed no code change -- just consistent
inputs on both sides.
Iteration-level granularity (not just data_id-level) chosen after checking
real n=100 telemetry: iteration_per_data_id varies 0..3 in practice, giving
73 distinct buckets vs. 31 at data_id-only resolution, ~11 events/bucket on
average. New PROGRESS_AXIS_LABEL constant applied everywhere a plot uses
this axis, including 3 pre-existing call sites whose label was never
updated when they were wired into progress_key() in an earlier session.
Verified end-to-end against synthetic fwdllm_plus-shaped telemetry (all
targeted plots populate with no errors) and a real async_cifar10 run (no
regression, progress_key() still a no-op there).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 6 (plot-coverage gap, fixed)
Root-caused why fwdllm's real n=100 plots were much thinner than a real
async_cifar10 run's despite Part 5's telemetry fixes: fwdllm's cache-reuse
selector fires once per round (not continuously), the trainer reports far
fewer timing fields, accuracy/loss stayed on plain round, and utility_belief
was never emitted. All four fixed and verified end-to-end (full suite: 470
passed, 7 skipped, 0 failed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: bound recv_fifo waits so max_runtime_s can't be starved
_aggregate_grads_async (fluxtune/async) and sync_collect_and_accumulate_
grads (fwdllm/fwdllm_plus/sync) both called channel.recv_fifo() with no
timeout, which defaults to blocking forever. If a selected/in-flight
trainer went quiet, the call never returned -- confirmed via a real
fluxtune run (run_20260701_225428_fluxtune_n10_smoke) that logged nothing
for 11+ minutes then had to be manually killed, despite --max-runtime-s 600.
Root cause: _check_early_stop_conditions() (which enforces max_runtime_s/
max_data_id_progress) only runs from _distribute_weights, on the other side
of the composer's put >> aggregate loop. While blocked inside the aggregate
call, the loop never cycles back to re-check it.
Both call sites now pass timeout=RECV_TIMEOUT_WAIT_S (30s), matching the
pattern asyncfl/top_aggregator.py's _aggregate_weights already uses for the
identical reason. A timed-out recv already degrades gracefully in both
methods (yields (None, ...), treated as "no data this tick") -- no other
behavior change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* MIGRATION_TO_LAUNCHER_FWDLLM.md: record Part 7 (max_runtime_s starvation fix)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* PR_CLEANUP_PLAN.md/MIGRATION_TO_LAUNCHER_FWDLLM.md: reconcile PR-readiness state
PR_CLEANUP_PLAN.md predated the post-migration investigation (Parts 2-7 of
MIGRATION_TO_LAUNCHER_FWDLLM.md) and no longer reflected reality: 3 more
real bugs found and fixed since its last update (fluxtune deadlock, asyncfl
duplicate-contribution gap, max_runtime_s starvation), the planned soak run
never actually happened (no niid_label_clients run exists), and the
originally-planned diff base (dg-fork-main) turns out to include an
unrelated ~20-commit async_cifar10 launcher initiative this branch was
built on top of, not this PR's own content. Rewrote "what's left before
merge" to reflect actual current state, flagged a scope decision (does the
telemetry/analysis tooling work belong in this PR), and corrected the
recommended diff base (35f8d65, not dg-fork-main). Also noted an exposed
GitHub PAT in the origin remote URL, unrelated to this PR but surfaced
while checking git state.
MIGRATION_TO_LAUNCHER_FWDLLM.md: updated the top summary through Part 7,
added a cross-reference to PR_CLEANUP_PLAN.md, corrected the "100%
availability" premise question with real trace data, and recorded the
Part 7 GPU re-validation run as in-progress.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm: retire PR_CLEANUP_PLAN.md, confirm Part 7 fix on GPU, explain n=100 gap
Part 7's max_runtime_s starvation fix is GPU-confirmed: the fluxtune re-run
(run_20260701_234833_fluxtune_n10_smoke, same mobiperf_3st_50 trace that
exposed the bug) self-terminated within budget with no manual kill needed.
Part 8: investigated why fwdllm's own n=100 run (Part 4 experiment) reached
only ~46% accuracy/data_id 29 while fwdllm_plus/fluxtune both passed 80%+
accuracy in the same 1.5h. Root-caused via direct log evidence (not
assumed): the aggregator's live candidate pool never grew past 30 of 100
trainers for the entire run ("Total ends: 30" on every cycle), and
"hasn't received weights" fired 9,680 times. Traces to --min-initial-
trainers 95 gating startup at n=100, combined with fwdllm's round-cached
reselection (Part 3) having no mechanism to replace a trainer that's
unresponsive but not formally departed -- unlike fwdllm_plus's per-
iteration reselect or fluxtune's continuous async reselection. A real,
distinct, scale-dependent gap in fwdllm's own selection design, not one of
bugs #4-7 -- documented as a non-blocking follow-up per user direction, the
convergence soak run is bypassed in favor of this existing n=100 evidence.
PR_CLEANUP_PLAN.md is removed from the branch (not just deferred to post-
merge as it originally said) -- its still-relevant content (bug list,
known limitations, scope decisions) is folded into a new "Merge readiness"
section in this doc. DELETION_CANDIDATES.md updated to match (it's the one
thing from that cleanup that survives, per the original plan).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: fold fwdllm's migration lessons into MIGRATING_TO_LAUNCHER.md, trim fwdllm doc to pending items only
MIGRATION_TO_LAUNCHER_FWDLLM.md had grown to ~1500 lines of investigation
narrative (evidence trails, exact log lines, commit-by-commit history) --
too big to read or maintain, and most of it was already resolved. Split:
- Durable, generalizable lessons (positive and negative) folded into
MIGRATING_TO_LAUNCHER.md's core sections: §2 gains aggregator gotchas
(unbounded recv_fifo blocking, selector reclaim ordering/completeness,
per-cycle dedup guards, cache-vs-selector state drift); §5 gains the
per-example telemetry manifest mechanism (progress_key()/
PROGRESS_AXIS_LABEL, model_param_count, event_categories) plus telemetry
gotchas (trainer-side selector artifact, subclasses must wire their own
telemetry.emit(), NO DATA placeholders looking populated at a glance);
§8 gains launcher/spawner gotchas (PYTHONPATH control, concurrent
trainer-wait polling, the await_join() race, sync/async compose()-branch
drift). §9 (fwdllm-specific) gains fwdllm's own specifics: run_sequential.sh's
controlled-comparison flags, the round-cached-vs-per-iteration reselection
design, the recv_fifo fix's concrete instance, and the telemetry manifest's
actual values.
- MIGRATION_TO_LAUNCHER_FWDLLM.md rewritten from scratch as a short pending-
items tracker (2 correctness/design follow-ups, 3 telemetry/analysis
improvements, 3 sanity checks) -- nothing blocking merge. Full history
stays in `git log` for that file, as the doc itself already pointed to.
Swept ~15 code comments across flame/selector/, flame/mode/, scripts/analysis/,
and various tests that referenced now-removed "Part N"/"P5.x" sections,
redirecting them to where the content actually landed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_aggregator: evict a round-cache member that's stuck, not just departed
The per-round reselection cache (_round_selected_ends, reselect_each_
iteration=False) only ever pruned an explicitly departed trainer
(disconnected or UN_AVL). A trainer that's still formally connected but
silently stuck -- e.g. never finished receiving its initial weights --
occupied a cache slot indefinitely, since nothing else in this design
could tell it apart from a trainer that's just slow.
Confirmed as the root cause of a real n=100 run's stall: the aggregator's
working set stayed capped at 30 of 100 trainers for the entire 90-minute
run ("hasn't received weights" fired ~9,700 times), ending at only ~46%
accuracy vs. fwdllm_plus/fluxtune's 80%+ in the same window -- those two
don't share this gap since they reselect continuously instead of caching
per round.
Fix: track _round_cache_activity_ts per cached end (stamped on cache entry,
reset on every real accepted contribution in
_process_single_trainer_message); _prune_departed_from_round_cache now also
evicts a member that's gone ROUND_CACHE_STUCK_TIMEOUT_S (5 min) without one,
same timeout-based reclaim pattern as SEND_TIMEOUT_WAIT_S/RECV_TIMEOUT_WAIT_S
used elsewhere in this codebase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* run_sequential.sh: add --avail-traces for unattended multi-trace overnight runs
--avail-trace only applies one trace to the whole --only baseline sequence
per invocation. Add --avail-traces (comma-separated) to loop the entire
sequence once per trace, all within one invocation/process -- for an
unattended overnight comparison (e.g. syn_0 then syn_20) without needing to
babysit and launch the next trace by hand.
Each (baseline, trace) run's name/job-id/log file are disambiguated by
trace when looping multiple (e.g. fwdllm_n100_syn_0_smoke vs.
fwdllm_n100_syn_20_smoke) so results never collide; single-trace/no-trace
invocations keep today's exact naming. --stop-on-fail now aborts the
remaining runs across all traces, not just the current one.
Dry-run verified against the real checked-in YAMLs (patch_yaml's python
heredoc extracted and run standalone, not an actual launch).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fwdllm_trainer: fix uncaught KeyError on end-of-training broadcast
_fetch_weights logged msg[MessageType.DATA_ID]/msg[MessageType.ITERATION_PER_DATA_ID]
unconditionally, a few lines above the code's own existence guard for those
same keys. The aggregator's end-of-training broadcast only carries
{MessageType.EOT: ...}, so any trainer still waiting in _fetch_weights when
the aggregator wrapped up hit an uncaught KeyError and crashed instead of
reaching the EOT handling that sets self._work_done and exits cleanly.
Confirmed across three n=100 overnight runs (2026-07-02): every
idle/unselected trainer (82-90 per run) crashed with this exact KeyError at
the run's stop timestamp. Harmless to run correctness (those trainers were
already about to be swept by the launcher), but every run ended with dozens
of uncaught-exception tracebacks instead of clean shutdowns.
Fix: switch the log line to msg.get(...), matching the guarded-access
pattern already used a few lines below for the analogous WEIGHTS log line.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: confirm fwdllm merge-readiness via 2026-07-02 n=100 runs, trim fwdllm doc to a redirect
Three fresh 2h n=100 runs (fwdllm, fwdllm_plus, fluxtune, all syn_0) confirm
the round-cache-stuck fix holds under real GPU load: 0 "hasn't received
weights" messages (vs ~9,700 before), steady progress to data_id 135/150,
accuracy climbing 30%->67%, using the full max_runtime_s budget instead of
stalling. Telemetry manifests clean on all three; no unhandled exceptions
after the fwdllm_trainer.py KeyError fix in the previous commit.
MIGRATING_TO_LAUNCHER.md: record the GPU-validation result on the
round-cache fix, fold in the new KeyError bug+fix, fold in the two
still-open (non-blocking) telemetry-improvement ideas that used to live in
the fwdllm-only doc, update the status table.
MIGRATION_TO_LAUNCHER_FWDLLM.md: no fwdllm-specific blockers remain, so
trim it to a short status note + redirect to MIGRATING_TO_LAUNCHER.md,
per the established fold-in-then-trim pattern (see d10bb53). Full
investigation history stays recoverable via git log on this path.
DELETION_CANDIDATES.md: fix a stale cross-reference to a "Merge readiness"
section that no longer exists in the fwdllm doc.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Dhruv Garg <dhruv.s.garg@hotmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu>
Co-authored-by: Gaurav Dadlaney <gaurav18n@gmail.com>
…ifar10) + fwdllm sim design (#69) * Sim unavailability: lock v1 scope (oracular-for-all) + file-level impl spec Rewrite UNAVAILABILITY_DESIGN.md to fold in the Jun25 design resolutions: - v1 scope locked: aggregator reads the shared trace (oracular) for ALL baselines, client_notify OFF. The old per-baseline oracle-vs-notify asymmetry collapses; aware vs unaware now differ only in WHEN a stalled slot is freed (aware: proactively at next selection boundary; unaware: reactively at the 90s abandon deadline). True avl_* transport + continuous mid-round scheduling deferred to Stage H. - Withhold-then-deliver corrected to a send-time gate, not a mid-compute interrupt: compute always completes, only the upload is gated, delivered stale at delivery_ts=max(sct,next_avail_ts). Real-side send-gate change documented; sim commits via the agg-side buffer. - 90s abandon + withhold are both true = two ledgers (slot vs delivery); the existing SEND/RECV_TIMEOUT_WAIT_S must re-clock to _vclock.now. - Three correctness invariants (no double-count; never re-select a still-down trainer; aware/unaware = trigger only). - Library mixin seam: flame/availability/{trace.py,availability_mixin.py} mixed into all four TopAggregators, consolidating the 3 duplicate read_trainer_unavailability; spans async_cifar10 + fwdllm. - New section 8: file-level v1 implementation spec (Stage A + C). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage A: trace resolver + AvailabilityMixin substrate New files: - flame/availability/trace.py: load_trace / state_at / next_avail_after (single bisect_right resolver; lru_cache for YAML files; replaces three inlined bisect_right copies) - flame/availability/availability_mixin.py: AvailabilityMixin with _init_availability, _avail_now (vclock in sim), read_trainer_unavailability, get_curr_unavail_trainers, free_stalled_slot (dormant hook for Stage C/D/H) Deletions (consolidation): - read_trainer_unavailability: removed from main_oort_sync_agg.py, main_asyncfl_agg.py, fwdllm_aggregator.py (all three dup copies) - get_curr_unavail_trainers: removed from syncfl/top_aggregator.py body (wall-clock impl) and main_oort_sync_agg.py (wall-time-in-sim bug); mixin provides the single vclock-correct version Wiring: - AvailabilityMixin mixed into syncfl TopAggregator → all four stacks inherit - _init_availability called from internal_init; sim_unavailability=False (default) ⇒ trainer_event_dict=None ⇒ byte-identical to all existing runs - flame/config.py: sim_unavailability, availability_aware, availability_trace_dir fields - Supports both new sim_unavailability gate and legacy track_trainer_avail path 389/389 unit tests pass. Syn_0 regression smoke pending on training node. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * stage A tests and wip stage B * stage B complete * Sim unavailability Stage C: delivery-ledger substrate + resume plan Land the reusable C.2/C.4/C.5 core (gated/dormant until the live commit loop calls it; default OFF => byte-identical): - AvailabilityMixin.compute_delivery_ts: earliest t>=sct with state AVL_* (immediate if already up at sct; inf if the trace never recovers). - free_stalled_slot activated: frees the selector slot ledger AND registers pending_withheld[end]=delivery_ts (two ledgers, Challenge 4 / invariant 1). - withheld_held_ends / ready_withheld / commit_withheld delivery-ledger helpers (ready_withheld orders by (delivery_ts, end_id) -> no past-dating). - invariant 2 wired: withheld_held_ends() unioned into the unavailable list in oort + asyncfl _distribute_weights so a held trainer stays out of the eligible pool until its delivery_ts. - asyncfl __init__: _sim_withheld_payload store added (consumed by C.2 next). 10 new unit tests (tests/availability/test_delivery_ledger.py); full suite 420/420. Detailed C.2/C.3 live-wiring resume plan added to UNAVAILABILITY_DESIGN.md (RESUME HERE subsection). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Sim unavailability Stage C: live commit-loop wiring (C.2/C.3) + parity rungs Wire the send-gate withhold / late re-commit (C.2) and the 90s vclock abandon (C.3) into both sim commit loops, as ONE shared library-level core in AvailabilityMixin so the asyncfl (felix/fedbuff) and oort (oort/refl) stacks ride identical logic instead of forking it: * _sim_withhold_if_unavail — per-update send-gate primitive (invariant-1 stash for an already-abandoned end; else hold if state_at(sct)=UN_AVL: stash payload, free_stalled_slot, register the delivery ledger). Pure per-update decision — does not consult round_start, which is why oort applies its carry-over gate first (Challenge 7). * _sim_pop_committable / _sim_reinject_ready_withheld — asyncfl pop loop + due re-injection at delivery_ts (ordered), slot-only ledger drop. * _sim_abandon_stalled — vclock 90s abandon over the selector slot ledger; _avail_free_slot_ledger / _avail_inflight_ends are robust to both selector shapes (fedbuff dict-by-requester + all_selected; oort/refl/feddance flat set). * _sim_take_withheld_delivering / _emit_withheld_delivery — late-stale-commit telemetry; asyncfl tags the "withheld" past-dating bucket. Call sites: asyncfl/_sim_recv_min pop site; oort/_sim_drain_buffer pop loop (composed after the §4.9 carry-over gate); both _distribute_weights run the abandon scan. All helpers are getattr-guarded ⇒ gate-off byte-identity holds for bare aggregators that never ran _init_availability. Run activation: trace-name normalization (avl_events_syn_20 → syn_20) in flame/availability/trace.py — the legacy oort/refl JSON configs resolved to 0 traces / silent-OFF before. Confirmed end-to-end: 300 traces load, 34/300 UN_AVL mid-window, compiled config carries enabled/type/trace. Telemetry: withheld_delivery + abandon_timeout events/builders (events.py). Parity rungs (checks.py, wired into run_all_parity + CHECK_META): * withheld_delivery (DIAG) — structural: delivery_ts≥sct, delay_s≥0, staleness≥0; reports delay/staleness dist + accept_frac. * abandon_timeout (CONTROL) — fails loudly on a wall-clock leak. * eligible_pool_reduction (DIAG) — candidates−eligible across modes. Loaders surface the new agg events + trainer avail_change; duty_cycle reads the real {old_state,new_state} format (A4 bug). observation_lag + A4b held (need run data — see §7). Tests: 436/436 lib (16 new in tests/availability/test_live_wiring.py) + 38 parity (14 new in scripts/parity/test_availability_rungs.py). Next: oort syn_20 smoke (sim+real, --runtime-s 1800 to clear the 600s down window) before felix/fedbuff or Stage D — see UNAVAILABILITY_DESIGN.md NEXT ACTION. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * wip fixes, context saved in unavail_design.md * Sim unavailability Stage C.6 complete + Stage D.1 boundary eviction wired C.6.4: add trainer_state_fractions_sorted.pdf (per-trainer UN_AVL sorted bar, A4dur visual companion) to analyze_run.py availability_plots. Syntax clean, crash-safe. Visual correctness needs a fresh syn_20 run. D.0: felix gate plumbing in parity yaml — sim_unavailability/availability_aware/ client_notify added to both felix entries so _init_availability activates for felix on syn_20 runs (client_notify.trace overridden by --trace flag). D.1: _sim_evict_unavail_inflight in AvailabilityMixin — proactive boundary eviction for availability_aware baselines: at each selection boundary, any in-flight trainer showing UN_AVL per the trace has its slot freed immediately (no 90s wait). Called from oort and asyncfl top_aggregators after _sim_abandon_stalled; guarded by _availability_aware so oort/refl (unaware) stay on the 90s-abandon path. build_abandon_timeout gains reason field to distinguish C.3 from D.1 events. 8 new unit tests; 444/444 total. Next: syn_0 byte-identity smoke (felix+oort) then felix syn_20 smoke to validate D.1 eviction fires and C.6.4 plots populate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Speed up smoke-test debug runs: parallel trainer shutdown + --num-trainers wait_all() waited up to 30s per straggler trainer sequentially, adding minutes of idle wait when several trainers got stuck post-aggregator-exit; now all trainers share one timeout window polled concurrently. Also add debug_run.sh --num-trainers to shrink the cohort below 300 for runs that need a real --runtime-s budget (e.g. a vclock floor for an availability trace) that the existing smoke mode's hardcoded rounds=4/runtime=240 would cut short. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sim unavailability: real send-gate, avl_state telemetry fix, D.2 task gating Lands the four implementation items from UNAVAILABILITY_DESIGN.md's Next actions: (1) real-side send-time gate (§8.3) — trainer task-start no longer skips on avl_state (compute always completes), and syncfl/trainer.py's existing send-time UN_AVL check is decoupled from client_notify["enabled"] (always False in v1) so it actually fires for real felix/oort; (2) fixes avail_composition/avl_state staying all-UNKNOWN by stamping PROP_AVL_STATE from the oracular trace onto every known end before each selection; (3) fixes withheld_delivery under-emission caused by the delivery ledger dropping a slot-only entry before its payload physically arrived; (4) D.2 AVL_TRAIN/ AVL_EVAL task-type eligibility gating in selection, wired into oort + asyncfl. 452/452 lib + 63/63 example-parity tests pass (was 444/444). Real/sim parity run validation still pending (see Next actions). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix D.2 hang: guard eval task-type exclusion against 2-state traces D.2's AVL_TRAIN-exclusion-from-eval rule made eval's eligible pool permanently empty on a 2-state trace (syn_0/syn_20/syn_50 never produce AVL_EVAL), which corrupted the selector's shared in-flight tracking for the train task too (async_oort.py/fedbuff.py _handle_send_state's disconnect-cleanup loop wipes selected_ends when handed an empty pool). Found via a real felix syn_0 smoke that hung with 0 AGG_RECV_WEIGHTS despite all 48 trainers training and sending. _init_availability now computes _trace_has_avl_eval once (does this trace ever produce AVL_EVAL for any trainer); the eval-side exclusion in get_curr_task_ineligible_trainers only applies when true, matching the design doc's own "2-state collapses the split" statement. Train-side exclusion is left unguarded (no 3-state trace exists yet to exercise the symmetric risk) and documented as a residual risk in UNAVAILABILITY_DESIGN.md (Challenge 13). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: crisp down UNAVAILABILITY_DESIGN.md + add syn_0 confirmation 838 → 452 lines. Completed stages compressed (mechanism + where it lives + exit met). Updated status: D.2 hang fix CONFIRMED via Jun 28 syn_0 runs (felix+oort, both complete 4 FL rounds, all avail rungs PASS, abandon/withheld correctly SKIP). Removed run-unconfirmed tags now folded into "pending syn_20" which is in progress. Added D.2 empty-pool hang to §9 dead-ends. Dropped verbose per-item prose for landed stages; kept the invariants and land-mines that bite if forgotten. parity_felix.json/parity_oort.json from Jun 28 syn_0 regression runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Parity fixes + oort syn_20 analysis: NEAR_ZERO_LAG, phase guard, new rungs Three parity check issues found and fixed during syn_20 validation: 1. Raise NEAR_ZERO_LAG_S 0.05→0.10s (U6): carry-over burst after avail windows inflates sim mean to ~70ms — still immediate-commit semantically. 2. Add point-mass guard in trainer_phase_split: both means ≤5ms → skip KS, pass on mean (pre_train_s, weights_to_gpu_s, weights_to_ram_s near-zero). 3. Wire four new avail rungs into report.py _SECTIONS: A4dur, Aa (eligible_pool_reduction), C.3 (abandon_timeout), C.2 (withheld_delivery). Fix abandon_timeout tier CONTROL→INV so _TIER_TAG renders correctly. UNAVAILABILITY_DESIGN.md: add §9.1 (oort parity settled diagnosis — K3b run-length, T2 pre-existing, A2 KS shape artifact from avail-window bimodal distribution; none block Stage E). Reorder Next Actions: Stage E first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: optimized batched plan for E/F/G Collapse remaining v1 work (E feddance, F starvation, G integration) into one code batch + one long-run batch. Non-conflicting code surfaces land together, gated on cheap signals (unit tests + syn_0 byte-identity + syn_20 feddance / syn_50 smokes). Share the F-validation syn_50 run with G.2's ramp rung; overlap the oort 3600s confirmation for free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Batch 1: syncfl E.1/E.2/E.3, Stage F vclock-advance, G.1 starvation rung Stage E (syncfl path — feddance + refl): - syncfl _distribute_weights: abandon/evict/stamp + scarcity-advance (calls _next_avail_vclock() and advances vclock instead of sleeping when no trainers selectable at round start) - syncfl _sync_sim_recv_first_k: withhold send-gate (_sim_withhold_if_unavail) + withheld bonus drain (_sim_reinject_ready_withheld + _emit_withheld_delivery) - syncfl internal_init: _sim_buffer (SimReorderBuffer) + _sim_committed init - E.2 = accept-stale (FedAvg has no staleness gate); E.3 = naturally handled (withheld updates excluded from committed, so U6 is over actual cohort) Stage F (starvation vclock-advance): - New _next_avail_vclock() mixin helper: min across all trace transitions + pending_withheld delivery timestamps; returns None when gate off - Wired into: (a) syncfl if-not-selected_ends path, (b) oort max_retries loop - Felix asyncfl (top_aggregator.py:639) gap noted; deferred to syn_50 observation G.1 (parity rungs): - starvation_advance_parity() in checks.py: DIAG rung, jump_factor=5x mean gap, gate = withheld_deliveries or abandon_timeouts or avail_composition subfield - Dispatched in Stage 2 Availability results dict - Added to report.py _SECTIONS section "2" as "Fst starvation_advance (diag)" - withheld_delivery deps chain updated to include abandon_timeout - 4 new starvation_advance unit tests; 67/67 parity tests pass (was 63) Felix master-gate (prereq): - debug_run.sh trace-override block now injects simUnavailability: True (+ availabilityAware: True if client_notify.enabled) for non-ORACULAR baselines when --trace syn_X is passed; oort/refl still use legacy path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: clarify ORACULAR naming, add baseline smoke rationale Terminology overhaul to eliminate the dual meaning of "ORACULAR": - "ORACULAR" in YAML/code now explicitly labelled as the legacy-gate config mechanism (field name in oort/refl YAML), not a knowledge model descriptor - v1 knowledge model renamed "trace-read" (agg reads trainer_event_dict directly) — applies to ALL baselines, including feddance - "aware/unaware" split renamed "proactive/reactive-90s" for slot-free timing (proactive = felix at next boundary; reactive-90s = oort/refl/feddance at 90s vclock deadline) - Stage H transport renamed "message-transport" (avl_* msgs) vs "trace-read" - Config-gate section now defines legacy-gate (oort/refl) vs simUnavail-gate (felix/feddance) and explains they activate the same trace-read code path - §0 existing paths table updated: path A = "trace-read", path C = "message-transport" - §8.4 config surface updated with mapping table Stage E smoke rationale added: feddance-only is sufficient because refl shares the same syncfl _distribute_weights/_sync_sim_recv_first_k code and its legacy-gate was already confirmed in Stage C (oort syn_20 run). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: oort 3600s syn_20 analysis + G.4 terminology-in-code todo oort n=300 3600s syn_20 results (Jun 28 Run 2: 40/48 PASS): - T2 training_budget: CONFIRMED self-corrected (ratio=1.133 ≤ 1.15) ✅ - K3b overhead_residual: did NOT self-correct as predicted (still rel=0.116, now gated downstream by P3); prediction revised - A2 eligibility: improving (KS 0.437→0.338) but still shape artifact; need more rounds (Batch 2 long run) - P3 trainer_speed: NEW marginal root cause at n=300 (ratio=1.153 vs tol 1.15, 1.3% over); clean at n=48; likely speed-tail noise at full cohort - Fst starvation_advance: PASS "no starvation advances detected" — correct (n=300 + syn_20 always leaves ~240 trainers available, Stage F never fires) G.4 added to Stage G: consolidate config-gate paths + rename legacy function `oracular_trainer_avail_check` → `_trace_read_avail_check` + update comments/logs to use new terminology (trace-read, proactive, reactive-90s, legacy-gate, simUnavail-gate). Deferred to after Batch 2 long runs pass. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: Stage E confirmed (feddance syn_20 1800s 46/47), Stage F in progress feddance syn_20 1800s parity check: 46/47 PASS. All availability and mechanism rungs pass (A1–A4, U3/U6/K8/U2). Sole fail = C2 loss (0.1696 vs 0.15, 2 eval pts, early-train noise at alpha=0.1/syn_20 — not a mechanism gap). Challenge 9 confirmed: no K8/U2 movement from withheld stale commits. Stage E exit met. Stage F syn_50 runs launched (feddance + oort, 45 min each, real+sim). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Stage F.2: unified pre-selection starvation gate + doc overhaul F.2 replaces the broken per-aggregator retry mechanisms with a single pre-selection return-early pattern across all three aggregators: - oort: was max_retries=5 + min_required=5 (50% of agg_goal) + "proceed anyway" fallback + 2s blocking sleep. Now: threshold=desired_selection (=int(aggr_num×overcommitment)=13), unbounded, non-blocking (0.5s + return). - syncfl (feddance/refl): was post-selection `if not selected_ends:` which FedDanceSelector's partial returns (3/10 eligible) never triggered. Now: pre-selection threshold=agg_goal=10. - asyncfl (felix/fedbuff): was time.sleep(0.5) in both modes at no-recv-ends path. Now: sim advances _next_avail_vclock(); real keeps 0.5s sleep. All three use the same pattern: num_eligible < threshold → sim vclock-advance + re-stamp + return; real: 0.5s sleep + return. Outer run() loop retries non-blocking. No max_retries ceiling; no "proceed anyway" fallback. At n=10 syn_50: min_avail≈3 at t≈1200s → eligible=3 < desired_selection=13 (oort) and < agg_goal=10 (feddance) → both starvation paths now exercisable. UNAVAILABILITY_DESIGN.md: 746 → 248 lines. Completed stages compressed to 2-3 lines each. Full detail kept only for active (Stage F exit) and next (Batch 2). Known parity failures moved to a single reference table (§7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Design doc: syn_0 regression passed; syn_50 n=10 starvation smoke in progress syn_0 results (n=48): Fst PASS (no starvation fires), C1/C2 diff=0.0, K1/K5 PASS on both baselines. Oort 43/50 (K3b pre-existing gates K2/K3); feddance 46/48 (A2 shape artifact + gpu_compute short-run noise, K2/K3/P3 PASS). F.2 threshold check confirmed not firing spuriously at n=48 with syn_0. n=10 syn_50 starvation smoke launched; results to check in morning. Next actions updated to morning check commands. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix syn_50 feddance sim deadlock + improve crash diagnostics; update doc debug_run.sh: simUnavailability was never injected for feddance (parity config has tracking_mode:client_notify at aggregator level but no HP-level tracking block, so neither the trackTrainerAvail nor client_notify branch fired). Result: trainer_event_dict=None → vclock deadlock at 0.0 for full 5400s. Fix: add third elif branch for non-oracular baselines with no HP tracking block, injecting availability_trace + simUnavailability=True. Also add simUnavailability to the client_notify-in-HP branch (felix path). runner.py: aggregator exit code was not logged, making silent crashes (e.g. feddance real crash at 12s) undiagnosable. Log exit code after wait(); also set PYTHONFAULTHANDLER=1 in child env so C-level crashes (segfault in paho/torch) write a traceback to _aggregator.log. UNAVAILABILITY_DESIGN.md: update status (n=10 bugs root-caused), replace next-actions with n=25 1800s run command + corrected rationale. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix real-mode recv deadlock + rename max_runtime_s → max_experiment_runtime_s Real-mode oort aggregator could block indefinitely in recv_fifo when syn_50 trainers went unavailable mid-round: no timeout meant max_runtime_s check in inc_round() was never reached, causing the process to run until manually killed. Fix: add trainer_recv_wall_timeout_s HP (default 90s, YAML-configurable) as a per-message wall-clock timeout on recv_fifo in real mode. On timeout recv_fifo yields None → progressed=False → while loop breaks → aggregator commits what it has and proceeds to inc_round() where the budget check fires. Also capped at remaining wall budget (min with max_experiment_runtime_s) so the process never overshoots. Sim mode unaffected (_recv_timeout=None; _oort_sim_recv/drain_ready is non-blocking). Rename max_runtime_s → max_experiment_runtime_s across all 41 source files (YAMLs, scripts, lib code, docs). The new name makes the experiment-level scope clear; comments document the dual-clock behavior (wall in real mode, vclock in sim mode). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * debug_run.sh: add 30s progress ticker + experiment count display Background loop prints elapsed/remaining/percent every 30s while run_node is running; tracks experiments-started via new run_* dir count. Budget passed as n_exps × runtime_s (upper bound; sim faster). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(availability): real-mode recv-barrier timeout + Challenge 13 cleanup + oort cohort floor Three correctness fixes for sim-unavailability under scarcity, plus design doc update. 1. syncfl real-mode aggregate recv had NO timeout: recv_fifo(first_k=agg_goal) blocked forever when unavailable trainers withhold their uploads (feddance n=12 real hung ~46 min on a 30 min budget, 0 rounds, killed only by the runner watchdog). Bound it with min(trainer_recv_wall_timeout_s=90s, remaining budget), mirroring oort/asyncfl which already had this. Only the syncfl base lacked it, which is why oort completed and feddance hung. (Challenge 16 / B2.0.1) 2. Challenge 13: _handle_send_state "invalid prior selection" cleanup keyed off the availability-filtered eligible pool, so an in-flight trainer that merely went UN_AVL (or wrong task-type) was dropped from selected_ends; an empty per-task pool wiped ALL shared in-flight tracking -> hang. Added connected_ends param (full connected pool) for the cleanup membership check in async_oort/async_random/fedbuff; new-candidate selection still uses the eligible pool. +3 regression tests (TestChallenge13SendStateCleanup). 3. oort cohort-floor guardrail: clamp the starvation threshold to min(desired_selection, connected) + one-time [COHORT_FLOOR] warning, so an undersized cohort (n < desired_selection) can't silently degenerate into an all-starvation, budget-exhausting run. Tests: 449 passed; 7 pre-existing failures (test_sync_sim_ordering / test_sim_barrier fixtures missing _sim_buffer, fail identically on clean HEAD); parity ladder 24/24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(availability): fix 7 stale sync-aggregator fixtures; suite now fully green The 7 long-standing failures in test_sync_sim_ordering (x6) and test_sim_barrier (x1) were stale fixtures, not product bugs. They __new__ an aggregator (bypassing __init__ + _init_availability), so the availability state the E/F stages added was never set: - _sync_sim_recv_first_k references self._sim_buffer directly (set in __init__) - the AvailabilityMixin helpers expect trainer_event_dict / pending_withheld - the oort real-recv timeout reads self.config.hyperparameters Fix: give the fixtures the gate-OFF availability state (_sim_buffer=SimReorderBuffer(), trainer_event_dict=None, pending_withheld={}, a minimal _HP with trainer_recv_wall_timeout_s / max_experiment_runtime_s). Byte-identical to a real gate-off run, so the sim-ordering / stale-reject logic is still exercised in isolation. Result: 456 passed / 0 failed / 7 skipped + parity ladder 24/24. Doc updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * latest unavail markdown file * feat(availability): T0-T5 pre-flight — starvation fix, two-axis flags, 6-baseline scaffold, 79 tests T0 (blocking): B2.0.2 starvation self-termination fixed in syncfl/oort/asyncfl. - Starvation else-branch sets _work_done=True at trace horizon or budget. - Budget check changed from > to >= so vclock==budget stops the run. - Real-mode wall-budget guard added to the scarcity sleep path. - 10 regression tests in test_starvation_termination.py. T1: Two-axis flag split (_availability_aware → avail_select_filter + proactive_inflight_evict). - avail_select_filter gates get_curr_task_ineligible_trainers (selection pool filter). - proactive_inflight_evict gates _sim_evict_unavail_inflight (felix only). - [ORACULAR] → [TRACE_READ]; oracular_trainer_avail_check → _trace_read_avail_check. - Legacy fallback comment clarified: getattr fallback is dead (pydantic default=None means attribute always exists; bool(None)=False is the correct safe default). T2: Per-baseline flags set in parity YAML to match canonical baseline matrix. felix: both ON. oort/fedbuff: both OFF. refl/feddance/oort_star: filter ON, evict OFF. T3: oort_star + fedbuff scaffolded (baselines.yaml + parity YAML sim+real entries, 12 total). - fedbuff lrDecay HPs added to match felix and all other async cifar10 runs. - debug_run.sh smoke default updated to all 6 baselines. - debug_run.sh proactiveInflightEvict auto-detect comment clarified (always dead; Stage H future; explicit YAML values are authoritative). T4: 41-test state-fidelity suite (test_state_fidelity.py): T-state-exact, T-eval-pool, T-withhold-deliver, T-aware-vs-reactive, T-starvation-sync. T5 pre-work: A5 state_timeline_agreement wired into checks.py + report.py (DIST tier, 0.95 tol). 28-test config-wiring suite (test_baseline_wiring.py): all 6 baselines, catches oort_star-missing class of bug. oort_star added to baselines.yaml. Tests: 536 pass / 7 skip. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(smoke): add smoke_suite.sh — timeout-guarded overnight smoke campaign debug_run.sh: LOGDIR is now env-overridable via FLAME_LOGDIR so each per-run invocation from the suite writes its Python agg output to an isolated directory instead of the shared /tmp/debug_run_logs. smoke_suite.sh: new parent orchestrator that runs 5 ordered steps (pytest → syn_0 sim → syn_20 sim → syn_20 both → syn_50 starvation) with individual per-(baseline, mode) wall-clock timeouts: - Each run launched via setsid so the whole process tree (launcher + 300 trainers) shares a session group; timeout kills via SIGTERM → 20s grace → SIGKILL on the group PGID. - Per-run grep checks on debug_run.out: stopping_run count (must be >0), SIM_WALL_CEILING count (must be 0), SIM_STARVATION count (informational). - Status: PASS / FAIL(no_stop) / FAIL(wall_ceil) / ERROR / TIMEOUT / SKIP. - Final report printed to stdout and written to <output-dir>/report.txt. - --dry-run shows every command without launching anything. - Configurable per-step runtimes (--runtime-syn{0,20,50}-s) and timeout buffer (--timeout-buffer-s); --steps to run a subset. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): replace setsid with set -m for reliable process-tree kill setsid forks when the calling process is already a process group leader (which interactive terminals guarantee for every background job). This makes runner_pid point to the dead parent that setsid discards, so kill -TERM -$runner_pid was targeting a defunct PGID — the entire stuck process tree survived as orphans. Verified with a 3-level (bash → launcher → trainers) simulated stuck runner: 4 orphan sleep 9999 processes confirmed after the setsid kill. Fix: use `set -m` (job control) immediately before the background launch and `set +m` immediately after. With job control active, bash assigns PGID = runner_pid to the job unconditionally, in both interactive and non-interactive contexts. Since flame/launch/spawner.py uses plain subprocess.Popen with no start_new_session or preexec_fn=os.setsid, all 300 trainer processes inherit the same PGID and are killed by kill -TERM/-KILL -$runner_pid. Verified: 6-process tree (runner, ticker, launcher, 3 trainers) all dead after kill, PGID=runner_pid in both interactive (bash -c) and non-interactive contexts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): clean termination + correct aggregator log grep Three fixes to _run_baseline: 1. Post-kill cleanup (timeout path) When SIGKILL fires, run_experiment_batch's finally block (_sweep_stragglers: pkill trainers/agg + 45s GPU wait) is in the killed process group and may not complete. Replicate it in bash: pkill -9 -f "trainer/pytorch/main.py" pkill -9 -f "aggregator/pytorch/main_" sleep $KILL_SETTLE_S # default 20s GPU memory drain so no stale trainer/aggregator processes or held GPU memory bleeds into the next run's allocation. 2. wait() order fixed Added wait after SIGKILL on timeout path; non-timeout path uses conditional wait so we never double-wait the same PID. 3. Grep checks scan the correct file AggregatorSpawner redirects the aggregator subprocess's stdout/stderr to experiments/run_*/..._aggregator.log (NOT $FLAME_LOGDIR/debug_run.out, which is only run_experiment.py's own print()s). Previous grep always returned 0 for "stopping run" / SIM_WALL_CEILING / SIM_STARVATION, causing every successful run to be classified as FAIL(no_stop). Fix: use -newer $ts_marker to find aggregator logs created during this run's lifetime; accumulate counts across them (handles multi-exp batches). Paths saved to $run_dir/agg_logs.txt for easy post-mortem access. Also adds --kill-settle-s CLI option and KILL_SETTLE_S=20 default. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(unavail): add T5-smoke section with 6h command + run count + hypotheses 23 jobs / 22 FL experiments (steps 1,2,4,5; step 3 skipped as redundant with step 4 sim half). Expected ~4h 38min wall, fits 6h with ~1h margin. Documents per-baseline predictions, starvation-at-n300 note (SIM_STARVATION absent is correct, not a failure), and failure watch-list for oort_star/fedbuff first live runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): grep-c double-output crash + in-place progress ticker - Fix arithmetic crash: grep -c always prints a count even for 0 matches and exits 1, causing || echo 0 to fire and produce "0\n0"; $(( N + 0\n0 )) is a bash syntax error that aborted _run_baseline before _record, silently dropping all runs after the first baseline. Fix: || echo 0 → || true. - Add per-run in-place ticker (stderr, \r): shows elapsed/budget/pct within the current run plus kill-in countdown and overall N/total done count. Overwrites in place each 5s tick — lets you spot stuck runs without log flood. - Add _compute_total_runs + TOTAL_RUNS/COMPLETED_RUNS globals; _record now increments COMPLETED_RUNS so the ticker stays accurate across all steps. - Move COMPLETED_RUNS increment into _record so pytest (step 1) also counts. - Drop nohup from the design-doc command (tmux keeps the session alive). - Expand T5-smoke hypotheses into a 23-row per-run table with Actual column. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(smoke): step4 ETA log used invalid ${#var//pat/} substitution Bash treats ${#var//x/y} as a parse error ("bad substitution"), which is fatal regardless of set -e/-u — it killed the whole suite right at the step 4 header, before any step 4/5 runs executed. Replace with wc -w to count baselines. * feat(availability): stamp real-mode vclock_now (A4dur fix) + rename ClientAvailability + B2.0.3 join-barrier re-anchor vclock_now was hardcoded None for real-mode selection telemetry, so A4dur's duration-weighted real/sim comparison fell back to a join-ramp-skewed origin; now stamped via ClientAvailability._avail_now() for both modes. Renamed AvailabilityMixin -> ClientAvailability (availability_mixin.py -> client_availability.py). Also fixes B2.0.3: agg_start_time_ts was stamped before the trainer join barrier, so real's trace-read clock baked in the join wait (~300s at n=300), reading the availability trace ~300s ahead of sim/ground-truth. Root-caused via feddance syn_50's A3 (avail_timebase) failure against the trace's own ground truth. Fixed via _mark_join_barrier_done(), which re-anchors agg_start_time_ts to real time once the join barrier resolves (real mode only, self-correcting to actual join duration) -- one fix point in syncfl/top_aggregator.py covers all six baselines via inheritance. Confirmed A4dur fix live on a fresh felix syn_20 real+sim pair (mean_err=0.0). B2.0.3 fix confirmation (fresh feddance syn_50 pair) still pending. 6 new regression tests in test_join_barrier_reanchor.py. Full suite green: 542 pass / 7 skip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --background flag for unattended overnight runs Re-execs the suite via nohup+disown and returns immediately, printing the PID, nohup log, and eventual report.txt path. Lets a smoke campaign survive the launching shell/SSH session closing, so it can be fired off and checked on later rather than needing a session held open for hours. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass -- A4dur confirmed, B2.0.3 root-caused+fixed, next-step command Whole-doc crisp pass per the Working Agreement: Status, T5-smoke, Challenges (#17 B2.0.2 was still marked open, now closed; new #18 for B2.0.3), and the known-parity-failures table all updated to reflect the A4dur confirmation and the new B2.0.3 root-cause + fix. Collapsed the old SCRATCH handoff section down to the still-open next steps (trackTrainerAvail cleanup, mobiperf live exercise, PR workflow) and dropped Q&A material now folded into permanent sections. Added a prominent NEXT STEP callout at the top of the doc with the exact unattended smoke_suite.sh command to confirm both fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(smoke): add --num-trainers to shrink the cohort below 300 Forwards --num-trainers through to every debug_run.sh invocation (debug_run.sh already auto-scales min_trainers_to_start accordingly). Lets a confirmation run use a smaller cohort under resource constraints without editing the parity config -- useful for mechanism-level fixes (e.g. B2.0.3's join-barrier re-anchor) that are n-independent by construction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): scope next-step confirmation run to n=100 Resource-constrained follow-up: A4dur and B2.0.3 are both mechanism-level fixes (already confirmed n-independent / self-correcting to actual join duration), so n=100 is sufficient to confirm them without reproducing the original n=300 failure magnitude. Notes the tradeoff -- not directly comparable to the existing n=300 parity_*.json snapshots for the still-open §7 checks, which stay gated on a real n=300 T5 run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(config): explicit encoding="utf-8" on YAML config reads (node-portability) open() without an explicit encoding falls back to the node's locale-preferred encoding. On a non-UTF-8 locale (e.g. C/POSIX without Python's UTF-8 mode coercion) this mis-decodes non-ASCII bytes in the parity YAML (an arrow character in a comment) byte-by-byte, and yaml.safe_load then rejects the resulting control characters with a ReaderError -- reproduced exactly with PYTHONUTF8=0 LC_ALL=C + encoding="latin-1". Fixed the crashing read (debug_run.sh's parity-config generator) plus the write/re-read round trip, and hardened the other YAML config reads in the availability substrate (trace.py's synthetic/mobiperf trace files, client_availability.py's trainer registry) against the same failure mode before it bites on a node with non-UTF-8 content in those files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(unavail): Batch 3 trace-fidelity & delay-enforcement overhaul (T3.0-T3.5) Adds absolute (vs. raw ground-truth trace) availability fidelity checks to complement the existing real-vs-sim relative checks, which is how the real-mode send-gate went dead code for the whole project without any check catching it (root-caused as T3.1a). - T3.0/T3.1a/T3.1b: shared trainer<->aggregator time origin (real mode), the debug_run.sh trace-wiring fix (trainer's own trace was never substituted), _refresh_avl_state() mode-dispatch cleanup. - T3.2: A6 trainer_trace_fidelity -- trainer's own avail_change telemetry vs. raw ground-truth trace. New scripts/parity/ground_truth.py; sim_now field added to avail_change. - T3.3: A7 agg_belief_fidelity -- aggregator's belief vs. ground truth, both selection and commit checkpoints. New agg_belief_change telemetry + ClientAvailability hooks. - T3.4: A8 send_gate_wait_fidelity -- observed vs. ground-truth-expected [SEND_GATE] wait duration. New send_gate_wait_s/send_gate_sct fields on task_send telemetry. - T3.5: K11 commit_promptness -- actual commit time vs. delivery_ts for withheld-then-delivered updates. New actual_commit_ts field on withheld_delivery telemetry. Each check ships with new telemetry, a parity rung in checks.py, a plot in analyze_run.py, and synthetic-data unit tests. Batch 3 is now code+test complete; UNAVAILABILITY_DESIGN.md's own plan calls for exactly one real run next (Phase 5) to generate live telemetry for all four new checks at once. Tests: lib/python/tests/ 574 passed / 7 skipped; examples/async_cifar10 scripts/parity/ + trainer/pytorch/ 121 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Batch 4 — felix real hang, sim-mode trainer clock freeze, A7 checker fix Phase 5/6's first real run against the new absolute fidelity checks (A6/A7/K11) surfaced three gaps; all three are root-caused, fixed, and unit-tested here: 1. felix real-mode TIMEOUT: D.1 proactive in-flight eviction (_sim_evict_unavail_inflight) was called only inside `if self.simulated:`, alongside the genuinely sim-only 90s vclock re-clock. But async_oort's selector derives `recv_ends` directly from `selected_ends`, so a stalled UN_AVL trainer never evicted in real mode never left recv_ends either, hanging the aggregator's 30s recv_fifo loop past its own budget. Un-nested D.1 from the sim-only gate in all three top_aggregator.py stacks (a no-op everywhere except felix, the only baseline with proactive_inflight_evict). 2. A6/A4dur/K6: sim-mode Trainer._sim_now() returns the frozen _sim_send_ts from its last dispatch, so a trainer withheld as UN_AVL can never observe later trace transitions while idle. Fixed in two zero-new-comms parts: (a) avail_change.sim_now now stamps the transition's own scheduled trace-time instead of _sim_now() at processing time; (b) inform_end_of_training's existing broadcast now piggybacks the aggregator's final vclock (sim mode only), and the trainer's shared _fetch_weights flushes one last _refresh_avl_state() catch-up on EOT. This also resolves the long-open K6 (`sim_send_ts==0`) mystery — same frozen-clock mechanism, not selector-starvation or a checker false positive. 3. A7 commit-checkpoint: not a system bug — the checker (_fidelity_score) extrapolated a trainer's last recorded commit-belief across the run's full span, wrongly scoring the silence after a trainer stops committing (typically because it went UN_AVL) as stale drift. Added extrapolate_tail=False for the commit-checkpoint call site, truncating the scoring window to [t_start, last observed t], symmetric with the existing start-side truncation. 12 new/updated tests (tests/ 586 pass/7 skip, scripts/parity/+trainer/pytorch/ 125 pass). Not yet confirmed on a live run — see UNAVAILABILITY_DESIGN.md's Batch 4 section for the re-confirmation command. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): ASCII-only comments in Batch 4 source files (node portability) * debug(unavail): temp diagnostic logging for D.1 eviction-miss mystery Live re-confirmation of the Batch 4 fix 1 (D.1 proactive eviction in real mode) found the fix is necessary but not sufficient: felix real still hangs at n=100, stuck at one round for 10+ minutes with only a handful of the ~20 stalled trainers ever getting evicted. Two isolated reproductions against the real AsyncOortSelector/ClientAvailability code (single stalled trainer, and 40/100 simultaneously UN_AVL) both show the eviction mechanism working correctly every cycle, so whatever is different about the live run isn't reproduced by either repro yet. Adds an [EVICT_DEBUG] summary log line to _sim_evict_unavail_inflight (inflight count, evicted count, and per-skip-reason counts: still available / buffered / already committed-or-withheld / no trace) so the next live run shows which guard is actually firing instead of inferring it from AWARE_EVICT's absence. Log-only change, no behavior change. Remove once root-caused (see UNAVAILABILITY_DESIGN.md's "Open A" for tracking). Also documents Open B: a separate, pre-existing bug found while investigating A -- trace loading falls back to the shared syn_20 "pattern" for most trainers instead of their individually-assigned per_trainer entry (hand-verified on two trainers whose own traces shouldn't transition until t=13800s/t=34200s, but which actually transitioned at t=600s/1200s matching the fallback exactly). Not yet root-caused or fixed; deliberately investigating Open A first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): add PR-readiness checklist for fresh-context handoff Explicit top-to-bottom checklist (Open A eviction-miss root cause, Open B trace-fallback root cause, cleanup, re-confirmation run, then the pre-existing Open items 1-2) so a new session can pick up exactly where this one left off without re-deriving status from chat history. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): Open A/B — selector abandon leak, trainer trace mismatch Open A (felix real never self-stops): the real-mode 90s SEND_TIMEOUT_WAIT_S abandon in async_oort.py/fedbuff.py only cleared all_selected, never selected_ends. recv_ends derives from selected_ends, not all_selected, so a stalled end never left the recv_fifo wait set and max_experiment_runtime_s's self-stop check (gated on recv_ends being empty) was unreachable. Both selectors now also discard the end from selected_ends. Open B (trainers ignore their per-trainer trace): spawner.py's get_synthetic_trace never took a trainer_id, so every trainer process baked in the shared cohort-wide `pattern` instead of its own registry-assigned per_trainer trace, even though the aggregator's own belief-tracking already resolved per-trainer correctly via flame.availability.trace.load_trace. get_synthetic_trace now delegates to that same resolver when given a trainer_id. Trainer startup also logs an [AVAIL_TRACE] signature (trace_hash) so a shared hash across trainer_ids is visible directly in the logs. debug_run.sh's --trace now accepts a space-separated list (one experiment set per trace) to make cross-trace validation runs a single invocation. Tests: tests/selector/test_send_timeout_frees_selected_ends.py, tests/launch/test_config_generator.py::TestSyntheticTracePerTrainer, tests/launch/test_debug_run_trace_substitution.py::TestMultiTraceSubstitution — all verified to fail without their respective fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): strip non-ASCII chars from this session's additions Extends 994f8187's node-portability cleanup to the lines added in the last two commits (0514d427, 8d145865) that it predates: em-dash/en-dash/multiply/ approx/bullet glyphs in debug_run.sh comments and UNAVAILABILITY_DESIGN.md's NEXT STEP section, replaced with ASCII equivalents (--, -, x, ~). Scoped to those additions only, not the doc's older pre-existing prose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(unavail): fwdllm agg telemetry test used stale property key _FakeChannel.get_end_property() matched on the literal string "round_duration", but _process_aggregation_goal_met() actually looks up PROP_CLIENT_TASK_TRAIN_DURATION ("client_task_train_duration_s"). The fake channel silently returned None instead of the seeded duration, so trainer_speed_s/agg_observed_s always came back empty -- a fixture bug, not a production one (traced back to a pre-existing mismatch already present on dg-fork-main before this branch's rebase). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs(unavail): crisp pass on UNAVAILABILITY_DESIGN.md per its own house rules Condense fully-resolved sections (Batch 3 T3.0-T3.5, Phase 5/6 results, T5-smoke/A4dur/B2.0.3) to the doc's own stated format for completed stages -- mechanism + files + tests + exit, 2-3 lines -- dropping narrative/discovery play-by-play that's already preserved in commit history. Also updates the NEXT STEP run command to the actual planned first-pass confirmation run (1800s, syn_50 only, felix+fedbuff) ahead of the full 7h campaign. 1375 -> 945 lines, no content loss (full detail remains in git log for every referenced fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): trim verbose comments in client_availability.py + syncfl top_aggregator.py Comment-only cleanup, no behavior change (verified via targeted availability/ sim-ordering/sim-barrier test runs, 141 passed): - client_availability.py: removed the TEMP DIAGNOSTIC eviction-miss counters and [EVICT_DEBUG] log line in _sim_evict_unavail_inflight -- explicitly marked for removal once Open A was root-caused, which UNAVAILABILITY_ DESIGN.md now confirms. Trimmed several multi-paragraph docstrings (get_curr_task_ineligible_trainers, _avail_stamp_end_states, _emit_withheld_delivery, _sim_reinject_ready_withheld) down to the causal fact + pointer, dropping session-narrative framing. - syncfl/top_aggregator.py: trimmed _mark_join_barrier_done's docstring and the EOT-broadcast piggyback comment the same way. First batch of a broader pass across the dg-fork-main diff; the other top_aggregator.py variants (oort/asyncfl), analyze_run.py, and the test files still have similar cleanup pending. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(unavail): finish comment-cleanup pass across the dg-fork-main diff Comment-only, no behavior change (full suite green: 728 passed / 7 skipped, same as before this batch): - oort/top_aggregator.py: trimmed the trainer_recv_wall_timeout_s explainer and the cohort-floor guardrail comment; dropped an internal "Next actions §2" task-tracking reference (also in asyncfl/top_aggregator.py). - syncfl/trainer.py: trimmed the AGG_START_TS caching comment, the EOT avl_state catch-up comment, and the [SEND_GATE] block's docstring-length comment down to the causal fact. - async_oort.py: reworded the Challenge 13 cleanup comment to match the shorter version already used in fedbuff.py/async_random.py (all three carry the identical fix; no reason for one copy to be 3x longer). - message.py, examples/async_cifar10/trainer/pytorch/main.py, analyze_run.py: trimmed remaining multi-paragraph docstrings/comments down to mechanism + one-line rationale, dropping session/phase-tracking references ("Batch 3 T3.0", "Batch 4 finding 2") in favor of plain causal language -- that history lives in UNAVAILABILITY_DESIGN.md and git log, not duplicated inline. Completes the comment-cleanup pass across the highest-density files in the diff (client_availability.py + the 3 top_aggregator.py variants + trainer.py + analyze_run.py). Remaining lower-density files and test-file docstrings were left as-is -- most already follow the repo's convention of a short per-test docstring naming the specific regression it guards against, which is exactly the kind of comment worth keeping. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat(launch): decouple spawn cohort size from dataset-split selector The launcher tied "how many trainers to spawn" and "which <name>_alpha<a>_n<N>.yaml split to read" to a single num_trainers knob, so a 10-trainer smoke demanded a dedicated n10 split file and KeyError-ed when only n300/n50/n48 existed. - experiment_config: add trainer.split_num_trainers (defaults to num_trainers) to pick the split partition independently of cohort size. - runner: split lookup uses split_num_trainers or num_trainers; the spawn cohort (trainer_ids) still uses num_trainers. - debug_run.sh: --num-trainers now preserves the config's native split size so a shrunk cohort reads the existing n300 split; add --alpha flag to choose dirichlet_alpha (0.1/1.0/10.0/100.0) against that n300 split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(parity): bound A7 commit-checkpoint scoring by max_gap_s _fidelity_score held each sparse commit-checkpoint observation constant across interior gaps between commits, penalizing the absence of a mid-gap commit rather than any wrong belief. Add max_gap_s (wired to lag_tol_s at the commit call site): each observation vouches for its own state only up to max_gap_s past itself, on both sides; a transition with no covering window is excluded from both the TVD score and the missed/spurious diagnostic. Generalizes the earlier extrapolate_tail tail-truncation to interior gaps. Took felix 59/61 -> 62/62 on the live n=300 run. Tests: test_agg_belief_fidelity.py (interior-gap + wrong-belief-at-own-instant). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(sim-unavail): fwdllm sim design, extend PARITY.md, trim UNAVAILABILITY_DESIGN, drop generated artifacts - PARITY.md: generalize to a multi-example parity methodology; add §F fwdllm rung catalog (baseline matrix, modified rungs, variance-cadence / dynamic-K/C / forward-grad layers). - simulate_fwdllm.md: rewrite as the design-only build plan -- correct code anchors to flame-core fwdllm_aggregator.py, resolve the §C baseline matrix from the landed launcher configs, integrate unavailability as first-class, surface decisions D1-D4. - UNAVAILABILITY_DESIGN.md: trim 1027 -> 283 lines to open items + next-steps + durable reference (baseline matrix, flags, v1 decisions, land-mines, dead-ends); collapse landed history to pointers; add the parallel-branch validation note. - Remove generated, unreferenced parity_{felix,oort,feddance}.json (-2152 lines; already covered by .gitignore). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Dhruv Garg <dgarg39@fyodorov.cc.gatech.edu> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.
Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.
Type of Change
Checklist