diff --git a/lib/python/examples/_metadata/baselines.yaml b/lib/python/examples/_metadata/baselines.yaml index 6de499971..8ee6e05df 100644 --- a/lib/python/examples/_metadata/baselines.yaml +++ b/lib/python/examples/_metadata/baselines.yaml @@ -212,6 +212,29 @@ baselines: hyperparameters: use_oort_loss_fn: "True" + oort_star: + description: >- + oort_star = oort sync + FedAvg + aware-at-selection (avail_select_filter=True), + no proactive in-flight eviction (reactive-90s). Differs from oort only in + avail_select_filter; the experiment config_overrides set avail_select_filter=True + (oort sets it False there). Uses same oracular tracking path. + example: + aggregator_main: aggregator/pytorch/main_oort_sync_agg.py + aggregator: + selector: + sort: oort + kwargs: {} + optimizer: + sort: fedavg + kwargs: {} + hyperparameters: + trackTrainerAvail: + enabled: "True" + type: ORACULAR + trainer: + hyperparameters: + use_oort_loss_fn: "True" + fedbuff: description: FedBuff = fedbuff selector (uniform async) + fedbuff optimizer. Async stack. example: diff --git a/lib/python/examples/async_cifar10/PARITY.md b/lib/python/examples/async_cifar10/PARITY.md index 2cf0c8f60..bb07ce2a7 100644 --- a/lib/python/examples/async_cifar10/PARITY.md +++ b/lib/python/examples/async_cifar10/PARITY.md @@ -1,9 +1,16 @@ # Real / Sim Parity — Methodical Causal Ladder -Living doc for the async_cifar10 parity checker. Kept in sync with +Living doc for the parity checker. Kept in sync with `scripts/parity/checks.py` (checks), `scripts/parity/report.py` (stage grouping + verdict), and the pytest suite. **ONE `## Status` section, updated in place.** +**Scope: this is the shared parity methodology, not one example's.** The ladder (§1), roles/ +tiers/dependency-gating, workflow policy, run-length budget, and mechanism reference (§3) are +**example-agnostic**. The async_cifar10 rung catalog (§2) is the reference instance; a second +example appends its own rung catalog rather than forking the method. **fwdllm** (forward-gradient, +variance-gated dynamic-K FL) is catalogued in **§F** — its build plan lives in +`examples/fwdllm/simulate_fwdllm.md`, which references §F for rung definitions. + **Comparator — give two run dirs, get a report JSON:** ```bash cd lib/python/examples/async_cifar10 @@ -702,3 +709,103 @@ refl overriding. | S | refl exploitation | FIXED: was deterministic top-k; now fork's cut_off_util-weighted `np.random.choice` | | D7 | UCB temporal-uncertainty `time_stamp` | **FIXED (Jun 23, §S.temporal).** Reference Oort+REFL: `sc += sqrt(0.1·log(round)/time_stamp)`, `time_stamp=self.epoch` (agg round of last RECEIPT), init at registration → never None, always contributes, up-weights under-selected/slower-returning clients. flame bug: refl's term was DEAD (0/7513) — `refl_oort.select()` let it divide by a None `time_stamp`; oort used last-SELECTED (dispatch round) with a None→0 guard. **Fix:** `PROP_LAST_RETURNED_ROUND` stamped at every receipt (fresh+stale) in oort/top_aggregator = agg round; selector reads it, registration-init lazy to current round; both oort+refl. `enable_temporal` kwarg (default True; False = ablation only). The legacy last-SELECTED machinery (`_record_last_selected_round`, D5) is REMOVED from OortSelector (no baseline used it); D5's MODEL_VERSION value was for staleness, not this UCB term. felix AsyncOortSelector is a separate class — untouched. | | D8 | `pacer()` round_threshold adaptation | **FIXED (Jun 24, §S.pacer).** Reference (`oort.py:184-199`) makes TWO symmetric moves on the exploited-utility trend: FLAT `|Δ|≤0.1·last` → `round_threshold += pacer_delta`, SHARP `|Δ|≥5·last` → `round_threshold = max(pacer_delta, −pacer_delta)`, keyed on `training_round`. flame's base `OortSelector.pacer()` raised on ANY dip (`last > curr`) with NO decrease branch → monotonic ratchet to 100, noise-sensitive → sim/real `round_threshold` diverged & back-half-compounded (the oort §S.pacer root). **Fix:** faithful both-branch port keyed on the current round, `pacer_step>0` guard; `REFLOortSelector.pacer` override (already faithful) REMOVED so oort+refl share the base; `round_threshold` added to selection telemetry. Guard `TestPacerFidelity`. **Cross-checked vs the REFL fork too** (`third_party/REFL/thirdparty/oort/oort.py`:176-201, byte-identical pacer, only round_threshold=30 default differs). **felix `AsyncOortSelector` (separate class) had the SAME bug + fired the pacer on its eval hand off a stale round → ALSO fixed** (faithful two-branch, train-gated; `test_async_oort_pacer_faithful`); changes felix dynamics → re-validate its 46/46 next run. | + +--- + +## §F FwdLLM extension -- variance-gated dynamic-K (forward-gradient FL) + +The method above (§1-§5) is example-agnostic; this is fwdllm's rung catalog. Rungs not +redefined here are inherited from §2 unchanged. `[NEW]` = to implement. The fwdllm build plan +(staged implementation, files, exit criteria, design decisions) is +`examples/fwdllm/simulate_fwdllm.md`; this section is the rung reference it points at. + +### §F.1 How FwdLLM differs (drives every new rung) +- **Aggregates GRADIENTS (JVPs), not weights.** Trainers send forward-gradient estimates; the + aggregator accumulates them into `grad_pool` and applies a server-LR SGD step at commit. + Gradient **values** depend on real GPU compute (run for real in sim) -> mode-invariant given + identical input + perturbation seed. What differs across modes is **which** gradients arrive, + **in what order**, against **which model version** = clock + selection + ordering fidelity. + This is what makes the §2 ladder applicable to fwdllm at all. +- **Commit cadence is ENDOGENOUS (variance-gated dynamic-K).** At each `_agg_goal` (=K) boundary, + `aggregate()` computes `var`; `var <= var_threshold` -> **commit** (server step, eval, + `data_id += 1`, `model_version += 1`, clear `cached_v`); else **roll back**, push grads to + `cached_v`, `iteration_per_data_id += 1`, **retry the same data_id**; + `max_iterations_per_data_id` force-commits despite a failed variance gate. So + updates-per-`model_version` is a **random variable** of the gradient-variance trajectory -- the + `model_version` clock is not a fixed function of update count. A round is the outer loop over + data bins; `_round += 1` only when `data_id == total_data_bins` (=150). +- **Dynamic K and C.** `DynamicKCController.step(metrics)` may change K (`_agg_goal`) and C + (concurrency) from observed metrics (var-pass ratio, eligible-ends) after each agg-goal cycle. +- **Eval per variance-pass.** `eval_model()` runs on **every** committed data_id, not a fixed + schedule; its modeled delay must be stamped separately (a stale-eval `sct` past-dates the clock). +- **Progress axis is `data_id`** (committed variance passes), not raw update count -- all + throughput/terminal rungs re-key to `data_id`. + +> **Crux:** in async_cifar10 clock and commit-count are loosely coupled; in fwdllm the commit +> (model-version) cadence is a *feedback function of gradient variance over the accumulated pool*. +> The sim must reproduce not just **when** updates arrive (clock) but the **variance trajectory** +> gating each commit. Variance is mode-invariant **iff** the contributing set + order + +> model-version of gradients matches -- reducing fwdllm parity back to clock + selection + ordering +> parity, **plus** a new variance-cadence verification layer (§F.4). + +### §F.2 Baseline matrix (fwdllm) +Filled from the landed `examples/fwdllm/expt_scripts/*_n10_smoke.yaml`. Maps onto the +async_cifar10 availability taxonomy (`avail_select_filter` / `proactive_inflight_evict`, +Unavailability §Baseline matrix). + +| baseline | sync/async | selector | agg | tracking_mode / avail | reselection | K (agg_goal) | dynamic_kc | +|---|---|---|---|---|---|---|---| +| **fluxtune** | async | `async_oort` | fedbuff (+server LR, JVP) | `client_notify` (3-tier, mobiperf_3st_50) | -- | 3 | disabled (fixed K/C) | +| **fwdllm** | sync | `random` | fedavg | `default` (unaware) | per-round | 10 (=c; all selected required) | -- | +| **fwdllm_plus** | sync | `random` | fedavg | `oracular` (`_metadata`, mobiperf_2st) | per-iteration (`reselect_each_iteration=True`) | 2 | -- | + +Decides scope: Stage 3 oort rungs (Sx/Sd/S2) run only for **fluxtune** (`async_oort`); the two +`random`-selector baselines skip them. Stage 5 sync-barrier rungs run for **fwdllm**/**fwdllm_plus** +(sync); fluxtune streams per-message. `client_notify` (fluxtune) is async_cifar10's deferred Stage-H +tracking model -- see simulate_fwdllm.md decision D1. + +### §F.3 Modified rungs (async_cifar10 meaning -> fwdllm redefinition) +| ID | async_cifar10 | FwdLLM redefinition | +|---|---|---| +| **K3a** | K-th fastest async commit advance | advance of `model_version` **per committed data_id** (clock delta between successive **variance passes**) | +| **K3b** | overhead residual ~= 0 | same, measured on the **variance-pass** boundary | +| **K2** | model versions / vsec | **committed data_ids / vsec** (throughput of *successful* variance passes) | +| **U3** | version gap at commit | gap of each contributing gradient vs `model_version` at the cycle it lands -- spread across **multiple iterations per data_id** | +| **K8 / U2** | @ matched model_version | @ matched **data_id** (the meaningful progress axis) | + +### §F.4 New rungs -- the variance-cadence layer (the fwdllm prize) +Append-only, deps let the engine localize. Never fix an EMERGENT rung directly (§1): walk down to +the lowest variance-cadence rung whose inputs are matched. + +**Stage 6' -- Variance-gated aggregation cadence** (dep Stage 5 ordering + Stage 1 clock) + +| ID | Check | Role/Tier | Isolates | Dep | +|---|---|---|---|---| +| **V1** `[NEW]` | iterations-per-data_id dist (realized dynamic K) | MECHANISM/DIST | # accumulation cycles to pass variance => contributing set/order diverged | U5,U4 | +| **V2** `[NEW]` | per-cycle `var` trajectory (at each agg-goal) | MECHANISM/DIST | variance *signal* diverges with matched inputs => grad-pool composition/order differs | V1 | +| **V3** `[NEW]` | `cached_v` pool size over time | MECHANISM/DIAG | rollback/cache bookkeeping diverges | V1 | +| **V4** `[NEW]` | force-commit freq (`max_iterations_per_data_id` bypass rate) | MECHANISM/DIST | cap hit at a different rate => chronic variance divergence | V1 | +| **V5** `[NEW]` | variance-pass ratio per window | EMERGENT/DIST | rollup feeding DynamicKC | V1,V2 | + +**Stage 3' -- Dynamic K/C trajectory** (dep Stage 3 selection + V5) + +| ID | Check | Role/Tier | Isolates | Dep | +|---|---|---|---|---| +| **DK1** `[NEW]` | K (`_agg_goal`) trajectory | MECHANISM/DIST | DynamicKC sees different metrics => K diverges (feeds back into cadence) | V5 | +| **DK2** `[NEW]` | C (`dynamic_c`) trajectory | MECHANISM/DIST | concurrency target diverges | V5,S3/4 | +| **DK3** `[NEW]` | eligible-ends-count metric fed to policy | CONTROL/DIST | the policy *input* differs (fix input, not policy) | A2 | + +**Stage 7' -- Forward-gradient quality** + +| ID | Check | Role/Tier | Isolates | Dep | +|---|---|---|---|---| +| **G1** `[NEW]` | per-update grad/JVP norm or SNR dist | EMERGENT/DIST | grad *quality* diverges (should be ~mode-invariant; FAIL = perturbation seed/order leaked) | S2,T_gpu | +| **G2** `[NEW]` | grad_pool size at commit (realized contributions) | EMERGENT/DIST | rollup of V1 x K | V1,DK1 | + +> **Decomposition:** `K2`(throughput)x but `K3a`(per-pass advance)ok -> clock fine, commit count +> diverged -> walk to V1/V5. `V1`x + `V2`ok-given-matched-input -> the *inputs* to variance differ +> -> walk to U5/S2. `V2`x with V1 inputs matched -> a true grad-pool accumulation-order bug. +> **DynamicKC coupling:** validate DK3 (policy *input*) before DK1/DK2 -- a diverging input means +> fix the metric, not the policy (CONTROL before MECHANISM). `var_threshold` / +> `max_iterations_per_data_id` are **baseline-defining config knobs, not parity levers** -- a +> cadence gap is always an upstream set/order/clock divergence. diff --git a/lib/python/examples/async_cifar10/UNAVAILABILITY_DESIGN.md b/lib/python/examples/async_cifar10/UNAVAILABILITY_DESIGN.md index c89361578..10177ea23 100644 --- a/lib/python/examples/async_cifar10/UNAVAILABILITY_DESIGN.md +++ b/lib/python/examples/async_cifar10/UNAVAILABILITY_DESIGN.md @@ -1,318 +1,283 @@ -# Sim Unavailability — Design & Staged Plan +# Sim Unavailability -- Design & Remaining Work -**Status:** design settled (all open questions resolved Jun 25); implementation NOT started. -Pickup-ready reference for adding trainer **unavailability** to the high-fidelity async_cifar10 -sim — and the template for the same feature in fwdllm ([simulate_fwdllm.md](../fwdllm/simulate_fwdllm.md) §7). +Design + status for modeling client **unavailability** in the FLAME FL simulator. The mechanism is +built and landed across all six async_cifar10 baselines; **felix is confirmed at full parity**. This +doc now carries mostly **open items + next-steps**; landed history is collapsed to pointers (full detail +in git). The fwdllm port has its own plan in +[../fwdllm/simulate_fwdllm.md](../fwdllm/simulate_fwdllm.md) (parity rungs in +[PARITY.md §F](PARITY.md)). -**Prerequisites (read first):** [PARITY.md](PARITY.md) §1–§2 (the causal ladder, role/tier tags, -dependency gating) and its §3 mechanism reference (`_vclock`, §3.drain, §3.resid, §4.5/§4.9, §S.dur). -This feature extends that ladder; every new rung and mechanism below assumes that vocabulary. +--- -**Goal:** let trainers drop in/out of `AVL_TRAIN` / `AVL_EVAL` / `UN_AVL` per their traces inside -the sim, emitting correct client-side notifications on the **virtual clock**, without (a) breaking -the parity already won at 100% availability, (b) a per-tick MQTT broadcast storm, or (c) -frozen-clock deadlocks. +## Status (Jul 2) -- felix CONFIRMED, fedbuff blocked, cross-baseline validation pending + +- **felix: FULL PARITY, 62/62 enforced checks** (live n=300 syn_50, `--runtime-s 3150`). Clock/timebase, + availability composition, selection, dispatch/training phases, staleness, and outcomes all PASS; real + self-stops cleanly (`"stopping run"`, no external kill, no `SIM_WALL_CEILING`). This is the reference + baseline the project set out to validate -- essentially PR-ready on its own. +- **fedbuff: NOT yet trustworthy.** Its run from the same campaign was **confounded** by a concurrent + felix-real run: fedbuff-real's per-round pacing collapsed ~3 min after felix-real launched (5-10s/round + -> 40-80s+/round) while felix-real stayed steady. The two ran on **separate nodes** with flat RAM / 0% + GPU on fedbuff's node, so it is **not** local CPU/GPU/RAM contention -- leading suspects are a shared + MQTT broker under ~600 concurrent real trainer connections, a shared network path, or shared + storage/telemetry-write contention. Not yet root-caused. + +### Next steps (this workstream) +1. **fedbuff shared-resource investigation** -- root-cause what collapsed fedbuff-real's pacing when + felix-real started despite separate nodes (broker / network / storage). Do this before a blind re-run. +2. **fedbuff isolated re-run** (+ sim) once understood, or at minimum confirmed no other n=300 real run is + active anywhere in the shared environment during its window; re-check parity. +3. **PR scoping** -- felix is PR-ready (62/62, clean self-stop). Decide whether to scope the PR to felix + now and follow up on fedbuff, or hold for both per the standing "land baselines together" agreement. + +### Next steps (parallel branch -- can proceed independently, merge fixes back as they land) +Exhaustive cross-baseline validation is **not** a blocker for landing the substrate. On a parallel branch, +run the full parity campaign for **oort / oort_star / refl / feddance** across **syn_0 / syn_20 / syn_50 / +mobiperf**, sim+real, at n=300; resolve the still-open §7 rows; fixes merge back into `dg-fork-main` as +they come. See "Open items" for the ordered gates (legacy `trackTrainerAvail` cleanup, mobiperf live +exercise) and PARITY.md's run-length budget. --- -## 0. What already exists (design *with* the grain) +## Preamble -- what this is + +**Goal.** Model client *unavailability* (devices dropping offline mid-training) so a fast **simulated** +run (virtual clock, no real sleeps) reproduces what a **real** run (wall-clock, MQTT, true delays) does -- +**sim/real parity** -- for every baseline, **config-gated and default-OFF** (byte-identical to today when off). + +**What was built (v1).** A shared availability substrate (`flame/availability/trace.py` + +`ClientAvailability`) mixed into the syncfl base and inherited by asyncfl/oort, so all baselines share one +trace-read effect path: +- **Send-time gate, deliver-late-stale.** A trainer that goes UN_AVL mid-flight *keeps computing*; its + upload is gated at send-time (real) / buffered to `delivery_ts = max(sct, next_avail)` (sim) and + committed later as a stale update. Nothing is cancelled or dropped. +- **Two ledgers, never conflated.** Slot ledger (90s vclock *abandon* frees the in-flight slot) + delivery + ledger (`pending_withheld[end]=delivery_ts`, commits through the existing staleness gate). +- **Proactive in-flight eviction** -- **felix only** (the one fully-aware baseline): frees a slot the trace + shows UN_AVL at the next selection boundary, no 90s wait. +- **Starvation / vclock-advance under scarcity.** When the eligible pool is too small to start a round, sim + advances the vclock to the next availability transition instead of spinning (self-terminating, B2.0.2). +- **Absolute (vs. ground-truth-trace) fidelity checks** on top of the relative (real-vs-sim) ones: A6 + (trainer state), A7 (aggregator belief, per selection/commit checkpoint), A8 (send-gate wait), K11 + (commit promptness). Relative checks answer "do the two modes agree?"; these answer "is either one + *correct*?" -- shared `scripts/parity/ground_truth.py`, one canonical trainer<->aggregator time origin. +- **Parity ladder** (`scripts/parity/`) -- availability rungs A1/A3/A4/A4dur/A5/A6/A7/A8/K11, + withheld_delivery, abandon_timeout, starvation_advance, eligible_pool_reduction. + +**Two orthogonal axes per baseline (keep separate).** +1. **Knowledge at selection** (`avail_select_filter`): does the selector read the trace to avoid + *selecting* currently-UN_AVL trainers? aware = yes, unaware = select blind. +2. **In-flight slot-free timing** (`proactive_inflight_evict`): when a *dispatched* trainer goes UN_AVL + mid-round, free its slot at the next boundary (proactive, felix only) or wait the 90s vclock abandon + (reactive-90s, everyone else). Aware-at-selection != in-flight eviction. + +The knowledge *model* is **trace-read** for all v1 baselines; message-transport (`client_notify`) and +predictive models are **Stage H** (future). + +--- + +## Working agreement (standing -- read every session) + +1. **Common first, one baseline first.** Land shared/library changes once, drive a single reference + baseline; don't fan out until it behaves. +2. **Short runs to debug, long runs to confirm.** Gate on unit tests + syn_0 byte-identity + shortest + syn_20 smoke. Long runs confirm; never find first bugs. +3. **Local deterministic tests before runs.** Prefer a synthetic-trace pytest that exhibits the bug over a + long run that hunts for it. +4. **Keep this doc crisp.** Completed stages: mechanism + where it lives + exit (2-3 lines). Full detail + only for active/next; dead-ends in §6. +5. **No stale content.** The moment a section is superseded -- a bug fixed, a task landed, a prediction + resolved, a next-step taken -- collapse it to a 2-3 line note (or delete) **in the same edit**. +6. **Whole-doc crisp pass on every edit.** Re-read top to bottom and push down anything the new result also + supersedes. The top of the doc carries only **open issues** + **next steps**; run commands, wall-time + estimates, and per-run hypothesis tables for finished work belong in raw logs. + +--- -The tree already has **three** availability paths; the 100%-avail runs left them dormant -(`trainer_event_dict`/`trainer_unavail_durations` default `None`). Reconcile these, don't add a fourth. +## Baseline matrix (CANONICAL) + +| baseline | sync/async | agg base / entry | knowledge @ selection (`avail_select_filter`) | in-flight slot-free (`proactive_inflight_evict`) | config-gate (as run) | +|---|---|---|---|---|---| +| **felix** | **async** | `asyncfl` (<- syncfl) / `main_asyncfl_agg.py` | aware | **proactive** (felix only) | `simUnavailability` | +| **fedbuff** | **async** | `asyncfl` / `main_asyncfl_agg.py` | unaware | reactive-90s | `simUnavailability` | +| **oort** | **sync** | `oort/top_aggregator` / `main_oort_sync_agg.py` | unaware | reactive-90s | `simUnavailability`+ | +| **oort_star** | **sync** | `oort/top_aggregator` / `main_oort_sync_agg.py` | aware | reactive-90s | `simUnavailability`+ | +| **refl** | **sync** | `syncfl` FedAvg / `main_fedavg_agg.py` | aware | reactive-90s | `simUnavailability`+ | +| **feddance** | **sync** | `syncfl` FedAvg / `main_fedavg_agg.py` | aware | reactive-90s | `simUnavailability` | + ++ oort/oort_star/refl's `baselines.yaml` catalog entry still carries the legacy `trackTrainerAvail: +{enabled: True, type: ORACULAR}` block (pre-dates this project). In every `debug_run.sh --trace`-launched +run the substitution sets `simUnavailability=True` for them too, so they run the same modern path. The +legacy block only matters if the parity YAML is loaded *without* that substitution -- untested territory, +and the reason it isn't cleaned up yet (removing it risks silently disabling availability in that path). +See Open item 1. + +**Notes.** (1) `ClientAvailability` lives in `flame/availability/client_availability.py`, mixed into +`syncfl/top_aggregator.py` (`class TopAggregator(ClientAvailability, Role)`); asyncfl/oort extend it -- all +six share the substrate. (2) trace-read is v1; the knowledge model becomes message-transport / predictive +in Stage H, but the select-filter / in-flight-evict *behavior* is unchanged. (3) **felix is the only +baseline that de-selects an in-flight trainer** when it goes UN_AVL; the aware-at-selection-only baselines +still hit the 90s abandon for mid-round drop-offs. + +### Flag reference +- `avail_select_filter: bool` -- selector excludes currently-UN_AVL trainers from the **selection** pool + (`get_curr_task_ineligible_trainers`). ON: felix/oort_star/refl/feddance. OFF: oort/fedbuff. +- `proactive_inflight_evict: bool` -- gates `_sim_evict_unavail_inflight` (in-flight boundary eviction). + ON: **felix only**. OFF: everyone else (reactive-90s). +- `tracking_mode` -- knowledge-model axis: `trace_read` (v1, live) | `client_notify` (Stage H) | + `predictive` (future). Replaces the `oracular` value at concept/log level (YAML field *value* compat kept). -| # | Path | Where | Time-base | Drives | -|---|---|---|---|---| -| **A. Agg pull (event trace)** | `get_curr_unavail_trainers()` binary-searches each trainer's `trainer_event_dict` (SortedDict `ts→state`) | syncfl/oort/asyncfl top_agg | `_vclock.now` (sim) / `time.time()−agg_start` (real) | `channel.set_curr_unavailable_trainers` at selection | -| **B. Agg pull (duration windows)** | `oracular_trainer_avail_check(end)` tests `trainer_unavail_durations[end]` `(start,dur)` | `asyncfl/top_aggregator.py:1264` | same | per-pick veto | -| **C. Trainer push (notifications)** | `check_and_update_state_avl` pops trace events → `channel.update_trainer_state` → backend → `Channel.update_state` (evicts `UN_AVL` / `AVL_TRAIN→AVL_EVAL` from `selected_ends`) | `trainer/pytorch/main.py` + `channel.py:1056` | `_sim_now()` = last dispatch `_sim_send_ts` (sim) / wall (real) | MQTT message | +--- -**Two anchoring facts:** -1. **A/B already key on `_vclock.now` in sim** (comment at `asyncfl/top_aggregator.py:1264`: - *"wall-clock would barely advance vs the sim timeline, so every unavailability window would be - missed"*). Aggregator-pull on the virtual clock = the no-comms, deterministic, never-freezes path. -2. **C has a frozen-clock defect in sim:** `_sim_now()` returns `_sim_send_ts`, which only updates - when the trainer is *dispatched*. An unselected trainer never advances → never pops events → never - notifies; a trainer that goes `UN_AVL` can't be selected → can't advance → **stuck `UN_AVL` - forever**. So C cannot be the source of truth for *selection* in sim until this is fixed. +## v1 core decisions (resolved -- durable reference) + +- **Knowledge model:** trace-read for all; one shared trace + `state_at(trainer, vclock)` + one effect path. +- **Mid-flight UN_AVL = compute-completes, gate the send, deliver-late (stale).** Real: gate at send-time. + Sim: buffer at `delivery_ts = max(sct, next_avail_ts)`. +- **Two ledgers, never conflated:** slot ledger (frees `selected_ends`) + delivery ledger + (`pending_withheld`, commits stale through the staleness gate). Order commits by `(delivery_ts, end_id)`, + never `sct`. +- **Busy != unavailable != withheld** -- three distinct non-pool states. Never route busy->UN_AVL. +- **All availability time on the vclock in sim.** Never wall, never a frozen per-trainer clock. +- **Config-gated, default OFF** => byte-identical. `simUnavailability` is the gate for all 6 baselines as + run (see Baseline matrix + note). +- **availability state** (`AVL_TRAIN/AVL_EVAL/UN_AVL`) x **busy?** x **has in-flight update?** are + orthogonal. `syn_0/20/50` are 2-state (no AVL_EVAL); `_trace_has_avl_eval` guard collapses D.2 for them. --- -## 1. Core decisions (all resolved) - -### The source of truth is PER-BASELINE — the asymmetry IS the design -- **Availability-UNAWARE (oort, refl):** real trainers go unavailable and **do not notify**; the - oracular aggregator **reads the traces**. So **A/B are the FAITHFUL model here, not a shortcut** — - real oort has no notification channel. The oracle gates **new selection only** (Q-new-1): it never - proactively evicts an in-flight trainer (it has no way to know), it just waits for the withheld - update to arrive on return. -- **Availability-AWARE (felix, fluxtune):** the aggregator does **not** read traces; it is informed - by **trainer→agg notification events** that **must take effect immediately** at the aggregator and - factor into selection AND `selected_ends`. So **C is the faithful model here**, with the - frozen-clock defect fixed. -- **One trace + one `state_at(trainer, vclock)` resolver + one effect path** shared by both, so - "what the agg believes" and "what the trainer is" cannot disagree. The only per-baseline difference - is *how the agg learns* a transition (oracle read vs event) and *whether* it acts on it (`awareness` - flag). **agg awareness** (does it factor availability in?) is orthogonal to **how it learns**. - -### Stop-gap for the aware path (Q4) -Model the aware event as **instant oracular reflection at the agg** (agg reads the shared trace on -the vclock at the transition instant, zero lag — behaviorally equal to instant push for selection). -**End goal = the true `avl_*` trainer→agg message.** Architecture must keep the -state-resolution + effect logic identical so the future swap changes only *transport*, not *effect* -(extensibility is a hard requirement). Per-tick broadcast (agg pings everyone each step) is -**rejected** (comms-heavy, induces sub-optimal decisions). - -### Mid-flight unavailability = COMPLETE-then-WITHHOLD-then-DELIVER (not a lost update) -When a trainer goes `UN_AVL` mid-compute it **finishes local compute, withholds the result, and -delivers it (now stale) on return to an available state.** Delayed delivery, not cancellation: -- Sim holds the completed update at `sct` but gates *delivery* on the next `AVL_*` window: - **`delivery_ts = max(sct, next_avail_ts)`**; it commits as a **stale** contribution (feeds - Stage-5/6 staleness, not a lost-update path). -- **AWARE agg:** on the `UN_AVL` event, **frees the slot** (replacement selectable) AND **tracks the - pending withheld delivery separately** (two ledgers — see Challenge 4). -- **UNAWARE agg:** no replacement; just waits for the delayed delivery. -- Late withheld update on return (Q-new-2): **async aware (felix) = accept-stale** through the - baseline's existing staleness path; **sync (feddance) = reject if staleness exceeds the baseline's - (lower) tolerance** — reuse the existing threshold, do NOT invent a new scalar. -- **Distinct from "busy"** (PARITY.md dead-end: do NOT route busy→`UN_AVL`). Busy = `AVL_*` but - occupied (hold slot, returns on time). Withheld = result exists, delivery deferred. Separate states. - -### Timing, comms, determinism -- **Transitions are continuous / event-scheduled at the exact transition vclock**, reflected at the - aware agg **immediately (lag = 0 first cut)** — not sampled-at-selection (a mid-round transition - must take effect mid-round). The clock-advance/commit path must consult the **next transition time** - so it can't skip a mid-window change: clamp the advance to `min(next_sct, next_transition_ts)`. -- If lag is ever modeled it lives on the **vclock** (per the §S.dur lesson: selector-fed quantities - must be intrinsic, never wall-contaminated). -- Oracular-pull is deterministic given trace+clock. The event path stays deterministic too (first cut - = instant oracular reflection); the true-message path must order events by vclock with a defined - tie-break to preserve `SEED=1234` real+sim parity (Challenge 6). -- **Everything config-gated, default OFF** ⇒ byte-identical to today's 46/46 scoreboard. Per-baseline - `availability_aware: bool` + `availability_trace: ` + master `sim_unavailability` gate. - -### Trace representation -- Collapse to **one event-trace representation** (`AVL_TRAIN/AVL_EVAL/UN_AVL`, strictly more - expressive than duration-windows; derive windows if a path still needs them). **Prefer 3-state - traces**; 2-state ({avail, unavail}) is allowed but **limited-utility for aware baselines** - (felix/fluxtune act on the `AVL_TRAIN↔AVL_EVAL` task-type split a 2-state trace collapses; oort is - indifferent). Surface trace granularity in telemetry. -- Single-source the trace + resolver like `client_duration.py` was single-sourced in §S.dur; both the - trainer side (transitions/telemetry) and agg side (oracular driver) read the identical object. - Traces (`mobiperf_2st/3st`, `syn_0`=100%, `syn_20`, `syn_50`) cover all n=300, loaded identically - both sides (Q-new-3). +## Parity rungs (availability tier -- what exists) + +- **A1** `avail_composition` (per-state counts, binned). **A3** `trace_time_base_consistency` -- CONTROL + hard gate (dep K3). **A4** `per_trainer_duty_cycle`. **A4dur** duration-weighted TVD + (`mean_err<=0.05`, `frac_within_tol(0.10)>=0.95`). **A5** `state_timeline_agreement` -- per-(trainer,t) + exact match. All five are **relative** (real vs sim). +- **A6** `trainer_trace_fidelity`, **A7** `agg_belief_fidelity` (tagged `selection`/`commit`), **A8** + `send_gate_wait_fidelity` (real-mode only) -- **absolute** (vs. ground-truth trace), independently per + mode. A7 is mechanism-agnostic (trace_read now, client_notify/predictive later). +- **K11** `commit_promptness` (INV) -- per-event: actual commit vs. earliest-legally-committable time, + generic over gate reason; primary promptness gate. `withheld_delivery` / `abandon_timeout` stay as + secondary distributional diagnostics. +- **eligible_pool_reduction** (`Aa`, HELD), **observation_lag** (live in v1 trace_read via A7), + **starvation_advance** (vclock jumps under scarcity). Calibrate HELD rungs at mobiperf. +- **Ramp:** syn_0 -> syn_20 -> syn_50 -> mobiperf_*. --- -## 2. First-principles factors (the why behind each decision) - -- **F1 Clock authority.** One monotonic vclock owns "now"; every availability decision is indexed by - it, never wall, never a per-trainer frozen clock. The trace is **sim-seconds since run start**; sim - (vclock) and real (wall-elapsed) must index the SAME windows — this is parity rung **A3** (the REFL - HIGH-1 hazard), a CONTROL for the whole feature. -- **F2 Source of truth** — per-baseline (§1). Both models share one trace + one resolver + one effect. -- **F3 Event semantics (what a transition CAUSES).** `→UN_AVL` aware: excluded from selection AND - freed from `selected_ends`/`all_selected` (`channel.update_state`), in-flight update withheld not - discarded. `→UN_AVL` unaware: agg waits (no proactive free). `AVL_TRAIN→AVL_EVAL`: train-pool - removal, eval-eligible only. `→AVL_TRAIN`: re-enters pool + triggers withheld delivery. The driver - must produce these effects deterministically at the transition instant, not merely filter the next - selection. -- **F4 Withhold-then-deliver** (§1). Return-stage fates: on-time / straggler-hold / withheld-then- - delivered. NO permanent cancellation. `_sim_hold_busy_slots` and oort `pending_after`/carry-over - must keep a withheld end accounted until its delayed delivery commits. -- **F5 Comms** — unaware: zero (oracular pull); aware: bounded by # real transitions (instant - reflection first cut), per-tick broadcast rejected. -- **F6 Timing** — continuous/event-scheduled, lag 0 first cut (§1). -- **F7 Regression surface (won mechanisms).** §3.resid `_sim_hold_busy_slots` (aware frees slot + - tracks pending; unaware holds until delivery); §4.5 `pending_after` / §4.9 carry-over (withheld end - must NOT re-enter pool during the down window, but its delivery must still commit, stale); §S.pacer - /§S.dur/A2c selector inputs (fewer candidates move the `pref` percentile — expect A2/S2/A2c shift; - score only genuinely-eligible trainers); per-baseline return stages each gain a "trainer vanished" - branch. All config-gated ⇒ default-off keeps byte-identity. -- **F8 Determinism** — oracular-pull seed-stable; event path ordered by vclock (Challenge 6). -- **F9 Trace** (§1). -- **F10 Starvation.** `oort/top_aggregator` already has a `max_retries` wait-retry for too-few- - available; under real unavailability it fires. In sim the wait must **advance the vclock** (jump to - next availability event / next in-flight `delivery_ts`), not spin on wall (Stage F). +## Challenges / land-mines (durable -- consult before the fwdllm port) + +Most are resolved; the still-open ones (3, 15) matter for cross-baseline validation and the port. + +1. Ordering on `delivery_ts`, not `sct` -- resolved (U6/U3 validated). +2. A3 time-base drift -- resolved; hard CONTROL gate; 90s abandon re-clocked to vclock. +3. **A2 two-tolerance trap (OPEN, watch).** Bimodal sim vs smoother real -> KS shape artifact; means match; + improving with run length (0.437->0.338). Expect <=0.2 at n=300/3h -- confirm at T5. +4-14. Resolved: busy/unavail/withheld three ledgers (4); real send-gate fidelity (5); determinism via + `(delivery_ts,end_id)` (6); compound straggler x UN_AVL (7); AVL_EVAL inert for oort + `_trace_has_avl_eval` + guard (8); staleness-on-sync cohort movement (9); scarcity advance skips no events (10); syn_0 + byte-identity discipline (11); library mixin spans examples, never example-local (12); empty per-task + pool corrupting `selected_ends` -- fixed by keying cleanup off `connected_ends` in all 3 selectors, + **still needs live mobiperf_3st exercise** (13); scarcity threshold via F.2 unified pattern (14). +15. **Per-baseline in-flight accounting + scenario sizing (OPEN, per-baseline).** In-flight is NOT constant: + oort (sync, over-selects) `in_flight ~= overcommitment*agg_goal - completed`; felix/fedbuff (async) + concurrency-bound, can exceed agg_goal; refl/feddance (sync FedAvg) clear `selected_ends` each round, + feddance returns *partial* selections so `eligible ~= (1-unavail)*n`. Manage in-flight per baseline; do + NOT assume "sync has no in-flight term." syn_50 caps ~43% unavail, so feddance's straddle window is + narrow (n~19) -- size `n ~= threshold / (1-unavail_frac)`. +16-18. Resolved: real syncfl recv-barrier bounded `timeout=min(90s,budget)` (B2.0.1, 16); sim starvation + self-termination `>`->`>=` budget check (B2.0.2, 17); real-mode trace-clock join-ramp re-anchor + (B2.0.3, 18) -- correct, but was masking item 20. +19. Resolved (Batch 4): oort/felix `K6 sim_send_ts` -- sim `Trainer._sim_now()` froze at last dispatch; + fixed via due-ts stamping + EOT final wake-up. K6/A6 PASS on the felix n=300 run. +20. Resolved (Batch 3 T3.1a): `debug_run.sh` never wired the trainer's own `client_notify.trace`, so real + trainers ran their send-gate against always-available `syn_0` regardless of `--trace` (sim was + aggregator-driven, so unaffected). Fixed + regression-tested + (`tests/launch/test_debug_run_trace_substitution.py`). This was the true cause behind the feddance A3/K3b + symptoms earlier blamed on clock-origin. --- -## 3. Concepts to keep crisp (naming discipline, PARITY.md) +## Dead-ends (settled -- do not retry) -- **availability state** (`AVL_TRAIN/AVL_EVAL/UN_AVL`) × **busy/occupied?** × **has in-flight update?** - — three orthogonal axes, never conflate (the busy→UN_AVL dead-end). -- **transition instant** (vclock the state changes) vs **observation instant** (vclock the agg acts); - lag = observation − transition (0 if continuous). -- Return fates: on-time / straggler-hold / withheld-then-delivered (stale). No cancellation. -- **agg awareness** ⊥ **how it learns** (oracle read vs event message). -- `_sim_now()` must stop meaning "last dispatch ts" — availability reads the **global vclock** (or is - fully agg-driven so the trainer never needs `_sim_now` for it). +- **busy -> UN_AVL routing** -- three distinct states. +- **Frozen per-trainer clock** (`_sim_now()` = last-dispatch ts) -- stuck UN_AVL forever; read `_vclock.now`. +- **Wall-clock in sim** for selection gate / 90s abandon -- wall barely advances vs vclock. +- **Per-tick MQTT broadcast** -- comms storm; v1 = trace-read pull (zero comms). +- **Ordering withheld commits by `sct`** -- past-dating; order by `(delivery_ts, end_id)`. +- **Forking withhold/abandon per stack** -- single shared `ClientAvailability`. +- **A4 bare transition fraction** -- brittle in trace-read mode; replaced by A4dur + Aa. +- **D.2 excluding AVL_TRAIN from eval on 2-state traces** -- empty eval pool wiped `selected_ends`; fixed by + `_trace_has_avl_eval` guard. +- **Subtracting starvation vclock-jumps from the budget** -- breaks parity (real polls scarcity on wall + budget; sim vclock jump consumes virtual budget symmetrically). Size `--runtime-s` accordingly. --- -## 4. New parity rungs (Stage 2 = Availability; extend the ladder) - -Each new mechanism leaves the finest-grained check that localizes it (PARITY.md Growth rule). -- **A1 avail_composition** (exists, trivial at 100%): now match per-state counts over the run, binned. -- **A3 trace_time_base_consistency** `[NEW]` (CONTROL/DIST, dep K3): same trace → same windows both - modes. **Hard gate — do not read A1/A2/A4 until A3 passes** (the REFL HIGH-1 / Challenge 2 lesson). -- **A4 per_trainer_duty_cycle** `[NEW]` (MECHANISM/DIST, dep A3): on/off fraction per trainer matches. -- **transition_effect** `[NEW]`: counts of UN_AVL slot-frees (aware), withheld-then-delivered updates, - AVL_TRAIN→AVL_EVAL demotions; sim vs real. -- **withheld_delivery** `[NEW]`: dist of `delivery_ts − sct` (down-window delay) + resulting staleness - (cross-checks F4 against Stage-5/6 U3). -- **observation_lag** `[NEW]`: transition→effect lag (tests F6; **must be ≈0** first cut for aware). -- **eligible_pool_reduction** `[NEW]`: A2 (`num_eligible`) tracks real's reduction, not just at 100%. -- **Regression guard:** re-run the syn_0 90-min all-baseline parity with availability OFF → current - scoreboard byte-for-byte (config-gating proof). -- **Ramp:** `syn_0` (regression) → **`syn_20` (first validation target)** → `syn_50` → `mobiperf_*`. - Shortest run per effect (run-length budget table); reserve long runs for C1/C2. First mechanism pass - = 45-min "one rung" budget, 5-min smoke first. +## Known parity failures (non-blocking -- resolve at T5 with long-run data) + +Resolved this project (full detail in git / §7 history): A4dur (all baselines, real selection missing +`vclock_now` -> fixed, `mean_err=0.0`); feddance A3 (item 20); K6/A6/A7-commit/asyncfl-TIMEOUT (Batch 4, +confirmed on felix n=300). Still open, for the parallel-branch campaign: + +| Check | Baseline | Status | +|---|---|---| +| A2 `eligibility` KS | oort | 0.437->0.338 (1.5h->3h); FAILs @ syn_20 n=300 short. Bimodal-vs-smooth shape; means match. Investigate with A4dur. | +| A2 `eligibility` KS | feddance | FAILs @ syn_20 n=300 smoke; earlier "clears at n=300" (n=25) not confirmed -- re-open. | +| K3b `overhead_residual` | oort | rel~=0.116; run-length sensitive; P3 gates at n=300. | +| P3 `trainer_speed` | oort | ratio=1.153 (tol 1.15); marginal tail at n=300; gates K3b. | +| `throughput` | oort | FAILs @ syn_20+syn_50 n=300 smoke; baseline-specific, lower priority. | +| `avail_composition`/`commit_visibility`/`total_commits` | fedbuff | FAILs @ syn_20 n=300 smoke -- but see fedbuff confound (Status); re-check after isolated re-run. | +| C2 `loss` | feddance | avg_diff~=0.16 (few eval pts); early-training noise at alpha=0.1; K8/C1/utility PASS. | +| U5 `inter-arrival` rho | feddance | 0.659->0.381 (syn_20->50); watch at mobiperf. | --- -## 5. Staged implementation + testing plan - -One mechanism per stage, each gated by its own tests + a syn_0 byte-identity regression + (where it -changes dynamics) a short syn_20 run. All config-gated, default OFF. Build the **unaware** (oort, -pure pull) path before the **aware** (felix, event+eviction+withhold) path — strictly simpler, shares -the substrate. Context-free names (`_ts`/`_time_s`, `_round`). - -### Stage A — Substrate: one trace, one resolver, one clock (NO behavior change) -- **A.1** New `flame/availability/trace.py` (mirrors `client_duration.py`): `load_trace(trainer_id) → - SortedDict[ts→state]` and `state_at(trace, t) → TrainerAvailState` (the binary-search currently - inlined in `get_curr_unavail_trainers` / `oracular_trainer_avail_check` / trainer - `check_and_update_state_avl`). Replace all three call sites; collapse `trainer_unavail_durations` - onto the event-trace. -- **A.2** Config surface: per-baseline `availability_aware`, `availability_trace`, master - `sim_unavailability` (default False). `syn_0` ⇒ no UN_AVL events ⇒ inert even when on. -- **A.3** Fix `_sim_now()` frozen clock (F1/F8): trainer availability reads the **global vclock**, not - `_sim_send_ts`. Route: make availability fully agg-driven; agg stamps the current vclock onto every - message the trainer receives, trainer uses that as "now" for its notification telemetry. -- **A.4** Telemetry (on the vclock): `avail_change` (move off wall), `agg_observed_state` (per-trainer - belief at each selection), trace granularity per run. -- **Tests:** resolver determinism + parity with the old inlined searches; frozen-clock deadlock cannot - recur; `sim_unavailability=False` ⇒ byte-identical. **Exit:** syn_0 90-min all-baseline parity holds - the scoreboard byte-for-byte. - -### Stage B — A3 time-base CONTROL (gate for everything above it) -- **B.1** A3 `trace_time_base_consistency` (CONTROL/DIST, dep K3): resolved on/off windows align - between modes within tolerance. **B.2** A4 `per_trainer_duty_cycle` (dep A3). -- **Tests + a 5-min syn_20 smoke** to populate A3/A4 (checker-side, validates instantly vs stored - dirs). **Exit:** A3 PASS on a syn_20 smoke for oort. - -### Stage C — UNAWARE oracular-pull driver (oort, refl) -- **C.1** Activate `get_curr_unavail_trainers()` for oort/refl via the Stage-A resolver on - `_vclock.now` → `channel.set_curr_unavailable_trainers`. **Gates new selection only** (Q-new-1). -- **C.2** Withheld-then-deliver for the pull path: an in-flight trainer entering `UN_AVL` is **not** - evicted; its modeled update is held and delivered at `delivery_ts = max(sct, next_avail_ts)`, - committing **stale**. Held end must NOT re-enter the pool during the down window (extend §4.5 - `pending_after`: exclude on `vclock < delivery_ts`, not just `< sct`). -- **C.3** Ordering: drain/commit order must key on **`delivery_ts`** for held ends (a withheld delivery - has `delivery_ts > sct`), or past-dating reappears (Challenge 1). -- **Tests:** unaware never frees the in-flight slot; withheld delivers at `max(sct,next_avail)`, - commits stale; held end excluded until delivery; determinism. -- **Validation:** syn_20, 45-min, oort+refl. Read A1, A2 (eligible-pool reduction), A4, withheld-delay - dist + staleness (U3). **Exit:** A1/A2/A3/A4 PASS, no new past-dating (U6), K2/K3b hold vs a syn_20 - real reference. - -### Stage D — AWARE immediate-event driver (felix; fluxtune if in scope) -- **D.1** Instant reflection: at every selection AND at each transition vclock the agg recomputes - per-trainer state from the resolver (stop-gap for the push message); effect path identical to a real - `avl_*` message (future swap = transport only). -- **D.2** Continuous/event-scheduled timing (F6): advancing the vclock toward the next commit, clamp to - `min(next_sct, next_transition_ts)` so a mid-window `UN_AVL` takes effect before the in-flight `sct`. -- **D.3** UN_AVL eviction + reselect: free the slot from `selected_ends`/`all_selected` (reuse - `channel.update_state` logic), make a replacement selectable, AND register the withheld delivery - separately (`pending_withheld[end]=delivery_ts`). Reconcile with §3.resid `_sim_hold_busy_slots` — - slot freed *and* pending delivery tracked, no leak, no double-count (Challenge 4). -- **D.4** `AVL_TRAIN↔AVL_EVAL`: task-type eligibility change via the resolver; only meaningful where - the baseline dispatches eval (sync oort dispatches 0 — felix-relevant; flag inert baselines, - Challenge 8). -- **D.5** Late withheld update = accept-stale (async) through felix's existing staleness path (no new - accept path; Q-new-2 async branch). -- **Tests:** UN_AVL frees slot + tracks pending (no leak); replacement selectable; event-scheduled - clamp fires a mid-window transition before the straggler's sct; AVL_EVAL gates task type; late update - commits stale. -- **Validation:** syn_20, 45-min, felix. A1/A2/A3/A4, transition-effect counts vs real, observation_lag - ≈0, withheld-delivery, U3, U6. **Exit:** mechanism rungs PASS at syn_20; then a full-length run to - re-confirm felix C1/C2 under unavailability. - -### Stage E — SYNC baselines + staleness-gated rejection (feddance) -- **E.1** Apply C/D to the sync path (feddance is availability-aware via its predictor; barrier - re-selects the cohort each round). -- **E.2** Staleness rejection: a late withheld update exceeding feddance's tolerance is **dropped** - (baseline's existing rule, no new threshold). Changes round composition → expect K8/U2 movement; - validate it's faithful (Challenge 9). -- **E.3** §6.u6 barrier-anchor: unavailability changes which K form the barrier; the barrier-anchored - U6 lag must be computed over the *actually contributing* cohort. -- **Tests + syn_20 45-min feddance.** **Exit:** A-rungs + U3/U6 + K8 PASS. - -### Stage F — Starvation / clock-advance under scarcity (F10) -- **F.1** In the `max_retries` wait-retry, when no one is selectable, **advance the vclock to the next - availability event (or next in-flight `delivery_ts`)** rather than wall-sleeping. Clamp to the - nearest of {next transition, next `delivery_ts`, next `sct`}; guard against an all-unavailable window - spinning forever (Challenge 10). -- **Validation:** syn_50, 45-min (heavier unavailability triggers scarcity), all baselines. **Exit:** - no stalls; K1 monotone; round cadence faithful at syn_50. - -### Stage G — Ladder integration + ramp + sign-off -- **G.1** Land all new rungs in `scripts/parity/{checks.py,report.py}` with deps (A1 now enforced, A3, - A4, transition_effect, withheld_delivery, observation_lag, eligible_pool_reduction). Append-only. -- **G.2** Ramp: syn_0 → syn_20 → syn_50 → mobiperf_*. **G.3** Per-baseline sign-off, now *with* - availability. - -### Stage H (FUTURE) — true `avl_*` message transport -Swap the aware-path instant-oracular reflection for real trainer→agg `avl_*` messages, processed -immediately, **without changing the effect logic** (D.1 was built for this). Preserve determinism by -ordering events on the vclock with a defined tie-break (Challenge 6). +## Open items -- pick up in order + +1. **Legacy `trackTrainerAvail` cleanup (oort/oort_star/refl) -- flagged, NOT done.** Their `baselines.yaml` + entries still carry `trackTrainerAvail: {enabled: True, type: ORACULAR}`. In every `debug_run.sh + --trace` run this resolves to the same `simUnavailability=True` path (a substitution quirk, not a static + value). Do **not** just zero `enabled`/`type` -- that risks silently disabling availability for any + invocation not going through the substitution script. Safe path: (a) add `simUnavailability: true` + statically wherever their trace is set; (b) only then zero the legacy block; (c) verify via an actual + generation + short real run (no pytest coverage of `debug_run.sh`'s generator). +2. **Minimal mobiperf live exercise before "mature enough to port."** The entire mobiperf (3-state, + AVL_EVAL) path is untested live so far (only syn_0/20/50, which are 2-state and collapse AVL_EVAL away). + Challenge 13's fix explicitly still needs live exercise at mobiperf_3st. ~30-45 min for one async + (felix) + one sync (refl or oort_star) baseline exercises AVL_EVAL + the empty-pool cleanup live and runs + Batch 3's checks against a 3-state trace. +3. **PR workflow.** felix is confirmed (62/62). Gated for a felix+fedbuff PR on the fedbuff isolated re-run + + concurrent-run investigation (Status). The fwdllm learnings are now carried into + [../fwdllm/simulate_fwdllm.md](../fwdllm/simulate_fwdllm.md) and [PARITY.md §F](PARITY.md); the design + decisions kept (this doc's v1 decisions / land-mines) and rejected (Dead-ends) are the durable record. + +### Minimal bar before porting fwdllm (not a full T5 gate) +Don't gate the port on the full 3h x 6-baseline x 4-trace campaign (that's for paper-quality numbers). Bar: +(1) Batch 3 (T3.0-T3.5) at least through felix -- **done** (A4dur confirmed; item 20 fixed; absolute +fidelity checks live). (2) mobiperf_2st for one async + one sync baseline (~30-45 min) to exercise AVL_EVAL ++ Challenge 13's cleanup live for the first time. (3) The full syn_50/mobiperf 6-baseline sweep is +separable -- parallel branch, not a pre-port blocker. --- -## 6. Challenges / land-mines - -1. **Ordering must key on `delivery_ts`, not `sct`, for withheld updates** (high risk of - re-introducing past-dating). §3.drain's min-`sct` gate and §4.9 carry-over assume `commit order == - sct order`; a withheld delivery commits at `max(sct,next_avail) > sct`. Re-validate U6/U3 after C/D. -2. **A3 time-base drift is the silent killer.** If sim vclock and real wall-elapsed advance at - different rates, the same trace makes a trainer unavailable at different real moments → every higher - rung diverges and mislocalizes. Hard CONTROL gate; do not read A1/A2/A4 until A3 passes. -3. **Two-tolerance trap on the eligible pool (A2 vs S3/4).** With availability ON, - `eligible = candidates − in_flight − unavailable`; a small gap can fail A2's tight KS while S3/4 - in_flight passes. Decompose the channel first; don't chase A2 as a separate bug. -4. **Busy vs unavailable vs withheld = three distinct non-pool states** (slot-leak / N≈300 ramp). The - §3.resid dead-end (busy→UN_AVL) ramped in-flight to ~300. Aware UN_AVL **frees** the slot but still - **tracks** the pending delivery — slot accounting and delivery accounting are separate ledgers; - conflating them leaks slots or double-commits. -5. **Stop-gap fidelity to real felix may not be lag-free.** Real felix uses MQTT notifications with - real delivery+processing lag; the stop-gap models lag 0. If real's lag is non-trivial, - `observation_lag` parity (sim 0 vs real >0) diverges and selection contexts won't match. Measure - real's notification lag early; if material, model it on the vclock or accelerate Stage H (the §S.dur - lesson: don't let a wall lag contaminate a vclock-indexed decision). -6. **Determinism / event tie-break at a shared vclock instant.** Transition, commit, and selection - events can coincide. Define a total order (e.g. transitions < commits < selections, then by - trainer_id) so `SEED=1234` real+sim parity + exact rungs stay enforceable. -7. **Compound states with existing carry-over.** An oort §4.9 carried-over straggler that ALSO goes - UN_AVL, or a §3.resid held slot whose trainer flips AVL_TRAIN→AVL_EVAL, are real cases. Enumerate - the (avail_state × occupied × in-flight) cross-product and assert each cell in tests. -8. **AVL_EVAL may be inert for some baselines** (sync oort dispatches 0 eval). A 3-state trace's eval - windows then do nothing; report which baselines exercise the eval split (ties to the 2-state - limited-utility flag, F9). -9. **Staleness-rejection on sync changes round composition (feddance).** Dropping over-stale late - updates shifts K8/U2/round-count — expect movement, validate it's faithful, reuse the existing - threshold (no new scalar). -10. **Scarcity clock-advance must not stall or fast-forward past events** (F10/Stage F). Clamp the jump - to the nearest of {next transition, next `delivery_ts`, next `sct`} and terminate on an - all-unavailable window. Guard K1 monotone + the K5 failsafe ceiling. -11. **Regression discipline.** Every stage re-runs the syn_0 90-min all-baseline parity and must hold - the scoreboard byte-for-byte before its syn_20 validation counts. A stage perturbing another - baseline serializes (one baseline per run round). +## Stage H (future -- out of scope) + +Two independent knowledge-model upgrades, both replacing `trace_read` on the `tracking_mode` axis; the +effect logic (select-filter / in-flight-evict) is unchanged -- only how the agg learns state changes: +- **H.1 Message-transport** (`client_notify` ON for aware baselines): trainers push avl-state changes over + MQTT instead of the agg reading the trace + a continuous/event-scheduled vclock clamp. Re-measure + `observation_lag` (must be ~0) once live. (fwdllm's fluxtune baseline already configures `client_notify` + -- see simulate_fwdllm.md D1.) +- **H.2 Predictive**: a learned/heuristic availability model (no trace read or message push). Not designed yet. --- -## 7. Open follow-ups (note here as work lands) +## History (collapsed -- full detail in git) -- **Q-new-2 sync confirmation:** verify feddance's existing staleness threshold is the right rejection - gate on a real syn_20 run before E.2 (don't assume the async tolerance transfers). -- **Real notification lag (Challenge 5):** measure on a real felix run early; decides whether lag 0 is - admissible or Stage H must move up. -- *(Append new open questions/info needs here as the substrate lands — keep this the single ledger.)* +- **A-G, C.6, D, E, F.2, B2.0.x, T0-T5, Batch 2/3/4** all landed. Substrate + A3 time-base CONTROL, + send-gate/deliver-late, two-ledger, proactive evict (felix), syncfl path, starvation self-termination, + the absolute ground-truth fidelity checks (Batch 3 T3.0-T3.5: A6/A7/A8/K11 + `ground_truth.py` + shared + canonical time origin), and Batch 4's four gating fixes (asyncfl real self-stop, sim trainer-clock freeze + / K6 / A6, A7-commit checker `max_gap_s`) are all in code + tests, confirmed on the felix n=300 run. + Per-task mechanism/file/exit detail lives in commit history; the durable decisions are in "v1 core + decisions", "Challenges", and "Dead-ends" above. diff --git a/lib/python/examples/async_cifar10/aggregator/pytorch/main_asyncfl_agg.py b/lib/python/examples/async_cifar10/aggregator/pytorch/main_asyncfl_agg.py index bfa3d6312..f8a7739d2 100644 --- a/lib/python/examples/async_cifar10/aggregator/pytorch/main_asyncfl_agg.py +++ b/lib/python/examples/async_cifar10/aggregator/pytorch/main_asyncfl_agg.py @@ -119,25 +119,9 @@ def __init__( self.learning_rate = self.config.hyperparameters.learning_rate self.batch_size = self.config.hyperparameters.batch_size or 16 - self.track_trainer_avail = ( - self.config.hyperparameters.track_trainer_avail or None - ) self.reject_stale_updates = ( self.config.hyperparameters.reject_stale_updates or False ) - self.trainer_event_dict = None - if ( - self.track_trainer_avail["enabled"] - and self.track_trainer_avail["type"] == "ORACULAR" - ): - self.trainer_event_dict = self.read_trainer_unavailability( - self.track_trainer_avail["trace"] - ) - else: - print( - f"Did not read oracular trainer jsons. Enabled value: {self.track_trainer_avail['enabled']}, type: {self.track_trainer_avail['type']}, trace: {self.track_trainer_avail.get('trace', '')}" - ) - print("self.trainer_event_dict: ", self.trainer_event_dict) self.loss_list = [] @@ -155,18 +139,6 @@ def initialize(self): _os.path.join(_os.path.dirname(_os.path.abspath(__file__)), "..", "..", "data")) - def read_trainer_unavailability(self, trace=None) -> None: - """ - Read availability trace pattern from central trace file. - - Currently returns None to disable oracular pre-loading. - The selector can still access traces via availability_trace_file config. - - This allows the aggregator to work with dynamic trainer spawning - without hardcoding the number of expected trainers. - """ - return None - def load_data(self) -> None: """Load a test dataset.""" transform_test = transforms.Compose( diff --git a/lib/python/examples/async_cifar10/aggregator/pytorch/main_oort_sync_agg.py b/lib/python/examples/async_cifar10/aggregator/pytorch/main_oort_sync_agg.py index 46ddbd794..f733093b1 100644 --- a/lib/python/examples/async_cifar10/aggregator/pytorch/main_oort_sync_agg.py +++ b/lib/python/examples/async_cifar10/aggregator/pytorch/main_oort_sync_agg.py @@ -22,14 +22,12 @@ import logging import time -from pathlib import Path import torch import torch.nn as nn import torch.nn.functional as F import torchvision import torchvision.transforms as transforms -import yaml # wandb setup import wandb @@ -41,15 +39,6 @@ import sys as _sys, os as _os _sys.path.insert(0, _os.path.dirname(_os.path.abspath(__file__))) from oracle_utility import OracleInjectMixin # noqa: E402 -from sortedcontainers import SortedDict - - -_METADATA_DIR = (Path(__file__).resolve().parents[3] / "_metadata") -_TRACE_KEY_TO_MOBIPERF_SUB = { - "mobiperf_2st": "states_2st", - "mobiperf_3st_50": "states_3st_50", - "mobiperf_3st_75": "states_3st_75", -} def initialize_wandb(run_name=None): @@ -130,26 +119,6 @@ def __init__( self.learning_rate = self.config.hyperparameters.learning_rate self.batch_size = self.config.hyperparameters.batch_size or 16 - self.track_trainer_avail = ( - self.config.hyperparameters.track_trainer_avail or None - ) - self.trainer_event_dict = None - if ( - self.track_trainer_avail["enabled"] - and self.track_trainer_avail["type"] == "ORACULAR" - ): - self.trainer_event_dict = self.read_trainer_unavailability( - self.track_trainer_avail["trace"] - ) - else: - print( - "Did not read oracular trainer jsons. " - f"enabled={self.track_trainer_avail.get('enabled')}, " - f"type={self.track_trainer_avail.get('type')}, " - f"trace={self.track_trainer_avail.get('trace', '')}" - ) - print("self.trainer_event_dict: ", self.trainer_event_dict) - self.loss_list = [] # Use wandb logging if enabled @@ -166,63 +135,6 @@ def initialize(self): _os.path.join(_os.path.dirname(_os.path.abspath(__file__)), "..", "..", "data")) - # Initialize aggregator start time for oracular availability tracking - self.agg_start_time_ts = time.time() - logger.info(f"Aggregator initialized at timestamp: {self.agg_start_time_ts}") - - def read_trainer_unavailability(self, trace=None) -> dict: - """Build trainer_id -> SortedDict(timestamp -> state) for `trace`. - - Reads from the shared examples/_metadata/ bundle (registry + traces), - not from legacy per-trainer JSON files. - """ - logger.info(f"Reading trainer unavailability for trace: {trace}") - - registry_path = _METADATA_DIR / "trainer_registry.yaml" - with open(registry_path) as f: - registry = yaml.safe_load(f)["trainers"] - - if trace in _TRACE_KEY_TO_MOBIPERF_SUB: - sub = _TRACE_KEY_TO_MOBIPERF_SUB[trace] - with open(_METADATA_DIR / "availability_traces/mobiperf_traces.yaml") as f: - traces = yaml.safe_load(f)["traces"] - - def lookup(tk: str, trainer_id: int) -> list: - return traces[f"device_{trainer_id:03d}"][sub] - - elif trace and trace.startswith("syn_"): - with open(_METADATA_DIR / "availability_traces/synthetic_traces.yaml") as f: - syn = yaml.safe_load(f)["traces"] - if trace not in syn: - logger.warning(f"trace {trace!r} not found in synthetic_traces.yaml") - return None - entry = syn[trace] - per_trainer = entry.get("per_trainer", {}).get("n300", {}) - pattern = entry.get("pattern", []) - - def lookup(tk: str, trainer_id: int) -> list: - return per_trainer.get(tk) or pattern - - else: - logger.warning(f"unsupported trace name: {trace!r}") - return None - - trainer_events_dict = {} - for tk, meta in registry.items(): - trainer_id = meta["trainer_id"] - task_id = meta["task_id"] - events = lookup(tk, trainer_id) - state_dict = SortedDict() - for timestamp, state in events: - state_dict[timestamp] = state - trainer_events_dict[task_id] = state_dict - - logger.info( - f"Loaded availability traces for {len(trainer_events_dict)} trainers " - f"(trace={trace})" - ) - return trainer_events_dict - def load_data(self) -> None: """Load a test dataset.""" transform_test = transforms.Compose( @@ -295,41 +207,6 @@ def _job(): import threading threading.Thread(target=_job, daemon=True).start() - def get_curr_unavail_trainers(self) -> list: - """Return trainer IDs currently in UN_AVL state based on oracular traces.""" - curr_unavail_trainer_list = [] - - if self.trainer_event_dict is None: - return curr_unavail_trainer_list - - agg_time_since_start_s = time.time() - self.agg_start_time_ts - - for trainer_id, event_dict in list(self.trainer_event_dict.items()): - if not event_dict: - continue - - idx = event_dict.bisect_right(agg_time_since_start_s) - 1 - if idx >= 0: - most_recent_event = event_dict.peekitem(idx) - most_recent_event_ts = most_recent_event[0] - most_recent_event_state = most_recent_event[1] - - if most_recent_event_state == "UN_AVL": - curr_unavail_trainer_list.append(trainer_id) - elif most_recent_event_state == "AVL_TRAIN": - pass - else: - logger.warning( - f"Trainer {trainer_id} has unknown state: {most_recent_event_state}" - ) - - logger.info( - f"[ORACULAR] Current unavailable trainers: {len(curr_unavail_trainer_list)} " - f"out of {len(self.trainer_event_dict)} total @ time={agg_time_since_start_s:.1f}s" - ) - - return curr_unavail_trainer_list - def check_and_sleep(self) -> None: """Induce transient unavailability""" pass diff --git a/lib/python/examples/async_cifar10/docs/EXPERIMENT_felix_streaming.md b/lib/python/examples/async_cifar10/docs/EXPERIMENT_felix_streaming.md index 887e2d15d..00312a1ca 100644 --- a/lib/python/examples/async_cifar10/docs/EXPERIMENT_felix_streaming.md +++ b/lib/python/examples/async_cifar10/docs/EXPERIMENT_felix_streaming.md @@ -114,7 +114,7 @@ each round). Two readings: | Streaming | **uniform (phase 1)** | one `full_after_s=10800`. Staggered (per-client onset≤5400 s, ±50% span) = phase 2 | | Time mode | **simulated** | virtual-clock time-to-accuracy, deterministic, fast | | Target accuracy | **60%** | | -| Stop rule | **20 consecutive evals ≥ 60%** | resets on any dip; `rounds`/`max_runtime_s` safety cap | +| Stop rule | **20 consecutive evals ≥ 60%** | resets on any dip; `rounds`/`max_experiment_runtime_s` safety cap | | Eval cadence | every **10** rounds | 20 evals ≈ 200 sustained rounds | **Arms (8).** `{felix, oort, refl, feddance} × {B, B_oracle}`. Each `B_oracle` uses @@ -193,7 +193,7 @@ oracle runs are suffixed `_node`). artifact of a single global unlock clock: staggering (clients' data arriving in different windows) should *amplify* the top-K churn and the baselines' disparity. - **"20 consecutive evals ≥ 60%"** yields a stable time-to-accuracy (not a lucky - spike); the `rounds`/`max_runtime_s` cap guarantees non-converging arms still + spike); the `rounds`/`max_experiment_runtime_s` cap guarantees non-converging arms still terminate. - **Simulated mode** gives a clean virtual-clock time axis, determinism, and speed. diff --git a/lib/python/examples/async_cifar10/docs/IMPLEMENTATION_felix_streaming.md b/lib/python/examples/async_cifar10/docs/IMPLEMENTATION_felix_streaming.md index 79521e459..26b4fde25 100644 --- a/lib/python/examples/async_cifar10/docs/IMPLEMENTATION_felix_streaming.md +++ b/lib/python/examples/async_cifar10/docs/IMPLEMENTATION_felix_streaming.md @@ -85,7 +85,7 @@ Follow‑ups at the bottom. **Config** (`config.py`, `Hyperparameters`): added `target_accuracy` (`targetAccuracy`, float, default `None` = disabled) and `stable_evals_above_target` (`stableEvalsAboveTarget`, int, default 20). The -existing `rounds` and `max_runtime_s` remain the safety cap (the latter is already +existing `rounds` and `max_experiment_runtime_s` remain the safety cap (the latter is already honored against the **virtual** clock in sim mode at `increment_round`). **Aggregator** (`syncfl/top_aggregator.py`, shared by sync + async stacks): @@ -105,7 +105,7 @@ rounds rather than tens of thousands. **Acceptance**: smoke run with `targetAccuracy=0.2, stableEvalsAboveTarget=2` terminates shortly after two consecutive evals ≥ 0.2; a high target falls through -to the `rounds`/`max_runtime_s` cap. +to the `rounds`/`max_experiment_runtime_s` cap. ## B. Staggered per-client streaming @@ -194,7 +194,7 @@ blocks on the aggregator config. Each arm: `num_trainers=50`, `data_streaming` (uniform `full_after_s=10800`; staggered adds the `stagger` block), `util_counterfactual` on, `checkpoint` on (offline oracle needs it), `aggGoal/aggr_num=10`, `targetAccuracy=0.60`, `stableEvalsAboveTarget=20`, -`evalEveryNRounds=10`, `rounds=20000`/`max_runtime_s=12600` cap. felix `c=10`; +`evalEveryNRounds=10`, `rounds=20000`/`max_experiment_runtime_s=12600` cap. felix `c=10`; `B_oracle` keeps B's own selector kwargs. Structure mirrors the working `felix_oort_refl_feddance_alpha0.1.yaml`. diff --git a/lib/python/examples/async_cifar10/docs/PLAN_felix_streaming_experiment.md b/lib/python/examples/async_cifar10/docs/PLAN_felix_streaming_experiment.md index adb688e24..a74711fcf 100644 --- a/lib/python/examples/async_cifar10/docs/PLAN_felix_streaming_experiment.md +++ b/lib/python/examples/async_cifar10/docs/PLAN_felix_streaming_experiment.md @@ -123,7 +123,7 @@ figures (see Verification below), `dg_flame` conda env. - **Config** — `lib/python/flame/config.py`: add hyperparameters `target_accuracy` (alias `targetAccuracy`, float, default `None` = disabled), `stable_evals_above_target` (alias, int, default 20). Reuse the existing `rounds` and - `max_runtime_s` as the safety cap (already honored in sim via the virtual clock). + `max_experiment_runtime_s` as the safety cap (already honored in sim via the virtual clock). - **Aggregator** — the eval result funnels through `_eval_emit(round, loss, acc)` in `lib/python/flame/mode/horizontal/syncfl/top_aggregator.py` (shared by sync + async stacks). Add a `_check_target_stop(acc)` called from `_eval_emit`: @@ -193,7 +193,7 @@ figures (see Verification below), `dg_flame` conda env. `data_streaming.enabled: 'True'` (+ `stagger` block for the staggered variant), `util_counterfactual.enabled: 'True'`, `checkpoint.enabled: 'True'` + `every_n_rounds` (required for offline oracle), `aggGoal/aggr_num: 10`, `targetAccuracy: 0.60`, - `stableEvalsAboveTarget: 20`, `evalEveryNRounds: 10`, `rounds`/`max_runtime_s` cap. felix + `stableEvalsAboveTarget: 20`, `evalEveryNRounds: 10`, `rounds`/`max_experiment_runtime_s` cap. felix `selector.kwargs.c: 10`; oracle arm `c: 50`. ### E. Analysis + claim figures diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_real.yaml index e646fd11c..443ab1326 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_real.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: k: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_sim.yaml index 52ff2a455..a5f49cc5b 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/fedavg_n48_parity_seeded_sim.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: k: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_real.yaml index 87087cf80..d48e87017 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_real.yaml @@ -43,7 +43,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_sim.yaml index d53d5d9de..89078f442 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/fedbuff_n48_parity_seeded_sim.yaml @@ -43,7 +43,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_real.yaml index e863ff5d2..2e57f4cc2 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_real.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: aggr_num: 5 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_sim.yaml index adef5f2a8..684ac8d0a 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/feddance_n48_parity_seeded_sim.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: aggr_num: 5 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n300_alpha0.1_syn0_STREAMING.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n300_alpha0.1_syn0_STREAMING.yaml index cc92117ea..128675a49 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n300_alpha0.1_syn0_STREAMING.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n300_alpha0.1_syn0_STREAMING.yaml @@ -39,7 +39,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 checkpoint: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_real.yaml index 4737f99ef..8be0df0f5 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_real.yaml @@ -46,7 +46,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_sim.yaml index 201b35207..7cc3ae224 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_parity_seeded_sim.yaml @@ -46,7 +46,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 8 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_real.yaml index f3ae65215..159521c83 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_real.yaml @@ -45,7 +45,7 @@ experiments: rounds: 40 aggGoal: 8 seed: 1234 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 12 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_sim.yaml index 653681627..9f07c7a72 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_n48_stress_sim.yaml @@ -45,7 +45,7 @@ experiments: rounds: 40 aggGoal: 8 seed: 1234 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 selector: kwargs: c: 12 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1.yaml index 5b266aedc..6fd5f8076 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1.yaml @@ -67,7 +67,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 evalEveryNRounds: 10 checkpoint: {enabled: "True", every_n_rounds: 10} @@ -111,7 +111,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 evalEveryNRounds: 10 checkpoint: {enabled: "True", every_n_rounds: 10} @@ -152,7 +152,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 checkpoint: {enabled: "True", every_n_rounds: 10} trackTrainerAvail: {trace: syn_0} @@ -192,7 +192,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 checkpoint: {enabled: "True", every_n_rounds: 10} trackTrainerAvail: {trace: syn_0} @@ -233,7 +233,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 checkpoint: {enabled: "True", every_n_rounds: 10} trackTrainerAvail: {trace: syn_0} @@ -273,7 +273,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 checkpoint: {enabled: "True", every_n_rounds: 10} trackTrainerAvail: {trace: syn_0} @@ -314,7 +314,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 evalEveryNRounds: 10 checkpoint: {enabled: "True", every_n_rounds: 10} @@ -354,7 +354,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) + max_experiment_runtime_s: 12600 # 3.5h wall-clock cap (data fully unlocks at 3h) aggGoal: 10 evalEveryNRounds: 10 checkpoint: {enabled: "True", every_n_rounds: 10} diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node1.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node1.yaml index 6b2b629ff..3d265b444 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node1.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node1.yaml @@ -38,7 +38,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 checkpoint: @@ -101,7 +101,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 checkpoint: enabled: 'True' diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node2.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node2.yaml index 1125a87c0..af125748a 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node2.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_CONTROL_node2.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 checkpoint: enabled: 'True' @@ -105,7 +105,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 checkpoint: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node1.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node1.yaml index 663adb0b2..db6f09f66 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node1.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node1.yaml @@ -38,7 +38,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 @@ -103,7 +103,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node2.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node2.yaml index 953dbd3ec..bc0825b98 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node2.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_SIMULATED_node2.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 @@ -110,7 +110,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml index 29e11e960..dd8944410 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node1.yaml @@ -38,7 +38,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 checkpoint: @@ -101,7 +101,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 checkpoint: enabled: 'True' diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node2.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node2.yaml index e1f914b75..0108a0e54 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node2.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_STREAMING_node2.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 checkpoint: enabled: 'True' @@ -105,7 +105,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 checkpoint: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_parity.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_parity.yaml index e588ed18d..c21c64518 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_parity.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_parity.yaml @@ -41,7 +41,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 @@ -91,6 +91,18 @@ experiments: checkpoint: enabled: 'True' every_n_rounds: 50 + # Stage C/D gate plumbing (§7): sim_unavailability activates the oracular + # delivery-ledger substrate for felix via the asyncfl commit loop. With + # syn_0 (100% avail) the mechanism fires but sees no UN_AVL → byte-identical + # behavior. client_notify.trace is overridden by debug_run.sh --trace; + # enabled stays False (Stage H). + # Two-axis flag split (T1): felix is the only baseline with both flags ON. + sim_unavailability: 'True' + avail_select_filter: 'True' + proactive_inflight_evict: 'True' + client_notify: + enabled: 'False' + trace: syn_0 selector: kwargs: c: 30 @@ -156,7 +168,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 50 checkpoint: @@ -164,6 +176,12 @@ experiments: every_n_rounds: 50 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 + sim_unavailability: 'True' + avail_select_filter: 'True' + proactive_inflight_evict: 'True' + client_notify: + enabled: 'False' + trace: syn_0 selector: kwargs: c: 30 @@ -233,7 +251,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 @@ -242,6 +260,10 @@ experiments: # arrival, so sim in_flight (~3 carry) matches real. Sim-only; validated # by this run (Sr residence rung). See PARITY.md Jun-17c. simInflightCarryover: true + # oort is unaware at selection (avail_select_filter=False); the trace is + # used for starvation-advance vclock tracking only, not selection filtering. + avail_select_filter: 'False' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -303,8 +325,12 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 + # oort is unaware at selection (avail_select_filter=False); the trace is + # used for starvation-advance vclock tracking only, not selection filtering. + avail_select_filter: 'False' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -368,7 +394,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 @@ -385,6 +411,9 @@ experiments: # in _advance_sim_clock (max(vclock,sct)+overhead, drift-instrumented), NOT a # clock ramp. Refine the value from the next run's K3b residual if over/under. simCommitOverheadSeconds: 0.011 + # refl is aware at selection but not proactive-evict (reactive-90s). + avail_select_filter: 'True' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -453,8 +482,11 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 + # refl is aware at selection but not proactive-evict (reactive-90s). + avail_select_filter: 'True' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -524,11 +556,14 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 min_trainers_to_start: 290 min_trainers_join_timeout_s: 600 aggGoal: 10 evalEveryNRounds: 50 + # feddance is aware at selection but not proactive-evict (reactive-90s). + avail_select_filter: 'True' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -588,9 +623,12 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 50 + # feddance is aware at selection but not proactive-evict (reactive-90s). + avail_select_filter: 'True' + proactive_inflight_evict: 'False' checkpoint: enabled: 'True' every_n_rounds: 50 @@ -610,3 +648,297 @@ experiments: ram_critical_percent: 90.0 gpu_warning_percent: 80.0 gpu_critical_percent: 90.0 + +# ── oort_star ───────────────────────────────────────────────────────────────── +# oort-sync base + avail_select_filter=True (T3 scaffold). +# Aware at selection (filters UN_AVL), reactive-90s in-flight (no proactive evict). +- name: oort_star_n300_alpha0.1_syn0_stream_sim + description: oort_star (oort sync + aware selection, no proactive evict) sim. + baseline: oort_star + trainer: + num_trainers: 300 + start_id: 1 + dataset: + name: cifar10 + dirichlet_alpha: 0.1 + availability: + mode: syn_0 + enable_training_delays: true + time_mode: simulated + hyperparameters: + batchSize: 10 + learningRate: 0.01 + lrDecayEnabled: true + lrDecayFactor: 0.98 + lrDecayEpoch: 10 + minLearningRate: 0.0001 + config_overrides: + hyperparameters: + data_streaming: + enabled: "True" + full_data_available_after_s: 10800 + util_counterfactual: + enabled: "True" + every_n_rounds: 50 + sample_size: 256 + aggregator: + config_template: ../_metadata/aggregator_base.json + selector: oort + tracking_mode: oracular + agg_goal: 10 + log_to_wandb: false + config_overrides: + job: + id: oort_star_n300_alpha0.1_syn0_stream_sim + hyperparameters: + batchSize: 10 + learningRate: 0.01 + rounds: 20000 + max_experiment_runtime_s: 12600 + min_trainers_to_start: 290 + min_trainers_join_timeout_s: 600 + aggGoal: 10 + simInflightCarryover: true + avail_select_filter: "True" + proactive_inflight_evict: "False" + checkpoint: + enabled: "True" + every_n_rounds: 50 + trackTrainerAvail: + trace: syn_0 + selector: + kwargs: + aggr_num: 10 + execution: + num_gpus: 8 + sleep_between_spawns: 1.0 + aggregator_warmup_time: 60 + monitoring: + enabled: true + check_interval_seconds: 30 + ram_warning_percent: 80.0 + ram_critical_percent: 90.0 + gpu_warning_percent: 80.0 + gpu_critical_percent: 90.0 +- name: oort_star_n300_alpha0.1_syn0_stream_real + description: oort_star (oort sync + aware selection, no proactive evict) real. + baseline: oort_star + trainer: + num_trainers: 300 + start_id: 1 + dataset: + name: cifar10 + dirichlet_alpha: 0.1 + availability: + mode: syn_0 + enable_training_delays: true + time_mode: real + hyperparameters: + batchSize: 10 + learningRate: 0.01 + lrDecayEnabled: true + lrDecayFactor: 0.98 + lrDecayEpoch: 10 + minLearningRate: 0.0001 + config_overrides: + hyperparameters: + data_streaming: + enabled: "True" + full_data_available_after_s: 10800 + util_counterfactual: + enabled: "True" + every_n_rounds: 50 + sample_size: 256 + aggregator: + config_template: ../_metadata/aggregator_base.json + selector: oort + tracking_mode: oracular + agg_goal: 10 + log_to_wandb: false + config_overrides: + job: + id: oort_star_n300_alpha0.1_syn0_stream_real + hyperparameters: + batchSize: 10 + learningRate: 0.01 + rounds: 20000 + max_experiment_runtime_s: 12600 + aggGoal: 10 + avail_select_filter: "True" + proactive_inflight_evict: "False" + checkpoint: + enabled: "True" + every_n_rounds: 50 + trackTrainerAvail: + trace: syn_0 + min_trainers_to_start: 290 + min_trainers_join_timeout_s: 600 + selector: + kwargs: + aggr_num: 10 + execution: + num_gpus: 8 + sleep_between_spawns: 1.0 + aggregator_warmup_time: 60 + monitoring: + enabled: true + check_interval_seconds: 30 + ram_warning_percent: 80.0 + ram_critical_percent: 90.0 + gpu_warning_percent: 80.0 + gpu_critical_percent: 90.0 +# ── fedbuff ─────────────────────────────────────────────────────────────────── +# asyncfl base + FedBuffSelector + both flags OFF (unaware, T3 scaffold). +- name: fedbuff_n300_alpha0.1_syn0_stream_sim + description: fedbuff (asyncfl + fedbuff selector, both flags OFF) sim. + baseline: fedbuff + trainer: + num_trainers: 300 + start_id: 1 + dataset: + name: cifar10 + dirichlet_alpha: 0.1 + availability: + mode: syn_0 + enable_training_delays: true + time_mode: simulated + hyperparameters: + # LR schedule matches felix and all other async cifar10 runs. + batchSize: 10 + learningRate: 0.01 + lrDecayEnabled: true + lrDecayFactor: 0.98 + lrDecayEpoch: 10 + minLearningRate: 0.0001 + config_overrides: + hyperparameters: + data_streaming: + enabled: "True" + full_data_available_after_s: 10800 + util_counterfactual: + enabled: "True" + every_n_rounds: 50 + sample_size: 256 + aggregator: + config_template: ../_metadata/aggregator_base.json + selector: fedbuff + tracking_mode: default + agg_goal: 10 + log_to_wandb: false + config_overrides: + job: + id: fedbuff_n300_alpha0.1_syn0_stream_sim + hyperparameters: + batchSize: 10 + learningRate: 0.01 + rounds: 20000 + max_experiment_runtime_s: 12600 + min_trainers_to_start: 290 + min_trainers_join_timeout_s: 600 + aggGoal: 10 + evalEveryNRounds: 50 + simSctOrderedDrain: true + simClockJumpClamp: true + simInflightResidence: true + avail_select_filter: "False" + proactive_inflight_evict: "False" + sim_unavailability: "True" + availability_trace: syn_0 + client_notify: + enabled: "False" + trace: syn_0 + checkpoint: + enabled: "True" + every_n_rounds: 50 + selector: + kwargs: + c: 30 + aggGoal: 10 + round_threshold: 70 + exploration_decay: 0.999 + execution: + num_gpus: 8 + sleep_between_spawns: 1.0 + aggregator_warmup_time: 60 + monitoring: + enabled: true + check_interval_seconds: 30 + ram_warning_percent: 80.0 + ram_critical_percent: 90.0 + gpu_warning_percent: 80.0 + gpu_critical_percent: 90.0 +- name: fedbuff_n300_alpha0.1_syn0_stream_real + description: fedbuff (asyncfl + fedbuff selector, both flags OFF) real. + baseline: fedbuff + trainer: + num_trainers: 300 + start_id: 1 + dataset: + name: cifar10 + dirichlet_alpha: 0.1 + availability: + mode: syn_0 + enable_training_delays: true + time_mode: real + hyperparameters: + # LR schedule matches felix and all other async cifar10 runs. + batchSize: 10 + learningRate: 0.01 + lrDecayEnabled: true + lrDecayFactor: 0.98 + lrDecayEpoch: 10 + minLearningRate: 0.0001 + config_overrides: + hyperparameters: + data_streaming: + enabled: "True" + full_data_available_after_s: 10800 + util_counterfactual: + enabled: "True" + every_n_rounds: 50 + sample_size: 256 + aggregator: + config_template: ../_metadata/aggregator_base.json + selector: fedbuff + tracking_mode: default + agg_goal: 10 + log_to_wandb: false + config_overrides: + job: + id: fedbuff_n300_alpha0.1_syn0_stream_real + hyperparameters: + batchSize: 10 + learningRate: 0.01 + rounds: 20000 + max_experiment_runtime_s: 12600 + aggGoal: 10 + evalEveryNRounds: 50 + avail_select_filter: "False" + proactive_inflight_evict: "False" + sim_unavailability: "True" + availability_trace: syn_0 + client_notify: + enabled: "False" + trace: syn_0 + checkpoint: + enabled: "True" + every_n_rounds: 50 + min_trainers_to_start: 290 + min_trainers_join_timeout_s: 600 + selector: + kwargs: + c: 30 + aggGoal: 10 + round_threshold: 70 + exploration_decay: 0.999 + execution: + num_gpus: 8 + sleep_between_spawns: 1.0 + aggregator_warmup_time: 60 + monitoring: + enabled: true + check_interval_seconds: 30 + ram_warning_percent: 80.0 + ram_critical_percent: 90.0 + gpu_warning_percent: 80.0 + gpu_critical_percent: 90.0 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node1.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node1.yaml index 33b55173f..df2fdcbd7 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node1.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node1.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -106,7 +106,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node2.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node2.yaml index c8321ff9e..df3542118 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node2.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node2.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -105,7 +105,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node3.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node3.yaml index 601a483b5..81c108d0b 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node3.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node3.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -105,7 +105,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node4.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node4.yaml index 1210e376f..1a9e9ba92 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node4.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_node4.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -105,7 +105,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_sim.yaml index ede512daa..f4c23b9b3 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/n50_alpha0.1_syn0_stream_unif_sim.yaml @@ -42,7 +42,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -106,7 +106,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -167,7 +167,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -220,7 +220,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -281,7 +281,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -334,7 +334,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -395,7 +395,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 @@ -448,7 +448,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 20000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 evalEveryNRounds: 10 targetAccuracy: 0.6 diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n300_alpha0.1_syn0_STREAMING.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n300_alpha0.1_syn0_STREAMING.yaml index e352df9d1..1bdc33f3e 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n300_alpha0.1_syn0_STREAMING.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n300_alpha0.1_syn0_STREAMING.yaml @@ -43,7 +43,7 @@ experiments: batchSize: 10 learningRate: 0.01 rounds: 1000 - max_runtime_s: 12600 + max_experiment_runtime_s: 12600 aggGoal: 10 checkpoint: enabled: 'True' diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_real.yaml index d1d066d86..1ac3fa384 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_real.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 trackTrainerAvail: trace: syn_0 selector: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_sim.yaml index f42ec358b..4b1881fa5 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/oort_n48_parity_seeded_sim.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 trackTrainerAvail: trace: syn_0 selector: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_real.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_real.yaml index 20390d30e..497552c7c 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_real.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_real.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 trackTrainerAvail: trace: syn_0 selector: diff --git a/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_sim.yaml b/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_sim.yaml index e26166e12..304e0e201 100644 --- a/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_sim.yaml +++ b/lib/python/examples/async_cifar10/expt_scripts_2026/refl_n48_parity_seeded_sim.yaml @@ -40,7 +40,7 @@ experiments: aggGoal: 5 seed: 1234 min_trainers_to_start: 48 - max_runtime_s: 1200 + max_experiment_runtime_s: 1200 trackTrainerAvail: trace: syn_0 selector: diff --git a/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md b/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md index 951528856..2290af0fa 100644 --- a/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md +++ b/lib/python/examples/async_cifar10/real-sim_parity_checker_plan.md @@ -7,7 +7,7 @@ - real = `experiments/run_20260606_182638_dbg_felix_n300_alpha0.1_syn0_stream_real` Both: Felix (`async_oort` + `fedbuff`), n=300, α=0.1, syn_0, agg_goal=10, -`max_runtime_s = sim_wall_ceiling_s = 10800s (3h)`, `min_trainers_to_start=290`. +`max_experiment_runtime_s = sim_wall_ceiling_s = 10800s (3h)`, `min_trainers_to_start=290`. --- @@ -107,7 +107,7 @@ Four findings: `rounds` cap, not the wall/vclock budget. **Fixed**: `rounds: 1000 → 20000` in `expt_scripts_2026/felix_oort_refl_feddance_alpha0.1_OVERNIGHT_node{1,2}.yaml` (canonical source, covers overnight + debug) plus an explicit `h["rounds"]=20000` - guard in `scripts/debug_run.sh` (non-smoke branch, alongside the `max_runtime_s` + guard in `scripts/debug_run.sh` (non-smoke branch, alongside the `max_experiment_runtime_s` override). New checker check **K9** below makes truncation-by-cap an explicit WARN so it can never silently distort a comparison again. @@ -277,7 +277,7 @@ gating; surface Jaccard as signal. | K6 | INV | sim mode: `task_recv.sim_send_ts` non-null & increasing after round 1; real mode: null | trainer task_recv | no violations | | K7 | INV | sim_rate (`vclock/wall`) in sane range [0.01, 100] | agg_round (sim) | in range | | K8 | DIST | **terminal-state parity**: at matched virtual budget V, both reached comparable FL-round count, total commits, total unique trainers used | agg_round | rounds within 10%, trainers within 5% | -| K9 | INV | **stopped-by-budget, not by cap**: neither run hit the `rounds` cap before `max_runtime_s` — else the comparison is truncated and downstream metrics are biased | agg_round + config | WARN if `max_round == rounds_cap` and wall/vclock < budget | +| K9 | INV | **stopped-by-budget, not by cap**: neither run hit the `rounds` cap before `max_experiment_runtime_s` — else the comparison is truncated and downstream metrics are biased | agg_round + config | WARN if `max_round == rounds_cap` and wall/vclock < budget | | K10 | INV | **vclock telemetry present**: a sim run's agg_round events must carry `vclock_now` (sync path currently omits it). FAIL-LOUD rather than silently SKIP K1–K3/K7 | agg_round (sim) | FAIL if sim run has zero `vclock_now` stamps | > **K2/K3/K4 are the checks that would have caught the 410-vs-673 regression on diff --git a/lib/python/examples/async_cifar10/scripts/debug_run.sh b/lib/python/examples/async_cifar10/scripts/debug_run.sh index 492e134dd..fe24b6326 100755 --- a/lib/python/examples/async_cifar10/scripts/debug_run.sh +++ b/lib/python/examples/async_cifar10/scripts/debug_run.sh @@ -17,7 +17,7 @@ # baseline is a no-op (not an error). # # Runtime: -# --runtime-s sets max_runtime_s for BOTH real and sim variants. +# --runtime-s sets max_experiment_runtime_s for BOTH real and sim variants. # Real mode: wall-clock seconds (passes directly). # Sim mode: virtual-clock seconds (vclock fix ensures sim stops at this # many virtual seconds, which completes in far less wall-clock @@ -49,20 +49,22 @@ echo "conda: base=$CB env=$ENVNAME python=$(which python)" EX="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" cd "$EX" || exit 1 SCR=expt_scripts_2026 -LOGDIR=/tmp/debug_run_logs; mkdir -p "$LOGDIR" +LOGDIR="${FLAME_LOGDIR:-/tmp/debug_run_logs}"; mkdir -p "$LOGDIR" export FLAME_BATCH_CONTINUE_ON_ERROR=1 # defaults RUNTIME_S=10800 BASELINES="felix refl" -SIM_WALL_CEILING_S="" # empty = max_runtime_s (1×, tight guard; sim should be faster than real) +SIM_WALL_CEILING_S="" # empty = max_experiment_runtime_s (1×, tight guard; sim should be faster than real) MODE="both" # sim | real | both — which time_mode variant(s) of each baseline to run +NUM_TRAINERS="" # empty = use whatever's in the parity config (300); non-smoke override only +ALPHA="" # empty = use the parity config's dirichlet_alpha (0.1); e.g. 100 for homogeneous usage() { - echo "usage: $0 [--baselines 'felix refl'] [--runtime-s 3600] [--mode sim|real|both] [--sim-wall-ceiling-s 2700]" - echo " $0 smoke [--baselines ...] [--mode sim|real|both]" + echo "usage: $0 [--baselines 'felix refl'] [--runtime-s 3600] [--mode sim|real|both] [--sim-wall-ceiling-s 2700] [--trace syn_20]" + echo " $0 smoke [--baselines ...] [--mode sim|real|both] [--trace syn_20]" echo "" - echo " --baselines which baselines to run (any of felix oort refl feddance);" + echo " --baselines which baselines to run (any of felix oort oort_star refl feddance fedbuff);" echo " filtered from the parity config, node-agnostic." echo " --mode which time_mode variant(s) to run for each baseline:" echo " 'sim' (only the simulated run), 'real' (only the real run)," @@ -71,17 +73,31 @@ usage() { echo " --sim-wall-ceiling-s wall-clock ceiling for sim mode (default: = runtime_s)." echo " A well-behaved sim finishes in <= real-mode wall time." echo " Fires [SIM_WALL_CEILING] warning + stops when exceeded." + echo " --trace availability trace name(s) to substitute, space-separated for" + echo " multiple (e.g. 'syn_20 syn_50' queues both, one experiment set each)." + echo " Default: use whatever is in the parity config (syn_0)." + echo " --num-trainers non-smoke only: shrink the cohort below the parity config's 300," + echo " scaling min_trainers_to_start down with it (gap of 8, same ratio as" + echo " smoke). Use this instead of 'smoke' when you need a real --runtime-s" + echo " budget (e.g. a vclock floor for an availability trace) that smoke's" + echo " hardcoded rounds=4/runtime=240 would cut short." + echo " --alpha Dirichlet alpha override (default: parity config's 0.1). Supported" + echo " values have an n300 split: 0.1 / 1.0 / 10.0 / 100.0 (100=homogeneous)." + echo " When set, the split lookup uses the n300 partition for that alpha." exit 2 } # parse args +TRACE="" # empty = use whatever is in the parity config (syn_0) if [ "${1:-}" = "smoke" ]; then SMOKE=1; shift - BASELINES="felix oort refl feddance" # smoke default: validate all + BASELINES="felix oort oort_star refl feddance fedbuff" # smoke default: validate all while [[ $# -gt 0 ]]; do case "$1" in --baselines) BASELINES="$2"; shift 2 ;; --mode) MODE="$2"; shift 2 ;; + --trace) TRACE="$2"; shift 2 ;; + --alpha) ALPHA="$2"; shift 2 ;; *) shift ;; esac done @@ -94,6 +110,9 @@ else --mode) MODE="$2"; shift 2 ;; --sim-wall-ceiling-s) SIM_WALL_CEILING_S="$2"; shift 2 ;; --wall-runtime-s) SIM_WALL_CEILING_S="$2"; shift 2 ;; # backward compat alias + --trace) TRACE="$2"; shift 2 ;; + --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; + --alpha) ALPHA="$2"; shift 2 ;; # --node is DEPRECATED (node1/node2 split removed): baselines are filtered # from a single node-agnostic parity config, so the node is irrelevant. # Accept+ignore so existing wrappers don't hard-error. @@ -106,15 +125,21 @@ case "$MODE" in sim|real|both) ;; *) echo "ERROR: --mode must be sim|real|both ( # Generate a single filtered+patched YAML from the parity source config. # $1 = baselines (space-separated), $2 = runtime_s, $3 = output path, -# [$4 = smoke: 1|0], [$5 = sim_wall_ceiling_s: int or ""], [$6 = mode: sim|real|both] +# [$4 = smoke: 1|0], [$5 = sim_wall_ceiling_s: int or ""], [$6 = mode: sim|real|both], +# [$7 = trace: trace name or ""], [$8 = num_trainers override: int or "", non-smoke only], +# [$9 = alpha override: float or ""] make_debug_yaml() { - python - "$SCR" "$1" "$2" "$3" "${4:-0}" "${5:-}" "${6:-both}" <<'PY' + python - "$SCR" "$1" "$2" "$3" "${4:-0}" "${5:-}" "${6:-both}" "${7:-}" "${8:-}" "${9:-}" <<'PY' import yaml, sys, copy, os scr, baselines_str, runtime_s, outpath, smoke, ceil_arg = ( sys.argv[1], sys.argv[2], int(sys.argv[3]), sys.argv[4], sys.argv[5] == "1", sys.argv[6] if len(sys.argv) > 6 else "" ) mode = (sys.argv[7] if len(sys.argv) > 7 else "both").lower() +# Space-separated list of trace names (e.g. "syn_20 syn_50"); "" -> [""] (no substitution). +trace_overrides = sys.argv[8].strip().split() if len(sys.argv) > 8 and sys.argv[8].strip() else [""] +num_trainers_override = int(sys.argv[9]) if len(sys.argv) > 9 and sys.argv[9].strip() else None +alpha_override = float(sys.argv[10]) if len(sys.argv) > 10 and sys.argv[10].strip() else None requested = set(baselines_str.lower().split()) # Deterministic selection seed (same for real+sim). Default 1234; SEED=none disables. _seed_env = os.environ.get("SEED", "1234").strip() @@ -141,44 +166,127 @@ def exp_mode(e): # feddance) × {sim, real}; filter it to the requested baselines/mode. src = f"{scr}/felix_oort_refl_feddance_alpha0.1_parity.yaml" try: - cfg = yaml.safe_load(open(src)) + # encoding="utf-8" explicit: the config has non-ASCII chars (e.g. "->" arrows + # in comments/descriptions); without this, open() falls back to the node's + # locale-preferred encoding, which mis-decodes them on non-UTF-8 locales + # (e.g. C/POSIX) and yaml.safe_load then rejects the resulting control chars. + cfg = yaml.safe_load(open(src, encoding="utf-8")) except FileNotFoundError: print(f"ERROR: parity config not found: {src}", flush=True) sys.exit(1) kept = [] -for e in cfg.get("experiments", []): - bl = e.get("baseline", "").lower() +for e_src in cfg.get("experiments", []): + bl = e_src.get("baseline", "").lower() if bl not in requested: continue - if mode != "both" and exp_mode(e) != mode: + if mode != "both" and exp_mode(e_src) != mode: continue - e = copy.deepcopy(e) - h = e["aggregator"]["config_overrides"]["hyperparameters"] - h["max_runtime_s"] = runtime_s - # Deterministic seed: the SAME value for every experiment so the real and sim - # variants of each baseline make identical selection draws (dedicated per- - # selector RNG, PARITY "Determinism / seeding"). Without this, real vs sim are - # two independent stochastic paths and participation/utility can never match. - # Override per-invocation with SEED=; SEED=none disables (legacy unseeded). - if seed_val is not None: - h["seed"] = seed_val - # sim_wall_ceiling_s: tight wall guard — sim must finish in <= this many - # wall-seconds (default = max_runtime_s = 1×; a healthy sim is faster). - h["sim_wall_ceiling_s"] = int(ceil_arg) if ceil_arg else runtime_s - if smoke: - e["trainer"]["num_trainers"] = 48 - h["rounds"] = 4 - h["min_trainers_to_start"] = 40 - h["min_trainers_join_timeout_s"] = 120 - e["name"] = "dbg_smoke_" + e["name"] - else: - # High round cap so the wall/vclock budget (max_runtime_s) is the - # binding stop condition, not an early round-count termination. - h["rounds"] = 20000 - e["name"] = f"dbg_{e['name']}" - e["aggregator"]["config_overrides"]["job"]["id"] = e["name"] - kept.append(e) + # One experiment per requested trace (trace_overrides has 1 entry, "", when + # --trace wasn't given, so this loop is a no-op pass-through by default). + for trace_override in trace_overrides: + e = copy.deepcopy(e_src) + h = e["aggregator"]["config_overrides"]["hyperparameters"] + h["max_experiment_runtime_s"] = runtime_s + # Deterministic seed: the SAME value for every experiment so the real and sim + # variants of each baseline make identical selection draws (dedicated per- + # selector RNG, PARITY "Determinism / seeding"). Without this, real vs sim are + # two independent stochastic paths and participation/utility can never match. + # Override per-invocation with SEED=; SEED=none disables (legacy unseeded). + if seed_val is not None: + h["seed"] = seed_val + # sim_wall_ceiling_s: tight wall guard -- sim must finish in <= this many + # wall-seconds (default = max_experiment_runtime_s = 1x; a healthy sim is faster). + h["sim_wall_ceiling_s"] = int(ceil_arg) if ceil_arg else runtime_s + if smoke: + e["trainer"]["num_trainers"] = 48 + h["rounds"] = 4 + h["min_trainers_to_start"] = 40 + h["min_trainers_join_timeout_s"] = 120 + e["name"] = "dbg_smoke_" + e["name"] + else: + # High round cap so the wall/vclock budget (max_experiment_runtime_s) is the + # binding stop condition, not an early round-count termination. + h["rounds"] = 20000 + if num_trainers_override: + # Shrink the cohort but keep runtime_s as the real budget (unlike + # smoke, which hardcodes rounds=4/runtime=240 -- too short for a + # trace-driven vclock floor like syn_20's first UN_AVL at t=600s). + # Same join-barrier slack ratio as smoke (gap of 8 below the count). + # Preserve the config's native partition size as split_num_trainers + # so the shrunk cohort reads the existing n split (e.g. n300) + # instead of demanding a dedicated n split file that may + # not exist (there is no cifar10_alpha0.1_n10 split, only n48/50/300). + # spawn_all spawns num_trainers trainers but keys the split lookup on + # split_num_trainers -- the two are independent by design. + orig_n = e["trainer"].get("num_trainers", 300) + e["trainer"]["num_trainers"] = num_trainers_override + e["trainer"]["split_num_trainers"] = orig_n + h["min_trainers_to_start"] = max(1, num_trainers_override - 8) + e["name"] = f"dbg_{e['name']}" + # --alpha override: repoint dirichlet_alpha and the split lookup. Only n300 + # splits exist for every alpha (0.1/1.0/10.0/100.0=homogeneous); n48/n50 + # exist for alpha0.1 only. So read the n300 partition for the chosen alpha + # (the cohort stays num_trainers, spawned as the first num_trainers of the + # 300-way split via the split_num_trainers decoupling). Name gets an + # alpha<..> tag so run dirs are distinguishable across alphas. + if alpha_override is not None: + e["trainer"].setdefault("dataset", {})["dirichlet_alpha"] = alpha_override + e["trainer"]["split_num_trainers"] = 300 + e["name"] = f"{e['name']}_alpha{str(alpha_override).replace('.', 'p')}" + # --trace override: substitute availability trace in trainer + aggregator config. + if trace_override: + # trainer.availability.mode is NOT read by anything (main.py/config.py never + # touch config.availability) -- vestigial from an earlier design, kept + # write-only here so as not to silently drop a field some other consumer may + # still expect. The trainer's ACTUAL trace selection comes from + # hyperparameters.client_notify.trace (see main.py's state_avl_event_ts + # assignment), which lives under trainer.config_overrides.hyperparameters, + # not trainer.hyperparameters (that block is base-model HP only: batchSize/ + # learningRate/etc, merged from configs/trainer_base.yaml's own client_notify + # default of trace=syn_0). Before this fix, ONLY the aggregator's own trace + # read (via `h` below) was ever overridden -- every debug_run.sh-launched + # trainer, real and sim, ran with client_notify.trace stuck at the + # trainer_base.yaml default (syn_0, always-available) regardless of the + # requested --trace, silently no-op'ing the trainer-side avl_state machinery + # (and hence the real-mode send-gate and all avail_change telemetry) for + # every trace-driven run this project has ever launched. Root-caused Jul 1 + # via UNAVAILABILITY_DESIGN.md Batch 3 T3.1. + avail = e["trainer"].setdefault("availability", {}) + old_trace = avail.get("mode", "syn_0") + avail["mode"] = trace_override + t_co_hp = e["trainer"].setdefault("config_overrides", {}).setdefault("hyperparameters", {}) + t_co_hp.setdefault("client_notify", {})["trace"] = trace_override + if "trackTrainerAvail" in h: + h["trackTrainerAvail"]["trace"] = trace_override + # For baselines NOT on the ORACULAR legacy path (felix, feddance, + # oracle, fedbuff): activate the new sim_unavailability gate so + # _init_availability picks up the trace (Sec 7 felix master-gate). + # ORACULAR baselines (oort, refl) already activate via the legacy path. + if h["trackTrainerAvail"].get("type", "").upper() != "ORACULAR": + h["simUnavailability"] = True + # proactive_inflight_evict is set directly in each experiment's + # config_overrides HP (T1 two-axis split); no auto-detection needed + # here. The client_notify.enabled check below is always False + # (Stage H is future), so proactiveInflightEvict is never set by + # this branch -- the explicit YAML value is authoritative. + t_hp = e.get("trainer", {}).get("hyperparameters", {}) + if str(t_hp.get("client_notify", {}).get("enabled", "False")).lower() == "true": + h["proactiveInflightEvict"] = True + elif "client_notify" in h and isinstance(h["client_notify"], dict): + h["client_notify"]["trace"] = trace_override + h["simUnavailability"] = True + elif e["aggregator"].get("tracking_mode", "oracular").lower() != "oracular": + # Non-oracular baseline with no HP-level tracking block (e.g. feddance + # in v1, which has no client_notify in HP and no trackTrainerAvail). + # Inject trace via availability_trace so _init_availability finds it. + h["availability_trace"] = trace_override + h["simUnavailability"] = True + # Rewrite syn_ or syn in the name so run dirs are identifiable. + import re + e["name"] = re.sub(r"syn_?[0-9]+", trace_override, e["name"]) + e["aggregator"]["config_overrides"]["job"]["id"] = e["name"] + kept.append(e) if not kept: print(f"WARNING: no experiments matched baselines={baselines_str} mode={mode}", @@ -186,19 +294,61 @@ if not kept: sys.exit(0) cfg["experiments"] = kept -yaml.safe_dump(cfg, open(outpath, "w"), sort_keys=False) +yaml.safe_dump(cfg, open(outpath, "w", encoding="utf-8"), sort_keys=False) print(f"Generated {outpath} with {len(kept)} experiment(s): " f"{[e['name'] for e in kept]}", flush=True) PY } +# Count experiments in a generated YAML (used to estimate budget and track progress). +_count_exps() { + python3 - "$1" <<'PY' +import yaml, sys +d = yaml.safe_load(open(sys.argv[1], encoding="utf-8")) +print(len(d.get('experiments', []))) +PY +} + run_node() { - local label="$1" cfg="$2" - echo "[$(date '+%F %T')] START $label -> $cfg" | tee -a "$LOGDIR/debug_run.log" + local label="$1" cfg="$2" budget_s="${3:-0}" n_exps="${4:-1}" + local start_ts; start_ts=$(date +%s) + # Baseline run-dir count — new dirs that appear are newly-started experiments. + local initial_runs; initial_runs=$(find experiments -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) + + echo "[$(date '+%F %T')] START $label ($n_exps exp(s), ~${budget_s}s budget)" | tee -a "$LOGDIR/debug_run.log" + + # Background progress ticker: fires every 30s, prints elapsed/remaining/percent + # and how many experiments have started (each start creates a new run_* dir). + ( + while true; do + sleep 30 + local now; now=$(date +%s) + local elapsed=$(( now - start_ts )) + local pct=0 remaining=0 + if [ "$budget_s" -gt 0 ]; then + pct=$(( elapsed * 100 / budget_s )) + remaining=$(( budget_s - elapsed )) + [ "$pct" -gt 100 ] && pct=100 + [ "$remaining" -lt 0 ] && remaining=0 + fi + local curr; curr=$(find experiments -maxdepth 1 -name "run_*" -type d 2>/dev/null | wc -l) + local started=$(( curr - initial_runs )) + [ "$started" -lt 0 ] && started=0 + printf " [%s] %s | %ds elapsed / ~%ds (%d%%) | exp started: %d/%d\n" \ + "$(date '+%T')" "$label" "$elapsed" "$budget_s" "$pct" "$started" "$n_exps" + done + ) & + local ticker_pid=$! + python -m flame.launch.run_experiment "$cfg" --example-dir "$EX" \ < /dev/null >> "$LOGDIR/${label}.out" 2>&1 local rc=$? - echo "[$(date '+%F %T')] DONE $label exit=$rc" | tee -a "$LOGDIR/debug_run.log" + + kill "$ticker_pid" 2>/dev/null + wait "$ticker_pid" 2>/dev/null + + local elapsed=$(( $(date +%s) - start_ts )) + echo "[$(date '+%F %T')] DONE $label exit=$rc (took ${elapsed}s / ~${budget_s}s budget)" | tee -a "$LOGDIR/debug_run.log" } # ---- smoke mode ---- @@ -208,8 +358,11 @@ if [ "$SMOKE" = "1" ]; then # Clear any stale config from a previous invocation so a no-match run is # skipped (not silently re-running a leftover config). rm -f "$cfg" - make_debug_yaml "$BASELINES" 240 "$cfg" 1 "$SIM_WALL_CEILING_S" "$MODE" - [ -f "$cfg" ] && run_node "dbg_smoke" "$cfg" + make_debug_yaml "$BASELINES" 240 "$cfg" 1 "$SIM_WALL_CEILING_S" "$MODE" "$TRACE" "" "$ALPHA" + if [ -f "$cfg" ]; then + _n=$(_count_exps "$cfg") + run_node "dbg_smoke" "$cfg" $(( _n * 240 )) "$_n" + fi echo "=== SMOKE RESULTS ===" for dd in experiments/run_*dbg_smoke_*; do [ -d "$dd" ] || continue @@ -221,18 +374,21 @@ if [ "$SMOKE" = "1" ]; then fi # ---- normal run mode ---- -echo "=== DEBUG RUN: baselines='$BASELINES' mode=$MODE runtime_s=$RUNTIME_S sim_wall_ceiling_s=${SIM_WALL_CEILING_S:-auto(=runtime_s)} ===" +echo "=== DEBUG RUN: baselines='$BASELINES' mode=$MODE runtime_s=$RUNTIME_S sim_wall_ceiling_s=${SIM_WALL_CEILING_S:-auto(=runtime_s)} num_trainers=${NUM_TRAINERS:-300(default)} alpha=${ALPHA:-0.1(default)} ===" cfg="$LOGDIR/debug_run.yaml" # Clear any stale config so a no-match run is skipped (not silently re-running # a previous baseline's leftover config). rm -f "$cfg" -make_debug_yaml "$BASELINES" "$RUNTIME_S" "$cfg" 0 "$SIM_WALL_CEILING_S" "$MODE" +make_debug_yaml "$BASELINES" "$RUNTIME_S" "$cfg" 0 "$SIM_WALL_CEILING_S" "$MODE" "$TRACE" "$NUM_TRAINERS" "$ALPHA" if [ ! -f "$cfg" ]; then echo "No experiments matched for baselines='$BASELINES'. Nothing to run." exit 0 fi -run_node "debug_run" "$cfg" +_n_exps=$(_count_exps "$cfg") +_budget=$(( _n_exps * RUNTIME_S )) +echo " queued: $_n_exps exp(s), estimated budget ~${_budget}s (sim finishes faster than real)" +run_node "debug_run" "$cfg" "$_budget" "$_n_exps" echo "Logs: $LOGDIR/debug_run.out" echo "Run dirs: experiments/run_*dbg_*" diff --git a/lib/python/examples/async_cifar10/scripts/gen_n50_experiment.py b/lib/python/examples/async_cifar10/scripts/gen_n50_experiment.py index 602a4a161..1e3a16b0c 100755 --- a/lib/python/examples/async_cifar10/scripts/gen_n50_experiment.py +++ b/lib/python/examples/async_cifar10/scripts/gen_n50_experiment.py @@ -83,7 +83,7 @@ def make_arm(baseline, staggered, oracle=False, suffix=""): name = f"{baseline}{ocl}_n50_alpha0.1_syn0_stream_{cond}_sim{suffix}" agg_hp = { "batchSize": 10, "learningRate": 0.01, - "rounds": ROUNDS_CAP, "max_runtime_s": MAX_RUNTIME_S, + "rounds": ROUNDS_CAP, "max_experiment_runtime_s": MAX_RUNTIME_S, "aggGoal": 10, "evalEveryNRounds": EVAL_EVERY, "targetAccuracy": TARGET_ACC, "stableEvalsAboveTarget": STABLE_EVALS, "min_trainers_to_start": N - 2, "min_trainers_join_timeout_s": 600, diff --git a/lib/python/examples/async_cifar10/scripts/parity/avail_state_series.py b/lib/python/examples/async_cifar10/scripts/parity/avail_state_series.py new file mode 100644 index 000000000..00344ce07 --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/avail_state_series.py @@ -0,0 +1,183 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Shared per-trainer availability-state series builder (Stage C.6.2). + +Single resolver, imported by both checks.py (Aa / A4dur / observation_lag rungs) +and analyze_run.py (the four plots re-pointed in C.6.4), so the forward-fill +semantics never diverge between the checker and the plotter (Challenge 12 +discipline — one function, not a duplicated copy in each consumer). + +Reads `per_trainer[end_id]["avl_state"]` on each `selection` event (C.6.1). +Time-base: both modes prefer `vclock_now`, stamped on the event via +`ClientAvailability._avail_now()` (sim: `_vclock.now`; real: wall-elapsed since +`agg_start_time_ts`) — the same shared origin the rest of the availability +substrate uses. Telemetry recorded before real-mode `vclock_now` stamping was +added falls back to `ts - t0` (t0 = the run's first selection event ts), which +is a *biased* estimate: real's first selection event fires only once enough +trainers have joined over MQTT (~300s for n=300), so it under-counts elapsed +time relative to the trace's true origin and skews any duration-weighted +comparison (A4dur) for trainers whose state changes mid-run. Kept only for +backward-compat with old runs — new telemetry should always carry vclock_now. +""" + +from __future__ import annotations + +from typing import Optional + + +def _event_time(e: dict, mode: str, t0: float) -> Optional[float]: + vclock = e.get("vclock_now") + if vclock is not None: + return vclock + if mode == "sim": + return None + ts = e.get("ts") + return None if ts is None else ts - t0 + + +def build_trainer_state_series( + selection_events: list, mode: str +) -> dict[str, list]: + """{end_id: [(t, avl_state), ...]} forward-fill series, sorted by t. + + mode: "sim" or "real", both preferring t = vclock_now, falling back to + ts - t0 only for real-mode telemetry recorded before vclock_now was + stamped on real selection events. One sample per + end_id per selection event it appears as a candidate in (whether or not + selected) — `avail_composition`/`per_trainer` already cover every + candidate in the pool, not just the chosen subset. Consecutive samples at + the same t collapse to the last write (same-instant events, e.g. carried + pacer state). + """ + real_ts = [e.get("ts") for e in selection_events if e.get("ts") is not None] + t0 = min(real_ts) if (mode == "real" and real_ts) else 0.0 + + series: dict[str, list] = {} + for e in sorted( + selection_events, key=lambda x: (x.get("round", 0), x.get("ts", 0.0)) + ): + t = _event_time(e, mode, t0) + if t is None: + continue + for end_id, cand in (e.get("per_trainer") or {}).items(): + state = cand.get("avl_state") + if state is None or state == "UNKNOWN": + continue + pts = series.setdefault(end_id, []) + if pts and pts[-1][0] == t: + pts[-1] = (t, state) + else: + pts.append((t, state)) + return series + + +def build_observed_timeline_from_avail_change(avail_change_events: list) -> list: + """Sorted [(sim_now, new_state), ...] from one trainer's own avail_change telemetry. + + ``sim_now`` (Batch 3 T3.2) is the trainer's own trace-time-basis clock at + the moment it applied the transition — sim: virtual-clock seconds; real: + wall-elapsed since the shared AGG_START_TS origin (T3.0). Distinct from + the record's ``ts`` (always wall time.time(), meaningless against a trace + indexed in trace-seconds). Events recorded before ``sim_now`` existed are + dropped; an entirely-empty result means "no fidelity signal for this + trainer" (old telemetry), not "trainer never transitioned" — the A6 + caller must treat empty-with-no-events differently only if it also has no + sim_now-tagged events at all across the whole run (checked once, not per + trainer). + """ + pts: list = [] + for e in sorted( + avail_change_events, key=lambda x: (x.get("round", 0), x.get("ts", 0.0)) + ): + t = e.get("sim_now") + state = e.get("new_state") + if t is None or state is None: + continue + if pts and pts[-1][0] == t: + pts[-1] = (t, state) + else: + pts.append((t, state)) + return pts + + +def build_observed_timeline_from_agg_belief(events: list) -> list: + """Sorted [(observed_at, state), ...] from one trainer's own + agg_belief_change telemetry (Batch 3 T3.3), already filtered by the + caller to a single end_id and checkpoint ("selection" | "commit"). + + ``observed_at`` is the trace-time-basis clock the belief was read at + (vclock seconds / wall-elapsed since the shared origin) -- same role as + avail_change's ``sim_now`` in build_observed_timeline_from_avail_change, + just a different telemetry stream (aggregator belief vs. trainer's own + self-report). + """ + pts: list = [] + for e in sorted( + events, key=lambda x: (x.get("round", 0), x.get("observed_at", 0.0)) + ): + t = e.get("observed_at") + state = e.get("state") + if t is None or state is None: + continue + if pts and pts[-1][0] == t: + pts[-1] = (t, state) + else: + pts.append((t, state)) + return pts + + +def selection_run_span(selection_events: list, mode: str) -> float: + """Max observed time across all selection events. + + Unlike run_span() (which needs a built per-trainer avl_state series), + this only needs the events' own ts/vclock_now -- usable by checks that + just need the run's overall time horizon and nothing about per-trainer + avail state (e.g. A6, which reads its own per-trainer state from + avail_change telemetry instead). Same t = vclock_now / (ts - t0) + time-base convention as build_trainer_state_series. + """ + real_ts = [e.get("ts") for e in selection_events if e.get("ts") is not None] + t0 = min(real_ts) if (mode == "real" and real_ts) else 0.0 + times = [ + t for t in (_event_time(e, mode, t0) for e in selection_events) + if t is not None + ] + return max(times) if times else 0.0 + + +def run_span(series: dict) -> float: + """Max observed t across all trainers — the run's own time horizon.""" + return max((pts[-1][0] for pts in series.values() if pts), default=0.0) + + +def state_fractions(series: dict, t_end: Optional[float] = None) -> dict: + """Per-trainer {state: fraction_of_span} from a forward-filled series. + + Dwell-time integration: each sample's state holds until the next sample + (or `t_end` for the trailing segment, default = the trainer's own last + sample — i.e. no tail credited beyond its last observation). Trainers + with < 2 samples are omitted (no observed dwell to integrate). Per-trainer + vectors sum to 1. + """ + out: dict = {} + for end_id, pts in series.items(): + if len(pts) < 2: + continue + durations: dict = {} + for (t_a, s_a), (t_b, _) in zip(pts, pts[1:]): + durations[s_a] = durations.get(s_a, 0.0) + max(0.0, t_b - t_a) + last_t, last_s = pts[-1] + end_t = t_end if t_end is not None else last_t + if end_t > last_t: + durations[last_s] = durations.get(last_s, 0.0) + (end_t - last_t) + total = sum(durations.values()) + if total <= 0: + continue + out[end_id] = {s: d / total for s, d in durations.items()} + return out + + +def total_variation_distance(a: dict, b: dict) -> float: + """0.5 * Σ_s |a_s - b_s| over the union of states; 0 if identical.""" + keys = set(a) | set(b) + return 0.5 * sum(abs(a.get(k, 0.0) - b.get(k, 0.0)) for k in keys) diff --git a/lib/python/examples/async_cifar10/scripts/parity/checks.py b/lib/python/examples/async_cifar10/scripts/parity/checks.py index effc43fc7..984939709 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/checks.py +++ b/lib/python/examples/async_cifar10/scripts/parity/checks.py @@ -26,6 +26,22 @@ import re import statistics from pathlib import Path + +from .avail_state_series import ( + build_observed_timeline_from_agg_belief, + build_observed_timeline_from_avail_change, + build_trainer_state_series, + run_span, + selection_run_span, + state_fractions, + total_variation_distance, +) +from .ground_truth import ( + by_short_id, + expected_send_gate_wait, + state_fractions_over_range, + transitions_in_range, +) from typing import Optional @@ -190,6 +206,9 @@ def load_agg_jsonl(path: str) -> dict: eval_commits: list = [] agg_evals: list = [] residence: list = [] + withheld_deliveries: list = [] + abandon_timeouts: list = [] + agg_belief_changes: list = [] with open(path) as f: for line in f: line = line.strip() @@ -199,6 +218,12 @@ def load_agg_jsonl(path: str) -> dict: ev = e.get("event") if ev == "selection" and e.get("task") == "train": selection_train.append(e) + elif ev == "withheld_delivery": + withheld_deliveries.append(e) + elif ev == "abandon_timeout": + abandon_timeouts.append(e) + elif ev == "agg_belief_change": + agg_belief_changes.append(e) elif ev == "agg_round": # Eval commits emit event=agg_round (tagged task=eval) so U6/U6e can # read their commit timeliness, but they carry no agg_goal_count and @@ -218,12 +243,22 @@ def load_agg_jsonl(path: str) -> dict: eval_commits.sort(key=lambda x: (x["round"], x["ts"])) agg_evals.sort(key=lambda x: x["round"]) residence.sort(key=lambda x: (x["round"], x["ts"])) + withheld_deliveries.sort(key=lambda x: (x.get("round", 0), x.get("ts", 0))) + abandon_timeouts.sort(key=lambda x: (x.get("round", 0), x.get("ts", 0))) + agg_belief_changes.sort(key=lambda x: (x.get("round", 0), x.get("observed_at", 0.0))) return { "selection_train": selection_train, "agg_rounds": agg_rounds, "eval_commits": eval_commits, "agg_evals": agg_evals, "residence": residence, + # Stage C availability events (sim-only): the send-gate late stale + # deliveries and the 90s vclock abandons. + "withheld_deliveries": withheld_deliveries, + "abandon_timeouts": abandon_timeouts, + # Batch 3 T3.3: aggregator belief-tracking (commit checkpoint only — + # the selection checkpoint is already in selection_train.per_trainer.avl_state). + "agg_belief_changes": agg_belief_changes, } @@ -241,6 +276,7 @@ def load_trainer_jsonl_dir(telemetry_dir: Optional[str]) -> dict: for f in sorted(d.glob("trainer_*.jsonl")): short_id = f.stem[-4:] task_recv_evs, trainer_round_evs, task_send_evs = [], [], [] + avail_change_evs: list = [] with open(f) as fp: for line in fp: line = line.strip() @@ -257,10 +293,15 @@ def load_trainer_jsonl_dir(telemetry_dir: Optional[str]) -> dict: trainer_round_evs.append(e) elif ev == "task_send": task_send_evs.append(e) + elif ev == "avail_change": + # Trainer availability transitions (A4 duty-cycle). Previously + # dropped here, so duty_cycle_parity was permanently SKIP. + avail_change_evs.append(e) result[short_id] = { "task_recv": task_recv_evs, "trainer_round": trainer_round_evs, "task_send": task_send_evs, + "avail_change": avail_change_evs, } return result @@ -1020,7 +1061,11 @@ def vals(agg_rounds): } -NEAR_ZERO_LAG_S = 0.05 # U6: both-modes mean lag <= this ⇒ immediate commit, KS uninformative +NEAR_ZERO_LAG_S = 0.10 # U6: both-modes mean lag <= this ⇒ immediate commit, KS uninformative +# 0.10s (raised from 0.05): sim with active availability windows sees ~70ms mean lag +# from carry-over burst commits right after an unavailability window ends — vclock +# advances through the stale queue before the next fresh update, inflating the per-round +# mean slightly above the old 50ms guard without indicating a real past-dating bug. def commit_visibility_parity(real: dict, sim: dict, warn_ks: float = 0.2, @@ -1034,20 +1079,37 @@ def commit_visibility_parity(real: dict, sim: dict, warn_ks: float = 0.2, the barrier wait, matching in both. Either way fidelity = sim dist == real dist, so we KS the two and also flag the mean gap. Upstream of staleness: a sim that commits updates late (past-dating) inflates staleness downstream. + + A withheld-then-delivered update (D.1/C.2) commits at + ``delivery_ts = max(sct, next_avail_ts)`` by construction (the down-window + delay), so its ``update_visibility_lag_s`` for that one commit equals the + delay already measured by the dedicated ``withheld_delivery`` rung + (`delivery_ts - sct`). Folding it into this distribution double-counts the + same signal and inflates the mean with an outlier this metric isn't + measuring (commit timeliness) — excluded by (round, end) cross-reference + against ``withheld_deliveries`` so the two rungs stay separable, per the + intended "withheld past-dating bucket" split (Stage C invariant notes). """ - def vals(agg_rounds): + def vals(agg_rounds, withheld_keys): out = [] for e in agg_rounds: v = e.get("update_visibility_lag_s") if v is None: continue + ends = e.get("contributing_trainers") or [] + if any((e.get("round"), end) in withheld_keys for end in ends): + continue if isinstance(v, (int, float)): out.append(float(v)) else: out.extend(float(x) for x in v if x is not None) return out - rv, sv = vals(real["agg_rounds"]), vals(sim["agg_rounds"]) + def withheld_keys(loaded): + return {(e.get("round"), e.get("end_id")) for e in loaded.get("withheld_deliveries", [])} + + rv = vals(real["agg_rounds"], withheld_keys(real)) + sv = vals(sim["agg_rounds"], withheld_keys(sim)) if not rv or not sv: return {"ok": True, "tier": "DIST", "skipped": True, "reason": "update_visibility_lag_s absent in one mode " @@ -2330,9 +2392,15 @@ def _traj(sel): def duty_cycle_parity(real_trainers: dict, sim_trainers: dict) -> dict: """A4 [DIST]: per-trainer availability duty-cycle parity. - Requires avail_change telemetry (on/off transitions per trainer), which - the current loader does not surface — SKIP placeholder per the append-only - growth rule; activates automatically once that telemetry exists. + Requires avail_change telemetry (per-trainer state transitions), now surfaced + by load_trainer_jsonl_dir. SKIP when absent (e.g. v1 oracular runs with + client_notify OFF, where the aggregator reads the trace directly and the + trainer emits no transitions — the trace-grounded A4b validator is the right + check there; see UNAVAILABILITY_DESIGN.md Stage B). + + NOTE (limitation, intentional): this counts the fraction of TRANSITIONS whose + new_state is AVL_*, not time-in-state. A duration-weighted duty cycle is the + A4b trace-vs-dispatch validator (to add with run data). """ def _has_avail_change(tr): return any(d.get("avail_change") for d in tr.values()) @@ -2340,14 +2408,15 @@ def _has_avail_change(tr): if not _has_avail_change(real_trainers) and not _has_avail_change(sim_trainers): return {"ok": True, "tier": "DIST", "status": "SKIP", "note": "avail_change telemetry not available; A4 inactive"} - # Telemetry present: compare per-trainer on-fraction. + # Telemetry present: compare per-trainer on-fraction. avail_change events carry + # {old_state, new_state} (build_avail_change), so "available" = new_state is AVL_*. def _on_frac(tr): out = {} for tid, d in tr.items(): evs = d.get("avail_change", []) if not evs: continue - on = sum(1 for e in evs if e.get("available")) + on = sum(1 for e in evs if str(e.get("new_state", "")).startswith("AVL")) out[tid] = on / len(evs) return out @@ -2359,6 +2428,870 @@ def _on_frac(tr): "max_dutycycle_diff": round(max_diff, 3), "n_trainers": len(keys)} +def duration_duty_cycle_parity(real: dict, sim: dict, + mean_tol: float = 0.05, + within_tau: float = 0.10, + frac_pass_tol: float = 0.95) -> dict: + """A4dur [DIST]: duration-weighted duty-cycle parity, real vs sim (C.6.3). + + Replaces A4's transition-FRACTION counting (a bare max over `avail_change` + — brittle, and blind in pure-oracular mode; see Dead-ends §9) with time- + IN-STATE: per-trainer {state: fraction_of_run} from `trainer_state_series` + (C.6.2, reading the C.6.1 per-trainer `avl_state` on selection events), + dwell-integrated over each mode's own run span. Per-trainer error = total- + variation distance between the real/sim fraction vectors. + + Population rollup is a DISTRIBUTION (mean/p50/p90/p99 + + frac_trainers_within_tol), not a single number — a systematic small drift + (mean) and a real diverging subset (tail) are different failure modes; + neither alone is robust (mirrors U6's "distribution + robust summary" + precedent already in this checker). + + Pass rule: mean_err <= mean_tol AND frac_within_tol >= frac_pass_tol — two + independent conditions for the two failure modes above. Kept alongside the + existing transition-count `duty_cycle_parity` (A4), which catches a + different failure mode (transitions stopping entirely) cheaply. SKIP if + either mode has no per-trainer avl_state samples (gate off, or telemetry + predates C.6.1). + """ + r_series = build_trainer_state_series(real["selection_train"], mode="real") + s_series = build_trainer_state_series(sim["selection_train"], mode="sim") + if not r_series or not s_series: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no per-trainer avl_state in selection telemetry " + "(gate off, or predates C.6.1)"} + + r_frac = state_fractions(r_series, t_end=run_span(r_series)) + s_frac = state_fractions(s_series, t_end=run_span(s_series)) + common = sorted(set(r_frac) & set(s_frac)) + if not common: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no trainers with >=2 avl_state samples in both modes"} + + errs = {tid: total_variation_distance(r_frac[tid], s_frac[tid]) for tid in common} + err_vals = list(errs.values()) + mean_err = sum(err_vals) / len(err_vals) + frac_within_tol = sum(1 for e in err_vals if e <= within_tau) / len(err_vals) + worst = sorted(errs.items(), key=lambda kv: -kv[1])[:5] + return { + "ok": mean_err <= mean_tol and frac_within_tol >= frac_pass_tol, + "tier": "DIST", + "n_trainers": len(common), + "mean_err": round(mean_err, 4), + "p50_err": round(percentile(err_vals, 50), 4), + "p90_err": round(percentile(err_vals, 90), 4), + "p99_err": round(percentile(err_vals, 99), 4), + "frac_within_tol": round(frac_within_tol, 3), + "within_tau": within_tau, + "mean_tol": mean_tol, + "frac_pass_tol": frac_pass_tol, + "worst_trainers": [{"end": short(tid), "err": round(e, 4)} for tid, e in worst], + } + + +def withheld_delivery_parity(real: dict, sim: dict) -> dict: + """withheld_delivery [NEW, sim characterization]: send-gated updates deliver + late and STALE, never before completion. + + The C.2 send-gate holds an update whose trainer is UN_AVL at completion (sct) + and re-commits it at delivery_ts = max(sct, next_avail) — stale, never + discarded. This rung asserts the STRUCTURAL invariants of that path (the + cross-mode staleness magnitude is owned by the `staleness` rung / U3): + * delivery_ts >= sct — never deliver before completion (no past-dating), + * delay_s = delivery_ts - sct >= 0, + * staleness >= 0. + Real mode emits no withheld_delivery (its trainer send-gate is a different + mechanism), so this is sim-only; SKIP when the gate is off (no events). + """ + evs = sim.get("withheld_deliveries", []) or [] + if not evs: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no withheld_delivery events (gate off or no withholds)"} + delays, stales, accepted, bad = [], [], 0, [] + for e in evs: + sct, dts = e.get("sct"), e.get("delivery_ts") + dly, st = e.get("delay_s"), e.get("staleness") + if dly is None and sct is not None and dts is not None: + dly = float(dts) - float(sct) + if sct is not None and dts is not None and float(dts) + 1e-6 < float(sct): + bad.append({"end": e.get("end_id"), "sct": sct, "delivery_ts": dts}) + if dly is not None: + delays.append(float(dly)) + if float(dly) < -1e-6: + bad.append({"end": e.get("end_id"), "delay_s": dly}) + if st is not None: + stales.append(int(st)) + if int(st) < 0: + bad.append({"end": e.get("end_id"), "staleness": st}) + if e.get("accepted"): + accepted += 1 + dmean, _ = mean_std(delays) if delays else (float("nan"), 0.0) + smean, _ = mean_std(stales) if stales else (float("nan"), 0.0) + return { + "ok": len(bad) == 0, + "tier": "DIAG", + "n_withheld": len(evs), + "mean_delay_s": round(dmean, 1) if delays else None, + "p95_delay_s": round(percentile(delays, 95), 1) if delays else None, + "mean_staleness": round(smean, 2) if stales else None, + "accept_frac": round(accepted / len(evs), 3), + "violations": bad[:10], + } + + +def commit_promptness_parity(sim: dict, early_tol_s: float = 1.0, + late_slack_tol_s: float = 30.0) -> dict: + """K11 [INV]: per-event hard invariant -- actual commit time vs. + earliest-legally-committable time (Batch 3 T3.5, "Pillar 3" aggregator + half — general/mechanism-agnostic, not availability-specific in principle, + though today's only telemetry source for it is the availability send-gate). + + Scope: the withheld-then-delivered population only (`withheld_deliveries`), + not every commit. A NORMAL (non-gated) commit's earliest-legally-committable + time is trivially its own `sct` — checking `actual_commit_ts >= sct` there + would just be re-measuring ordinary round-batching queue depth (multiple + commits sharing one monotonic vclock inside a round always show positive + "slack" against their own sct, by construction of `_advance_sim_clock`'s + max()), not a real promptness bug. The withheld population is where + `earliest_legally_committable_time` is a MEANINGFUL constraint distinct + from `sct` — and, per `compute_delivery_ts` (client_availability.py, T3.5 + finding, see UNAVAILABILITY_DESIGN.md), `delivery_ts` there already IS + `earliest_legally_committable_time` (the max of whichever gates are + active, generalized as far as v1 has more than one gate type to take a + max over) — no separate bookkeeping field was needed, just an + `actual_commit_ts` stamp to compare it against. + + `commit_slack_s = actual_commit_ts - delivery_ts`. Two distinct failure + modes, reported separately: + * EARLY (`slack < -early_tol_s`): committed before it was legally + available — a correctness bug (past-dating), same class as + `withheld_delivery`'s `delivery_ts >= sct` check but stricter (against + the ACTUAL commit instant, not just the registered delivery_ts). + * LATE (`slack > late_slack_tol_s`): held longer than the gate strictly + required — a promptness/scheduling bug (e.g. coarse reinjection + polling — `_sim_reinject_ready_withheld` only runs once per + commit-loop invocation). + Both gate `ok`; `late_slack_tol_s`'s default is a starting point pending + real-run calibration (Phase 6), same caveat as A6/A7/A8's DIST tolerances. + + Sim-only: real emits no `withheld_delivery` at all (its trainer send-gate + is a different, trainer-side mechanism — see `withheld_delivery_parity`). + SKIP when no withheld_delivery event carries `actual_commit_ts` (gate off, + no withholds, or telemetry predates T3.5). + """ + evs = sim.get("withheld_deliveries", []) or [] + slacks: list = [] + early_violations: list = [] + late_violations: list = [] + for e in evs: + act, dts = e.get("actual_commit_ts"), e.get("delivery_ts") + if act is None or dts is None: + continue + slack = float(act) - float(dts) + slacks.append(slack) + if slack < -early_tol_s: + early_violations.append({"end": e.get("end_id"), "slack_s": round(slack, 2)}) + elif slack > late_slack_tol_s: + late_violations.append({"end": e.get("end_id"), "slack_s": round(slack, 2)}) + + if not slacks: + return {"ok": True, "tier": "INV", "status": "SKIP", + "note": "no withheld_delivery events with actual_commit_ts " + "(gate off, no withholds, or predates T3.5)"} + + mean_slack, _ = mean_std(slacks) + return { + "ok": not early_violations and not late_violations, + "tier": "INV", + "n_events": len(slacks), + "mean_slack_s": round(mean_slack, 2), + "max_slack_s": round(max(slacks), 2), + "min_slack_s": round(min(slacks), 2), + "n_early_violations": len(early_violations), + "n_late_violations": len(late_violations), + "early_tol_s": early_tol_s, + "late_slack_tol_s": late_slack_tol_s, + "early_violations": early_violations[:10], + "late_violations": late_violations[:10], + } + + +def abandon_timeout_parity(real: dict, sim: dict, + threshold_s: float = 90.0, + wall_leak_ceiling_s: float = 1e7) -> dict: + """abandon_timeout [NEW, CONTROL]: the 90s abandon fires on the VCLOCK. + + Each C.3 abandon (``reason="abandon_90s_vclock"``) frees a stalled + in-flight slot at age >= SEND_TIMEOUT_WAIT_S. Control purpose + (Challenge 2): the deadline must be measured on the vclock, not the wall — + a wall-clock leak surfaces as an age in epoch-scale seconds (~1.7e9) + instead of sim-seconds. Fails loudly if any C.3 age is wall-scale or below + the threshold. + + D.1 boundary evictions (``reason="aware_boundary_eviction"``) are a + *different* mechanism — they free the slot proactively at the next + selection boundary specifically to avoid the 90s wait, so a low age is + their correct, expected behavior, not a violation. They're tracked + separately and never measured against ``threshold_s``; only a wall-clock + leak (age epoch-scale) would be a bug for them too. + + Sim-only (real uses the wall selector abandon); SKIP when no abandons + fired. + """ + evs = sim.get("abandon_timeouts", []) or [] + if not evs: + return {"ok": True, "tier": "INV", "status": "SKIP", + "note": "no abandon_timeout events (gate off or none stalled)"} + + def _age(e): + age = e.get("age_s") + if age is None: + sst, now = e.get("sim_send_ts"), e.get("vclock_now") + if sst is not None and now is not None: + age = float(now) - float(sst) + return None if age is None else float(age) + + c3_ages, d1_ages, wall_leak, below = [], [], [], [] + for e in evs: + age = _age(e) + if age is None: + continue + if age >= wall_leak_ceiling_s: + wall_leak.append(e.get("end_id")) + continue + if e.get("reason") == "aware_boundary_eviction": + d1_ages.append(age) + else: + c3_ages.append(age) + if age + 1e-6 < threshold_s: + below.append({"end": e.get("end_id"), "age_s": round(age, 1)}) + c3_mean, _ = mean_std(c3_ages) if c3_ages else (float("nan"), 0.0) + out = { + "ok": not wall_leak and not below, + "tier": "INV", + "n_abandon": len(c3_ages), + "mean_age_s": round(c3_mean, 1) if c3_ages else None, + "max_age_s": round(max(c3_ages), 1) if c3_ages else None, + "threshold_s": threshold_s, + "wall_leak_ends": wall_leak[:10], + "below_threshold": below[:10], + } + if d1_ages: + d1_mean, _ = mean_std(d1_ages) + out["n_aware_boundary_eviction"] = len(d1_ages) + out["aware_boundary_eviction_mean_age_s"] = round(d1_mean, 1) + out["aware_boundary_eviction_max_age_s"] = round(max(d1_ages), 1) + if wall_leak: + out["note"] = "WALL-CLOCK LEAK: abandon age is epoch-scale; vclock not used" + return out + + +def starvation_advance_parity(real: dict, sim: dict, + jump_factor: float = 5.0) -> dict: + """starvation_advance [NEW, DIAG, Stage F]: vclock-advance events under scarcity. + + Stage F replaces wall-sleeping with vclock-advances when no trainers are + selectable. This rung detects such advances from the sim's agg_round timeline: + a vclock jump between consecutive rounds that exceeds ``jump_factor × mean_advance`` + suggests a starvation advance fired (the round completed without a commit). + + SKIP when the gate is off (no avail events) or when fewer than 3 rounds are + present (too few points to establish a baseline). PASS when no anomalous jumps + are detected (or syn_0 / 100%-availability runs where F never fires). + """ + rounds = sim.get("agg_rounds", []) + if not rounds: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no agg_round events"} + # Check if availability gate was active: any withheld_deliveries or + # abandon_timeouts events indicate the sim-unavailability path ran. + gate_active = bool( + sim.get("withheld_deliveries") or sim.get("abandon_timeouts") + or any(e.get("avail_composition") for e in sim.get("selection_train", [])) + ) + if not gate_active: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "availability gate off (syn_0 / 100%-avail)"} + vclocks = sorted( + [float(r["vclock_now"]) for r in rounds if r.get("vclock_now") is not None] + ) + if len(vclocks) < 3: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": f"only {len(vclocks)} vclock points; need ≥3"} + gaps = [vclocks[i + 1] - vclocks[i] for i in range(len(vclocks) - 1)] + mean_gap = sum(gaps) / len(gaps) + threshold = jump_factor * mean_gap + jumps = [(i, g) for i, g in enumerate(gaps) if g > threshold] + return { + "ok": True, # informational only — starvation advances are expected + "tier": "DIAG", + "n_rounds": len(vclocks), + "mean_advance_s": round(mean_gap, 2), + "jump_threshold_s": round(threshold, 2), + "n_starvation_jumps": len(jumps), + "max_jump_s": round(max(g for _, g in jumps), 1) if jumps else 0.0, + "note": (f"{len(jumps)} starvation advance(s) detected " + f"(jump > {threshold:.1f}s = {jump_factor}× mean)") if jumps + else "no starvation advances detected", + } + + +def eligible_pool_reduction_parity(real: dict, sim: dict, + tol_rel: float = 0.25) -> dict: + """eligible_pool_reduction [NEW, DIAG]: availability shrinks the eligible pool + by the same amount in both modes. + + Complements A2 (absolute num_eligible) by isolating the REDUCTION + (num_candidates - num_eligible) — what availability + in-flight remove from + the pool. Under unavailability this is > 0 and should track across modes; at + 100% availability it is ~the in-flight count and A2 already covers it. + """ + def _red(sel): + out = [] + for e in sel: + nc, ne = e.get("num_candidates"), e.get("num_eligible") + if nc is not None and ne is not None: + out.append(max(0, int(nc) - int(ne))) + return out + + rr, sr = _red(real["selection_train"]), _red(sim["selection_train"]) + if not rr or not sr: + return {"ok": True, "tier": "DIAG", "status": "SKIP", + "note": "no num_candidates/num_eligible to compute reduction"} + rm, sm = sum(rr) / len(rr), sum(sr) / len(sr) + ref = max(rm, sm, 1.0) + rel = abs(rm - sm) / ref + return {"ok": rel <= tol_rel, "tier": "DIAG", + "real_mean_reduction": round(rm, 1), "sim_mean_reduction": round(sm, 1), + "rel_diff": round(rel, 3), "tol_rel": tol_rel} + + +def state_timeline_agreement(real: dict, sim: dict, + n_bins: int = 20, + tol: float = 0.95) -> dict: + """A5 [DIST]: per-(trainer, t) avl_state agreement between real and sim. + + Real and sim both read availability from the SAME trace, so at any + normalised time t ∈ [0, 1], a trainer's avl_state should be identical in + both runs. Forward-fills the per-trainer series (from C.6.1 avl_state on + selection events) at n_bins equally-spaced normalised time points, compares + the result per (trainer, bin), and reports match_frac. + + Time is normalised within each mode (t / run_span) so wall-time vs vclock + differences are removed before comparison. SKIP if either mode has no + per-trainer avl_state, or if the two modes share no common trainers. + PASS when match_frac >= tol (default 0.95). + """ + r_series = build_trainer_state_series(real["selection_train"], mode="real") + s_series = build_trainer_state_series(sim["selection_train"], mode="sim") + if not r_series or not s_series: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no per-trainer avl_state in selection telemetry " + "(gate off, or predates C.6.1)"} + + r_span = run_span(r_series) + s_span = run_span(s_series) + if r_span <= 0 or s_span <= 0: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "degenerate run span (zero duration)"} + + common = sorted(set(r_series) & set(s_series)) + if not common: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no common trainers between real and sim series"} + + def _state_at_frac(pts, frac, span): + """Forward-fill: trainer state at absolute time frac*span.""" + target = frac * span + state = None + for t, s in pts: + if t <= target: + state = s + else: + break + return state + + bin_fracs = [(b + 0.5) / n_bins for b in range(n_bins)] + matched = 0 + total = 0 + mismatched: list = [] + + for tid in common: + r_pts, s_pts = r_series[tid], s_series[tid] + if not r_pts or not s_pts: + continue + for frac in bin_fracs: + rs = _state_at_frac(r_pts, frac, r_span) + ss = _state_at_frac(s_pts, frac, s_span) + if rs is None or ss is None: + continue + total += 1 + if rs == ss: + matched += 1 + elif len(mismatched) < 5: + mismatched.append({ + "trainer": short(tid), + "frac": round(frac, 2), + "real": rs, "sim": ss, + }) + + if total == 0: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no (trainer, bin) pairs with data in both modes"} + + match_frac = matched / total + return { + "ok": match_frac >= tol, + "tier": "DIST", + "match_frac": round(match_frac, 4), + "matched": matched, + "total": total, + "tol": tol, + "n_bins": n_bins, + "n_trainers": len(common), + "mismatched_examples": mismatched, + } + + +def _match_transitions(gt_transitions: list, obs_transitions: list, + lag_tol_s: float) -> tuple: + """Greedy in-order pairing of ground-truth vs observed transition events. + + Both lists are chronological per-trainer transition sequences, so an + accurate observer's transitions should appear in the same order as ground + truth's (states alternate the same way in both). Walks ground truth in + order; each gt event consumes the *next* unconsumed observed event if it + has the same new_state and lands within lag_tol_s, else it's counted + missed. Any observed events never consumed are spurious. Returns + (lags, n_missed, n_spurious). + """ + lags: list = [] + missed = 0 + j = 0 + for gt_t, gt_s in gt_transitions: + if j < len(obs_transitions): + obs_t, obs_s = obs_transitions[j] + if obs_s == gt_s and abs(obs_t - gt_t) <= lag_tol_s: + lags.append(abs(obs_t - gt_t)) + j += 1 + continue + missed += 1 + spurious = len(obs_transitions) - j + return lags, missed, spurious + + +def _pad_tail(obs: list, span: float) -> list: + """Ensure a (t, state) series reaches `span` with >= 2 points. + + state_fractions() drops single-point series outright (no dwell segment to + integrate) -- append a synthetic tail point at the run's own span so a + trainer with exactly one observed point (its only transition landed at + t=0, or it never changed again after one early change) still contributes + a full-span dwell estimate instead of being silently excluded. + """ + if not obs: + return obs + if obs[-1][0] < span: + return obs + [(span, obs[-1][1])] + return obs + + +def _covered_intervals(obs: list, t_start: float, t_end: float, + max_gap_s: float) -> list: + """[(a, b, state), ...] -- the union of windows each observation + vouches for: itself forward to the next observation, or `max_gap_s` past + itself, whichever is sooner (capped at `t_end`). A gap longer than + `max_gap_s` on both sides of a given instant has NO covering + observation and is excluded from the returned intervals entirely. + + This is the interior-gap generalization of the tail truncation below: a + sparse, event-triggered observation stream (A7 commit-checkpoint) can't + be blamed for silence beyond its own validity window, whether that + silence is at the end of the run or between two observations. The + caller uses these same intervals to restrict BOTH the duration-weighted + TVD score and the missed/spurious-transition diagnostic, so a + transition with no nearby observation on either side is consistently + excluded from both (not scored as an error, not flagged as missed) -- + it is simply not fair to score what nothing was there to observe. + """ + pts = [p for p in obs if t_start <= p[0] <= t_end] + intervals: list = [] + for i, (t_a, s_a) in enumerate(pts): + nxt = pts[i + 1][0] if i + 1 < len(pts) else t_end + seg_end = min(nxt, t_a + max_gap_s, t_end) + if seg_end > t_a: + intervals.append((t_a, seg_end, s_a)) + return intervals + + +def _covered_fractions(intervals: list, gt) -> tuple: + """Duration-weighted {state: fraction} for obs and gt, integrated only + over `intervals` (see `_covered_intervals`).""" + obs_durations: dict = {} + gt_durations: dict = {} + for t_a, seg_end, s_a in intervals: + obs_durations[s_a] = obs_durations.get(s_a, 0.0) + (seg_end - t_a) + for s, frac in state_fractions_over_range(gt, t_a, seg_end).items(): + gt_durations[s] = gt_durations.get(s, 0.0) + frac * (seg_end - t_a) + obs_total = sum(obs_durations.values()) + gt_total = sum(gt_durations.values()) + if obs_total <= 0 or gt_total <= 0: + return None, {} + obs_frac = {s: d / obs_total for s, d in obs_durations.items()} + gt_frac = {s: d / gt_total for s, d in gt_durations.items()} + return obs_frac, gt_frac + + +def _fidelity_score(raw_obs: list, gt, span: float, lag_tol_s: float = 30.0, + seed_state: Optional[str] = None, + extrapolate_tail: bool = True, + max_gap_s: Optional[float] = None) -> Optional[tuple]: + """Shared A6/A7 core: one trainer's duration-weighted TVD vs ground truth, + plus event-level diagnostics (missed/spurious transitions, lags) from a + greedy in-order match against the raw trace's own transition points. + + `seed_state`: prepend (0.0, seed_state) when raw_obs doesn't already + start at/before t=0 -- the known initial-belief anchor (A6: a trainer + inits AVL_TRAIN, see main.py; A7 commit-checkpoint belief has no such + anchor -- a trainer with zero commits has no belief to seed, pass None). + Without a seed, the window before the first observation is EXCLUDED from + both sides of the comparison (t_start = first observed t) rather than + penalizing a belief that couldn't exist yet — a commit-checkpoint belief + only starts at the first commit, always > 0, so scoring against [0, span) + would otherwise blame a fixed, unavoidable "missing prefix" as if it were + genuine drift. + + `extrapolate_tail`: when True (A6, A7-selection -- continuously/densely + refreshed observation streams), `_pad_tail` carries the last observation + forward to `span`, matching the historical behavior. When False (A7 + commit-checkpoint -- Batch 4 finding, UNAVAILABILITY_DESIGN.md), the + window is instead truncated to `[t_start, last observed t]`: "commit" is + an inherently event-triggered sample, not a continuous one, and a + trainer that legitimately stops committing (typically because it went + UN_AVL -- exactly the state this check cares about) has no way to record + a belief for the un-observed tail. Extrapolating "still believed X" + across that silence blamed the *absence of a later commit* as if it were + a stale belief, systematically worst for the trainers this check most + wants to catch. Symmetric with the existing start-side truncation above. + + `max_gap_s`: when set (A7 commit-checkpoint -- Batch 4 live-run finding, + UNAVAILABILITY_DESIGN.md), extends the same "don't extrapolate a sparse + observation" reasoning to INTERIOR gaps, not just the tail: each + observation only vouches for its own state up to `max_gap_s` past + itself, not all the way to the next commit (subsuming and superseding + `extrapolate_tail`'s truncation -- the last observation's own + `max_gap_s` window already bounds the tail the same way). Without this, + a trainer that commits correctly at t=100 (AVL_TRAIN) and again + correctly at t=590 (AVL_TRAIN) but flips through UN_AVL and back in + between (e.g. [200,400)) was scored as if it believed AVL_TRAIN for the + whole [100,590) gap -- penalizing the *absence of a mid-gap commit*, the + same class of error the tail fix already exempts. The missed/spurious + transition diagnostic is filtered the same way: a ground-truth + transition with no covering observation window on either side is + excluded from both the score AND the diagnostic, not scored as 0 error + while simultaneously flagged "missed" (self-contradictory). `None` + (default) preserves the historical hold-until-next-observation behavior + for A6 and A7-selection, both dense enough that this rarely matters and + byte-identical scoring is wanted. + + Returns None if there's nothing to score (empty input, or ground-truth / + observed fraction computation comes up empty). + """ + if not raw_obs: + return None + obs = raw_obs + t_start = 0.0 + if seed_state is not None and raw_obs[0][0] > 0.0: + obs = [(0.0, seed_state)] + raw_obs + elif seed_state is None and raw_obs[0][0] > 0.0: + t_start = raw_obs[0][0] + if max_gap_s is None: + t_end = span if extrapolate_tail else min(span, obs[-1][0]) + obs = _pad_tail(obs, t_end) + obs_frac = state_fractions({"_": obs}, t_end=t_end).get("_") + gt_frac = state_fractions_over_range(gt, t_start, t_end) + gt_transitions = transitions_in_range(gt, t_start, t_end) + else: + t_end = span + intervals = _covered_intervals(obs, t_start, t_end, max_gap_s) + obs_frac, gt_frac = _covered_fractions(intervals, gt) + gt_transitions = [ + (ts, s) for ts, s in transitions_in_range(gt, t_start, t_end) + if any(a <= ts <= b for a, b, _ in intervals) + ] + if obs_frac is None or not gt_frac: + return None + tvd = total_variation_distance(obs_frac, gt_frac) + lags, missed, spurious = _match_transitions(gt_transitions, raw_obs, lag_tol_s) + return tvd, lags, missed, spurious + + +def _fidelity_result(errs: dict, n_missed: int, n_spurious: int, max_lag: float, + mode: str, mean_tol: float, within_tau: float, + frac_pass_tol: float, skip_note: str) -> dict: + """Shared A6/A7 result shape: DIST-tier population rollup (mean/p50/p90/ + p99 + frac_within_tol) over a {short_id: tvd_error} map, same pass rule + and worst-trainers reporting for both rungs.""" + if not errs: + return {"ok": True, "tier": "DIST", "status": "SKIP", "note": skip_note} + err_vals = list(errs.values()) + mean_err = sum(err_vals) / len(err_vals) + frac_within_tol = sum(1 for e in err_vals if e <= within_tau) / len(err_vals) + worst = sorted(errs.items(), key=lambda kv: -kv[1])[:5] + return { + "ok": mean_err <= mean_tol and frac_within_tol >= frac_pass_tol, + "tier": "DIST", + "mode": mode, + "n_trainers": len(errs), + "mean_err": round(mean_err, 4), + "p50_err": round(percentile(err_vals, 50), 4), + "p90_err": round(percentile(err_vals, 90), 4), + "p99_err": round(percentile(err_vals, 99), 4), + "frac_within_tol": round(frac_within_tol, 3), + "within_tau": within_tau, + "mean_tol": mean_tol, + "frac_pass_tol": frac_pass_tol, + "n_missed_transitions": n_missed, + "n_spurious_transitions": n_spurious, + "max_lag_s": round(max_lag, 1), + "worst_trainers": [{"end": tid, "err": round(e, 4)} for tid, e in worst], + } + + +def trainer_trace_fidelity_parity(trainer_dict: dict, selection_events: list, + mode: str, ground_truth: Optional[dict], + mean_tol: float = 0.05, + within_tau: float = 0.10, + frac_pass_tol: float = 0.95, + lag_tol_s: float = 30.0) -> dict: + """A6 [DIST]: per-trainer observed-vs-ground-truth-trace fidelity, ONE + mode at a time (Batch 3 T3.2 — "Pillar 1"). + + Unlike A4dur/A5 (real vs sim compared *to each other*), this compares a + single mode's own trainer-side telemetry (avail_change, i.e. what the + trainer itself believes/logs about its availability) against the raw + trace file directly — the absolute check that would have caught + Challenges §5 item 20 (real trainers running their send-gate against a + trivial always-available trace) on its own, without needing a companion + sim run to diff against. + + SKIP if no ground-truth trace was resolved for this run (predates T3.2 / + aggregator_config.json missing), the run span is degenerate, or no + trainer carries sim_now-tagged avail_change telemetry (predates T3.2). + """ + if not ground_truth: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no ground-truth trace resolved for this run"} + + span = selection_run_span(selection_events, mode) + if span <= 0: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "degenerate run span"} + + gt_by_short = by_short_id(ground_truth) + errs: dict = {} + n_missed = n_spurious = 0 + max_lag = 0.0 + for short_id, d in trainer_dict.items(): + gt = gt_by_short.get(short_id) + if gt is None: + continue + raw_obs = build_observed_timeline_from_avail_change(d.get("avail_change", [])) + scored = _fidelity_score(raw_obs, gt, span, lag_tol_s, seed_state="AVL_TRAIN") + if scored is None: + continue + tvd, lags, missed, spurious = scored + errs[short_id] = tvd + n_missed += missed + n_spurious += spurious + if lags: + max_lag = max(max_lag, max(lags)) + + return _fidelity_result( + errs, n_missed, n_spurious, max_lag, mode, mean_tol, within_tau, frac_pass_tol, + skip_note="no trainers with both ground-truth and sim_now-tagged " + "avail_change telemetry (gate off, or predates T3.2)") + + +def agg_belief_fidelity_parity(agg: dict, mode: str, ground_truth: Optional[dict], + mean_tol: float = 0.05, within_tau: float = 0.10, + frac_pass_tol: float = 0.95, + lag_tol_s: float = 30.0) -> dict: + """A7 [DIST]: aggregator BELIEF vs ground-truth trace, ONE mode, BOTH + checkpoints (Batch 3 T3.3 — "Pillar 2"). Returns + {"selection": {...}, "commit": {...}}; the caller flattens each into its + own top-level result key so the causal ladder can localize a + selection-only vs commit-only divergence separately. + + **selection checkpoint** reuses the EXISTING per-candidate avl_state + stamped every selection cycle (`PROP_AVL_STATE` via + `_avail_stamp_end_states`, already read into `selection_train`'s + `per_trainer.avl_state` by `flame/selector/__init__.py`'s + `emit_selection`) — already the aggregator's belief, no new telemetry + (found while building T3.3: emitting a *fresh* `agg_belief_change` here + too would have doubled telemetry volume — up to 300 events/round — for + data that's already fully persisted; same class of "verify the doc's + assumption against actual code" correction as T3.2's `avail_change`/ + `sim_now` finding). + + **commit checkpoint** reads the NEW `agg_belief_change` telemetry + (`checkpoint="commit"`), emitted by `_record_commit_belief` from every + stack's real receive loop and from `_sim_withhold_if_unavail` in sim — + meaningful for ALL baselines, including unaware ones (oort/fedbuff) that + don't filter at selection but still get a commit-time belief reading. + """ + if not ground_truth: + skip = {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no ground-truth trace resolved for this run"} + return {"selection": skip, "commit": skip} + + sel_events = agg.get("selection_train", []) + span = selection_run_span(sel_events, mode) + if span <= 0: + skip = {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "degenerate run span"} + return {"selection": skip, "commit": skip} + + gt_by_short = by_short_id(ground_truth) + + # --- selection checkpoint: reuse the existing per-candidate avl_state series --- + sel_series = build_trainer_state_series(sel_events, mode=mode) + sel_errs: dict = {} + sel_missed = sel_spurious = 0 + sel_max_lag = 0.0 + for end_id, raw_obs in sel_series.items(): + short_id = str(end_id)[-4:] + gt = gt_by_short.get(short_id) + if gt is None: + continue + scored = _fidelity_score(raw_obs, gt, span, lag_tol_s, seed_state="AVL_TRAIN") + if scored is None: + continue + tvd, lags, missed, spurious = scored + sel_errs[short_id] = tvd + sel_missed += missed + sel_spurious += spurious + if lags: + sel_max_lag = max(sel_max_lag, max(lags)) + sel_result = _fidelity_result( + sel_errs, sel_missed, sel_spurious, sel_max_lag, mode, mean_tol, within_tau, + frac_pass_tol, + skip_note="no trainers with ground-truth-matched avl_state in " + "selection telemetry") + + # --- commit checkpoint: NEW agg_belief_change telemetry --- + commit_by_end: dict = collections.defaultdict(list) + for e in agg.get("agg_belief_changes", []): + if e.get("checkpoint") == "commit": + commit_by_end[e.get("end_id")].append(e) + commit_errs: dict = {} + commit_missed = commit_spurious = 0 + commit_max_lag = 0.0 + for end_id, evs in commit_by_end.items(): + short_id = str(end_id)[-4:] + gt = gt_by_short.get(short_id) + if gt is None: + continue + raw_obs = build_observed_timeline_from_agg_belief(evs) + # extrapolate_tail=False + max_gap_s=lag_tol_s: "commit" is + # event-triggered, not continuous (Batch 4 finding, + # UNAVAILABILITY_DESIGN.md) -- don't score the silence after a + # trainer's last commit (tail) OR between two commits (interior gap) + # as if it were stale belief; each commit only vouches for its own + # state within lag_tol_s of itself. + scored = _fidelity_score(raw_obs, gt, span, lag_tol_s, seed_state=None, + extrapolate_tail=False, max_gap_s=lag_tol_s) + if scored is None: + continue + tvd, lags, missed, spurious = scored + commit_errs[short_id] = tvd + commit_missed += missed + commit_spurious += spurious + if lags: + commit_max_lag = max(commit_max_lag, max(lags)) + commit_result = _fidelity_result( + commit_errs, commit_missed, commit_spurious, commit_max_lag, mode, mean_tol, + within_tau, frac_pass_tol, + skip_note="no trainers with ground-truth-matched commit-checkpoint " + "agg_belief_change telemetry (gate off, or predates T3.3)") + + return {"selection": sel_result, "commit": commit_result} + + +def send_gate_wait_fidelity_parity(trainer_dict: dict, ground_truth: Optional[dict], + mean_tol_s: float = 10.0, + within_tau_s: float = 30.0, + frac_pass_tol: float = 0.95) -> dict: + """A8 [DIST, real mode only]: observed [SEND_GATE] wait vs. ground-truth- + expected wait (Batch 3 T3.4 — "Pillar 3", trainer half). + + For every real-mode task_send event carrying both send_gate_wait_s (the + actual wall-time spent blocked in _send_weights's UN_AVL wait loop) and + send_gate_sct (the trainer's own trace-time-basis clock, sampled right + before the gate check — same clock T3.2's avail_change.sim_now uses), + computes the ground-truth-expected wait directly from the raw trace + (expected_send_gate_wait, ground_truth.py) and compares it against the + observed wait. This validates the WAIT DURATION matches what the trace + says it should be — not just that some wait happened (a weaker property + already visible in the real send-gate's own [SEND_GATE] log line). + + Real mode only: sim's send-time gate is agg-side (Stage C); trainer-side + send_gate_wait_s/send_gate_sct are always None in sim telemetry by + construction, so a sim run naturally contributes nothing here. + + SKIP if no ground truth resolved, or no event carries both fields + (gate off, gate never engaged in this run, or telemetry predates T3.4). + Events whose trace never recovers after sct (expected wait undefined, + "would wait forever") are excluded from scoring, not treated as error. + """ + if not ground_truth: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no ground-truth trace resolved for this run"} + + gt_by_short = by_short_id(ground_truth) + errs: list = [] + n_events = 0 + n_uncomparable = 0 + for short_id, d in trainer_dict.items(): + gt = gt_by_short.get(short_id) + if gt is None: + continue + for e in d.get("task_send", []): + obs = e.get("send_gate_wait_s") + sct = e.get("send_gate_sct") + if obs is None or sct is None: + continue + n_events += 1 + expected = expected_send_gate_wait(gt, float(sct)) + if expected is None: + n_uncomparable += 1 + continue + errs.append(abs(float(obs) - expected)) + + if not errs: + return {"ok": True, "tier": "DIST", "status": "SKIP", + "note": "no real-mode task_send events with both " + "send_gate_wait_s and send_gate_sct (gate off, gate " + "never engaged, or predates T3.4)"} + + mean_err = sum(errs) / len(errs) + frac_within_tol = sum(1 for e in errs if e <= within_tau_s) / len(errs) + return { + "ok": mean_err <= mean_tol_s and frac_within_tol >= frac_pass_tol, + "tier": "DIST", + "n_events": n_events, + "n_scored": len(errs), + "n_uncomparable": n_uncomparable, + "mean_err_s": round(mean_err, 2), + "p50_err_s": round(percentile(errs, 50), 2), + "p90_err_s": round(percentile(errs, 90), 2), + "p99_err_s": round(percentile(errs, 99), 2), + "frac_within_tol": round(frac_within_tol, 3), + "within_tau_s": within_tau_s, + "mean_tol_s": mean_tol_s, + "frac_pass_tol": frac_pass_tol, + } + # ═══════════════════════════════════════════════════════════════════ # §3.4x Training input control & per-phase split (T2 / T_*) — Stage 4 # ═══════════════════════════════════════════════════════════════════ @@ -2432,9 +3365,24 @@ def _collect(tr, field): ks = ks_stat(rv, sv) rm, _ = mean_std(rv) sm, _ = mean_std(sv) - res = {"ok": ks <= ks_tol, "tier": "DIST", "phase": f, + # Point-mass guard: when both modes are sub-5ms the distribution is a + # near-zero spike; KS→1 is a statistical artifact of comparing two + # point masses at slightly different zero-proxies (0.001s real vs 0.0s + # sim). Pass on mean_diff instead — a real past-dating divergence clears + # 5ms by orders of magnitude. + _near_zero_phase_s = 0.005 + if abs(rm) <= _near_zero_phase_s and abs(sm) <= _near_zero_phase_s: + ok = True + note = (f"near-zero point mass (both means <={_near_zero_phase_s*1000:.0f}ms): " + "KS uninformative — passed on mean") + else: + ok = ks <= ks_tol + note = None + res = {"ok": ok, "tier": "DIST", "phase": f, "ks_stat": round(ks, 3), "ks_tol": ks_tol, "real_mean_s": round(rm, 3), "sim_mean_s": round(sm, 3)} + if note: + res["note"] = note # mqtt_fetch is pure network-I/O wall time: the sim serves weights from # an in-memory cache and folds the trainer cycle into budget+leg, so this # phase is deliberately NOT part of the virtual clock. Comparing it @@ -2485,7 +3433,9 @@ def run_all_parity(real_agg: dict, sim_agg: dict, agg_goal: int = 0, max_rounds: Optional[int] = None, rounds_cap: Optional[int] = None, - budget_s: Optional[float] = None) -> dict: + budget_s: Optional[float] = None, + real_ground_truth: Optional[dict] = None, + sim_ground_truth: Optional[dict] = None) -> dict: """Run the full parity + invariant battery; returns {name: result_dict}. Ordered HIGH → MID → LOW so coarse failures surface first: @@ -2523,6 +3473,24 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["eligible_speed"] = eligible_speed_composition_parity(real_agg, sim_agg) results["avail_timebase"] = avail_timebase_parity(real_agg, sim_agg) results["duty_cycle"] = duty_cycle_parity(real_trainers, sim_trainers) + results["duty_cycle_duration"] = duration_duty_cycle_parity(real_agg, sim_agg) + results["eligible_pool_reduction"] = eligible_pool_reduction_parity( + real_agg, sim_agg) + results["abandon_timeout"] = abandon_timeout_parity(real_agg, sim_agg) + results["starvation_advance"] = starvation_advance_parity(real_agg, sim_agg) + results["state_timeline_agreement"] = state_timeline_agreement(real_agg, sim_agg) + results["trainer_trace_fidelity_real"] = trainer_trace_fidelity_parity( + real_trainers, real_agg["selection_train"], "real", real_ground_truth) + results["trainer_trace_fidelity_sim"] = trainer_trace_fidelity_parity( + sim_trainers, sim_agg["selection_train"], "sim", sim_ground_truth) + _a7_real = agg_belief_fidelity_parity(real_agg, "real", real_ground_truth) + _a7_sim = agg_belief_fidelity_parity(sim_agg, "sim", sim_ground_truth) + results["agg_belief_fidelity_real_selection"] = _a7_real["selection"] + results["agg_belief_fidelity_real_commit"] = _a7_real["commit"] + results["agg_belief_fidelity_sim_selection"] = _a7_sim["selection"] + results["agg_belief_fidelity_sim_commit"] = _a7_sim["commit"] + results["send_gate_wait_fidelity_real"] = send_gate_wait_fidelity_parity( + real_trainers, real_ground_truth) # ── Stage 3 Selection ── results["selection_detail"] = selection_detail_parity(real_agg, sim_agg) @@ -2552,6 +3520,8 @@ def run_all_parity(real_agg: dict, sim_agg: dict, results["commit_visibility"] = commit_visibility_parity(real_agg, sim_agg) results["eval_commit_timeliness"] = eval_commit_timeliness(sim_agg) results["staleness"] = staleness_parity(real_agg, sim_agg) + results["withheld_delivery"] = withheld_delivery_parity(real_agg, sim_agg) + results["commit_promptness"] = commit_promptness_parity(sim_agg) results["aggregation_sequence"] = aggregation_sequence_parity( real_agg, sim_agg, max_rounds) @@ -2603,6 +3573,18 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "eligible_speed": {"stage": 2, "role": "MECHANISM", "deps": ("eligibility",)}, "avail_timebase": {"stage": 2, "role": "CONTROL", "deps": ("per_round_advance",)}, "duty_cycle": {"stage": 2, "role": "MECHANISM", "deps": ("avail_timebase",)}, + "duty_cycle_duration": {"stage": 2, "role": "MECHANISM", "deps": ("avail_timebase",)}, + "eligible_pool_reduction": {"stage": 2, "role": "DIAG", "deps": ("eligibility",)}, + "abandon_timeout": {"stage": 2, "role": "CONTROL", "deps": ("avail_timebase",)}, + # A6/A7 (Batch 3 T3.2/T3.3) are absolute (vs. ground truth), not real-vs-sim + # — no dependency on avail_timebase (A3), unlike the relative Stage 2 checks above. + "trainer_trace_fidelity_real": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "trainer_trace_fidelity_sim": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "agg_belief_fidelity_real_selection": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "agg_belief_fidelity_real_commit": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "agg_belief_fidelity_sim_selection": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "agg_belief_fidelity_sim_commit": {"stage": 2, "role": "MECHANISM", "deps": ()}, + "send_gate_wait_fidelity_real": {"stage": 2, "role": "MECHANISM", "deps": ()}, # ── Stage 3 Selection ── "selection_detail": {"stage": 3, "role": "MECHANISM", "deps": ("eligibility",)}, "residence": {"stage": 3, "role": "MECHANISM", "deps": ("eligibility",)}, @@ -2632,6 +3614,8 @@ def run_all_parity(real_agg: dict, sim_agg: dict, "commit_visibility": {"stage": 6, "role": "MECHANISM", "deps": ("per_round_advance",)}, "eval_commit_timeliness": {"stage": 6, "role": "MECHANISM", "deps": ("commit_visibility",)}, "staleness": {"stage": 6, "role": "MECHANISM", "deps": ("per_round_advance", "inter_arrival_order", "commit_visibility")}, + "withheld_delivery": {"stage": 6, "role": "DIAG", "deps": ("staleness", "abandon_timeout")}, + "commit_promptness": {"stage": 6, "role": "CONTROL", "deps": ("withheld_delivery",)}, "aggregation_sequence": {"stage": 6, "role": "EMERGENT", "deps": ("participation", "inter_arrival_order")}, "first_divergence_summary": {"stage": 6, "role": "DIAG", "deps": ()}, # ── Stage 7 Statistical utility ── @@ -2644,6 +3628,8 @@ def run_all_parity(real_agg: dict, sim_agg: dict, # ── Stage 9 Budget / stop sanity (orthogonal) ── "budget_not_cap": {"stage": 9, "role": "DIAG", "deps": ()}, "failsafe": {"stage": 9, "role": "MECHANISM", "deps": ()}, + # ── Stage F Starvation clock-advance ── + "starvation_advance": {"stage": 2, "role": "DIAG", "deps": ("abandon_timeout",)}, } # Checks whose FAIL is downgraded to WARN regardless of tier (expected-noisy). diff --git a/lib/python/examples/async_cifar10/scripts/parity/cli.py b/lib/python/examples/async_cifar10/scripts/parity/cli.py index a069b957c..7a4b2bbc6 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/cli.py +++ b/lib/python/examples/async_cifar10/scripts/parity/cli.py @@ -57,6 +57,7 @@ def _run_pair(real_dir: str, sim_dir: str, from parity.checks import ( load_run_dir, run_all_parity, first_divergence, ) + from parity.ground_truth import load_ground_truth, resolve_trace_name from parity.report import print_report, write_json, write_plot real_label = real_label or os.path.basename(real_dir.rstrip("/")) @@ -67,6 +68,9 @@ def _run_pair(real_dir: str, sim_dir: str, print(f"[parity] Loading sim: {sim_label}") sim_agg, sim_trainers = load_run_dir(sim_dir) + real_ground_truth = load_ground_truth(resolve_trace_name(real_dir)) + sim_ground_truth = load_ground_truth(resolve_trace_name(sim_dir)) + print(f"[parity] real: {len(real_agg['agg_rounds'])} agg_round events, " f"{len(real_agg['selection_train'])} selection events, " f"{len(real_agg['agg_evals'])} eval events, " @@ -81,6 +85,8 @@ def _run_pair(real_dir: str, sim_dir: str, agg_goal=agg_goal, rounds_cap=rounds_cap, budget_s=budget_s, + real_ground_truth=real_ground_truth, + sim_ground_truth=sim_ground_truth, ) # Add first_divergence as a diagnostic summary entry (always ok — index=0 is expected for async) @@ -120,7 +126,7 @@ def main() -> None: parser.add_argument("--rounds-cap", type=int, default=None, help="rounds cap from config (enables K9 truncation check)") parser.add_argument("--budget-s", type=float, default=None, - help="max_runtime_s / sim_wall_ceiling_s (enables K5/K9)") + help="max_experiment_runtime_s / sim_wall_ceiling_s (enables K5/K9)") parser.add_argument("--strict", action="store_true", help="Treat WARN as FAIL") parser.add_argument("--lenient", action="store_true", diff --git a/lib/python/examples/async_cifar10/scripts/parity/ground_truth.py b/lib/python/examples/async_cifar10/scripts/parity/ground_truth.py new file mode 100644 index 000000000..d52cc94bd --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/ground_truth.py @@ -0,0 +1,155 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Ground-truth trace lookups — shared infra for the Batch 3 absolute-fidelity +rungs (A6 trainer_trace_fidelity, A7 agg_belief_fidelity, A8 send_gate_wait_fidelity). + +Deliberately thin (see UNAVAILABILITY_DESIGN.md Batch 3 ▶ Implementation +phases → Phase 2): reuses flame.availability.trace's load_trace/state_at/ +read_trainer_unavailability almost as-is — no need to reimplement trace +loading, it already exists in exactly the shape needed. New code here is: + (a) a trace-*name* resolver — given a run dir, which of 3 possible config + keys holds the trace name actually used (varies by baseline, mirrors + debug_run.sh's own 3-way substitution branch), and + (b) a duration-weighted, time-range query wrapper around state_at, since the + checkers need "fraction of [t_start, t_end] spent in each state", not a + single point lookup. +""" + +from __future__ import annotations + +import json +import math +from pathlib import Path +from typing import Optional + +from sortedcontainers import SortedDict + +from flame.availability.trace import ( + next_avail_after, + read_trainer_unavailability, + state_at, +) +from flame.config import TrainerAvailState + +_AVL_STATES = frozenset({TrainerAvailState.AVL_TRAIN, TrainerAvailState.AVL_EVAL}) + + +def resolve_trace_name(run_dir: str) -> Optional[str]: + """Read aggregator_config.json in run_dir and return the configured trace name. + + Priority mirrors debug_run.sh's own ``--trace`` substitution branch (see + UNAVAILABILITY_DESIGN.md Baseline matrix ⁺ note): oort/oort_star/refl write + trackTrainerAvail.trace; felix/oort_star (HP-level client_notify path) + write client_notify.trace; feddance/fedbuff write availability_trace. Try + in that order so a run with more than one key set (e.g. a stale YAML + default alongside the live debug_run.sh override) resolves to the one + debug_run.sh itself would have picked, not an arbitrary one. + + Returns None if the config file is missing/unreadable or no key is set + (gate off / syn_0-style always-available). + """ + path = Path(run_dir) / "aggregator_config.json" + try: + with open(path, encoding="utf-8") as f: + cfg = json.load(f) + except (FileNotFoundError, json.JSONDecodeError): + return None + hp = cfg.get("hyperparameters", cfg) + + track = hp.get("trackTrainerAvail") or {} + trace = track.get("trace") + if trace: + return trace + + client_notify = hp.get("client_notify") or {} + trace = client_notify.get("trace") + if trace: + return trace + + return hp.get("availability_trace") or None + + +def load_ground_truth( + trace_name: Optional[str], base_dir: Optional[str] = None +) -> Optional[dict]: + """{task_id: SortedDict[ts -> state_str]} for every registered trainer. + + Thin re-export of flame.availability.trace.read_trainer_unavailability — + the canonical per-trainer trace loader, already used aggregator-side. + Returns None when trace_name is falsy or the registry/trace can't be read. + """ + return read_trainer_unavailability(trace_name, base_dir=base_dir) + + +def by_short_id(ground_truth: Optional[dict]) -> dict: + """Re-key {task_id: trace} to {task_id[-4:]: trace}. + + Matches the short-id convention scripts/parity/checks.py uses for trainer + telemetry (load_trainer_jsonl_dir keys on f.stem[-4:]; short() truncates + the same way elsewhere in this module) — lets A6/A7 join ground truth + directly against per-trainer telemetry dicts without a separate registry + lookup at call time. + """ + return {task_id[-4:]: trace for task_id, trace in (ground_truth or {}).items()} + + +def state_fractions_over_range( + trace: SortedDict, t_start: float, t_end: float +) -> dict: + """Duration-weighted {state: fraction_of[t_start, t_end]}, read directly + off the raw ground-truth trace. + + Exact dwell-time integration over the trace's own transition points + within the range (not sampled/binned) — the ground-truth counterpart to + avail_state_series.state_fractions(), which does the same integration + over an *observed* (telemetry-derived) series instead. + """ + if t_end <= t_start: + return {} + boundaries = [t_start] + boundaries.extend(trace.irange(t_start, t_end, inclusive=(False, False))) + boundaries.append(t_end) + durations: dict = {} + for a, b in zip(boundaries, boundaries[1:]): + if b <= a: + continue + s = state_at(trace, a).value + durations[s] = durations.get(s, 0.0) + (b - a) + total = sum(durations.values()) + if total <= 0: + return {} + return {s: d / total for s, d in durations.items()} + + +def transitions_in_range(trace: SortedDict, t_start: float, t_end: float) -> list: + """[(ts, state_str), ...] ground-truth transitions within (t_start, t_end].""" + return [ + (ts, trace[ts]) for ts in trace.irange(t_start, t_end, inclusive=(False, True)) + ] + + +def expected_send_gate_wait(trace: SortedDict, sct: float) -> Optional[float]: + """Ground-truth-expected [SEND_GATE] wait (Batch 3 T3.4, A8): the trainer + became ready to send at ``sct`` (its own trace-time-basis clock) -- how + long should it have had to wait, per the raw trace, before the gate + releases it? + + 0 if the trace already shows the trainer AVL_* at ``sct`` (no reason to + wait), else the gap to the next AVL_* transition. Same state-check-first + logic as ``ClientAvailability.compute_delivery_ts`` (client_availability.py, + the sim-side send-gate) -- next_avail_after(trace, sct) alone is NOT + enough, since it always returns the *next* transition even when the + trainer is already available right now -- expressed as a wait DURATION + (this - sct) rather than an absolute delivery time. + + Returns None if the trace never recovers after ``sct`` + (``next_avail_after`` == inf) -- an unanswerable "would wait forever" + case; the caller excludes it from scoring instead of treating it as a + numeric error. + """ + if state_at(trace, sct) in _AVL_STATES: + return 0.0 + nxt = next_avail_after(trace, sct) + if math.isinf(nxt): + return None + return max(0.0, nxt - sct) diff --git a/lib/python/examples/async_cifar10/scripts/parity/report.py b/lib/python/examples/async_cifar10/scripts/parity/report.py index 78cea17b4..5cdd1b824 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/report.py +++ b/lib/python/examples/async_cifar10/scripts/parity/report.py @@ -50,6 +50,18 @@ ("A2b eligible-pool speed composition", "eligible_speed"), ("A3 trace time-base consistency", "avail_timebase"), ("A4 per-trainer duty-cycle", "duty_cycle"), + ("A4dur duration duty-cycle parity", "duty_cycle_duration"), + ("Aa eligible-pool reduction (diag)", "eligible_pool_reduction"), + ("C.3 abandon_timeout (vclock CTRL)", "abandon_timeout"), + ("Fst starvation_advance (diag)", "starvation_advance"), + ("A5 state_timeline_agreement", "state_timeline_agreement"), + ("A6r trainer_trace_fidelity (real, abs)", "trainer_trace_fidelity_real"), + ("A6s trainer_trace_fidelity (sim, abs)", "trainer_trace_fidelity_sim"), + ("A7r.sel agg_belief_fidelity (real, selection, abs)", "agg_belief_fidelity_real_selection"), + ("A7r.com agg_belief_fidelity (real, commit, abs)", "agg_belief_fidelity_real_commit"), + ("A7s.sel agg_belief_fidelity (sim, selection, abs)", "agg_belief_fidelity_sim_selection"), + ("A7s.com agg_belief_fidelity (sim, commit, abs)", "agg_belief_fidelity_sim_commit"), + ("A8r send_gate_wait_fidelity (real, abs)", "send_gate_wait_fidelity_real"), ]), ("3", "Selection", [ ("S3/4 num_chosen / in_flight / eff_c", "selection_detail"), @@ -82,6 +94,8 @@ ("6", "Aggregation", [ ("U6 commit visibility lag", "commit_visibility"), ("U3 staleness distribution", "staleness"), + ("C.2 withheld_delivery (diag)", "withheld_delivery"), + ("K11 commit_promptness (sim, INV)", "commit_promptness"), ("P1 aggregation sequence", "aggregation_sequence"), ("U1 first divergence", "first_divergence_summary"), ]), @@ -413,6 +427,92 @@ def _fmt_metric(name: str, res: dict) -> list: f" max_dutycycle_diff={res.get('max_dutycycle_diff')} " f"n_trainers={res.get('n_trainers')}" ) + elif name == "duty_cycle_duration": + if res.get("mean_err") is not None: + lines.append( + f" mean_err={res.get('mean_err')} (<={res.get('mean_tol')}) " + f"frac_within_tol={res.get('frac_within_tol')} (>={res.get('frac_pass_tol')} " + f"@ tau={res.get('within_tau')}) n_trainers={res.get('n_trainers')}" + ) + elif name == "eligible_pool_reduction": + if res.get("real_mean_reduction") is not None: + lines.append( + f" real_mean_reduction={res.get('real_mean_reduction')} " + f"sim_mean_reduction={res.get('sim_mean_reduction')} " + f"rel_diff={res.get('rel_diff')} (<={res.get('tol_rel')})" + ) + elif name == "state_timeline_agreement": + if res.get("match_frac") is not None: + lines.append( + f" match_frac={res.get('match_frac')} (>={res.get('tol')}) " + f"matched={res.get('matched')}/{res.get('total')} bins " + f"n_trainers={res.get('n_trainers')} n_bins={res.get('n_bins')}" + ) + if res.get("mismatched_examples"): + lines.append(f" mismatch_examples={res.get('mismatched_examples')}") + elif name in ("trainer_trace_fidelity_real", "trainer_trace_fidelity_sim", + "agg_belief_fidelity_real_selection", "agg_belief_fidelity_real_commit", + "agg_belief_fidelity_sim_selection", "agg_belief_fidelity_sim_commit"): + if res.get("mean_err") is not None: + lines.append( + f" mean_err={res.get('mean_err')} (<={res.get('mean_tol')}) " + f"frac_within_tol={res.get('frac_within_tol')} (>={res.get('frac_pass_tol')} " + f"@ tau={res.get('within_tau')}) n_trainers={res.get('n_trainers')}" + ) + lines.append( + f" n_missed_transitions={res.get('n_missed_transitions')} " + f"n_spurious_transitions={res.get('n_spurious_transitions')} " + f"max_lag_s={res.get('max_lag_s')}" + ) + if res.get("worst_trainers"): + lines.append(f" worst_trainers={res.get('worst_trainers')}") + elif name == "send_gate_wait_fidelity_real": + if res.get("mean_err_s") is not None: + lines.append( + f" mean_err_s={res.get('mean_err_s')} (<={res.get('mean_tol_s')}) " + f"frac_within_tol={res.get('frac_within_tol')} (>={res.get('frac_pass_tol')} " + f"@ tau={res.get('within_tau_s')}s) n_scored={res.get('n_scored')}/" + f"{res.get('n_events')} n_uncomparable={res.get('n_uncomparable')}" + ) + elif name == "abandon_timeout": + if res.get("n_abandon") is not None: + parts = [f" n_abandon={res.get('n_abandon')}"] + if res.get("mean_age_s") is not None: + parts.append(f"mean_age_s={res.get('mean_age_s')}") + if res.get("n_aware_boundary_eviction") is not None: + parts.append( + f"aware_evictions={res.get('n_aware_boundary_eviction')} " + f"(mean_age={res.get('aware_boundary_eviction_mean_age_s')}s)" + ) + lines.append(" ".join(parts)) + elif name == "withheld_delivery": + if res.get("n_withheld") is not None: + lines.append( + f" n_withheld={res.get('n_withheld')} " + f"mean_delay_s={res.get('mean_delay_s')} " + f"mean_staleness={res.get('mean_staleness')} " + f"accept_frac={res.get('accept_frac')}" + ) + if res.get("violations"): + lines.append(f" violations={res.get('violations')}") + elif name == "commit_promptness": + if res.get("n_events") is not None: + lines.append( + f" n_events={res.get('n_events')} " + f"mean_slack_s={res.get('mean_slack_s')} " + f"max_slack_s={res.get('max_slack_s')} " + f"min_slack_s={res.get('min_slack_s')}" + ) + lines.append( + f" n_early_violations={res.get('n_early_violations')} " + f"(<={res.get('early_tol_s')}s) " + f"n_late_violations={res.get('n_late_violations')} " + f"(<={res.get('late_slack_tol_s')}s)" + ) + if res.get("early_violations"): + lines.append(f" early_violations={res.get('early_violations')}") + if res.get("late_violations"): + lines.append(f" late_violations={res.get('late_violations')}") elif name == "training_budget": if res.get("ks_stat") is not None: defer = " [mix-deferred to A2c]" if res.get("mix_deferred") else "" diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_agg_belief_fidelity.py b/lib/python/examples/async_cifar10/scripts/parity/test_agg_belief_fidelity.py new file mode 100644 index 000000000..67f9d5e1c --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/test_agg_belief_fidelity.py @@ -0,0 +1,209 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 3 T3.3 — A7 agg_belief_fidelity_parity. + +Same synthetic-drift unit-test pattern as T3.2's test_ground_truth.py, applied +to the aggregator's BELIEF (not the trainer's own avail_change self-report). +Two independent checkpoints (selection / commit) are tested separately since +that's the whole point of tagging them apart -- an unaware baseline (oort/ +fedbuff) only ever gets a meaningful commit-checkpoint score. +""" + +from __future__ import annotations + +import os +import sys + +from sortedcontainers import SortedDict + +_SCRIPTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +from parity.checks import agg_belief_fidelity_parity # noqa: E402 + + +def _sel(round_num, vclock_now, per_trainer): + return {"event": "selection", "task": "train", "round": round_num, + "ts": vclock_now, "vclock_now": vclock_now, "per_trainer": per_trainer} + + +def _belief(round_num, end_id, state, observed_at, checkpoint="commit"): + return {"event": "agg_belief_change", "round": round_num, "end_id": end_id, + "state": state, "observed_at": observed_at, "checkpoint": checkpoint} + + +def test_a7_skips_without_ground_truth(): + agg = {"selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": []} + res = agg_belief_fidelity_parity(agg, "sim", None) + assert res["selection"]["ok"] and res["selection"].get("status") == "SKIP" + assert res["commit"]["ok"] and res["commit"].get("status") == "SKIP" + + +def test_a7_skips_on_degenerate_span(): + agg = {"selection_train": [], "agg_belief_changes": []} + gt = {"t1_0001": SortedDict()} + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert res["selection"].get("status") == "SKIP" + assert res["commit"].get("status") == "SKIP" + + +def test_a7_selection_checkpoint_passes_when_matched(): + # Ground truth: AVL_TRAIN [0,600), UN_AVL [600,900). Aggregator's own + # selection-time belief (per_trainer.avl_state) tracks it exactly. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + agg = { + "selection_train": [ + _sel(1, 0.0, {"0001": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 600.0, {"0001": {"avl_state": "UN_AVL"}}), + _sel(3, 900.0, {"0001": {"avl_state": "UN_AVL"}}), + ], + "agg_belief_changes": [], + } + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert res["selection"]["ok"], res["selection"] + assert res["selection"]["mean_err"] == 0.0 + # No commit-checkpoint telemetry at all in this run -> SKIP, not a false pass. + assert res["commit"].get("status") == "SKIP" + + +def test_a7_selection_checkpoint_fails_when_diverged(): + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + agg = { + "selection_train": [ + _sel(1, 0.0, {"0001": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 900.0, {"0001": {"avl_state": "AVL_TRAIN"}}), # never saw UN_AVL + ], + "agg_belief_changes": [], + } + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert not res["selection"]["ok"], res["selection"] + assert res["selection"]["mean_err"] > 0.0 + + +def test_a7_commit_checkpoint_passes_when_matched(): + # Aggregator's own commit-time belief (agg_belief_change, from + # _record_commit_belief) reads the same state ground truth has at that + # instant -- meaningful even for an unaware baseline with no selection + # filter at all (selection_train carries no per_trainer avl_state here). + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0, checkpoint="commit"), + _belief(2, "0001", "UN_AVL", 600.0, checkpoint="commit"), + ], + } + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert res["commit"]["ok"], res["commit"] + assert res["commit"]["mean_err"] == 0.0 + # No per_trainer.avl_state anywhere -> selection checkpoint has no signal. + assert res["selection"].get("status") == "SKIP" + + +def test_a7_commit_checkpoint_fails_on_belief_wrong_at_its_own_instant(): + # Ground truth transitions at 600s; the aggregator's LAST commit lands at + # 900 but still claims AVL_TRAIN -- i.e. a belief-recording bug where the + # recorded state doesn't even match ground truth AT its own claimed + # instant (unlike the interior-gap case below, where both endpoints are + # individually correct and only the un-observed middle differs). This + # must still fail post-interior-gap-fix: `max_gap_s` only excuses + # UN-observed periods, not a wrong reading at an observed one. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 1000.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0, checkpoint="commit"), + _belief(2, "0001", "AVL_TRAIN", 900.0, checkpoint="commit"), # still wrong + ], + } + res = agg_belief_fidelity_parity(agg, "sim", gt, lag_tol_s=30.0) + assert not res["commit"]["ok"], res["commit"] + assert res["commit"]["mean_err"] > 0.05 + + +def test_a7_commit_checkpoint_ignores_selection_events(): + # A selection-checkpoint agg_belief_change (if one were ever emitted) + # must not leak into the commit-checkpoint score. + gt = {"t1_0001": SortedDict()} + agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "UN_AVL", 100.0, checkpoint="selection"), + ], + } + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert res["commit"].get("status") == "SKIP" + + +def test_a7_commit_checkpoint_does_not_extrapolate_past_last_commit(): + # Batch 4 finding (UNAVAILABILITY_DESIGN.md): the trainer's last commit + # lands at 550 while still AVL_TRAIN (correctly so -- ground truth is + # AVL_TRAIN up to 600), then it goes UN_AVL at 600 and never commits + # again for the rest of the 900s span. Extrapolating "still AVL_TRAIN" + # across [550, 900) would blame ~300/900 of the span on a belief that + # was simply never re-sampled, not wrong -- the window must truncate to + # [t_start, last observed t] instead. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0, checkpoint="commit"), + _belief(2, "0001", "AVL_TRAIN", 550.0, checkpoint="commit"), + ], + } + res = agg_belief_fidelity_parity(agg, "sim", gt) + assert res["commit"]["ok"], res["commit"] + assert res["commit"]["mean_err"] == 0.0 + # The untruncated window would have counted the 600s ground-truth + # transition as "missed" too (it falls after the last observation) -- + # truncation excludes it from the diagnostic count as well. + assert res["commit"]["n_missed_transitions"] == 0 + + + +def test_a7_commit_checkpoint_does_not_extrapolate_across_interior_gap(): + # Batch 4 live-run finding (UNAVAILABILITY_DESIGN.md, felix n=300 syn_50): + # both commits are individually CORRECT -- t=100 reads AVL_TRAIN (true, + # ground truth is AVL_TRAIN on [0,200)), t=590 reads AVL_TRAIN (true, + # ground truth is AVL_TRAIN on [400,600)) -- but the trace dips to UN_AVL + # on [200,400) in between, with no commit to observe it. Holding the + # first commit's belief all the way to the second (old behavior) would + # blame ~200/490s of that gap on "wrong belief", when neither commit was + # ever wrong at its own instant -- same class of over-penalization the + # tail fix (extrapolate_tail=False) already exempts, just mid-run instead + # of at the end. + gt = {"t1_0001": SortedDict({200.0: "UN_AVL", 400.0: "AVL_TRAIN"})} + agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 600.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0, checkpoint="commit"), + _belief(2, "0001", "AVL_TRAIN", 590.0, checkpoint="commit"), + ], + } + res = agg_belief_fidelity_parity(agg, "sim", gt, lag_tol_s=30.0) + assert res["commit"]["ok"], res["commit"] + assert res["commit"]["mean_err"] == 0.0 + + +def test_a7_real_and_sim_scored_independently(): + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + real_agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0), + _belief(2, "0001", "AVL_TRAIN", 700.0), # real never saw UN_AVL + ], + } + sim_agg = { + "selection_train": [_sel(1, 0.0, {}), _sel(2, 900.0, {})], + "agg_belief_changes": [ + _belief(1, "0001", "AVL_TRAIN", 100.0), + _belief(2, "0001", "UN_AVL", 600.0), + ], + } + real_res = agg_belief_fidelity_parity(real_agg, "real", gt) + sim_res = agg_belief_fidelity_parity(sim_agg, "sim", gt) + assert not real_res["commit"]["ok"] + assert sim_res["commit"]["ok"] diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_avail_state_series.py b/lib/python/examples/async_cifar10/scripts/parity/test_avail_state_series.py new file mode 100644 index 000000000..0f5917347 --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/test_avail_state_series.py @@ -0,0 +1,119 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Unit tests for the C.6.2 shared trainer_state_series resolver.""" + +from __future__ import annotations + +import os +import sys + +_SCRIPTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +from parity.avail_state_series import ( # noqa: E402 + build_trainer_state_series, + run_span, + state_fractions, + total_variation_distance, +) + + +def _sel(round_num, ts, vclock_now, per_trainer): + return { + "event": "selection", "task": "train", "round": round_num, "ts": ts, + "vclock_now": vclock_now, "per_trainer": per_trainer, + } + + +def test_build_series_sim_uses_vclock(): + events = [ + _sel(1, 100.0, 0.0, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 110.0, 600.0, {"t1": {"avl_state": "UN_AVL"}}), + _sel(3, 120.0, 900.0, {"t1": {"avl_state": "AVL_TRAIN"}}), + ] + series = build_trainer_state_series(events, mode="sim") + assert series["t1"] == [ + (0.0, "AVL_TRAIN"), (600.0, "UN_AVL"), (900.0, "AVL_TRAIN"), + ] + + +def test_build_series_real_uses_ts_minus_t0(): + events = [ + _sel(1, 1000.0, None, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 1300.0, None, {"t1": {"avl_state": "UN_AVL"}}), + ] + series = build_trainer_state_series(events, mode="real") + assert series["t1"] == [(0.0, "AVL_TRAIN"), (300.0, "UN_AVL")] + + +def test_build_series_real_prefers_vclock_now_over_ts_minus_t0(): + # Real-mode first selection event fires well after the true run start + # (e.g. ~300s of MQTT join-ramp for n=300) — ts - t0 would understate + # elapsed time relative to the trace's agg_start-anchored origin. Once + # vclock_now is stamped on real events too (ClientAvailability._avail_now()), + # it must win over the ts - t0 fallback. + events = [ + _sel(1, 1000.0, 600.0, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 1300.0, 900.0, {"t1": {"avl_state": "UN_AVL"}}), + ] + series = build_trainer_state_series(events, mode="real") + assert series["t1"] == [(600.0, "AVL_TRAIN"), (900.0, "UN_AVL")] + + +def test_build_series_skips_unknown_state(): + events = [_sel(1, 0.0, 0.0, {"t1": {"avl_state": "UNKNOWN"}})] + series = build_trainer_state_series(events, mode="sim") + assert series == {} + + +def test_build_series_same_t_last_write_wins(): + events = [ + _sel(1, 0.0, 100.0, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel(2, 0.0, 100.0, {"t1": {"avl_state": "UN_AVL"}}), + ] + series = build_trainer_state_series(events, mode="sim") + assert series["t1"] == [(100.0, "UN_AVL")] + + +def test_run_span(): + series = {"t1": [(0.0, "AVL_TRAIN"), (900.0, "UN_AVL")], + "t2": [(0.0, "AVL_TRAIN"), (500.0, "UN_AVL")]} + assert run_span(series) == 900.0 + + +def test_state_fractions_dwell_time(): + # AVL_TRAIN for 600s, then UN_AVL for the remaining 300s of a 900s span. + series = {"t1": [(0.0, "AVL_TRAIN"), (600.0, "UN_AVL"), (900.0, "UN_AVL")]} + fracs = state_fractions(series) + assert fracs["t1"]["AVL_TRAIN"] == 600.0 / 900.0 + assert fracs["t1"]["UN_AVL"] == 300.0 / 900.0 + + +def test_state_fractions_credits_trailing_segment_to_t_end(): + series = {"t1": [(0.0, "AVL_TRAIN"), (600.0, "UN_AVL")]} + fracs = state_fractions(series, t_end=900.0) + assert fracs["t1"]["AVL_TRAIN"] == 600.0 / 900.0 + assert fracs["t1"]["UN_AVL"] == 300.0 / 900.0 + + +def test_state_fractions_omits_single_sample_trainers(): + series = {"t1": [(0.0, "AVL_TRAIN")]} + assert state_fractions(series) == {} + + +def test_total_variation_distance_identical_is_zero(): + a = {"AVL_TRAIN": 0.7, "UN_AVL": 0.3} + assert total_variation_distance(a, dict(a)) == 0.0 + + +def test_total_variation_distance_disjoint_is_one(): + a = {"AVL_TRAIN": 1.0} + b = {"UN_AVL": 1.0} + assert total_variation_distance(a, b) == 1.0 + + +def test_total_variation_distance_partial(): + a = {"AVL_TRAIN": 0.6, "UN_AVL": 0.4} + b = {"AVL_TRAIN": 0.5, "UN_AVL": 0.5} + assert abs(total_variation_distance(a, b) - 0.1) < 1e-9 diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_availability_rungs.py b/lib/python/examples/async_cifar10/scripts/parity/test_availability_rungs.py new file mode 100644 index 000000000..0a84ef2db --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/test_availability_rungs.py @@ -0,0 +1,334 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Stage C parity rungs: withheld_delivery / abandon_timeout / eligible_pool_reduction. + +These cover the run-independent (structural) logic of the new availability rungs +and the loader fields that feed them. Tolerance calibration vs a real syn_20 +reference is deferred to an actual run (these only assert what is true regardless +of run data: ordering invariants, the vclock wall-leak guard, and SKIP/empty +handling). +""" + +from __future__ import annotations + +import json +import os +import sys + +_SCRIPTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +from parity.checks import ( # noqa: E402 + abandon_timeout_parity, + commit_promptness_parity, + duration_duty_cycle_parity, + eligible_pool_reduction_parity, + load_agg_jsonl, + load_trainer_jsonl_dir, + starvation_advance_parity, + withheld_delivery_parity, +) + + +def _sel(round_num, nc, ne): + return {"event": "selection", "task": "train", "round": round_num, + "ts": float(round_num), "num_candidates": nc, "num_eligible": ne} + + +# --------------------------------------------------------------------------- +# withheld_delivery +# --------------------------------------------------------------------------- + +def test_withheld_delivery_skips_when_empty(): + res = withheld_delivery_parity({}, {"withheld_deliveries": []}) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_withheld_delivery_passes_well_formed(): + evs = [ + {"end_id": "t1", "sct": 150.0, "delivery_ts": 200.0, "delay_s": 50.0, + "staleness": 3, "accepted": True}, + {"end_id": "t2", "sct": 260.0, "delivery_ts": 260.0, "delay_s": 0.0, + "staleness": 0, "accepted": True}, + ] + res = withheld_delivery_parity({}, {"withheld_deliveries": evs}) + assert res["ok"], res + assert res["n_withheld"] == 2 + assert res["accept_frac"] == 1.0 + assert res["mean_delay_s"] == 25.0 + + +def test_withheld_delivery_fails_on_pre_completion_delivery(): + # delivery_ts < sct would be past-dating — must fail loudly. + evs = [{"end_id": "t1", "sct": 200.0, "delivery_ts": 150.0, "staleness": 1}] + res = withheld_delivery_parity({}, {"withheld_deliveries": evs}) + assert not res["ok"] and res["violations"] + + +def test_withheld_delivery_fails_on_negative_staleness(): + evs = [{"end_id": "t1", "sct": 100.0, "delivery_ts": 120.0, "staleness": -2}] + res = withheld_delivery_parity({}, {"withheld_deliveries": evs}) + assert not res["ok"] + + +# --------------------------------------------------------------------------- +# commit_promptness (K11, Batch 3 T3.5) +# --------------------------------------------------------------------------- + +def test_commit_promptness_skips_when_empty(): + res = commit_promptness_parity({"withheld_deliveries": []}) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_commit_promptness_skips_when_no_actual_commit_ts(): + # Telemetry predating T3.5: delivery_ts present, actual_commit_ts absent. + evs = [{"end_id": "t1", "sct": 100.0, "delivery_ts": 150.0}] + res = commit_promptness_parity({"withheld_deliveries": evs}) + assert res.get("status") == "SKIP" + + +def test_commit_promptness_passes_near_zero_slack(): + evs = [ + {"end_id": "t1", "delivery_ts": 200.0, "actual_commit_ts": 200.0}, + {"end_id": "t2", "delivery_ts": 300.0, "actual_commit_ts": 300.4}, + ] + res = commit_promptness_parity({"withheld_deliveries": evs}) + assert res["ok"], res + assert res["n_events"] == 2 + assert res["n_early_violations"] == 0 + assert res["n_late_violations"] == 0 + + +def test_commit_promptness_fails_on_early_violation(): + # Committed BEFORE its legal delivery_ts -- a hard correctness bug. + evs = [{"end_id": "t1", "delivery_ts": 200.0, "actual_commit_ts": 150.0}] + res = commit_promptness_parity({"withheld_deliveries": evs}) + assert not res["ok"], res + assert res["n_early_violations"] == 1 + assert res["early_violations"] == [{"end": "t1", "slack_s": -50.0}] + assert res["n_late_violations"] == 0 + + +def test_commit_promptness_fails_on_late_violation(): + # Held far longer than delivery_ts required -- a promptness/scheduling bug. + evs = [{"end_id": "t1", "delivery_ts": 200.0, "actual_commit_ts": 260.0}] + res = commit_promptness_parity({"withheld_deliveries": evs}, late_slack_tol_s=30.0) + assert not res["ok"], res + assert res["n_late_violations"] == 1 + assert res["late_violations"] == [{"end": "t1", "slack_s": 60.0}] + assert res["n_early_violations"] == 0 + + +def test_commit_promptness_early_and_late_scored_independently(): + evs = [ + {"end_id": "t1", "delivery_ts": 200.0, "actual_commit_ts": 150.0}, # early + {"end_id": "t2", "delivery_ts": 200.0, "actual_commit_ts": 260.0}, # late + {"end_id": "t3", "delivery_ts": 200.0, "actual_commit_ts": 200.5}, # fine + ] + res = commit_promptness_parity({"withheld_deliveries": evs}, late_slack_tol_s=30.0) + assert not res["ok"] + assert res["n_events"] == 3 + assert res["n_early_violations"] == 1 + assert res["n_late_violations"] == 1 + + +def test_commit_promptness_ignores_events_missing_delivery_ts(): + evs = [{"end_id": "t1", "actual_commit_ts": 200.0}] + res = commit_promptness_parity({"withheld_deliveries": evs}) + assert res.get("status") == "SKIP" + + +# --------------------------------------------------------------------------- +# abandon_timeout (vclock wall-leak control) +# --------------------------------------------------------------------------- + +def test_abandon_skips_when_empty(): + res = abandon_timeout_parity({}, {"abandon_timeouts": []}) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_abandon_passes_vclock_scale_ages(): + evs = [{"end_id": "t1", "sim_send_ts": 100.0, "vclock_now": 195.0, "age_s": 95.0}, + {"end_id": "t2", "sim_send_ts": 50.0, "vclock_now": 200.0, "age_s": 150.0}] + res = abandon_timeout_parity({}, {"abandon_timeouts": evs}) + assert res["ok"], res + assert res["n_abandon"] == 2 and res["mean_age_s"] >= 90 + + +def test_abandon_fails_on_wall_clock_leak(): + # epoch-scale age => the deadline was measured on the wall, not the vclock. + evs = [{"end_id": "t1", "sim_send_ts": 0.0, "vclock_now": 1.75e9, "age_s": 1.75e9}] + res = abandon_timeout_parity({}, {"abandon_timeouts": evs}) + assert not res["ok"] and res["wall_leak_ends"] == ["t1"] + assert "WALL-CLOCK LEAK" in res.get("note", "") + + +def test_abandon_fails_below_threshold(): + evs = [{"end_id": "t1", "sim_send_ts": 100.0, "vclock_now": 150.0, "age_s": 50.0}] + res = abandon_timeout_parity({}, {"abandon_timeouts": evs}) + assert not res["ok"] and res["below_threshold"] + + +def test_abandon_age_derived_when_missing(): + evs = [{"end_id": "t1", "sim_send_ts": 100.0, "vclock_now": 195.0}] # no age_s + res = abandon_timeout_parity({}, {"abandon_timeouts": evs}) + assert res["ok"] and res["max_age_s"] == 95.0 + + +# --------------------------------------------------------------------------- +# eligible_pool_reduction +# --------------------------------------------------------------------------- + +def test_eligible_pool_reduction_skips_without_fields(): + real = {"selection_train": [{"event": "selection", "round": 1, "ts": 1.0}]} + sim = {"selection_train": [{"event": "selection", "round": 1, "ts": 1.0}]} + res = eligible_pool_reduction_parity(real, sim) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_eligible_pool_reduction_passes_when_matched(): + real = {"selection_train": [_sel(1, 300, 240), _sel(2, 300, 250)]} # red 60,50 + sim = {"selection_train": [_sel(1, 300, 245), _sel(2, 300, 248)]} # red 55,52 + res = eligible_pool_reduction_parity(real, sim) + assert res["ok"], res + + +def test_eligible_pool_reduction_fails_when_divergent(): + real = {"selection_train": [_sel(1, 300, 290), _sel(2, 300, 292)]} # red ~9 + sim = {"selection_train": [_sel(1, 300, 200), _sel(2, 300, 210)]} # red ~95 + res = eligible_pool_reduction_parity(real, sim) + assert not res["ok"] + + +# --------------------------------------------------------------------------- +# duty_cycle_duration (A4dur, C.6.3) +# --------------------------------------------------------------------------- + +def _sel_avl(round_num, ts, vclock_now, per_trainer): + return {"event": "selection", "task": "train", "round": round_num, "ts": ts, + "vclock_now": vclock_now, "per_trainer": per_trainer} + + +def test_duty_cycle_duration_skips_without_avl_state(): + real = {"selection_train": [{"event": "selection", "round": 1, "ts": 1.0, + "per_trainer": {"t1": {}}}]} + sim = {"selection_train": [{"event": "selection", "round": 1, "ts": 1.0, + "vclock_now": 0.0, "per_trainer": {"t1": {}}}]} + res = duration_duty_cycle_parity(real, sim) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_duty_cycle_duration_passes_when_matched(): + # t1: AVL_TRAIN for [0,600), UN_AVL for [600,900) in both modes. + real = {"selection_train": [ + _sel_avl(1, 1000.0, None, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel_avl(2, 1600.0, None, {"t1": {"avl_state": "UN_AVL"}}), + _sel_avl(3, 1900.0, None, {"t1": {"avl_state": "UN_AVL"}}), + ]} + sim = {"selection_train": [ + _sel_avl(1, 0.0, 0.0, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel_avl(2, 0.0, 600.0, {"t1": {"avl_state": "UN_AVL"}}), + _sel_avl(3, 0.0, 900.0, {"t1": {"avl_state": "UN_AVL"}}), + ]} + res = duration_duty_cycle_parity(real, sim) + assert res["ok"], res + assert res["n_trainers"] == 1 + assert res["mean_err"] == 0.0 + + +def test_duty_cycle_duration_fails_when_divergent(): + # real: AVL_TRAIN the whole span. sim: UN_AVL the whole span -> TVD = 1.0. + real = {"selection_train": [ + _sel_avl(1, 1000.0, None, {"t1": {"avl_state": "AVL_TRAIN"}}), + _sel_avl(2, 1900.0, None, {"t1": {"avl_state": "AVL_TRAIN"}}), + ]} + sim = {"selection_train": [ + _sel_avl(1, 0.0, 0.0, {"t1": {"avl_state": "UN_AVL"}}), + _sel_avl(2, 0.0, 900.0, {"t1": {"avl_state": "UN_AVL"}}), + ]} + res = duration_duty_cycle_parity(real, sim) + assert not res["ok"] + assert res["mean_err"] == 1.0 + + +# --------------------------------------------------------------------------- +# loaders surface the new events +# --------------------------------------------------------------------------- + +def test_load_agg_jsonl_surfaces_new_events(tmp_path): + p = tmp_path / "aggregator_x.jsonl" + lines = [ + {"event": "selection", "task": "train", "round": 1, "ts": 1.0, + "num_candidates": 300, "num_eligible": 250}, + {"event": "withheld_delivery", "round": 2, "ts": 2.0, "end_id": "t1", + "sct": 10.0, "delivery_ts": 20.0, "delay_s": 10.0, "staleness": 1}, + {"event": "abandon_timeout", "round": 3, "ts": 3.0, "end_id": "t2", + "sim_send_ts": 5.0, "vclock_now": 100.0, "age_s": 95.0}, + ] + p.write_text("\n".join(json.dumps(x) for x in lines) + "\n") + agg = load_agg_jsonl(str(p)) + assert len(agg["withheld_deliveries"]) == 1 + assert len(agg["abandon_timeouts"]) == 1 + assert agg["withheld_deliveries"][0]["end_id"] == "t1" + + +# --------------------------------------------------------------------------- +# starvation_advance +# --------------------------------------------------------------------------- + +def _agg_round(vclock_now): + return {"event": "agg_round", "vclock_now": vclock_now} + + +def _sim_with_avail(rounds, withheld=True): + """Build a sim dict with gate-active marker and given agg_round list.""" + return { + "agg_rounds": rounds, + "withheld_deliveries": [{"end_id": "t0"}] if withheld else [], + "abandon_timeouts": [], + "selection_train": [], + } + + +def test_starvation_advance_skips_without_avail(): + # no withheld/abandon/avail_composition → gate off → SKIP + sim = {"agg_rounds": [_agg_round(10.0), _agg_round(20.0), _agg_round(30.0)], + "withheld_deliveries": [], "abandon_timeouts": [], "selection_train": []} + res = starvation_advance_parity({}, sim) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_starvation_advance_skips_with_no_agg_rounds(): + sim = _sim_with_avail([]) + res = starvation_advance_parity({}, sim) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_starvation_advance_no_jumps(): + sim = _sim_with_avail([_agg_round(float(i * 100)) for i in range(10)]) + res = starvation_advance_parity({}, sim) + assert res["ok"] and res["n_starvation_jumps"] == 0 + + +def test_starvation_advance_detects_jump(): + # 8 normal 100s gaps then one 3000s gap (30× mean) + vclocks = [float(i * 100) for i in range(8)] + [800.0 + 3000.0] + sim = _sim_with_avail([_agg_round(v) for v in vclocks]) + res = starvation_advance_parity({}, sim, jump_factor=5.0) + assert res["ok"] and res["n_starvation_jumps"] >= 1 + assert res["max_jump_s"] > 1000.0 + + +def test_load_trainer_jsonl_dir_surfaces_avail_change(tmp_path): + p = tmp_path / "trainer_0001.jsonl" + lines = [ + {"event": "task_recv", "round": 1}, + {"event": "avail_change", "round": 1, "old_state": "AVL_TRAIN", + "new_state": "UN_AVL"}, + ] + p.write_text("\n".join(json.dumps(x) for x in lines) + "\n") + tr = load_trainer_jsonl_dir(str(tmp_path)) + assert "0001" in tr + assert len(tr["0001"]["avail_change"]) == 1 + assert tr["0001"]["avail_change"][0]["new_state"] == "UN_AVL" diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_ground_truth.py b/lib/python/examples/async_cifar10/scripts/parity/test_ground_truth.py new file mode 100644 index 000000000..4c51adb9f --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/test_ground_truth.py @@ -0,0 +1,245 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 3 T3.2 — ground_truth.py + A6 trainer_trace_fidelity_parity. + +Unlike the relative (real-vs-sim) rungs in test_availability_rungs.py, A6 +compares one mode's own trainer telemetry against a synthetic ground-truth +trace directly — these tests inject a *known* drift between "observed" and +"ground truth" and assert the fidelity calc measures and flags it correctly, +per the working agreement (synthetic-drift unit test before touching real data). +""" + +from __future__ import annotations + +import json +import math +import os +import sys + +from sortedcontainers import SortedDict + +_SCRIPTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +from parity.checks import trainer_trace_fidelity_parity # noqa: E402 +from parity.ground_truth import ( # noqa: E402 + by_short_id, + expected_send_gate_wait, + load_ground_truth, + resolve_trace_name, + state_fractions_over_range, + transitions_in_range, +) + + +def _sel(round_num, vclock_now): + return {"event": "selection", "task": "train", "round": round_num, + "ts": vclock_now, "vclock_now": vclock_now} + + +def _avail_change(round_num, old_state, new_state, sim_now): + return {"event": "avail_change", "round": round_num, "old_state": old_state, + "new_state": new_state, "sim_now": sim_now} + + +# --------------------------------------------------------------------------- +# resolve_trace_name +# --------------------------------------------------------------------------- + +def _write_config(tmp_path, hp: dict): + p = tmp_path / "aggregator_config.json" + p.write_text(json.dumps({"hyperparameters": hp})) + return tmp_path + + +def test_resolve_trace_name_missing_file(tmp_path): + assert resolve_trace_name(str(tmp_path)) is None + + +def test_resolve_trace_name_no_keys_set(tmp_path): + _write_config(tmp_path, {"trackTrainerAvail": {"enabled": "False", "type": "NA"}}) + assert resolve_trace_name(str(tmp_path)) is None + + +def test_resolve_trace_name_track_trainer_avail_priority(tmp_path): + # All three keys set -> trackTrainerAvail wins (debug_run.sh's own priority). + _write_config(tmp_path, { + "trackTrainerAvail": {"enabled": "True", "type": "ORACULAR", "trace": "syn_20"}, + "client_notify": {"enabled": "False", "trace": "syn_50"}, + "availability_trace": "syn_0", + }) + assert resolve_trace_name(str(tmp_path)) == "syn_20" + + +def test_resolve_trace_name_client_notify_fallback(tmp_path): + _write_config(tmp_path, { + "trackTrainerAvail": {"enabled": "False", "type": "NA"}, + "client_notify": {"enabled": "False", "trace": "syn_20"}, + }) + assert resolve_trace_name(str(tmp_path)) == "syn_20" + + +def test_resolve_trace_name_availability_trace_fallback(tmp_path): + _write_config(tmp_path, {"availability_trace": "syn_50"}) + assert resolve_trace_name(str(tmp_path)) == "syn_50" + + +def test_load_ground_truth_none_when_no_trace(): + assert load_ground_truth(None) is None + + +# --------------------------------------------------------------------------- +# state_fractions_over_range / transitions_in_range +# --------------------------------------------------------------------------- + +def test_state_fractions_over_range_always_available(): + trace = SortedDict() # syn_0-style empty trace -> always AVL_TRAIN + frac = state_fractions_over_range(trace, 0.0, 100.0) + assert frac == {"AVL_TRAIN": 1.0} + + +def test_state_fractions_over_range_single_transition(): + trace = SortedDict({600.0: "UN_AVL"}) + frac = state_fractions_over_range(trace, 0.0, 900.0) + assert math.isclose(frac["AVL_TRAIN"], 600.0 / 900.0) + assert math.isclose(frac["UN_AVL"], 300.0 / 900.0) + + +def test_state_fractions_over_range_degenerate(): + trace = SortedDict({600.0: "UN_AVL"}) + assert state_fractions_over_range(trace, 100.0, 100.0) == {} + + +def test_transitions_in_range(): + trace = SortedDict({100.0: "UN_AVL", 200.0: "AVL_TRAIN", 500.0: "UN_AVL"}) + assert transitions_in_range(trace, 0.0, 300.0) == [ + (100.0, "UN_AVL"), (200.0, "AVL_TRAIN"), + ] + # left-exclusive: a transition exactly at t_start is not "within" the range. + assert transitions_in_range(trace, 100.0, 300.0) == [(200.0, "AVL_TRAIN")] + + +def test_by_short_id(): + gt = {"505f9fc483cf4df68a2409257b5fad7d3c580370": SortedDict()} + out = by_short_id(gt) + assert list(out) == ["0370"] + + +# --------------------------------------------------------------------------- +# expected_send_gate_wait (Batch 3 T3.4, A8) +# --------------------------------------------------------------------------- + +def test_expected_send_gate_wait_already_available(): + trace = SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"}) + # sct before the drop -> already AVL_TRAIN there (default), no wait. + assert expected_send_gate_wait(trace, 500.0) == 0.0 + + +def test_expected_send_gate_wait_mid_gap(): + trace = SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"}) + # sct lands inside the UN_AVL window -> wait until the recovery transition. + assert math.isclose(expected_send_gate_wait(trace, 700.0), 200.0) + + +def test_expected_send_gate_wait_exact_transition_boundary(): + trace = SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"}) + # sct exactly at the recovery point -> already available, zero wait. + assert expected_send_gate_wait(trace, 900.0) == 0.0 + + +def test_expected_send_gate_wait_never_recovers(): + trace = SortedDict({600.0: "UN_AVL"}) + assert expected_send_gate_wait(trace, 700.0) is None + + +# --------------------------------------------------------------------------- +# trainer_trace_fidelity_parity (A6) +# --------------------------------------------------------------------------- + +def test_a6_skips_without_ground_truth(): + trainers = {"0001": {"avail_change": [_avail_change(1, "AVL_TRAIN", "UN_AVL", 100.0)]}} + sel = [_sel(1, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", None) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_a6_skips_on_degenerate_span(): + trainers = {"0001": {"avail_change": []}} + gt = {"t1_0001": SortedDict()} + res = trainer_trace_fidelity_parity(trainers, [], "sim", gt) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_a6_skips_when_no_matching_trainers(): + # ground truth keyed by a short id that never shows up in trainer telemetry. + trainers = {"0001": {"avail_change": []}} + gt = {"t1_9999": SortedDict()} + sel = [_sel(1, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", gt) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_a6_passes_when_observed_matches_ground_truth(): + # Ground truth: AVL_TRAIN [0,600), UN_AVL [600,900). Trainer observes and + # logs the exact same transition at the exact same trace-time. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"avail_change": [ + _avail_change(1, "AVL_TRAIN", "UN_AVL", 600.0), + ]}} + sel = [_sel(1, 0.0), _sel(2, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", gt) + assert res["ok"], res + assert res["n_trainers"] == 1 + assert res["mean_err"] == 0.0 + assert res["n_missed_transitions"] == 0 + assert res["n_spurious_transitions"] == 0 + assert res["max_lag_s"] == 0.0 + + +def test_a6_fails_when_trainer_never_tracked_the_trace(): + # This is exactly Challenges §5 item 20: the trainer's own trace was + # wired to a trivial always-available one, so it never logs any + # transition, while ground truth says it should have gone UN_AVL at 600s + # for the back half of a 900s run -- a full state-fraction mismatch. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"avail_change": []}} + sel = [_sel(1, 0.0), _sel(2, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", gt) + # No sim_now-tagged avail_change events anywhere in the run -> SKIP, not a + # false "perfect fidelity" pass -- distinguishing "never transitioned" from + # "no fidelity signal at all" is the whole point of this rung. + assert res["ok"] and res.get("status") == "SKIP" + + +def test_a6_fails_on_injected_lag_drift(): + # Ground truth transitions at 600s; the trainer's own clock is skewed and + # only logs the transition (and its corresponding entry into the observed + # timeline) 300s late -- same class of bug as B2.0.3, but on the trainer + # side. Duration-weighted TVD should catch the resulting fraction skew. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"avail_change": [ + _avail_change(1, "AVL_TRAIN", "UN_AVL", 900.0), # 300s late + ]}} + sel = [_sel(1, 0.0), _sel(2, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", gt, lag_tol_s=30.0) + assert not res["ok"], res + assert res["mean_err"] > 0.05 + assert res["n_missed_transitions"] == 1 # 300s lag exceeds lag_tol_s + assert res["n_spurious_transitions"] == 1 + + +def test_a6_syn0_regression_perfect_fidelity(): + # syn_0-style always-available trace: empty ground-truth SortedDict, and a + # trainer that (correctly) never transitions -- but DOES carry at least + # one sim_now-tagged avail_change event (e.g. a same-state re-stamp), + # otherwise there's no fidelity signal to distinguish from "never ran + # T3.2's telemetry fix" (see test_a6_fails_when_trainer_never_tracked_the_trace). + gt = {"t1_0001": SortedDict()} + trainers = {"0001": {"avail_change": [ + _avail_change(1, "AVL_TRAIN", "AVL_TRAIN", 0.0), + ]}} + sel = [_sel(1, 0.0), _sel(2, 900.0)] + res = trainer_trace_fidelity_parity(trainers, sel, "sim", gt) + assert res["ok"], res + assert res["mean_err"] == 0.0 diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_ladder.py b/lib/python/examples/async_cifar10/scripts/parity/test_ladder.py index 2ff3bb397..c04421659 100644 --- a/lib/python/examples/async_cifar10/scripts/parity/test_ladder.py +++ b/lib/python/examples/async_cifar10/scripts/parity/test_ladder.py @@ -214,6 +214,63 @@ def test_avail_timebase_detects_trajectory_shift(): assert not results["avail_timebase"]["ok"], "A3 should catch the shift" +def test_avail_timebase_passes_aligned(): + """A3 passes when sim eligible-count trajectory matches real within tolerance.""" + real_agg, real_tr = _build_mode(40, advance=10.0, with_vclock=False) + sim_agg, sim_tr = _build_mode(40, advance=10.0, with_vclock=True) + # Identical num_eligible (already set to 10 by _build_mode) — must pass. + results, _ = _verdict(real_agg, sim_agg, real_tr, sim_tr) + assert results["avail_timebase"]["ok"], "aligned trajectory should pass A3" + + +def test_avail_timebase_skips_without_eligible_data(): + """A3 skips when selection events carry no num_eligible field.""" + from parity.checks import avail_timebase_parity + real_agg = {"selection_train": [{"event": "selection", "round": r} for r in range(1, 21)]} + sim_agg = {"selection_train": [{"event": "selection", "round": r} for r in range(1, 21)]} + res = avail_timebase_parity(real_agg, sim_agg) + assert res.get("status") == "SKIP" or res["ok"], "no num_eligible → A3 must skip or pass" + + +def test_duty_cycle_skips_without_avail_change_telemetry(): + """A4 skips when no avail_change telemetry is present (default for syn_0 runs).""" + from parity.checks import duty_cycle_parity + real_tr, sim_tr = {}, {} + for tid in TRAINERS: + real_tr[tid] = {"task_recv": [], "trainer_round": []} + sim_tr[tid] = {"task_recv": [], "trainer_round": []} + res = duty_cycle_parity(real_tr, sim_tr) + assert res["ok"] and res.get("status") == "SKIP", "no telemetry → A4 must skip" + + +def test_duty_cycle_passes_matched_fractions(): + """A4 passes when sim and real duty-cycles match. + + avail_change telemetry carries {old_state, new_state} (build_avail_change), + so "available" = new_state startswith AVL_*. + """ + from parity.checks import duty_cycle_parity + # 3 transitions per trainer: →AVL_TRAIN / →UN_AVL / →AVL_TRAIN → on_frac = 2/3 + evs = [{"new_state": "AVL_TRAIN"}, {"new_state": "UN_AVL"}, + {"new_state": "AVL_TRAIN"}] + real_tr = {tid: {"avail_change": evs} for tid in TRAINERS} + sim_tr = {tid: {"avail_change": evs} for tid in TRAINERS} + res = duty_cycle_parity(real_tr, sim_tr) + assert res["ok"], f"matched duty-cycles must pass A4: {res}" + + +def test_duty_cycle_fails_mismatch(): + """A4 fails when sim duty-cycle diverges from real by more than tolerance.""" + from parity.checks import duty_cycle_parity + # Real: mostly available (on_frac=0.8); sim: mostly unavailable (on_frac=0.2). + real_evs = [{"new_state": "AVL_TRAIN"}] * 8 + [{"new_state": "UN_AVL"}] * 2 + sim_evs = [{"new_state": "AVL_TRAIN"}] * 2 + [{"new_state": "UN_AVL"}] * 8 + real_tr = {tid: {"avail_change": real_evs} for tid in TRAINERS} + sim_tr = {tid: {"avail_change": sim_evs} for tid in TRAINERS} + res = duty_cycle_parity(real_tr, sim_tr) + assert not res["ok"], f"duty-cycle mismatch (0.8 vs 0.2) must fail A4: {res}" + + def test_eligibility_pointmass_passes_on_mean(): """A2: real num_eligible is a constant point-mass (300), sim 298.9 ± tiny. KS saturates to ~1 but the means match — must PASS on the mean (PARITY.md §3i), diff --git a/lib/python/examples/async_cifar10/scripts/parity/test_send_gate_wait_fidelity.py b/lib/python/examples/async_cifar10/scripts/parity/test_send_gate_wait_fidelity.py new file mode 100644 index 000000000..ccc6ff104 --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/parity/test_send_gate_wait_fidelity.py @@ -0,0 +1,108 @@ +# Copyright 2026 Cisco Systems, Inc. and its affiliates +# SPDX-License-Identifier: Apache-2.0 +"""Batch 3 T3.4 — A8 send_gate_wait_fidelity_parity. + +Same synthetic-drift unit-test pattern as T3.2/T3.3 (test_ground_truth.py / +test_agg_belief_fidelity.py), applied to the trainer-side [SEND_GATE] wait +duration: does the observed send_gate_wait_s match what the raw trace says +the wait SHOULD have been, given send_gate_sct (the trainer's own +trace-time-basis clock sampled right before the gate check)? +""" + +from __future__ import annotations + +import os +import sys + +from sortedcontainers import SortedDict + +_SCRIPTS = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +if _SCRIPTS not in sys.path: + sys.path.insert(0, _SCRIPTS) + +from parity.checks import send_gate_wait_fidelity_parity # noqa: E402 + + +def _task_send(round_num, send_gate_wait_s, send_gate_sct): + return { + "event": "task_send", "round": round_num, "task_to_perform": "train", + "send_gate_wait_s": send_gate_wait_s, "send_gate_sct": send_gate_sct, + } + + +def test_a8_skips_without_ground_truth(): + trainers = {"0001": {"task_send": [_task_send(1, 200.0, 700.0)]}} + res = send_gate_wait_fidelity_parity(trainers, None) + assert res["ok"] and res.get("status") == "SKIP" + + +def test_a8_skips_when_no_matching_trainer(): + gt = {"t1_9999": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"task_send": [_task_send(1, 200.0, 700.0)]}} + res = send_gate_wait_fidelity_parity(trainers, gt) + assert res.get("status") == "SKIP" + + +def test_a8_skips_when_no_gate_fields_present(): + # sim telemetry (or telemetry predating T3.4): send_gate_wait_s/sct absent. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"task_send": [ + {"event": "task_send", "round": 1, "task_to_perform": "train"}, + ]}} + res = send_gate_wait_fidelity_parity(trainers, gt) + assert res.get("status") == "SKIP" + + +def test_a8_passes_when_observed_matches_ground_truth(): + # Ground truth: AVL_TRAIN [0,600), UN_AVL [600,900), AVL_TRAIN [900,inf). + # Trainer completed compute at sct=700 (mid-outage) -> should wait 200s. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"})} + trainers = {"0001": {"task_send": [_task_send(1, 200.0, 700.0)]}} + res = send_gate_wait_fidelity_parity(trainers, gt) + assert res["ok"], res + assert res["mean_err_s"] == 0.0 + assert res["n_scored"] == 1 + + +def test_a8_zero_wait_when_already_available(): + gt = {"t1_0001": SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"})} + trainers = {"0001": {"task_send": [_task_send(1, 0.0, 100.0)]}} + res = send_gate_wait_fidelity_parity(trainers, gt) + assert res["ok"], res + assert res["mean_err_s"] == 0.0 + + +def test_a8_fails_on_injected_wait_drift(): + # A promptness bug: observed wait is 300s short of what ground truth says. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"})} + trainers = {"0001": {"task_send": [_task_send(1, 50.0, 700.0)]}} # expect 200.0 + res = send_gate_wait_fidelity_parity(trainers, gt, mean_tol_s=10.0) + assert not res["ok"], res + assert res["mean_err_s"] == 150.0 + + +def test_a8_excludes_uncomparable_never_recovers_events(): + # Trace never recovers after sct -> excluded from scoring, not penalized. + gt = {"t1_0001": SortedDict({600.0: "UN_AVL"})} + trainers = {"0001": {"task_send": [ + _task_send(1, 50.0, 700.0), # uncomparable (no recovery in trace) + _task_send(2, 0.0, 100.0), # already available -> comparable, err=0 + ]}} + res = send_gate_wait_fidelity_parity(trainers, gt) + assert res["ok"], res + assert res["n_events"] == 2 + assert res["n_scored"] == 1 + assert res["n_uncomparable"] == 1 + + +def test_a8_population_rollup_across_multiple_events(): + gt = {"t1_0001": SortedDict({600.0: "UN_AVL", 900.0: "AVL_TRAIN"})} + trainers = {"0001": {"task_send": [ + _task_send(1, 200.0, 700.0), # exact match, err=0 + _task_send(2, 190.0, 700.0), # err=10 + ]}} + res = send_gate_wait_fidelity_parity(trainers, gt, mean_tol_s=10.0, + within_tau_s=15.0) + assert res["n_scored"] == 2 + assert res["mean_err_s"] == 5.0 + assert res["ok"] diff --git a/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh b/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh index 7f5b94a1e..d639ed5a8 100755 --- a/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh +++ b/lib/python/examples/async_cifar10/scripts/run_felix_streaming.sh @@ -69,7 +69,7 @@ src, out = sys.argv[1], sys.argv[2] d = yaml.safe_load(open(src)) for e in d["experiments"]: h = e["aggregator"]["config_overrides"]["hyperparameters"] - h.update(rounds=6, max_runtime_s=180, evalEveryNRounds=2, + h.update(rounds=6, max_experiment_runtime_s=180, evalEveryNRounds=2, targetAccuracy=0.2, stableEvalsAboveTarget=2, min_trainers_to_start=48, min_trainers_join_timeout_s=120) h["checkpoint"]["every_n_rounds"] = 2 diff --git a/lib/python/examples/async_cifar10/scripts/smoke_suite.sh b/lib/python/examples/async_cifar10/scripts/smoke_suite.sh new file mode 100755 index 000000000..a1fe02753 --- /dev/null +++ b/lib/python/examples/async_cifar10/scripts/smoke_suite.sh @@ -0,0 +1,467 @@ +#!/usr/bin/env bash +# ============================================================================ +# smoke_suite.sh — Sequential smoke campaign with per-run timeout guard. +# +# Runs five ordered steps. Each individual (baseline, mode) invocation of +# debug_run.sh is wrapped in a wall-clock timeout: if the run does not +# self-terminate within (runtime_s + timeout_buffer_s), it is killed via +# SIGTERM → SIGKILL on the whole process group so no orphan trainers remain. +# The suite then continues with the next run. +# +# Steps +# 1 pytest — unit tests (flame/tests/), expected: 536p/7s +# 2 syn_0 sim — byte-identity regression, all 6 baselines +# 3 syn_20 sim — all 6 baselines (availability active, fast mode) +# 4 syn_20 both — all 6 baselines × sim + real (parity gate) +# 5 syn_50 starvation — feddance + oort, both modes (B2.0.2 regression) +# Expected: [SIM_STARVATION] events present, no [SIM_WALL_CEILING], +# self-stops via "stopping run" in both modes. +# +# Per-run checks (applied to the Python agg output log): +# stopping_run count of "stopping run" lines — must be > 0 +# wall_ceiling count of [SIM_WALL_CEILING] lines — must be 0 +# starvation count of [SIM_STARVATION] lines — informational +# +# Report is written to /report.txt and printed at the end. +# +# Usage: +# smoke_suite.sh [OPTIONS] +# +# Options: +# --runtime-syn0-s N Budget (wall/vclock) for syn_0 runs [default: 900] +# --runtime-syn20-s N Budget for syn_20 runs [default: 1800] +# --runtime-syn50-s N Budget for syn_50 starvation runs [default: 3600] +# --timeout-buffer-s N Extra wall-sec before force-kill [default: 600] +# --kill-settle-s N GPU memory settle wait after force-kill [default: 20] +# --steps LIST Comma-separated steps to run (1–5) [default: 1,2,3,4,5] +# --baselines NAMES Space-separated baseline list (steps 2–4) [default: all 6] +# --starvation-baselines NAMES Baselines for step 5 [default: feddance oort] +# --num-trainers N Shrink the cohort below the parity config's 300 +# (forwarded to debug_run.sh --num-trainers on every +# run) [default: "" = use the parity config's 300] +# --output-dir DIR Log + report directory [default: /tmp/smoke_suite_] +# --background Re-exec via nohup+disown and return immediately; +# survives the launching shell/SSH session closing. +# Prints the PID, nohup log, and report path, then +# exits 0 right away — the suite keeps running +# detached. Use this for unattended/overnight runs. +# --dry-run Print commands without running them +# --help +# ============================================================================ +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +EX_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" # async_cifar10/ +LIB_DIR="$(cd "$EX_DIR/../.." && pwd)" # lib/python/ +DEBUG_RUN="$SCRIPT_DIR/debug_run.sh" + +# ── Defaults ───────────────────────────────────────────────────────────────── +RUNTIME_SYN0_S=900 +RUNTIME_SYN20_S=1800 +RUNTIME_SYN50_S=3600 +TIMEOUT_BUFFER_S=600 +KILL_SETTLE_S=20 # GPU settle wait (seconds) after a force-killed run +ALL_BASELINES="felix oort oort_star refl feddance fedbuff" +STARV_BASELINES="feddance oort" +STEPS="1,2,3,4,5" +OUTPUT_DIR="/tmp/smoke_suite_$(date +%Y%m%d_%H%M%S)" +DRY_RUN=0 +BACKGROUND=0 +NUM_TRAINERS="" + +# ── Progress counters (set after arg-parse via _compute_total_runs) ─────────── +TOTAL_RUNS=0 +COMPLETED_RUNS=0 + +# ── Arg parsing ────────────────────────────────────────────────────────────── +usage() { + grep '^#' "$0" | grep -v '^#!/' | sed 's/^# \{0,1\}//' + exit 0 +} + +ORIG_ARGS=("$@") # preserved pre-shift for the --background re-exec below + +while [[ $# -gt 0 ]]; do + case "$1" in + --runtime-syn0-s) RUNTIME_SYN0_S="$2"; shift 2 ;; + --runtime-syn20-s) RUNTIME_SYN20_S="$2"; shift 2 ;; + --runtime-syn50-s) RUNTIME_SYN50_S="$2"; shift 2 ;; + --timeout-buffer-s) TIMEOUT_BUFFER_S="$2"; shift 2 ;; + --kill-settle-s) KILL_SETTLE_S="$2"; shift 2 ;; + --steps) STEPS="$2"; shift 2 ;; + --baselines) ALL_BASELINES="$2"; shift 2 ;; + --starvation-baselines) STARV_BASELINES="$2"; shift 2 ;; + --output-dir) OUTPUT_DIR="$2"; shift 2 ;; + --num-trainers) NUM_TRAINERS="$2"; shift 2 ;; + --dry-run) DRY_RUN=1; shift ;; + --background) BACKGROUND=1; shift ;; + --help|-h) usage ;; + *) echo "Unknown arg: $1" >&2; usage ;; + esac +done + +# ── --background: re-exec detached, return control immediately ─────────────── +# Guarded by SMOKE_SUITE_BG so the re-exec'd child (which still sees +# --background in ORIG_ARGS) runs the real suite instead of looping. +if [[ "$BACKGROUND" == "1" && -z "${SMOKE_SUITE_BG:-}" ]]; then + mkdir -p "$OUTPUT_DIR" + NOHUP_LOG="$OUTPUT_DIR/nohup.log" + SMOKE_SUITE_BG=1 nohup bash "$0" "${ORIG_ARGS[@]}" >"$NOHUP_LOG" 2>&1 < /dev/null & + disown + echo "[smoke_suite] backgrounded — PID $! (survives this shell/SSH session closing)" + echo "[smoke_suite] nohup log : $NOHUP_LOG" + echo "[smoke_suite] report : $OUTPUT_DIR/report.txt (written when the suite finishes)" + echo "[smoke_suite] progress : tail -f $NOHUP_LOG" + exit 0 +fi + +# ── Setup ──────────────────────────────────────────────────────────────────── +mkdir -p "$OUTPUT_DIR/runs" +SUITE_LOG="$OUTPUT_DIR/suite.log" +REPORT="$OUTPUT_DIR/report.txt" +SUITE_START=$(date +%s) + +_log() { echo "[$(date '+%F %T')] $*" | tee -a "$SUITE_LOG"; } +_step() { _log ""; _log "══════ STEP $* ══════"; } + +# ── Result tracking ────────────────────────────────────────────────────────── +# Each entry: "label|status|stopping_run|wall_ceiling|starvation" +declare -a RUN_RESULTS=() + +_record() { + RUN_RESULTS+=("${1}|${2}|${3}|${4}|${5}") + COMPLETED_RUNS=$(( COMPLETED_RUNS + 1 )) +} + +# ── Total-run count (computed once after arg-parse) ────────────────────────── +_compute_total_runs() { + local total=0 + local n_all; n_all=$(echo "$ALL_BASELINES" | wc -w) + local n_starv; n_starv=$(echo "$STARV_BASELINES" | wc -w) + local _s + IFS=',' read -ra _sa <<< "$STEPS" + for _s in "${_sa[@]}"; do + _s="${_s// /}" + case "$_s" in + 1) total=$(( total + 1 )) ;; + 2) total=$(( total + n_all )) ;; + 3) total=$(( total + n_all )) ;; + 4) total=$(( total + 2 * n_all )) ;; + 5) total=$(( total + 2 * n_starv )) ;; + esac + done + TOTAL_RUNS=$total +} + +# ── Per-run timeout wrapper ─────────────────────────────────────────────────── +# _run_baseline