Fix sim/real parity: unified max(gpu,D) timing model + GPU contention…

[dhruvsgarg]

… detection (#62)

Unified timing model (trainer, aggregator, OORT):

• sim_round_duration = max(gpu_time, D) in both modes, mirroring real recv_ts-sent_ts. No contention: max(~0.5s,D)=D; overrun: gpu>D so OORT correctly deprioritizes contended trainers.
• Real mode sleep = max(0, D-gpu); sim mode virtual advance = max(gpu,D).
• Adds MessageType.TRAINING_BUDGET_S (37) to carry D to the aggregator.

GPU contention detection:

• Trainer logs [TIMING_OVERRUN] when gpu > D with advice to reduce trainers-per-GPU or add GPUs.
• Aggregator logs [TIMING_OVERRUN_AGG] from budget vs elapsed in both modes.

OORT crash fix (sample_by_util ValueError):

• Root cause: timedelta(0) round duration -> system_utility=0 for all trainers -> np.random.choice(replace=False, p=[...]) fails. Fixed at source (max(gpu,D) is always >0) plus a defensive zero-prob filter in sample_by_util.

compare_parity.py enhancements:

• Check 9: GPU contention analysis (overrun fraction; FAIL>25%, WARN>10%).
• --plot-out PATH: 2-panel timing sanity plot (mean GPU vs budget bars, round-by-round deviation). Graceful on old runs without training_budget_s.

Docs + first-task guides:

• TELEMETRY_AND_SPEEDUP_PLAN.md: new 2026-06-01 update section; Q2 corrected to reflect max(gpu,D) not just D.
• david_first_task.md: add per-trainer lat/long mobility trace, emit as telemetry each round (real mode).
• seshu_first_task.md: sinusoidal training_delay_s variation ±20% of mean, 0.5s floor, config-gated (real mode).
• README.md: next-steps table with all three first-task files.

[Copilot]

Pull request overview

Unifies the per-round timing model used by trainer, async aggregator, and OORT so that round_duration = max(gpu_time, D) in both real and simulated modes (mirroring real recv_ts − sent_ts). Adds explicit GPU-contention detection/logging, fixes an OORT np.random.choice crash when all utilities are zero, propagates the trainer's training-budget to the aggregator via a new TRAINING_BUDGET_S message field, and extends compare_parity.py with round-duration / sim_send_ts / GPU-contention checks plus an optional sanity plot. Also adds two new contributor first-task guides and a task_recv telemetry event.

Changes:

• Unified max(gpu, D) round-duration model + TRAINING_BUDGET_S=37 message, with trainer/aggregator overrun warnings.
• OORT sample_by_util defensive zero-probability filter and additional per-trainer/extra fields in selection telemetry.
• New task_recv telemetry event, compare_parity.py checks 7–9, --plot-out, and new docs / first-task guides.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
lib/python/flame/mode/message.py	Adds TRAINING_BUDGET_S=37 and updates SIM_* docstrings to max(gpu,D) semantics.
lib/python/flame/mode/horizontal/syncfl/trainer.py	Sends TRAINING_BUDGET_S and emits task_recv telemetry on weight receipt.
lib/python/flame/mode/horizontal/asyncfl/top_aggregator.py	Stamps SIM_SEND_TS, computes overrun against budget, simplified _distribute_weights, comment cleanup.
lib/python/examples/async_cifar10/trainer/pytorch/main.py	Implements max(gpu,D) timing, [TIMING_OVERRUN] warning, sleep=max(0,D-gpu).
lib/python/flame/selector/async_oort.py	Adds per_trainer_extra/extra in selection emit; zero-prob filter in sample_by_util.
lib/python/flame/telemetry/events.py	Adds EVENT_TASK_RECV builder.
lib/python/examples/async_cifar10/scripts/compare_parity.py	Adds checks 7–9 (round duration, sim_send_ts, GPU contention) and --plot-out timing plot.
lib/python/examples/async_cifar10/TELEMETRY_AND_SPEEDUP_PLAN.md	Documents 2026-06-01 update and corrects Q2.
lib/python/examples/async_cifar10/README.md	New first-task table linking ava/david/seshu guides.
lib/python/examples/async_cifar10/david_first_task.md	New: location-trace first task guide.
lib/python/examples/async_cifar10/seshu_first_task.md	New: sinusoidal training_delay_s first task guide.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.