P0-#5: EAGLE watchdog — degrade to plain decode, never a silent hang#120
Merged
Conversation
#5 L1) The serial EAGLE draft chain is coordinator-local compute — the one leg of a decode round no socket timeout sees (F6 bounds the ring recv, the job timeout bounds prefill). A wedged or pathologically slow drafter therefore crawled the job forever while still streaming: the 2026-07-14 residential-tail silent-hang shape. Watchdog: an eagle-routed draft step over M25_DRAFT_BUDGET_S (default 1.0s, ~80x the healthy 10-13ms; 0 disables) twice consecutively — first step of a job exempt (warmup) — or cumulative drafting >20x the budget AND >half the decode wall, flips the job to n-gram-only IN PLACE: HybridDrafter latches via disable_eagle() (flag-based; an object swap would orphan the pending request/fetch pairing), cur_depth returns to pipelined depth, extend() stops. No wire change; the degrade is invisible to the stages. Result dict gains eagle_degraded/eagle_degraded_at; coordinate_pipe gains an optional on_progress tick (each reply recv) for the job-level stall backstop. tests/test_eagle_degrade.py: slow-forever drafter -> lossless completion, drafter out of the loop after the trip, pipelining restored (ring.backlog); healthy drafter never trips; 0-disables; inert without EAGLE. 42 tests green incl. the F6 heartbeat + fake-ring regressions.
The ret socket was untimed mid-session: a coordinator-side wedge (dead conntrack after a relay path change, a receive buffer that never drains) left the tail blocked INSIDE _ret_send's send — unable to select, accept a fresh hello_return, or see the next reset. One stuck send wedged the whole warm ring. This is the strongest candidate for the 2026-07-14 residential-tail EAGLE hang: EAGLE's aux frames are the big return payload, and the CPU fake ring's infinite-bandwidth loopback is exactly why it never reproduced there. ret now gets settimeout(M25_RET_STALL_S) (default 180s; 0 disables = the old untimed behavior) at both adoption sites. With the libp2p transport the send loop is per-sendmsg-call (_sendall_vectored), so the bound means ZERO bytes accepted for that long — a slow-but-draining residential uplink resets the clock with every chunk and never trips, however big the frame (pinned by test; sendall's total-deadline semantics were verified empirically and rejected). On trip the existing _ret_send EDGE absorb does the designed recovery: drop ret, keep predecessor+KV, job stale, next coordinator re-adopts mid-session. tests/test_ret_stallguard.py: per-progress vs zero-progress send semantics pinned at the transport level; the REAL serve() tail survives a wedged ret (stage timeout 30s so only the guard explains the fast recovery), re-adopts a fresh return channel, and serves the next reset; 0-disables pinned.
…agle:0 A degraded coordinator (post-watchdog, or the shard.coordinate retry) could only stop DRAFTING — the stages kept producing and shipping aux hidden states (3 layers x [K+1,H] bf16, the dominant EAGLE return payload and the prime suspect for the 2026-07-14 residential wedge) because the aux attach was gated only by each stage's own launch env. The degraded arm therefore did not equal the proven-good plain ring on the wire. _reset_op now stamps eagle:0/1 from the coordinator's arm; every stage (tail reply sites, head/middle forward sites, both reset and reset_batch sessions) gates its aux attach on the session's flag. Field absent = old coordinator = launch env decides, so every deployed build interops unchanged; old stages ignore the field. Side effect: a plain-launched coordinator against an eagle-launched ring stops paying for aux it never reads. tests/test_reset_eagle_flag.py: reset stamping; the REAL tail replies plain toks under eagle:0 and restores aux on a flag-less reset (per-session, not sticky); the REAL middle stage forwards the flag down the ring and drops aux from forwarded frames. 57 tests green across the ring suites.
L2 — run_job now runs the mitigation ladder: coordinate_pipe is called resumable, and an EDGE fault while EAGLE was armed (the residential-tail wedge class) triggers ONE degraded retry on a FRESH ring dial. The re-dial is mandatory, not hygiene: the tail only goes stale on a fresh hello_return, so a plain reset on the old sockets would eat a late in-flight reply as its ack and commit garbage. The retry resumes the committed tokens under the SAME settlement nonce with the same delta state (stream continues, no dup/gap; receipts sweep exactly once, on the surviving attempt), and flips M25_EAGLE (+M25_TREE) off process-wide — sticky by policy: the ring just proved EAGLE-hostile, re-arming per job would burn a failed attempt on every request; a daemon restart re-arms from env. The reset then stamps eagle:0 so the stages silence aux ring-wide — the degraded arm equals the proven plain ring on the wire. JobRejected (a reply, not a death) never retries. A retry that also fails bails rc=1 for a clean daemon restart — never a loop on poisoned sockets. SHARD_JOB_DONE gains degraded; SHARD_JOB_RETRY marks the failover. L3 — a per-job stall watchdog thread: no progress (reply recv / commit / redial — coordinate_pipe's new on_progress ticks on every prefill reply, so a legitimately slow thin-uplink prefill counts as ALIVE) for M25_JOB_STALL_S (auto: job timeout + 60s; 0 disables) -> SHARD_JOB_FATAL + os._exit(1) -> daemon restart + fail-closed complete. Catches the classes nothing socket-level bounds: a drafter wedged inside torch/CUDA, a stuck send. _emit is now one locked atomic write so a watchdog FATAL can never splice a TOKEN line (complete-lines-only NDJSON contract). --prefill-chunk added. fake_ring gains mute_after_decode (wedged tail: consume, never reply) and reject_decode (structured H1 error). 10 new tests incl. a real-os._exit subprocess proof and losslessness/nonce/receipt continuity across the retry. Full suite 584 passed, 1 skipped.
…ial-verify D1) The backstop's own FATAL emit could defeat it: a main thread wedged INSIDE a locked stdout write (a full/unread daemon pipe — the very stuck-write class L3 exists to cover) left the watchdog blocked on _EMIT_LOCK before _hard_exit; a dead stdout raised BrokenPipeError and killed the watchdog thread, orphaning a torch-wedged job. The emit is now best-effort — bounded lock wait, stderr fallback, every exception swallowed — and _hard_exit(1) sits in a finally: the last line of defense exits no matter what. Also: the coordinator's drain loop and receipts sweep now tick on_progress (several sequential full-timeout waits post-done must not sum past the stall budget under M25_REPLY_TIMEOUT=0). 64 tests green.
This was referenced Jul 20, 2026
leyten
added a commit
that referenced
this pull request
Jul 20, 2026
…her (#122) The watchdog env flags from #120 were unreachable in a real deployment — the launcher only forwards ENG_ENV to stages/gateway, and M25_RET_STALL_S (tail), M25_DRAFT_BUDGET_S/M25_JOB_STALL_S (coordinator), M25_REPLY_TIMEOUT werent in it. Adds them so the tail gets its stall bound and a --serve gateway coordinator can tune the budgets.
leyten
added a commit
that referenced
this pull request
Jul 20, 2026
…123) Controlled-ring validation closes P0-#5: shipped #120 build + #122 knobs serve M2.5 over a real 3-region WAN ring (degraded=false, receipts chained under one nonce); the sustained-backpressure M1 wedge is not reproducible on a fast datacenter ring by construction (depth-1 EAGLE, 4-6MB buffers, no NET_ADMIN) — proven offline in test_ret_stallguard.py. Marks P0-#5 DONE, adds the receipt.
leyten
added a commit
that referenced
this pull request
Jul 22, 2026
… silent hang (#120) Mitigation ladder for the 2026-07-14 residential-tail EAGLE hang; root cause unknown by design — every EAGLE-implicated stall class now ends in worst-case-slower or a clean fast fail. L1: draft-budget watchdog flips a slow drafter to n-gram mid-job (M25_DRAFT_BUDGET_S, flag-latch, pipelining restored). M1: the tail ret socket gets a per-progress stall bound (M25_RET_STALL_S; per-sendmsg semantics — a slow-but-draining uplink never trips) so one wedged send can no longer hang the whole warm ring. Reset now carries eagle:0/1 and every stage silences aux for degraded sessions — the degraded arm equals the plain ring on the wire. L2: shard.coordinate runs one degraded retry on a mandatory fresh re-dial, resuming committed tokens under the same settlement nonce (no delta dup/gap; receipts sweep once); EAGLE/TREE sticky-off after; JobRejected never retries. L3: per-job stall backstop (M25_JOB_STALL_S, auto timeout+60s; prefill replies count as progress) with an unconditional exception-proof os._exit so the daemon restarts and fail-closes. Verified by a design panel + independent adversarial pass (1 defect found and fixed: the backstop emit could block its own kill). 22 new tests incl. a real-os._exit subprocess proof; full suite 584 passed, 1 skipped.
leyten
added a commit
that referenced
this pull request
Jul 22, 2026
…her (#122) The watchdog env flags from #120 were unreachable in a real deployment — the launcher only forwards ENG_ENV to stages/gateway, and M25_RET_STALL_S (tail), M25_DRAFT_BUDGET_S/M25_JOB_STALL_S (coordinator), M25_REPLY_TIMEOUT werent in it. Adds them so the tail gets its stall bound and a --serve gateway coordinator can tune the budgets.
leyten
added a commit
that referenced
this pull request
Jul 22, 2026
…123) Controlled-ring validation closes P0-#5: shipped #120 build + #122 knobs serve M2.5 over a real 3-region WAN ring (degraded=false, receipts chained under one nonce); the sustained-backpressure M1 wedge is not reproducible on a fast datacenter ring by construction (depth-1 EAGLE, 4-6MB buffers, no NET_ADMIN) — proven offline in test_ret_stallguard.py. Marks P0-#5 DONE, adds the receipt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mitigation-first fix for the 2026-07-14 residential-tail EAGLE hang (launch blocker P0-#5). Root cause is still unknown — this ladder makes every EAGLE-implicated stall class end in worst case slower, or a clean fast fail — never a silent hang, on the deployment path (
python -m shard.coordinate→coordinate_pipe→ libp2p transport).The ladder (each layer covers a class the others cannot):
M25_DRAFT_BUDGET_S(default 1.0s ≈ 80× healthy; first step exempt; 2 consecutive breaches; cumulative guard) flips the job to n-gram-only in place: flag-latch onHybridDrafter(no object swap — the pending request/fetch pairing survives), pipelining restored, zero wire changes.sendall— unable to select, re-accept, or see a reset. The whole warm ring hung behind it (strongest root-cause candidate; the CPU fake ring never reproduced it because loopback has infinite bandwidth).M25_RET_STALL_S(default 180s) is a per-progress bound — the libp2p transport sends per-sendmsgcall, so it means zero bytes accepted for 180s; a slow-but-draining residential uplink never trips it (sendalltotal-deadline semantics were verified empirically and rejected). Trip → the existing_ret_sendEDGE absorb: drop ret, keep predecessor+KV, stale until the next coordinator.shard.coordinate): an EDGE fault while EAGLE was armed → close + re-dial (mandatory: the tail only goes stale on a freshhello_return; a plain reset on old sockets eats a late in-flight reply as its ack), resume committed tokens under the same settlement nonce, same delta state (no dup/gap; receipts sweep exactly once). EAGLE/TREE stay off for the process (sticky; a daemon restart re-arms from env);SHARD_JOB_DONEcarriesdegraded.JobRejectednever retries; a failed retry bails rc=1 for a clean daemon restart.M25_JOB_STALL_S(auto = job timeout + 60s) → best-effort FATAL + unconditionalos._exit(1)→ daemon restart + fail-closed complete. Covers what nothing socket-level can: a drafter wedged inside torch/CUDA, a stuck send. The kill is exception-proof and lock-bypass-proof (adversarial-verify D1).Verification: design panel (adversarial correctness + test design) before build; independent adversarial verify pass after (1 confirmed defect, fixed in the last commit; all other attack lines held). 22 new tests across 4 files, incl. a real-
os._exitsubprocess proof, losslessness/nonce/receipt continuity across the retry, and the realserve()tail surviving a wedged return channel. Full suite 584 passed, 1 skipped.Known-accepted (documented, not built): attempt-1 EAGLE prefill on a thin-uplink tail stays legally slow until the first fault (fp8 prefill aux / admission-gating EAGLE by tail uplink = post-mitigation levers). Daemon-side escalation (restart coordinator with
M25_EAGLE=0after a stall-kill) = small c0mpute follow-up. Note:S.M25_EAGLEarmed with a non-extend drafter now pipelines at full depth (was pinned to 1); no test pinned the old behavior and the new semantics are the intended ones.Next: controlled-ring validation — tc-throttled tail uplink, master arm (expect the wedge) vs this branch (expect degrade + survive).