Skip to content

P0-#5: EAGLE watchdog — degrade to plain decode, never a silent hang#120

Merged
leyten merged 5 commits into
masterfrom
eagle/watchdog
Jul 20, 2026
Merged

P0-#5: EAGLE watchdog — degrade to plain decode, never a silent hang#120
leyten merged 5 commits into
masterfrom
eagle/watchdog

Conversation

@leyten

@leyten leyten commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Mitigation-first fix for the 2026-07-14 residential-tail EAGLE hang (launch blocker P0-#5). Root cause is still unknown — this ladder makes every EAGLE-implicated stall class end in worst case slower, or a clean fast fail — never a silent hang, on the deployment path (python -m shard.coordinatecoordinate_pipe → libp2p transport).

The ladder (each layer covers a class the others cannot):

  • L1 — draft-budget watchdog (coordinator): the serial EAGLE draft chain is coordinator-local compute no socket timeout sees — a slow/wedged drafter crawled forever while still streaming. An eagle-routed draft step over M25_DRAFT_BUDGET_S (default 1.0s ≈ 80× healthy; first step exempt; 2 consecutive breaches; cumulative guard) flips the job to n-gram-only in place: flag-latch on HybridDrafter (no object swap — the pending request/fetch pairing survives), pipelining restored, zero wire changes.
  • M1 — tail ret stall bound: the return socket was untimed; one send wedged against a dead/blocked return path parked the tail inside sendall — unable to select, re-accept, or see a reset. The whole warm ring hung behind it (strongest root-cause candidate; the CPU fake ring never reproduced it because loopback has infinite bandwidth). M25_RET_STALL_S (default 180s) is a per-progress bound — the libp2p transport sends per-sendmsg call, so it means zero bytes accepted for 180s; a slow-but-draining residential uplink never trips it (sendall total-deadline semantics were verified empirically and rejected). Trip → the existing _ret_send EDGE absorb: drop ret, keep predecessor+KV, stale until the next coordinator.
  • eagle:0 on the reset wire: a degraded coordinator now silences aux ring-wide (every stage attach site honors the session flag) — the degraded arm equals the proven-good plain ring on the wire, not just coordinator-side. Old builds interop unchanged (absent field = launch env).
  • L2 — one degraded retry (shard.coordinate): an EDGE fault while EAGLE was armed → close + re-dial (mandatory: the tail only goes stale on a fresh hello_return; a plain reset on old sockets eats a late in-flight reply as its ack), resume committed tokens under the same settlement nonce, same delta state (no dup/gap; receipts sweep exactly once). EAGLE/TREE stay off for the process (sticky; a daemon restart re-arms from env); SHARD_JOB_DONE carries degraded. JobRejected never retries; a failed retry bails rc=1 for a clean daemon restart.
  • L3 — job stall backstop: no progress (reply/commit/redial; prefill replies count — a 75MB aux chunk on a thin uplink is legitimately slow) for M25_JOB_STALL_S (auto = job timeout + 60s) → best-effort FATAL + unconditional os._exit(1) → daemon restart + fail-closed complete. Covers what nothing socket-level can: a drafter wedged inside torch/CUDA, a stuck send. The kill is exception-proof and lock-bypass-proof (adversarial-verify D1).

Verification: design panel (adversarial correctness + test design) before build; independent adversarial verify pass after (1 confirmed defect, fixed in the last commit; all other attack lines held). 22 new tests across 4 files, incl. a real-os._exit subprocess proof, losslessness/nonce/receipt continuity across the retry, and the real serve() tail surviving a wedged return channel. Full suite 584 passed, 1 skipped.

Known-accepted (documented, not built): attempt-1 EAGLE prefill on a thin-uplink tail stays legally slow until the first fault (fp8 prefill aux / admission-gating EAGLE by tail uplink = post-mitigation levers). Daemon-side escalation (restart coordinator with M25_EAGLE=0 after a stall-kill) = small c0mpute follow-up. Note: S.M25_EAGLE armed with a non-extend drafter now pipelines at full depth (was pinned to 1); no test pinned the old behavior and the new semantics are the intended ones.

Next: controlled-ring validation — tc-throttled tail uplink, master arm (expect the wedge) vs this branch (expect degrade + survive).

leyten added 5 commits July 20, 2026 14:07
#5 L1)

The serial EAGLE draft chain is coordinator-local compute — the one leg of
a decode round no socket timeout sees (F6 bounds the ring recv, the job
timeout bounds prefill). A wedged or pathologically slow drafter therefore
crawled the job forever while still streaming: the 2026-07-14
residential-tail silent-hang shape.

Watchdog: an eagle-routed draft step over M25_DRAFT_BUDGET_S (default 1.0s,
~80x the healthy 10-13ms; 0 disables) twice consecutively — first step of a
job exempt (warmup) — or cumulative drafting >20x the budget AND >half the
decode wall, flips the job to n-gram-only IN PLACE: HybridDrafter latches
via disable_eagle() (flag-based; an object swap would orphan the pending
request/fetch pairing), cur_depth returns to pipelined depth, extend()
stops. No wire change; the degrade is invisible to the stages. Result dict
gains eagle_degraded/eagle_degraded_at; coordinate_pipe gains an optional
on_progress tick (each reply recv) for the job-level stall backstop.

tests/test_eagle_degrade.py: slow-forever drafter -> lossless completion,
drafter out of the loop after the trip, pipelining restored (ring.backlog);
healthy drafter never trips; 0-disables; inert without EAGLE. 42 tests
green incl. the F6 heartbeat + fake-ring regressions.
The ret socket was untimed mid-session: a coordinator-side wedge (dead
conntrack after a relay path change, a receive buffer that never drains)
left the tail blocked INSIDE _ret_send's send — unable to select, accept a
fresh hello_return, or see the next reset. One stuck send wedged the whole
warm ring. This is the strongest candidate for the 2026-07-14
residential-tail EAGLE hang: EAGLE's aux frames are the big return payload,
and the CPU fake ring's infinite-bandwidth loopback is exactly why it never
reproduced there.

ret now gets settimeout(M25_RET_STALL_S) (default 180s; 0 disables = the
old untimed behavior) at both adoption sites. With the libp2p transport the
send loop is per-sendmsg-call (_sendall_vectored), so the bound means ZERO
bytes accepted for that long — a slow-but-draining residential uplink
resets the clock with every chunk and never trips, however big the frame
(pinned by test; sendall's total-deadline semantics were verified
empirically and rejected). On trip the existing _ret_send EDGE absorb does
the designed recovery: drop ret, keep predecessor+KV, job stale, next
coordinator re-adopts mid-session.

tests/test_ret_stallguard.py: per-progress vs zero-progress send semantics
pinned at the transport level; the REAL serve() tail survives a wedged ret
(stage timeout 30s so only the guard explains the fast recovery), re-adopts
a fresh return channel, and serves the next reset; 0-disables pinned.
…agle:0

A degraded coordinator (post-watchdog, or the shard.coordinate retry) could
only stop DRAFTING — the stages kept producing and shipping aux hidden
states (3 layers x [K+1,H] bf16, the dominant EAGLE return payload and the
prime suspect for the 2026-07-14 residential wedge) because the aux attach
was gated only by each stage's own launch env. The degraded arm therefore
did not equal the proven-good plain ring on the wire.

_reset_op now stamps eagle:0/1 from the coordinator's arm; every stage
(tail reply sites, head/middle forward sites, both reset and reset_batch
sessions) gates its aux attach on the session's flag. Field absent = old
coordinator = launch env decides, so every deployed build interops
unchanged; old stages ignore the field. Side effect: a plain-launched
coordinator against an eagle-launched ring stops paying for aux it never
reads.

tests/test_reset_eagle_flag.py: reset stamping; the REAL tail replies plain
toks under eagle:0 and restores aux on a flag-less reset (per-session, not
sticky); the REAL middle stage forwards the flag down the ring and drops
aux from forwarded frames. 57 tests green across the ring suites.
L2 — run_job now runs the mitigation ladder: coordinate_pipe is called
resumable, and an EDGE fault while EAGLE was armed (the residential-tail
wedge class) triggers ONE degraded retry on a FRESH ring dial. The re-dial
is mandatory, not hygiene: the tail only goes stale on a fresh hello_return,
so a plain reset on the old sockets would eat a late in-flight reply as its
ack and commit garbage. The retry resumes the committed tokens under the
SAME settlement nonce with the same delta state (stream continues, no
dup/gap; receipts sweep exactly once, on the surviving attempt), and flips
M25_EAGLE (+M25_TREE) off process-wide — sticky by policy: the ring just
proved EAGLE-hostile, re-arming per job would burn a failed attempt on
every request; a daemon restart re-arms from env. The reset then stamps
eagle:0 so the stages silence aux ring-wide — the degraded arm equals the
proven plain ring on the wire. JobRejected (a reply, not a death) never
retries. A retry that also fails bails rc=1 for a clean daemon restart —
never a loop on poisoned sockets. SHARD_JOB_DONE gains degraded;
SHARD_JOB_RETRY marks the failover.

L3 — a per-job stall watchdog thread: no progress (reply recv / commit /
redial — coordinate_pipe's new on_progress ticks on every prefill reply, so
a legitimately slow thin-uplink prefill counts as ALIVE) for M25_JOB_STALL_S
(auto: job timeout + 60s; 0 disables) -> SHARD_JOB_FATAL + os._exit(1) ->
daemon restart + fail-closed complete. Catches the classes nothing
socket-level bounds: a drafter wedged inside torch/CUDA, a stuck send.
_emit is now one locked atomic write so a watchdog FATAL can never splice
a TOKEN line (complete-lines-only NDJSON contract). --prefill-chunk added.

fake_ring gains mute_after_decode (wedged tail: consume, never reply) and
reject_decode (structured H1 error). 10 new tests incl. a real-os._exit
subprocess proof and losslessness/nonce/receipt continuity across the
retry. Full suite 584 passed, 1 skipped.
…ial-verify D1)

The backstop's own FATAL emit could defeat it: a main thread wedged INSIDE
a locked stdout write (a full/unread daemon pipe — the very stuck-write
class L3 exists to cover) left the watchdog blocked on _EMIT_LOCK before
_hard_exit; a dead stdout raised BrokenPipeError and killed the watchdog
thread, orphaning a torch-wedged job. The emit is now best-effort — bounded
lock wait, stderr fallback, every exception swallowed — and _hard_exit(1)
sits in a finally: the last line of defense exits no matter what.

Also: the coordinator's drain loop and receipts sweep now tick on_progress
(several sequential full-timeout waits post-done must not sum past the
stall budget under M25_REPLY_TIMEOUT=0). 64 tests green.
@leyten
leyten merged commit 4d5df7f into master Jul 20, 2026
2 checks passed
@leyten
leyten deleted the eagle/watchdog branch July 20, 2026 14:50
leyten added a commit that referenced this pull request Jul 20, 2026
…her (#122)

The watchdog env flags from #120 were unreachable in a real deployment — the launcher only forwards ENG_ENV to stages/gateway, and M25_RET_STALL_S (tail), M25_DRAFT_BUDGET_S/M25_JOB_STALL_S (coordinator), M25_REPLY_TIMEOUT werent in it. Adds them so the tail gets its stall bound and a --serve gateway coordinator can tune the budgets.
leyten added a commit that referenced this pull request Jul 20, 2026
…123)

Controlled-ring validation closes P0-#5: shipped #120 build + #122 knobs serve M2.5 over a real 3-region WAN ring (degraded=false, receipts chained under one nonce); the sustained-backpressure M1 wedge is not reproducible on a fast datacenter ring by construction (depth-1 EAGLE, 4-6MB buffers, no NET_ADMIN) — proven offline in test_ret_stallguard.py. Marks P0-#5 DONE, adds the receipt.
leyten added a commit that referenced this pull request Jul 22, 2026
… silent hang (#120)

Mitigation ladder for the 2026-07-14 residential-tail EAGLE hang; root cause unknown by design — every EAGLE-implicated stall class now ends in worst-case-slower or a clean fast fail. L1: draft-budget watchdog flips a slow drafter to n-gram mid-job (M25_DRAFT_BUDGET_S, flag-latch, pipelining restored). M1: the tail ret socket gets a per-progress stall bound (M25_RET_STALL_S; per-sendmsg semantics — a slow-but-draining uplink never trips) so one wedged send can no longer hang the whole warm ring. Reset now carries eagle:0/1 and every stage silences aux for degraded sessions — the degraded arm equals the plain ring on the wire. L2: shard.coordinate runs one degraded retry on a mandatory fresh re-dial, resuming committed tokens under the same settlement nonce (no delta dup/gap; receipts sweep once); EAGLE/TREE sticky-off after; JobRejected never retries. L3: per-job stall backstop (M25_JOB_STALL_S, auto timeout+60s; prefill replies count as progress) with an unconditional exception-proof os._exit so the daemon restarts and fail-closes. Verified by a design panel + independent adversarial pass (1 defect found and fixed: the backstop emit could block its own kill). 22 new tests incl. a real-os._exit subprocess proof; full suite 584 passed, 1 skipped.
leyten added a commit that referenced this pull request Jul 22, 2026
…her (#122)

The watchdog env flags from #120 were unreachable in a real deployment — the launcher only forwards ENG_ENV to stages/gateway, and M25_RET_STALL_S (tail), M25_DRAFT_BUDGET_S/M25_JOB_STALL_S (coordinator), M25_REPLY_TIMEOUT werent in it. Adds them so the tail gets its stall bound and a --serve gateway coordinator can tune the budgets.
leyten added a commit that referenced this pull request Jul 22, 2026
…123)

Controlled-ring validation closes P0-#5: shipped #120 build + #122 knobs serve M2.5 over a real 3-region WAN ring (degraded=false, receipts chained under one nonce); the sustained-backpressure M1 wedge is not reproducible on a fast datacenter ring by construction (depth-1 EAGLE, 4-6MB buffers, no NET_ADMIN) — proven offline in test_ret_stallguard.py. Marks P0-#5 DONE, adds the receipt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant