feat(pipe): adaptive round-length policy (M25_ADAPTIVE_K, coordinator-only, lossless)#31
Open
463464q435q43 wants to merge 48 commits into
Open
feat(pipe): adaptive round-length policy (M25_ADAPTIVE_K, coordinator-only, lossless)#31463464q435q43 wants to merge 48 commits into
463464q435q43 wants to merge 48 commits into
Conversation
Eagle/chain diagnostics
fix(ring): break tail bring-up handshake deadlock
Feat/topology select ring
The measured per-hop bottleneck is moving the bf16 activation over WAN (a 120KB round-trip is 6-8x the connect-ping; transfer dominates, not relay/compute). Transporting the inter-stage hidden state as fp8 (e4m3, per-tensor scale to keep the residual-stream outlier channels in range) halves bytes/hop -> ~2x tok/s on a transfer-bound ring. Opt-in: the bf16 path is byte-for-byte unchanged when off. Lossy but deterministic + verifiable: receipts hash the fp8 engine's activations, so trustless verification is intact (it just attests an fp8 engine, not bf16-exact). Quality is a measured question (residual-stream outliers can bite) -> validate with a bf16-vs-fp8 A/B on a real ring before claiming "usable M2.5 at 2x". - transport: register float8_e4m3fn/e5m2 in the codec dtype table - m25_pipe: _pack_h/_unpack_h + _hsend/_hrecv wrap the 5 activation wire sites - scatter launcher passes M25_FP8_WIRE through to stages Offline-validated: codec round-trips fp8 bit-exact, bytes exactly 0.50x, 2.26% per-hop quant error on an outlier-heavy activation (per-channel scaling is the tightening lever if the A/B shows quality loss).
…placement Consolidate the 2026-07-01 state onto master: everything (handshake, select_ring, EAGLE-chain, fp8) is now on master, branches deleted. Records: the measured fp8 A/B (+9% on high-bw vast, ~2x is the residential bytes-bound regime; quality preserved but not bit-exact); the 3-agent residential-bandwidth research (the bind is the sender's UPLOAD; decode survives, long-context prefill is the wall on cable/DSL but fine on fiber; you can't conjure upload, the fix is selection + use the download direction + fewer bytes/hops); and the admission-vs-placement decision (coarse proven floor in c0mpute + per-role capability matching in select_ring, not a join threshold). Next: make select_ring upload-aware, then the validation + residential-throttle runs.
RESUME HERE was current but the lower sections lagged: ROADMAP leyten#1 still said 'vLLM tree GO/NO-GO NEXT' (superseded), PROVEN lacked select_ring/fp8/residential, IN-FLIGHT still said 'reconfirm EAGLE accept'. Fixed: ROADMAP redirects to the RESUME HERE next-actions; PROVEN adds select_ring, fp8, residential-bottleneck rows and marks EAGLE merged; IN-FLIGHT marks EAGLE confirmed+merged.
select_ring minimized decode step-time only, blind to the residential wall: prefill ships the [S,H] activation (~100MB/hop @16k) over home uplinks and the bind is the SENDER'S UPLOAD, so a slow-cable node on a load-bearing hop costs minutes of TTFT while decode's tiny activation survives. The objective is now total request time (prefill + D*decode) with per-node upload a first-class, sender-bound cost. Prefill transport is the engine's chunked+ pipelined makespan (sum_fwd(u)+(C-1)*max_fwd(u))/C, interpolating serial-blob SUM and bottleneck MAX. The selector tails the lowest-upload node (the tail forwards nothing), drops nodes whose uplink would dominate prefill, and relegates them to off-critical roles (weight-seeder / aggregator / hot-standby / decode-only-replica / spot-check-verifier) derived from the drop reason, not a hard Mbps gate -- the placement half of admission-vs-placement. Stays pure; byte-identical to the old decode-only path when up_mbps is omitted (golden-snapshot tested). Validated offline (scratchpad/sim_network.py, volunteer/residential pool): aware/ oracle ~0.98 across context while upload-blind collapses 0.98->0.80 as context grows; TTFT speedup 2.5-5x (p95 19x); request-time 1.0->1.32x (2k->64k); the fat- uplink rental pool shows a smaller gap. tests/test_topology.py: 10 tests, all pass. Adversarial review also found a pre-existing false-"infeasible": the >TRIM latency funnel's must-set was subnet-blind, trimming feasibility-critical distinct-subnet cards when the fattest cards are co-located. Fixed (must-set keeps a distinct- subnet cover) + regression-tested.
…a-blocked Rented 8 real EU boxes for the residential upload A/B; measured real hop bandwidth heterogeneity (8-127 Mbps, i.e. residential-tier slow hops exist and would wall prefill), confirming the premise. The fully-automated per-node upload aware-vs-blind comparison was blocked by vast-container limits (no NET_ADMIN for tc, NAT hairpin, ssh-proxy rate-limiting under repeated runs). Boxes torn down (0 live, ~$4). Clean number needs one gentle non-rate-limited run; natural uplinks already span the residential range so no throttle needed.
M25_FP8_WIRE ships float8_e4m3fn tensors, but only shard/transport.py (the libp2p codec) had the fp8 entries in its dtype table — the raw-TCP PSK path rejected every fp8 frame as 'unknown tensor dtype' and reset the edge in a loop. Mirror the two entries; the tables must stay in lockstep (regression-tested in tests/test_eagle_draft.py).
… GQA broadcast The committed-context cache was a Python list of per-token [1,NKV,HD] tensors: every propose() re-cat'ed the WHOLE context + repeat_interleave GQA-expanded it, and extend() ran ~8 tiny kernels per position in a Python loop. With EAGLE at depth=1 all of that sits serially between recv and the next send, so it is pure per-traversal latency. Now: preallocated [CAP,NKV,HD] buffers (amortized doubling), one-shot batched extend, draft-chain k/v written into the scratch tail past ctx_len (next extend overwrites it — the committed cache is never mutated), and attention broadcasts the GRP query groups against NKV-head K/V instead of copying the context per step. CPU bench at toy dims, ctx 3000: prefill-extend 327ms -> 2.1ms (156x), decode round (extend K+1 + propose K) 16.4ms -> 4.35ms (3.8x); the GPU win is larger (the old path was launch-overhead-bound). Proposals are regression-locked to the original implementation (kept verbatim as the test reference) across rounds, dtypes, buffer growth and the legacy single-aux path: tests/test_eagle_draft.py, 6 tests. Also adds prefill_pair_tokens — the pure per-chunk EAGLE pairing the coordinator uses to feed the whole prompt (next commit).
… serial-path profile Three coordinator/stage-side costs on the EAGLE path (the ~7 tok/s reasoning product path), found by tracing one decode traversal: * aux rode every hop UN-quantized: 3 layers x [K+1,H] bf16 ~ 166KB/hop — 3x the activation payload itself — while h got fp8. _merge_aux now packs each stage's aux as [fp8,scale] (M25_FP8_AUX, defaults to the M25_FP8_WIRE setting); _unpack dequantizes once coordinator-side. Aux only feeds the DRAFTER, so losslessness is untouched by construction — only accept could move, and only when the flag is on. * the drafter saw only the LAST prefill chunk (default 512 tokens) of context: coordinate_pipe kept just the final chunk's aux. Every chunk now extends the EAGLE context as its response arrives (pairing = prefill_pair_tokens; the extend hides in the next chunk's WAN wait), so accept on long prompts stops being capped by a 512-token window. _eagle_aux_range also drops its per-position Python stack loop. * the decode wall was unattributed: t_draft/t_recv were measured but never returned. The result now carries decode_s/draft_s/ring_wait_s and coord prints the breakdown — the profile that ranks the next serial-path fix (measure first).
The 2026-06-27 warm sweep found K=8 ~2x over K=6 on the scattered EU ring, and every EAGLE measurement since ran K=8 — but both entry points still defaulted to 6, so any launch that forgot --K served below the known optimum. Defaults now match the receipts.
GraphRunner replays a captured graph and never re-runs run_block's aux capture, so with both flags set the EAGLE drafter silently feeds on stale prefill aux — accept craters and any measurement taken that way lies. The combination now refuses to start instead of quietly corrupting a run. (Found by the 2026-07-02 review fleet.)
…ram routing Two ways the hybrid wasted its drafting budget, both review-fleet finds: * every divergence (and the end-of-job drain) called d_fetch() on a request known to be stale — computing a full proposal (on the EAGLE path a K-step serial chain) just to discard it, then re-drafting. Drafters now expose cancel(); the coordinator uses it (falls back to fetch() for drafters without one). * NgramDrafter set matched=True on ANY ng-token anchor recurrence, even with zero preceding-context agreement (best_len=0). On novel/ reasoning text those are coincidences, so garbage n-gram drafts displaced the EAGLE drafts the hybrid exists for — accept 0 where EAGLE gets 2-4 tokens. matched now requires best_len >= min_match (default 1; M25_NGRAM_MINMATCH=0 restores any-anchor routing). The n-gram proposal itself is unchanged — only the routing signal moves; standalone n-gram deployments are unaffected. Needs the warm A/B to quantify the accept lift; the routing logic is regression-tested.
…findings, ranked next actions
… worktree-safe REPO Three measurement-integrity holes in m25_scatter_pipe, all review-fleet finds, all of which would have poisoned the upcoming branch-vs-master warm A/B: * M25_* flags were forwarded to the STAGES only — the coordinator and gateway processes got none, so e.g. M25_EAGLE=1 warmed aux-capturing stages while the coordinator silently drafted n-gram-only. One eng_env() prefix now feeds all three launch sites (only flags the operator actually set; unset ones fall through to the identical per-process defaults). Also brings the launcher's --K default to the measured 8. * push_code ignored scp failures — one flaky vast SSH silently launched a stale or MIXED-VERSION ring. Now retries once, then fails the launch loudly. * REPO was hardcoded to the main checkout — running the launcher from a worktree (the A/B: branch arm vs master arm) would push the WRONG tree's files. Derived from the launcher's own location now (SHARD_REPO overrides).
…rag-quote accept 13->44% Six arms on a scattered 6x5090 EU ring (rental-lottery order, same ring all arms). Serial-path branch vs master, EAGLE reasoning-hybrid, honest bench. Both warm orderings agree branch>master (fwd 4.0->5.9, rev 4.6->5.5) = +33% decode-weighted, jitter-robust. Accept identical on every cell EXCEPT rag-quote, where the whole-prompt drafter-context fix lifts accept 13->44% (g 2.0->4.5, ~4->10.5 tok/s) reproducibly across both branch passes. fp8-aux quality-safe (accept unchanged). min-match unproven on one pass.
…13->44%, merge-worthy
Perf/eagle serial path
…ail channel lifecycles The warm-ring wedge (re-warm before every coordinator, the session tax): the tail closed its predecessor whenever the coordinator-return died, the EOF cascaded upstream, and no stage ever re-dialed nxt_sock — one coordinator exit bricked the whole warm ring. Ports specpipe's proven recovery: head/middle deliberately drop the forward link on any edge error and rebuild it (dial-forever, 0.5s) before re-accepting a predecessor; the tail gives pred and ret independent lifecycles (ret death keeps pred+KV and accepts a new hello_return mid-session via select; pred death tears down both so a reset-ok can never reach a dead coordinator) and drops a dead job's in-flight replies until the next reset. Warm smoke-test: coord -> kill -> new coord on the same ring, no re-warm.
…not the receipts' CRITICAL trust hole (2026-07-02 fleet, 3 reviewers): verify_coverage's target layer_count was max(layer_end) over the receipts under test, so a ring that omitted layers still "tiled fully" and passed — skip-compute-and-still-get-paid. The m25 coordinator now passes S.cfg.num_hidden_layers (62) and fails CLOSED on an empty receipt list when SHARD_RECEIPTS is on; specpipe gains --n-layers and warns loudly when the target is self-derived (legacy path, coordinator never loads the model). tests/test_receipt_coverage.py pins the fixed semantics (truncated/gap/overlap/out-of-range/pinning/tamper all raise).
…live, guarded acks Two adversarial review passes on the churn-recovery port traced four real failure modes; all four are closed here: - A conn that speaks a JOB frame is adopted as the NEW predecessor with its frame queued (specpipe fill()'s semantic) instead of closed as junk — closing it kill-looped a stage replacement after a silent pred death, both at bring-up and mid-session. - TCP keepalive (60/20/3) on every ring socket bounds silent-peer death (no FIN: power loss, NAT expiry) to ~2min; the select()-based tail can't otherwise see a half-open predecessor now that the old 600s recv-timeout teardown (the idle wedge itself) is gone. - ret_ok sends are guarded at both loci: a coordinator dying between hello and ack was misclassified as predecessor death and cost a full ring cascade. - Pred-death keeps a JUST-reconnected coordinator's return channel (stale set, no reset consumed): the old-session EOF racing the gateway's retry used to kill the exact retry churn recovery exists for. Reset is only honored once a live ret exists; mid-session ret drops its greeting timeout (parity with bring-up); the pending pool is capped at 8. Also ports wire.py's malformed-frame guard to shard/transport.py recv_msg (the production path): the resilient tail now recv's stranger conns all session, so one unparseable frame crashing the stage process went from a bring-up window to a standing exposure. ConnectionError -> EDGE_ERRORS -> edge reset, not process death.
…al GQA kernel, receipts Port of eagle/tree-verify onto the merged serial-path base, with the 2026-07-02 fleet's payload verdict applied (the measured tok/s loss was ~6-7x self-inflicted wire bytes, not physics). What changed vs the measured v1: - propose_tree is best-first top-M by cumulative draft log-prob (M25_TREE_M/ _TOPB/_DEPTH; default 12 nodes) on the O(1) drafter internals — kills the fixed-2^d shape (62 nodes bought +0.7 g) that made the 1.23MB aux return. - Tree traffic rides _hsend/_hrecv: fp8 activations + fp8 aux per hop. - attn_tree always runs the manual broadcast-GQA kernel (tree_spec._gqa_masked_ attend): a dense mask knocks SDPA off flash on sm_120, and there is no repeat_interleave copy of the context; mask built once per stage-call on device in h's dtype. - Tree blocks are attested (signer.observe both branches + receipt sweep + coverage verify in coordinate_pipe_tree) — verification no longer silently off under M25_TREE, and the tree A/B carries the same receipt cost as chain. - Whole-prompt EAGLE prefill context (per-chunk extend), draft_s/ring_wait_s breakdown, loud no-aux guard, launcher forwards M25_TREE* + pushes tree_spec. - tests/test_tree_spec.py (8): mask ancestor property vs brute force, walk vs exhaustive, chain degeneracy, kernel-vs-reference, RoPE gather, and propose_tree topb=1 == propose() exactly (the offline losslessness gate).
…edge fixed) 6xEU 5090 ring: coord kill -9 mid-decode (depth=4 in flight) -> new coordinator completes on the same warm ring (6.70 -> 6.60 tok/s, weights never reloaded). Stage logs show the ported recovery choreography (drop+rebuild forward links, tail re-accepts channels). The re-warm-per-coordinator tax is gone.
…chain-tree The tree coordinator drove the EAGLE drafter directly, silently dropping the HybridDrafter's n-gram half; verbatim-reuse cells paid full EAGLE accept (2026-07-02 warm A/B: rag-quote 5.6 -> 3.0 tok/s while every novel cell won). Route per round: n-gram first, a real match verifies as a 1-wide tree (and now banks the +1 correction/bonus token the chain path drops); miss -> EAGLE tree.
…+18% Same-warm 6xEU ring, honest bench: chain 3.9 -> tree-hybrid 4.6 decode-weighted (reason-math 4.8->6.0, reason-logic 3.0->4.7, code-edit 4.5->6.1). Flips the v1 "tree loses tok/s on tight rings" verdict — the loss was wire payload, as the 2026-07-02 fleet concluded. Losslessness gate: 76 identical tokens then one near-tie kernel flip (documented class). rag-quote stays chain's win (pipelined n-gram vs sync tree round) — depth-aware hybrid is the measured next lever.
Fold the 2026-07-02 review fleet (79 confirmed) into a clear tiered roadmap: TIER 0 done (merged serial-path), TIER 1 perf (wedge, tree- verify, topology, cheap levers), TIER 2 trust/moat (the CRITICAL self- referential coverage check + 4 high), TIER 3 robustness (gateway/wire/ contained bugs), TIER 4 cleanup. Records the key signal: after the merge only 1 remaining HIGH is perf — risk moved off the single-stream hot path onto trust + gateway/wire. Adds the honest tok/s projection (~5.7-7 today -> ~10-12 good tight ring post-tree-verify+topology, ~5-6 global scatter; most findings are NOT tok/s). Full per-finding detail in .claude/plans/fleet-findings-20260702.md.
…erify wins EU +18% RESUME HERE now points at the two validated branches (wedge+trust, tree-v2), PROVEN gains the tree A/B + churn-survival rows, the trust row drops its self-referential warning, and TIER-1.1/1.2 + TIER-2.1 are ticked. Next: land both PRs, then the high-RTT global-scatter cell and the depth-aware hybrid.
leyten call (2026-07-02): steady-state rings are regional by construction (select_ring picks close subsets; a global ring is a placement failure). The global measure was go/no-go only while tree lost on tight rings; v2 wins the tight EU ring, so nothing hangs on a global number. NEXT re-ranked: depth-aware hybrid, then the RTT-ordered warm run for the absolute 10-12 check.
Fix/ring wedge receipt truth
Eagle/tree verify v2
…ith a pinned node select_ring's >_TRIM latency funnel built its feasibility cover require-blind: the pinned node's subnet slot went to the pool's fattest same-subnet card (which can never co-ring with require), and k_min stopped the cover walk before the other-subnet fat nodes a require-compatible ring needs — so the widened fallback searched a starved pool and returned None while a feasible ring existed. Fix: a second cover walk seeded with require (its subnet spent, its capacity counted), purely additive to the trimmed pool. Regression test fails pre-fix, passes post-fix; the legacy golden snapshot is byte-identical (the funnel never fires there). Unblocks the RTT-ordered launch (TIER 1.3).
fix(topology): require-aware trim funnel — no more false-infeasible w…
…ke-ring harness (leyten#14) Squash of perf/tree-depth-hybrid (11 commits): usability-report + CPU fake-ring harnesses, honest g + full-accept bonus, M25_STAGE_TIMING transport/compute split, select_ring head-first deployable orientation, depth-aware hybrid (pipelined n-gram bursts + sync tree rounds) with adversarial-review fixes, good-ring receipts (3 arms). 64 CPU tests green.
…he public internet The c0mpute technical report (author: leyten), final copy: page-one hero chart, the accept-gated pipelining law with a both-ends-calibrated Monte-Carlo, the depth-aware hybrid router, the transport/compute split and the CPU-launch-bound discovery, verification measured at 0.4% of stage compute, and a live coordinator-kill recovery timeline. Ships with the interleaved multi-arm bench harness (research/m25_paper_bench.py), four receipt sets from the test evening (docs/receipts/m25-paper-*), and every figure generated from those receipts (docs/paper/figures.py). 64 CPU tests green.
Paper published on Zenodo (CC-BY-4.0, v1.0). DOI added to the paper title block and the README, with the report linked from the top of the README.
Centered labels in the two segments collided where the segments were narrower than the text. Transport labels now sit outside the bar end, stage-compute labels stay inside their segment at a size that fits.
Reviewer feedback: one plain byline (leyten · c0mpute.ai · July 3, 2026) replaces the author and links lines; no links on the front page. Abstract now states the stages are untrusted, the defensible form of the trustless claim (Limitations still names the freshness/replay gaps). Title unchanged.
The concept DOI resolves to the latest report version; version DOIs stay valid for citations of a specific revision.
Reviewer feedback round two. 'Decentralized inference' placed in the intro and related work, 'decentralized AI' once in the close; three placements total, no stuffing. Conclusion gains a roadmap paragraph in which every future item traces to a measured finding (drafter training toward the 0.8 crossover, freshness/spot-recompute verification, CPU-aware selection and graph capture, right-sized models on the same fabric) and ends on the idle consumer-GPU supply plus a commitment to keep publishing receipts.
* feat(stage): graph-captured EAGLE aux buffers
The M25_CUDA_GRAPH+M25_EAGLE SystemExit guard existed because run_block
captures EAGLE aux as a Python side-effect that graph replay bypasses,
so the drafter would feed on stale prefill aux. Bake the capture into
the graph instead: _GraphState gains a static [s,H] bf16 buffer per aux
layer in the runner's range, GraphRunner._layers copy_()s each aux
layer's output residual into it during capture (the copy is part of the
graph, so every replay refreshes it), and run() publishes the buffers
into _AUX after replay, mirroring run_block's contract. _AUX aliases
the static buffers — safe because the serve loop consumes them
synchronously via _merge_aux before the next replay. Guard removed;
M25_CUDA_GRAPH => M25_STATIC_KV kept.
Why: slow-CPU stages are kernel-launch-bound (35-50ms/block vs 11.5ms,
2026-07-03 receipt); the graph route recovers ~2-4x of block time but
was hard-blocked on the EAGLE/tree serve path.
Evidence: full CPU suite green (64 passed); on-box bit-equality + aux
freshness proof lands in research/graph_aux_check.py in this branch.
* feat(stage): bound cuda-graph capture set + OOM-safe fallback
Hybrid refeed frames have variable s (K+1 up to ~K+TREE_DEPTH+2) and
each (s, bucket) pair is one captured graph pinning its own workspace
memory, so unbounded lazy capture could OOM a stage over a long serve.
M25_GRAPH_MAX (default 16) caps total captured graphs process-wide:
_block captures a new (s,bucket) only within budget, past it the block
runs eager (counted in _GRAPH_SKIPPED, never a crash) while captured
shapes keep replaying. GraphRunner.run additionally survives a capture
failure (torch.cuda.OutOfMemoryError/RuntimeError): the bucket is
marked permanently eager and the block falls back to run_block — a
stage must never die from graph capture.
Evidence: full CPU suite green (64 passed); bounding-logic unit tests
land with tests/test_graph_aux.py in this branch.
* feat(pipe): per-job runtime graph toggle via reset op
The graph A/B must interleave graph-on/graph-off arms per job on ONE
warm ring — a stage relaunch per arm reintroduces the time-of-day
drift the interleaved methodology exists to kill. So the hot path
(_block) now consults S.M25_CUDA_GRAPH_ACTIVE (initialized from the
M25_CUDA_GRAPH env) instead of the env constant, and the coordinator's
reset op may carry {"graph": true/false}: stages apply it via
S.set_graph before ack'ing/propagating (the forwarded reset carries the
field down the ring), absent field = keep the current setting, so old
coordinators change nothing. Enabling requires M25_STATIC_KV (+SDPA)
from process start (static KV is allocated at Layer construction): the
A/B ring launches with M25_STATIC_KV=1 M25_EAGLE=1 and flips per job;
eager-with-static-KV is bit-identical to the cat path
(research/m25_statickv_test.py) so the off arm stays representative.
Without the prereqs a graph=true request is refused loudly and ignored.
Coordinator side: M25_GRAPH_JOB (env or in-process) stamps the field.
Evidence: full CPU suite green (64 passed); toggle-plumbing tests land
with tests/test_graph_aux.py in this branch.
* test: graph-aux CPU tests + GPU check script
tests/test_graph_aux.py (CPU, no GPU/model): _block's M25_GRAPH_MAX
bounding — capture within budget, eager fallback past it (counted),
captured shapes keep replaying, prefill never graphed — via a fake
GraphRunner recording captures; the per-job toggle plumbing (reset dict
-> set_graph -> _block routing, refusal without M25_STATIC_KV, absent
field = no change); coordinator stamps 'graph' via M25_GRAPH_JOB
(_reset_op unit + end-to-end over the fake ring, which now logs the
reset's graph field); subprocess import with M25_CUDA_GRAPH=1
M25_EAGLE=1 proves the SystemExit guard is gone; _merge_aux consumes
_AUX values unchanged (the contract the graph's aliased publish rests
on).
research/graph_aux_check.py (on-box): real-layer bit-equality of graph
replay vs eager-manual (h + aux) at 3 start_pos across 2 buckets, aux
FRESHNESS at a new start_pos with the same input (the stale-aux
regression), OOM-safe capture fallback, and eager-vs-replay ms/block
timing (the launch-overhead recovery on that box's CPU).
Evidence: python3 -m pytest tests/ -q -> 74 passed (10 new).
* chore(ops): forward graph knobs to ring processes
M25_GRAPH_MAX (stage-side capture cap) and M25_GRAPH_JOB (coordinator
reset stamp for the per-job A/B) must ride ENG_ENV or a scatter-launched
ring silently drops them — the exact set-locally-but-not-forwarded
measurement poisoning the list's comment warns about.
* fix(stage): drain device before eager fallback on capture failure
Review F1: a warmup/capture failure propagates before the side stream
is joined (current_stream().wait_stream(side)), so warmup's garbage
kc.index_copy_ at [alen-s, alen) can still be in flight when the
except-path run_block writes real KV at [start, start+s) on the default
stream — overlapping whenever alen < total+s -> corrupt KV -> wrong
committed tokens WITH valid receipts. torch.cuda.synchronize() at the
top of the failure handler drains all streams before any eager work.
The eager-marked fast path needs no sync: it runs no side-stream work
and the first failure already drained.
Evidence: full CPU suite green (74 passed); the OOM-safety probe in
research/graph_aux_check.py exercises this exact path on-box.
* fix(pipe): verify graph toggle ack + surface arm counters
Review F2/F3/F6/F9 — measurement validity for the per-job graph A/B:
F2: a refused toggle (one forgotten M25_STATIC_KV=1) was loud only in
that stage's own log; the coordinator discarded the bare reset ack, so
every graph:true job would silently run eager and the paid A/B would
bank 'graph doesn't help'. Now a graph-stamped reset is acked as
{ok, graph:<applied>, counters} (plain resets keep bare 'ok' for
old-coordinator compat) and _check_reset_ack raises LOUDLY on any
mismatch, including a bare-ok from a pre-toggle stage build. Only the
tail's applied value rides the ack — head/middle refusals stay
invisible (fine when the ring shares one env), so the refusal string
is grep-stable ('GRAPH REFUSED') and the runbook must grep every stage
log before trusting an arm.
F3: cap-exceeded skips now log once per new (s,bucket) shape, and
graph-A/B jobs' receipt sweeps return {receipts, graph, graph_captured,
graph_skipped}; both coordinators unpack it and surface it as
result['graph_arm'] — the job record shows how graphed the arm was.
F6: M25_GRAPH_JOB parses via an explicit truthy set ('no'/'off'/'0'
are now False). F9: added the missing set_graph refusal test for
M25_SDPA=0 with M25_STATIC_KV=1.
Evidence: python3 -m pytest tests/ -q -> 77 passed (3 new).
…leyten#26) The tail closed a LIVE coordinator's return channel on ANY predecessor-edge blip (kept ret only when already stale). An internal-ring leg hiccup while the coordinator was alive therefore killed its return channel, forcing a full reconnect that raced the return-tunnel recovery and wedged the ring. On a permissionless ring internal-leg blips are the steady state, so this made churn fatal — a demonstrated wedge that required a full re-warm to clear. Keep ret across a predecessor blip and hold the session stale until the coordinator's next reset re-arms it (stale=carried on re-accept); the interrupted job's in-flight replies are dropped meanwhile. Symmetric to _ret_send's ret-dies-keep-predecessor path. Adversarially reviewed: no interleaving computes a stale frame on reset KV (the stale gate drops every non-reset op; both coordinators reset-first after reconnect); coordinator-kill recovery is unchanged. Validated live: a warm ring healed a mid-decode blip via coordinator retry instead of wedging. 77 tests pass. Follow-ups (tracked): a short per-reply recv heartbeat so blip failover is seconds not up-to-timeout; an in-process serve()-tail churn test.
Raw `pkill -f PATTERN` self-terminates whenever the launching command line itself contains PATTERN — the classic kill-then-relaunch one-liner (`ssh box "pkill -f foo.py; python foo.py ..."`) kills its own shell before the relaunch runs, silently wiping the launch. This has bitten repeatedly. safe_kill walks its own PID + full ancestor chain (shell/ssh/sshd/...) and excludes them, so it can never kill the command that invoked it, while still killing every other cmdline match. For long-lived daemons prefer `pkill -x <name>` (argv0 match); use safe_kill when you must match the full command line. push_code ships it to every ring box for ad-hoc ops. tests/test_safe_kill.py (6): kills the target; the self-match caller SURVIVES while a separate same-pattern target still dies (the exact footgun); -s routes a real catchable signal; nothing-to-kill is exit 0. 83 tests pass.
…eyten#28) * feat(pipe): cwnd keep-warm noop sender on idle ring legs TCP slow-start-after-idle collapses cwnd when a connection idles past RTO, and serial decode idles every ring leg a full traversal between frames. Measured on a 40ms-RTT GB-BG vast leg (2026-07-05 leg probe): at idle<=300ms, 30KB-1.6MB frames complete in ~1 RTT (40-56ms); at idle=900ms the same frames take 80/120/160ms (2-4 RTTs, cwnd_p50 167->94); idle=100ms preserves cwnd fully. The kernel fix (net.ipv4.tcp_slow_start_after_idle) is read-only in vast containers (verified on 5 boxes), so the engine warms its own sockets. _KeepWarm wraps every sending socket: stage nxt_sock (incl. churn re-dials), tail ret (incl. mid-session re-adoption), and the coordinator pipe inside coordinate_pipe/_tree/_batch — wrapped there rather than at the call boundary so every caller (coord CLI, gateway, sweep) inherits one lock per socket with zero call-site changes. The lock serializes real sends against a daemon thread that posts a tiny {"op":"noop"} frame after M25_CWND_KEEPWARM_MS idle (default 0 = OFF; master behavior unchanged). Noops are leg-local: recv_data() and inline skips in the stage loops drop them without popping inflight, answering, forwarding, attesting, or timing them; the stage loops skip inline (not recv_data) so a noop-only idle predecessor cannot starve the tail's select set. recv_data enforces the socket timeout as an overall deadline so a noop-flooding wedged peer still trips supervision. _tail_accept classifies a noop greeting as a speaking predecessor (its frame is handed back and skipped), so stage replacement cannot kill-loop on a warm sender. Offline suite: 64 passed (unchanged from master). * feat(pipe): keepwarm runtime toggle via reset op M25_KEEPWARM_JOB on the coordinator arms its own pipe wrapper (coordinate_pipe/_tree/_batch) and rides the job's reset op as "keepwarm_ms"; stage reset handlers apply it to their wrapped senders (tail ret, head/middle nxt) and propagate the reset down the ring unchanged, so one field flips all legs. Absent env/field = no change; "0" is an explicit OFF arm. This lets the interleaved warm A/B toggle keep-warm per job on a live ring without relaunching stages (the graph-toggle pattern, implemented independently off master). reset_batch carries no field — stage toggling stays reset-only. * test: keepwarm cadence, interleave safety, recv skip, toggle, tail-accept CPU tests for the cwnd keep-warm lever (tests/test_cwnd_keepwarm.py): >=2 noops in 150ms at a 30ms interval on an idle socketpair, zero at interval 0 (and no thread); 300 tensor frames hammered against a 1ms noop thread decode intact and in order (the one-lock-per-socket guarantee); a dead socket never crashes the noop thread; recv_data skips noops and still times out under a noop flood; a noop injected before EVERY fake-ring reply leaves pipelined and divergence-trap runs byte-lossless (noops never pop inflight); M25_KEEPWARM_JOB=25 rides the reset as keepwarm_ms=25 and the coordinator noops during a 250ms ring stall, absent env sends neither; _tail_accept classifies a noop greeting as a speaking predecessor and hands it back as first_msg. fake_ring's oracle now logs+skips noops like serve() and records the reset's keepwarm field. Full suite: 77 passed. * fix(pipe): keepwarm noop send must never block reconnect/teardown Adversarial review: the noop daemon held self.lock across send_msg-> sendall, and attach()/set_interval()/stop() took the same lock. On the likely paid-ring fault — a GPU/driver hang whose TCP stays alive — the noop's sendall blocks up to the socket's PRODUCTION timeout (1800s on the gateway, forever on the tail's untimed ret after coordinator reconnect), so a subsequent attach(new_ret)/stop() blocked indefinitely, regressing churn-resilience (a stuck send pinned the reconnect). Reproduced live. Three composing guards: 1. the noop thread acquires the send lock NON-BLOCKING and skips the tick if a real send holds it (a send in progress => the leg isn't idle anyway), so it never queues behind — and gets pinned by — a send; 2. _noop_once bounds the send to <=2s (save/settimeout(<=2s)/restore in finally, re-reading self.sock once so an attach swap mid-tick is safe), so even a full-buffer peer releases the lock in ~2s; 3. attach() is a lockless atomic ref swap, stop()/close() set a lockless _stop flag, and runner spawn/exit use a separate _life lock — so NO teardown path takes the send lock. Max wait to reconnect/tear down is now ~0s; the stuck noop thread itself exits within the 2s bound. Tests: stuck-noop-send does not block attach()/stop() (full send buffer, untimed socket; asserts <3s and thread exits); non-blocking acquire keeps the noop thread responsive to a lockless stop() while a real send holds the lock; interleave test now asserts real-send forward progress is never stalled by the noop path. Full suite: 79 passed. * feat(pipe): ship cwnd keep-warm, default-ON for the interactive gateway Merges the reviewed+blocker-fixed keep-warm lever onto master. Keep-warm sends a tiny noop on each idle ring leg (~150ms cadence) so TCP slow-start-after-idle never collapses the congestion window between tokens — the single-stream reasoning regime idles legs 250-900ms, straddling the RTO, so a chunk of traversals otherwise pay 2-4 extra RTTs (measured: idle-900ms legs run 2-4x slower, scratchpad leg probe). Function for the PoC: a CONSISTENCY / tail-latency lever for jittery public-internet paths, not a throughput lever — it stops the occasional traversal from stuttering, it does not speed up a warm one. On a calm EU ring it is neutral (4-rep rotated A/B: chain+GW/chain+G ratio 1.01-1.04 across all cells incl. pipelined — an earlier single -rep 'breaks pipelining' read was WAN-window drift). Default-off in code; the --serve (interactive gateway) path defaults it to 150ms since idle-leg cwnd collapse is exactly its regime and cost is bounded (asymmetric payoff). Measurement paths stay off. VALIDATION DEBT: the ring-level benefit is proven only at the TCP layer (leg probe); a keep-warm ON/OFF A/B on a genuinely jittery/residential path (ties to the DoubleZero pilot) is still owed before defaulting it on beyond interactive. 98 tests pass.
* feat(pipe): cwnd keep-warm noop sender on idle ring legs
TCP slow-start-after-idle collapses cwnd when a connection idles past
RTO, and serial decode idles every ring leg a full traversal between
frames. Measured on a 40ms-RTT GB-BG vast leg (2026-07-05 leg probe):
at idle<=300ms, 30KB-1.6MB frames complete in ~1 RTT (40-56ms); at
idle=900ms the same frames take 80/120/160ms (2-4 RTTs, cwnd_p50
167->94); idle=100ms preserves cwnd fully. The kernel fix
(net.ipv4.tcp_slow_start_after_idle) is read-only in vast containers
(verified on 5 boxes), so the engine warms its own sockets.
_KeepWarm wraps every sending socket: stage nxt_sock (incl. churn
re-dials), tail ret (incl. mid-session re-adoption), and the
coordinator pipe inside coordinate_pipe/_tree/_batch — wrapped there
rather than at the call boundary so every caller (coord CLI, gateway,
sweep) inherits one lock per socket with zero call-site changes. The
lock serializes real sends against a daemon thread that posts a tiny
{"op":"noop"} frame after M25_CWND_KEEPWARM_MS idle (default 0 = OFF;
master behavior unchanged). Noops are leg-local: recv_data() and
inline skips in the stage loops drop them without popping inflight,
answering, forwarding, attesting, or timing them; the stage loops skip
inline (not recv_data) so a noop-only idle predecessor cannot starve
the tail's select set. recv_data enforces the socket timeout as an
overall deadline so a noop-flooding wedged peer still trips
supervision. _tail_accept classifies a noop greeting as a speaking
predecessor (its frame is handed back and skipped), so stage
replacement cannot kill-loop on a warm sender.
Offline suite: 64 passed (unchanged from master).
* feat(pipe): keepwarm runtime toggle via reset op
M25_KEEPWARM_JOB on the coordinator arms its own pipe wrapper
(coordinate_pipe/_tree/_batch) and rides the job's reset op as
"keepwarm_ms"; stage reset handlers apply it to their wrapped senders
(tail ret, head/middle nxt) and propagate the reset down the ring
unchanged, so one field flips all legs. Absent env/field = no change;
"0" is an explicit OFF arm. This lets the interleaved warm A/B toggle
keep-warm per job on a live ring without relaunching stages (the
graph-toggle pattern, implemented independently off master).
reset_batch carries no field — stage toggling stays reset-only.
* test: keepwarm cadence, interleave safety, recv skip, toggle, tail-accept
CPU tests for the cwnd keep-warm lever (tests/test_cwnd_keepwarm.py):
>=2 noops in 150ms at a 30ms interval on an idle socketpair, zero at
interval 0 (and no thread); 300 tensor frames hammered against a 1ms
noop thread decode intact and in order (the one-lock-per-socket
guarantee); a dead socket never crashes the noop thread; recv_data
skips noops and still times out under a noop flood; a noop injected
before EVERY fake-ring reply leaves pipelined and divergence-trap runs
byte-lossless (noops never pop inflight); M25_KEEPWARM_JOB=25 rides the
reset as keepwarm_ms=25 and the coordinator noops during a 250ms ring
stall, absent env sends neither; _tail_accept classifies a noop
greeting as a speaking predecessor and hands it back as first_msg.
fake_ring's oracle now logs+skips noops like serve() and records the
reset's keepwarm field. Full suite: 77 passed.
* fix(pipe): keepwarm noop send must never block reconnect/teardown
Adversarial review: the noop daemon held self.lock across send_msg->
sendall, and attach()/set_interval()/stop() took the same lock. On the
likely paid-ring fault — a GPU/driver hang whose TCP stays alive — the
noop's sendall blocks up to the socket's PRODUCTION timeout (1800s on
the gateway, forever on the tail's untimed ret after coordinator
reconnect), so a subsequent attach(new_ret)/stop() blocked
indefinitely, regressing churn-resilience (a stuck send pinned the
reconnect). Reproduced live.
Three composing guards:
1. the noop thread acquires the send lock NON-BLOCKING and skips the
tick if a real send holds it (a send in progress => the leg isn't
idle anyway), so it never queues behind — and gets pinned by — a send;
2. _noop_once bounds the send to <=2s (save/settimeout(<=2s)/restore in
finally, re-reading self.sock once so an attach swap mid-tick is
safe), so even a full-buffer peer releases the lock in ~2s;
3. attach() is a lockless atomic ref swap, stop()/close() set a lockless
_stop flag, and runner spawn/exit use a separate _life lock — so NO
teardown path takes the send lock. Max wait to reconnect/tear down is
now ~0s; the stuck noop thread itself exits within the 2s bound.
Tests: stuck-noop-send does not block attach()/stop() (full send buffer,
untimed socket; asserts <3s and thread exits); non-blocking acquire
keeps the noop thread responsive to a lockless stop() while a real send
holds the lock; interleave test now asserts real-send forward progress
is never stalled by the noop path. Full suite: 79 passed.
* feat(pipe): ship cwnd keep-warm, default-ON for the interactive gateway
Merges the reviewed+blocker-fixed keep-warm lever onto master. Keep-warm sends a
tiny noop on each idle ring leg (~150ms cadence) so TCP slow-start-after-idle
never collapses the congestion window between tokens — the single-stream reasoning
regime idles legs 250-900ms, straddling the RTO, so a chunk of traversals otherwise
pay 2-4 extra RTTs (measured: idle-900ms legs run 2-4x slower, scratchpad leg probe).
Function for the PoC: a CONSISTENCY / tail-latency lever for jittery public-internet
paths, not a throughput lever — it stops the occasional traversal from stuttering,
it does not speed up a warm one. On a calm EU ring it is neutral (4-rep rotated A/B:
chain+GW/chain+G ratio 1.01-1.04 across all cells incl. pipelined — an earlier single
-rep 'breaks pipelining' read was WAN-window drift). Default-off in code; the --serve
(interactive gateway) path defaults it to 150ms since idle-leg cwnd collapse is exactly
its regime and cost is bounded (asymmetric payoff). Measurement paths stay off.
VALIDATION DEBT: the ring-level benefit is proven only at the TCP layer (leg probe);
a keep-warm ON/OFF A/B on a genuinely jittery/residential path (ties to the DoubleZero
pilot) is still owed before defaulting it on beyond interactive. 98 tests pass.
* docs(engine): RESUME HERE — graph-aux/churn/safe_kill/keep-warm shipped; ring destroyed
Session living-state update: four PRs landed (leyten#25 graph-aux +74% mech-verified,
leyten#26 return-channel churn fix, leyten#27 safe_kill, leyten#28 keep-warm default-on interactive),
DoubleZero assessment banked, ring torn down (0 live). Records the open debt (churn
F6 heartbeat + serve-tail test, keep-warm jitter validation, FWD_RET tunnel
robustness) and the next-session pickup. Adds the interleaved lever-bench tool
(research/m25_lever_bench.py) used for the graph/keep-warm A/Bs.
…-only, lossless) The chain coordinator's round length K is per-FRAME, not per-connection: each verify frame is [anchor]+ds at a start, stages crop KV to start and accept any s (prefill already ships s=512 chunks). Acceptance is streaky (survivorship: per-depth accept probability rises inside accepted runs), so a two-state policy upsizes the NEXT draft request to M25_AK_HI (default 16) after a round that accepted the whole frame or >= M25_AK_UP (default 6) tokens, and decays straight back to base K otherwise. K changes what is DRAFTED per round, never what is committed - the ring greedy-verifies every proposal, so losslessness is structural, not statistical. Coordinator-only, zero wire/stage change (stages already accept arbitrary chunk lengths); mixed coordinator/stage versions stay compatible. Default OFF (M25_ADAPTIVE_K unset) = master byte-exact: the policy never moves, next_k() always returns base K, and the result row carries no new fields. Opt-in M25_AK_TRACE=1 adds a per-round (kf, n) trace to the result row for acceptance-structure analysis. Measured on a scattered-EU WAN 5-stage RTX 5090 ring (274.8 ms loop RTT, MiniMax-M2.5 NVFP4, base K=8 depth 4, interleaved A/B, pre-registered): verbatim/n-gram cell +45% tok/s (118.0 -> 171.5, n=2/arm, output byte-identical to the OFF arm, 25/31 rounds upsized) - recovering essentially all of fixed-16's gain while keeping the base-8 payload on bust rounds; novel/EAGLE cell +10% tokens-per-round (2.32 -> 2.55, deterministic 3/3, only 3/100 rounds upsized - the policy pays the larger payload only where acceptance justifies it). CPU tests (tests/test_adaptive_k.py, no GPU needed): flag-OFF bit-exact vs a master-loop mirror across 60 K x depth x mask decodes (commits, rounds, wasted, frame traces), losslessness asserted at every commit on the ON path, round-economics bounds, policy transition table, env overrides, 180-decode fuzz.
463464q435q43
force-pushed
the
mosaic/adaptive-k
branch
from
July 7, 2026 07:09
1e9e803 to
aba20ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR: Adaptive round-length policy (
M25_ADAPTIVE_K) — lossless, coordinator-only, +45% on the verbatim cellTarget:
leyten/shardmaster (based on182e93b, applies clean — verified locally).Branch:
mosaic/adaptive-k· Files:phase0/m25_pipe.py(+70/−13),tests/test_adaptive_k.py(new, 250 lines, CPU-only).Motivation
The chain coordinator's round length K is per-frame, not per-connection: every verify frame is
[anchor]+dsat astart, stages crop KV tostartand accept any chunk length (prefill alreadyships s=512 chunks). Acceptance on real workloads is streaky — measured per-depth accept
probability rises inside an accepted run (survivorship). A fixed K therefore leaves throughput on
the table exactly where acceptance is best. The paper's §4.2 law names α and T as the only levers on
the draftable number; K is a third, and it is free.
Mechanism
A two-state policy (
_AdaptiveK, ~40 lines, no torch, no I/O) over the base K:M25_AK_UP(default 6) tokens, the nextdraft request is
M25_AK_HI(default 16) tokens;(request→fetch is snapshot-only on both drafters, so this costs nothing);
Losslessness is structural, not statistical: K changes what is drafted per round, never what
is committed — the ring greedy-verifies every proposal. Zero wire/stage change; mixed
coordinator/stage versions stay compatible. Default OFF = master byte-exact (policy state never
moves,
next_k()always returns base K, result row carries no new fields). Opt-inM25_AK_TRACE=1adds a per-round
(kf, n)trace to the result row for acceptance-structure analysis.Measured results (receipts, pre-registered A/B before data)
Regime: MiniMax-M2.5 NVFP4, single-stream B=1, scattered-EU WAN 5-stage RTX 5090 ring,
274.8 ms loop RTT, engine
182e93b, bf16 wire, keep-warm OFF, greedy, base K=8 depth 4,hybrid n-gram+EAGLE drafter. Receipt
mosaic-m25wan-ring2-ak-adaptive-20260706.Mechanism diagnostic: 25/31 rounds upsized on the verbatim cell vs 3/100 on the novel cell — the
policy pays the K=16 payload only where acceptance justifies it. Adaptive recovers essentially all
of fixed-16's verbatim-cell gain while keeping the base-8 payload on bust rounds.
Supporting structure receipt (
mosaic-m25wan-ring2-ak-trace-20260707, usesM25_AK_TRACE):i.i.d. per-token acceptance rejected at p=0.0044 (exact DP test, pre-registered α=0.01);
verbatim-cell acceptance is position-deterministic. This is why a streak policy beats any
fixed K: the streak signal is real, not noise.
Caveats we state up front
(greedy tie-flips when the drafted continuation length changes the tie context). Within-config
it is deterministic (3/3 identical). On the verbatim cell byte-identity holds across policies.
If upstream wants byte-identity across policies as a hard invariant on all cells, the flag
documents this as its one behavioral note.
round-time share is smaller and the win will be smaller. The mechanism (fewer rounds) is
ring-independent; the magnitude is not.
mechanism-grade evidence there.
Verification protocol (how a reviewer checks losslessness)
python3 -m pytest tests/test_adaptive_k.py -q— CPU-only, no GPU: flag-OFF bit-exactness vs amaster-loop mirror across 60 K×depth×mask decodes (commits, rounds, wasted counts AND frame
traces), losslessness asserted at every commit on the ON path, policy transition table, env
overrides, 180-decode fuzz.
M25_ADAPTIVE_Kunset and=1on a verbatim-classprompt;
sha256the outputs — identical. Flag unset: result row is byte-identical to master's(no new fields).
License / transfer
Contributed under the repository's Apache-2.0 license, no strings attached. The measured claims
above are ours (Mosaic Intelligence), backed by signed receipts + raw per-run JSONs we retain;
we're happy to share raw artifacts, re-run on request, or assist a maintainer replication on a
tight-EU ring before merge.
Prepared 2026-07-07 by the authors (Mosaic Intelligence).