perf(stage): tree-frame CUDA graphs — padded-template capture re-enables per-stream trees#138
Conversation
…-stream trees Per-stream trees raised g +15-70% committed/round but lost -29..-63%/stream live (receipt perstream-trees-ab-20260712): tree frames ran eager stage-side, 154ms summed stage compute vs 45ms for graph-replayed chains. The variable node count N (trunk + M drafted) defeated fixed-shape capture. This pads every de-lockstep tree frame to ONE fixed template (M25_TREE_PAD, default trunk_max+M rounded up to 8 = 24) and captures run_block_tree_row per context bucket in a TreeRowGraphRunner; _TGraphState statics (row, wcp, RoPE rows, padded ancestor mask) are refreshed per replay from host wire lists, composed on device. Dummy nodes write k/v contiguously at [start+n, start+npad) — the speculative-junk class the dirty-frontier contract already overwrites-before-read — and clone node 0's input/mask/position so their 62-layer trajectory is a real token's (no NaN ingress via masked reads). Tree graphs spend their own M25_TREE_GRAPH_MAX budget so variable-s chain shapes can't starve them; N>npad, hatch off, over-budget, or failed capture degrade to the eager tree path. M25_TREE_GRAPH=0 = escape hatch. CPU gate 0 green (tests/test_tree_graph.py, 16): template math verbatim vs build_tree_mask, contiguous row-local writes, masked-junk inertness, padded attention bit-equal to unpadded on real rows, routing/budget semantics. On-box gates (bit-equality vs eager-padded, argmax agreement, timing) ride research/tree_graph_check.py next.
Gates 1/2/2b of the padded-capture correctness hierarchy on real layers: capture faithfulness (replay bit-equal to the eager-padded reference incl. aux), padded-vs-unpadded numerics class (cosine bar 0.999 + the dummy- magnitude probe for cross-token MoE scale coupling), row/span-local KV no-poison, and the eager/graph/chain timing split the lever exists for.
The independent adversarial pass returned no BROKEN verdicts (junk-slot invariant and dummy finiteness confirmed safe as the single contract: overwrite-before-committed-read + masked-junk-finiteness) but flagged two unproven residues; both now have gates: 1c reads the live GPU statics back and compares against an independent CPU-composed _TGraphState (a device-side compose bug is invisible to gate 1, which shares statics between its arms), 1d proves a padding-only overflow near the KV cap degrades to eager instead of raising or capturing. Also a doc nit (pid H2D) from the same pass.
…s = shipped class Single 5090 (layers 29-31, cutlass sm_120): graph replay bit-equal to the eager-padded reference across 6 shapes incl. aux (maxabs 0.0); GPU statics == CPU compose; padding-only overflow degrades to eager; writes bit-confined to the frame's row+span; MoE activation scales proven token-local (x3-dummy delta exactly 0). Padded-vs-unpadded diverges at 2 layers (worst cos 0.9717) but per-op deltas are 3e-2/2e-2 and the compounding is MoE router near-tie amplification — the same measured class as the SHIPPED solo chain-graph lever on the same layers (cos 0.9873), so gate 2 is precedent-anchored and the ring accept/g stays the end judge, exactly as it was for chain graphs. tree_pad_isolate.py and chain_class_ref.py carry the isolation + precedent measurements. Timing on the idle box: eager 3.77ms -> graph 2.20ms per tree frame (2 layers); the contended-ring round decomposition is bar 3's number.
The vast API returned one ssh proxy endpoint for two different instances (proven 2026-07-23: same hostname + GPU UUID over both iids' mappings), so bootstrap, code push, and key collection silently hit the same box twice — caught only by the duplicate-receipt-pubkey guard. SHARD_SSH_OVERRIDE= "iid=host:port" pins an explicit endpoint (the direct 22/tcp map keeps working when the proxy map breaks).
…n the head can't dial the tail A consumer-ISP peering hole left the tail's advertised addr unreachable from the head only (reachable from everywhere else, incl. this box); the return forward's dial set is exactly one maddr, so the ring warms but every coordinator return connection resets. The override points the head's return forward at the tail's /p2p-circuit addr on a launch relay; DCUtR upgrades to direct where possible, and the tail's head-PeerId allowlist is unaffected (RemotePeer survives a circuit).
A live A/B arm was numerically ambiguous between graphs-never-fired and graphs-fired-but-bytes-bind: captures were silent on success and the tree counter is separate from the graph_arm stamp. One loud line per capture makes every stage log carry the fired-or-not proof.
…p8-aux chain win Capture-proven tree graphs on a 6x5090 EU ring: the g lever reproduces (+13..+76% committed/round) but per-stream tok/s stays net-negative (-22..-61% at B=4) because tree rounds pay a flat ~207ms — the naked ring traversal (sync round drains the pipeline) — vs 60-87ms for chains hidden behind de-lockstep + keep-warm. The fp8-aux control pass isolates the terms: halved aux lifted chains +20..+125%/stream (a standalone win, g held) and moved trees not at all, ruling bytes out and compute out (capture logs), leaving coordinator scheduling as the scoped next lever. Kill criterion stands for this ring class; the solo +18% tree win is unaffected.
|
Gate 3 (ring A/B) is in — docs/receipts/tree-graph-ring-ab-20260724.json. Verdict: capture-PROVEN tree graphs on a 6x5090 EU ring ('tree capture OK' on every stage). The g lever reproduces (+13..+76% committed/round) but per-stream tok/s remains net-negative (-22..-61% at B=4): tree rounds cost a flat ~207ms — the naked ring traversal, paid because a tree round drains the pipeline — vs 60-87ms for chains hidden behind de-lockstep + keep-warm. An M25_FP8_AUX=1 control pass isolates the terms: halved aux lifted CHAIN arms +20..+125%/stream (standalone win, g held, receipts valid) and moved tree rounds not at all. So: compute — fixed by this PR (proven); bytes — not the tree binder; sync-round latency — the binder, a coordinator-scheduling lever now precisely scoped (pipeline tree rounds across streams). The kill criterion stands for this ring class; solo trees (+18% banked) unaffected. This PR's capture machinery is correct and required for any future tree scheduling work; the two launcher hardening fixes (SHARD_SSH_OVERRIDE, SHARD_RET_MADDR circuit return) came out of real WAN failures found during the A/B and stand on their own. |
…ever M25_FP8_AUX already defaults to 1 under M25_FP8_WIRE=1; the 07-12 trees receipt env pinned it to 0 (contradicting that session's own runbook) and this A/B copied that env for apples-to-apples. So the +14..+54%/stream at B=4 is evidence the BANKED SCORECARD IS UNDERSTATED, not a default to flip. Also records the methodology caveat the first write-up omitted: unlike the interleaved 3-rep tree A/B, the aux comparison is cross-launch and single-rep, so it needs an interleaved SWEEP_FP8AUX_ARMS confirmation before anyone re-quotes the scorecard.
|
Correction to the gate-3 comment above, before merge: the fp8-aux number is a receipt correction, not a new lever.
Methodology caveat now recorded in the receipt too: unlike the tree A/B (interleaved off,on inside one process, 3 reps), the aux comparison is cross-launch and single-rep. Direction is consistent across all 10 arms, but it is suggestive rather than banked — confirmation is an interleaved |
1 participant
Per-stream trees raised g +15-70% committed/round but hit the kill criterion live (-29..-63%/stream, receipt perstream-trees-ab-20260712): tree frames ran EAGER stage-side - 154ms summed stage compute vs 45ms graph-replayed chains - because the variable node count defeats fixed-shape capture.
This pads every de-lockstep tree frame to ONE fixed template (M25_TREE_PAD, derived trunk_max+M -> 24) and captures run_block_tree_row per context bucket (TreeRowGraphRunner/_TGraphState). Per-replay statics carry (row, start, topology) from host wire lists; the mask is composed on-device with build_tree_mask's values verbatim. Dummy nodes write KV contiguously at [start+n, start+npad) - the speculative-junk-past-the-frontier class chain verify frames already leave at rejected slots - and clone node 0 (input/mask/RoPE) so their trajectory is a real token's (bounded; no NaN ingress via masked reads). Tree graphs spend their own M25_TREE_GRAPH_MAX so variable-s chain shapes can't starve them. Hatches: M25_TREE_GRAPH=0; oversize/over-budget/capture-fail/near-cap -> eager, never fatal. Solo-mode trees stay eager (same machinery applies later).
Correctness (receipts in-tree):
Timing (idle box, 2 layers): eager tree 3.77ms -> graphed 2.20ms (1.72x recovery); the decisive number is the ring round decomposition - the perstream-trees-ab A/B re-run (gate 3) lands as a follow-up receipt.
Also: coordinator row-mode ctx headroom now covers the pad span (max(16, trunk+M, npad)). Pre-existing solo-tree headroom (flat 16 < N=21) noted, not touched.