feat: multi-node comm data (world 8/16, gb200) + data-driven beyond-node custom-AR fan-out - #1593
Open
Arsene12358 wants to merge 1 commit into
Open
feat: multi-node comm data (world 8/16, gb200) + data-driven beyond-node custom-AR fan-out#1593Arsene12358 wants to merge 1 commit into
Arsene12358 wants to merge 1 commit into
Conversation
… multi-node comm rows (world 8/16) query_custom_allreduce_scaled previously hard-clamped tp to num_gpus_per_node and extrapolated beyond it with a p2p-bandwidth ratio, because comm collection was single-node and no table recorded larger fan-outs. On gb200 (modeled as 4-GPU nodes) that priced tp16 allreduces at ~2.7us while 16-rank serving (sglang pynccl ring across the NVL72 segment) measures ~48-50us per AR (~185 ARs/decode step), driving a 37% median TPOT under-prediction for Qwen3.8-Max sglang fp8 tp16. Change: when the custom_allreduce table records fan-outs beyond the node size (multi-node collection), use the largest recorded fan-out <= tp directly and only bandwidth-scale beyond it. Tables that stop at or below the node size keep the legacy clamp+scale behavior bit-for-bit (pinned by the new unit test; full rust suite and sdk pytest unchanged). Data (collected on oci-hsg GB200 NVL72, 4 nodes x 4 GPUs, segment-pinned, slurm 6515005/6516705): - gb200 comm/nccl/2.23: +336 rows — nccl-tests sweeps (all_reduce, all_gather, reduce_scatter, alltoall x half/int8) at num_gpus 8 and 16 (512B..512MB), NCCL 2.27.7. Consumed by query_nccl_scaled users (alltoall/AG/RS at >4 ranks) which previously bandwidth-scaled from the 4-rank curves. - gb200 comm/sglang/0.5.14: +92 rows — collect_all_reduce.py --backend sglang under srun (--use-slurm) at world 8/16 on the sglang v0.5.17 image; at >node fan-outs sglang custom AR disables itself and the rows measure the pynccl fallback, i.e. the exact serving lane. vLLM rows at world 8/16 were collected in the same jobs but deliberately NOT landed: the vllm 0.27.1 benchmark measures ~47us at 16 ranks (its custom AR and symm-mem lanes both refuse cross-node groups), yet vllm serving at tp16 measures 8.96 ms/step bs1 with 11% prediction accuracy under the legacy pricing — repricing its ARs to the benchmarked value would break a validated arm. Evidence banked in AIC-1808. Validation (Qwen3.8-Max fp8 tp16 gb200 vs measured serving, stacked on the qwen3.8 support branch): sglang TPOT errs move from -54/-37/-18% to -12.7/-14.6/-5.7% (conc 1/8/32); static per-step bs1 -57% -> -17.5%, bs32 -26% -> -11.7%; vllm predictions byte-identical. Linear: AIC-1808 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Yiming Liu <yimingl@nvidia.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (3)
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review. 📜 Recent review details⏰ Context from checks skipped due to timeout. (16)
🧰 Additional context used📓 Path-based instructions (1)- Prefer applicable inline comments. When the correct fix is clear, small, and limited to the commented diff hunk, include it as a GitHub Suggested Change so the author can apply it with one click.⚙️ CodeRabbit configuration file Files:
WalkthroughChangesThe custom-allreduce scaling query now uses recorded beyond-node fan-outs before extrapolating overflow. Regression and parity coverage add TP16 Qwen3.5-397B-A17B execution modes. Custom Allreduce Scaling
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to This change updates multi-node communication pricing and adds measured GB200 data with parity coverage; no actionable merge-blocking risk remains based on the supplied evidence, so it is merge-ready after normal checks and review. Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Full details: Description checkExplanation The description is detailed and covers the change scope, implementation details, validation results, exclusions, and related Linear issue. It does not use the template headings and does not explicitly identify where reviewers should start, but it is mostly complete. Comment |
Contributor
Perf Parquet Diff Report
Compared
Per-File Row Diff PreviewShowing the first 3 rows per diff kind for each changed parquet file. Full exact CSVs are in aic-core/src/aiconfigurator_core/systems/data/gb200/comm/nccl/2.23/nccl_perf.parquet
added rows - full CSV: framework,version,device,op_name,kernel_source,nccl_dtype,num_gpus,message_size,latency
TRTLLM,2.27.7,NVIDIA GB200,all_gather,NCCL,half,16,1024,0.02846
TRTLLM,2.27.7,NVIDIA GB200,all_gather,NCCL,half,16,1048576,0.03176
TRTLLM,2.27.7,NVIDIA GB200,all_gather,NCCL,half,16,131072,0.03131aic-core/src/aiconfigurator_core/systems/data/gb200/comm/sglang/0.5.14/custom_allreduce_perf.parquet
added rows - full CSV: framework,version,device,op_name,kernel_source,allreduce_dtype,num_gpus,message_size,latency,backend
SGLang,0.5.17,NVIDIA GB200,all_reduce,SGLang_CustomAllReduce_eager,bfloat16,16,1024,0.1678281593322754,sglang_eager
SGLang,0.5.17,NVIDIA GB200,all_reduce,SGLang_CustomAllReduce_eager,bfloat16,16,1048576,0.1297692775726318,sglang_eager
SGLang,0.5.17,NVIDIA GB200,all_reduce,SGLang_CustomAllReduce_eager,bfloat16,16,128,0.1257126426696777,sglang_eagerArtifact Contents
|
Contributor
Sanity Check Chart Generation Report📥 Download all sanity charts from workflow artifacts New perf data files were detected in this PR. Please use the link above to Below is a report of whether the chart generation was successful for each op. Chart Generation Report for system: gb200, backend: sglang, backend_version: 0.5.14
|
Arsene12358
marked this pull request as ready for review
August 26, 2026 05:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the sglang multi-node TPOT under-prediction on gb200 (Linear AIC-1808). One rust change plus measured multi-node comm data; one new parity golden pins the repriced path end-to-end.
query_custom_allreduce_scaledhonors recorded beyond-node fan-outs. The query previously hard-clamped tp tonum_gpus_per_nodeand extrapolated beyond it with a p2p-bandwidth ratio, because comm collection was single-node and no table recorded larger fan-outs. On gb200 (modeled as 4-GPU nodes; the NVL72 reroute is gated onnum_gpus_per_node == 72and never fires) that priced tp16 allreduces at ~2.7µs while 16-rank serving measures ~48–50µs per AR across the NVL72 segment (~185 ARs per decode step). Now, when the custom-AR table records fan-outs beyond the node size, the query uses the largest recorded fan-out ≤ tp directly and only bandwidth-scales beyond it. Tables that stop at or below the node size keep the legacy clamp+scale behavior bit-for-bit (pinned by a new unit test; the legacy formula is algebraically unchanged in that case).comm/nccl/2.23: +336 rows — nccl-tests sweeps (all_reduce, all_gather, reduce_scatter, alltoall × half/int8) at num_gpus 8 and 16, 512B–512MB, collected on oci-hsg GB200 NVL72 (4 nodes × 4 GPUs, segment-pinned, srun--mpi=pmix, NCCL 2.27.7). Feedsquery_nccl_scaledconsumers (alltoall/AG/RS at >4 ranks) which previously bandwidth-scaled from the 4-rank curves.comm/sglang/0.5.14: +92 rows —collector/network/collect_all_reduce.py --backend sglang --use-slurmat world 8/16 on the sglang v0.5.17 image. At beyond-node fan-outs sglang custom AR disables itself ("process group spans across nodes"), so the rows measure the pynccl fallback — the exact serving lane. Three-way agreement at 16 ranks: 48.8µs (this collector, graph @128b) ≈ ~50µs (nsys-audited serving decode) ≈ ~48µs (nccl-tests).Validation
Stacked on the Qwen3.8 support branch (#1583) and compared against measured serving (GB200 NVL72 tp16 fp8, direct engine, jobs 6491036/6506559/6512879 — details in the AIC-1782/AIC-1808 Linear trail):
Suites: cargo 515 passed; parity 451 passed (4 new golden records for the added tp16 gb200/sglang case, append-only pin); sdk pytest 2027 passed.
Deliberately not included
versioncolumn (nccl 2.27.7, sglang 0.5.17) while living in the resolution-keyed dirs (2.23,0.5.14) — same precedent as the existing2.23dir whose rows self-report 2.29.2; loaders ignore the column.Linear: AIC-1808
🤖 Generated with Claude Code
Summary by CodeRabbit
Improvements
Validation