Add Qwen3.5-397B-A17B B200 BF16 aggregated recipes for 1k1k#180
Add Qwen3.5-397B-A17B B200 BF16 aggregated recipes for 1k1k#180nlevin-ui wants to merge 3 commits into
Conversation
Two recipes for the Qwen3.5-397B-A17B model on B200 GPUs (BF16, TP8): - max-tpt.yaml: high concurrency (16/32/64) with scheduler-recv-interval=30 - low-latency.yaml: low concurrency (4/8) with scheduler-recv-interval=10 Incorporates SGLang optimizations (chunked-prefill, flashinfer allreduce fusion, CUDA graph tuning) validated to match/exceed colleague benchmarks.
📝 WalkthroughWalkthroughAdded two new deployment recipes for the b200-bf16 1k1k benchmark: Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
recipes/b200-bf16/1k1k/low-latency.yaml (1)
33-34:cuda-graph-max-bsandmax-running-requestsare over-provisioned relative to target concurrencies.With
concurrencies: "4x8"the maximum in-flight request count is 8, yetcuda-graph-max-bs: 128precompiles CUDA graphs for all batch sizes up to 128 andmax-running-requests: 128reserves scheduler slots for 16× the actual peak. This consumes extra GPU memory (CUDA graphs) and startup time without providing any benefit for this workload. Aligning these to a value like 16–32 would reduce warm-up overhead and memory pressure.♻️ Suggested change
- cuda-graph-max-bs: 128 - max-running-requests: 128 + cuda-graph-max-bs: 16 + max-running-requests: 16🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/b200-bf16/1k1k/low-latency.yaml` around lines 33 - 34, The values for cuda-graph-max-bs and max-running-requests are over-provisioned for the declared concurrencies ("4x8"); reduce both settings to a value in the 16–32 range (instead of 128) so CUDA graph precompilation and scheduler slots align with the actual max in-flight requests (~8), cutting warm-up time and GPU memory use; update the entries for cuda-graph-max-bs and max-running-requests accordingly.recipes/b200-bf16/1k1k/max-tpt.yaml (1)
5-5:sglang-nightlycontainer is unversioned — consider pinning for reproducible benchmarks.Using a rolling nightly tag makes it impossible to reproduce or compare benchmark results across runs if the container image updates between executions. Consider pinning to a specific nightly digest or dated tag (e.g.,
sglang-nightly-2026-02-19) when publishing validated benchmark results.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/b200-bf16/1k1k/max-tpt.yaml` at line 5, The container field currently uses the unpinned tag "sglang-nightly"; update the container value to a reproducible, pinned identifier (either a dated nightly tag like "sglang-nightly-YYYY-MM-DD" or an immutable digest sha256:<digest>) so benchmark runs are reproducible; locate the container: "sglang-nightly" entry in max-tpt.yaml and replace it with the chosen pinned tag or digest and document the tag/digest used for the published benchmark.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Line 36: The recipe’s streaming flush interval is too high for a low-latency
profile: update the stream-interval key in the low-latency recipe (alongside the
already reduced scheduler-recv-interval) from 30 to a much smaller value (e.g.,
1 or 2) so the server flushes decode tokens frequently for streaming clients;
locate the stream-interval entry in the low-latency recipe and change its value
accordingly.
---
Nitpick comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Around line 33-34: The values for cuda-graph-max-bs and max-running-requests
are over-provisioned for the declared concurrencies ("4x8"); reduce both
settings to a value in the 16–32 range (instead of 128) so CUDA graph
precompilation and scheduler slots align with the actual max in-flight requests
(~8), cutting warm-up time and GPU memory use; update the entries for
cuda-graph-max-bs and max-running-requests accordingly.
In `@recipes/b200-bf16/1k1k/max-tpt.yaml`:
- Line 5: The container field currently uses the unpinned tag "sglang-nightly";
update the container value to a reproducible, pinned identifier (either a dated
nightly tag like "sglang-nightly-YYYY-MM-DD" or an immutable digest
sha256:<digest>) so benchmark runs are reproducible; locate the container:
"sglang-nightly" entry in max-tpt.yaml and replace it with the chosen pinned tag
or digest and document the tag/digest used for the published benchmark.
| cuda-graph-max-bs: 128 | ||
| max-running-requests: 128 | ||
|
|
||
| stream-interval: 30 |
There was a problem hiding this comment.
stream-interval: 30 is inconsistent with the low-latency intent.
scheduler-recv-interval is correctly reduced to 10 for low-latency, but stream-interval remains at 30 (identical to max-tpt.yaml). A stream-interval of 30 means the server accumulates 30 decode tokens before flushing to streaming clients, adding observable streaming latency. If streaming clients will use this recipe, consider lowering stream-interval (e.g., to 1 or 2) to align with the low-latency goal.
If sa-bench measures only aggregate TTFT/TPOT and does not exercise the streaming path, this is a no-op for benchmarking but may mislead users who copy this config for streaming deployments.
🔧 Suggested change
- stream-interval: 30
+ stream-interval: 1
scheduler-recv-interval: 10📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| stream-interval: 30 | |
| stream-interval: 1 | |
| scheduler-recv-interval: 10 |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/b200-bf16/1k1k/low-latency.yaml` at line 36, The recipe’s streaming
flush interval is too high for a low-latency profile: update the stream-interval
key in the low-latency recipe (alongside the already reduced
scheduler-recv-interval) from 30 to a much smaller value (e.g., 1 or 2) so the
server flushes decode tokens frequently for streaming clients; locate the
stream-interval entry in the low-latency recipe and change its value
accordingly.
Enable hardware-specialized backends for B200: - attention-backend: trtllm_mha (TRT-LLM fused MHA kernel) - moe-runner-backend: flashinfer_trtllm (grouped GEMM for MoE experts) - enable-symm-mem: true (symmetric memory for faster all-reduce) trtllm backends showed +2-5% throughput improvement across all concurrency levels. Symmetric memory run pending validation.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
recipes/b200-bf16/1k1k/low-latency.yaml (1)
36-37:cuda-graph-max-bsandmax-running-requestsare over-provisioned for a low-latency profile.Both values are 128, identical to
max-tpt.yaml, but the maximum concurrency here is only 8. Two effects:
cuda-graph-max-bs: 128causes SGLang to compile and cache CUDA graphs for batch sizes 1, 2, 4, …, 128 at server startup — substantially increasing warm-up time for a low-latency deployment where the decode batch will never exceed ~8.max-running-requests: 128permits deep queuing. Whilesa-benchcaps load externally and benchmark numbers are unaffected, production users inheriting this recipe can inadvertently allow high queue depths that destroy tail latency.♻️ Proposed tighter low-latency values
- cuda-graph-max-bs: 128 - max-running-requests: 128 + cuda-graph-max-bs: 16 + max-running-requests: 16🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/b200-bf16/1k1k/low-latency.yaml` around lines 36 - 37, The low-latency profile is over-provisioned: reduce cuda-graph-max-bs from 128 to a value matching expected max decode batch (e.g., 8) so SGLang only precompiles graphs up to needed sizes, and lower max-running-requests from 128 to a small queue cap (e.g., 8 or 16) to prevent deep queuing and protect tail latency; update the values for the keys cuda-graph-max-bs and max-running-requests accordingly in the recipe.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Line 42: The recipe enables unvalidated symmetric memory via the
enable-symm-mem key in low-latency.yaml; change it to a safe default (disable or
remove) until the all-reduce/symmetric-memory path is validated to avoid silent
correctness regressions at low concurrency. Locate the enable-symm-mem entry in
the low-latency.yaml recipe (same setting referenced in max-tpt.yaml) and either
set enable-symm-mem: false or remove the key, and add a short inline comment
referencing the pending validation ticket so future readers know why it’s
disabled.
In `@recipes/b200-bf16/1k1k/max-tpt.yaml`:
- Line 42: The recipe currently sets enable-symm-mem: true which is unvalidated;
remove this key from the YAML (or set it to false) until symmetric-memory is
fully validated, or if you must keep it temporarily add a clear inline comment
beside enable-symm-mem referencing "provisional - pending validation" and link
to the PR/issue that tracks validation; update any documentation or README
references in this recipe that mention symmetric-memory to reflect the
provisional status.
---
Duplicate comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Line 39: Update the streaming interval to match the low-latency intent by
reducing the stream-interval key (currently set to 30) to the same lower value
used for scheduler-recv-interval (e.g., 10) or another appropriately small
value; locate the stream-interval entry in the recipe
(recipes/b200-bf16/1k1k/low-latency.yaml) and change its value so streaming
flushes occur more frequently and do not add the 30-token decode delay.
---
Nitpick comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Around line 36-37: The low-latency profile is over-provisioned: reduce
cuda-graph-max-bs from 128 to a value matching expected max decode batch (e.g.,
8) so SGLang only precompiles graphs up to needed sizes, and lower
max-running-requests from 128 to a small queue cap (e.g., 8 or 16) to prevent
deep queuing and protect tail latency; update the values for the keys
cuda-graph-max-bs and max-running-requests accordingly in the recipe.
- Remove enable-symm-mem from both recipes: benchmarked -13 to -18% regression on B200 at low concurrency - Low-latency: reduce cuda-graph-max-bs and max-running-requests 128->16 to match max concurrency of 8 - Low-latency: reduce stream-interval 30->10 to match latency-sensitive scheduler-recv-interval Final validated results (trtllm_mha + flashinfer_trtllm, no symm-mem): CONC 4: 116.5 tok/s/GPU, 123.0 tok/s/user CONC 8: 199.7 tok/s/GPU, 104.7 tok/s/user CONC 16: 317.4 tok/s/GPU, 83.5 tok/s/user CONC 32: 428.2 tok/s/GPU, 56.8 tok/s/user CONC 64: 718.9 tok/s/GPU, 47.3 tok/s/user
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
recipes/b200-bf16/1k1k/low-latency.yaml (1)
5-5: Pincontainerto a specific versioned tag for reproducible benchmarks.
sglang-nightlyis rebuilt daily; two runs of this recipe on different days may use different SGLang versions, making benchmark comparisons unreliable and regressions hard to bisect. Pinning the container digest or a dated tag (e.g.,sglang-nightly-2026-02-19) ensures results are tied to a known build.♻️ Proposed change
- container: "sglang-nightly" + container: "sglang-nightly-2026-02-19" # pin to a specific dated/versioned tag🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/b200-bf16/1k1k/low-latency.yaml` at line 5, The recipe currently uses a floating image tag in the container field (`container: "sglang-nightly"`), which makes benchmarks non-reproducible; update the container value to a specific, versioned tag or digest (e.g., `sglang-nightly-YYYY-MM-DD` or an immutable SHA digest) so runs are tied to a known build—edit the `container` entry in this YAML to use the chosen dated tag or content-addressable digest and commit that change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Around line 36-37: Add an inline comment next to the cuda-graph-max-bs and
max-running-requests entries explaining that they are intentionally set to 2x
the actual max concurrency (e.g., 16 vs concurrency 8) to provide headroom for
batching bursts; update the entries for cuda-graph-max-bs and
max-running-requests in low-latency.yaml to include that clarifying comment so
future maintainers understand the doubling is intentional and matches the same
pattern used in max-tpt.yaml.
---
Duplicate comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Around line 39-40: The two interval settings stream-interval and
scheduler-recv-interval should be equal; set both to 10 (as done) and add a
brief inline comment near these keys noting this is for benchmarking and that
streaming deployments can lower stream-interval to 1–2 to reduce token flush
latency; also remove the stray [duplicate_comment] marker left in the YAML
content.
---
Nitpick comments:
In `@recipes/b200-bf16/1k1k/low-latency.yaml`:
- Line 5: The recipe currently uses a floating image tag in the container field
(`container: "sglang-nightly"`), which makes benchmarks non-reproducible; update
the container value to a specific, versioned tag or digest (e.g.,
`sglang-nightly-YYYY-MM-DD` or an immutable SHA digest) so runs are tied to a
known build—edit the `container` entry in this YAML to use the chosen dated tag
or content-addressable digest and commit that change.
| cuda-graph-max-bs: 16 | ||
| max-running-requests: 16 |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare cuda-graph-max-bs, max-running-requests, and concurrencies across all 1k1k recipes
echo "=== Values in 1k1k recipes ==="
rg -n "cuda-graph-max-bs|max-running-requests|concurrencies" --type yaml recipes/b200-bf16/1k1k/Repository: ishandhanani/srt-slurm
Length of output: 513
🏁 Script executed:
#!/bin/bash
# Search for recipe-loading or validation logic
echo "=== Searching for recipe loading/parsing logic ==="
fd -e py -e go -e js | head -20 | xargs -I {} sh -c 'rg -l "cuda-graph-max-bs|max-running-requests|recipe" {} 2>/dev/null' | head -10
echo ""
echo "=== Looking for validation logic ==="
rg -i "cuda-graph-max-bs|max-running-requests" --type py --type go -B 2 -A 2 | head -50
echo ""
echo "=== Checking for comments about concurrency ratio ==="
rg -i "headroom|2x|double|batch|prefix" recipes/b200-bf16/1k1k/Repository: ishandhanani/srt-slurm
Length of output: 273
Add a clarifying comment about the 2x headroom ratio in cuda-graph-max-bs and max-running-requests.
The PR description claims these values were "reduced to match max concurrency of 8", but they're set to 16 (2x the max concurrency). This 2x ratio is consistent across both low-latency.yaml (16 vs 8) and max-tpt.yaml (128 vs 64), suggesting intentional headroom for batching bursts. Add an inline comment explaining the doubling to clarify the design intent for future maintainers.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/b200-bf16/1k1k/low-latency.yaml` around lines 36 - 37, Add an inline
comment next to the cuda-graph-max-bs and max-running-requests entries
explaining that they are intentionally set to 2x the actual max concurrency
(e.g., 16 vs concurrency 8) to provide headroom for batching bursts; update the
entries for cuda-graph-max-bs and max-running-requests in low-latency.yaml to
include that clarifying comment so future maintainers understand the doubling is
intentional and matches the same pattern used in max-tpt.yaml.
1 participant
Two recipes for the Qwen3.5-397B-A17B model on B200 GPUs (BF16, TP8):
Incorporates SGLang optimizations (chunked-prefill, flashinfer allreduce fusion, CUDA graph tuning) validated to match/exceed colleague benchmarks.
Summary by CodeRabbit