Add Qwen3.5 FP4 B300 AgentX MTP - #2421
Conversation
Add the initial SGLang NEXTN AgentX recipe with golden synthetic acceptance and 256k traces.\n\n中文：新增 Qwen3.5 FP4 B300 AgentX MTP 初始配置，使用 SGLang NEXTN、黄金合成接受长度和 256k 轨迹数据集。
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
Append the Qwen3.5 FP4 B300 AgentX MTP benchmark entry.\n\n中文：追加 Qwen3.5 FP4 B300 AgentX MTP 基准测试触发记录。
cquil11
changed the title
Add strict two-pool HiCache accounting, cache reporting, session-sticky DEP8 routing, AgentX request/graph limits, and measured-cache guidance for the B300 frontier.\n\n中文：为 B300 前沿调优加入严格的双池 HiCache 容量核算、缓存统计、会话粘性的 DEP8 路由、AgentX 请求与 CUDA Graph 上限，并记录实测缓存容量依据。
Add isolated TP2 c72 and TP4 c80 DRAM HiCache points so each cache-cliff recovery topology can be measured independently. 中文：添加独立的 TP2 c72 与 TP4 c80 DRAM HiCache 测试点，以便分别测量各拓扑在缓存工作集拐点后的恢复效果。
Account for SGLang target KV, Mamba, and native NEXTN draft host pools as H * 31/15 per rank, with page-alignment reserve, so projected use stays within runner-generated DRAM. 中文：按每个 rank 的 H * 31/15 计入 SGLang 目标 KV、Mamba 与原生 NEXTN 草稿主机缓存，并预留页对齐空间，确保预计用量不超过 runner 生成的 DRAM 上限。
中文：将最新 main 合并到 B300 提交分支，并将本 PR 的性能变更记录重新追加到文件末尾。
Add TP4 concurrency points around the measured HiCache crossover and extend the DRAM-offload curve beyond it.\n\n中文：在实测 HiCache 切换区间细化 B300 Qwen TP4 并发点，并将 DRAM 卸载曲线扩展到拐点之后。
Remove the measured no-offload thrashing tail and add dense TP2 points across the expected HiCache crossover.\n\n中文：移除实测发生抖动的无卸载尾部配置，并在预期 HiCache 切换区间细化 B300 TP2 并发点。
Drop the attention-DP arm after live AgentX profiling showed fragmented prefix caches and throughput below the TP frontier. Remove the now-unused router path.\n\n中文：AgentX 实测表明注意力 DP 会导致前缀缓存碎片化且吞吐低于 TP 前沿，因此移除该配置及不再使用的路由路径。
Probe three post-c72 offload points to locate saturation of the measured TP2 efficiency frontier.\n\n中文：在 c72 之后增加三个卸载并发点，用于定位实测 B300 TP2 效率前沿的饱和位置。
Remove the no-offload c96 point after it was dominated by HiCache c80 in both throughput and latency, and refresh the measured crossover note.\n\n中文：无卸载 c96 在吞吐与延迟上均被 HiCache c80 支配，因此移除该点并更新实测切换区间说明。
Remove offload points that exceed the measured host-cache validity boundary and densify the final safe interval below c80.\n\n中文：移除超过实测主机缓存有效边界的卸载配置，并在 c80 以下的安全区间细化最终并发点。
Remove no-offload c64 after it stalled under the long-context working set; retain the matched HiCache points beyond c56.\n\n中文：无卸载 c64 在长上下文工作集下停滞，因此移除该点，并保留 c56 之后对应的 HiCache 配置。
中文：在缓存工作集拐点测试 B300 TP4 HiCache。
中文：裁剪 B300 TP2 无卸载配置的缓存拐点尾部。
中文：优化 B300 缓存悬崖附近的并发点，并使用 SGLang 默认单 tokenizer 启动路径以避免 HiCache 初始化期间的多 tokenizer 共享内存竞态。
中文：在 B300 HiCache 缓存悬崖附近增加 c84、c92 和 c94 测试点，以更细粒度覆盖有效的帕累托前沿。
中文：在 B300 TP2 的 c32 至 c56 区间增加 HiCache 重叠测试点，以覆盖一小时 AgentX 预热下可能提前出现的缓存悬崖。
Use six tokenizer workers for TP4 long-context replay while retaining SGLang default single-worker startup for TP2 HiCache. 中文：TP4 长上下文回放使用 6 个 tokenizer worker；TP2 HiCache 保持 SGLang 默认单 worker 启动路径，避免共享内存初始化竞态。
Add HiCache concurrency 34 and 38 around the measured c36-to-c40 transition. 中文：在实测 c36 到 c40 的转折区间补充 HiCache 并发点 34 和 38。
Densify the TP4 no-offload cliff and remove TP4 HiCache points that are dominated by the TP4 no-offload and TP2 HiCache arms. Keep TP2 HiCache dense through the measured host-KV saturation region.\n\n中文：细化 B300 缓存帕累托前沿；加密 TP4 无卸载缓存临界区间，移除被 TP4 无卸载与 TP2 HiCache 方案支配的 TP4 HiCache 测点，并保留覆盖主机 KV 饱和区间的密集 TP2 HiCache 测点。
|
Claude encountered an error after 11s —— View job I'll analyze this and get back to you. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — approving despite the nit; it's a metadata-only bilingual-title requirement, not a functional issue.
Extended reasoning...
Overview
This PR adds a new benchmark recipe (qwen3.5-fp4-b300-sglang-agentic-mtp): a new shell script under benchmarks/single_node/agentic/, a corresponding config entry in configs/nvidia-master.yaml, and a perf-changelog.yaml entry. It follows the same structure as the neighboring qwen3.5-fp8-b300-sglang-agentic-hicache entry and other spec-decoding: mtp recipes already in the file, and does not touch any shared library code (benchmark_lib.sh), CI workflow, or routing logic.
Security risks
None. This is a benchmark recipe for launching an SGLang server with fixed CLI flags and env vars; there is no user-controlled input, no auth/crypto/permissions code, and no new shared code paths that other recipes depend on.
Level of scrutiny
Low-to-moderate is appropriate here. New recipe additions are additive and isolated: a bug in the HiCache sizing math or a bad conc-list bound would fail loudly in the benchmark sweep (the script has explicit error checks for capacity/alignment overflow) rather than silently corrupting other recipes. The one issue surfaced (missing bilingual title/description per AGENTS.md) is a checked-in process requirement, not a functional defect, so it's flagged as a nit rather than blocking.
Other factors
The HiCache byte-budget arithmetic in the script is unusually well-commented, explaining the 15/31 draft-pool ratio and alignment reserve, and the config's inline comments document the empirical concurrency-sweep rationale (why certain conc values were excluded, e.g. TP4 HiCache dominated by no-offload). This matches the level of rigor seen in adjacent, previously-merged recipes in the same file. No prior reviews exist on this PR from this bot.
| @@ -0,0 +1,173 @@ | |||
| #!/usr/bin/env bash | |||
There was a problem hiding this comment.
🟡 The PR title and description are English-only, but AGENTS.md line 7 mandates bilingual PR titles (<English title> / <中文标题>) and a Chinese translation section (e.g. ## 中文说明) in the body for every PR. / PR 标题和描述仅为英文，但 AGENTS.md 第 7 行要求每个 PR 的标题必须为双语格式（<English title> / <中文标题>），且正文需包含中文翻译部分（如 ## 中文说明）。请在标题追加中文标题，并在 Summary 后补充对应的中文说明段落。
Extended reasoning...
AGENTS.md explicitly and prominently states (line 7, bolded): "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...). This applies to every PR and every issue."
PR #2421's title is Add Qwen3.5 FP4 B300 AgentX MTP — English only, with no / <中文标题> suffix. The description body contains only an English ## Summary section:
Add Qwen3.5-397B-A17B NVFP4 AgentX coverage on B300 with SGLang native NEXTN MTP, golden synthetic acceptance, prefix caching, and the 256k trace dataset.
There is no ## 中文说明 section or any Chinese translation anywhere in the title or body, which is a direct, checked-in convention violation rather than a matter of subjective style.
Step-by-step verification:
- Open
AGENTS.mdat the repo root and read line 7 — it defines the exact required title format (<English title> / <中文标题>) and body structure (English content followed by a## 中文说明section). - Compare against PR Add Qwen3.5 FP4 B300 AgentX MTP #2421's actual title:
Add Qwen3.5 FP4 B300 AgentX MTP. There is no/separator and no Chinese characters anywhere in the title. - Compare against the PR body: it contains exactly one section,
## Summary, entirely in English. No## 中文说明(or any Chinese-language section) is present. - Conclusion: both the title-format rule and the body-translation rule from AGENTS.md line 7 are violated by this PR as submitted.
This convention exists so that the bilingual engineering audience described in the Code Conventions section (and referenced tooling like the CODEOWNER sign-off bot) can review PRs and issues without an English-only barrier; it is a checked-in, project-wide requirement rather than a stylistic nicety, so per review guidance it should be surfaced even though it does not affect any runtime or benchmark behavior in this recipe addition.
Impact: No functional or benchmark risk — the .sh script, configs/nvidia-master.yaml entry, and perf-changelog.yaml entry are all unaffected. This is purely a PR-metadata/process gap.
Fix: Update the PR title to Add Qwen3.5 FP4 B300 AgentX MTP / 新增 Qwen3.5 FP4 B300 AgentX MTP 基准测试, and append a ## 中文说明 section to the description body that mirrors the English ## Summary content in idiomatic technical Chinese (per the translation-quality bar and terminology table also given in AGENTS.md), keeping model names, hardware SKUs, framework names, and CLI/env-var identifiers in English.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30529799842 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30529799842 |
1 participant
Summary
Add Qwen3.5-397B-A17B NVFP4 AgentX coverage on B300 with SGLang native NEXTN MTP, golden synthetic acceptance, prefix caching, and the 256k trace dataset.