[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning - #2254
[AMD][MI355X] Add DSv4 FP4 agentic MTP recipe with DPA tuning#2254seungrokj wants to merge 50 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
added
AMD
full-sweep-fail-fast
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| dsv4-fp4-mi355x-sglang-agentic-hicache: | ||
| image: lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 | ||
| model: deepseek-ai/DeepSeek-V4-Pro | ||
| model-prefix: dsv4 | ||
| runner: cluster:mi355x-amds | ||
| precision: fp4 | ||
| framework: sglang | ||
| multinode: false | ||
| scenarios: | ||
| agentic-coding: | ||
| - dram-utilization: 0.80 | ||
| search-space: | ||
| - { tp: 8, kv-offloading: none, conc-list: [1, 2, 4, 8] } | ||
| - { tp: 8, dp-attn: true, kv-offloading: none, conc-list: [16, 32, 48, 64] } | ||
| - { tp: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [64] } |
There was a problem hiding this comment.
🟡 This PR accidentally leaves a second, byte-for-byte identical dsv4-fp4-mi355x-sglang-agentic-hicache: block in configs/amd-master.yaml (lines 2468-2482), duplicating the pre-existing entry directly above it. It's a copy-paste artifact from using the hicache block as a template for the new dsv4-fp4-mi355x-sglang-agentic-mtp entry — please delete the duplicate block.
Extended reasoning...
The diff adds a second top-level dsv4-fp4-mi355x-sglang-agentic-hicache: mapping key immediately below the pre-existing one, right before the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry that this PR is actually meant to introduce. In the resulting file this shows up as two adjacent, byte-for-byte identical blocks: the pre-existing entry at lines 2452-2466, followed by the newly-added duplicate at lines 2468-2482.
Code path: configs/amd-master.yaml is loaded via yaml.safe_load in load_config_files() (utils/matrix_logic/validation.py:919). PyYAML's default loader does not error on duplicate mapping keys within a single document — it silently keeps only the last occurrence. Since both dsv4-fp4-mi355x-sglang-agentic-hicache blocks are identical, the "last one wins" collapse produces the exact same effective config as if the duplicate had never been added, so there is no behavioral difference today.
Why the repo's own duplicate-key guard doesn't catch it: load_config_files() does have a duplicate-key check at validation.py:930-936, but it only detects a key being duplicated across different config files (it intersects config_data.keys() from the current file against all_config_data.keys() accumulated from previously-loaded files). By the time that check runs, yaml.safe_load has already silently collapsed the intra-file duplicate down to one key, so the duplication within amd-master.yaml itself is invisible to it. There's also no .yamllint config in the repo that would otherwise flag duplicate keys at CI time.
Impact: Purely cosmetic/dead config today, since the two blocks are identical and the parser deterministically resolves to one value. However, this is exactly the kind of artifact that causes real confusion later: if a future PR edits one of the two blocks (e.g. bumping the image tag or tweaking the search space) without noticing the sibling duplicate, the edit could silently have no effect (if the untouched copy happens to load last) or silently take effect while looking like a no-op in a diff review. It's clearly unintentional — the PR description says it's adding only the new -mtp entry, and the mechanism is obviously that the hicache block was copied as a scaffold for the new entry and the copy of the original block was never deleted.
Proof / walkthrough:
- Before this PR:
dsv4-fp4-mi355x-sglang-agentic-hicache:appears once, at (pre-PR) line ~2465. - This PR's diff inserts 32 new lines directly after it, starting with
dsv4-fp4-mi355x-sglang-agentic-hicache:again (identical image, model, scenarios, search-space), followed by the newdsv4-fp4-mi355x-sglang-agentic-mtp:block. - Post-PR,
grep -n '^dsv4-fp4-mi355x-sglang-agentic-hicache:' configs/amd-master.yamlreturns two hits (lines 2452 and 2468 in the final file). - Running the file through
yaml.safe_loadcollapses these to a single key with the (identical) value of the second occurrence — no functional change today, but a landmine for future edits and a source of reviewer confusion.
Fix: Delete the duplicated dsv4-fp4-mi355x-sglang-agentic-hicache: block introduced by this PR (lines 2468-2482), leaving the original entry and the new dsv4-fp4-mi355x-sglang-agentic-mtp: entry intact.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29548395719 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29548395719 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29555826772 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29562088440 |
d895957 to
4ddd584
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29635054984 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29635071153 |
5324d24 to
3040b2a
Compare
seungrokj
added
evals-only
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29678199226 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29678318533 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29679892938 |
seungrokj
added
full-sweep-enabled
and removed
full-sweep-fail-fast
evals-only
…to throughput runs Only export SGLANG_SIMULATE_ACC_* when EVAL_ONLY=false. Simulated acceptance bypasses real target verification, so it must be off for accuracy (EVAL_ONLY) runs or it corrupts the eval score. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iAnalysisAI/InferenceX into amd/agentx_dsv4_sgl_mtp_0717
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30372581813 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30372778418 |
seungrokj
removed
the
evals-only
…c 1,2 arm Add a pure-TP (dp-attn false) low-concurrency arm (conc 1,2, spec-decoding mtp) alongside the existing DEP conc 4-32 arm. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30408908888 |
…nc 1,2 arm Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30409338435 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30412031207 |
…_static to 0.85 and chunked_prefill_size to 4096 to avoid runtime OOM Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30502761391 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30502792935 |
… [48,64] and add hicache dram arm Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@seungrokj the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
Pin the shared AIPerf submodule to the synchronized main revision with additive per-lane warmup after mandatory primers while retaining timing-faithful spread phase starts. Align the workflow documentation with the request-count fast preset. 中文：将共享 AIPerf 子模块固定到已同步的 main 版本，在必需预热请求之后执行额外的逐 lane 预热，并保留符合轨迹时序的分散阶段启动。同步更新工作流文档，使其准确描述基于请求数量的快速预设。
* fix(kimik3): keep only TRTLLM_RAGGED in the DSpark recipe, revert the rest
Reduce the flag alignment to its one surviving piece and rebuild on main.
The full upstream Blackwell/NVIDIA block killed the engine core on every
completed concurrency -- c1/c2/c4/c8 across both KV arms in run 30378755933 --
roughly 13-16 min in, during agentic warmup:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None
AssertionError (all 8 TP ranks simultaneously)
K3 declares num_shared_experts 2 and this image resolves its MoE backend to
FLASHINFER_TRTLLM_MXFP4_MXFP8 / TrtLlmMxfp4ExpertsMonolithic, so that reads as an
image-side invariant violation rather than a bad configuration value. The same
crash hit the non-DSpark recipe, so it is not spec-decoding specific.
Retained: --attention-config mla_prefill_backend=TRTLLM_RAGGED
Reverted: gpu-memory-utilization 0.95 -> 0.90
max-num-seqs 32 -> 2*CONC
max-model-len 1000000 -> 1048576
max-num-batched-tokens 32768 -> dropped (8192 default)
--no-enable-flashinfer-autotune -> dropped
VLLM_USE_V2_MODEL_RUNNER=1 -> dropped (V2 is on by default anyway)
cudagraph capture consequently enumerates 2*CONC token-multiples of (1+7)=8
again, instead of a fixed 32-entry ladder to 256.
Rebased onto main rather than amending, since main had advanced four commits and
the changelog diff had started reading main's tail as deletions.
* fix(kimik3): enable VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION on the DSpark recipe
The upstream recipe sets this on BOTH its blackwell and nvidia paths
(recipes.vllm.ai/moonshotai/Kimi-K3), and this repo has never set it. It
defaults to 0.
It matters because of where it lands: vllm/models/kimi_k3/nvidia/model.py:549
threads it into LatentMoERunner as
runner_args={"enable_k3_latent_moe_tail_fusion": ...}, and that is the same
runner whose shared-experts output buffer asserted when the wider flag alignment
was tried:
vllm/model_executor/layers/fused_moe/runner/shared_experts.py:165
assert self._output[self._output_idx] is None (all 8 TP ranks)
So with it unset we have been running a MoE tail path upstream never exercises,
on every K3 run here including the green ones. That is a better explanation for
the crash than any of the six flags reverted earlier.
Enabled on its own so its effect is isolated. If this holds, the remaining
upstream deltas can be re-added in order: gpu-memory-utilization 0.95,
max-num-seqs 32 (spec arm), --no-enable-flashinfer-autotune,
VLLM_USE_V2_MODEL_RUNNER=1.
Not re-adding --max-num-batched-tokens 32768: re-reading the recipe YAML shows
that value appears nowhere for blackwell (AMD is 4096, PD prefill 16384, PD
decode 32), so it was introduced in error. --max-model-len is likewise 1048576
upstream, not 1000000, and already correct here.
* feat(kimik3): DSpark at level 2 with synthetic acceptance pinned to golden AL
Adopt the AgentX-prescribed measurement methodology for the DSpark arm, per
golden_al_distribution/README.md: "a submission may choose any supported draft
length, but it may not substitute a different acceptance target", because
"InferenceX is evaluating inference-system performance, not the ability to
fine-tune a benchmark-specific speculative head". The README also states up front
that AL is workload-dependent, so pinning the committed acceptance is what makes
submissions comparable -- running the acceptance this corpus happens to produce
would be the non-conforming choice.
throughput: rejection_sample_method synthetic, synthetic_acceptance_length 2.51
(committed golden AL at K=2, probabilistic curve)
EVAL_ONLY: rejection_sample_method block (real verification)
The eval split matters: synthetic acceptance commits drafts regardless of target
logits, so text is wrong and SWE-bench scores 0.0000. This follows
dsv4_fp4_b300_vllm_mtp.sh; kimik2.5_fp4_b300_mtp.sh omits the split.
Level 2, not 7. Draft length IS ours to choose, and verify width is a real system
cost synthetic acceptance does not remove -- at K=7 the target verifies 8
positions per step, which loses to no-speculation once the GPU saturates even at
the forced golden AL of 3.84. With cost ~ 1 + m(K-1) + f: K=2 clears break-even
across the m range (~1.7x low conc, ~1.2x saturated), K=3 is a wash at saturation
(~0.95x), K=7 is negative (~0.54x). cudagraph capture therefore enumerates
multiples of (1+2)=3.
Recorded in the script and config headers, because the throughput numbers will
otherwise be misread: the draft's REAL acceptance on this corpus is 1.16-2.01
with 13-41% position-1 acceptance. Its native window is 32k (rope yarn factor 32
over original_max_position_embeddings 32768, stretched to the declared 1M), and
the identical image, draft and spec config reach AL 4.18 / 0.826 position-1 on
short-context work. The synthetic numbers measure the system at a prescribed
acceptance; they are not evidence the draft predicts well at 100k+ context.
* chore: prepare PR 2380 ingest recovery
* fix: recover PR 2380 ingest
…o v0.5.16-20260729 and set kvnone conc-list to [72] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nto amd/agentx_dsv4_sgl_mtp_0717 # Conflicts: # perf-changelog.yaml
… to v0.5.15.post1 and set kvnone conc-list to [64] Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iAnalysisAI/InferenceX into amd/agentx_dsv4_sgl_mtp_0717
4 participants
Summary
dsv4-fp4-mi355x-sglang-agentic-mtpconfig (separate from hicache)dsv4_fp4_mi355x_sglang_mtp.shwith EAGLE spec decoding (3 steps, topk 1, 4 draft tokens)--enable-prefill-delayer,GPU_MAX_HW_QUEUES=5, chunked prefill8192*TPMAX_RUNNING_REQUESTS = 2*CONC,CUDA_GRAPH_MAX_BS = CONC(cap 128)MEM_FRACTION_STATICreduced by 0.10 when spec decoding is activelmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260714Test plan
🤖 Generated with Claude Code