[AMD][AgentX] DSv4 MI355X agentic disagg: add high-concurrency dots using DEP8 - #2308
[AMD][AgentX] DSv4 MI355X agentic disagg: add high-concurrency dots using DEP8#2308ichbinblau wants to merge 22 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
ichbinblau
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
ichbinblau
added
AMD
agentx
|
hi @ichbinblau any chance u can priority doing disagg mtp for agentic |
Hi @functionstackx, this PR fixes the SGL router settings issues for high-concurrency use cases. I'll open another PR for the MTP changes. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29989948268 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29990310046 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29992400324 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30255202619 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30255202619 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30255202619 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30255202619 |
ichbinblau
added
the
evals-only
Ensure ~/.modal.toml is written for agentic runs even when SWEBENCH_USE_MODAL is unset, and forward SCENARIO_TYPE/EVAL_LIMIT into the Slurm container alongside the Modal/HF tokens. Co-authored-by: Cursor <cursoragent@cursor.com>
Route all RUN_EVAL paths through lm-eval instead of SWE-bench, export CONC and disagg topology fields before run_eval, and avoid duplicate append_lm_eval_summary for agentic eval-only. Co-authored-by: Cursor <cursoragent@cursor.com>
Exclude leaky g10/g16 nodes, improve docker pre-clean and gpu_sanity drain, bridge disagg EP/DPA into lm-eval meta_env.json, and filter eval-only rows by eval-min-prefill-ep so DEP8 c128 jobs dispatch with correct metadata. Co-authored-by: Cursor <cursoragent@cursor.com>
Append a second evals-only changelog row filtered with eval-max-prefill-ep: 1 so TP8/EP1 low-conc (eval-conc 32) runs separately from the existing DEP8/DPA c128 eval, matching one-eval-per-topology policy without mutating prior entries. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit b9d85e5.
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
adc0ec2 to
4aa0713
Compare
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30417609648 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
8 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
|
@ichbinblau the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30419156341 |
Double gpu_sanity VRAM drain retries so docker teardown has more time to release memory before pre-flight fails. Drop g14-g15 from the default SLURM exclude list. Pin utils/aiperf to be758d62. Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30523296250 |
aiperf>=0.12.0 requires Python >=3.11,<3.14 (dropped 3.10 support in
SemiAnalysisAI/aiperf#1107). The sglang-rocm/vllm-rocm container images
ship Python 3.10.12 as the default python3, which caused `uv pip install`
to fail silently during venv creation, leaving the aiperf/hf CLIs
uninstalled and causing agentic trace replay to fail with exit 127.
Changed `uv venv --python "$(command -v python3)"` to
`uv venv --python "${AIPERF_PYTHON_VERSION:-3.11}"` to allow uv's
auto-download of a standalone Python 3.11 interpreter when the system
lacks one. This makes agentic benchmarks work on any container image
that has uv available, regardless of bundled Python version.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30523602354 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30526922977 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30526922977 |
3 participants
Summary
Follow-on fixes for the DSv4 FP4 MI355X SGLang PD-disaggregated agentic benchmark (
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache), rebased on top of currentmain(which already absorbed the bulk of this branch's earlier history via #2170). Two logical changes:What's changed
HiCache sizing (ratio vs size)
dsv4_fp4_mi355x_sglang-disagg.sh: setHICACHE_RATIO=4andFORCE_HICACHE_RATIO=1so this recipe explicitly opts into ratio-based sizing.job.slurm: forwardFORCE_HICACHE_RATIOinto the container's HiCache env file (hicache_mc_<job>.env) alongside the otherHICACHE_*vars — it was previously silently dropped, so the override never reached the server.NCCL / decode cuda-graph crash workaround
models.yaml: disable prefill cuda-graph (disable_cuda_graph: true) forDeepSeek-V4-Pro-AgentXas a complementary mitigation, drop--enable-two-batch-overlapfromdp_flags, lower prefillmem_fraction_static0.8→0.72 and no-dpmax_running_requests128→64.lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710→v0.5.15.post1-rocm720-mi35x-20260719.RDMA QoS/DCQCN + GPU-sanity pre-flight gates (
job.slurm)helpers/rdma_check.sh: validates NIC QoS (PFC/DSCP) and DCQCN configuration on every allocated node viasrunbefore any container/GPU time is spent. A misconfigured NIC otherwise only shows up much later as unexplained tail latency or throughput variance under MoRI cross-node KV transfer. Override withSKIP_RDMA_CHECK=1.helpers/gpu_sanity.sh: after container pre-clean/teardown, fails fast (naming the offending process) if a bare, non-containerized process is still holding GPU VRAM — instead of surfacing as an opaque OOM during model load ~30 minutes later.Router policy fix + DPA (ep=8/dp-attn) hicache arm
server_sglang.sh(agentic branch only — DI branch untouched): fixes a bug where the script readROUTER_PREFILL_POLICYbut recipe scripts exportPREFILL_ROUTER_POLICY, so the recipe's policy override was silently ignored and the router always fell back to the hardcoded default. Also switches the default prefill router policy fromcache_aware→consistent_hashing, adds--dp-awareto the router flags, and comments out the separate--decode-policyoverride (superseded by--dp-aware).configs/amd-master.yaml: bumpdsv4-fp4-mi355x-sglang-disagg-agentic-hicacheimage tov0.5.15.post1-rocm720-mi35x-20260719; re-enable (previously commented out) the TP8/EP8/DPA (ep=8, dp-attn=true) hicache search-space arm atconc-list: [64, 96, 128].trace_replay.sh: whenPREFILL_ENABLE_DP=true, exportAIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=trueso aiperf's HTTP routing key is derived from the correlation ID under DP-attention.Test plan
bash -non all touched shell scripts (env.sh,job.slurm,server_sglang.sh,setup_deps.sh,helpers/gpu_sanity.sh,helpers/rdma_check.sh) — syntax OKconfigs/amd-master.yaml,benchmarks/multi_node/amd_utils/models.yaml— YAML parses cleanlyorigin/mainto confirm no unintended regression in theIS_AGENTIC/DI router split or the router-readiness canary (wait_or_die) added in a prior PRe2e Test - dsv4-fp4-mi355x-sglang-disagg-agentic-hicache(multi-node, mi355x-amds) — new TP8/EP8/DPA arm at conc 64/96/128