fix: MI355X (gfx950) + Ionic RDMA support for MoRI EP#1
Open
raviguptaamd wants to merge 26 commits into
Open
Conversation
…scripts Dockerfile: - Fix sed pattern to avoid corrupting https URLs (s/http/https -> s|http://|https://|) - Replace silent `|| true` on vLLM build with logged warning - Remove commented-out dead code (old base image, CMAKE_PREFIX_PATH, git checkout) - Use consistent absolute path for versions.txt, add comment documenting base image dependency - Fix trailing whitespace on multiple lines - Combine version-tracking RUN layers into single layer - Move py-spy and flask installs from runtime into Dockerfile Slurm script (run_xPyD_models.slurm): - Require explicit DOCKER_IMAGE_NAME instead of defaulting to stale image - Remove unreachable empty-string check - Remove duplicate NUM_NODES echo MoRI EP script (vllm_disagg_mori_ep.sh): - Refactor 4 near-identical vllm serve blocks into shared functions (setup_mori_env, build_kv_transfer_config, launch_vllm_worker, wait_for_proxy_and_cleanup, print_node_info), eliminating ~150 lines - Extract hardcoded ports into named variables at top of script - Fix typo "untill" -> "until" - Fix --gpu_memory_utilization to --gpu-memory-utilization for consistency - Remove commented-out compilation-config lines - Add graceful kill with `kill ... 2>/dev/null || true` Server script (vllm_disagg_server.sh): - Add comments noting DeepSeek-R1 shares architecture with V3 Made-with: Cursor
etcd is not needed for the MoRI EP disaggregation path. Remove: - etcd, etcd-server, etcd-client apt packages - etcd v3.6.0-rc.5 binary download and PATH entry - etcd-cpp-apiv3 source build Made-with: Cursor
This reverts commit 68b3935.
Remove etcd from both Dockerfile and vllm_disagg_server.sh: - Dockerfile: remove etcd/etcd-server/etcd-client apt packages, etcd binary download, and etcd-cpp-apiv3 source build - vllm_disagg_server.sh: remove ETCD Server Setup section (start_etcd, etcdctl health/status checks, barrier waits) and etcd_pid kill at exit Fix spelling typos: - "untill" -> "until" in vllm_disagg_server.sh and sglang_disagg_server.sh - "Dissagerated" -> "Disaggregated" in vllm_dissag/README.MD - "Disaggerated" -> "Disaggregated" in sglang_disagg/README.MD - "disggregation" -> "disaggregation" in README.md Note: the directory name scripts/vllm_dissag/ is also misspelled (should be vllm_disagg) but renaming it is a larger refactor best done in a separate PR. Made-with: Cursor
Made-with: Cursor
Port DeepEP server script and dispatch logic from MAD-private PR #208. Three run modes now available via RUN_MORI/RUN_DEEPEP flags with mutual exclusion. Adds host RDMA library bind-mounts, docker pull, and DeepEP env var passthrough to the slurm launcher. Updates README with all three modes, node topology, and DeepEP configuration reference. Made-with: Cursor
Made-with: Cursor
Expose BENCHMARK_ITR, BENCHMARK_CON, and BENCHMARK_COMBINATIONS as environment variables passed through slurm -> Docker -> benchmark script, matching sglang's pattern. Add a small warmup run before the main sweep to stabilize results. Made-with: Cursor
Refactors all three server scripts and the Slurm launcher to run the
proxy/router on the same node as the Prefill master (rank 0), reducing
the required node count from xP+yD+1 to xP+yD.
Key changes:
- NUM_NODES = xP + yD (was xP + yD + 1)
- SBATCH defaults: -N 2 -n 2 (was -N 3 -n 3)
- Rank 0 now runs both prefill vLLM serve and proxy/router
- ROUTER_PORT defaults to 8001 (avoids collision with SERVER_PORT=2584)
- IP array indexing shifted: prefill at [0..xP-1], decode at [xP..end]
- DP start ranks adjusted: prefill = NODE_RANK*8, decode = (NODE_RANK-xP)*8
- Log file names updated: prefill_NODE0.log, decode_NODE{xP}.log
Made-with: Cursor
…script Replace hard-fail checks with auto-detection via ibstat (matching the DeepEP script pattern) so the script works without external env vars. Made-with: Cursor
Kill stale listeners on the barrier port before starting the socket_barrier, matching the pattern already used in the DeepEP script. Prevents 'Address already in use' errors after failed runs. Made-with: Cursor
…ript The vLLM version in the Docker image does not support this flag, causing 'unrecognized arguments' errors on startup. Made-with: Cursor
…decode-url The current vllm-router API uses --prefill and --decode (without -url suffix and without /v1 path). Matches the DeepEP script's pattern. Made-with: Cursor
Port 8001 is occupied by a system service on the cluster nodes, causing 'Address already in use' errors when the co-located proxy tries to bind. Use 18001 which is available. Made-with: Cursor
Made-with: Cursor
- Update model lists in top-level README and vllm_dissag README to match VALID_MODELS in Slurm (add DeepSeek-R1, DeepSeek-V3-5layer, use correct amd- prefix) - Add DeepSeek-V3-5layer config to MODEL_*_CONFIGS and MODEL_ENVS in vllm_disagg_server.sh (was falling back to --tp-size 4) - Fix Docker build path in vllm_dissag README - Document MoRI-specific ports (20005/10001) and PROXY_TYPE scope - Add barrier port cleanup for ports 5000 and 2222 in Slurm launcher - Fix unset $prompts variable in benchmark_xPyD.sh echo - Remove duplicate HSA_ENABLE_SDMA=1 lines in vllm_disagg_server.sh Made-with: Cursor
Use dirname of the script to cd into the correct directory before setting NIXL_REPO_DIR=$(pwd). Falls back to SLURM_SUBMIT_DIR if dirname fails. Validates that all required server scripts exist, giving a clear error message if the directory is wrong. Made-with: Cursor
…k docs - Fix UCX_NET_DEVICES: auto-detected value was being overridden by hardcoded mlx5_0:1 in all vllm serve commands; now uses the auto-detected value consistently (matching DeepEP script behavior) - Fix exit 0 on MODEL_NAME error path to exit 1 - Remove dead code: ORIGINAL_MODEL_PATH, timestamp, BENCHMARK_FILE, SELECTED_NODELIST_STR, duplicate SLURM echo block, no-op sed, commented-out trap, duplicate ls NIXL_COOKBOOK_PATH - Remove README self-reference line - Add BENCHMARK_ITR/CON/COMBINATIONS documentation to README Made-with: Cursor
Slurm copies the batch script to /var/spool/slurmd/, so dirname $0 resolves to the spool path instead of the original directory. Use SLURM_SUBMIT_DIR (always set by Slurm to the sbatch invocation dir) as the primary source, falling back to dirname for non-Slurm runs. Made-with: Cursor
Adapt vLLM disaggregated inference scripts for AMD Instinct MI355X clusters with Pensando Ionic RDMA NICs: - Add DeepSeek-V3/V3-5layer to MORI_EP_VALID_MODELS - Auto-detect default-route interface for IP resolution instead of hostname -I (fixes multi-homed nodes where eno1 is unreachable) - Add Ionic RDMA library bind-mounts (libionic-rdmav34.so, libionic.so) - Pass MoRI env vars (MORI_DISABLE_TOPO, MORI_IB_GID_INDEX, MORI_RDMA_DEVICES, MORI_NUM_QP_PER_PE, MORI_DISABLE_P2P) - Add --jobid/--overlap flags for srun in interactive salloc sessions - Auto-detect GLOO_SOCKET_IFNAME and NCCL_SOCKET_IFNAME (was hardcoded to eth0) - Add /it-share volume mount for shared model storage - Skip multi-node model path check when MODEL_PATH is pre-set - Reduce gpu-memory-utilization to 0.7 for MoRI EP Made-with: Cursor
raviguptaamd
force-pushed
the
fix/pr209-review-cleanup
branch
4 times, most recently
from
b31ef10 to
be9e3fc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adapt vLLM disaggregated inference scripts for AMD Instinct MI355X (gfx950) clusters with Pensando Ionic RDMA NICs. These changes are tested on a 2-node SLURM cluster (mia1-p02) and are intended to be merged into PR ROCm#143 (
fix/pr209-review-cleanup).Changes
DeepSeek-V3andDeepSeek-V3-5layertoMORI_EP_VALID_MODELSip route+ip addr) instead ofhostname -Ito avoid unreachable management IPs on multi-homed nodeslibionic-rdmav34.so,libionic.sobind-mounts for Pensando/Ionic NICsMORI_DISABLE_TOPO,MORI_IB_GID_INDEX,MORI_RDMA_DEVICES,MORI_NUM_QP_PER_PE,MORI_DISABLE_P2Pinto Docker containers--jobid=$SLURM_JOB_ID --overlaptosruncalls forsalloccompatibilityGLOO_SOCKET_IFNAMEandNCCL_SOCKET_IFNAMEinstead of hardcodingeth0/it-sharefor shared model storageMODEL_PATHis pre-set externally--gpu-memory-utilizationfrom 0.8 to 0.7 for MoRI EPKnown blocker
MoRI EP hits a
VM_L2_PROTECTION_FAULTon MI355X during rocshmem's symmetric memory initialization (hipDeviceMallocUncached). This is a rocshmem/HIP runtime issue, not a script issue — these script changes are independently correct and needed for Ionic RDMA + MI355X support.Files changed
scripts/vllm_dissag/run_xPyD_models.slurmscripts/vllm_dissag/vllm_disagg_mori_ep.shscripts/vllm_dissag/vllm_disagg_server.shTest plan
Made with Cursor