[fix] add ep ipc fix, deepep works without nvl_bytes=0

.gitignore

@@ -362,4 +362,7 @@ marimo/_lsp/
 __marimo__/
 
 # Streamlit
-.streamlit/secrets.toml
+.streamlit/secrets.toml
+tests/deepep_matrix_work/
+deepep_fabric_archive/
+hook_archive/

docs/vllm/direct-edits.md

@@ -55,6 +55,8 @@ class CompilationConfig:
 
 This is the earliest activation point in the parent process. As soon as a `CompilationConfig` is fully constructed, foundry's runtime patches are in place.
 
+> Note: vLLM folds `graph_extension_config_path` into `CompilationConfig.compute_hash()` (the torch.compile cache key) even though it never affects codegen. This is harmless **as long as every SAVE/LOAD phase passes the same path string** — run all phases from one canonical CWD (or always an absolute path). If pass 1 and pass 2 see the path via different mount aliases (`/home/...` vs `/data/...`), pass 2 misses pass 1's warm cache and inductor recompiles inside the cuda-graph capture window → `cudaErrorStreamCaptureInvalidated`. We keep vLLM unpatched and rely on consistent invocation instead; the EP recipe scripts are launched from the workspace root. (An optional `ignored_factors` exclusion would harden this at the source but is not applied.)
+
 ## 3. `vllm/v1/engine/core.py` (~6 lines)
 
 ```python

python/foundry/__init__.py

@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright contributors to the Foundry project
+# The ops wildcard must come FIRST: .graph's CUDAGraph (and the
+# allocation_region helpers) intentionally override the raw pybind names.
 from .allocation_region import (
     allocation_region,
     free_preallocated_region,

python/foundry/integration/vllm/hooks.py

@@ -695,12 +695,22 @@ def _patch_deepep() -> None:
     def patched(self, *args, **kwargs):
         out = orig(self, *args, **kwargs)
         if isinstance(out, dict) and get_graph_extension_mode() != CUDAGraphExtensionMode.NONE:
-            # Force VMM-fabric buffers
-            # so the foundry cuMem hook tracks them, and disable the
-            # NVLink buffer (LL-mode uses RDMA) to avoid fabric-cuMemCreate
-            # collisions with preallocated region.
-            out["use_fabric"] = True
-            out["num_nvl_bytes"] = 0
+            if os.environ.get("FOUNDRY_DEEPEP_NVL_IPC", "0") == "1":
+                # Keep upstream's NVLink buffer (legacy cudaIpc sharing). The
+                # hook's VMM-IPC layer transports the fds via SCM_RIGHTS and
+                # maps peers at relocated VAs, which DeepEP absorbs through
+                # its buffer_ptrs_gpu device table - peer VAs are never baked
+                # into captured graphs. use_fabric stays off: fabric
+                # cuMemCreate needs IMEX, and exercising the IPC path is the
+                # point of this mode.
+                out["use_fabric"] = False
+            else:
+                # Default: force VMM-fabric buffers
+                # so the foundry cuMem hook tracks them, and disable the
+                # NVLink buffer (LL-mode uses RDMA) to avoid fabric-cuMemCreate
+                # collisions with preallocated region.
+                out["use_fabric"] = True
+                out["num_nvl_bytes"] = 0
         return out
 
     cls._make_all2all_kwargs = patched

recipe/experimental/README.md

@@ -0,0 +1,129 @@
+# Foundry recipe — experimental (DeepEP legacy CUDA-IPC NVL buffer)
+
+End-to-end SAVE / LOAD recipe for **Qwen3-30B-A3B** expert-parallel where DeepEP's
+intranode NVLink buffer stays on the **legacy CUDA-IPC** path
+(`cudaIpcGetMemHandle` / `cudaIpcOpenMemHandle`) under foundry, instead of the
+default fabric/NVSHMEM-only path used by `recipe/vllm/serve_qwen3-30ba3b_ep.sh`.
+
+This exercises foundry's **VMM-IPC translation layer**: DeepEP's NVL buffer is a
+foundry VMM (`cuMemCreate`) allocation that legacy IPC can't share, so the hook
+exports it as a POSIX fd and transports that fd to the peer rank via
+`SCM_RIGHTS` over a per-process unix socket, then maps it into the importer's
+own VA range. This is the path that lets foundry SAVE/LOAD work on machines
+**without fabric / IMEX**, where DeepEP would otherwise fail at `Buffer.sync`
+with `cuMemImportFromShareableHandle ... error 999`.
+
+```
+recipe/experimental/
+├── README.md                          # this file
+├── foundry_save.toml                  # SAVE config (workspace_root = "foundry_archive_ipc")
+├── foundry_load.toml                  # LOAD config (same workspace_root)
+└── serve_qwen3-30ba3b_ipc_ep.sh       # Qwen3-30B-A3B (MoE)  EP, DeepEP NVL/IPC
+```
+
+It is the standard `recipe/vllm` EP recipe plus one switch — `FOUNDRY_DEEPEP_NVL_IPC=1`
+— so read [`../vllm/README.md`](../vllm/README.md) first for installation, the
+two-pass SAVE workflow, the archive layout, and the shared EP flags. Only the
+IPC-specific deltas are documented here.
+
+## Required code
+
+The IPC path needs only **Foundry** changes beyond the standard install — no
+vLLM edit:
+
+- **Foundry** — `foundry/csrc/hook.cpp`: the SCM_RIGHTS VMM-IPC fd transport +
+  whole-chunk peer mapping (handles LOAD-mode buffers carved from the
+  preallocated chunk). **C++ change → rebuild required:**
+  `uv pip install -e . --no-build-isolation` in `foundry/`.
+- **Foundry** — `foundry/python/foundry/integration/vllm/hooks.py`: the
+  `FOUNDRY_DEEPEP_NVL_IPC` env knob in `_patch_deepep` (Python, no rebuild).
+
+### Run all phases from one consistent path (no vLLM edit needed)
+
+vLLM folds the foundry TOML path (`graph_extension_config_path`) into its
+torch.compile cache key even though the path never affects codegen. If SAVE
+pass 1 and pass 2 see that path with *different spellings* — e.g. two mount
+aliases of the same dir (`/data/...` vs `/home/...`), or you move the TOML
+between passes — pass 2 misses pass 1's warm compile cache and inductor
+recompiles **inside** the cuda-graph capture window, where its combo-kernel
+benchmark does an illegal `torch.randn` → `cudaErrorStreamCaptureInvalidated`.
+
+The fix is operational, not code: **invoke the script the same way for every
+phase** (same shell / same `cd`, or always an absolute path), so SAVE pass 1,
+SAVE pass 2, and LOAD all pass the identical path string → identical cache hash
+→ pass 2 reuses pass 1's compiled kernels. Run from the workspace root, e.g.:
+
+```bash
+cd <workspace>          # one canonical path for all three phases
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+bash foundry/recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh 2 --load
+```
+
+## Workflow
+
+Same two-pass SAVE → LOAD as the base recipe; `<ep_size>` is the first arg:
+
+```bash
+# 0. Fresh start (distinct workspace from the base recipe)
+rm -rf foundry_archive_ipc
+
+# 1. SAVE pass 1 — memory profile + capture
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+# wait for "Application startup complete", then Ctrl-C
+
+# 2. SAVE pass 2 — deterministic re-capture
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --save
+# wait for "Application startup complete", then Ctrl-C
+
+# 3. LOAD — preallocate, re-import IPC buffers, replay graphs
+bash serve_qwen3-30ba3b_ipc_ep.sh 2 --load
+# leave running
+
+# 4. Query (separate shell)
+bash ../../../experimental/query.sh 12000 Qwen/Qwen3-30B-A3B
+```
+
+Uncomment `nvshmem_host_path` in both TOMLs first (EP still needs NVSHMEM for the
+DeepEP RDMA buffer; the IPC path only changes the NVL buffer).
+
+## What `FOUNDRY_DEEPEP_NVL_IPC=1` does
+
+`_patch_deepep` (`foundry/python/foundry/integration/vllm/hooks.py`) normally
+forces `use_fabric=True, num_nvl_bytes=0` so the only cross-GPU buffer is the
+NVSHMEM symmetric heap. With `FOUNDRY_DEEPEP_NVL_IPC=1` it instead keeps
+upstream's nonzero `num_nvl_bytes` with `use_fabric=False`, so the DeepEP Buffer
+allocates the NVLink buffer via `cudaMalloc` + `cudaIpcGetMemHandle` on **both**
+SAVE and LOAD. The hook then:
+
+- **SAVE**: exports each VMM-backed NVL buffer as a POSIX fd, served to peers
+  over `\0foundry-vmm-ipc.<pid>` via `SCM_RIGHTS` (same-uid `SO_PEERCRED` check,
+  per-process token vs PID reuse).
+- **LOAD**: buffers are carved from the preallocated chunk (no individual
+  handle), so the hook exports the **whole chunk** fd + offset; the peer maps the
+  entire chunk once and returns an interior pointer.
+- Peer mappings land at a **relocated** VA (logged `[HOOK] INFO: VMM-IPC import
+  relocated`) — correct, because DeepEP resolves peers through its device-side
+  `buffer_ptrs_gpu` table (refreshed by `Buffer.sync`), never through addresses
+  baked into captured graphs.
+
+A healthy run logs two `VMM-IPC import relocated` lines per phase (one per peer)
+and **no** `error 999`.
+
+## Validation status
+
+SAVE → SAVE → LOAD → query verified on Qwen3-30B-A3B EP=2 (2× H200, no IMEX):
+per-rank `final_alloc_offset` identical across passes, LOAD replays at the saved
+offset, query returns coherent completions. LOAD reaches a serving server in
+~27 s; the IPC import itself is sub-second (inside the ~7.5 s `load_model`) and
+has no steady-state serving cost — peer addresses are resolved once at init via
+the device pointer table, not per token.
+
+## IPC-specific troubleshooting
+
+| Symptom | Likely cause |
+|---|---|
+| `[HOOK] ERROR: cuMemImportFromShareableHandle failed with error 999` at `deep_ep.cpp` `Buffer::sync` | Foundry hook not rebuilt with the SCM_RIGHTS transport — it's still packing a raw fd. Rebuild `foundry/` (`uv pip install -e . --no-build-isolation`). |
+| `operation failed due to a previous error during capture` on SAVE **pass 2** | vLLM compile-cache over-keying not applied (`graph_extension_config_path` still hashed) → recompile-in-capture. Apply the `compilation.py` `ignored_factors` edit, or set `FOUNDRY_DISABLE_COMBO_KERNELS=1` (opt-in belt-and-suspenders, wired in the experimental serve script under `experimental/expert-parallel/`). |
+| LOAD `illegal memory access` at replay with an NVL buffer present | Relocated peer import collided with the NVSHMEM heap hint — ensure the hook build includes the dedicated import-VA zone (`0x300000000000`). |
+| `error 999` only with `--deepep-mode auto`/`normal` | That's expected for non-LL modes here; this recipe pins `deepep_low_latency`. |

recipe/experimental/foundry_load.toml

@@ -0,0 +1,9 @@
+mode = "load"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc"
+scratch_space_size = "4096MB"
+# REQUIRED for the EP/IPC recipe — DeepEP low-latency still allocates its RDMA
+# buffer on the NVSHMEM symmetric heap (the NVL buffer is the cudaIpc path).
+# Point this at the libnvshmem_host.so built by vllm/tools/ep_kernels.
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_load_fp8.toml

@@ -0,0 +1,6 @@
+mode = "load"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc_fp8"
+scratch_space_size = "4096MB"
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_save.toml

@@ -0,0 +1,9 @@
+mode = "save"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc"
+scratch_space_size = "4096MB"
+# REQUIRED for the EP/IPC recipe — DeepEP low-latency still allocates its RDMA
+# buffer on the NVSHMEM symmetric heap (the NVL buffer is the cudaIpc path).
+# Point this at the libnvshmem_host.so built by vllm/tools/ep_kernels.
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/foundry_save_fp8.toml

@@ -0,0 +1,6 @@
+mode = "save"
+base_addr = 0x600000000000
+region_size = "256GB"
+workspace_root = "foundry_archive_ipc_fp8"
+scratch_space_size = "4096MB"
+nvshmem_host_path = "/data/liuxs/workarea/foundry-org/vllm/tools/ep_kernels/ep_kernels_workspace/nvshmem/lib/libnvshmem_host.so"

recipe/experimental/serve_qwen3-30ba3b_ipc_ep.sh

@@ -0,0 +1,92 @@
+#!/bin/bash
+# Usage: bash serve_qwen3-30ba3b_ipc_ep.sh <ep_size> [--save|--load]
+#
+# EXPERIMENTAL: DeepEP expert-parallel with the legacy CUDA-IPC NVLink buffer
+# (cudaIpcGet/OpenMemHandle) kept ON under foundry, instead of the default
+# fabric/NVSHMEM-only path. This exercises foundry's VMM-IPC translation layer
+# (SCM_RIGHTS fd transport + whole-chunk peer mapping) — the path that lets
+# foundry SAVE/LOAD work on machines without fabric/IMEX, where DeepEP shares
+# its intranode NVL buffers via legacy IPC.
+#
+# Differs from recipe/vllm/serve_qwen3-30ba3b_ep.sh by exactly one env var:
+#   FOUNDRY_DEEPEP_NVL_IPC=1  -> _patch_deepep keeps upstream's nonzero
+#   num_nvl_bytes with use_fabric=False (legacy cudaIpc NVL buffer on SAVE and
+#   LOAD), instead of forcing use_fabric=True / num_nvl_bytes=0.
+#
+# Requires the foundry hook rebuilt with the SCM_RIGHTS VMM-IPC transport and
+# the vLLM compile-cache fix (see this directory's README "Required code").
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+EP_SIZE=${1:?Usage: $0 <ep_size> [--save|--load]}
+MODEL_NAME="Qwen/Qwen3-30B-A3B"
+HOST="0.0.0.0"
+PORT=12000
+GPU_MEMORY_UTILIZATION=0.8
+
+FOUNDRY_ARGS=()
+if [[ "$2" == "--save" ]]; then
+    FOUNDRY_ARGS+=( --compilation-config.graph_extension_config_path "${SCRIPT_DIR}/foundry_save.toml" )
+    # Foundry pins NCCL to the IPC/ring transports — the CUMEM P2P and NVLS
+    # multicast fast paths cuMemMap with driver-capability flags the foundry
+    # VMM region doesn't carry.
+    export NCCL_CUMEM_ENABLE=0
+    export NCCL_NVLS_ENABLE=0
+    # Foundry only patches the V1 model runner; pin V1 explicitly so vLLM
+    # doesn't quietly route Qwen3ForCausalLM-class architectures to V2.
+    export VLLM_USE_V2_MODEL_RUNNER=0
+    # Keep DeepEP's NVLink buffer on the legacy cudaIpc path under foundry.
+    # Must be identical on SAVE and LOAD (it changes the comm-buffer alloc
+    # trajectory and the captured graphs).
+    export FOUNDRY_DEEPEP_NVL_IPC=1
+    echo "Using foundry SAVE (DeepEP NVL/IPC): ${SCRIPT_DIR}/foundry_save.toml"
+elif [[ "$2" == "--load" ]]; then
+    FOUNDRY_ARGS+=( --compilation-config.graph_extension_config_path "${SCRIPT_DIR}/foundry_load.toml" )
+    export NCCL_CUMEM_ENABLE=0
+    export NCCL_NVLS_ENABLE=0
+    export VLLM_USE_V2_MODEL_RUNNER=0
+    export FOUNDRY_DEEPEP_NVL_IPC=1
+    echo "Using foundry LOAD (DeepEP NVL/IPC): ${SCRIPT_DIR}/foundry_load.toml"
+elif [[ -n "$2" ]]; then
+    echo "Usage: $0 <ep_size> [--save|--load]"
+    exit 1
+else
+    echo "Running without foundry (baseline vLLM)"
+fi
+
+# LD_PRELOAD of libcuda_hook.so / libnvshmem_host.so is set by foundry's
+# setup_ld_preload_env at worker spawn time (uses the paths in the TOML
+# config). Baseline runs don't need either preloaded by the shell.
+
+# Foundry only patches the V1 model runner. vLLM defaults certain
+# architectures (e.g. Qwen3ForCausalLM) to the V2 runner, which our
+# patches don't touch — pin V1 here.
+export VLLM_USE_V2_MODEL_RUNNER=0
+
+export VLLM_USE_FLASHINFER_SAMPLER=1
+export VLLM_DISABLE_SHARED_EXPERTS_STREAM=1
+
+CUDAGRAPH_CAPTURE_SIZES=($(seq 1 256))
+
+ARGS=(
+    --trust-remote-code
+    --host "$HOST"
+    --port "$PORT"
+    --tensor-parallel-size 1
+    --data-parallel-size "$EP_SIZE"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --distributed-executor-backend uni
+    --enable-expert-parallel
+    --all2all-backend deepep_low_latency
+    --no-enable-prefix-caching
+    # max-num-batched-tokens == max-num-seqs satisfies vLLM's batched>=seqs check
+    # and keeps DeepEP's (tokens+1)*2 <= NVSHMEM_QP_DEPTH(=1024) — no QP override.
+    --max-num-batched-tokens 256
+    --max-num-seqs 256
+    --attention-config.backend FLASH_ATTN
+    --compilation-config.cudagraph_mode FULL_DECODE_ONLY
+    --compilation-config.cudagraph_num_of_warmups 0
+    --chat-template-content-format string
+    --cudagraph-capture-sizes ${CUDAGRAPH_CAPTURE_SIZES[@]}
+)
+
+vllm serve "$MODEL_NAME" "${ARGS[@]}" "${FOUNDRY_ARGS[@]}"

recipe/experimental/serve_qwen3-30ba3bfp8_ipc_ep.sh

@@ -0,0 +1,69 @@
+#!/bin/bash
+# Usage: bash serve_qwen3-30ba3bfp8_ipc_ep.sh <ep_size> [--save|--load]
+#
+# EXPERIMENTAL: FP8 Qwen3-30B-A3B with DeepGEMM MoE + DeepEP expert-parallel,
+# keeping the legacy CUDA-IPC NVLink buffer ON under foundry (FOUNDRY_DEEPEP_NVL_IPC=1).
+# Same as serve_qwen3-30ba3b_ipc_ep.sh but: FP8 model + VLLM_USE_DEEP_GEMM=1
+# (blockscale FP8 MoE), and its own archive (foundry_archive_ipc_fp8).
+#
+# Run every phase from one consistent path (or absolute path) so SAVE pass 1/2
+# and LOAD pass the identical graph_extension_config_path — see README.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+EP_SIZE=${1:?Usage: $0 <ep_size> [--save|--load]}
+MODEL_NAME="Qwen/Qwen3-30B-A3B-FP8"
+HOST="0.0.0.0"
+PORT=12000
+GPU_MEMORY_UTILIZATION=0.8
+
+FOUNDRY_ARGS=()
+if [[ "$2" == "--save" ]]; then
+    FOUNDRY_ARGS+=( --compilation-config.graph_extension_config_path "${SCRIPT_DIR}/foundry_save_fp8.toml" )
+    export NCCL_CUMEM_ENABLE=0
+    export NCCL_NVLS_ENABLE=0
+    export VLLM_USE_V2_MODEL_RUNNER=0
+    export FOUNDRY_DEEPEP_NVL_IPC=1
+    echo "Using foundry SAVE (FP8/DeepGEMM, DeepEP NVL/IPC): ${SCRIPT_DIR}/foundry_save_fp8.toml"
+elif [[ "$2" == "--load" ]]; then
+    FOUNDRY_ARGS+=( --compilation-config.graph_extension_config_path "${SCRIPT_DIR}/foundry_load_fp8.toml" )
+    export NCCL_CUMEM_ENABLE=0
+    export NCCL_NVLS_ENABLE=0
+    export VLLM_USE_V2_MODEL_RUNNER=0
+    export FOUNDRY_DEEPEP_NVL_IPC=1
+    echo "Using foundry LOAD (FP8/DeepGEMM, DeepEP NVL/IPC): ${SCRIPT_DIR}/foundry_load_fp8.toml"
+elif [[ -n "$2" ]]; then
+    echo "Usage: $0 <ep_size> [--save|--load]"
+    exit 1
+else
+    echo "Running without foundry (baseline vLLM)"
+fi
+
+export VLLM_USE_V2_MODEL_RUNNER=0
+export VLLM_USE_FLASHINFER_SAMPLER=1
+export VLLM_DISABLE_SHARED_EXPERTS_STREAM=1
+# FP8 blockscale MoE via DeepGEMM.
+export VLLM_USE_DEEP_GEMM=1
+
+CUDAGRAPH_CAPTURE_SIZES=($(seq 1 256))
+
+ARGS=(
+    --trust-remote-code
+    --host "$HOST"
+    --port "$PORT"
+    --tensor-parallel-size 1
+    --data-parallel-size "$EP_SIZE"
+    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION"
+    --distributed-executor-backend uni
+    --enable-expert-parallel
+    --all2all-backend deepep_low_latency
+    --no-enable-prefix-caching
+    --max-num-batched-tokens 256
+    --max-num-seqs 256
+    --attention-config.backend FLASH_ATTN
+    --compilation-config.cudagraph_mode FULL_DECODE_ONLY
+    --compilation-config.cudagraph_num_of_warmups 0
+    --chat-template-content-format string
+    --cudagraph-capture-sizes ${CUDAGRAPH_CAPTURE_SIZES[@]}
+)
+
+vllm serve "$MODEL_NAME" "${ARGS[@]}" "${FOUNDRY_ARGS[@]}"