feat(deepep-efa): NVIDIA Dynamo DeepEP-V2 MoE all-to-all over EFA (NCCL-GIN CPU-proxy)

[dmvevents]

What this adds

A new self-contained inference example: NVIDIA Dynamo (dynamo.frontend OpenAI ingress +
dynamo.vllm engine) serving a Mixture-of-Experts model with DeepEP-V2 (ElasticBuffer)
expert-parallel all-to-all, routed over AWS EFA via the NCCL-GIN CPU-proxy path
(NCCL_GIN_TYPE=2), on p5en.48xlarge (H200).

Path: examples/inference/nvidia-dynamo/deepep-v2-efa/

It is the Dynamo-frontend counterpart to the vLLM DeepEP-V2 sample (examples/inference/vllm/deepep-v2-efa/,
open in #1230). dynamo.vllm wraps the same vLLM engine — it forwards every unknown CLI flag straight into
vLLM's AsyncEngineArgs — so nothing about the DeepEP-V2 / EFA transport changes; this sample adds only
an OpenAI-compatible frontend and a DP/EP-aware worker wrapper on top of that proven substrate.

Why this is not duplicating an existing PR

• Not the other DeepEP-family PRs. The open DeepEP-EFA family covers different serving engines:
  feat(deepep-efa): vLLM DeepEP-V2 MoE all-to-all over EFA (eager + non-eager) #1230 (vLLM), feat(deepep-efa): vLLM DeepEP-V2 MoE all-to-all over EFA via NCCL-GIN GDAKI (GPU-initiated) #1231 (vLLM + UCCL/NIXL), feat(deepep-efa): TensorRT-LLM NcclEP MoE all-to-all over EFA (NCCL-GIN CPU-proxy) #1240 (TRT-LLM), feat(deepep-efa): NeMo-RL GRPO + Megatron Shape-Y MoE all-to-all over EFA (DeepEP-V2 NCCL-GIN) #1242 (NeMo-RL). None serves the DeepEP-V2/EFA
  engine behind an NVIDIA Dynamo front. This is the Dynamo member of that family.
• Not the existing nvidia-dynamo/ platform sample. The current examples/inference/nvidia-dynamo/
  folder is a SageMaker HyperPod-EKS deployment driven by the Dynamo operator and its
  DynamoGraphDeployment (DGD) CRD, serving GPT-OSS / Qwen3.6 in aggregated + disaggregated (NIXL)
  flavors on ml.g6e.4xlarge. That is a materially different execution model (operator + CRD + etcd/NATS
  platform, L40S, KV-transfer disagg) from this sample's raw two-node DeepEP-V2 MoE all-to-all serving on
  p5en/H200. See the "extend-before-create" note below.
• Duplicate-work checks (AGENTS.md §1) run clean: no open PR adds a Dynamo DeepEP-V2/EFA example.

Extend-before-create (AGENTS.md §2)

Closest existing sibling: examples/inference/nvidia-dynamo/ (matheus's Dynamo-platform sample).
This new content is placed as a variant subdirectory inside it (nvidia-dynamo/deepep-v2-efa/), not as
a parallel top-level sibling — so it extends the existing nvidia-dynamo/ engine folder rather than
creating a new one.

A new subdirectory (rather than a README section or model recipe inside the platform sample) is justified
because the execution model differs materially from the platform sample:

	nvidia-dynamo/ (existing)	nvidia-dynamo/deepep-v2-efa/ (this PR)
Orchestration	Dynamo operator + DGD CRD + etcd/NATS platform (Helm)	raw 2-node StatefulSet; file discovery backend; no operator/etcd/NATS
Cross-node fan-out	operator-wired discovery	vLLM's native DP coordinator (--headless, --data-parallel-address) over EFA
MoE transport	(agg / NIXL-disagg KV transfer)	DeepEP-V2 ElasticBuffer expert all-to-all over EFA (NCCL-GIN CPU-proxy)
Hardware	ml.g6e.4xlarge (L40S 48GB)	p5en.48xlarge (H200), 16 EFA NICs/node

The two share the same engine name (nvidia-dynamo) — which is why this lives under that folder — but
not the same dependencies or run path, which is the AGENTS.md test for a new variant subdirectory.

How it works (one paragraph)

DeepEP's default transport is NVSHMEM/IBGDA, which EFA does not provide. The V2 (ElasticBuffer) path runs
its dispatch/combine over aws-ofi-nccl's GIN CPU-proxy (NCCL_GIN_TYPE=2, OFI_NCCL_GIN_GDAKI=0) on the
efa-direct fabric. Three things make it work: EP_REUSE_NCCL_COMM=0 (or serve init segfaults — torch's
NCCL comm is lazy/null under vLLM before ElasticBuffer construction); the gdrcopy forced-PCIe capability
for the GIN plugin on gdrdrv-2.4 hosts (OFI_NCCL_GDRCOPY_FORCED_PCIE_COPY=1, the parameter from
aws/aws-ofi-nccl#1351, cherry-picked at a pinned SHA); and DeepEP-V2 source pinned to b306af06 + PR#612
(EFA auto-QP cap) at its immutable head SHA. Full detail + the eager/non-eager status table are in the folder
README. NGC-from-scratch: the Dockerfile is FROM nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu22.04, all deps
built from public source; no private image is referenced anywhere.

The one Dynamo-specific delta beyond the vLLM substrate is a readiness contract documented in the README
("Readiness on Dynamo — why /health alone lies"): dynamo.frontend's /health returns 200 as soon as the
HTTP server binds (its ServiceObserver defaults to Ready and nothing flips it on engine registration), so
the Kubernetes leader probe requires /health 200 and a populated endpoints array — otherwise the
probe would pass hours before the model finishes loading and defeat the startup budget.

Test Results

Cluster: Amazon EKS, 2× p5en.48xlarge (H200, 8 GPU + 16 EFA NIC/node), EFA K8s device plugin,
ap-southeast-3.
Model: Qwen/Qwen3-30B-A3B-FP8 (public), DP16 / EP16, --enforce-eager (the shipped default).
Image: NGC-from-scratch, built from the in-tree Dockerfile
(FROM nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu22.04); vLLM pinned to the DeepEP-V2-backend merge commit
e2f993dc4 (0.22.1rc1.dev283) + ai-dynamo{,-runtime}==1.3.1. Run 2026-09-04.

1. EFA transport proof, before model load

recipe/run-kernel-test.sh on both nodes runs DeepEP/tests/elastic/test_ep.py (DeepEP-V2 dispatch +
combine) and prints KERNEL-TEST PASS only when the NCCL_DEBUG=INFO log shows the efa-direct banner,
so a green result cannot be a silent TCP/SHM fallback:

--- LEADER (node_rank=0) ---
KERNEL-TEST PASS (node_rank=0) — DeepEP-V2 dispatch/combine over EFA verified
dynamo-deepep-v2-0 NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 16 nics)

--- WORKER (node_rank=1) ---
KERNEL-TEST PASS (node_rank=1) — DeepEP-V2 dispatch/combine over EFA verified
dynamo-deepep-v2-1 NCCL INFO NET/OFI Selected provider is efa, fabric is efa-direct (found 16 nics)

(This gate exercises DeepEP + aws-ofi-nccl + EFA only — not vLLM or Dynamo — so it was captured on an image
whose transport layers, Dockerfile Layers 1–4, are byte-identical to the shipped image; only the vLLM/Dynamo
layers above them differ. The serve-path proof on the shipped image is §3.)

2. End-to-end serving — both pods Ready, coherent chat completion

Two-node StatefulSet (kubernetes/). The leader turns Ready only after /health carries a populated
endpoints array (post-weight-load engine registration — see README "why /health alone lies"):

$ kubectl -n dynamo-deepep get pods -o wide
NAME                 READY   STATUS    RESTARTS      AGE   NODE
dynamo-deepep-v2-0   1/1     Running   0             23m   ip-10-240-84-22    (leader)
dynamo-deepep-v2-1   1/1     Running   1 (15m ago)   23m   ip-10-240-79-100   (worker)
#   worker RESTARTS=1: HF anonymous-pull rate limit (HTTP 429) on one shard while both pods pulled
#   ~30 GB in parallel; recovered on retry from the per-pod cache (see kubernetes/ manifest note).

$ kubectl -n dynamo-deepep exec dynamo-deepep-v2-0 -- curl -s http://127.0.0.1:8000/health
{"status":"healthy",
 "endpoints":["dyn://dynamo.backend.clear_kv_blocks","dyn://dynamo.backend.generate",
              "dyn://dynamo.backend.get_perf_metrics"],
 "instances":[{"component":"backend","endpoint":"generate","namespace":"dynamo", ...}]}

The vLLM engine came up as the pinned DeepEP-V2 substrate across all 16 DP ranks with no CUDA launch fault
(the exact failure mode that ruled out the vLLM 0.26 line on this DeepEP/EFA substrate — see README
"eager vs non-eager"):

Initializing a V1 LLM engine (v0.22.1rc1.dev283+ge2f993dc4) with ... data_parallel_size=16,
  enforce_eager=True, quantization=fp8
GPU KV cache size: 1,015,152 tokens
INFO utils.launch_core_engines: Started DP Coordinator process (PID: 2507)
INFO coordinator.py:249 All engine subscriptions received by DP coordinator
init engine (profile, create kv cache, warmup model) took 226.42 s

Chat completion via the leader's OpenAI endpoint:

$ curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' -d \
  '{"model":"Qwen/Qwen3-30B-A3B-FP8","max_tokens":96,"temperature":0,
    "messages":[{"role":"user",
      "content":"In one sentence, what is expert parallelism in a Mixture-of-Experts model?"}]}'
{"id":"chatcmpl-306c1629-8779-48e8-9641-1eb13b8a3566","model":"Qwen/Qwen3-30B-A3B-FP8",
 "object":"chat.completion",
 "choices":[{"index":0,"finish_reason":"length","message":{"role":"assistant","content":
   "<think>\nOkay, the user is asking for a one-sentence definition of expert parallelism in a
    Mixture-of-Experts (MoE) model. Let me start by recalling what I know about MoE models. \n\nMoE
    models are a type of neural network architecture where multiple \"experts\" (sub-networks) are
    used, and a gating mechanism decides which experts to activate for each input. Now, expert
    parallelism... I think this refers to how the experts"}}],
 "usage":{"prompt_tokens":26,"completion_tokens":96,"total_tokens":122}}

3. EFA carried the expert all-to-all during serving

On this cluster's EFA kernel driver (3.3.0g) the rdma_*_bytes hw_counters sysfs node is not exposed
(/sys/class/infiniband/rdmap*/ports/1/hw_counters/ is absent even though all 16 rdmap* devices
enumerate under ibv_devices), so a per-request byte delta cannot be read on this host. The serve-path
proof instead is that DeepEP-V2 brought its expert all-to-all up on EFA on all 16 DP ranks across both
nodes — the per-rank banner printed during engine init, under --all2all-backend deepep_v2,
NCCL_GIN_TYPE=2, FI_PROVIDER=efa:

(Worker_DP0_EP0  ...) [DeepEP] EFA detected: capping num_allocated_qps 129 -> 2 ...   # leader node, DP0–DP7
   ... DP1_EP1 … DP7_EP7 identical ...
(Worker_DP8_EP8  ...) [DeepEP] EFA detected: capping num_allocated_qps 129 -> 2 ...   # worker node, DP8–DP15
   ... DP9_EP9 … DP15_EP15 identical ...

All 16 ranks (DP0–7 on the leader node, DP8–15 on the worker node) initialized the DeepEP-V2 EFA path;
combined with the kernel-test efa-direct (found 16 nics) banner in §1, the expert traffic is on EFA, not a
TCP/SHM fallback.

Provenance note: the benchmarks/ tables in the folder are the vLLM twin's numbers measured at this same
pin (e2f993dc4); they are carried as historical reference and are not re-measured on this Dynamo
image. This PR's verification is the functional E2E above (transport proof + coherent completion under the
Dynamo front), not a re-measured throughput table — stated honestly per the folder README's "Known
limitations".

Checklist

• New example is self-contained (own README, pinned versions, no latest tags).
• NGC-from-scratch Dockerfile (FROM nvcr.io/nvidia/*); no private image referenced.
• npx markdownlint-cli2 clean on changed markdown.
• Extend-before-create justified (closest sibling named; execution model differs).
• ## Test Results filled with real E2E output on 2× p5en (DP16/EP16, coherent completion, EFA verified).