chore(dynamo-platform): coordinate NATS removal with Dynamo 1.4+ bump #1983
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,40 +1,54 @@ | ||
| # EKS Dynamo Networking Prerequisites | ||
|
|
||
| For `*-eks-ubuntu-inference-dynamo` recipes, AICR configures | ||
| `dynamo-platform` with Kubernetes-native discovery. As of the Dynamo 1.4+ | ||
| bump, AICR no longer installs bundled NATS by default: the request plane | ||
| defaults to TCP and the KV event plane defaults to ZMQ | ||
| (`ai-dynamo/dynamo#11951`). This removes the old `4222` NATS requirement, | ||
| but it does **not** remove the underlying cross-nodegroup networking | ||
| requirement — the request plane and KV events are now **direct | ||
| frontend↔worker pod-to-pod connections** instead of both sides talking to a | ||
| `dynamo-platform-nats` StatefulSet on the system nodegroup, and Frontend | ||
| pods still run on the system nodegroup while workers run on the GPU | ||
| nodegroup, so traffic still crosses the same GPU↔system nodegroup SG | ||
| boundary as before. | ||
|
|
||
| > **TODO before merging (tracked in NVIDIA/aicr#1836):** the port(s) below | ||
| > are not yet confirmed against a real Dynamo 1.4+ EKS deployment. What's | ||
| > known from the AICR recipes: the ZMQ KV-event endpoint is set explicitly | ||
| > per worker via `--kv-events-config`, e.g. | ||
| > `{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5557"}` | ||
| > (see `tests/manifests/dynamo-vllm-smoke-test.yaml`), offset by `+dp_rank` | ||
| > for dp_rank > 0. The TCP request plane does not have one fixed, | ||
| > documented port the way NATS had `4222` — confirm the actual listening | ||
| > port(s) on a live cluster before finalizing the SG rule below: | ||
| > ```shell | ||
|
Contributor
There was a problem hiding this comment. 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win Add a blank line before the fenced block.
🧰 Tools🪛 markdownlint-cli2 (0.23.1)[warning] 25-25: Fenced code blocks should be surrounded by blank lines (MD031, blanks-around-fences) 🤖 Prompt for AI AgentsSource: Linters/SAST tools
Member
There was a problem hiding this comment. 🔵 Nitpick — MD031 — fenced block not preceded by a blank line Inside the blockquote, line 24 is immediately followed by the opening ```shell fence at line 25 with no blank Blast radius: markdownlint / doc build; cosmetic. Fix: Insert a blank blockquote line ( |
||
| > kubectl exec -n dynamo-system <frontend-pod> -- ss -tlnp | ||
| > kubectl exec -n dynamo-system <worker-pod> -- ss -tlnp | ||
| > ``` | ||
|
|
||
| If the GPU and system node groups sit in different security groups, these | ||
| ports may be blocked from GPU nodes to the frontend's node (and vice versa). | ||
| Typical symptoms: | ||
| - Dynamo frontend and vLLM worker pods stuck in `CrashLoopBackOff`, or a | ||
| frontend that starts cleanly but never successfully routes a request | ||
| through to a worker | ||
| - Worker startup probes failing with `connection refused` because the | ||
| process exits before serving | ||
| - The `inference-perf` performance validator failing after its | ||
| workload-readiness (10 min) and health (5 min) gates lapse — roughly | ||
| 15 min — while `deployment` and `conformance` pass; the workload never | ||
| reaches a ready state | ||
|
|
||
| You can confirm reachability directly from a GPU node before re-running. The | ||
| toleration is required because the GPU node groups on these clusters are | ||
| tainted (`NoSchedule`/`NoExecute`); without it the probe pod stays `Pending` | ||
| and never runs: | ||
|
|
||
| ```shell | ||
| kubectl run tcp-probe --rm -i --restart=Never --image=busybox:1.36 \ | ||
| --overrides='{"spec":{"nodeSelector":{"<gpu-node-label-key>":"<value>"},"tolerations":[{"operator":"Exists"}]}}' \ | ||
| -- sh -c 'nc -zv -w 5 <worker-pod-ip-or-svc> <PORT>' | ||
|
Comment on lines
43
to
+51
Contributor
There was a problem hiding this comment. 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- target file structure ---'
ast-grep outline docs/integrator/eks-dynamo-networking.md --lang markdown || true
printf '%s\n' '--- target lines and surrounding content ---'
sed -n '1,120p' docs/integrator/eks-dynamo-networking.md
printf '%s\n' '--- related networking and probe references ---'
rg -n -i -C 3 'GPU|system|nodegroup|node group|request port|event port|security.?group|tcp-probe|worker endpoint|frontend|worker' docs/integrator docs README.md 2>/dev/null | head -300Repository: NVIDIA/aicr Length of output: 32806 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- relevant tracked files ---'
git ls-files | rg -i 'dynamo|eks|manifest|validator|prometheus' | head -250
printf '%s\n' '--- Dynamo deployment and scheduling definitions ---'
rg -n -i -C 4 'dynamo-platform|frontend|worker|nodeSelector|toleration|tolerations|kv-events-config|5557|request.*port|service.*port|port:' recipes tests pkg manifests components 2>/dev/null | head -500
printf '%s\n' '--- exact target documentation line numbers ---'
nl -ba docs/integrator/eks-dynamo-networking.md | sed -n '1,115p'Repository: NVIDIA/aicr Length of output: 48555 🏁 Script executed: #!/bin/bash
set -e
printf '%s\n' '--- smoke-test manifest ---'
sed -n '35,100p' tests/manifests/dynamo-vllm-smoke-test.yaml
printf '%s\n' '--- Dynamo component values and EKS overlays ---'
sed -n '1,220p' recipes/components/dynamo-platform/values.yaml
for f in recipes/overlays/gb200-eks-ubuntu-inference-dynamo.yaml recipes/overlays/h100-eks-ubuntu-inference-dynamo.yaml recipes/overlays/rtx-pro-6000-eks-ubuntu-inference-dynamo.yaml recipes/overlays/eks-inference.yaml; do
if [ -f "$f" ]; then
printf '\n--- %s ---\n' "$f"
sed -n '1,220p' "$f"
fi
done
printf '%s\n' '--- focused diff for the documentation ---'
git diff -- docs/integrator/eks-dynamo-networking.md
printf '%s\n' '--- exact target line numbers ---'
awk '{printf "%5d %s\n", NR, $0}' docs/integrator/eks-dynamo-networking.md | sed -n '35,70p'Repository: NVIDIA/aicr Length of output: 18517 🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import re, json
text = Path("docs/integrator/eks-dynamo-networking.md").read_text()
block = re.search(r"```shell\n(.*?)\n```", text, re.S).group(1)
override = re.search(r"--overrides='([^']+)'", block).group(1)
spec = json.loads(override)
target = re.search(r"nc -zv -w 5 (\S+) (\S+)", block).groups()
assert spec["spec"]["nodeSelector"] == {
"<gpu-node-label-key>": "<value>"
}
assert target[0] == "<worker-pod-ip-or-svc>"
print("probe node selector:", spec["spec"]["nodeSelector"])
print("probe target:", target[0])
print("probe port:", target[1])
print("result: source is constrained to the GPU nodegroup and destination is constrained to a worker endpoint; this does not exercise a GPU-to-system path")
PYRepository: NVIDIA/aicr Length of output: 281 🏁 Script executed: #!/bin/bash
set -e
python3 - <<'PY'
from pathlib import Path
import json, re
text = Path("docs/integrator/eks-dynamo-networking.md").read_text()
start = text.index("kubectl run tcp-probe")
block = text[start:text.index("```", start)]
override = re.search(r"--overrides='([^']+)'", block).group(1)
spec = json.loads(override)
target = re.search(r"nc -zv -w 5 (\S+) (\S+)", block).groups()
print("probe node selector:", spec["spec"]["nodeSelector"])
print("probe tolerations:", spec["spec"]["tolerations"])
print("probe target:", target[0])
print("probe port:", target[1])
assert "<gpu-node-label-key>" in spec["spec"]["nodeSelector"]
assert target[0] == "<worker-pod-ip-or-svc>"
print("result: the documented source is constrained to the GPU nodegroup and the documented destination is a worker endpoint; this is not a GPU-to-system probe")
PYRepository: NVIDIA/aicr Length of output: 460 Add cross-nodegroup probes. The current probe runs on the GPU nodegroup and targets a worker endpoint, so it does not test GPU-to-system or system-to-GPU traffic. After identifying the actual request and KV-event listeners, run probes from each nodegroup to an endpoint in the other nodegroup. 🤖 Prompt for AI Agents |
||
| ``` | ||
|
|
||
| The conformance validator's `ai-service-metrics` check adds a third requirement: | ||
|
|
@@ -64,8 +78,9 @@ SG rule below remains the reliable cluster-side guarantee. | |
|
|
||
| ## Required Security Group Rules | ||
|
|
||
| Allow ingress from the GPU node security group to the system node security | ||
| group on: | ||
| - TCP `<PORT>` - Dynamo request plane + KV events (dynamo-platform) — confirm exact port(s) on-cluster, see TODO above | ||
|
mohityadav8 marked this conversation as resolved.
Member
There was a problem hiding this comment. 🟠 Major — Networking doc ships placeholders + 'TODO before merging' Lines 16 (TODO before merging, #1836), 51, 83, 114 carry unresolved placeholders in the probe, the required SG-rule list, and the AWS CLI example. This doc is authoritative — linked from docs/README.md, docs/integrator/index.md, and deep-linked from docs/user/validation.md#required-security-group-rules — so an operator with split GPU/system SGs cannot open the correct rule from it, and hits the exact CrashLoop/timeout symptoms it describes. Blast radius: Operators cannot complete the SG rule; overlaps mchmarny's standing MAJOR. Fix: Confirm the real request-plane + ZMQ ports on a live 1.4.1 EKS cluster (per the doc's own ss -tlnp step), fill all three s and remove the TODO — or hold the rewrite until #1836 resolves ports. |
||
| - TCP `9090` - Prometheus (required for the `ai-service-metrics` conformance check) | ||
|
|
||
| The `9090` rule is required as a fallback guarantee: the orchestrator *prefers* | ||
|
|
@@ -94,9 +109,9 @@ aws ec2 describe-instances \ | |
| --query "Reservations[0].Instances[0].SecurityGroups[*].GroupId" \ | ||
| --output text | ||
|
|
||
| # 2) Allow Dynamo request/event-plane + Prometheus from GPU SG -> system SG | ||
| aws ec2 authorize-security-group-ingress --group-id <system-sg-id> \ | ||
| --protocol tcp --port <PORT> --source-group <gpu-sg-id> | ||
|
|
||
| aws ec2 authorize-security-group-ingress --group-id <system-sg-id> \ | ||
| --protocol tcp --port 9090 --source-group <gpu-sg-id> | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,11 +20,11 @@ | |
| # serving Qwen/Qwen3-0.6B (public, ungated, ~1.2GB). | ||
| # No HuggingFace token required. | ||
| # | ||
| # Uses the Dynamo 1.4+ Kubernetes path: Kubernetes-native discovery, with | ||
| # the request plane on TCP (default) and the KV event plane on ZMQ | ||
| # (default) — no bundled NATS relay in between. The frontend runs | ||
| # KV-cache-aware routing; the local vLLM engine publishes KV events over | ||
| # ZMQ and the KV router consumes them directly, end-to-end. | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
| # | ||
| # Usage: | ||
| # kubectl apply -f tests/manifests/dynamo-vllm-smoke-test.yaml -n dynamo-system | ||
|
|
@@ -49,12 +49,16 @@ spec: | |
| spec: | ||
| containers: | ||
| - name: main | ||
| image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 | ||
| env: | ||
| - name: SERVED_MODEL_NAME | ||
| value: Qwen/Qwen3-0.6B | ||
| - name: DYN_ROUTER_MODE | ||
| value: kv | ||
| - name: DYN_REQUEST_PLANE | ||
| value: tcp | ||
| - name: DYN_EVENT_PLANE | ||
| value: zmq | ||
|
Comment on lines
+60
to
+61
Contributor
There was a problem hiding this comment. 🗄️ Data Integrity & Integration | 🔵 Trivial 🔎 Supported by static analysis🏁 Script executed: printf '%s\n' '--- relevant repository conventions ---'
find /tmp/coderabbit-repo-knowledge/nvidia-aicr-7c4812cf -type f -name '*.md' -maxdepth 3 -print
printf '%s\n' '--- target manifest ---'
cat -n tests/manifests/dynamo-vllm-smoke-test.yaml | sed -n '1,110p'
printf '%s\n' '--- directly bound transport and KV-event references ---'
rg -n -C 3 'DYN_(REQUEST|EVENT)_PLANE|kv-events-config|5557|KV events|ZMQ|zmq' \
tests/manifests . --glob '!tests/manifests/dynamo-vllm-smoke-test.yaml' --glob '*.yaml' --glob '*.yml' --glob '*.md' | head -240Repository: NVIDIA/aicr Length of output: 28658 🏁 Script executed: printf '%s\n' '--- scoped conventions and learnings ---'
head -80 /tmp/coderabbit-repo-knowledge/nvidia-aicr-7c4812cf/conventions/recipes.md
head -80 /tmp/coderabbit-repo-knowledge/nvidia-aicr-7c4812cf/conventions/pkg.md
head -80 /tmp/coderabbit-repo-knowledge/nvidia-aicr-7c4812cf/learnings/yml.md
head -80 /tmp/coderabbit-repo-knowledge/nvidia-aicr-7c4812cf/learnings/tests-chainsaw.md
printf '%s\n' '--- networking contract ---'
cat -n docs/integrator/eks-dynamo-networking.md | sed -n '1,95p'
printf '%s\n' '--- chart defaults and related manifest structure ---'
cat -n recipes/components/dynamo-platform/values.yaml | sed -n '20,42p'
cat -n validators/performance/testdata/inference/dynamo-deployment.yaml | sed -n '70,108p'Repository: NVIDIA/aicr Length of output: 11502 🌐 Web query:
💡 Result: In Dynamo v1.4, the ZMQ endpoint Citations:
🌐 Web query:
💡 Result: The string tcp://:5557 refers to the network endpoint configuration for the ZMQ (ZeroMQ) transport used by the Dynamo event plane [1][2]. In the context of the Dynamo system, DYN_EVENT_PLANE is an environment variable used to specify the transport mechanism for publishing and subscribing to events (such as KV cache events) [1][2]. The two supported transports are: 1. ZMQ (default): Uses ZeroMQ for peer-to-peer communication, requiring no external infrastructure [2]. When using this transport, workers bind their ZMQ sockets to a specified endpoint (e.g., tcp://:5557) and register their addresses via a discovery backend [2]. 2. NATS: Requires an external NATS server, which is configured via the NATS_SERVER environment variable [2]. When configuring backend engines like vLLM for use with Dynamo, you can specify the ZMQ endpoint as part of the KV events configuration, for example: --kv-events-config '{"publisher": "zmq", "endpoint": "tcp://:5557", "enable_kv_cache_events": true}' [1]. The asterisk () in the address indicates that the socket will bind to all available network interfaces on port 5557 [2]. Citations:
🌐 Web query:
💡 Result: In NVIDIA Dynamo v1.4, ZMQ (ZeroMQ) serves as a transport option for the event plane, which facilitates communication between workers and router components, including KV cache event propagation and worker load metrics [1][2]. The ZMQ event plane configuration, including the use of endpoint 5557, works as follows: 1. Worker Endpoint Binding: When Citations:
Confirm ZMQ reachability before merge. Dynamo 1.4.1 documents 🤖 Prompt for AI Agents |
||
| - name: VllmDecodeWorker | ||
| type: worker | ||
| replicas: 1 | ||
|
|
@@ -67,9 +71,14 @@ spec: | |
| effect: NoSchedule | ||
| containers: | ||
| - name: main | ||
| image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.4.1 | ||
| workingDir: /workspace/examples/backends/vllm | ||
| command: ["python3", "-m", "dynamo.vllm"] | ||
| env: | ||
| - name: DYN_REQUEST_PLANE | ||
| value: tcp | ||
| - name: DYN_EVENT_PLANE | ||
| value: zmq | ||
| args: | ||
| - --model | ||
| - Qwen/Qwen3-0.6B | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
MAJOR: The replacement networking contract is explicitly unfinished: the probe and security-group commands retain
<PORT>, AWS UAT still allows only the removed NATS port 4222, and current user guidance still describes a NATS-backed event plane. On supported EKS installs with separate GPU and system security groups, AICR can install while TCP requests or ZMQ KV events remain blocked, and current UAT will not detect it.Minimum correction: establish the exact 1.4.1 listener ports and traffic directions, replace every placeholder with actionable rules and probes, and synchronize UAT and user guidance to that tested TCP/ZMQ contract.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mchmarny I've pushed the 1.4.1 fix for the registry blocker.
For the networking doc - I don't have access to a live 1.4.1 EKS cluster to confirm the TCP request-plane port. The ZMQ KV-events port 5557 is confirmed from the smoke test manifest, but the request-plane port needs a live ss -tlnp run on the actual cluster. Could you or @yuanchen8911 share that output? Once I have the real port I can immediately replace , update the SG rules, fix the probe commands, and push the final doc.