Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions ci/k8s/client/sgl/manifest-azure-mooncake.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,18 @@ spec:
- --modelexpress-config
- '{"transport": "transfer_engine"}'
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MX_SERVER_ADDRESS
value: "mx-server:8000"
- name: MX_RDMA_NIC_PIN
Expand Down
12 changes: 12 additions & 0 deletions ci/k8s/client/sgl/manifest-azure.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,18 @@ spec:
- --modelexpress-config
- '{"transport": "nixl"}'
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: MX_SERVER_ADDRESS
value: "mx-server:8000"
- name: MX_P2P_METADATA
Expand Down
12 changes: 12 additions & 0 deletions ci/k8s/client/vllm/dynamo/manifest-azure-aggregated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,18 @@ spec:
custom:
rdma/ib: "1"
envs:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: VLLM_PLUGINS
value: "modelexpress"
- name: MX_SERVER_ADDRESS
Expand Down
24 changes: 24 additions & 0 deletions ci/k8s/client/vllm/dynamo/manifest-azure-disaggregated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,18 @@ spec:
custom:
rdma/ib: "1"
envs:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: VLLM_PLUGINS
value: "modelexpress"
- name: MX_SERVER_ADDRESS
Expand Down Expand Up @@ -206,6 +218,18 @@ spec:
custom:
rdma/ib: "1"
envs:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: VLLM_PLUGINS
value: "modelexpress"
- name: MX_SERVER_ADDRESS
Expand Down
12 changes: 12 additions & 0 deletions ci/k8s/client/vllm/manifest-azure.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,18 @@ spec:
- --port
- "${WORKER_PORT}"
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: VLLM_PLUGINS
value: "modelexpress"
- name: MX_SERVER_ADDRESS
Expand Down
27 changes: 27 additions & 0 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,33 @@ reachable.

To use the Kubernetes backend, apply `examples/crds.yaml` at cluster install time (installs both the `ModelMetadata` P2P CRD and the `ModelCacheEntry` registry CRD), then either enable `serviceAccount.rbac.enabled=true` on the Helm chart or apply `examples/p2p_transfer_k8s/server/kubernetes_backend/rbac-modelmetadata.yaml`.

For automatic cleanup of P2P metadata, expose the client Pod identity through
the Kubernetes Downward API. The checked-in vLLM, SGLang, and Dynamo manifests
already include these fields:

```yaml
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
```

With the Kubernetes metadata backend, a complete same-namespace identity makes
the Pod the owner of every weight or artifact `ModelMetadata` CR it publishes.
Kubernetes then garbage-collects those records when the Pod is deleted. Missing,
partial, or cross-namespace identity does not block publication; ModelExpress
omits the owner reference and retains the previous reaper-based lifecycle. Redis
and non-Kubernetes deployments ignore these fields.

#### Storage access modes

MX has one configurable filesystem path, the model weights cache (`MODEL_EXPRESS_CACHE_DIRECTORY`, default `./cache`). Its access-mode requirement depends on deployment topology, not on MX itself:
Expand Down
27 changes: 26 additions & 1 deletion docs/metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,18 @@ Uses `ModelMetadata` CRDs for P2P source metadata, `ConfigMap`s for tensor descr

**ConfigMap name format**: `mx-source-{source_id}-{worker_id}-tensors-worker-{rank}`

ConfigMaps use `ownerReferences` pointing to the parent CRD so they are garbage-collected automatically.
When a client publishes a complete Kubernetes Pod identity (`POD_NAME`,
`POD_UID`, and `POD_NAMESPACE`) and the Pod is in the metadata namespace, the
`ModelMetadata` CR uses that Pod as an owner. This applies to both weight and
artifact metadata because both use the same `PublishMetadata` RPC. Deleting the
Pod therefore garbage-collects its `ModelMetadata` CR. Tensor `ConfigMap`s use a
second owner reference pointing to the parent CR, so they are collected with it.

Kubernetes owner references cannot cross namespaces. If the identity is
missing, partial, or names a different namespace, publication still succeeds
without a Pod owner reference. This preserves behavior for older clients and
non-Kubernetes environments; the server-side stale metadata reaper remains the
cleanup path in those cases.

**Model lifecycle CRD name format**: `mx-cache-{sanitized-model-name}-{hash}`

Expand All @@ -353,6 +364,13 @@ apiVersion: modelexpress.nvidia.com/v1alpha1
kind: ModelMetadata
metadata:
name: mx-source-a1b2c3d4e5f67890-f3a2b1c4
ownerReferences:
- apiVersion: v1
kind: Pod
name: mx-vllm-7d9f8f6c8b-k2m4p
uid: 8c69d55f-3e40-4b6e-a16b-6f0c1168e171
controller: false
blockOwnerDeletion: false
labels:
modelexpress.nvidia.com/mx-source-id: a1b2c3d4e5f67890
modelexpress.nvidia.com/mx-worker-id: f3a2b1c4
Expand Down Expand Up @@ -385,6 +403,13 @@ apiVersion: modelexpress.nvidia.com/v1alpha1
kind: ModelMetadata
metadata:
name: mx-source-b2c3d4e5f67890a1-f3a2b1c4
ownerReferences:
- apiVersion: v1
kind: Pod
name: mx-vllm-7d9f8f6c8b-k2m4p
uid: 8c69d55f-3e40-4b6e-a16b-6f0c1168e171
controller: false
blockOwnerDeletion: false
spec:
modelName: artifact-transfer-e2e
sourceType: deep_gemm_cache
Expand Down
10 changes: 10 additions & 0 deletions examples/p2p_transfer_k8s/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,16 @@ On startup, the engine-specific ModelExpress loader auto-detects the best loadin

After loading, every worker publishes its metadata so future instances can discover it as an RDMA source.

When the server uses the Kubernetes metadata backend, the example manifests
also inject `POD_NAME`, `POD_UID`, and `POD_NAMESPACE` through the Downward API.
ModelExpress uses a complete same-namespace identity to make the Pod the owner
of each `ModelMetadata` CR it publishes, including artifact metadata. This
ownership behavior is independent of the inference engine.
Deleting the Pod then removes its metadata through Kubernetes garbage
collection. If any identity field is unavailable, publication continues without
an owner reference so older clients and non-Kubernetes deployments remain
compatible.

## Prerequisites

- ModelExpress server deployed (see [`../server/`](../server/))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,18 @@ spec:
port: 8000
timeoutSeconds: 10
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: HF_HOME
value: "/models"
- name: MODEL_NAME
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,18 @@ spec:
port: 8000
timeoutSeconds: 10
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: HF_HOME
value: "/models"
- name: MODEL_NAME
Expand Down
8 changes: 8 additions & 0 deletions examples/p2p_transfer_k8s/client/vllm/vllm-multi-node.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,14 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: HF_TOKEN
valueFrom:
secretKeyRef:
Expand Down
8 changes: 8 additions & 0 deletions examples/p2p_transfer_k8s/client/vllm/vllm-single-node.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,14 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_UID
valueFrom:
fieldRef:
fieldPath: metadata.uid
- name: HF_TOKEN
valueFrom:
secretKeyRef:
Expand Down
40 changes: 37 additions & 3 deletions modelexpress_client/go/gen/modelexpress/p2p/p2p.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions modelexpress_client/python/modelexpress/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,9 @@ def publish_metadata(
identity=identity,
worker=worker,
worker_id=worker_id,
pod_name=envs.POD_NAME,
pod_uid=envs.POD_UID,
pod_namespace=envs.POD_NAMESPACE,
)
response = self.stub.PublishMetadata(request, timeout=30)
if not response.success:
Expand Down
4 changes: 4 additions & 0 deletions modelexpress_client/python/modelexpress/envs.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@
VLLM_ATTENTION_BACKEND: str
HOSTNAME: str
POD_NAMESPACE: str
POD_NAME: str
POD_UID: str

_TRUTHY = {"1", "true", "yes", "on"}

Expand Down Expand Up @@ -219,6 +221,8 @@ def _env_float(name: str, default: float) -> float:
"VLLM_ATTENTION_BACKEND": lambda: os.environ.get("VLLM_ATTENTION_BACKEND", "auto"),
"HOSTNAME": lambda: os.environ.get("HOSTNAME", ""),
"POD_NAMESPACE": lambda: os.environ.get("POD_NAMESPACE", ""),
"POD_NAME": lambda: os.environ.get("POD_NAME", ""),
"POD_UID": lambda: os.environ.get("POD_UID", "")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
"POD_UID": lambda: os.environ.get("POD_UID", "")
"POD_UID": lambda: os.environ.get("POD_UID", ""),

}


Expand Down
Loading
Loading