diff --git a/docs/README.md b/docs/README.md index da5da7873..f79e8e32f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -46,7 +46,7 @@ For pipelines and platforms that call AICR programmatically or host | Add or modify recipe metadata | [Recipe Development](integrator/recipe-development.md) | | Verify artifacts (SLSA, SBOM, attestations) | [Supply Chain Verification](integrator/supply-chain-verification.md) | | Ship custom validators via `--data` | [Validator Extension](integrator/validator-extension.md) | -| Cloud-specific GPU setup | [AKS](integrator/aks-gpu-setup.md), [GKE](integrator/gke-gpu-setup.md), [EKS networking](integrator/eks-dynamo-networking.md), [GKE networking](integrator/gke-tcpxo-networking.md), [Talos](integrator/talos-integration.md) | +| Cloud-specific GPU setup | [AKS](integrator/aks-gpu-setup.md), [GKE](integrator/gke-gpu-setup.md), [OKE](integrator/oke-gpu-setup.md), [EKS networking](integrator/eks-dynamo-networking.md), [GKE networking](integrator/gke-tcpxo-networking.md), [Talos](integrator/talos-integration.md) | ### Contributor Guide diff --git a/docs/integrator/index.md b/docs/integrator/index.md index c554a1909..5466b0c30 100644 --- a/docs/integrator/index.md +++ b/docs/integrator/index.md @@ -24,6 +24,7 @@ This section is for integrators who: | [GKE TCPXO Networking](gke-tcpxo-networking.md) | GPUDirect TCPXO prerequisites for GKE training overlays | | [AKS GPU Setup](aks-gpu-setup.md) | AKS prerequisites: Kubernetes 1.34+ (DRA GA), GPU driver setup, DRA configuration | | [GKE GPU Setup](gke-gpu-setup.md) | GKE device-plugin ownership: the `gpuStack` profile, node-pool setup for both values, verification, and troubleshooting | +| [OKE GPU Setup](oke-gpu-setup.md) | OKE GPU stack ownership: the `gpuStack` profile (three values), the device-plugin add-on / disable label, and bring-your-own-image pools | | [Talos Integration](talos-integration.md) | Running AICR on Talos Linux | | [OpenShift Deployment](openshift.md) | OpenShift/OCP-specific Helm and OLM integration and two-phase operator deployment | | [Recipe Development](recipe-development.md) | Creating and modifying recipe metadata for custom environments | diff --git a/docs/integrator/oke-gpu-setup.md b/docs/integrator/oke-gpu-setup.md new file mode 100644 index 000000000..e9bf35708 --- /dev/null +++ b/docs/integrator/oke-gpu-setup.md @@ -0,0 +1,102 @@ +# OKE GPU Setup + +## GPU Stack Ownership + +OKE installs NVIDIA's device plugin automatically on every cluster, and +Oracle's GPU node images preinstall the NVIDIA driver, container toolkit, and +host MOFED. Which of those a GPU node pool actually has depends on how it was +provisioned, and the AICR recipe must match it — the `gpuStack` configuration +profile on the OKE family names the three qualified combinations: + +| Value | Driver / toolkit | `nvidia.com/gpu` advertiser | Pool shape | +|---|---|---|---| +| `oci-default` (default) | Oracle GPU node image | OKE's auto-installed device plugin | stock OKE with Oracle GPU images | +| `operator-plugin` | Oracle GPU node image | GPU Operator's device plugin | OKE plugin disabled (node label or add-on removed) | +| `operator-managed` | GPU Operator installs both | GPU Operator's device plugin | bring-your-own driverless image | + +MOFED is host-supplied in every value — Oracle's GPU images and the common +bring-your-own images alike carry it, so `network-operator` never deploys +`ofedDriver` on OKE, and the device plugin runs with `MOFED_ENABLED=false` +(without it, k8s-device-plugin >= 0.19.0 with CDI floods every host RDMA +uverb into every GPU pod and breaks NCCL). + +Select the mode at recipe generation; the paths it owns are locked at every +output boundary: + +```shell +# Stock OKE cluster (the default) — no flag needed +aicr recipe --service oke --accelerator l40s --os ol --intent training -o recipe.yaml + +# OKE plugin disabled, image-supplied driver +aicr recipe --service oke --accelerator l40s --os ol --intent training \ + --profile gpuStack=operator-plugin -o recipe.yaml + +# Bring-your-own driverless image (e.g. a custom Ubuntu image) +aicr recipe --service oke --accelerator gb200 --os ubuntu --intent training \ + --profile gpuStack=operator-managed -o recipe.yaml +``` + +## Default: Stock OKE (`oci-default`) + +A default-provisioned OKE cluster with Oracle GPU images needs zero setup: +the image supplies the driver and toolkit, and OKE's device plugin advertises +`nvidia.com/gpu`. The GPU Operator manages the rest of the stack with its own +plugin disabled — running both plugins double-advertises the same GPUs, which +the #1327 exactly-one-advertiser policy forbids. + +## Alternative: Let the GPU Operator's Plugin Advertise (`operator-plugin`) + +If you prefer the GPU Operator's device plugin (feature discovery, MIG, CDI +control), disable OKE's plugin on the GPU pools and select the value: + +- **Per node pool (recommended, snapshot-visible):** add the node label + `oci.oraclecloud.com/disable-gpu-device-plugin=true` to the pool's initial + node labels at creation. +- **Per cluster:** remove the `NvidiaGpuPlugin` cluster add-on + (Terraform `addons = { NvidiaGpuPlugin = { remove = true } }`, or the + add-on lifecycle API). Note this leaves no on-node marker. + +The driver still comes from the Oracle image — the GPU Operator installs +nothing under this value. + +## Alternative: Bring-Your-Own Image (`operator-managed`) + +Custom images (OKE Ubuntu pools are always custom images) may ship no NVIDIA +stack at all. Under `operator-managed` the GPU Operator installs the driver +and toolkit, its device plugin advertises, and the DRA kubelet plugin reads +the driver userspace from the operator install path +(`/run/nvidia/driver` — the profile moves `nvidiaDriverRoot` in lockstep; +see the driver-ownership coherence rules). Disable OKE's device plugin on +these pools exactly as under `operator-plugin`. + +If your custom image DOES bake a driver (Oracle publishes downloadable +Ubuntu GPU images with driver + CUDA + DOCA-OFED), use `operator-plugin` +instead — a second, operator-installed driver on top of an image driver is +an unqualified hybrid. + +## Validation + +Qualification is partial, by design (tracked in +[#2363](https://github.com/NVIDIA/aicr/issues/2363)): + +| Axis | Signal | Status | +|---|---|---| +| Driver (image vs operator) | `GPU.hardware.driver-loaded` on the sampled GPU node | **Qualified.** `oci-default` and `operator-plugin` require it `true` at generation — durable, since the image driver persists and the `aicr validate` pre-flight re-evaluates the same constraint. A driverless (bring-your-own image) cluster fails both with remediation steering to `operator-managed`. `operator-managed` carries the post-deployment form as a readiness-scoped **deployment-outcome check**: it verifies the operator's driver install executed; it does not prove the value was the right choice. | +| Advertiser (OKE plugin vs operator plugin) | undecided | **Deferred.** Candidate signals — the `oci.oraclecloud.com/disable-gpu-device-plugin` node label vs a cluster add-on projection — are unresolved, and the node-label constraint evaluator's universe is GKE-only today. `oci-default` vs `operator-plugin` is currently an explicit `--profile` choice; verify the plugin state on your pools yourself (`kubectl get ds -n kube-system | grep nvidia-gpu-device-plugin`). | + +The snapshot samples a single GPU node; mixed pools are out of scope. + +## Oracle Add-on Interactions + +Do **not** enable Oracle's `NvidiaGpuOperator` or `NvidiaNetworkOperator` +managed add-ons alongside AICR bundles — the bundle deploys both operators +itself, and two lifecycle managers fight over the same releases. The only +Oracle GPU add-on compatible with AICR bundles is the device plugin +(`NvidiaGpuPlugin`), and only under `oci-default`. + +## References + +- [OKE: Running GPU Workloads](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengrunninggpunodes.htm) +- [OKE cluster add-ons](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm) +- [oci-hpc-oke quickstart](https://github.com/oracle-quickstart/oci-hpc-oke) — worker images, RDMA manifests +- [AKS GPU Setup](aks-gpu-setup.md), [GKE GPU Setup](gke-gpu-setup.md) — the sibling `gpuStack` families diff --git a/docs/user/cli-reference.md b/docs/user/cli-reference.md index 7b4ad7416..71fa997ca 100644 --- a/docs/user/cli-reference.md +++ b/docs/user/cli-reference.md @@ -365,6 +365,14 @@ values govern advertisement, the #1327 allocation-policy paths are closure-locked in addition to the declared owned paths — see [GKE GPU setup](../integrator/gke-gpu-setup.md#gpu-device-plugin-ownership) and [Component Catalog › GKE Device-Plugin Ownership](component-catalog.md#gke-device-plugin-ownership). +The OKE family declares `gpuStack` with values `oci-default` (default; +Oracle's GPU node image supplies the driver and OKE's auto-installed device +plugin advertises — `advertiser: external`), `operator-plugin` (image driver, +GPU Operator's plugin advertises; OKE's plugin disabled via the +`oci.oraclecloud.com/disable-gpu-device-plugin=true` node label or add-on +removal), and `operator-managed` (bring-your-own driverless image; the +operator installs driver, toolkit, and plugin, with the DRA driver root in +lockstep) — see [OKE GPU setup](../integrator/oke-gpu-setup.md). Profiles can also be exercised through a versioned external overlay. Selection and verification are independent: `--profile` (or the default) @@ -464,7 +472,7 @@ Generate recipes using direct system parameters: | `--intent` | | string | Workload intent: training, inference | | `--os` | | string | OS family: ubuntu, rhel, cos, amazonlinux, ol, talos | | `--platform` | | string | Platform/framework type: dynamo, kubeflow, nim, runai, slurm | -| `--profile` | | string | Profile selection in exact `name=value` form (e.g. `gpuStack=operator-managed` on AKS or `gpuStack=bundle-installer` on GKE); omit to use the declaration's default (`gpuStack=azure-managed` on AKS, `gpuStack=gke-default` on GKE) | +| `--profile` | | string | Profile selection in exact `name=value` form (e.g. `gpuStack=operator-managed` on AKS/OKE or `gpuStack=bundle-installer` on GKE); omit to use the declaration's default (`gpuStack=azure-managed` on AKS, `gpuStack=gke-default` on GKE, `gpuStack=oci-default` on OKE) | | `--slurm-accounting-mode` | | string | Slurm accounting ownership: disabled (default), customer-managed, aicr-provided | | `--runtime-inventory` | | string | Runtime AI inventory (`k8s-aibom`) selection: `enabled`, `disabled`. Recorded in the generated recipe | | `--nodes` | | int | Number of GPU nodes in the cluster | diff --git a/docs/user/component-catalog.md b/docs/user/component-catalog.md index e62bce2a2..e236b1af5 100644 --- a/docs/user/component-catalog.md +++ b/docs/user/component-catalog.md @@ -150,15 +150,17 @@ The recipes now carry that value wherever it is needed ([#2181](https://github.c | AKS `gpuStack=operator-managed` | the operator's driver pod | `false` | the `gpuStack` profile | | GKE COS `gpuStack=gke-default` (default) | none the labeler can observe — the driver is finalized by an init container of GKE's kube-system DaemonSet | `true` | the `gpuStack` profile | | GKE COS `gpuStack=bundle-installer` | the bundle's `gcp-driver-installer` DaemonSet | `false` | the `gpuStack` profile | -| OKE | none — driver is in the node image | `true` | the overlay (OKE has no profile) | +| OKE `gpuStack=oci-default` (default) | none — driver is in the node image; OKE's auto plugin advertises | `true` | the `gpuStack` profile | +| OKE `gpuStack=operator-plugin` | none — driver is in the node image (OKE plugin disabled; operator plugin advertises) | `true` | the `gpuStack` profile | +| OKE `gpuStack=operator-managed` | the operator's driver pod | `false` | the `gpuStack` profile | | EKS | the operator's driver pod | unset (chart default `false`) | — | | Kind (nvkind) | none — driver is host-installed | `true` | the overlay (Kind has no profile) | The explicit `false` on the operator-managed variants is deliberate rather than redundant: it keeps the path profile-owned, so it cannot be flipped into an unsafe hybrid later. Do **not** assume a preinstalled driver where the GPU Operator installs one — skipping detection there would keep the label applied across an unloaded or unhealthy driver. -**NVSentinel is mandatory on the profiled families.** Because the AKS and GKE-COS `gpuStack` profiles name nvsentinel, its presence is profile-owned: `--set nv-sentinel:enabled=false` and a `bundlers=` list that omits it are both rejected on those platforms. That is intended — NVSentinel is a required component for these deployments. It remains optional on platforms with no `gpuStack` profile, such as OKE and EKS. +**NVSentinel is mandatory on the profiled families.** Because the AKS, GKE-COS, and OKE `gpuStack` profiles name nvsentinel, its presence is profile-owned: `--set nv-sentinel:enabled=false` and a `bundlers=` list that omits it are both rejected on those platforms. That is intended — NVSentinel is a required component for these deployments. It remains optional on platforms with no `gpuStack` profile, such as EKS. -Only AKS and GKE-COS get the install-time profile lock; OKE and Kind set the value at overlay level, so a bundle-time or declared-dynamic change is still rejected by the gate below, but a manual post-generation edit to the rendered Helm values is not. +AKS, GKE-COS, and OKE get the install-time profile lock; Kind sets the value at overlay level, so a bundle-time or declared-dynamic change is still rejected by the gate below, but a manual post-generation edit to the rendered Helm values is not. If you do need to set it yourself on an unlisted platform, it is an ordinary override: diff --git a/docs/user/container-images.md b/docs/user/container-images.md index 112201971..cf9055c90 100644 --- a/docs/user/container-images.md +++ b/docs/user/container-images.md @@ -20,7 +20,7 @@ A machine-readable **CycloneDX 1.6 JSON** companion to this page is produced by ## Summary - Components: **44** -- Unique images: **100** +- Unique images: **99** - Distinct registries: **11** Registries: `602401143452.dkr.ecr.us-west-2.amazonaws.com`, `cr.agentgateway.dev`, `docker.io`, `gcr.io`, `ghcr.io`, `gke.gcr.io`, `nvcr.io`, `public.ecr.aws`, `quay.io`, `registry.k8s.io`, `us-docker.pkg.dev` @@ -63,7 +63,7 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip | nfd-ocp | manifest | — | — | 0 | | nfd-ocp-olm | manifest | — | — | 0 | | nodewright-customizations | manifest | — | — | 5 | -| nodewright-operator | helm | nodewright | v0.17.1 | 3 | +| nodewright-operator | helm | nodewright | v0.18.0 | 2 | | nvidia-dra-driver-gpu | helm | dra-driver-nvidia-gpu | 0.4.1 | 1 | | nvidia-dra-driver-gpu-ocp | helm | dra-driver-nvidia-gpu | 0.4.1 | 1 | | nvsentinel | helm | nvsentinel | v1.20.0 | 6 | @@ -275,9 +275,8 @@ _No images extracted._ ### nodewright-operator -- `alpine/kubectl:1.36.2@sha256:01d138ce994b684abc62d9cfdff44de42a4c8996dcc12626dd0193afc3fb5a95` -- `ghcr.io/nvidia/nodewright/operator:v0.17.0@sha256:1511449bf51f2844b6bb3a03bde3d5590caf2ca283e3e39c0745a8016af2132f` -- `quay.io/brancz/kube-rbac-proxy:v0.15.0@sha256:2c7b120590cbe9f634f5099f2cbb91d0b668569023a81505ca124a5c437e7663` +- `docker.io/alpine/kubectl:1.36.2@sha256:01d138ce994b684abc62d9cfdff44de42a4c8996dcc12626dd0193afc3fb5a95` +- `ghcr.io/nvidia/nodewright/operator:v0.18.0@sha256:49ace92de35703f6da84c75147e2f93803d8b70f4f48dc9ae5213f7a6788ddfb` ### nvidia-dra-driver-gpu diff --git a/pkg/bundler/testdata/stock_render_golden.yaml b/pkg/bundler/testdata/stock_render_golden.yaml index d43d9f13f..d7380d12c 100644 --- a/pkg/bundler/testdata/stock_render_golden.yaml +++ b/pkg/bundler/testdata/stock_render_golden.yaml @@ -3,51 +3,51 @@ # # One entry per leaf overlay: a digest over its fully rendered helm-deployer # bundle tree (sorted relative paths paired with per-file content hashes). -a100-aks-ubuntu-training-kubeflow: 265940a7ef6f82bfe9752e5e488b416d5d54aa0245b29f0262796978547af68d -a100-any: 457dd24ba4f8b9d44af7f68ddc1f4e9d46d0e6eac30b0eceb2e979e1e9c2d2db -a100-eks-ubuntu-training-kubeflow: 9a77e69e7ea639be6100c4c2c0cb49bf87059348020c98f5c561158a22d452e7 -a100-gke-cos-training-kubeflow: 0fb904af67e4ad85cb1b71efed94c8cd3966fc36f14cc80d9b50d5ffe7babde5 -a100-oke-ubuntu-training-kubeflow: 92c1a420bccca84c7ca7e74d8960af6c3662bce4c708c45ccaa386f209dc1c9e -b200-any: b8e028ff78b7f142c3e157143a258510c4f44579e31f13c183796767de53d862 -b200-gke-cos-inference-dynamo: cd3c475c9bb709b8164571be18ec625b38263a762d91d16f89049e75fc934eb4 -b200-gke-cos-training-kubeflow: f35da200bec023578aaaeebe337775d9b62718cfd2ac0c568c4a56f69575fbb3 -bcm-inference: 2a7d525459b585c3b1639eb556de8feb682b27f6bb8be3a7015de7773ba7c235 -gb200-any: ff2c8128bc91fce4a48d650d160659b08272a346a044f7955d3b23109fde6e8d -gb200-eks-ubuntu-inference-dynamo: 637a4fd32933c220453d3732363c9a56bd75dd91cdd886a9b0571b666670744d -gb200-eks-ubuntu-training-kubeflow: e4d2b1cc86cf33b7741e881a984f6943353bf2e1b85d918b8d3e82cf50c8fa60 -gb200-eks-ubuntu-training-slurm: d356fec2562b9d3f8adcccb2344402f41270ca301f353affd6529aaa86e430d4 -gb200-oke-ubuntu-inference-dynamo: 0304fffcd27d04827ae561b32ced55b9ff87e0f6049bda30a075afd49f24ad11 -gb200-oke-ubuntu-training-kubeflow: 3ed8582775ca602b48eb88ff05d228db3f4189f4230f31f72cf355be8d5241b0 -gb300-any: a67ce96e16a0a22d85f22c2803f029e49b1befb4226a60cdf9820fb6b6e77ea2 -gb300-eks-ubuntu-inference-dynamo: c725636167448e64ef1d567613aded4bcd7ede0137a3d1e4b9c70be1ec20c61a -gb300-eks-ubuntu-training-kubeflow: e84019cb236bae67eef79a8166214295b066d1514ebb869619898925be48522d -h100-aks-ubuntu-inference-dynamo: 54061313326c300b83a0627f415002bd080c7c0e338e19bcce7996446a791065 -h100-aks-ubuntu-training-kubeflow: ea0180c91126bdda49e3641538cfef03961358ebe4d883e893d53a82b26d4bd7 -h100-aks-ubuntu-training-slurm: 7f9c80295d770456adca24b66bc7681e42fd9db54fcb97d2541c657c939645c9 -h100-any: 7e98352317f4d2a753255ec776e7b7bc605ad288f3856b4a467a15f8844aa903 -h100-bcm-ubuntu-training: c73ab59f00b5425a0be7c8aaac6107469c8fb59e54eecce9cf4937704821611d -h100-eks-ubuntu-inference-dynamo: 6a3e47afa2018591b22f2fd96105d75ec23b631fafb4c0f30f42ba77af6f537a -h100-eks-ubuntu-inference-nim: 3b9b4fe53c967fa95e56e2cf6e631a55f044d50f76423e11ac176d839e005d2d -h100-eks-ubuntu-training-kubeflow: 18be43e357c98753a069d5db99f2dbe79f6bbd7c98c91286715968f183639cb0 -h100-eks-ubuntu-training-slurm: fdf26391dd6d2e10b9e853f3bd480c8968f2749948cb58a4a1cf5a1b34b85325 -h100-gke-cos-inference-dynamo: a8527e38ea90f346eee045c6b1b448263c83258c3d99864f73b4e9f86d28d520 -h100-gke-cos-training-kubeflow: 304639b08cf1899fc5cb8bb7a5c1a4f9ff652896ae18a92123c802b2f4cfc341 -h100-gke-cos-training-slurm: 0efee10e13589930111b9f5064430d2dcac463d2d2cf6032a6a7687be2085958 -h100-kind-inference-dynamo: 8cea3d208001c5d7bb542d83a75e2ab641722981b279db9391a300b8f9d1f20c -h100-kind-training-kubeflow: 1203d2e475a51a3e07d3a62a87ad2dca91c24eda3aea800aca86bf2dfee5a80a -h100-kind-training-slurm: 8a27b1ae4e91cda80c4a32d456552952f2ec0a28af10e9e0317219b551ddd133 -h200-any: ceee05a6ed9dd0f218919a90187de851e16e93cf89293c9746d5f0e46bbda222 -h200-eks-inference: 2900b7d96bc48a27eb5fedda75b99b010b9165bc6058372d2de3b991f64c4982 -h200-eks-training: 4b3074954d10a81d057ac746c7bddf3664637dfd002cd0955348ec3dee11eb20 -l40s-any: dabbb23da635f2b6db7803be55841a7305f884ce7798dbf79d952686004f09d8 -l40s-oke-inference: 8a8884dcf64b9feaa708b152cc4eaf1b9c09d790de295cf2b0e5a5ca464154f1 -l40s-oke-training: f9b741b390f9f29451c7ad16da56602bbf6497bec067be6b19242cffdf1e1de7 -monitoring-hpa: 32917a470d982044c3d49960dbd5e62ab56cd870375bfac23c35d7a4b3f4298a +a100-aks-ubuntu-training-kubeflow: 553baccfd5525642e70d014a1a5ce431e9224194ef0643a01fe8b1cd8c7335f7 +a100-any: 24f4b2957aada8a9e82c9859ff6d20b658959f5dd2b71f896fef2bc3be96b1ae +a100-eks-ubuntu-training-kubeflow: 1118cc0eb8b4d5fc173a127d9bb2ad978d7bf3cb17cb4e9b49261289a2a895e9 +a100-gke-cos-training-kubeflow: e635f4cc761f3228f08ae0780d1ea079ff7140072a001a5e22c5b687011ecae6 +a100-oke-ubuntu-training-kubeflow: 28d940ae535057479eb74c725886236ba98aca089ad0d1453a9ffe990068d976 +b200-any: 4d52ad96e70c3fc20370551ae6ac27b0fe361a9190c24e7ff600a0d2359934fe +b200-gke-cos-inference-dynamo: a7773eacacaf9d840c936c084f95f9c66b3f5996ec3c65b7231c543fab7ff57a +b200-gke-cos-training-kubeflow: 3c23b03c4dc8d7a91423a19a5d5db26f2ca73f4333788bb5772c482b63f70ae2 +bcm-inference: 31980d30b41ca5e93641a10132de4a0696ea1b6b3c4c01e465f31643aec249cc +gb200-any: b142c472d4ab3cd7d089c948935e688a6b26cec8656ef2622fea827b4265f973 +gb200-eks-ubuntu-inference-dynamo: e6c0d3837b99ad66586170448dcb45a9918acd4ae24224c48af1a149b5ad54d8 +gb200-eks-ubuntu-training-kubeflow: ba426f4ef5cbf4694240672e03bd9e4ca8db611c905271019597945b09ed0d36 +gb200-eks-ubuntu-training-slurm: a9261f38f32d264191d271266c683d25bdd9a0a9e5ed874724a51a3b59e30675 +gb200-oke-ubuntu-inference-dynamo: 354f1904b1c09f9216f29786cbcdb2c4238b0b6e807244b6b6c8ddbc1087c900 +gb200-oke-ubuntu-training-kubeflow: fae79d77329020822102973f9ba01049077bb51ab44b043f2e5c0c80ba3ee248 +gb300-any: a707d3e822046e6cfbed77deecf18e7017dd827f026d23fa6128a8ae038eaf66 +gb300-eks-ubuntu-inference-dynamo: 98c6b918c3a8c2e8ca1f2254957a219296add77494b1a8613b31b9ab91b782da +gb300-eks-ubuntu-training-kubeflow: d8f9462fe98c6b1224e92fc16e3f3e1c8f35f433e659599ea0c252c9cd4f9a56 +h100-aks-ubuntu-inference-dynamo: 8e4cb2e8d58ba308b99a2439e05cdd31df55c8957a4a2dd4dab23b262186914e +h100-aks-ubuntu-training-kubeflow: 7987042d92f550cc5f44bd9b6f90fbd8567084ca5aaa7ebff2929616258aaf9f +h100-aks-ubuntu-training-slurm: 44e4011a9307486d863e7552d0b820dcfc5751b2802078396c050a40829b8c54 +h100-any: ce3e367d0dcdc7fa4b67fea7bc88d294eed86129274c4e8e42ca01911d828cc2 +h100-bcm-ubuntu-training: 04982936726dc56fa3c11fff7e0aa95907713591d8fe07948b328a6933cdd253 +h100-eks-ubuntu-inference-dynamo: 5003b7543b68b9156806d0e7ebf44cd936485b1b43e60bc5ec45b36078ca53f1 +h100-eks-ubuntu-inference-nim: 4535c2662c143107bf2bebdfda965ab1bcdb3f2ef818e43db94783066b077fca +h100-eks-ubuntu-training-kubeflow: 4101b4aeaf6b5b8c746fc6965ec5264309d5363b9d7f2f8469b4c09fa520165c +h100-eks-ubuntu-training-slurm: 512a9c23eaab3a4fc8c6a7ff0d76ce11c6015d7d3bbe546ced062a658bf8f4af +h100-gke-cos-inference-dynamo: a5476300fba818719e9ee5bcbe49ad81ed0576755791bc312349e1c579f9e925 +h100-gke-cos-training-kubeflow: 3179390b0855fc95a02f96fc417adef4ee333ca4bcb5235a6f19d157934cdd5c +h100-gke-cos-training-slurm: a8e55d200e185c14ff35063401409e4e047c6514ef82d438640ff436c8b49171 +h100-kind-inference-dynamo: 3eb0d6f1bf412a136deb307890ff90b3655b9b1f71c5894d3e505c467f4d7e9a +h100-kind-training-kubeflow: 91816b57d36ba3f663a83f42178dfff656f1cd22b5c5afc4605213259cd5540a +h100-kind-training-slurm: 2fb89f3da8ccf3cc50107161521345fcb2a3ac499488741efc6cdca5f89fc125 +h200-any: 1b98e3df862e5f9c4715a676b94b576ebfb57f7b5fe0f60e92c588caaccb24ab +h200-eks-inference: dfba31bc08abd37181aad00bf68cc9983b82f1abe9172700d2d184c48f1a4984 +h200-eks-training: 44bfaf2d63b2e21797095a5b2467294eb4f3ee6b148314a0a96d08054e06bc7b +l40s-any: ec32a4b689457a781f99fe75cad4182c20ee8b22c90fa665bce52ebdc6971213 +l40s-oke-inference: 15e05a20bee6cd7e4b760d26c480f87a706fc8c176cb0ab8a91fa95dbfafdd29 +l40s-oke-training: 4ee65b66ec8d0967094ab91b29efa12b2062cac25b9f2ca7e36a00ecb0639056 +monitoring-hpa: a0f8fbabea4a7cb67c0c57317ef17cd8bc18c1f801f84cd752bb14df8f7e8055 ocp-inference-nim: 016fc59c13e901de5556f09f3cb3d5a90f91b9826d34f126fde1e066b311605f ocp-training: 652f7ccdf52009adafe993da111857aa67390678f382b4c0caf0aecd0d613635 -rtx-pro-6000-any: 9115c5d61d9941130e448f29b9fbcd834f6d15a46fa4edf3239afc44fcdd36f0 -rtx-pro-6000-eks-ubuntu-inference-dynamo: 8c35c21289dc4a56f3e035da9d35427c9a1f2402c643febda2c6b397692b2a49 -rtx-pro-6000-eks-ubuntu-inference-nim: feb5f1d72ebd95459c9df0891420d20fac42c6b8c16f0e8fa551da61a19aef1d -rtx-pro-6000-eks-ubuntu-training-kubeflow: 1dfe905ac8d891fba4ce797de6532320024fe2616653e4703be5fc8e29716d77 -rtx-pro-6000-lke-ubuntu-inference: c5d53e0adbe9c500ecd5eb45d2482e632f1c414657a08ebef053a8aab90fc3cb -rtx-pro-6000-lke-ubuntu-training: 39e9cc1636a11d2568d9d50ecf3744f5773c4870772727661c40be63bac33280 +rtx-pro-6000-any: 25de9980a9c0470b1cb576364ca9db636d1830f01d6471f5e6e7e697ce9066c3 +rtx-pro-6000-eks-ubuntu-inference-dynamo: 8e7d63fd563ea806304fd965a459d0df735d0ba94f6adebf64edc91a91fa45d7 +rtx-pro-6000-eks-ubuntu-inference-nim: 79a561f2f4883c247a0dc6c97453c2c370eed3f549b3d6f4bc9a20ccd0138071 +rtx-pro-6000-eks-ubuntu-training-kubeflow: 364ffa48c09249d5492205a0040b8a8cc44197d701ec5f93cd1b34b65d441479 +rtx-pro-6000-lke-ubuntu-inference: 81163821ed76ed2a6ff10e587bb448815d0eec089282a50e283d77aaa5baaa27 +rtx-pro-6000-lke-ubuntu-training: c4b6bf452b3b95b1e06bf2bf6b4fbadb89d02df790cba78e671c1734ec2fabdf diff --git a/pkg/recipe/nvsentinel_gpustack_test.go b/pkg/recipe/nvsentinel_gpustack_test.go index 629d2e5b4..dc4e1410c 100644 --- a/pkg/recipe/nvsentinel_gpustack_test.go +++ b/pkg/recipe/nvsentinel_gpustack_test.go @@ -16,7 +16,6 @@ package recipe import ( "slices" - "sort" "strings" "testing" ) @@ -84,15 +83,38 @@ func TestNVSentinelConfigurationMatrix(t *testing.T) { wantAssume: ptr(false), }, { - name: "OKE: node image installs the driver (overlay-level, no profile)", + name: "OKE oci-default: image driver + OKE's plugin, no driver pod", criteria: &Criteria{ Service: CriteriaServiceOKE, Accelerator: CriteriaAcceleratorA100, OS: CriteriaOSOracleLinux, Intent: CriteriaIntentTraining, }, + profile: "gpuStack=oci-default", wantAssume: ptr(true), }, + { + name: "OKE operator-plugin: image driver, operator plugin, still no driver pod", + criteria: &Criteria{ + Service: CriteriaServiceOKE, + Accelerator: CriteriaAcceleratorA100, + OS: CriteriaOSOracleLinux, + Intent: CriteriaIntentTraining, + }, + profile: "gpuStack=operator-plugin", + wantAssume: ptr(true), + }, + { + name: "OKE operator-managed: the operator's driver pod is the evidence", + criteria: &Criteria{ + Service: CriteriaServiceOKE, + Accelerator: CriteriaAcceleratorA100, + OS: CriteriaOSOracleLinux, + Intent: CriteriaIntentTraining, + }, + profile: "gpuStack=operator-managed", + wantAssume: ptr(false), + }, { name: "Kind: nvkind host-installs the driver (overlay-level, no profile)", criteria: &Criteria{ @@ -359,20 +381,12 @@ func TestNVSentinelPresenceLockedOnProfiledFamilies(t *testing.T) { } } -// TestOKENVSentinelValueIsNotProfileOwned documents an asymmetry the docs -// must not paper over. -// -// OKE has no gpuStack profile, so its assumeDriverInstalled value lives on -// the overlay's componentRefs and is NOT covered by ADR-015's generated -// install-time profile lock — unlike the equivalent AKS and GKE-COS values. -// The NVSentinel driver-label bundle gate still rejects bundle-time and -// declared-dynamic changes that would recreate #2175, so the normal paths -// are protected; a manual post-generation edit to the rendered Helm values -// is outside that guarantee. -// -// Stating the asymmetry here keeps "profile-owned" from being read as a -// uniform property of every platform that sets the value. -func TestOKENVSentinelValueIsNotProfileOwned(t *testing.T) { +// TestOKENVSentinelValueIsProfileOwned pins the OKE family's conversion to +// the gpuStack profile: assumeDriverInstalled moved from the overlay's +// componentRefs into the profile values, so OKE now receives the same +// generated install-time profile lock as AKS and GKE-COS — the asymmetry the +// previous version of this test documented is gone. +func TestOKENVSentinelValueIsProfileOwned(t *testing.T) { t.Parallel() result, err := NewBuilder().BuildFromCriteriaWithProfile(t.Context(), &Criteria{ @@ -385,28 +399,27 @@ func TestOKENVSentinelValueIsNotProfileOwned(t *testing.T) { t.Fatalf("BuildFromCriteriaWithProfile() error = %v", err) } - // The value is present ... + // The value is present under the declared default ... values, err := result.GetValuesForComponentWithContext(t.Context(), nvsentinelComponent) if err != nil { t.Fatalf("GetValuesForComponentWithContext(nvsentinel): %v", err) } if assume, ok := nestedBool(values, "labeler", "assumeDriverInstalled"); !ok || !assume { - t.Fatalf("OKE labeler.assumeDriverInstalled = %v (set: %v), want true", assume, ok) + t.Fatalf("OKE labeler.assumeDriverInstalled = %v (set: %v), want true under oci-default", assume, ok) } - // ... and deliberately unprofiled. - if result.Metadata.SelectedProfile != nil { - t.Fatalf("OKE resolved with a profile (%#v); if a gpuStack profile is ever added to OKE, "+ - "move the value into it and update the docs that describe this asymmetry", - result.Metadata.SelectedProfile) + // ... and profile-owned. + selected := result.Metadata.SelectedProfile + if selected == nil || selected.Name != "gpuStack" || selected.Value != "oci-default" { + t.Fatalf("selectedProfile = %#v, want gpuStack=oci-default", selected) + } + owned := selected.OwnedPaths[nvsentinelComponent] + wantOwned := []string{"enabled", "labeler.assumeDriverInstalled"} + if len(owned) != len(wantOwned) || owned[0] != wantOwned[0] || owned[1] != wantOwned[1] { + t.Fatalf("ownedPaths[nvsentinel] = %v, want %v", owned, wantOwned) } - if len(result.EffectiveLockSet()) != 0 { - lockedComponents := make([]string, 0, len(result.EffectiveLockSet())) - for component := range result.EffectiveLockSet() { - lockedComponents = append(lockedComponents, component) - } - sort.Strings(lockedComponents) - t.Fatalf("OKE has a non-empty profile lock set %v, want none", lockedComponents) + if len(result.EffectiveLockSet()) == 0 { + t.Fatal("OKE profile lock set is empty, want the gpuStack-owned paths") } } diff --git a/pkg/recipe/profile_oke_test.go b/pkg/recipe/profile_oke_test.go new file mode 100644 index 000000000..fa1a27673 --- /dev/null +++ b/pkg/recipe/profile_oke_test.go @@ -0,0 +1,207 @@ +// Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package recipe + +import ( + "testing" + + "github.com/NVIDIA/aicr/pkg/allocpolicy" +) + +func okeCriteria() *Criteria { + return &Criteria{ + Service: CriteriaServiceOKE, + Accelerator: CriteriaAcceleratorL40S, + OS: CriteriaOSOracleLinux, + Intent: CriteriaIntentTraining, + } +} + +// TestOKEGpuStackProfileResolution pins the OKE family conversion: the +// oke-ol overlay declares gpuStack with default oci-default (the stock OKE +// cluster — Oracle image driver + OKE's auto-installed device plugin as the +// external advertiser) and alternatives operator-plugin (image driver, GPU +// Operator's plugin) and operator-managed (bring-your-own driverless image; +// the operator installs driver, toolkit, and plugin, and the DRA root moves +// in lockstep). Every value's DD-style distinguisher is readiness-scoped +// (deployed ClusterPolicy state), so this also pins the readiness routing. +func TestOKEGpuStackProfileResolution(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + selection string + wantValue string + wantAdvertiser string + wantDriver bool + wantToolkit bool + wantPlugin bool + wantDRARoot string + wantAssume bool + // wantGenDriverLoaded: the value carries the grounded generation + // constraint GPU.hardware.driver-loaded=true (image-supplied + // driver, durable across validate re-evaluation). + wantGenDriverLoaded bool + // wantOutcomeCheck: the value carries the readiness-scoped + // deployment-outcome check (driver-loaded=true post-deploy), + // routed into validation.readiness — never spec.constraints. + wantOutcomeCheck bool + }{ + { + name: "default selection is oci-default with the external advertiser", + selection: "", + wantValue: "oci-default", + wantAdvertiser: allocpolicy.AdvertiserExternal, + wantDriver: false, + wantToolkit: false, + wantPlugin: false, + wantDRARoot: "/", + wantAssume: true, + wantGenDriverLoaded: true, + }, + { + name: "operator-plugin keeps the image driver and advertises via the operator", + selection: "gpuStack=operator-plugin", + wantValue: "operator-plugin", + wantAdvertiser: "", + wantDriver: false, + wantToolkit: false, + wantPlugin: true, + wantDRARoot: "/", + wantAssume: true, + wantGenDriverLoaded: true, + }, + { + name: "operator-managed owns driver, toolkit, plugin, and the DRA root", + selection: "gpuStack=operator-managed", + wantValue: "operator-managed", + wantAdvertiser: "", + wantDriver: true, + wantToolkit: true, + wantPlugin: true, + wantDRARoot: "/run/nvidia/driver", + wantAssume: false, + wantOutcomeCheck: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + result, err := NewBuilder().BuildFromCriteriaWithProfile( + t.Context(), okeCriteria(), tt.selection) + if err != nil { + t.Fatalf("BuildFromCriteriaWithProfile() failed: %v", err) + } + selected := result.Metadata.SelectedProfile + if selected == nil { + t.Fatal("metadata.selectedProfile is nil") + return + } + if selected.Name != "gpuStack" || selected.Value != tt.wantValue { + t.Errorf("selectedProfile = %s=%s, want gpuStack=%s", selected.Name, selected.Value, tt.wantValue) + } + if selected.Advertiser != tt.wantAdvertiser { + t.Errorf("advertiser = %q, want %q", selected.Advertiser, tt.wantAdvertiser) + } + if result.APIVersion != RecipeProfileAPIVersion { + t.Errorf("apiVersion = %q, want %q", result.APIVersion, RecipeProfileAPIVersion) + } + + // Declaration-wide ownedPaths: identical for every selection. + wantOwned := map[string][]string{ + "gpu-operator": { + "devicePlugin.enabled", "driver.enabled", + "driver.useOpenKernelModules", "enabled", + "hostPaths.driverInstallDir", "toolkit.enabled", + }, + "nvidia-dra-driver-gpu": {"enabled", "nvidiaDriverRoot"}, + "nvsentinel": {"enabled", "labeler.assumeDriverInstalled"}, + } + for component, want := range wantOwned { + got := selected.OwnedPaths[component] + if len(got) != len(want) { + t.Errorf("ownedPaths[%s] = %v, want %v", component, got, want) + continue + } + for i := range want { + if got[i] != want[i] { + t.Errorf("ownedPaths[%s] = %v, want %v", component, got, want) + break + } + } + } + + gpuValues, err := result.GetValuesForComponentWithContext(t.Context(), "gpu-operator") + if err != nil { + t.Fatalf("GetValuesForComponentWithContext(gpu-operator): %v", err) + } + if v, ok := nestedBool(gpuValues, "driver", "enabled"); !ok || v != tt.wantDriver { + t.Errorf("driver.enabled = %v (set: %v), want %v", v, ok, tt.wantDriver) + } + if v, ok := nestedBool(gpuValues, "toolkit", "enabled"); !ok || v != tt.wantToolkit { + t.Errorf("toolkit.enabled = %v (set: %v), want %v", v, ok, tt.wantToolkit) + } + if v, ok := nestedBool(gpuValues, "devicePlugin", "enabled"); !ok || v != tt.wantPlugin { + t.Errorf("devicePlugin.enabled = %v (set: %v), want %v", v, ok, tt.wantPlugin) + } + + draValues, err := result.GetValuesForComponentWithContext(t.Context(), "nvidia-dra-driver-gpu") + if err != nil { + t.Fatalf("GetValuesForComponentWithContext(nvidia-dra-driver-gpu): %v", err) + } + if root, _ := draValues["nvidiaDriverRoot"].(string); root != tt.wantDRARoot { + t.Errorf("nvidiaDriverRoot = %q, want %q", root, tt.wantDRARoot) + } + + nvsValues, err := result.GetValuesForComponentWithContext(t.Context(), nvsentinelComponent) + if err != nil { + t.Fatalf("GetValuesForComponentWithContext(nvsentinel): %v", err) + } + if v, ok := nestedBool(nvsValues, "labeler", "assumeDriverInstalled"); !ok || v != tt.wantAssume { + t.Errorf("assumeDriverInstalled = %v (set: %v), want %v", v, ok, tt.wantAssume) + } + + // Driver-axis constraint placement (#2363: only the driver axis + // is groundable today; the advertiser axis is deferred). + const driverLoaded = "GPU.hardware.driver-loaded" + genHas := false + for _, c := range result.Constraints { + if c.Name == driverLoaded && c.Value == "true" { + genHas = true + } + } + if genHas != tt.wantGenDriverLoaded { + t.Errorf("generation constraint %s present = %v, want %v", driverLoaded, genHas, tt.wantGenDriverLoaded) + } + readinessHas := false + if result.Validation != nil && result.Validation.Readiness != nil { + for _, c := range result.Validation.Readiness.Constraints { + if c.Name == driverLoaded && c.Value == "true" { + readinessHas = true + } + } + } + if readinessHas != tt.wantOutcomeCheck { + t.Errorf("readiness outcome check %s present = %v, want %v", driverLoaded, readinessHas, tt.wantOutcomeCheck) + } + if tt.wantOutcomeCheck && genHas { + // The self-falsifying-precondition trap: operator-managed + // must never gate generation on driver state its own + // deployment changes. + t.Errorf("operator-managed carries a generation driver gate — self-falsifying") + } + }) + } +} diff --git a/pkg/recipe/testdata/catalog_parity_golden.yaml b/pkg/recipe/testdata/catalog_parity_golden.yaml index 95fe07389..f3569018f 100644 --- a/pkg/recipe/testdata/catalog_parity_golden.yaml +++ b/pkg/recipe/testdata/catalog_parity_golden.yaml @@ -3,51 +3,51 @@ # # One entry per leaf overlay: sha256 of its deterministically-marshalled # resolved recipe. A moved digest means that recipe's resolved bytes changed. -a100-aks-ubuntu-training-kubeflow: 3d8d77ac7ac29bf13253410e0b0486bc322d69d66d8778884df3375619566d58 -a100-any: 62d1581b21fecd3465b69659bac3dc9e5438eb2612c98c1acdd7b19cc9237084 -a100-eks-ubuntu-training-kubeflow: 0196adfa0c77f230d95ed3ebee2bdade8ac720f631cd3bbe2b8cec7d55094a10 -a100-gke-cos-training-kubeflow: 514f996360b251a6ef657ba1a715b5705be03d6b6dccb53887ee96554a83ffb6 -a100-oke-ubuntu-training-kubeflow: 310a6ee2abdd7f8508882429d657c01bd23458ba635236c410934825e4f617dc -b200-any: 63dbc4fe27b84321e8b1991cd98c13f85395a3919d13e0c6891599eaa1f1e729 -b200-gke-cos-inference-dynamo: 8174db8270ea447d9b307f21be9945b919ed3bcdacc702d2961d2ff75fb60c35 -b200-gke-cos-training-kubeflow: 96723738cb0baaaec300c95d5cfbc08a18d999fd0adb52aef3c88436561da1e7 -bcm-inference: 92fa64d3af61026891b37cbf692d2663be7b2484078ab65e2653d18fc2bfe6f8 -gb200-any: 9ee6ab187b4c1b93cd859d86ada35e922d0873ffc6fe892cb084ad98baf09727 -gb200-eks-ubuntu-inference-dynamo: 8babae6413d104da16060f18d919c80fd97ff983bd32f9876bd5f72da7fd6be6 -gb200-eks-ubuntu-training-kubeflow: 63b2fbd3a195bac682bb45263e5d5d0ec479a6571dd38ef0505eeb4f287b29e9 -gb200-eks-ubuntu-training-slurm: c35e33618c0a8bb548bbe3c5d6765faf90cebe1427478ef97089f7a30bb37a6c -gb200-oke-ubuntu-inference-dynamo: 6f01848824dcbefb0f58ec56ee9e911fff1470c9f67c0d0a9ba0de1dd1e648d2 -gb200-oke-ubuntu-training-kubeflow: 3959f47474edfac9b53a65939117d4364551fde0c9324246f195f144fe3beb12 -gb300-any: 7f68607dffbbfc912f19b00ffa833caa81ca675d25bc22cf612dbd0b76f39475 -gb300-eks-ubuntu-inference-dynamo: b14a3d99503949b8bfcdc81455da4af389a339deb24ea784637a7901297d70ad -gb300-eks-ubuntu-training-kubeflow: 5303e5ebf8dc2ec13c2cff60c7caba2ef53d1a65bfd69d06f4f78cbb93835c49 -h100-aks-ubuntu-inference-dynamo: 9fe4d82412d250d4004efc56fa4ddd6b8b1a9166387455e96ac2c2284d323107 -h100-aks-ubuntu-training-kubeflow: 72a8ca777c333e0db82717f53a706ae9155163ff0c73fd4e5ed8d71badda5811 -h100-aks-ubuntu-training-slurm: ef48d77171817bb5b24815d128457c246e0e5dc0fe850aa6881276952e590055 -h100-any: f5a55f03948075adde9b1b24fd650d98fa4d35b85f394d071ad3365ba05588a5 -h100-bcm-ubuntu-training: 76746f45add1fb6b0d7b6441b4d9581acceb3b5d3d4800b943c1cf460c92fbc5 -h100-eks-ubuntu-inference-dynamo: 0a31f03010849f76b385ce81c21e499af884f41bee89de47b9bacf03f8b6e311 -h100-eks-ubuntu-inference-nim: 9c6a4839bf5b59f620093427de0e138a9fafaf1f55f3fa7f555b1ee33b914a3a -h100-eks-ubuntu-training-kubeflow: bfcbd5e14fe43209f2401e429d39b5f7a52d049832db07e5b9595d130ab6e2fc -h100-eks-ubuntu-training-slurm: 8d29c2be5c3c0dc8396b0df53111a485f6d57771394df8ef135e4b052d681064 -h100-gke-cos-inference-dynamo: 45190af6c5b5d76f69a4f36d9c0cb800c450bdef54ca5b522ad49f605d1bad2a -h100-gke-cos-training-kubeflow: 43de2291cc0adb9b8d89494179d86eeef35994b2ad4fd4062a87265d6753ec6b -h100-gke-cos-training-slurm: 414f95e1a231b1c86bcbe70729b820a95533367dec0c9e3f7e36514431fad1d9 -h100-kind-inference-dynamo: 0e2d552b62b55b91f960c9721ed128b1d07f68be8517799473fbaa3baa1cc6e5 -h100-kind-training-kubeflow: 4bb03659faaa6a9a66d58dfc95572bdb54b75010c219c4bbefbd0eafeb8ed867 -h100-kind-training-slurm: 0245c786638ade1023ba4dc49e0f2efd56d2f789bdf447e946bbc3a3e652e734 -h200-any: acf1986d76037dddd025125eaa34eb7ecf3d31fe62cb80e7dc89dd042ebb42f5 -h200-eks-inference: d1ff148001722e9d5e23606bd7770f8120506f08c10f0c132e0c379f535609c5 -h200-eks-training: 98d04b3c81a059a0321451a39772af8d0d3310ad1fb861837f87d82dd997c4a1 -l40s-any: 89210ba5815f93ce3d7b1a2a78f7ba75d083556e940603a1bc812eac9aeccd0a -l40s-oke-inference: 6c75ef93e86fb21dbdf73ac97089b894ca8da832164dc807e94592493ab7f7cc -l40s-oke-training: 2a35011bd59cf3c727cea158d504c51bd59cba069620d3d6a4988d68972dee74 -monitoring-hpa: 15e93304e7e68997e7b15a0be4af360a29f10c869502e6dccb6a6cefb6519fd5 -ocp-inference-nim: 49245ecfc91f67fcc2ed0b305b00046a4164628cafc75ab98542e6dafa626821 -ocp-training: aaa8cd7f45ef42235b075936461d715e2e017ea2802df9e8017de38304a17915 -rtx-pro-6000-any: 9639d67639256a44be8566f3f8eda49ceb4025b6b140a3bc2a4b6d4768d07ffa -rtx-pro-6000-eks-ubuntu-inference-dynamo: cbe11051d979ae20e83471142c4582e1504fd24f55898a00afcededbe143bba6 -rtx-pro-6000-eks-ubuntu-inference-nim: 08b73d58d0d60681be55ee1402bdec5bf1367a0cb137a273ca27bec0788cc020 -rtx-pro-6000-eks-ubuntu-training-kubeflow: c2595e2994d56a735714103eedd30c6afd45921f458bf026c3432017b5eba60e -rtx-pro-6000-lke-ubuntu-inference: 28cdab0531f8ed6018391aba7cab1fbfad8ba551392de1b9138f3b88042f5340 -rtx-pro-6000-lke-ubuntu-training: 312c4cf49d0e8cec0377cd9858338d7106226a2252d81828a75c1d426a194a2f +a100-aks-ubuntu-training-kubeflow: 99ba4c7a1be4e4d30d8acd0fa4350557d4b535d5878f7cba682d7641c65a9e13 +a100-any: d16eb5c003d5c735deff2ea1285526b98d1d9c7a20e7cd5405c5679c6b67e793 +a100-eks-ubuntu-training-kubeflow: 59bf5f14afe030801abbf5627215ab0bb3a09117f833f77a2e970b1e9ae26ce4 +a100-gke-cos-training-kubeflow: e967da677f206e7c622d45591b3e36aa3b6b8ba0b6616d1ca20890c28d619fc5 +a100-oke-ubuntu-training-kubeflow: a1b49ff4c93644c4243a73ed6da64a803518011b8be9eb5f5712ea5ac43cc427 +b200-any: 45ea61bdb679127e140bd41b30fb01fdd91e64936b9ebfeb40c64a085cc59a1f +b200-gke-cos-inference-dynamo: d433152fa0235aff02565dc6007ef9ee5d56bb28b8a6e7dd1df55b174ab7a165 +b200-gke-cos-training-kubeflow: 8e3195e401ace43b650a5edea4625dba0aa3b04244e8bf90371ef79135cb87a5 +bcm-inference: 532d0caef0c570ea7483cac0ec1f31848cdf87175cb191a81a95c4159a8ce45d +gb200-any: a2ee1f87f28e8fd3191c5eb676e98ada72228dd2a88d6dc9bdf96a6ae55757d1 +gb200-eks-ubuntu-inference-dynamo: 540292f9c9c4c0a59bfd8ab7cd03738d32a96fe380babd72d6e593e5b27e6c1e +gb200-eks-ubuntu-training-kubeflow: bf2be525e4bf0ec8fa98988f6d01939dac21cabe06cf2526cd42488b9961ee2d +gb200-eks-ubuntu-training-slurm: 2ed987ee1adf483fa04ceebf6ac6f5433db69eb65a138085744e3a4f11efd68e +gb200-oke-ubuntu-inference-dynamo: 55b2a4514e5f774d819d02c2370254152ba1febc59947e6ec74d89a7c07d59d1 +gb200-oke-ubuntu-training-kubeflow: 29c49c4ec3bc40c711b40fae53d7be9a0261ff1ca7ad5b971878192bc5a0085d +gb300-any: 86219d16100b8708cf7fa8f24d7ca5992afd8677be5195e5f2637592ff00be31 +gb300-eks-ubuntu-inference-dynamo: c30b877b0670396b1f6182e0fe7e491ccd1ea4b4a55379c53f9e22d51adcf40b +gb300-eks-ubuntu-training-kubeflow: f80319d540d3f814edf72a896a90c02e1f6eca85590d224e2ddb84a92d9b50ce +h100-aks-ubuntu-inference-dynamo: 70e2334461cb1daa68d5936ce825a299db1313865d37e940b75ef4301c8d8f33 +h100-aks-ubuntu-training-kubeflow: d6318b715adb3b6d5eac6af84dcb81e7dd499810e4609b03e14c4d2c78bee9bb +h100-aks-ubuntu-training-slurm: fd7a7bdd61cd224bdcc74ae1b12d0f9fe6512fbed283950b2c6223d814a36596 +h100-any: 79808d6bec1caf50d87e91d48a2529d956b6345157983853c8a81722e6dc9f61 +h100-bcm-ubuntu-training: 5fc737cec4bdb9050be7f123eea1c891367917fe519f26e25adbe618830902c9 +h100-eks-ubuntu-inference-dynamo: 5867a33041edb58a61ac2f03614df47073fce2abf28a1e0038427eff222514c9 +h100-eks-ubuntu-inference-nim: 85152631df6e2d899131cd27f37a809187684017a73d0239f7aff538d9a2a6e2 +h100-eks-ubuntu-training-kubeflow: 709e67b84c68d6c28ff39ddfd5903d73d2e261502dc2b7e62b851a00e13b6a57 +h100-eks-ubuntu-training-slurm: 4b4003b5f6bbe7ed703964d841e66d84173af3362bf689d09a1fef78ac47f767 +h100-gke-cos-inference-dynamo: a6eba0e9c2cfbe2c75363061b6e382eabf250bf155b14f1eada8f09b3167a287 +h100-gke-cos-training-kubeflow: 89fa136b412f48674676c3e429d773d8c28154a488bfb489df2ea7a4750c01fa +h100-gke-cos-training-slurm: 5e8e9fa3ccf53577452738b000c8601695e7d708c2428e18a11eea3e8351b37c +h100-kind-inference-dynamo: 41ef03a2b379b2bd4ad0e860e499463d29b003aa947d0eeff3600ee1da36ec55 +h100-kind-training-kubeflow: 34c7d5342fc7792f28859fa5c2d07b79eb8e57999fffc98ce232ce2cd96413a0 +h100-kind-training-slurm: d427f1a59ce4e45cc127cdb050acf2473cf228b7209102a84256920a1287a45d +h200-any: bc580ca6d8efad198c9901d29d306ff99c01bae95e4fae2af0c84d2a4471983d +h200-eks-inference: e99c57d64be024d5180e70c58be17f4d3dfc08ac3b730840d924de807783a72f +h200-eks-training: 61d684ba87b90e60dd27bfba231c811f020e4577127df1ca30505920c1efc37b +l40s-any: 0210e041b3ffc84e6b7414ba40e5156496ccd61253483074fbc27a9e09957ba0 +l40s-oke-inference: b4f3de89404d53779a87d24cd601775487f39124d4e84d8139f9a91e7d80dcd9 +l40s-oke-training: 2c7bdcb28b68f12075f05104ee2d77919f7492b79f67432fe0de90fbbb8b995e +monitoring-hpa: 6ddb7b4bc162aa46e2b4e565853c83ca4e9b89cf49ebcc5fa66e6f1bc7b66502 +ocp-inference-nim: 85d8abf6b1c516fbfbc6285367ef55f0a4c25b1d0b93e99d94ca1e146f9be46e +ocp-training: 335a9679e6de0b0e643deb55f34507220b378f3965a119fdb72d9d0c9adad147 +rtx-pro-6000-any: 4ede34331ac02ba2f6bc559a2c4a34a97d4644de722d01297e399da6ee410d62 +rtx-pro-6000-eks-ubuntu-inference-dynamo: 7b65cc4c20c24cf8e24d0c250cf0846a61cf2759c099235d86b9c94d1fba9d79 +rtx-pro-6000-eks-ubuntu-inference-nim: 57679988eecb06e5b057d05b65ccac208eb73b0ea8ccc7ed8552b6cf0f9a56f7 +rtx-pro-6000-eks-ubuntu-training-kubeflow: b50a27d111971f5dc2f720fd052c38752d2ff09b53ea3072ab2e722171a8df84 +rtx-pro-6000-lke-ubuntu-inference: cf3a434bc85ec6e5f7d4d4e37bd9463f1fcb00bcbbf63744a303ca54b980c1cb +rtx-pro-6000-lke-ubuntu-training: a71487a1cc18f5aa3123e6f00179fb7386d51f0917b1ea4099f0125754daa7d0 diff --git a/recipes/components/gpu-operator/values-oke.yaml b/recipes/components/gpu-operator/values-oke.yaml index 3a0c3ed7e..9f1e9ff22 100644 --- a/recipes/components/gpu-operator/values-oke.yaml +++ b/recipes/components/gpu-operator/values-oke.yaml @@ -15,18 +15,14 @@ # GPU Operator Helm values # OKE base configuration overrides # -# OKE pre-installs NVIDIA drivers and container toolkit on GPU bare-metal -# shapes (BM.GPU.B200, BM.GPU.H100, etc.). Disable both to avoid conflicts. -# To use GPU Operator-managed drivers instead, provision GPU nodes without -# the Oracle-managed GPU driver and set driver.enabled=true, -# toolkit.enabled=true, AND dradriver:nvidiaDriverRoot=/run/nvidia/driver — -# the Oracle GPU image bundles the driver and the container toolkit together, -# so a node provisioned without it has neither. The DRA root must move too: -# the OKE DRA base pins nvidiaDriverRoot: / for the host-installed driver, but -# once gpu-operator manages the driver, Rule 2 of the bundle-time -# driver-ownership coherence check requires nvidiaDriverRoot to equal -# hostPaths.driverInstallDir (/run/nvidia/driver). The same check also rejects -# a driver-only flip on a cluster whose snapshot recorded no NVIDIA driver. +# Driver, toolkit, and device-plugin ownership is selected by the gpuStack +# configuration profile declared on recipes/overlays/oke-ol.yaml (ADR-015): +# oci-default (Oracle image driver + OKE's auto device plugin), +# operator-plugin (image driver + operator plugin), and operator-managed +# (bring-your-own driverless image; the operator installs all three and the +# DRA root moves to /run/nvidia/driver in lockstep). Those paths are +# profile-owned and locked — select the mode at recipe generation +# (--profile gpuStack=), never with bundle-time --set overrides. # hostPaths.driverInstallDir is intentionally left at the gpu-operator base # default /run/nvidia/driver: it is the host path the operator-validator @@ -41,10 +37,24 @@ cdi: enabled: true default: false -# OKE pre-installs NVIDIA drivers on GPU bare-metal shapes -driver: - enabled: false +# driver.enabled / toolkit.enabled / devicePlugin.enabled are profile-owned +# (see header) and deliberately absent here — the selected gpuStack value's +# fragment is authoritative. -# OKE pre-installs NVIDIA container toolkit -toolkit: - enabled: false +# Universal OKE bare-metal device-plugin env (independent of the gpuStack +# value; consumed only when the operator's plugin runs): +# - DEVICE_LIST_STRATEGY lists cdi-annotations FIRST: with cri-o + CDI on +# OKE bare metal, CDI device IDs must flow via pod annotations — the +# cdi-cri-first default routes them differently and breaks NCCL on +# SR-IOV clusters. +# - MOFED_ENABLED=false: OKE nodes carry host MOFED, and k8s-device-plugin +# >= 0.19.0 defaults MOFED_ENABLED=true when CDI is enabled, flooding +# every host RDMA uverb into every GPU pod and breaking NCCL. +# Helm REPLACES lists: an overlay overriding devicePlugin.env must restate +# these entries. +devicePlugin: + env: + - name: DEVICE_LIST_STRATEGY + value: cdi-annotations,cdi-cri,volume-mounts + - name: MOFED_ENABLED + value: "false" diff --git a/recipes/components/nvidia-dra-driver-gpu/values-oke.yaml b/recipes/components/nvidia-dra-driver-gpu/values-oke.yaml index d03581e7f..3ab8f37a6 100644 --- a/recipes/components/nvidia-dra-driver-gpu/values-oke.yaml +++ b/recipes/components/nvidia-dra-driver-gpu/values-oke.yaml @@ -15,8 +15,9 @@ # NVIDIA DRA Driver GPU Helm values # OKE configuration overrides # -# OKE pre-installs NVIDIA drivers on GPU bare-metal shapes, so the driver -# root is `/` (host-installed) rather than `/run/nvidia/driver` (containerized). +# The driver root follows the selected gpuStack value (see oke-ol.yaml): +# `/` where the image supplies the driver, `/run/nvidia/driver` where the +# GPU Operator installs it. # Reference: https://github.com/oracle-quickstart/oci-hpc-oke/blob/grace-blackwell/manifests/dra/values.yaml # # GPU allocation defaults are inherited from values.yaml (this file is merged @@ -29,8 +30,9 @@ nameOverride: nvidia-dra-driver-gpu fullnameOverride: nvidia-dra-driver-gpu -# Host-installed driver (OKE pre-installs NVIDIA drivers) -nvidiaDriverRoot: / +# nvidiaDriverRoot is profile-owned (gpuStack on oke-ol.yaml): / for the +# image-supplied driver values, /run/nvidia/driver under operator-managed +# (TestDriverRootLockstep, issue #1087). Deliberately absent here. controller: priorityClassName: "" diff --git a/recipes/overlays/oke-ol.yaml b/recipes/overlays/oke-ol.yaml index 57f4687d9..4c681b25f 100644 --- a/recipes/overlays/oke-ol.yaml +++ b/recipes/overlays/oke-ol.yaml @@ -13,7 +13,7 @@ # limitations under the License. kind: RecipeMetadata -apiVersion: aicr.run/v1alpha2 +apiVersion: aicr.run/v1alpha3 metadata: name: oke-ol @@ -54,23 +54,14 @@ spec: resources: requests: storage: 50Gi - # NVSentinel on OKE: the node image supplies the driver, so no driver pod - # exists for the labeler to watch. Without assumeDriverInstalled the - # nvsentinel.dgxc.nvidia.com/driver.installed label is never applied and - # metadata-collector plus both syslog-health-monitor DaemonSets sit at zero - # desired pods — silently, with no error and no event (#2175). - # - # OKE has no gpuStack profile, so this value is overlay-level and is NOT - # profile-owned: it does not receive the generated install-time profile - # lock that protects the equivalent AKS and GKE-COS values. The NVSentinel - # driver-label bundle gate still rejects bundle-time and declared-dynamic - # changes that would recreate #2175, but a manual post-generation edit to - # the rendered Helm values is outside that guarantee. + # NVSentinel's labeler.assumeDriverInstalled is owned by the gpuStack + # profile below (#2175): whether a driver pod exists for the labeler to + # watch depends on the selected value, so the polarity is per value — + # true where the driver ships in the node image (oci-default, + # operator-plugin), false where the operator's driver pod is the + # evidence (operator-managed). No overlay-level value here. - name: nvsentinel type: Helm - overrides: - labeler: - assumeDriverInstalled: true # OKE DRA driver: host-installed driver root, compute domains enabled - name: nvidia-dra-driver-gpu @@ -81,6 +72,171 @@ spec: affinity: nodeAffinity: null + # GPU stack shape for OKE GPU node pools (ADR-015). OKE auto-installs + # NVIDIA's device plugin on every cluster (per-node opt-out is the + # oci.oraclecloud.com/disable-gpu-device-plugin=true node label; removing + # the NvidiaGpuPlugin cluster add-on is the API-level equivalent), and + # Oracle's GPU node images preinstall the driver, container toolkit, and + # host MOFED. Which of those a pool actually has depends on how it was + # provisioned — the three values below name the qualified combinations. + # + # Qualification (issue #2363): only the DRIVER axis is grounded today — + # image-driver values require GPU.hardware.driver-loaded=true at + # generation (durable: the image driver persists, so the validate + # pre-flight's re-evaluation holds), and operator-managed carries its + # post-deployment form as a readiness-scoped deployment-outcome check. + # The ADVERTISER axis (OKE's plugin vs the operator's) has no evaluable + # external signal yet: the disable-gpu-device-plugin node label is not in + # the gpu-nodes evaluator's universe (GKE-only today), and DGXC removes + # the add-on without setting the label. Until #2363 selects a signal, + # oci-default vs operator-plugin is an explicit --profile choice. + # + # Every leaf OKE overlay (OL and Ubuntu) inherits this declaration + # unchanged. + profile: + name: gpuStack + description: >- + Who installs the GPU driver and which device plugin advertises + nvidia.com/gpu on OKE GPU node pools. "oci-default" (the default) is + the stock OKE cluster: Oracle's GPU node image supplies the driver and + toolkit, and OKE's auto-installed device plugin advertises — the GPU + Operator installs none of the three. "operator-plugin" keeps the + image-supplied driver and toolkit but the OKE device plugin is + disabled (node label or add-on removal), so the GPU Operator's plugin + advertises. "operator-managed" is a bring-your-own driverless image: + the GPU Operator installs driver, toolkit, and device plugin, and the + DRA driver reads the operator install path. MOFED is host-supplied in + every value (Oracle and BYO images alike) — network-operator never + deploys ofedDriver on OKE. + default: oci-default + values: + # Stock OKE: Oracle GPU image + OKE's auto-installed device plugin. + # The external plugin is THE nvidia.com/gpu advertiser in the #1327 + # exactly-one invariant; the operator's plugin must be off or the two + # double-advertise. + oci-default: + advertiser: external + componentRefs: + - name: gpu-operator + overrides: + driver: + enabled: false + useOpenKernelModules: false + toolkit: + enabled: false + devicePlugin: + enabled: false + hostPaths: + driverInstallDir: /run/nvidia/driver + - name: nvidia-dra-driver-gpu + overrides: + nvidiaDriverRoot: / + - name: nvsentinel + overrides: + labeler: {assumeDriverInstalled: true} + # Driver axis, grounded: Oracle GPU node images ship the driver, so + # a loaded kernel module on the sampled node is a durable + # pre-existing fact — it holds again when the validate pre-flight + # re-evaluates this constraint post-deployment (no + # self-falsification; the image driver persists). Single-node + # sampling caveat: mixed pools are out of scope (#464). + # + # Advertiser axis (OKE's plugin vs the operator's) is deliberately + # UNQUALIFIED pending #2363: the candidate signals — the + # oci.oraclecloud.com/disable-gpu-device-plugin node label vs a + # cluster add-on projection — are undecided, and the gpu-nodes + # label evaluator's universe is GKE-only today. Until then, + # oci-default vs operator-plugin is an explicit --profile choice. + constraints: + - name: GPU.hardware.driver-loaded + value: "true" + remediation: >- + The sampled GPU node has no NVIDIA kernel driver loaded, but + the oci-default profile assumes Oracle GPU node images with + the driver preinstalled. For bring-your-own driverless + images, regenerate with --profile gpuStack=operator-managed. + # Oracle GPU image driver + the GPU Operator's device plugin: pools + # where OKE's auto plugin is disabled (per-node label + # oci.oraclecloud.com/disable-gpu-device-plugin=true, or the + # NvidiaGpuPlugin add-on removed at the cluster). + operator-plugin: + componentRefs: + - name: gpu-operator + overrides: + driver: + enabled: false + useOpenKernelModules: false + toolkit: + enabled: false + devicePlugin: + enabled: true + hostPaths: + driverInstallDir: /run/nvidia/driver + - name: nvidia-dra-driver-gpu + overrides: + nvidiaDriverRoot: / + - name: nvsentinel + overrides: + labeler: {assumeDriverInstalled: true} + # Same grounded driver-axis fact as oci-default (image-supplied + # driver, durable across validate re-evaluation); the advertiser + # axis that distinguishes this value from oci-default is deferred + # to #2363 — see the comment on oci-default. + constraints: + - name: GPU.hardware.driver-loaded + value: "true" + remediation: >- + The sampled GPU node has no NVIDIA kernel driver loaded, but + the operator-plugin profile assumes Oracle GPU node images + with the driver preinstalled. For bring-your-own driverless + images, regenerate with --profile gpuStack=operator-managed. + # Bring-your-own driverless image: the GPU Operator installs driver, + # toolkit, and device plugin; the DRA kubelet plugin reads the driver + # userspace from the operator install path (TestDriverRootLockstep, + # issue #1087). + operator-managed: + componentRefs: + - name: gpu-operator + overrides: + driver: + enabled: true + # AOR-aligned: proprietary kernel modules on OKE. + useOpenKernelModules: false + toolkit: + enabled: true + devicePlugin: + enabled: true + hostPaths: + driverInstallDir: /run/nvidia/driver + - name: nvidia-dra-driver-gpu + overrides: + nvidiaDriverRoot: /run/nvidia/driver + # The operator's driver pod is the labeler's evidence — assuming a + # preinstalled driver here would label GPU nodes before the driver + # is actually ready. + - name: nvsentinel + overrides: + labeler: {assumeDriverInstalled: false} + # No generation-time driver gate: "no driver loaded" is the + # self-falsifying pre-condition the ADR forbids — the operator's + # own install erases it, so it would fail every post-deployment + # validate on a healthy cluster. Its post-deployment form lives + # below as a readiness-scoped DEPLOYMENT-OUTCOME CHECK per the + # ADR-015 contract: it verifies the operator's driver install + # actually executed (unlike a rendered .spec readback it can + # fail), but it does NOT qualify the value — every value's own + # success satisfies it. The advertiser/ownership qualification + # for this family is tracked in #2363. + readinessConstraints: + - name: GPU.hardware.driver-loaded + value: "true" + remediation: >- + The operator-managed profile deploys the GPU Operator as the + driver owner, but post-deployment the sampled GPU node still + has no NVIDIA kernel driver loaded — the operator's driver + install has not completed (or failed). Inspect the + nvidia-driver DaemonSet in the gpu-operator namespace. + validation: conformance: checks: diff --git a/recipes/registry.yaml b/recipes/registry.yaml index abbf297a2..d37f32c9f 100644 --- a/recipes/registry.yaml +++ b/recipes/registry.yaml @@ -261,7 +261,7 @@ components: helm: defaultRepository: oci://ghcr.io/nvidia/nodewright/charts defaultChart: nodewright - defaultVersion: v0.17.1 + defaultVersion: v0.18.0 defaultNamespace: skyhook nodeScheduling: nodeCountPaths: @@ -354,15 +354,16 @@ components: # nvsentinel.dgxc.nvidia.com/driver.installed by watching for a GPU # driver pod. Where the driver ships in the node image (or the host) # and no driver pod exists (AKS gpuStack=azure-managed, GKE COS - # gpuStack=gke-default, OKE, Kind), the label is never applied and + # gpuStack=gke-default, OKE gpuStack=oci-default/operator-plugin, + # Kind), the label is never applied and # metadata-collector plus both syslog-health-monitor DaemonSets sit # at 0 desired pods — with no error and no event, while # gpu-health-monitor stays healthy on the DCGM label. The chart # automates the remedy via labeler.assumeDriverInstalled # (--assume-driver-installed, NVSentinel design 018's Manual # Labeling Procedure). Every one of those recipes now assigns it - # itself — from the gpuStack profile on AKS and GKE-COS, at overlay - # level on OKE and Kind (#2181) — so this gate is defense in depth + # itself — from the gpuStack profile on AKS, GKE-COS, and OKE, at + # overlay level on Kind (#2181) — so this gate is defense in depth # rather than the primary mechanism. GKE # gpuStack=bundle-installer and AKS gpuStack=operator-managed # assign an explicit false: a driver pod exists there for the