Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/user/component-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
| **cert-manager-ocp-olm** | OLM installer for cert-manager on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified cert-manager Operator via the Operator Lifecycle Manager. Paired with `cert-manager-ocp`. OCP-specific. | [cert-manager (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5ec3f5a5eebc3d6acb0ee71c) |
| **cert-manager-ocp** | cert-manager CertManager CR for OpenShift. The operand Deployments (controller, cainjector, webhook) land in a hardcoded `cert-manager` namespace regardless of the operator's own namespace. Deployed after `cert-manager-ocp-olm`. OCP-specific. | [cert-manager](https://github.com/cert-manager/cert-manager) |
| **prometheus-adapter-ocp** | Prometheus Adapter for OpenShift. Reuses the same upstream chart as `prometheus-adapter`, pointed at OCP's built-in Thanos Querier instead of kube-prometheus-stack (which stays disabled on OCP). No certified OCP operator exists for this component. OCP-specific. | [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) |
| **nvidia-dra-driver-gpu-ocp** | NVIDIA DRA GPU driver for OpenShift. Reuses the same upstream chart as `nvidia-dra-driver-gpu`, with an added SCC RoleBinding granting the kubelet-plugin DaemonSet the host device access OCP's default restricted-v2 SCC forbids. No certified OCP operator exists for this component. OCP-specific. Known limitation: some GPU-driver rollout protections and remedy hints do not yet cover the OCP aliases (`gpu-operator-ocp`, `nvidia-dra-driver-gpu-ocp`) — the deployer's stale-NVML migration wait/restart, driver-version annotation injection, and the driver-absent remedy's `gpuoperator:`/`dradriver:` override keys; tracked in [#2136](https://github.com/NVIDIA/aicr/issues/2136). | [NVIDIA DRA Driver](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) |
| **nvidia-dra-driver-gpu-ocp** | NVIDIA DRA GPU driver for OpenShift. Reuses the same upstream chart as `nvidia-dra-driver-gpu`, with an added SCC RoleBinding granting the kubelet-plugin DaemonSet the host device access OCP's default restricted-v2 SCC forbids. No certified OCP operator exists for this component. OCP-specific. Known limitation: the driver-version annotation injected onto the DRA pod templates falls back to the `gpu-operator-ocp-olm` Subscription channel, which changes on a channel re-pin but not on every in-channel OLM auto-upgradeso the stale-NVML rollout gate (#973) can still miss an in-channel driver bump on OCP; tracked in [#2135](https://github.com/NVIDIA/aicr/issues/2135). | [NVIDIA DRA Driver](https://github.com/kubernetes-sigs/dra-driver-nvidia-gpu) |

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor — "Fixes: #2135" closes the issue the docs + comment cite as the tracker for the remaining limitation

The PR body says Fixes: #2135, so merging closes #2135. But this "Known limitation" row (and the mirroring comment at pkg/bundler/bundler.go:2954, "See #2135") both cite #2135 as the tracker for the still-open in-channel OLM auto-upgrade gap — the annotation keys off the Subscription channel string, which stays stable across in-channel CSV bumps under installPlanApproval: Automatic. After merge the link resolves to a closed issue, implying the limitation is resolved when it isn't. This is the same defect class mchmarny already got fixed once, when the row cited (now-closed) #2136.

Blast radius: User-facing catalog page + a durable code comment. An OCP operator hitting stale NVML after an in-channel auto-upgrade follows the link, finds a closed issue, and concludes it's a resolved bug rather than a documented standing limitation.

Fix: File a narrower follow-up issue for the in-channel-auto-upgrade gap and repoint both the catalog row and the bundler.go:2954 comment at it; or drop Fixes: #2135 from the PR body so #2135 stays open as the tracker. This is the one item worth doing before merge.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor — "Fixes: #2135" closes the issue the docs + comment still cite as the OPEN tracker

Not addressed since the prior review. This catalog row and the mirroring comment at pkg/bundler/bundler.go:2954 ("See #2135") both cite #2135 as the tracker for the still-open in-channel OLM auto-upgrade gap — the annotation keys off the Subscription channel string, which stays stable across in-channel CSV bumps under installPlanApproval: Automatic. But the PR body says Fixes: #2135, so merging closes #2135 and both links resolve to a closed issue, implying the limitation is resolved when it isn't. Same defect class mchmarny already got fixed once, when the row cited (now-closed) #2136.

Blast radius: User-facing catalog page + a durable code comment. An OCP operator hitting stale NVML after an in-channel auto-upgrade follows the link, finds a closed issue, and concludes it's a resolved bug rather than a documented standing limitation.

Fix: File a narrower follow-up issue for the in-channel-auto-upgrade gap and repoint both this row and the bundler.go:2954 comment at it; or drop Fixes: #2135 from the PR body so #2135 stays open as the tracker. The one item worth doing before merge.

| **k8s-nim-operator-ocp** | NVIDIA NIM Operator for OpenShift. Reuses the same upstream chart as `k8s-nim-operator`, with OCP-specific RBAC. Requires `cert-manager-ocp` for admission-webhook TLS. OCP-specific. | [K8s NIM Operator](https://github.com/NVIDIA/k8s-nim-operator) |

## How Components Are Selected
Expand Down
31 changes: 29 additions & 2 deletions pkg/bundler/bundler.go
Original file line number Diff line number Diff line change
Expand Up @@ -2837,8 +2837,10 @@ const draChartVersionAnnotation = header.Domain + "/gpu-operator-chart-version"
// enabled in the filtered resolved recipe before the annotation is
// written; recipes that disable either remain untouched.
const (
gpuOperatorComponentName = "gpu-operator"
draComponentName = "nvidia-dra-driver-gpu"
gpuOperatorComponentName = "gpu-operator"
draComponentName = "nvidia-dra-driver-gpu"
gpuOperatorOCPComponentName = "gpu-operator-ocp"
gpuOperatorOCPOLMComponentName = "gpu-operator-ocp-olm"
)

var (
Expand Down Expand Up @@ -2933,6 +2935,31 @@ func (b *DefaultBundler) injectDRAChartVersionAnnotation(
// is exercised by the disabled-component unit tests.
return
}
if gpuOperatorComponentName == gpuOperatorOCPComponentName && gpuOperatorVersion == "" {
// gpu-operator-ocp is a ClusterPolicy CR, not a Helm chart, so
// ComponentRef.Version is never populated for it — the empty
// check below would always skip injection on OCP. Fall back to
// the OLM Subscription channel (gpu-operator-ocp-olm) as the
// rollout-trigger value instead.
//
// KNOWN LIMITATION: the channel pin (e.g. "v25.10") only
// changes on a channel re-pin, not on every operator update.
// With installPlanApproval: Automatic (the default —
// components/gpu-operator-ocp-olm/values.yaml), OLM can
// upgrade to newer CSVs inside the same channel — reloading
// the driver — without the channel string changing, so this
// annotation catches bundle-driven operator bumps (a recipe
// regenerated against a different channel) but NOT in-channel
// auto-upgrades. The stale-NVML gap this annotation exists to
// close (#973) remains open for that case on OCP. See #2135.
if olmValues, ok := componentValues[gpuOperatorOCPOLMComponentName]; ok {
if sub, ok := olmValues["subscription"].(map[string]any); ok {
if channel, ok := sub["channel"].(string); ok {
gpuOperatorVersion = channel
}
}
}
}
Comment thread
mohityadav8 marked this conversation as resolved.
if gpuOperatorVersion == "" {
// gpu-operator is enabled but the resolver produced an empty
// Version string. This shouldn't happen in normal recipe
Expand Down
41 changes: 41 additions & 0 deletions pkg/bundler/bundler_dra_annotation_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,47 @@ func TestInjectDRAChartVersionAnnotation_PreservesExistingValues(t *testing.T) {
}
}

// TestInjectDRAChartVersionAnnotation_OCPFallbackToOLMChannel pins the
// OCP fallback added for #2135: gpu-operator-ocp is a ClusterPolicy
// CR, not a Helm chart, so ComponentRef.Version is always empty for
// it. Instead of skipping injection (the pre-fix behavior), the
// helper reads the OLM Subscription channel from the
// gpu-operator-ocp-olm component's values and mirrors that onto both
// nvidia-dra-driver-gpu-ocp pod templates.
func TestInjectDRAChartVersionAnnotation_OCPFallbackToOLMChannel(t *testing.T) {
b, err := New()
if err != nil {
t.Fatalf("New() error = %v", err)
}

const draOCPComponentName = "nvidia-dra-driver-gpu-ocp"
componentValues := map[string]map[string]any{
gpuOperatorOCPComponentName: {},
draOCPComponentName: {},
gpuOperatorOCPOLMComponentName: {
"subscription": map[string]any{
"channel": "v25.10",
},
},
}
rr := &recipe.RecipeResult{
ComponentRefs: []recipe.ComponentRef{
{Name: gpuOperatorOCPComponentName, Version: ""},
{Name: draOCPComponentName, Version: "0.4.1"},
},
}

b.injectDRAChartVersionAnnotation(componentValues, rr)

for _, podPath := range []string{"controller", "kubeletPlugin"} {
got := dig(componentValues[draOCPComponentName], podPath, "podAnnotations", draChartVersionAnnotation)
if got != "v25.10" {
t.Errorf("podAnnotations[%s][%s] = %v, want v25.10 (OLM channel fallback)",
podPath, draChartVersionAnnotation, got)
}
}
}

// TestInjectDRAChartVersionAnnotation_OverridesUserSet pins the
// "internal annotation always reflects the actual chart version"
// invariant. A user --set that wrote a stale value into the
Expand Down
64 changes: 56 additions & 8 deletions pkg/bundler/deployer/helm/helm.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,15 @@ type ComponentData struct {
IsOCI bool
Tag string // Git ref for Kustomize-typed components (tag/branch/commit)
Path string // Path within the repository to the kustomization

// DriverOperatorManaged is true when the bundle's effective values
// select an operator-managed NVIDIA driver — gpu-operator's or
// gpu-operator-ocp's driver.enabled is true. deploy.sh's DRA
// migration-wait block (see #2135, #973) uses this to tell "driver
// is host-managed" apart from "driver is operator-managed but the
// DaemonSet/node-label migration signal isn't observable yet",
// which live cluster state alone cannot distinguish.
DriverOperatorManaged bool
}

// compile-time interface check
Expand Down Expand Up @@ -265,6 +274,42 @@ func (g *Generator) Generate(ctx context.Context, outputDir string) (*deployer.O

// buildComponentDataList builds a sorted list of ComponentData from the recipe.
// It validates that all component names are safe for use as directory names.
// driverOperatorManaged reports whether this bundle's effective values
// select an operator-managed NVIDIA driver: gpu-operator's or
// gpu-operator-ocp's driver.enabled is true. Checks both component names
// since only one is ever enabled in a given recipe (see
// pkg/bundler/bundler.go's gpuOperatorComponentNames for the canonical
// list this mirrors).
// gpuOperatorComponentName and gpuOperatorOCPComponentName are this
// package's copy of the canonical/OCP gpu-operator component names (a
// 4th duplicate alongside pkg/bundler/bundler.go, pkg/bundler/validations
// /checks.go, and their override-key constants — this package cannot
// import pkg/bundler due to the dependency cycle noted at
// componentOverrideKeys' godoc equivalent). Named here, rather than an
// inline literal, so a `grep gpuOperatorOCPComponentName` across the repo
// surfaces every copy that needs updating together.
const (
gpuOperatorComponentName = "gpu-operator"
gpuOperatorOCPComponentName = "gpu-operator-ocp"
)

func (g *Generator) driverOperatorManaged() bool {
for _, name := range []string{gpuOperatorComponentName, gpuOperatorOCPComponentName} {
values, ok := g.ComponentValues[name]
if !ok {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nitpick — driverOperatorManaged() gpu-operator-ocp branch untested

The gpu-operator-ocp arm of driverOperatorManaged() is never exercised: both cases of TestGenerate_DeployScript_DRARestartGatedOnDriverOperatorManaged populate only ComponentValues["gpu-operator"]. Removing gpu-operator-ocp from the slice would leave OCP operator-managed bundles ungated and the suite would stay green.

Blast radius: OCP operator-managed driver bundles — the DRA restart gate would not engage.

Fix: Add a table case with ComponentValues["gpu-operator-ocp"] = {driver:{enabled:true}} (no gpu-operator entry) asserting the block-restart wording renders.

continue
}
driver, ok := values["driver"].(map[string]any)
if !ok {
continue
}
if enabled, ok := driver["enabled"].(bool); ok && enabled {
return true
}
}
return false
}

// Only the fields consumed by the orchestration templates are populated.
func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
// Sort by deployment order
Expand All @@ -273,6 +318,8 @@ func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
g.RecipeResult.DeploymentOrder,
)

driverOperatorManaged := g.driverOperatorManaged()

components := make([]ComponentData, 0, len(sorted))
for _, ref := range sorted {
if !deployer.IsSafePathComponent(ref.Name) {
Expand All @@ -283,14 +330,15 @@ func (g *Generator) buildComponentDataList() ([]ComponentData, error) {
chartName := ref.EffectiveChart()

components = append(components, ComponentData{
Name: ref.Name,
Namespace: ref.Namespace,
Repository: ref.Source,
ChartName: chartName,
Version: ref.Version,
IsOCI: strings.HasPrefix(ref.Source, "oci://"),
Tag: ref.Tag,
Path: ref.Path,
Name: ref.Name,
Namespace: ref.Namespace,
Repository: ref.Source,
ChartName: chartName,
Version: ref.Version,
IsOCI: strings.HasPrefix(ref.Source, "oci://"),
Tag: ref.Tag,
Path: ref.Path,
DriverOperatorManaged: driverOperatorManaged,
})
}

Expand Down
170 changes: 170 additions & 0 deletions pkg/bundler/deployer/helm/helm_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,176 @@ func TestGenerate_DeployScriptExecutable(t *testing.T) {
}
}

// TestGenerate_DeployScript_DRARestartGatedOnDriverOperatorManaged pins the
// fix for #2135's review follow-up: live cluster state alone (absent
// DaemonSet + no labeled node) cannot tell "driver is host-managed" apart
// from "driver is operator-managed but the migration gate hasn't converged
// yet" — the latter must block the DRA kubelet-plugin restart rather than
// running it unguarded, or it reproduces the invalid-CDI/ContainerCreating
// failure (#973). DriverOperatorManaged is derived at bundle time from
// gpu-operator's/gpu-operator-ocp's effective driver.enabled and threaded
// into the rendered script, so this only needs to check the generated
// text — no live cluster required.
func TestGenerate_DeployScript_DRARestartGatedOnDriverOperatorManaged(t *testing.T) {
recipeResult := func() *recipe.RecipeResult {
return &recipe.RecipeResult{
Kind: "RecipeResult",
APIVersion: "aicr.run/v1alpha2",
Metadata: recipe.RecipeResultMetadata{Version: "v0.1.0"},
Criteria: &recipe.Criteria{
Service: "eks",
Accelerator: "h100",
Intent: "training",
},
ComponentRefs: []recipe.ComponentRef{
{
Name: "gpu-operator",
Namespace: "gpu-operator",
Chart: "gpu-operator",
Version: "v25.3.3",
Source: "https://helm.ngc.nvidia.com/nvidia",
},
{
Name: "nvidia-dra-driver-gpu",
Namespace: "nvidia-dra-driver",
Chart: "nvidia-dra-driver-gpu",
Version: "0.4.1",
Source: "https://helm.ngc.nvidia.com/nvidia",
},
},
DeploymentOrder: []string{"gpu-operator", "nvidia-dra-driver-gpu"},
}
}

tests := []struct {
name string
recipeResultOCP bool // when true, uses OCP component names throughout instead of canonical
componentValues map[string]map[string]any
wantContains []string
wantNotContains []string
}{
{
name: "operator-managed driver blocks restart until gate is observable",
componentValues: map[string]map[string]any{
"gpu-operator": {
"driver": map[string]any{"enabled": true},
},
"nvidia-dra-driver-gpu": {},
},
wantContains: []string{
`SKIP_RESTART="false"`,
`blocking the DRA plugin restart until the driver rollout is detectable`,
`SKIP_RESTART=true`,
`if [[ -n "${DRA_DS}" && "${SKIP_RESTART}" != "true" ]]; then`,
`no nodes labeled nvidia.com/gpu.deploy.driver=true yet; skipping migration wait and DRA restart`,
},
wantNotContains: []string{
`nvidia-driver-daemonset not present (host-managed driver); skipping migration wait"`,
},
},
{
name: "host-managed driver still skips the wait without blocking restart",
componentValues: map[string]map[string]any{
"gpu-operator": {
"driver": map[string]any{"enabled": false},
},
"nvidia-dra-driver-gpu": {},
},
wantContains: []string{
`nvidia-driver-daemonset not present (host-managed driver); skipping migration wait"`,
},
wantNotContains: []string{
`blocking the DRA plugin restart until the driver rollout is detectable`,
},
},
{
name: "OCP DRA component renders its own guard and is gated by gpu-operator-ocp's driver.enabled",
recipeResultOCP: true,
componentValues: map[string]map[string]any{
"gpu-operator-ocp": {
"driver": map[string]any{"enabled": true},
},
"nvidia-dra-driver-gpu-ocp": {},
},
wantContains: []string{
`if [[ "${name}" == "nvidia-dra-driver-gpu-ocp" ]]; then`,
`SKIP_RESTART="false"`,
`blocking the DRA plugin restart until the driver rollout is detectable`,
`SKIP_RESTART=true`,
},
wantNotContains: []string{
`if [[ "${name}" == "nvidia-dra-driver-gpu" ]]; then`,
`nvidia-driver-daemonset not present (host-managed driver); skipping migration wait"`,
},
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ctx := context.Background()
outputDir := t.TempDir()

rr := recipeResult()
if tt.recipeResultOCP {
rr = &recipe.RecipeResult{
Kind: "RecipeResult",
APIVersion: "aicr.run/v1alpha2",
Metadata: recipe.RecipeResultMetadata{Version: "v0.1.0"},
Criteria: &recipe.Criteria{
Service: "ocp",
Accelerator: "h100",
Intent: "training",
},
ComponentRefs: []recipe.ComponentRef{
{
Name: "gpu-operator-ocp",
Namespace: "gpu-operator",
Chart: "gpu-operator",
Version: "",
Source: "",
},
{
Name: "nvidia-dra-driver-gpu-ocp",
Namespace: "nvidia-dra-driver",
Chart: "nvidia-dra-driver-gpu",
Version: "0.4.1",
Source: "https://helm.ngc.nvidia.com/nvidia",
},
},
DeploymentOrder: []string{"gpu-operator-ocp", "nvidia-dra-driver-gpu-ocp"},
}
}

g := &Generator{
RecipeResult: rr,
ComponentValues: tt.componentValues,
Version: "v1.0.0",
}

if _, err := g.Generate(ctx, outputDir); err != nil {
t.Fatalf("Generate failed: %v", err)
}

content, err := os.ReadFile(filepath.Join(outputDir, "deploy.sh"))
if err != nil {
t.Fatalf("failed to read deploy.sh: %v", err)
}
script := string(content)

for _, want := range tt.wantContains {
if !strings.Contains(script, want) {
t.Errorf("deploy.sh missing %q", want)
}
}
for _, notWant := range tt.wantNotContains {
if strings.Contains(script, notWant) {
t.Errorf("deploy.sh unexpectedly contains %q", notWant)
}
}
})
}
}

// ---------------------------------------------------------------------------
// Property tests (helpers and data-shape preservation)
// ---------------------------------------------------------------------------
Expand Down
Loading