Recipes declare no host GPU driver floor, leaving the #1995 enforcement hook unused

[yuanchen8911]

Problem

check-nvidia-smi gained a deployment-phase constraint, Deployment.gpu-driver.version, in #1995 (closed 2026-08-21). It is evaluated against the nvidia-smi banner on every verified GPU node and fails closed when declared but unmeasurable.

No recipe declares it. Not one overlay, not one mixin. The enforcement hook is built and idle.

The consequence is sharpest on the families where AICR does not control the driver at all — driver.enabled: false, so the GPU Operator installs nothing and the host driver is whatever the provider's node image ships:

Platform	Driver owner	Notes
AKS, gpuStack=azure-managed	node pool image	provider-owned
GKE COS, gpuStack=gke-default	node image	provider-owned
GKE COS, gpuStack=bundle-installer	AICR — gcp-driver-installer, pinned 580.173.02 (COS-qualified)	see below
OKE	preinstalled on bare-metal shapes	provider-owned

driver.enabled: false only means the GPU Operator installs no driver; it does not imply the provider owns it. After #2360, gpuStack=bundle-installer enables gcp-driver-installer, so AICR owns and pins the driver on that value.

bundle-installer still needs a floor: the installer's lsmod fast path exits 0 when any nvidia module is already loaded, so a pinned bump does not upgrade live nodes — an older driver persists until node replacement or reboot. A floor is what catches that.

On these, a cluster with an arbitrarily old host driver validates green: the check confirms nvidia-smi runs and reports a version, and compares it to nothing.

The example from #1995's own text is still unenforced — GKE A4X Max (GB300) requires host driver R580.95.05+, documented by Google.

This is the same missing-requirement-expression pattern as #2402 (Kubernetes floors below the DRA chart's kubeVersion): AICR installs components carrying real minimum requirements and does not express those requirements as constraints.

It is not the same failure mechanism, and this issue should not be read as claiming one. Deployment.gpu-driver.version is a post-deployment validation constraint: its absence means aicr validate cannot fail on an out-of-spec driver. It does not reject the recipe or stop Helm from deploying, whereas #2402's chart kubeVersion causes Helm to refuse installation outright.

Why now

The GPU Operator v26.7.0 bump (#2383) raises operand versions on exactly the platforms where the driver does not move:

• DCGM 4.6.0-1, DCGM Exporter 4.6.0
• device plugin 0.20.0, GFD 0.20.0
• NVIDIA DRA driver 0.5.0 kubelet plugin
• container toolkit 1.20.0 — note AKS runs toolkit.enabled: false, so the node image's toolkit must also be compatible

Each runs against a host driver AICR does not control and does not gate.

What already exists

From #1995, in validators/deployment/nvidia_smi.go:

• Constraint name Deployment.gpu-driver.version; the value must carry a comparison operator (typically >=) to behave as a floor — a bare version is an exact string match.
• Banner parsing handles both Driver Version: and the renamed KMD Version:, case-insensitively, including the pipe-delimited table-row layout (check-nvidia-smi fails on drivers renaming nvidia-smi banner fields #1667). Four-component versions such as 580.95.05.1 are rejected rather than silently truncated.
• failClosedIfUnmeasurableGPUDriverFloor converts the ordinarily non-blocking Skip paths (no GPU nodes, all cordoned, nodes busy) into a blocking error when the constraint is declared: "a declared gate must not PASS unenforced."
• With no constraint declared, the check keeps its original banner-presence behaviour and does not invent a floor.

So this issue requires no new validator code. The work is choosing floors and writing constraint lines.

Options

A. Per-family floors on the driver.enabled: false overlays (recommended)

Declare a floor on each provider-installed family, set from what that provider ships and what the accelerator requires.

• Pros: floors are meaningful per platform; GB300-on-GKE can carry a genuinely high floor without holding a generic L40S cluster to it. Targets exactly the families with no other guarantee.
• Cons: one decision per family; floors need revisiting as providers move their node images.

B. A single global floor in base.yaml

• Pros: one line, uniform coverage including the operator-managed families.
• Cons: must be the lowest common denominator across every supported accelerator and provider, so it is weakest precisely where it matters most. A floor that GB300 needs would break older L40S/A100 clusters; a floor they tolerate is meaningless for GB300.

C. Global baseline plus per-family overrides

Low floor in base.yaml, raised on families that need more.

• Pros: guarantees no family is entirely ungated while allowing precision.
• Cons: two places to reason about; a leaf overlay silently inheriting a weak baseline can read as "gated" when it is not.

D. Extend to the operator-managed families as well

Orthogonal to A–C. On EKS the pinned driver.version expresses intent, but the constraint detects something the pin cannot: the pinned driver not actually being what runs — a failed driver upgrade, a node that did not roll, or a pre-existing host driver shadowing the operator's.

Recommendation

Option A only. Option D is deferred — see "Option D cannot be expressed per profile value today" below.

1. Define floors on concrete service x accelerator leaves where a documented compatibility minimum exists, applying a floor across all profile values of an overlay only when the same minimum is valid for every value.

   Note the earlier framing of "the driver.enabled: false families" was wrong and is withdrawn: on AKS and GKE COS, driver ownership is profile-selected (gpuStack), but deployment constraints are not profile-selectable today. A constraint placed on the shared overlay applies to every profile value, so it cannot selectively target the provider-managed one. The correct unit is the affected resolved family with a documented minimum.

2. Key floors to documented requirements, not shipped or validated versions. A provider's currently shipped version proves a floor is satisfiable; it is not the minimum, and a floor keyed to it would move whenever a node image moves. Likewise a passing 26.7.0 run with 595.91.07 proves that version works — it does not establish the lowest compatible version. NVIDIA lists R580 and R535 as supported for v26.7 alongside 595.91.07 as the recommended default, so the chart default is not the compatibility floor.

   The effective floor should be the maximum documented minimum imposed by the accelerator, the active component stack, and any provider-specific configuration.

3. Resolve the last-wins placement question first and land matrix tests covering every combination affected by each floor (see the blocker section below). Do not gate documented floors on an exhaustive compatibility matrix for all supported recipes — that is ongoing work, not a prerequisite.

Record the source of each floor in a comment next to it (provider documentation, operand requirement, or accelerator requirement) so a later reader can tell whether it is still current.

Blocker: constraints merge last-wins, not max

This must be resolved before any floor lands. Constraints merge by name with the later overlay winning — pkg/recipe/validation.go:140-172: "Non-empty overlay unions by Name; overlay value wins on same-name." There is no version comparison and no max.

So if an accelerator leaf and a platform overlay both declare Deployment.gpu-driver.version, whichever applies later wins regardless of which floor is higher. A lower platform floor can silently downgrade a higher accelerator floor. Since GB300 requires more than its platform baseline, at least some floors are accelerator-driven, and a leaf floor must be able to raise a platform floor — which last-wins does not guarantee by construction.

Any implementation needs table-driven resolved-recipe tests asserting the final effective floor for every combination affected by that floor, including alternate profile values — not merely that a constraint exists somewhere in the chain.

Option D cannot be expressed per profile value today

Option D is a different contract — rollout convergence against the configured pin, not minimum compatibility — and is split out of this issue's scope.

It also cannot be routed through profiles as things stand, for two independent reasons:

1. ProfileValue.constraints are snapshot (generation-time) measurement constraints. Catalog load validates them via measurement.ValidatePath (pkg/recipe/constraint_paths.go:97), and Deployment is not a measurement Type (pkg/measurement/types.go defines K8s, GPU, OS, SystemD, NodeTopology). So Deployment.gpu-driver.version in a profile value is rejected at load.
2. Even bypassing that, it would land in the wrong place. Profile constraints append to top-level mergedSpec.Constraints (pkg/recipe/profile_resolution.go), while check-nvidia-smi reads only validation.deployment.constraints via findDeploymentConstraint (validators/deployment/gpu_operator_version.go:67-77). Two distinct fields.

Consequence: adding a KeyGPUDriver producer would not unblock profile-specific deployment floors. It would enable a generation-time snapshot constraint for an already-installed driver — and that would additionally require adding KeyGPUDriver to TypeGPU's closed key set in pkg/measurement/catalog.go:153-160, which today lists only KeyGPUPresent, KeyGPUCount, KeyGPUDriverLoaded, KeyGPUDetectionSource, KeyGPUModel.

A snapshot-side floor is a weaker and different guarantee than a deployment-phase one (it checks the snapshotted cluster at generation, not the running cluster after install), and profile constraints fail closed when unmeasurable, so older snapshots lacking the field would break generation.

Profile-specific deployment floors therefore need explicit profile-to-deployment-phase routing or selected-profile-aware validator logic. Option D also interacts with the documented --set gpuoperator:driver.version=... override: a floor pinned to the configured version would reject a downward override, which may be desirable but is a documented behaviour change.

Related

#1995 (validator hook, closed — this is the consuming half), #2402 (same defect class, Kubernetes floors), #2383 (v26.7.0 bump and driver 595.91.07, whose validation should inform the floors), #1716 (driver installation on GKE COS), #1873 (driver ownership coherence).

[yuanchen8911]

Revisions after review

Four corrections to the issue body above, all verified against origin/main at 33f0860fc.

1. This is a missing validation contract, not a demonstrated deploy failure

The body says "the recipe validates clean and the deploy fails." That is wrong for this constraint and should be struck.

Deployment.gpu-driver.version is a post-deployment validation constraint. Its absence means aicr validate cannot fail on an out-of-spec driver — it does not reject the recipe or stop Helm from deploying. That differs from #2402, where the chart's own kubeVersion causes Helm to refuse installation outright.

"Same missing-requirement-expression pattern" is accurate. "Same deploy-failure mechanism" is not, and overstates severity.

2. The GKE driver-ownership row is wrong

The table lists GKE COS as provider-owned. That is only true for one profile value. After #2360, recipes/overlays/gke-cos.yaml declares gpuStack with:

• bundle-installer — enables gcp-driver-installer (installer: {enabled: true}), so AICR owns and pins the driver (driverVersion: "580.173.02", COS-qualified).
• gke-default — Google-provisioned driver; genuinely provider-owned.

driver.enabled: false only means the GPU Operator installs no driver; it does not imply the provider owns it.

bundle-installer is still worth gating: the installer's lsmod fast path exits 0 when any nvidia module is already loaded, so a pinned bump does not upgrade live nodes — an older driver persists until node replacement or reboot. The floor is what would catch that.

3. Floors must key to documented requirements, not shipped versions

A provider's currently shipped version proves a floor is satisfiable; it is not itself the minimum. Keying floors to shipped versions would make them move whenever a node image moves, which is not what a floor means.

The effective floor should be the maximum documented minimum imposed by the accelerator, the active component stack, and any provider-specific configuration.

Related correction: the body says the 26.7.0 validation runs will "establish what the new operands actually tolerate." A passing run with 595.91.07 proves that version works; it does not establish the lowest compatible version. NVIDIA lists R580 and R535 as supported for v26.7 alongside 595.91.07 as the recommended default, so the chart default is not the compatibility floor.

4. Option A has a last-wins hazard that must be resolved first

This is the substantive finding. Constraints merge by name, last overlay wins — pkg/recipe/validation.go:140-172: "Non-empty overlay unions by Name; overlay value wins on same-name." There is no version comparison and no max.

So if an accelerator leaf and a platform overlay both declare Deployment.gpu-driver.version, whichever applies later wins regardless of which floor is higher. A lower platform floor can silently downgrade a higher accelerator floor. Per-family floors (Option A) walk straight into this.

Any implementation needs a table-driven resolved-recipe test asserting the final effective floor for every supported service x accelerator x profile combination, not just that the constraint exists somewhere.

5. Option D cannot be implemented via profiles as written

Option D (match the operator-managed floor to the configured pin) is a different contract — rollout convergence, not minimum compatibility — and should be split out.

CORRECTION (superseded). The mechanism originally given in this section was wrong. It claimed the blocker was profile_resolution.go's runtime fail-closed-on-NotFound branch, and that adding a KeyGPUDriver producer would unblock profile-specific floors. Neither is right:

• ProfileValue.constraints are snapshot constraints, validated as measurement paths at catalog load (pkg/recipe/constraint_paths.go:97 → measurement.ValidatePath). Deployment is not a measurement Type, so Deployment.gpu-driver.version is rejected there and the runtime branch is never reached.
• Even bypassing that, profile constraints append to top-level mergedSpec.Constraints, while check-nvidia-smi reads only validation.deployment.constraints. Two distinct fields.
• Therefore a KeyGPUDriver producer would not unblock Option D. It would also require adding KeyGPUDriver to TypeGPU's closed key set in pkg/measurement/catalog.go.

See the issue body for the corrected analysis.

Revised scope

1. Resolve the last-wins placement question and add matrix tests asserting the final effective floor per service x accelerator x profile — including alternate profile values and accidental downgrades.
2. Add floors where a documented minimum already exists rather than waiting on an exhaustive matrix. Google documents R570 for GKE B200 and R580.95.05 for A4X Max; those are actionable now.
3. Treat actual-driver-versus-effective-pin convergence (Option D) as a separate, profile- and override-aware follow-up.

Option A remains directionally right. Bundling Option D into it conflates two contracts and understates the work.

One scope caution

Step 1 as originally framed — "build an authoritative compatibility-floor matrix for every supported resolved recipe" — is open-ended research that could stall the cheap correctness win. The floors in step 2 are already documented by vendors and need no matrix to land. Suggest sequencing step 2 behind only the placement/merge fix from step 1, and treating the exhaustive matrix as ongoing rather than a gate.

[yuanchen8911]

The corrected analysis has been folded into the issue body, which is now the canonical description. The revisions comment above is retained only as the record of what changed and why; read the body, not the comment, for current state.

[yuanchen8911]

Implementation outcome: the mechanism lands, the motivating cases do not

Implemented on feat/2438-host-driver-floors. Recording what could and could not be done, because the headline example in this issue turned out to be unattachable and the issue should not read as resolved by the PR.

What landed

One floor — Deployment.gpu-driver.version: ">= 575.57.08" — on four overlays: rtx-pro-6000-eks-training, rtx-pro-6000-eks-inference, rtx-pro-6000-lke-training, rtx-pro-6000-lke-inference.

Source, quoted in a comment beside each declaration: NVIDIA GPU Operator platform support — "NVIDIA RTX PRO 6000 Blackwell Server Edition notes: Driver versions 575.57.08 or later is required." This is a product-level minimum, so unlike the Google numbers below it is OS- and provider-independent and holds across the whole chain.

The placement rule, and why it is the durable part

Declare a floor on the maximal-leaf overlay for the affected service x accelerator x intent family. Never on an *-any wildcard, never on base, never on an accelerator-unbound service overlay.

FindMatchingOverlays sorts candidates by specificity ascending, then merges each candidate's full inheritance chain root to leaf. Resolved through the production path:

criteria(service=eks, accelerator=rtx-pro-6000, intent=training)
  applied=[base monitoring-hpa rtx-pro-6000-any eks eks-training rtx-pro-6000-eks-training]

The accelerator wildcard is applied before the service overlays. Combined with last-wins merging and no max comparison, *-any is the weakest position for a floor, not the broadest — the inverse of the natural intuition, and exactly the silent-downgrade path this issue predicted.

Three tests enforce it (pkg/recipe/gpu_driver_floor_test.go): the final effective value for every affected resolved combination, the ordering fact the rule depends on, and an invariant rejecting any future floor placed on a wildcard, base, or an accelerator-unbound overlay. Both controls were demonstrated to fire — injecting a lower >= 550.00.00 on a deeper leaf failed the inheriting rows with a last-wins diagnostic, and moving a floor onto rtx-pro-6000-any failed the invariant.

Documented minimums that could NOT be attached

GB300 / GKE A4X Max — the headline example in this issue. R580.95.05 is genuinely documented by Google. But every gb300-* overlay in the catalog is EKS; there is no GKE GB300 overlay. The requirement is specific to A4X Max, so there is no family to attach it to. This is the one part of the spec above that could not be implemented as written.

GKE COS — all 14 GKE overlays. Google's per-accelerator numbers (L4/H100 >= 535, H200 >= 550, B200 >= 570) sit under "GPUs on Ubuntu nodes -> Ubuntu driver compatibility". Every AICR GKE overlay declares os: cos; there is no Ubuntu GKE family. Applying those numbers to COS leaves would assert a minimum neither Google nor NVIDIA has stated for COS, which is precisely what this issue's Recommendation forbids. Deliberately not done.

H200 >= 550 — same Ubuntu/GKE scoping; AICR's H200 overlays are EKS.

A100 — no documented minimum. Google lists R470/R525/R535/R550 as choices, not a floor.

AKS and OKE — neither provider publishes a per-accelerator host-driver minimum that could be found.

HGX B200 >= 570.133.20 — NVIDIA documents this as a driver container version, i.e. the GPU Operator's driver image. It does not transfer to host-managed leaves, so it was not applied to the GKE COS B200 overlays.

Consequence

The PR delivers the enforcement mechanism and one real floor, but GKE COS remains ungated — which is the gap this issue was filed to close. Mitigating factors on GKE specifically: under gpuStack=bundle-installer AICR owns the driver via gcp-driver-installer (pinned, COS-qualified), and under gke-default Google ships current drivers on the A3/A4 families. The residual exposure is mainly the stale-node case, since the installer's lsmod fast path leaves an already-loaded older driver in place until node replacement.

Follow-up needed

• Obtain a COS-scoped driver minimum from Google or NVIDIA, or confirm none is published. Without one, GKE COS cannot be gated honestly.
• Revisit GB300 if a GKE A4X Max overlay is ever added.
• Option D (matching a floor to the configured driver.version pin) remains deferred and unstarted; it is a different contract and needs profile-to-deployment-phase routing.

This issue should stay open, or be closed only alongside a follow-up that names the GKE COS gap explicitly.