YP6M-3270 Scope the flex GPU device plugin to flex nodes only

[brandonli-ybor]

Problem

gpu.nodeSelector matched p6m.dev/node-type=gpu-shared alone. That label is not flex-specific — the platform's Karpenter time-sliced-gpu NodePool stamps the same label on cloud (AKS) GPU nodes, where the gpu-operator already runs its own device plugin. The chart's comment stated the assumption directly:

the nodeSelector (p6m.dev/node-type=gpu-shared, auto-applied by cloud-init to GPU-family flex nodes) means this schedules ZERO pods until a GPU flex node exists

On ybor-playground-dev-westus2 that is false: this DaemonSet schedules onto Karpenter-provisioned T4 nodes. Both plugins bind-mount /var/lib/kubelet/device-plugins and register the resource nvidia.com/gpu; kubelet's device manager keeps only the last registrant. So a time-sliced node advertises 4 GPUs or 1, depending on which plugin happened to start last — and it never self-heals.

This is the real cause of YP6M-3270, which had been recorded as a Karpenter label race. It is not: the config label is set by kubelet at first registration (proven via managedFields), and the gpu-operator's SIGHUP reload correctly re-registers in ~149 ms.

Evidence (live, ybor-playground-dev-westus2, node aks-time-sliced-gpu-9nlmk)

Fresh node, both plugins present — operator registered last, so it won:

17:30:26       flex plugin pod SCHEDULED onto the Karpenter time-sliced node
17:32:00.697   flex plugin registered nvidia.com/gpu (1 device, no sharing)
17:32:03.399   operator plugin registered nvidia.com/gpu (config: any)
17:32:03.548   operator plugin RE-registered with replicas:4   -> capacity 4

Forcing the reverse order — delete the flex pod, the DaemonSet recreates it so it registers last, operator's plugin untouched (0 restarts, still loaded with replicas:4):

17:34:43.752   flex plugin re-registered nvidia.com/gpu
17:35:19       kubelet capacity 4 -> 1
17:39:06       still capacity=1 allocatable=1, no self-heal
               node labels still replicas=4 / Tesla-T4-SHARED / sharing-strategy=time-slicing

Fix

Require both labels. flex.azure.com/node=true is set by terraform/ec2/main.tf:227 in this repo as a kubelet --node-labels entry on every flex node, and p6m.dev/node-type=gpu-shared is added at :231 for GPU-family instance types — same locals block, so a GPU flex node always has both, and an AKS node can never have flex.azure.com/node. The two DaemonSets become mutually exclusive by construction, with no cloud-init change and no new label.

Also updates the two comments that encoded the failed assumption, and bumps the chart 0.3.0 → 0.3.1.

Verification

Applied as a live DaemonSet patch on ybor-playground-dev-westus2 first, then verified end to end:

17:42:37   selector patched -> flex pod evicted from the T4; capacity still 1 (stale endpoint)
17:43:54   operator plugin restarted -> capacity 4
17:46:49   held at 4, flex pods on node = 0

Then the acceptance case — a brand-new Karpenter time-sliced node, zero intervention:

18:00:10   new node registered: aks-time-sliced-gpu-pc9vv
18:02:25   cap=4  flexplugins=0  opplugin=Running   -> CONVERGED with no manual step
18:04:43   held at 4 for 2+ min; GPU pod Running; only device-plugin pod on the node is the operator's

helm template with values-ybor-playground.yaml renders the DaemonSet with nodeSelector: {flex.azure.com/node: "true", p6m.dev/node-type: gpu-shared}, matching what was validated live.

Risk notes

• Blast radius is a narrowing, not a widening: the DaemonSet can only schedule on strictly fewer nodes than before. The failure mode to watch is the inverse — a GPU flex node provisioned by a path that does not set flex.azure.com/node would no longer get the plugin. Today that label comes from this repo's own terraform, so the invariant is co-located; if flex nodes ever gain a second provisioning path, it must set both labels.
• No test added. This chart has no helm unittest suite (no *_test.yaml anywhere in the repo), so a selector assertion would mean introducing a test framework — out of scope for a one-line fix. Verified by live run + render instead.
• The dev cluster currently carries this change as an out-of-band DaemonSet patch, which Helm will revert on the next flex-node-system upgrade. Merging this is what makes it permanent.
• No image, action, or dependency refs changed. No permission/RBAC changes. No secrets or data in the diff.

Refs: YP6M-3270