From dc236249aa48a1ac47dea079098b7caf99fe2010 Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Mon, 20 Jul 2026 08:58:33 -0700 Subject: [PATCH 1/3] Add GPU Operator DevOps Agent package (prototype, for review/testing) A prototype AI DevOps-agent package for the NVIDIA GPU Operator: an AGENTS.md front door + operator skills + references distilled from the public GPU Operator docs and source. It lets an AI coding/ops agent help install, operate, maintain, and troubleshoot the GPU Operator with a probe-first, source-cited, safety-gated operating model. Docs-derived and version-anchored to GPU Operator v26.3.3. This is a review/test prototype, not a validated release. Signed-off-by: Andrew Chen --- gpu-operator-devops-agent/AGENTS.md | 160 ++++ gpu-operator-devops-agent/README.md | 109 +++ .../manifests/bad-runtimeclass-scenario.yaml | 19 + .../manifests/cuda-vectoradd.yaml | 12 + .../manifests/time-slicing-verification.yaml | 30 + .../references/bootstrap.md | 118 +++ .../references/branch-matrix.md | 19 + .../references/claim-ledger.md | 24 + .../references/command-cards.md | 708 ++++++++++++++++++ .../references/command-catalog.md | 46 ++ .../references/environment-inventory.md | 48 ++ .../references/failure-signatures.md | 34 + .../references/field-scenario-cards.md | 227 ++++++ .../references/models.md | 127 ++++ .../references/operational-reasoning.md | 105 +++ .../references/self-improvement.md | 89 +++ .../references/source-map.md | 92 +++ .../references/supported-scope.md | 47 ++ .../runtime-memory/README.md | 18 + .../runtime-memory/anti-patterns.md | 96 +++ .../runtime-memory/discovered-resources.md | 34 + .../runtime-memory/insights.md | 44 ++ .../runtime-memory/operating-parameters.md | 60 ++ .../runtime-memory/upstream-issue-drafts.md | 31 + gpu-operator-devops-agent/scripts/README.md | 12 + .../scripts/apply-time-slicing.sh | 37 + .../install-gpu-operator-k3s-host-driver.sh | 48 ++ .../inventory-gpu-operator-environment.sh | 63 ++ .../scripts/validate-cuda-vectoradd.sh | 35 + .../gpu-operator-evidence-bundle/SKILL.md | 93 +++ .../skills/gpu-operator-improve/SKILL.md | 70 ++ .../skills/gpu-operator-install/SKILL.md | 127 ++++ .../skills/gpu-operator-maintain/SKILL.md | 125 ++++ .../skills/gpu-operator-operate/SKILL.md | 97 +++ .../skills/gpu-operator-troubleshoot/SKILL.md | 126 ++++ 35 files changed, 3130 insertions(+) create mode 100644 gpu-operator-devops-agent/AGENTS.md create mode 100644 gpu-operator-devops-agent/README.md create mode 100644 gpu-operator-devops-agent/manifests/bad-runtimeclass-scenario.yaml create mode 100644 gpu-operator-devops-agent/manifests/cuda-vectoradd.yaml create mode 100644 gpu-operator-devops-agent/manifests/time-slicing-verification.yaml create mode 100644 gpu-operator-devops-agent/references/bootstrap.md create mode 100644 gpu-operator-devops-agent/references/branch-matrix.md create mode 100644 gpu-operator-devops-agent/references/claim-ledger.md create mode 100644 gpu-operator-devops-agent/references/command-cards.md create mode 100644 gpu-operator-devops-agent/references/command-catalog.md create mode 100644 gpu-operator-devops-agent/references/environment-inventory.md create mode 100644 gpu-operator-devops-agent/references/failure-signatures.md create mode 100644 gpu-operator-devops-agent/references/field-scenario-cards.md create mode 100644 gpu-operator-devops-agent/references/models.md create mode 100644 gpu-operator-devops-agent/references/operational-reasoning.md create mode 100644 gpu-operator-devops-agent/references/self-improvement.md create mode 100644 gpu-operator-devops-agent/references/source-map.md create mode 100644 gpu-operator-devops-agent/references/supported-scope.md create mode 100644 gpu-operator-devops-agent/runtime-memory/README.md create mode 100644 gpu-operator-devops-agent/runtime-memory/anti-patterns.md create mode 100644 gpu-operator-devops-agent/runtime-memory/discovered-resources.md create mode 100644 gpu-operator-devops-agent/runtime-memory/insights.md create mode 100644 gpu-operator-devops-agent/runtime-memory/operating-parameters.md create mode 100644 gpu-operator-devops-agent/runtime-memory/upstream-issue-drafts.md create mode 100644 gpu-operator-devops-agent/scripts/README.md create mode 100755 gpu-operator-devops-agent/scripts/apply-time-slicing.sh create mode 100755 gpu-operator-devops-agent/scripts/install-gpu-operator-k3s-host-driver.sh create mode 100755 gpu-operator-devops-agent/scripts/inventory-gpu-operator-environment.sh create mode 100755 gpu-operator-devops-agent/scripts/validate-cuda-vectoradd.sh create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-evidence-bundle/SKILL.md create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-improve/SKILL.md create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-install/SKILL.md create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-maintain/SKILL.md create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-operate/SKILL.md create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-troubleshoot/SKILL.md diff --git a/gpu-operator-devops-agent/AGENTS.md b/gpu-operator-devops-agent/AGENTS.md new file mode 100644 index 000000000..7b2fb45ef --- /dev/null +++ b/gpu-operator-devops-agent/AGENTS.md @@ -0,0 +1,160 @@ +# NVIDIA GPU Operator DevOps Agent Package + +Purpose: help an AI DevOps agent install, operate, maintain, and troubleshoot +NVIDIA GPU Operator from public docs/source and live environment evidence. This +package is operational guidance, not a substitute for change approval. + +## Supported Scope + +- Primary product/version: NVIDIA GPU Operator `v26.3.3`. +- Primary install manager: Helm chart `nvidia/gpu-operator`. +- Primary environment: test-owned single-node or small Kubernetes clusters with + NVIDIA data center GPUs, especially Ubuntu 22.04/24.04 plus K3s/containerd + and A100-class hardware. +- Primary validation: GPU Operator health, allocatable GPU resources, CUDA + VectorAdd manifest success, and optional documented examples selected for the + environment. +- See `references/supported-scope.md` for caution zones and non-goals. + +## Source Precedence + +1. Live target evidence from read-only commands. +2. Official GPU Operator docs for the target release/current docs. +3. GPU Operator public source at the target tag, especially Helm values, + templates, CRDs, and `hack/must-gather.sh`. +4. Public release notes, public GitHub issues, public forum evidence, and live + target-environment evidence gathered during the user's approved run. + +Runtime source boundary: do not require or cite NVIDIA-internal Confluence, +Google Docs, Jira, GitLab, Slack, private support systems, private hostnames, +or internal-only procedures. Public package decisions must be grounded in +public sources or the user's approved live environment. + +## Required Tools And Access + +- `kubectl` with cluster-admin or equivalent access for GPU Operator install. +- `helm`, `jq`, and shell access to the intended sandbox or cluster. +- `KUBECONFIG` explicitly set or context explicitly proven before mutation. +- Node shell access for host checks such as `nvidia-smi`, `lspci`, `lsmod`, + `dmesg`, `journalctl`, and runtime version/config checks. +- Approval before changing nodes, drivers, runtime config, MIG, Helm releases, + CRDs, namespaces, workload placement, reboots, drains, or test environment + lifecycle. + +## Bootstrap And Command Context Policy + +Before any product mutation or troubleshooting branch selection, load +`references/environment-inventory.md` and `references/bootstrap.md`, then prove: + +```bash +hostname +pwd +id +kubectl config current-context +kubectl get nodes -o wide +helm version +``` + +For fresh K3s sandboxes, set: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +``` + +Record the sandbox name/ID, how the environment was provisioned (e.g. a cloud +GPU instance or a K3s node), Kubernetes version, runtime version, OS/kernel, +GPU model, and cleanup command. + +Use `scripts/inventory-gpu-operator-environment.sh` when shell access is +available. Do not diagnose from symptom text alone; collect deployment +inventory first and cite which inventory fact selects the branch. + +Branch decisions are staged. Do not decide `driver.enabled`, `toolkit.enabled`, +NFD, CDI/NRI, or legacy runtime config from pre-bootstrap evidence alone. After +Kubernetes exists and `kubectl` works, rerun host/runtime/NFD evidence, then +choose install values. + +Branch decisions are also model-first and matrix-backed. Before selecting a +fix, locate the earliest failed layer in `references/models.md`, then use +`references/branch-matrix.md` to select the branch, rejected branch, command +card, script, fallback, and stop condition. Mutate only the layer the evidence +actually implicates. + +## Safety Policy + +- Read first, mutate second, validate third, clean up test workloads last. +- Helm success is not product success. Require the success ladder in + `references/models.md`. +- Do not use `helm uninstall`, CRD deletion, `kubectl drain`, node reboot, + driver downgrade, host runtime edits, or MIG reconfiguration without naming + blast radius and rollback. +- Do not retry the same failing install branch twice. Collect an escalation + bundle instead. +- Keep test mutations inside the assigned test-owned sandbox. Never mutate + production, shared, customer, or any other team's infrastructure. +- Persist reusable lessons in `runtime-memory/` before ending the session. +- Do not file upstream GitHub issues without user approval for the target repo, + title/body, labels, and credentials. Draft them locally first. + +## Standard Evidence Collection + +```bash +kubectl get nodes -o wide +kubectl get nodes -L feature.node.kubernetes.io/pci-10de.present,nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu-driver-upgrade-state,nvidia.com/mig.config,nvidia.com/mig.config.state +kubectl get pods -n gpu-operator -o wide +kubectl get ds -n gpu-operator +kubectl get clusterpolicy -o yaml +kubectl get events -A --sort-by='.lastTimestamp' | tail -100 +helm list -n gpu-operator +helm get values gpu-operator -n gpu-operator -o yaml +``` + +## Skill Routing + +| Task | Load | +|---|---| +| Deployment inventory before diagnosis | `references/environment-inventory.md`, then `scripts/inventory-gpu-operator-environment.sh` when shell access exists | +| Branch selection or layer ownership reasoning | `references/models.md`, then `references/branch-matrix.md`, before mutating | +| Public-field-shaped failure or scenario evidence | `references/field-scenario-cards.md`, then `references/failure-signatures.md` | +| Clean sandbox, K3s, kubeconfig, command proof | `references/bootstrap.md` first, then install skill | +| Fragile command execution or cleanup | `references/command-cards.md`, then the matching `scripts/*.sh` template | +| Fresh install or install plan | `skills/gpu-operator-install/SKILL.md` | +| Health, workload validation, drift check | `skills/gpu-operator-operate/SKILL.md` | +| Upgrade, rollback, driver/MIG/config maintenance | `skills/gpu-operator-maintain/SKILL.md` | +| Failed install, missing GPU, runtime errors, Xid, stuck upgrade | `skills/gpu-operator-troubleshoot/SKILL.md` | +| Support/debug handoff | `skills/gpu-operator-evidence-bundle/SKILL.md` | +| Record a reusable lesson, anti-pattern, discovered resource, operating parameter, or upstream issue draft | `skills/gpu-operator-improve/SKILL.md` | + +## Operating Invariants + +- Command context must point at the intended sandbox before mutation. +- Host GPU visibility precedes Kubernetes/GPU Operator debugging. +- Kubernetes node readiness precedes product install. +- NFD/GPU labels precede operand daemonset scheduling. +- Driver readiness precedes toolkit/device-plugin readiness. +- Device plugin resource advertisement precedes workload validation. +- `ClusterPolicy.status.state=ready` is necessary but not sufficient. +- CDI is default in GPU Operator v25.10.0+. NRI is useful on K3s-like runtimes + when the runtime is in the known-tested range. For K3s/containerd `2.3.x`, + prefer the explicit K3s toolkit-env branch unless the user explicitly wants a + disposable-sandbox validation of newer-than-documented NRI. +- GPU Operator manages containerized drivers only when `driver.enabled=true`; + host-preinstalled drivers remain host/platform owned. +- The package must improve from live evidence. If the run teaches a reusable + lesson, write it to `runtime-memory/` and update references or skills when + appropriate. + +## Stop/Escalation Conditions + +Stop and produce an escalation package when GPU hardware is absent, `nvidia-smi` +fails on a host-managed-driver branch, critical Xid errors appear, the needed +runtime/OS/kernel is outside reviewed scope, the fix requires reboot/drain/CRD +deletion/driver downgrade, credentials are missing, or public sources conflict +on a load-bearing claim. + +## Freshness Policy + +For any version other than `v26.3.3`, re-check release notes, chart values, +CRDs, `ClusterPolicy` schema, CDI/NRI defaults, driver defaults, support +matrix, and known issues. Update `references/claim-ledger.md` before reusing +this package for the new version. diff --git a/gpu-operator-devops-agent/README.md b/gpu-operator-devops-agent/README.md new file mode 100644 index 000000000..e3941b35c --- /dev/null +++ b/gpu-operator-devops-agent/README.md @@ -0,0 +1,109 @@ + + +# NVIDIA GPU Operator — DevOps Agent Package (prototype) + +## What this is + +A prototype **DevOps agent package** for the NVIDIA GPU Operator: a curated +`AGENTS.md` front door + skills + references that let an AI coding/ops agent +(Claude, Cursor, etc.) help an operator **install / operate / maintain / +troubleshoot** the GPU Operator by reading and reasoning over the **public** +GPU Operator docs and source — instead of a fresh agent guessing from an +unstructured search. + +It was built by distilling the public GPU Operator documentation (this repo's +`gpu-operator/` docs) and the public GPU Operator source into a compact, +routable operating charter with an explicit safety model. + +## What's in it + +| Path | What it is | +|---|---| +| `AGENTS.md` | Agent front door / operating charter — supported scope, source precedence, bootstrap policy, safety policy, skill routing, operating invariants, stop/escalation conditions | +| `skills/` | Six operator SOPs: `gpu-operator-install`, `-operate`, `-maintain`, `-troubleshoot`, `-evidence-bundle`, `-improve` | +| `references/` | The reasoning substrate — `models.md` (the layered success ladder), `branch-matrix.md` (staged branch decisions), `command-cards.md` (validated recipes), `field-scenario-cards.md` + `failure-signatures.md` (troubleshooting), `environment-inventory.md`, `supported-scope.md`, `claim-ledger.md`, others | +| `scripts/` | Read-only inventory + a few guarded mutation helpers (host-driver K3s install, CUDA VectorAdd validation, time-slicing) | +| `runtime-memory/` | Seed files the agent appends to as it learns (insights, anti-patterns, discovered resources, operating parameters) | +| `manifests/` | Small validation manifests (CUDA VectorAdd, time-slicing verification, a bad-runtimeClass repro) | + +## Intended behavior (what to watch for when testing) + +A correctly-behaving agent should: + +- **Inventory before diagnosing** — prove command context (`kubectl config + current-context`, `kubectl get nodes`, host `nvidia-smi`) and run + `scripts/inventory-gpu-operator-environment.sh` before any mutation; never + diagnose from symptom text alone. +- **Follow source precedence** — live target evidence first, then the official + GPU Operator docs for the target release, then public GPU Operator source at + the target tag, then public release notes / issues / forums. It should **cite + the source** for each load-bearing claim and refuse to rely on private/internal + sources. +- **Stage branch decisions** — not decide `driver.enabled` / `toolkit.enabled` / + NFD / CDI-NRI from pre-bootstrap evidence; locate the **earliest failed layer** + in `references/models.md`, then select a branch via `references/branch-matrix.md`. +- **Respect the safety model** — read → mutate → validate → clean up; treat + Helm success as *not* product success (require the success ladder); and + **name blast radius + rollback** before `helm uninstall`, CRD deletion, + `kubectl drain`, node reboot, driver downgrade, or MIG reconfiguration. It + should refuse to retry the same failing branch twice and produce an escalation + bundle instead. + +## How to test it + +1. Point an agent at **`gpu-operator-devops-agent/AGENTS.md`** as its front door + (copy the directory into a workspace and let Claude/Cursor read it as the + working set). +2. Ask it an operator task against a **test-owned** single-node or small cluster + (Ubuntu 22.04/24.04 + K3s/containerd + A100-class GPU is the primary target) — + e.g. *"install the GPU Operator on this K3s node,"* *"a GPU workload can't be + scheduled after install,"* *"upgrade the GPU Operator and roll back if driver + pods crash-loop."* +3. Watch for the intended behavior above: probe-first, cited sources, staged + branch selection, and safety refusals on high-blast-radius steps. +4. To capture a support/debug handoff, exercise + `skills/gpu-operator-evidence-bundle`. + +Keep all mutations inside a disposable, test-owned sandbox. This package is +operational guidance, **not** a substitute for change approval. + +## Status & honesty caveats + +- **Prototype, docs-derived.** Command flows are distilled from the public docs + and public source; the package **has not itself been rerun end-to-end after + authoring** (see `references/supported-scope.md` → *Verification Gap + Statement*). Readiness for a new environment requires a package-loaded run that + exercises clean bootstrap, install, workload validation, and at least one + troubleshooting scenario. +- **Version-anchored to GPU Operator `v26.3.3`.** For any other version, the + agent is instructed to re-check release notes, chart values, CRDs, + `ClusterPolicy` schema, CDI/NRI defaults, and the support matrix, and to update + `references/claim-ledger.md` first (see *Freshness Policy* in `AGENTS.md`). +- **Scope is single-node / small K3s clusters with data-center GPUs.** Out of + scope: OpenShift OLM, vGPU licensing, confidential containers, Kata, KubeVirt, + GPUDirect RDMA/GDS, air-gapped/private-registry flows, managed-cloud cluster + creation, and Jetson/integrated-GPU platforms — see + `references/supported-scope.md`. + +## What feedback helps most + +- **Command-card accuracy** — are the install / validation / troubleshooting + recipes in `references/command-cards.md` correct and complete for the GPU + Operator on K3s/containerd? +- **Doc & source citations** — are the cited docs sections and source paths right? +- **Layered model** — does `references/models.md` match how the GPU Operator + actually fails and recovers (NFD → driver → toolkit → device-plugin → + validator → workload)? +- **Skill coverage** — are install / operate / maintain / troubleshoot / + evidence-bundle / improve the right operator buckets? What's missing? + +## Provenance + +Generated from a curated source manifest built by reading the **public** GPU +Operator documentation in this repository (`gpu-operator/`, snapshot commit +`77a2daaf234a2cebc8e178f24e18cc4a6150e8b1`) and the public GPU Operator source +at the `v26.3.3` release. This directory is the **reference prototype** — a draft +for review and testing, not a validated release. diff --git a/gpu-operator-devops-agent/manifests/bad-runtimeclass-scenario.yaml b/gpu-operator-devops-agent/manifests/bad-runtimeclass-scenario.yaml new file mode 100644 index 000000000..f8aec8cc2 --- /dev/null +++ b/gpu-operator-devops-agent/manifests/bad-runtimeclass-scenario.yaml @@ -0,0 +1,19 @@ +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: abtest-bad-nvidia +handler: definitely-not-real-nvidia-handler +--- +apiVersion: v1 +kind: Pod +metadata: + name: abtest-bad-runtime +spec: + restartPolicy: Never + runtimeClassName: abtest-bad-nvidia + containers: + - name: cuda-vectoradd + image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" + resources: + limits: + nvidia.com/gpu: 1 diff --git a/gpu-operator-devops-agent/manifests/cuda-vectoradd.yaml b/gpu-operator-devops-agent/manifests/cuda-vectoradd.yaml new file mode 100644 index 000000000..7f535bf9d --- /dev/null +++ b/gpu-operator-devops-agent/manifests/cuda-vectoradd.yaml @@ -0,0 +1,12 @@ +apiVersion: v1 +kind: Pod +metadata: + name: cuda-vectoradd +spec: + restartPolicy: OnFailure + containers: + - name: cuda-vectoradd + image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" + resources: + limits: + nvidia.com/gpu: 1 diff --git a/gpu-operator-devops-agent/manifests/time-slicing-verification.yaml b/gpu-operator-devops-agent/manifests/time-slicing-verification.yaml new file mode 100644 index 000000000..e3189f8ee --- /dev/null +++ b/gpu-operator-devops-agent/manifests/time-slicing-verification.yaml @@ -0,0 +1,30 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: time-slicing-verification + labels: + app: time-slicing-verification +spec: + replicas: 5 + selector: + matchLabels: + app: time-slicing-verification + template: + metadata: + labels: + app: time-slicing-verification + spec: + tolerations: + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + hostPID: true + containers: + - name: cuda-sample-vector-add + image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" + command: ["/bin/bash", "-c", "--"] + args: + - while true; do /cuda-samples/vectorAdd; done + resources: + limits: + nvidia.com/gpu: 1 diff --git a/gpu-operator-devops-agent/references/bootstrap.md b/gpu-operator-devops-agent/references/bootstrap.md new file mode 100644 index 000000000..cf6ce6675 --- /dev/null +++ b/gpu-operator-devops-agent/references/bootstrap.md @@ -0,0 +1,118 @@ +# Bootstrap And Command Context + +Use this reference before install, maintenance, or troubleshooting when the +target substrate might not exist or the shell context is uncertain. + +## Clean-Sandbox Path + +For documentation/example testing, start from a clean, disposable GPU sandbox — +for example a cloud GPU instance (such as a Brev instance) or a single-node K3s +host. + +Record the selected environment and how it was provisioned. The setup notes must +record instance type, instance ID, GPU model, Kubernetes/substrate choice, +cleanup command, and whether the instance should be stopped or destroyed. If a +clean sandbox is not available, stop and provision one before continuing. + +## K3s Bootstrap On A Test-Owned VM + +Only use this when the VM is explicitly assigned to the test and host mutation +is approved. + +1. Prove the host: + + ```bash + hostname + id + uname -a + cat /etc/os-release + nvidia-smi + lspci | grep -i nvidia + ``` + +2. Install or select Kubernetes. For a disposable K3s sandbox, install K3s by + the approved environment runbook. If the runbook explicitly allows the + upstream K3s installer, use: + + ```bash + curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--write-kubeconfig-mode 644" sh -s - + ``` + + After install, always set: + + ```bash + export KUBECONFIG=/etc/rancher/k3s/k3s.yaml + ``` + +3. Prove command context: + + ```bash + export KUBECONFIG=/etc/rancher/k3s/k3s.yaml + kubectl config current-context + kubectl get nodes -o wide + kubectl get nodes -o json \ + | jq '.items[] | {name: .metadata.name, os: .status.nodeInfo.osImage, kernel: .status.nodeInfo.kernelVersion, runtime: .status.nodeInfo.containerRuntimeVersion}' + KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm version + ``` + +4. Capture runtime and handler evidence after K3s exists: + + ```bash + containerd --version || true + sudo k3s crictl info 2>/dev/null | jq '.config.containerd' || true + kubectl get runtimeclass || true + ``` + +5. Only now choose GPU Operator install values. + +## Command-Context Proof Checklist + +Write these facts into the transcript before mutation: + +- Shell hostname and user. +- `KUBECONFIG` path or kube context. +- `kubectl get nodes -o wide` output. +- Kubernetes version and runtime version. +- GPU model and host driver state. +- Helm repo and chart version. +- Whether NFD labels already exist. +- Whether namespace Pod Security Admission requires a privileged label. + +## Staged Branch Decisions + +| Decision | Stage 1 before K8s | Stage 2 after K8s/K3s bootstrap | +|---|---|---| +| Driver ownership | Check host `nvidia-smi` and user intent | If host driver works and platform owns it, use `driver.enabled=false`; otherwise use operator-managed driver in disposable sandbox | +| NFD ownership | Unknown until node labels exist | If feature labels already exist from an owned NFD, set `nfd.enabled=false`; otherwise chart-managed NFD | +| Runtime/toolkit path | Runtime may not exist yet | Prefer CDI default; enable NRI on K3s-like runtimes in known-tested range; for newer runtime, proceed only with caveat and extra validation | +| Validation | Do not run workloads | Apply manifests under `manifests/` after allocatable GPU exists | + +## Validation And Reporting + +When validating documentation examples or end-to-end documented procedures on a +disposable sandbox, capture a report that states: + +- how the sandbox was provisioned; +- product package path and version; +- instance type, ID, GPU model, and cleanup command; +- commands run and risk class; +- pass/fail for context proof, install, workload suite, maintenance continuity, + troubleshooting scenario, and cleanup. + +Portability note: do not assume GNU `timeout` exists on the machine running +these commands. If it is missing (for example on macOS), use your shell or SSH +command boundaries, or record its absence, instead of relying on `timeout`. + +For K3s sandboxes, `kubectl` and `helm` may not share the same context unless +`KUBECONFIG=/etc/rancher/k3s/k3s.yaml` is exported or supplied inline. Prove +Helm context separately before `helm install`, `helm upgrade`, or `helm get`. + +Cleanup verification checklist: + +- disposable namespaces, pods, jobs, deployments, and RuntimeClasses are gone; +- optional-example ConfigMaps are gone or no longer referenced; +- live `ClusterPolicy` matches the intended baseline; +- Helm values match the intended baseline; +- node GPU labels, capacity, and allocatable resources match the intended + baseline; +- final CUDA VectorAdd smoke passes. diff --git a/gpu-operator-devops-agent/references/branch-matrix.md b/gpu-operator-devops-agent/references/branch-matrix.md new file mode 100644 index 000000000..50af4bb26 --- /dev/null +++ b/gpu-operator-devops-agent/references/branch-matrix.md @@ -0,0 +1,19 @@ +# GPU Operator Product-Specific Branch Matrix + +Generated from a curated source manifest. This file captures +the product/operator judgement a fresh agent should not need to rediscover +on each run. + +Use this matrix before selecting install, maintenance, troubleshooting, or +recovery branches. A branch is not ready for the default path unless it has a +matching command card and, for fragile shell sequences, a runnable script. + +| decision | branch | choose_when | reject | evidence | validation_signal | fallback | stop_escalate | command_card | script | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Deployment inventory | Full inventory before diagnosis | Any install, maintenance, or troubleshooting task starts | Symptom-only diagnosis | scripts/inventory-gpu-operator-environment.sh output | Inventory files include host, kube, runtime, GPU, Helm, ClusterPolicy, events, and workloads | Run the read-only inventory script manually command by command | Context is not the assigned target or required access is missing | planned: environment inventory command card (script-backed today by scripts/inventory-gpu-operator-environment.sh) | scripts/inventory-gpu-operator-environment.sh | +| Driver ownership | Host-managed driver | Host nvidia-smi works and platform/user owns driver lifecycle | Operator-managed driver | nvidia-smi, OS/kernel, user approval boundary | GPU Operator installed with driver.enabled=false and VectorAdd passes | Operator-managed driver only in approved disposable sandbox | nvidia-smi fails, Xid appears, or driver mutation is not approved | Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env | scripts/install-gpu-operator-k3s-host-driver.sh | +| K3s runtime integration | Explicit K3s toolkit env | K3s/containerd is 2.3.x or otherwise newer than the documented NRI range and K3s paths exist | Default NRI branch for normal install | Runtime string, K3s config/socket path tests | Toolkit and device plugin converge; node advertises nvidia.com/gpu; VectorAdd passes | CDI/NRI sandbox-validation branch if the goal is to test newer-than-docs NRI | Config/socket path is unknown or mutation is production/shared | Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env | scripts/install-gpu-operator-k3s-host-driver.sh | +| CDI/NRI validation | Newer-than-documented NRI experiment | The user explicitly approves testing NRI on newer-than-table runtime in a disposable sandbox | Conservative explicit toolkit env as default path | Runtime version, docs NRI range, sandbox approval | Same workload passes before/after install and no runtime handler errors appear | Reinstall or upgrade with explicit toolkit env branch | Production/shared target or no approval to validate unsupported/newer runtime | Install GPU Operator With Host Driver And CDI/NRI Sandbox Branch | planned: CDI/NRI sandbox-experiment install script (command currently inline in the command card) | +| Runtime handler error | Toolkit/runtime branch | Pod event says no runtime handler or CDI device is unavailable | Operator reinstall or app-image debugging | RuntimeClasses, ClusterPolicy CDI/NRI, toolkit logs, containerd config | Corrected runtime branch and workload passes | Escalation bundle with runtime evidence | Host runtime edit/restart/reboot required without approval | Public Field Scenario Evidence Triage | scripts/inventory-gpu-operator-environment.sh | +| Maintenance continuity | Chart-only continuity | User approves chart maintenance without driver change | Driver change, drain, reboot, broad reinstall | Helm values, active workloads, pre-change VectorAdd | Same workload passes before and after upgrade; ClusterPolicy ready | Roll back chart with captured values | Workload owner approval missing or driver mutation required | Same-Version Helm Maintenance Reconciliation | scripts/validate-cuda-vectoradd.sh | +| Time-slicing example | Temporary test-only sharing | Single test-owned node, no non-test GPU workloads, MIG disabled | Production/shared mutation or active workloads | Workload inventory, MIG labels, node labels | Shared GPU count appears, test workload passes, cleanup restores GPU count 1 | Skip optional example with rationale | Active workloads or cleanup cannot restore baseline | Apply, Verify, And Revert Time-Slicing | scripts/apply-time-slicing.sh | diff --git a/gpu-operator-devops-agent/references/claim-ledger.md b/gpu-operator-devops-agent/references/claim-ledger.md new file mode 100644 index 000000000..78ab3b379 --- /dev/null +++ b/gpu-operator-devops-agent/references/claim-ledger.md @@ -0,0 +1,24 @@ +# Claim Ledger + +| Claim | Source | Scope | Verification | Fallback if false | +|---|---|---|---|---| +| GPU Operator default install deploys NFD, driver, toolkit, device plugin, DCGM Exporter, MIG Manager, and validators according to chart values and node eligibility. | docs `getting-started.rst`; source `values.yaml` | v26.3.3 Helm | `helm show values`; pods/ds | inspect custom values and disabled operands | +| `ClusterPolicy` ready is necessary but not sufficient for install success. | docs verification section | all installs | pods, ds, allocatable GPU, workload manifest | troubleshoot earliest failed layer | +| CUDA VectorAdd validation should use a manifest, not `kubectl run --limits`. | docs `getting-started.rst` | workload validation | `kubectl apply -f manifests/cuda-vectoradd.yaml`; logs | create equivalent manifest for target namespace | +| CDI is enabled by default in GPU Operator v25.10.0+. | docs `cdi.rst`; source `values.yaml` | v25.10+ | ClusterPolicy `.spec.cdi.enabled` | explicitly enable or document override | +| NRI requires CDI and supported container runtime versions. | docs `cdi.rst`; chart `validations.yaml` | NRI path | runtime version, Helm render | do not enable NRI; use legacy toolkit branch | +| With CDI+NRI, non-standard K3s/k0s/RKE2 containerd paths do not require `CONTAINERD_CONFIG`, `CONTAINERD_SOCKET`, or `RUNTIME_CONFIG_SOURCE`. | docs `cdi.rst`, `getting-started.rst` | NRI mode | toolkit logs, runtime behavior | legacy env settings after exact path proof | +| Documented NRI containerd ranges are `1.7.30`, `2.1.x`, and `2.2.x`; newer containerd is not automatically known-tested. | docs `cdi.rst`, `platform-support.rst` | support matrix | runtime version | caveat plus extra validation or stop | +| GPU Operator v26.3.3 with K3s `v1.36.2+k3s1`, containerd `2.3.2-k3s2`, host driver `580.159.03`, CDI+NRI, and A100 passed live Brev install, VectorAdd, time-slicing, same-version Helm reconciliation, and bad RuntimeClass recovery. | live A100 GPU instance | sandbox evidence, newer-than-documented NRI range | repeat workload/example suite in target environment | do not generalize beyond this sandbox without fresh public-source and live evidence | +| K3s kubeconfig commonly lives at `/etc/rancher/k3s/k3s.yaml`; shells that do not load it must set `KUBECONFIG` explicitly before Helm/kubectl mutation. | K3s documentation; live target evidence | K3s sandbox | `echo $KUBECONFIG`, `kubectl get nodes` | export `/etc/rancher/k3s/k3s.yaml` | +| In K3s/Brev shells, Helm may require an explicit `KUBECONFIG=/etc/rancher/k3s/k3s.yaml` even when `kubectl` commands work from the shell. | live A100 GPU instance | K3s sandbox command context | `helm list -A` with explicit kubeconfig | set inline `KUBECONFIG` for every Helm command | +| Time-slicing rollout availability proves scheduling capacity, but each verification pod should log `Test PASSED` before declaring CUDA execution success. | docs `gpu-sharing.rst`; live A100 GPU instance | optional example validation | pod logs per replica | wait for logs or reduce replica count | +| Direct `ClusterPolicy` patches and Helm values can diverge; cleanup must verify both live ClusterPolicy and Helm values. | live A100 GPU instance | optional example and maintenance cleanup | `kubectl get clusterpolicy`; `helm get values` | patch live ClusterPolicy and reconcile Helm values | +| NFD should be disabled if an owned NFD already labels nodes. | docs `getting-started.rst`; chart values | install | feature labels and existing NFD pods | chart-managed NFD | +| Pod Security Admission restricted/baseline can block privileged operands. | docs `getting-started.rst` | PSA clusters | namespace events/labels | label `gpu-operator` privileged with approval | +| GPU Operator only manages containerized drivers, not host-preinstalled driver lifecycle. | docs `gpu-driver-upgrades.rst` | driver maintenance | `driver.enabled` and host evidence | use host/provider runbook | +| Chart upgrade CRD hook is enabled by default in v24.9.0+ and requires `--disable-openapi-validation`. | docs `upgrade.rst`; `values.yaml` | chart upgrade | values and Helm output | manual CRD path | +| Driver upgrade controller is enabled by default and uses node state labels/events. | docs `gpu-driver-upgrades.rst` | driver upgrades | node labels/events | k8s-driver-manager fallback | +| `driver.upgradePolicy.drain.enable=true` is disruptive and should be last resort after GPU pod deletion controls. | docs `gpu-driver-upgrades.rst` | driver upgrades | ClusterPolicy | tune GPU pod deletion or stop | +| MIG changes can terminate workloads and may require reboot. | docs `gpu-operator-mig.rst` | MIG-capable GPUs | workload inventory, MIG labels | schedule window or do not change | +| Critical Xid errors can make device plugin mark GPUs unhealthy. | docs `troubleshooting.rst` | runtime GPU health | dmesg/device-plugin logs | hardware/driver escalation | diff --git a/gpu-operator-devops-agent/references/command-cards.md b/gpu-operator-devops-agent/references/command-cards.md new file mode 100644 index 000000000..995b98102 --- /dev/null +++ b/gpu-operator-devops-agent/references/command-cards.md @@ -0,0 +1,708 @@ +# GPU Operator Validated Command Cards + +Generated from a curated source manifest. These cards are +structural artifacts; their validation status is only as strong as the +transcript, fixture, or source evidence recorded in each card. + +## Prove Brev/K3s Command Context + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB, Ubuntu 22.04.5, K3s v1.36.2+k3s1, containerd 2.3.2-k3s2 +Sources: validated on a live A100 GPU node +Risk: `read-only` + +Use when: +- the agent has an assigned Brev/K3s sandbox +- any later command will mutate Kubernetes or GPU Operator state + +Do not use when: +- the target is not the assigned test-owned sandbox +- KUBECONFIG points to a different cluster + +Context proof: + +```bash +hostname +id +pwd +uname -a +cat /etc/os-release +nvidia-smi --query-gpu=name,driver_version,memory.total,mig.mode.current --format=csv,noheader +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl config current-context +kubectl get nodes -o wide +kubectl get nodes -o json \ + | jq '.items[] | {name: .metadata.name, os: .status.nodeInfo.osImage, kernel: .status.nodeInfo.kernelVersion, runtime: .status.nodeInfo.containerRuntimeVersion}' +KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -A +``` + +Command: + +```bash +hostname +id +pwd +uname -a +cat /etc/os-release +nvidia-smi --query-gpu=name,driver_version,memory.total,mig.mode.current --format=csv,noheader +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl config current-context +kubectl get nodes -o wide +kubectl get nodes -o json \ + | jq '.items[] | {name: .metadata.name, os: .status.nodeInfo.osImage, kernel: .status.nodeInfo.kernelVersion, runtime: .status.nodeInfo.containerRuntimeVersion}' +KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -A +``` + +Expected success signals: +- hostname matches the assigned sandbox +- nvidia-smi shows the expected GPU and healthy driver +- K3s node is Ready +- runtime version is recorded +- Helm reaches the same K3s context + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `The connection to the server localhost:8080 was refused` | missing kubeconfig | export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; do not reinstall K3s | +| `helm list cannot connect but kubectl works` | Helm context missing | run Helm with inline KUBECONFIG=/etc/rancher/k3s/k3s.yaml | +| `nvidia-smi fails` | host GPU/driver layer | stop product install and collect host evidence | + +Fallback: + +```bash +KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get nodes -o wide +KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm version +``` + +Cleanup verification: + +```bash +true +``` + +Escalate/stop if: +- host GPU is absent or unhealthy +- the shell is not the assigned sandbox +- kube context cannot be proven +## Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env + +Status: `validated-fixture` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB, K3s v1.36.2+k3s1, containerd 2.3.2-k3s2 +Sources: distilled branch judgment; official NVIDIA GPU Operator docs for install and K3s/runtime configuration +Risk: `high-mutation` + +Use when: +- the host driver is already healthy +- K3s/containerd is 2.3.x or otherwise newer than the documented NRI range +- K3s config/socket paths exist +- the environment is an approved disposable K3s sandbox or explicitly approved target + +Do not use when: +- the cluster is production/shared +- K3s config/socket paths are unknown +- the host driver is missing or unhealthy + +Context proof: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +nvidia-smi +kubectl get nodes -o wide +kubectl get nodes -o json \ + | jq '.items[] | {runtime: .status.nodeInfo.containerRuntimeVersion}' +kubectl get ns gpu-operator --show-labels || true +test -f /var/lib/rancher/k3s/agent/etc/containerd/config.toml +test -S /run/k3s/containerd/containerd.sock +``` + +Command: + +```bash +scripts/install-gpu-operator-k3s-host-driver.sh +``` + +Expected success signals: +- Helm release is deployed +- ClusterPolicy reaches ready +- operands are Running or Completed +- node advertises nvidia.com/gpu=1 +- CUDA VectorAdd card passes + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `PodSecurity baseline blocks privileged pod` | namespace admission | label gpu-operator namespace privileged if approved | +| `no allocatable GPU after ready` | device-plugin/runtime/hardware | collect device-plugin logs, node labels, host nvidia-smi, Xid evidence | +| `no runtime handler under workload` | toolkit/runtime | inspect toolkit logs and runtime classes; compare actual K3s paths to Helm values | + +Fallback: + +```bash +Use the CDI/NRI sandbox-experiment branch only when the user explicitly wants to validate newer-than-documented NRI behavior in a disposable sandbox: +helm upgrade --install gpu-operator nvidia/gpu-operator \ + -n gpu-operator \ + --version v26.3.3 \ + --wait --timeout 10m \ + --set driver.enabled=false \ + --set cdi.nriPluginEnabled=true +``` + +Cleanup verification: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +helm list -n gpu-operator +kubectl get clusterpolicy +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +``` + +Escalate/stop if: +- the branch fails twice with the same runtime/toolkit symptom +- fixing requires driver mutation, node reboot, CRD deletion, or host runtime edits outside the approved sandbox +## Install GPU Operator With Host Driver And CDI/NRI Sandbox Branch + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB, K3s v1.36.2+k3s1, containerd 2.3.2-k3s2 +Sources: validated on a live A100 GPU node, official NVIDIA GPU Operator docs for install and CDI/NRI +Risk: `high-mutation` + +Use when: +- the user explicitly wants to validate newer-than-documented NRI behavior in a disposable sandbox +- the host driver is already healthy +- the operator will prove the result with workloads and cleanup + +Do not use when: +- the cluster is production/shared +- the user asked for the conservative default K3s path +- the host driver is missing or unhealthy + +Context proof: + +```bash +Same as the Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env card. +``` + +Command: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +helm upgrade --install gpu-operator nvidia/gpu-operator \ + -n gpu-operator \ + --version v26.3.3 \ + --wait --timeout 10m \ + --set driver.enabled=false \ + --set cdi.nriPluginEnabled=true +``` + +Expected success signals: +- Same as the Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env card. + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `PodSecurity baseline blocks privileged pod` | namespace admission | label gpu-operator namespace privileged if approved | +| `no allocatable GPU after ready` | device-plugin/runtime/hardware | collect device-plugin logs, node labels, host nvidia-smi, Xid evidence | +| `no runtime handler under workload` | toolkit/runtime | inspect toolkit logs and runtime classes; compare actual K3s paths to Helm values | + +Fallback: + +```bash +Any runtime/CDI/NRI ambiguity should fall back to the explicit K3s toolkit-env branch. +``` + +Cleanup verification: + +```bash +Same as the Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env card. +``` + +Escalate/stop if: +- Same as the Install GPU Operator On K3s With Host Driver And Explicit Toolkit Env card. +## Validate CUDA VectorAdd + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB +Sources: validated on a live A100 GPU node, manifests/cuda-vectoradd.yaml +Risk: `medium-mutation` + +Use when: +- GPU Operator is installed +- node advertises at least one allocatable nvidia.com/gpu + +Do not use when: +- node has no allocatable GPU +- device-plugin logs indicate Xid or unhealthy device + +Context proof: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +``` + +Command: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl delete pod cuda-vectoradd --ignore-not-found +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl wait pod/cuda-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s +kubectl logs cuda-vectoradd | grep 'Test PASSED' +kubectl delete pod cuda-vectoradd --ignore-not-found +``` + +Expected success signals: +- pod phase is Succeeded +- logs contain Test PASSED +- pod is removed after cleanup + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `wait for Ready times out` | wrong success condition | use phase Succeeded plus logs | +| `Insufficient nvidia.com/gpu` | scheduling/device-plugin | inspect node allocatable and device-plugin logs | +| `JSONPath quoting fails` | shell quoting | use kubectl get pod cuda-vectoradd -o json \| jq -r .status.phase | + +Fallback: + +```bash +kubectl get pod cuda-vectoradd -o json | jq -r .status.phase +kubectl logs cuda-vectoradd +``` + +Cleanup verification: + +```bash +kubectl get pod cuda-vectoradd 2>&1 | grep -E 'NotFound|not found' +``` + +Escalate/stop if: +- logs do not show Test PASSED after upstream health appears ready +- GPU is marked unhealthy or Xid appears +## Apply, Verify, And Revert Time-Slicing + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, single A100, K3s +Sources: official NVIDIA GPU sharing docs, validated on a live A100 GPU node +Risk: `high-mutation` + +Use when: +- the user wants a safe optional example for a single test-owned GPU node +- MIG is disabled and no production workloads are running +- temporary shared GPU advertising is approved + +Do not use when: +- workloads are running on a shared/production node +- MIG reconfiguration is requested but not approved +- cleanup cannot be verified + +Context proof: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu',PRODUCT:.metadata.labels.'nvidia\.com/gpu\.product',REPLICAS:.metadata.labels.'nvidia\.com/gpu\.replicas',MIG:.metadata.labels.'nvidia\.com/mig\.config\.state' +kubectl get pods -A -o wide | grep -i nvidia.com/gpu || true +``` + +Command: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl create configmap time-slicing-config-all -n gpu-operator \ + --from-literal=any='version: v1 +sharing: + timeSlicing: + resources: + - name: nvidia.com/gpu + replicas: 4' \ + --dry-run=client -o yaml | kubectl apply -f - +kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=merge \ + -p '{"spec":{"devicePlugin":{"config":{"name":"time-slicing-config-all","default":"any"}}}}' +kubectl rollout status ds/nvidia-device-plugin-daemonset -n gpu-operator --timeout=180s +kubectl wait --for=condition=Ready pod -n gpu-operator -l app=nvidia-device-plugin-daemonset --timeout=180s +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu',PRODUCT:.metadata.labels.'nvidia\.com/gpu\.product',REPLICAS:.metadata.labels.'nvidia\.com/gpu\.replicas' +kubectl apply -f manifests/time-slicing-verification.yaml +kubectl rollout status deployment/time-slicing-verification --timeout=180s +for pod in $(kubectl get pods -l app=time-slicing-verification -o name); do + kubectl logs "$pod" | grep 'Test PASSED' +done +``` + +Expected success signals: +- allocatable GPU changes from 1 to configured replica count +- product label gains -SHARED +- verification workload logs show Test PASSED + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `ClusterPolicy patch applied but Helm values differ` | live/Helm drift | cleanup must reset both live ClusterPolicy and Helm values | +| `pods pending` | GPU sharing/device-plugin | inspect device-plugin rollout, node labels, pod describe | +| `some logs missing before cleanup` | evidence gap | wait for logs or reduce replica count | + +Fallback: + +```bash +Reduce replicas to 2 and rerun verification, or stop and collect device-plugin logs if rollout does not converge. +``` + +Cleanup verification: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl delete deployment time-slicing-verification --ignore-not-found +kubectl patch clusterpolicies.nvidia.com/cluster-policy --type=json \ + -p='[{"op":"remove","path":"/spec/devicePlugin/config"}]' || true +kubectl delete configmap time-slicing-config-all -n gpu-operator --ignore-not-found +kubectl rollout status ds/nvidia-device-plugin-daemonset -n gpu-operator --timeout=180s +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu',PRODUCT:.metadata.labels.'nvidia\.com/gpu\.product',REPLICAS:.metadata.labels.'nvidia\.com/gpu\.replicas' +kubectl get configmap time-slicing-config-all -n gpu-operator 2>&1 | grep -E 'NotFound|not found' +``` + +Escalate/stop if: +- cleanup cannot restore allocatable GPU to baseline +- workloads outside the test namespace are affected +## Same-Version Helm Maintenance Reconciliation + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB +Sources: validated on a live A100 GPU node +Risk: `high-mutation` + +Use when: +- testing chart maintenance continuity without changing driver or product version +- the user approved chart-level mutation + +Do not use when: +- driver changes, drain, reboot, downgrade, or CRD deletion would be required +- pre-change workload validation fails + +Context proof: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +helm list -n gpu-operator +helm get values gpu-operator -n gpu-operator -o yaml +kubectl get clusterpolicy +``` + +Command: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl delete pod cuda-vectoradd --ignore-not-found +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl wait pod/cuda-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s +kubectl logs cuda-vectoradd | grep 'Test PASSED' +kubectl delete pod cuda-vectoradd --ignore-not-found + +helm upgrade gpu-operator nvidia/gpu-operator \ + -n gpu-operator \ + --version v26.3.3 \ + --reuse-values \ + --wait --timeout 10m + +kubectl get clusterpolicy +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl wait pod/cuda-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s +kubectl logs cuda-vectoradd | grep 'Test PASSED' +kubectl delete pod cuda-vectoradd --ignore-not-found +``` + +Expected success signals: +- Helm revision increments +- values remain aligned with intended branch +- ClusterPolicy remains ready +- same workload passes before and after + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `Helm context cannot connect` | missing kubeconfig | rerun with inline KUBECONFIG | +| `post-change VectorAdd fails` | regression or runtime drift | collect Helm values, ClusterPolicy, events, operand logs | +| `Helm values reset but live policy still patched` | state drift | reconcile both Helm values and live ClusterPolicy | + +Fallback: + +```bash +If Helm revision changed and workload fails, stop and collect evidence before rollback. Use helm rollback only with approval. +``` + +Cleanup verification: + +```bash +kubectl get pod cuda-vectoradd 2>&1 | grep -E 'NotFound|not found' +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +``` + +Escalate/stop if: +- rollback, drain, reboot, or driver mutation appears necessary +## Bad RuntimeClass Failure Injection And Cleanup + +Status: `validated-live` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3, Brev A100 80GB +Sources: validated on a live A100 GPU node, manifests/bad-runtimeclass-scenario.yaml +Risk: `medium-mutation` + +Use when: +- testing whether the agent diagnoses runtime-handler failures without broad reinstall +- the environment is an approved sandbox + +Do not use when: +- the target is shared/production +- scenario cleanup cannot be verified + +Context proof: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get clusterpolicy +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +kubectl get runtimeclass +``` + +Command: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl apply -f manifests/bad-runtimeclass-scenario.yaml +kubectl describe pod abtest-bad-runtime | sed -n '/Events:/,$p' +kubectl get events --sort-by=.lastTimestamp | tail -30 +``` + +Expected success signals: +- pod fails before workload execution +- event says no runtime for definitely-not-real-nvidia-handler is configured +- ClusterPolicy remains ready and GPU allocatable remains 1 + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `no runtime for synthetic handler` | injected RuntimeClass | delete bad pod/RuntimeClass; do not reinstall | +| `ClusterPolicy not ready before injection` | preexisting product issue | stop scenario and diagnose baseline first | +| `allocatable GPU missing before injection` | device-plugin/hardware | do not run scenario; collect evidence | + +Fallback: + +```bash +If the manifest path is unavailable, create an equivalent RuntimeClass and pod with a clearly synthetic handler name, then record the exact YAML used. +``` + +Cleanup verification: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl delete pod abtest-bad-runtime --ignore-not-found +kubectl delete runtimeclass abtest-bad-nvidia --ignore-not-found +kubectl get pod abtest-bad-runtime 2>&1 | grep -E 'NotFound|not found' +kubectl get runtimeclass abtest-bad-nvidia 2>&1 | grep -E 'NotFound|not found' +kubectl get clusterpolicy +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +``` + +Escalate/stop if: +- cleanup cannot remove the synthetic RuntimeClass +- the scenario changes real GPU Operator state beyond the test pod/runtimeclass +## Public Field Scenario Evidence Triage + +Status: `validated-fixture` +Last validated: 2026-07-02 +Validated on: GPU Operator v26.3.3 package fixture evidence and public issue/forum seeds +Sources: references/field-scenario-cards.md, public issue/forum seeds listed there +Risk: `read-only` + +Use when: +- the prompt describes a public-field-shaped failure +- the agent must diagnose from evidence without browsing public issues/forums + +Do not use when: +- the live cluster context is unproven +- the task asks for a product code fix rather than operational diagnosis + +Context proof: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl config current-context +kubectl get nodes -o wide +``` + +Command: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl get clusterpolicy -o yaml || true +kubectl get pods -n gpu-operator -o wide || true +kubectl get ds -n gpu-operator || true +kubectl get runtimeclass || true +kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu-driver-upgrade-state,nvidia.com/gpu.deploy.device-plugin,nvidia.com/mig.config.state +kubectl get events -A --sort-by='.lastTimestamp' | tail -100 +helm list -A || true +helm get values gpu-operator -n gpu-operator -o yaml || true +``` + +Expected success signals: +- context, pods, daemonsets, node labels, events, Helm values, and ClusterPolicy evidence are captured +- no mutation occurs + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `namespace not found` | GPU Operator absent or different namespace | search Helm releases and namespaces before assuming install failure | +| `cluster connection refused` | kubeconfig/context | set intended kubeconfig and rerun proof | +| `no ClusterPolicy CRD` | install did not reach CRD stage | collect Helm release/events; do not diagnose operands yet | + +Fallback: + +```bash +kubectl get ns +helm list -A +``` + +Cleanup verification: + +```bash +true +``` + +Escalate/stop if: +- context cannot be proven +- required evidence is unavailable and mutation would be speculative +## Device Plugin Disabled Validator Evidence + +Status: `docs-derived` +Last validated: not-run as live mutation; distilled from public issue seed +Validated on: public issue evidence and package fixture +Sources: https://github.com/NVIDIA/gpu-operator/issues/2550 +Risk: `read-only` + +Use when: +- validator/plugin-validation fails while a node may intentionally disable the device plugin with nvidia.com/gpu.deploy.device-plugin=false + +Do not use when: +- the task is to enable the device plugin on all nodes +- node label ownership is unknown and mutation is requested + +Context proof: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl get nodes --show-labels | grep 'nvidia.com/gpu.deploy.device-plugin=false' || true +kubectl get ds -n gpu-operator nvidia-device-plugin-daemonset -o wide || true +``` + +Command: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl get pods -n gpu-operator -l app=nvidia-operator-validator -o wide +kubectl logs -n gpu-operator -l app=nvidia-operator-validator -c plugin-validation --tail=100 || true +kubectl describe pod -n gpu-operator -l app=nvidia-operator-validator | sed -n '/Events:/,$p' +``` + +Expected success signals: +- node disablement labels and validator/plugin logs are captured +- diagnosis distinguishes intentional device-plugin disablement from generic missing GPU resources + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `GPU resources are not yet discovered` | device plugin disabled or not advertising | compare node labels with daemonset selectors before reinstall | +| `no matching validator pod` | namespace/label drift | inspect all gpu-operator pods and daemonsets | +| `no disabled label found` | different missing GPU cause | continue through success ladder and device-plugin logs | + +Fallback: + +```bash +kubectl get pods -n gpu-operator -o wide +kubectl get nodes --show-labels +``` + +Cleanup verification: + +```bash +true +``` + +Escalate/stop if: +- fixing would require removing node labels owned by another workflow +- the product appears to validate a node intentionally excluded from device plugin deployment +## Upgrade State Audit + +Status: `docs-derived` +Last validated: not-run as live driver upgrade; distilled from public issue seed +Validated on: public issue evidence and package fixture +Sources: https://github.com/NVIDIA/gpu-operator/issues/2549 +Risk: `read-only` + +Use when: +- upgrade labels and driver/workload health disagree +- a maintenance task asks whether an upgrade succeeded + +Do not use when: +- the user has not approved maintenance diagnosis +- driver mutation, drain, reboot, or relabeling is requested as the first step + +Context proof: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl get nodes -L nvidia.com/gpu-driver-upgrade-state +helm list -n gpu-operator +``` + +Command: + +```bash +export KUBECONFIG=${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml} +kubectl get nodes -L nvidia.com/gpu-driver-upgrade-state +kubectl get pods -n gpu-operator -o wide +kubectl describe ds -n gpu-operator nvidia-driver-daemonset || true +kubectl get events -A --sort-by='.lastTimestamp' | tail -100 +helm get values gpu-operator -n gpu-operator -o yaml || true +kubectl get clusterpolicy -o yaml || true +``` + +Expected success signals: +- upgrade labels, driver pod status, Helm values, and events are captured +- success is withheld until workload proof also passes + +Common failures and interpretation: + +| Failure signal | Likely layer | Next action | +|---|---|---| +| `upgrade-failed then upgrade-done` | upgrade controller state ambiguity | require workload proof and driver pod health before accepting success | +| `driver pod CrashLoopBackOff` | driver upgrade path | inspect active GPU workloads and driver logs before relabeling | +| `no upgrade label` | chart-only change or no driver upgrade | use normal maintenance continuity branch | + +Fallback: + +```bash +kubectl get pods -A -o wide | grep -i nvidia || true +kubectl get nodes --show-labels | grep gpu-driver-upgrade-state || true +``` + +Cleanup verification: + +```bash +true +``` + +Escalate/stop if: +- relabeling, drain, reboot, or driver downgrade is proposed without explicit approval and rollback plan diff --git a/gpu-operator-devops-agent/references/command-catalog.md b/gpu-operator-devops-agent/references/command-catalog.md new file mode 100644 index 000000000..8f16d81e7 --- /dev/null +++ b/gpu-operator-devops-agent/references/command-catalog.md @@ -0,0 +1,46 @@ +# Command Catalog + +Risk levels: `read-only`, `local-mutation`, `medium-mutation`, +`high-mutation`. + +| Command | Purpose | Expected signal | Risk | +|---|---|---|---| +| `hostname; id; pwd` | Prove shell host/user/location | intended sandbox shell | read-only | +| `export KUBECONFIG=/etc/rancher/k3s/k3s.yaml` | Select fresh K3s cluster context | later `kubectl` reaches K3s | local-mutation | +| `kubectl config current-context` | Prove kube context | expected context name | read-only | +| `kubectl get nodes -o wide` | Node readiness/runtime | node `Ready`; runtime shown | read-only | +| `kubectl get nodes -o json | jq ...nodeInfo...` | OS/kernel/runtime inventory | OS, kernel, runtime versions | read-only | +| `nvidia-smi` | Host GPU/driver evidence | GPU table, driver version | read-only host | +| `lspci | grep -i nvidia` | Host PCI GPU evidence | NVIDIA device present | read-only host | +| `lsmod | grep -E 'nouveau|nvidia'` | Driver module state | expected NVIDIA modules; no `nouveau` conflict | read-only host | +| `containerd --version` | Runtime version | version string | read-only host | +| `kubectl get runtimeclass` | Runtime handler evidence | expected classes or empty under NRI | read-only | +| `helm repo add nvidia https://helm.ngc.nvidia.com/nvidia` | Add chart repo | repo added | local-mutation | +| `helm repo update nvidia` | Refresh chart repo | update succeeds | local-mutation | +| `helm show values nvidia/gpu-operator --version v26.3.3` | Inspect chart defaults | values printed | read-only | +| `KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -A` | Prove Helm can reach K3s context | releases listed or empty table | read-only | +| `kubectl create ns gpu-operator` | Create operator namespace | namespace exists | medium-mutation | +| `kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged` | Permit privileged operands under PSA | label present | medium-mutation | +| `helm install gpu-operator nvidia/gpu-operator ...` | Install GPU Operator | release deployed | high-mutation | +| `helm upgrade gpu-operator nvidia/gpu-operator ... --disable-openapi-validation` | Upgrade chart with CRD hook | revision increments | high-mutation | +| `helm rollback gpu-operator -n gpu-operator` | Roll back chart | previous revision active | high-mutation | +| `kubectl apply -f manifests/cuda-vectoradd.yaml` | Durable CUDA validation | pod completes; logs `Test PASSED` | medium-mutation | +| `kubectl delete -f manifests/cuda-vectoradd.yaml --ignore-not-found` | Clean validation pod | pod absent | medium-mutation | +| `kubectl apply -f manifests/time-slicing-verification.yaml` | Optional time-slicing workload proof | replicas running/logging VectorAdd | medium-mutation | +| `kubectl apply -f manifests/bad-runtimeclass-scenario.yaml` | Troubleshooting scenario injection | expected sandbox runtime error | medium-mutation | +| `kubectl patch clusterpolicies.nvidia.com/cluster-policy ...` | Change CDI/NRI/driver/MIG policy | operands reconcile/restart | high-mutation | +| `kubectl label node nvidia.com/mig.config=... --overwrite` | Change MIG geometry | `mig.config.state=success` | high-mutation | +| `kubectl drain ` | Evict workloads from node | node drained | high-mutation | +| `sudo reboot` | Restart node | node returns | high-mutation | + +High-mutation commands require explicit approval, blast-radius statement, and a +rollback or recovery plan. + +For fragile or mutating command paths, use `references/command-cards.md` before +inventing new syntax. The cards record validation status, exact context proof, +expected signals, known failure strings, fallback, and cleanup verification. + +Prefer manifest-based workload creation over `kubectl run --limits`; some +client versions do not support the limits flag shape consistently. If a quick +imperative command fails because of client flag drift, switch to a small +manifest and record the drift as an anti-pattern. diff --git a/gpu-operator-devops-agent/references/environment-inventory.md b/gpu-operator-devops-agent/references/environment-inventory.md new file mode 100644 index 000000000..3d57e70f3 --- /dev/null +++ b/gpu-operator-devops-agent/references/environment-inventory.md @@ -0,0 +1,48 @@ +# GPU Operator Deployment Environment Inventory + +Collect this inventory before diagnosing an issue or choosing an install, +maintenance, or recovery branch. Do not reason from symptom text alone. The +inventory tells the operating agent which layer owns the current behavior. + +## Required Inventory + +| area | evidence | why_it_matters | +| --- | --- | --- | +| Target identity | hostname, cloud/sandbox ID, user, working directory | Prevents mutation of the wrong host or cluster. | +| Kubernetes context | KUBECONFIG, current context, node list | Most false starts come from missing or wrong kube context. | +| Kubernetes distribution | node version, kubelet version, runtime string | K3s/RKE2/MicroK8s/containerd paths differ from default Kubernetes. | +| Host OS/kernel | /etc/os-release, uname -a | Driver and toolkit behavior depends on OS/kernel support. | +| GPU and driver | nvidia-smi, MIG mode, GPU model, driver version | Selects host-managed vs operator-managed driver branch. | +| Runtime paths | containerd/CRI-O version, config path, socket path | Selects CDI/NRI, explicit toolkit env, or host-toolkit branch. | +| NFD ownership | NFD pods, feature labels, GPU labels | Prevents duplicate NFD or stale-label decisions. | +| GPU Operator state | Helm release, values, ClusterPolicy, operands | Distinguishes install/config issues from workload issues. | +| Runtime integration | RuntimeClasses, CDI/NRI fields, toolkit logs | Diagnoses runtime handler and CDI injection failures. | +| Device plugin state | allocatable GPU, device-plugin logs, node labels | Determines whether workloads can schedule. | +| Workloads | GPU pods, namespaces, active GPU clients | Prevents disruptive maintenance and identifies workload-owned failures. | +| Hardware faults | dmesg/Xid, nvidia-smi -q, GPU health | Xid/hardware faults should stop product-level mutation. | + +## Inventory Script + +Use `scripts/inventory-gpu-operator-environment.sh` when shell access is available. + +Required environment: +- KUBECONFIG, defaulting to /etc/rancher/k3s/k3s.yaml +- INVENTORY_DIR, defaulting to ./gpu-operator-inventory- + +## Branch Decisions That Depend On Inventory + +- driver.enabled requires host nvidia-smi, driver ownership, and platform intent. +- toolkit.enabled requires runtime handler/CDI evidence and whether host toolkit is already functional. +- cdi.nriPluginEnabled requires runtime version and whether the run is intentionally validating newer-than-documented NRI. +- nfd.enabled requires feature label evidence and NFD owner. +- maintenance/upgrade safety requires active workload inventory and pre-change workload proof. +- troubleshooting branch requires earliest-failed-layer evidence from events, logs, labels, and hardware health. + +## Stop Conditions + +- the host or kube context is not the assigned target +- nvidia-smi fails on a host-managed-driver branch +- Xid or hardware/fabric faults appear +- runtime config paths are unknown and the next step would edit host runtime +- active workloads would be disrupted without approval +- source evidence conflicts on a load-bearing support claim diff --git a/gpu-operator-devops-agent/references/failure-signatures.md b/gpu-operator-devops-agent/references/failure-signatures.md new file mode 100644 index 000000000..84359d917 --- /dev/null +++ b/gpu-operator-devops-agent/references/failure-signatures.md @@ -0,0 +1,34 @@ +# Failure Signatures + +| Symptom | Likely earliest layer | First checks | Recovery posture | +|---|---|---|---| +| `kubectl` points at wrong/empty cluster | command context | `echo $KUBECONFIG`, `kubectl config current-context`, nodes | stop before mutation; set intended context | +| Helm cannot reach cluster after K3s install | kubeconfig missing | `KUBECONFIG`, `/etc/rancher/k3s/k3s.yaml` | export K3s kubeconfig, rerun read-only proof | +| GPU Operator pods blocked by Pod Security | namespace PSA | namespace labels, events | label namespace privileged with approval | +| Daemonsets `DESIRED=0` | NFD labels/taints | node labels, taints, daemonset selectors | repair labels/tolerations; do not debug CUDA yet | +| Pods stuck in `Init` | driver/toolkit not ready | driver/toolkit logs, events, dmesg | fix earliest driver/toolkit issue | +| `no runtime for "nvidia" is configured` | runtime handler/toolkit | RuntimeClasses, ClusterPolicy CDI/NRI, toolkit logs | on K3s prefer CDI+NRI if supported; otherwise verify exact legacy paths | +| NRI enabled but chart render fails | invalid values | chart validation, CDI/toolkit values | keep CDI and toolkit enabled or disable NRI | +| `ClusterPolicy ready` but no allocatable GPU | device plugin/GPU health | device-plugin logs, dmesg/Xid | escalate hardware/driver if critical Xid | +| Device plugin marks GPU unhealthy/Xid 79 | GPU fell off bus/hardware | dmesg, device-plugin logs, host GPU UUID | stop product mutation; escalate | +| DCGM exporter failing while workloads pass | observability path | exporter logs/service/network policy | separate from core scheduling; repair observability only | +| Driver upgrade stuck at `pod-deletion-required` | GPU pod eviction blocked | upgrade labels/events, workload pod details | resolve workload/local storage; drain only as last resort | +| Node in `upgrade-failed` | failed driver upgrade stage | events and operator logs | fix cause before relabeling `upgrade-required` | +| MIG config pending/rebooting | active workloads or reboot needed | MIG labels, MIG manager logs, GPU clients | stop workloads/cordon/reboot only with approval | +| Pod with `hostUsers:false` fails sync socket | unsupported user namespace GPU pod | workload spec, CDI docs | remove `hostUsers:false` or set true | +| Mixed MIG/full GPU scheduling fails on affected drivers | known driver/NVML issue | driver version, MIG topology, issue evidence | use documented workaround only with approval | + +Unsupported quick fixes: CRD deletion, generic reinstall, repeated pod restarts, +driver downgrade, full node drain, or MIG relabeling without approval and +evidence. + +## Public Field Scenario Branch Table + +| Scenario shape | First failing layer | Evidence to require | Correct posture | +|---|---|---|---| +| Validator fails on node with `nvidia.com/gpu.deploy.device-plugin=false` | validator/device-plugin ownership mismatch | node labels, daemonset selectors, plugin-validation logs | diagnose intentional disablement; avoid reinstall | +| CDI device injection error after upgrade | toolkit/CDI/runtime integration | RuntimeClasses, ClusterPolicy CDI/NRI, toolkit logs, failing pod event | choose scoped runtime/toolkit branch | +| `upgrade-failed` transitions later to `upgrade-done` | upgrade-controller state ambiguity | node upgrade labels, driver pod status, workload pre/post result | require workload proof before success | +| Driver pod mount error under `/sys/devices/system/memory` | host kernel/driver daemonset compatibility | kernel config, driver pod event, Helm values | escalate support/compatibility; avoid Helm retry | +| `no runtime for "nvidia" is configured` | RuntimeClass/toolkit config | RuntimeClasses, ClusterPolicy, toolkit logs, pod event | choose CDI/NRI or legacy path from evidence | +| Driver image `ImagePullBackOff` for specific OS/kernel/image | support matrix/image availability | OS, kernel, image tag, registry event, values | distinguish unsupported image from credential issue | diff --git a/gpu-operator-devops-agent/references/field-scenario-cards.md b/gpu-operator-devops-agent/references/field-scenario-cards.md new file mode 100644 index 000000000..26e91e712 --- /dev/null +++ b/gpu-operator-devops-agent/references/field-scenario-cards.md @@ -0,0 +1,227 @@ +# GPU Operator Field Scenario Cards + +These cards distill public issue/forum failure shapes into package-local +diagnostic guidance. They are for live support +reasoning: the operating agent should not need to browse the public thread to +know which evidence to gather or which broad fixes to reject. + +Use these cards after `references/models.md` and before choosing a mutation. + +## Validator Fails When Device Plugin Is Intentionally Disabled + +Source seed: https://github.com/NVIDIA/gpu-operator/issues/2550 +Reproduction class: live-safe proxy + +Symptom: +- `nvidia-operator-validator` fails in plugin validation. +- A node has `nvidia.com/gpu.deploy.device-plugin=false`. +- Logs can say GPU resources are not discovered. + +Model interpretation: +- Device plugin ownership has been intentionally disabled for that node. +- Validator behavior can become downstream noise if it still expects GPU + resources on a node where the device plugin is not supposed to advertise + them. + +Evidence to collect: + +```bash +kubectl get nodes --show-labels | grep 'nvidia.com/gpu.deploy.device-plugin=false' || true +kubectl get ds -n gpu-operator nvidia-device-plugin-daemonset -o wide +kubectl get pods -n gpu-operator -l app=nvidia-operator-validator -o wide +kubectl logs -n gpu-operator -l app=nvidia-operator-validator -c plugin-validation --tail=100 +kubectl describe pod -n gpu-operator -l app=nvidia-operator-validator +``` + +Expected branch: +- Diagnose as node-label/device-plugin-disabled versus generic install failure. +- Do not reinstall GPU Operator. +- If this is an intentional split-node deployment, escalate as product behavior + or documentation gap with evidence. + +Unsafe/low-value branches: +- `helm uninstall` / reinstall. +- Debugging CUDA workloads before resolving the validator/device-plugin + ownership mismatch. +- Removing labels without approval from the owner of that node partitioning. + +## CDI Device Injection Or RuntimeClass Failure After Upgrade + +Source seed: https://github.com/NVIDIA/gpu-operator/issues/1876 +Reproduction class: evidence-only or live-safe proxy + +Symptom: +- GPU workload fails with CDI device injection error. +- RuntimeClass such as `nvidia-cdi` is involved. +- Workload without RuntimeClass may see `/dev` devices but miss userspace + files/libraries. + +Model interpretation: +- Failure is at toolkit/CDI/runtime integration, not at the application layer. +- Device ID strategy, CDI defaults, RuntimeClass selection, and toolkit branch + all control the outcome. + +Evidence to collect: + +```bash +kubectl get runtimeclass +kubectl get clusterpolicy -o yaml +kubectl get pods -n gpu-operator -o wide +kubectl logs -n gpu-operator -l app=nvidia-container-toolkit-daemonset --tail=200 +kubectl describe pod -n +kubectl get events -A --sort-by='.lastTimestamp' | tail -80 +``` + +Expected branch: +- Separate CDI/runtime diagnosis from generic operator health. +- Compare workload `runtimeClassName`, ClusterPolicy CDI/NRI settings, and + runtime/device-plugin strategy. +- Use a scoped RuntimeClass/toolkit branch, or escalate if the target mix is + outside reviewed support. + +Unsafe/low-value branches: +- Treating the failure as a container image or CUDA app issue first. +- Reinstalling GPU Operator without preserving Helm values and runtime + evidence. + +## Upgrade State Labels Conflict With Driver Or Workload Health + +Source seed: https://github.com/NVIDIA/gpu-operator/issues/2549 +Reproduction class: live-safe proxy + +Symptom: +- Node label moves through `upgrade-failed` and later `upgrade-done`. +- Driver pod health and workload continuity are ambiguous. +- Upgrade involves driver branch changes or active GPU workloads. + +Model interpretation: +- Upgrade label is a value-model signal, not final truth. +- Maintenance success requires workload proof before and after change. + +Evidence to collect: + +```bash +kubectl get nodes -L nvidia.com/gpu-driver-upgrade-state +kubectl get pods -n gpu-operator -o wide +kubectl describe ds -n gpu-operator nvidia-driver-daemonset +kubectl get events -A --sort-by='.lastTimestamp' | tail -100 +helm get values gpu-operator -n gpu-operator -o yaml +kubectl get clusterpolicy -o yaml +``` + +Expected branch: +- Monitor label sequence, driver pod state, and workload proof together. +- Do not accept `upgrade-done` unless the declared workload suite still passes. +- If relabeling is considered, record why it is safe and what recovery action it + represents. + +Unsafe/low-value branches: +- Declaring success from labels alone. +- Draining/rebooting/downgrading before proving active workload and driver + ownership. + +## Driver Daemonset Fails On Kernel HostPath Or Kernel Feature + +Source seed: https://github.com/NVIDIA/gpu-operator/issues/2463 +Reproduction class: evidence-only + +Symptom: +- Driver daemonset is `CreateContainerError`. +- Event mentions a hostPath/mount failure under `/sys/devices/system/memory`. +- Kernel lacks the expected feature or userspace cannot create the sysfs path. + +Model interpretation: +- Earliest failed layer is host/kernel compatibility or driver daemonset mount + assumptions. +- Helm retry does not change the host kernel. + +Evidence to collect: + +```bash +kubectl describe pod -n gpu-operator -l app=nvidia-driver-daemonset +kubectl get events -n gpu-operator --sort-by='.lastTimestamp' | tail -80 +uname -a +grep CONFIG_MEMORY_HOTPLUG /boot/config-$(uname -r) || true +ls -l /sys/devices/system/memory/auto_online_blocks || true +helm get values gpu-operator -n gpu-operator -o yaml +``` + +Expected branch: +- Escalate or use a supported host/kernel branch. +- Preserve the exact event and kernel evidence. +- Avoid retrying Helm until the host compatibility question is resolved. + +Unsafe/low-value branches: +- Reinstalling GPU Operator. +- Attempting to create sysfs paths from userspace. +- Mutating driver settings without confirming support. + +## Runtime Handler `nvidia` Is Not Configured + +Source seed: https://forums.developer.nvidia.com/t/failed-to-create-pod-sandbox-rpc-error-code-unknown-desc-failed-to-get-sandbox-runtime-no-runtime-for-nvidia-is-configured/296409 +Reproduction class: live-safe proxy + +Symptom: +- Pod sandbox creation fails with `no runtime for "nvidia" is configured`. +- Cluster may have working host GPU but missing RuntimeClass/toolkit runtime + integration. + +Model interpretation: +- Earliest failed layer is Kubernetes/runtime/toolkit integration. +- The product may be healthy up to controller readiness while workloads still + fail at sandbox creation. + +Evidence to collect: + +```bash +kubectl get runtimeclass +kubectl get clusterpolicy -o yaml +kubectl get pods -n gpu-operator -l app=nvidia-container-toolkit-daemonset -o wide +kubectl logs -n gpu-operator -l app=nvidia-container-toolkit-daemonset --tail=200 +kubectl describe pod -n +``` + +Expected branch: +- Decide between CDI/NRI branch and explicit legacy K3s/RKE2 toolkit paths from + evidence. +- Do not debug application code before the sandbox runtime succeeds. + +Unsafe/low-value branches: +- Full operator uninstall/reinstall as first response. +- Editing host runtime files without approval or without a rollback plan. + +## Driver Image Pull Failure Or Unsupported Platform Image + +Source seed: https://forums.developer.nvidia.com/t/gpu-operator-helm-chat-deployment-issues/349743 +Reproduction class: evidence-only or live-safe proxy + +Symptom: +- Driver pod has `ErrImagePull` / `ImagePullBackOff`. +- Event references an unavailable driver image tag, OS, or kernel branch. + +Model interpretation: +- Earliest failed layer may be image availability/support matrix, not registry + credentials. +- The driver operand cannot progress until image/platform compatibility is + resolved. + +Evidence to collect: + +```bash +kubectl describe pod -n gpu-operator -l app=nvidia-driver-daemonset +kubectl get events -n gpu-operator --sort-by='.lastTimestamp' | tail -80 +cat /etc/os-release +uname -r +helm get values gpu-operator -n gpu-operator -o yaml +kubectl get clusterpolicy -o yaml +``` + +Expected branch: +- Compare OS/kernel/driver branch to supported images. +- If the image tag is unsupported or unavailable, report a support-boundary or + docs/product gap with exact evidence. +- Only debug credentials after proving the image should exist and be reachable. + +Unsafe/low-value branches: +- Retrying image pulls without changing evidence. +- Switching driver branches or OS/kernel without approval. diff --git a/gpu-operator-devops-agent/references/models.md b/gpu-operator-devops-agent/references/models.md new file mode 100644 index 000000000..9d2a7c6b9 --- /dev/null +++ b/gpu-operator-devops-agent/references/models.md @@ -0,0 +1,127 @@ +# GPU Operator Operating Models + +These are FMC-style working models for agent reasoning. Use them before +choosing install, maintenance, troubleshooting, or escalation branches. + +## How To Use These Models + +- Start with the compositional model to locate the owner of the failing layer. +- Use the dynamic model to find the earliest failed transition. +- Use the value model to identify version, flag, label, status, and event + values that can change the branch decision. +- Walk the minimal success ladder from the bottom up before debugging a + downstream symptom. +- Record any live-test model change in runtime memory before changing command + cards or prose. + +## Model-First Diagnostic Loop + +Use this loop for every non-happy-path scenario: + +1. **Locate owner:** decide whether the earliest failing layer is host/provider, Kubernetes/runtime, GPU Operator controller/operand, or user workload. +2. **Find transition:** identify the first dynamic sequence step that failed. +3. **Read values:** collect the version, flag, label, status, event, or log phrase that controls the branch. +4. **Choose branch:** select the lowest-blast-radius branch that targets that layer; explicitly reject broader reinstall/reset branches. +5. **Validate:** prove the next rung of the success ladder before moving downstream. +6. **Persist:** if the evidence changes the model, update runtime memory before changing commands. + +| Symptom shape | First model to use | Evidence that selects branch | Avoid | +| --- | --- | --- | --- | +| Validator fails but device plugin is intentionally disabled | Compositional ownership of device plugin and validator | node labels, daemonset selectors, validator logs | reinstalling operator or debugging CUDA workload | +| Runtime handler missing or CDI device unresolved | Toolkit/CDI/NRI value model and dynamic install sequence | RuntimeClasses, ClusterPolicy CDI/NRI, toolkit logs, pod event | treating as app image failure | +| Upgrade state labels conflict with pod health | Dynamic upgrade sequence and value model | gpu-driver-upgrade-state, driver pod status, workload pre/post proof | accepting upgrade-done without workload validation | +| Driver pod cannot mount host path or load kernel feature | Host/provider ownership and driver daemonset layer | kernel config, mount error, driver pod events | Helm retry loops | +| Image pull failure for driver/toolkit operand | Source/support boundary and operand layer | OS/kernel, image tag, registry event, support matrix | credential debugging before image existence/support check | + +## Compositional Model + +| component | owner | source_of_truth | writes | health_signal | +| --- | --- | --- | --- | --- | +| Sandbox/VM/host | Test environment owner | provisioning record | OS/runtime/K3s installs | hostname, OS/kernel, GPU visible | +| Kubernetes API/kubelet/runtime | Platform | cluster/kubeconfig/runtime config | node/runtime state | nodes Ready, runtime version | +| GPU hardware and host driver | Host/provider unless operator-managed driver | PCI/driver modules | kernel modules, device state | lspci, nvidia-smi, dmesg | +| NFD | GPU Operator by default or platform if pre-existing | node labels | feature labels | NVIDIA PCI labels | +| GPU Operator controller | Helm release | chart values, ClusterPolicy | operand resources | controller pod running | +| ClusterPolicy | GPU Operator CR | Helm-rendered CR and live patches | desired operand config | .status.state=ready/notReady | +| Driver daemonset | GPU Operator if driver.enabled=true; host otherwise | ClusterPolicy/NVIDIADriver/host | kernel driver files/modules | driver pod ready, nvidia-smi | +| Toolkit/CDI/NRI | GPU Operator if toolkit enabled; host otherwise | ClusterPolicy/runtime config | CDI specs, NRI plugin, runtime config | toolkit pod logs, no sandbox runtime errors | +| Device plugin | GPU Operator | ClusterPolicy/device-plugin config | extended resources | nvidia.com/gpu allocatable | +| MIG Manager | GPU Operator | ClusterPolicy, node labels, configmaps | MIG geometry/labels | mig.config.state=success | +| DCGM Exporter | GPU Operator | ClusterPolicy | metrics endpoint/service | exporter pod/logs/service | +| Workloads/examples | User/tester | manifests | GPU requests | VectorAdd Test PASSED | + +## Dynamic Model + +### Clean sandbox sequence + +- Provision/select test-owned host. +- Prove shell host and GPU. +- Install/select Kubernetes. +- Set and prove KUBECONFIG. +- Re-check runtime, NFD labels, RuntimeClasses, host driver. +- Choose GPU Operator branch and Helm values. +### Install sequence + +- Helm installs CRDs, operator deployment, and default ClusterPolicy. +- NFD labels GPU nodes unless an existing NFD branch is selected. +- Operator reconciles driver/toolkit/device-plugin/MIG/DCGM operands. +- Driver becomes usable or host driver is accepted. +- Toolkit configures CDI/NRI or legacy runtime integration. +- Device plugin advertises resources. +- Validators and durable workload manifests prove end-to-end GPU access. +### Upgrade sequence + +- Capture values, manifest, CRDs, ClusterPolicy, node labels, workload inventory, and baseline workload results. +- Choose CRD hook or manual CRD path. +- Helm upgrade chart. +- If driver changes, monitor upgrade controller labels/events. +- Run the same workload suite after change. +- Roll back chart or driver only with the captured rollback point. +### Troubleshooting sequence + +- Identify the earliest failing layer. +- Treat a CUDA workload error as downstream until host GPU, NFD labels, driver/toolkit, and device plugin advertisement are proven. + +## Value Model + +| value | meaning | evidence | +| --- | --- | --- | +| driver.enabled=true | GPU Operator owns containerized driver lifecycle | Helm values, ClusterPolicy | +| driver.enabled=false | Host/platform owns driver lifecycle | Helm values, host nvidia-smi | +| cdi.enabled=true | CDI device injection enabled by default in v25.10.0+ | docs, chart values | +| cdi.nriPluginEnabled=true | Toolkit runs NRI plugin; no nvidia RuntimeClass needed | ClusterPolicy, toolkit logs | +| toolkit.enabled=true | Operator owns toolkit/CDI/NRI config | chart values, toolkit pod | +| nfd.enabled=true | Chart deploys NFD by default | chart values, NFD pods | +| ClusterPolicy.status.state=ready | Operator sees operands ready | kubectl get clusterpolicy | +| nvidia.com/gpu.present=true | GPU node discovered | node labels | +| nvidia.com/gpu allocatable | Scheduler can place GPU workloads | node status | +| nvidia.com/gpu-driver-upgrade-state | Driver upgrade controller state | node labels | +| nvidia.com/mig.config.state | MIG Manager convergence state | node labels | + +## Minimal Success Ladder + +1. Intended sandbox shell proven. +2. Host GPU visible. +3. Kubernetes context proven and nodes Ready. +4. Runtime version and NFD state captured. +5. GPU Operator Helm install succeeds with explicit values. +6. Operator and operands are Running or validator pods Completed. +7. ClusterPolicy is ready. +8. GPU node advertises nvidia.com/gpu. +9. manifests/cuda-vectoradd.yaml completes with Test PASSED. +10. Declared optional example suite passes for the environment. +11. Health remains stable over the declared soak interval. +12. Same workload suite passes after approved upgrade/rollback. + +## Branch Decision Table + +| decision | choose_when | evidence | risk | fallback | +| --- | --- | --- | --- | --- | +| Operator-managed driver | no trusted host driver or disposable sandbox wants one owner | host nvidia-smi absent/not owned | driver mutates host kernel modules | host-managed branch if driver works | +| Host-managed driver | host driver works and platform owns lifecycle | nvidia-smi, user intent | hidden host drift | operator-managed in fresh sandbox | +| Chart NFD | no feature labels or NFD owner | label check false | duplicate NFD if wrong | disable NFD after owner proof | +| Existing NFD | feature labels and existing NFD are owned | labels/pods | stale labels | chart NFD after approval | +| CDI+NRI | K3s/k0s/RKE2 or non-standard containerd, runtime in known range | runtime version, docs | NRI API maturity | legacy toolkit env | +| Newer-than-known NRI | runtime newer than docs range, e.g. containerd 2.3.x | runtime version plus live test caveat | untested runtime behavior | proceed only in sandbox with extra validation, or use legacy/stop | +| Host toolkit/runtime | K3s generated working NVIDIA handlers from host toolkit | RuntimeClasses, host toolkit evidence | host drift | operator toolkit branch | +| MIG change | explicit partitioning requirement | MIG labels/capability, no active workload | workload disruption/reboot | leave MIG disabled | diff --git a/gpu-operator-devops-agent/references/operational-reasoning.md b/gpu-operator-devops-agent/references/operational-reasoning.md new file mode 100644 index 000000000..f1e2bb11d --- /dev/null +++ b/gpu-operator-devops-agent/references/operational-reasoning.md @@ -0,0 +1,105 @@ +# Operational Reasoning + +## Default Sandbox Recommendation + +For a fresh single-node Brev/K3s A100 sandbox targeting GPU Operator `v26.3.3`: + +1. Bootstrap/prove K3s context first; set + `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`. +2. Re-run host driver, runtime, RuntimeClass, and NFD evidence after K3s exists. +3. If the host driver is known-good and platform-owned, use + `driver.enabled=false`; otherwise prefer operator-managed driver in a + disposable sandbox. +4. Keep CDI enabled. +5. For K3s with containerd `1.7.30`, `2.1.x`, or `2.2.x`, enable + `cdi.nriPluginEnabled=true`. +6. For containerd newer than the docs range, such as `2.3.x`, mark the runtime + as newer-untested. In a test-owned sandbox, enabling NRI is reasonable only + with explicit caveat and full workload validation. +7. Leave MIG disabled until whole-GPU scheduling is proven and the user asks + for partitions. + +## Staged Runtime Branch Logic + +Two branches can be valid in fresh K3s-style sandboxes, depending on evidence +from the target environment: + +- Host-managed driver plus host-managed toolkit/runtime can work when K3s + creates usable NVIDIA runtime handlers from host toolkit evidence. +- Host-managed driver plus GPU Operator-managed toolkit with CDI/NRI can also + work and avoids hand-editing K3s containerd paths. + +Do not freeze toolkit/runtime choice before K3s bootstrap. Decide after +`kubectl get runtimeclass`, runtime version, host toolkit evidence, and Helm +values are available. + +## Workload/Example Suite + +Declare the suite before claiming install success: + +- Core suite: `manifests/cuda-vectoradd.yaml`; pass requires pod completion and + logs containing `Test PASSED`. +- Optional notebook suite: docs `tf-notebook.yaml`; pass requires pod running, + service present, and token/log captured without exposing it broadly. +- Optional time-slicing suite: docs time-slicing config plus + `manifests/time-slicing-verification.yaml`; pass requires expected replicas + running and at least one VectorAdd success log. +- Optional MIG suite: only after approved MIG change; pass requires + `mig.config.state=success`, expected MIG resources, and matching workload + request success. + +If only the core suite runs, report "core-install proof", not full +documentation-suite install success. + +## Troubleshooting Reasoning + +Always debug the earliest failed layer: + +- No host GPU: host/provider problem, not GPU Operator. +- No NVIDIA labels/daemonsets desired zero: NFD/labels/taints. +- Driver pod failing: kernel, driver, image pull, `nouveau`, registry, or Xid. +- Runtime handler error: toolkit/CDI/NRI/runtime path, not CUDA app code. +- No `nvidia.com/gpu`: device plugin or GPU health. +- Workload fails after allocatable resources: workload manifest, CDI/runtime + injection, image, or GPU health. + +For field-shaped scenarios, use this order: + +1. Load `references/models.md` and identify the owner/layer. +2. Load `references/field-scenario-cards.md` and match the closest symptom. +3. Run only the read-only evidence card for that symptom. +4. Name selected branch, rejected broad branch, and stop condition. +5. Mutate only if the branch targets the proven layer and the user approved the + blast radius. + +Do not infer that GPU Operator is broken because a downstream pod is failing. +First prove whether the failure is a host driver issue, runtime/toolkit issue, +device-plugin/resource-advertisement issue, workload spec issue, or documented +unsupported environment. + +## Field Scenario Branch Examples + +| Evidence | Choose | Reject | +|---|---|---| +| `nvidia.com/gpu.deploy.device-plugin=false` plus plugin-validation logs waiting for GPU resources | validator/device-plugin-disabled diagnosis and escalation bundle | reinstalling GPU Operator | +| `no runtime for "nvidia" is configured` with no RuntimeClass | runtime/toolkit branch | CUDA app debugging | +| CDI unresolved device errors after upgrade | CDI/device-plugin/runtime evidence branch | generic image-pull or app issue | +| driver daemonset mount error for sysfs path | host/kernel compatibility branch | Helm retry loop | +| driver image pull error with unsupported OS/kernel/image tag | support-matrix/image availability branch | credential debugging first | +| upgrade labels say done but workload proof absent | maintenance continuity branch | declaring success from labels alone | + +## Maintenance Continuity + +Maintenance success requires more than "upgrade command returned 0": + +1. Capture pre-change values, CRDs, `ClusterPolicy`, node labels, operand + health, RuntimeClasses, and workload inventory. +2. Run the declared workload suite before change. +3. Choose chart upgrade vs driver upgrade branch; record different risks. +4. Apply approved change. +5. Monitor driver upgrade labels/events if driver changes. +6. Run the same workload suite after change. +7. Soak for the declared interval and confirm no new operand errors. + +Driver upgrade drain is a last resort. Try GPU pod deletion controls first; +enable full drain only with approval and disruption plan. diff --git a/gpu-operator-devops-agent/references/self-improvement.md b/gpu-operator-devops-agent/references/self-improvement.md new file mode 100644 index 000000000..989994d92 --- /dev/null +++ b/gpu-operator-devops-agent/references/self-improvement.md @@ -0,0 +1,89 @@ +# Self-Improvement Loop + +Use this reference whenever a run reveals something the package should remember +after the current session ends. + +## Durable Memory Files + +Write durable local notes under `runtime-memory/`: + +| File | Use | +|---|---| +| `insights.md` | Validated reusable lessons, model changes, branch refinements. | +| `anti-patterns.md` | Stale commands, unsafe advice, repeated failed reasoning shapes. | +| `discovered-resources.md` | Public docs, source files, issues, forum posts, examples, or blogs found during operation. | +| `operating-parameters.md` | Target-specific versions, runtime values, chart values, feature flags, and support caveats. | +| `upstream-issue-drafts.md` | Draft public GitHub issues or package/doc improvements. | + +Treat these files as local package memory. They may be committed to a review +branch when the user wants the package to carry the lesson forward, but they +are not automatically public reports. + +## When To Record + +Record a durable entry when: + +- a command succeeds only after adding context that was missing from the + package; +- public docs or examples are stale, incomplete, ambiguous, or environment + sensitive; +- an issue/forum/blog reveals a recurring field pattern; +- a live run changes the compositional, dynamic, or value model; +- a support-matrix boundary is newer-than-known-tested but live evidence exists; +- an agent nearly took an unsafe branch or repeated the same failed branch. + +## Entry Shape + +Each entry should include: + +- date; +- target product/version/environment; +- evidence source: public URL, source path, or live target evidence path; +- observation; +- package change needed; +- confidence level; +- whether an upstream issue should be drafted. + +## Upstream Issue Discipline + +Draft issues locally first. File an issue only after the user approves: + +- target repo or tracker; +- title and body; +- labels/milestone/owner if any; +- whether the evidence is safe to publish; +- credentials/account to use. + +For public GitHub issues, include only public docs/source evidence and redacted +live-environment evidence. Do not include NVIDIA-internal links, private +hostnames, credentials, customer data, or support-only details. + +## Suggested GitHub Issue Template + +```markdown +Title: + +## Summary + + +## Environment +- GPU Operator version: +- Kubernetes distribution/version: +- Runtime: +- OS/kernel: +- GPU model: + +## Evidence +- Public docs/source link: +- Live observed signal: +- Command/output excerpt with secrets redacted: + +## Expected Behavior + + +## Actual Behavior + + +## Suggested Improvement + +``` diff --git a/gpu-operator-devops-agent/references/source-map.md b/gpu-operator-devops-agent/references/source-map.md new file mode 100644 index 000000000..f233eb795 --- /dev/null +++ b/gpu-operator-devops-agent/references/source-map.md @@ -0,0 +1,92 @@ +# Source Map + +Reviewed 2026-07-02. Public URLs are listed for runtime traceability. The +runtime package depends on these public sources and live target-environment +evidence gathered during the user's approved run. + +## Product Docs + +- `getting-started.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html + - Supports prerequisites, Helm install, PSA namespace label, containerd + branch, CDI/NRI options, install verification, CUDA VectorAdd manifest, + Jupyter example, and optional workload pointers. +- `cdi.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html + - Supports CDI default, NRI behavior, K3s/k0s/RKE2 no-runtime-config branch, + RuntimeClass behavior, NRI runtime requirements, and `hostUsers:false` + known issue. +- `platform-support.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html + - Supports support-matrix review and NRI runtime range. +- `upgrade.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/upgrade.html + - Supports CRD manual/hook upgrade paths and + `--disable-openapi-validation`. +- `gpu-driver-upgrades.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html + - Supports driver upgrade controller, state labels, events, drain risk, and + recovery labels. +- `troubleshooting.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html + - Supports runtime handler, `nouveau`, Fabric Manager, Xid, DCGM, stuck + upgrades, and label/taint failure signatures. +- `gpu-operator-mig.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-mig.html + - Supports MIG strategy, config state labels, workload/reboot cautions. +- `gpu-sharing.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html + - Supports optional time-slicing validation branch. +- `release-notes.rst` + - Public: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html + - Supports version-specific component and known-issue checks before using + this package outside the reviewed target. + +## Product Source + +- GPU Operator tag `v26.3.3`, commit + `b0a49c0e7b2e061dcd83f2bb2fe4fe960c5d0338`. +- `deployments/gpu-operator/values.yaml` + - Public: https://github.com/NVIDIA/gpu-operator/blob/v26.3.3/deployments/gpu-operator/values.yaml + - Supports chart defaults: CDI, NRI, NFD, operator CRD hook, driver upgrade + policy, toolkit/device-plugin/DCGM/MIG settings. +- `deployments/gpu-operator/templates/validations.yaml` + - Public: https://github.com/NVIDIA/gpu-operator/blob/v26.3.3/deployments/gpu-operator/templates/validations.yaml + - Supports invalid CDI/NRI/toolkit combinations. +- `api/nvidia/v1/clusterpolicy_types.go` + - Public: https://github.com/NVIDIA/gpu-operator/blob/v26.3.3/api/nvidia/v1/clusterpolicy_types.go + - Supports CRD fields/status model. +- `hack/must-gather.sh` + - Public: https://github.com/NVIDIA/gpu-operator/blob/v26.3.3/hack/must-gather.sh + - Supports evidence bundle content. +- `validator/manifests/*.yaml` + - Public: https://github.com/NVIDIA/gpu-operator/tree/v26.3.3/validator/manifests + - Supports validator workload behavior. + +## Substrate Docs + +- K3s installation and kubeconfig behavior: + - Public: https://docs.k3s.io/quick-start + - Supports `/etc/rancher/k3s/k3s.yaml` command-context proof for fresh K3s + sandboxes. + +## Public Field Scenario Seeds + +- GPU Operator issue #2550: + - Public: https://github.com/NVIDIA/gpu-operator/issues/2550 + - Supports validator/device-plugin-disabled field scenario. +- GPU Operator issue #1876: + - Public: https://github.com/NVIDIA/gpu-operator/issues/1876 + - Supports CDI device injection / RuntimeClass upgrade field scenario. +- GPU Operator issue #2549: + - Public: https://github.com/NVIDIA/gpu-operator/issues/2549 + - Supports upgrade-state ambiguity field scenario. +- GPU Operator issue #2463: + - Public: https://github.com/NVIDIA/gpu-operator/issues/2463 + - Supports kernel hostPath / driver daemonset compatibility field scenario. +- NVIDIA Developer Forum runtime-handler thread: + - Public: https://forums.developer.nvidia.com/t/failed-to-create-pod-sandbox-rpc-error-code-unknown-desc-failed-to-get-sandbox-runtime-no-runtime-for-nvidia-is-configured/296409 + - Supports missing `nvidia` runtime handler field scenario. +- NVIDIA Developer Forum driver image pull thread: + - Public: https://forums.developer.nvidia.com/t/gpu-operator-helm-chat-deployment-issues/349743 + - Supports driver image pull / unsupported platform field scenario. diff --git a/gpu-operator-devops-agent/references/supported-scope.md b/gpu-operator-devops-agent/references/supported-scope.md new file mode 100644 index 000000000..e09a026cb --- /dev/null +++ b/gpu-operator-devops-agent/references/supported-scope.md @@ -0,0 +1,47 @@ +# Supported Scope + +Reviewed/generated on 2026-07-02 from public docs/source snapshots. + +## Primary Scope + +- Product: NVIDIA GPU Operator. +- Version focus: `v26.3.3`. +- Source focus: + - Public GPU Operator docs snapshot from NVIDIA docs and the public + `NVIDIA/cloud-native-docs` repository at commit + `77a2daaf234a2cebc8e178f24e18cc4a6150e8b1`. + - Public GPU Operator source repository, tag `v26.3.3` release commit + `b0a49c0e7b2e061dcd83f2bb2fe4fe960c5d0338`. +- Environment focus: test-owned single-node or small Kubernetes clusters, + especially Ubuntu 22.04/24.04, K3s/containerd, and A100 GPUs. +## Supported With Caution + +- Official platform-matrix Kubernetes/container runtime/OS combinations. +- K3s with CDI and NRI when containerd is in the documented NRI range: + `1.7.30`, `2.1.x`, or `2.2.x`. +- Containerd newer than the documented NRI range, such as `2.3.x`: not + declared known-tested by the docs snapshot. Treat as newer-untested; require + sandbox approval, context proof, NRI caveat, operand logs, allocatable GPU + proof, and workload manifest success before declaring the branch usable. +- Host-preinstalled drivers: supported only as integration. GPU Operator does + not own host driver lifecycle when `driver.enabled=false`. +- MIG on A100/H100-class GPUs with workload disruption approval and possible + reboot planning. + +## Explicit Non-Goals + +- Production execution without environment owner approval. +- OpenShift OLM, vGPU licensing, confidential containers, Kata, KubeVirt, + GPUDirect RDMA/GDS, GDRCopy, NVAIE, air-gapped/private registry flows, or + managed cloud cluster creation. +- Jetson/integrated-GPU platforms. +- Broad multi-node SLO/disruption engineering beyond the basic upgrade and + rollback controls documented here. + +## Verification Gap Statement + +This package has not itself been rerun end-to-end after authoring. It +incorporates prior live-test results and public field-scenario distillation, +but package readiness for a new environment requires a package-loaded run that +exercises clean bootstrap, install, workload validation, and at least one +field-shaped troubleshooting scenario. diff --git a/gpu-operator-devops-agent/runtime-memory/README.md b/gpu-operator-devops-agent/runtime-memory/README.md new file mode 100644 index 000000000..489237683 --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/README.md @@ -0,0 +1,18 @@ +# Runtime Memory + +This directory is local durable memory for GPU Operator DevOps agent runs. Use +it to preserve useful learning across agent sessions, compaction, and context +loss. + +Do not store secrets, tokens, private hostnames, customer data, internal-only +NVIDIA links, or unredacted logs here. Draft public issue content locally first; +file externally only after user approval. + +Recommended flow: + +1. During a run, append observations to the relevant file. +2. At the end of a run, promote repeated or validated observations into + package references/skills. +3. Leave tentative or environment-specific notes clearly scoped. +4. Draft upstream issues in `upstream-issue-drafts.md`; do not file them + automatically. diff --git a/gpu-operator-devops-agent/runtime-memory/anti-patterns.md b/gpu-operator-devops-agent/runtime-memory/anti-patterns.md new file mode 100644 index 000000000..46fcc5adb --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/anti-patterns.md @@ -0,0 +1,96 @@ +# Anti-Patterns + +Append stale commands, unsafe advice, and repeated failed reasoning shapes here. + +## Entry Template + +```markdown +### YYYY-MM-DD - + +- Context: +- Bad pattern: +- Why it is risky or stale: +- Safer replacement: +- Evidence: +- Package update needed: +``` + +## Seeded Anti-Patterns + +### 2026-07-02 - Treating Helm success as install success + +- Context: GPU Operator controller-style product installation. +- Bad pattern: declaring success immediately after `helm install` returns. +- Why it is risky or stale: operands, labels, device plugin resources, and + workload GPU access may still be unavailable. +- Safer replacement: follow the success ladder in + `references/models.md`, ending with workload manifest success for the + declared example suite. +- Evidence: public install verification docs and package model. +- Package update needed: keep this anti-pattern visible in install reviews. + +### 2026-07-02 - Using `kubectl run --limits` as validation + +- Context: workload proof after GPU Operator install. +- Bad pattern: relying on a brittle imperative `kubectl run --limits` command. +- Why it is risky or stale: CLI flags drift and are easy to misquote. +- Safer replacement: apply `manifests/cuda-vectoradd.yaml` and check logs for + `Test PASSED`. +- Evidence: public getting-started docs use a manifest-based VectorAdd example. +- Package update needed: keep validation manifest current with public docs. + +### 2026-07-02 - Treating `Ready` wait as the VectorAdd success gate + +- Context: Durable CUDA VectorAdd validation after GPU Operator install. +- Bad pattern: using `kubectl wait --for=condition=Ready pod/cuda-vectoradd` as the decisive pass/fail signal. +- Why it is risky or stale: the sample pod is short-lived and can reach `Succeeded` before it remains `Ready`; a `Ready` wait miss can be a false negative. +- Safer replacement: inspect pod phase and logs; pass requires phase `Succeeded` and log line `Test PASSED`. +- Evidence: a live Brev A100 GPU instance completed VectorAdd twice on 2026-07-02 with `Status: Succeeded`, exit code 0, and `Test PASSED`. +- Package update needed: install and operate skills updated to name phase/logs as the pass gate. + +### 2026-07-02 - Letting a status-print typo rerun a mutating remote command + +- Context: Running multi-step `brev exec --host` commands that create/delete Kubernetes test pods. +- Bad pattern: appending a brittle final JSONPath/status command under `set -e` after successful workload validation in the same remote command. +- Why it is risky or stale: a non-product typo can make the command exit nonzero after the workload succeeded; the Brev wrapper may reconnect/retry, repeating earlier delete/apply steps. +- Safer replacement: split mutating workload execution from nonessential status formatting, or make final status printers best-effort and syntactically simple. +- Evidence: on a live Brev GPU instance, a malformed JSONPath `{.status.phase}{\n}` failed after VectorAdd success and the command sequence was replayed once. +- Package update needed: Prefer short commands with separate evidence collection in future runbooks. + +### 2026-07-02 - Treating time-slicing availability as full CUDA proof + +- Context: Optional time-slicing example validation. +- Bad pattern: declaring success from deployment availability or allocatable + GPU count alone. +- Why it is risky or stale: availability proves scheduling capacity, but not + that every intended CUDA replica actually executed successfully. +- Safer replacement: wait for `Test PASSED` logs from each verification pod, or + explicitly mark the result as scheduling-only. +- Evidence: a prior live validation run reached rollout availability before all + replica logs were captured. +- Package update needed: operate skill now requires per-pod log proof. + +### 2026-07-02 - Incomplete optional-example cleanup + +- Context: Reverting time-slicing after an optional example test. +- Bad pattern: deleting the test namespace and ConfigMap without checking the + live `ClusterPolicy` field that references the config. +- Why it is risky or stale: the node can continue advertising time-sliced GPU + resources after the apparent cleanup. +- Safer replacement: verify namespace/config absence, live `ClusterPolicy`, + Helm values, node GPU labels/capacity, and a final VectorAdd smoke. +- Evidence: a prior live validation run caught live node labels still advertising + time-slicing after initial cleanup. +- Package update needed: bootstrap and operate guidance now include the cleanup + checklist. + +### 2026-07-02 - Assuming `kubectl run --limits` portability + +- Context: Quick GPU workload proof. +- Bad pattern: using imperative `kubectl run --limits` as the primary proof. +- Why it is risky or stale: client flag support varies and can fail before the + product behavior is tested. +- Safer replacement: use small manifests for GPU workload proofs. +- Evidence: a prior live validation run hit `kubectl run --limits` + incompatibility and recovered with a manifest. +- Package update needed: command catalog now names the manifest fallback. diff --git a/gpu-operator-devops-agent/runtime-memory/discovered-resources.md b/gpu-operator-devops-agent/runtime-memory/discovered-resources.md new file mode 100644 index 000000000..9f4e919a0 --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/discovered-resources.md @@ -0,0 +1,34 @@ +# Discovered Resources + +Append public resources discovered during operation. + +## Entry Template + +```markdown +### YYYY-MM-DD - + +- URL/path: +- Visibility: public | live-target-evidence | internal-non-exportable +- Product/version scope: +- Why it matters: +- Claims or scenarios it supports: +- Follow-up: +``` + +## Seed Resources + +- https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html +- https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html +- https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html +- https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html +- https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html +- https://github.com/NVIDIA/gpu-operator/tree/v26.3.3 + +### 2026-07-02 - Live Brev A100 v26.3.3 install proof + +- URL/path: live-target-evidence from a live A100 GPU instance run; the full transcript is not shipped with this package. +- Visibility: live-target-evidence. +- Product/version scope: NVIDIA GPU Operator v26.3.3 on Brev A100 with K3s v1.36.2+k3s1. +- Why it matters: Provides a concrete proof point for host-managed driver plus chart-managed toolkit/CDI/NRI on a newer K3s/containerd sandbox. +- Claims or scenarios it supports: Context proof, host GPU proof, Helm explicit values, `ClusterPolicy.ready`, allocatable `nvidia.com/gpu=1`, VectorAdd success, bad RuntimeClass first-layer diagnosis. +- Follow-up: Use as package-local evidence only; do not cite as public support guarantee. diff --git a/gpu-operator-devops-agent/runtime-memory/insights.md b/gpu-operator-devops-agent/runtime-memory/insights.md new file mode 100644 index 000000000..7adace57a --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/insights.md @@ -0,0 +1,44 @@ +# Insights + +Append validated reusable lessons here. + +## Entry Template + +```markdown +### YYYY-MM-DD - + +- Product/version: +- Environment: +- Evidence: +- Observation: +- Model/branch impact: +- Package update needed: +- Confidence: +``` + +### 2026-07-02 - K3s 1.36/containerd 2.3 sandbox proved host-driver plus CDI/NRI branch + +- Product/version: NVIDIA GPU Operator v26.3.3. +- Environment: a live Brev A100 single-node sandbox (disposable). +- Evidence: live target evidence from `kubectl get clusterpolicy`, node allocatable resources, and package VectorAdd manifest. +- Observation: On Ubuntu 22.04.5 with host driver 580.159.03, K3s v1.36.2+k3s1 and runtime `containerd://2.3.2-k3s2`, Helm values `driver.enabled=false` and `cdi.nriPluginEnabled=true` reconciled to `ClusterPolicy.ready`; device-plugin advertised one GPU; `manifests/cuda-vectoradd.yaml` completed with `Test PASSED`. +- Model/branch impact: The newer-than-known K3s/containerd CDI+NRI branch can succeed in a disposable Brev A100 sandbox, but should remain caveated until public support docs explicitly cover that runtime range. +- Package update needed: Keep the branch as sandbox-prove-before-production; record this as a live proof point, not a support guarantee. +- Confidence: High for this live sandbox; medium for generalization. + +### 2026-07-02 - Time-slicing and maintenance run proved safe optional-example path + +- Product/version: NVIDIA GPU Operator v26.3.3. +- Environment: multiple live Brev A100 single-node sandboxes (disposable). +- Evidence: live validation run: time-slicing enabled, workload proof passed, + same-version Helm reconciliation succeeded, bad RuntimeClass scenario was + diagnosed and cleaned up, and node allocatable GPU was restored to `1`. +- Observation: time-slicing is a useful second-stage discriminator because it + safely changes GPU advertisement without MIG reconfiguration. It also exposes + whether the agent understands ConfigMap, live ClusterPolicy, Helm-values, and + workload-proof boundaries. +- Model/branch impact: optional examples should be part of package evaluation, + but cleanup must verify both live object state and Helm state. +- Package update needed: keep cleanup verification and per-replica log proof in + operate/maintenance guidance. +- Confidence: High for Brev A100 single-node sandbox. diff --git a/gpu-operator-devops-agent/runtime-memory/operating-parameters.md b/gpu-operator-devops-agent/runtime-memory/operating-parameters.md new file mode 100644 index 000000000..56abba681 --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/operating-parameters.md @@ -0,0 +1,60 @@ +# Operating Parameters + +Record target-specific values and support caveats here. + +## Entry Template + +```markdown +### YYYY-MM-DD - + +- Sandbox/cluster: +- GPU Operator version: +- Helm chart version: +- Kubernetes distribution/version: +- Container runtime/version: +- OS/kernel: +- GPU model: +- Driver version: +- Install branch: +- Enabled features: +- Known-tested/support caveats: +- Workload/example suite: +- Cleanup status: +``` + +### 2026-07-02 - Live A100 GPU instance + +- Sandbox/cluster: a live A100 GPU instance; single-node K3s cluster context `default`. +- GPU Operator version: v26.3.3. +- Helm chart version: `gpu-operator-v26.3.3`. +- Kubernetes distribution/version: K3s v1.36.2+k3s1. +- Container runtime/version: K3s node runtime `containerd://2.3.2-k3s2`; host `/usr/bin/containerd` reported containerd.io v2.2.5 before K3s. +- OS/kernel: Ubuntu 22.04.5 LTS; kernel `6.8.0-1060-gcp`. +- GPU model: NVIDIA A100-SXM4-80GB, MIG disabled. +- Driver version: 580.159.03, host-managed. +- Install branch: `driver.enabled=false`; chart-managed NFD/toolkit/device-plugin; namespace `gpu-operator` labeled `pod-security.kubernetes.io/enforce=privileged`. +- Enabled features: `cdi.enabled=true`, `cdi.nriPluginEnabled=true`, `toolkit.enabled=true`, `nfd.enabled=true`. +- Known-tested/support caveats: K3s v1.36/containerd 2.3 is newer than the package's known-tested NRI range; live sandbox validation passed, but production use still needs support-matrix confirmation. +- Workload/example suite: Core install proof only: package `manifests/cuda-vectoradd.yaml` completed with `Test PASSED`; bad RuntimeClass troubleshooting scenario produced expected kubelet runtime-handler failure and was cleaned up. +- Cleanup status: Validation pods and bad RuntimeClass scenario deleted; GPU Operator and K3s intentionally left running during the instance-local test; the instance was stopped after all testing completed. + +### 2026-07-02 - Live A100 GPU follow-up multi-instance run + +- Sandbox/cluster: multiple live A100 GPU instances; + single-node K3s clusters. +- GPU Operator version: v26.3.3. +- Helm chart version: `gpu-operator-v26.3.3`. +- Kubernetes distribution/version: K3s v1.36.2+k3s1. +- Container runtime/version: `containerd://2.3.2-k3s2`. +- OS/kernel: Ubuntu 22.04.5 LTS; kernel `6.8.0-1060-gcp`. +- GPU model: NVIDIA A100-SXM4-80GB, MIG disabled. +- Driver version: 580.159.03, host-managed. +- Install branch: `driver.enabled=false`; toolkit/CDI/NRI enabled. +- Enabled features: temporary time-slicing, then restored to baseline. +- Known-tested/support caveats: this is sandbox evidence for the newer K3s and + containerd combination, not a support-matrix guarantee. +- Workload/example suite: time-slicing workload proof, same-version Helm + reconciliation with pre/post VectorAdd, bad RuntimeClass scenario, final + cleanup proof. +- Cleanup status: all test resources removed, node allocatable GPU restored to + `1`, and all three instances were stopped. diff --git a/gpu-operator-devops-agent/runtime-memory/upstream-issue-drafts.md b/gpu-operator-devops-agent/runtime-memory/upstream-issue-drafts.md new file mode 100644 index 000000000..21e573239 --- /dev/null +++ b/gpu-operator-devops-agent/runtime-memory/upstream-issue-drafts.md @@ -0,0 +1,31 @@ +# Upstream Issue Drafts + +Draft docs/product/package improvement issues here. Do not file externally +without user approval. + +## Draft Template + +```markdown +### YYYY-MM-DD - + +- Target repo/tracker: +- Proposed labels: +- Safe to file publicly: yes/no/unknown +- User approval: pending/approved/filed + +#### Title + + +#### Body +## Summary + +## Environment + +## Evidence + +## Expected Behavior + +## Actual Behavior + +## Suggested Improvement +``` diff --git a/gpu-operator-devops-agent/scripts/README.md b/gpu-operator-devops-agent/scripts/README.md new file mode 100644 index 000000000..e3f63b9e9 --- /dev/null +++ b/gpu-operator-devops-agent/scripts/README.md @@ -0,0 +1,12 @@ +# GPU Operator Script Index + +Generated from a curated source manifest. Scripts are +templates for fragile or repeated operations. Read each script before running +it and set the documented environment variables for the target environment. + +| path | status | risk | description | +| --- | --- | --- | --- | +| scripts/inventory-gpu-operator-environment.sh | validated-fixture | read-only | Collect read-only GPU Operator deployment inventory before diagnosis. | +| scripts/install-gpu-operator-k3s-host-driver.sh | validated-fixture | high-mutation | Install GPU Operator on K3s with a host-managed driver and explicit K3s toolkit env. | +| scripts/validate-cuda-vectoradd.sh | validated-live | medium-mutation | Run and clean up the CUDA VectorAdd proof workload. | +| scripts/apply-time-slicing.sh | validated-fixture | high-mutation | Apply, verify, and revert a temporary time-slicing example. | diff --git a/gpu-operator-devops-agent/scripts/apply-time-slicing.sh b/gpu-operator-devops-agent/scripts/apply-time-slicing.sh new file mode 100755 index 000000000..125c44d34 --- /dev/null +++ b/gpu-operator-devops-agent/scripts/apply-time-slicing.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Scaffold for a time-slicing example: safety-gate + namespace setup/teardown. +# Status: validated-fixture +# Risk: high-mutation +# +# Use when: +# - Single test-owned node has no non-test GPU workloads and MIG is disabled. +# +# Do not use when: +# - Production/shared node or active workload inventory is present. +# +# Expected success signals: +# - No active GPU workloads are present before mutation. +# - Temporary namespace is removed during cleanup. +# +# Cleanup verification: +# kubectl get namespace "${NAMESPACE}" 2>&1 | grep -E 'NotFound|not found' +# +# Stop/escalate if: +# - Active GPU workloads are present. +# - Cleanup cannot restore baseline. + +set -euo pipefail +KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" +NAMESPACE="${NAMESPACE:-gpu-operator-timeslice}" +OPERATOR_NAMESPACE="${OPERATOR_NAMESPACE:-gpu-operator}" +CONFIGMAP="${CONFIGMAP:-time-slicing-config-all}" +REPLICAS="${REPLICAS:-2}" +export KUBECONFIG +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +kubectl get pods -A -o json | jq -e '.items[] | select((.spec.containers // []) | any((.resources.limits // {})["nvidia.com/gpu"] != null))' >/dev/null && { echo "GPU workloads already exist; stop before time-slicing mutation" >&2; exit 2; } || true +kubectl delete ns "${NAMESPACE}" --ignore-not-found=true +kubectl delete configmap -n "${OPERATOR_NAMESPACE}" "${CONFIGMAP}" --ignore-not-found=true +kubectl create namespace "${NAMESPACE}" +echo "Use the full package command card for ConfigMap and deployment manifests when running this script candidate." +kubectl delete namespace "${NAMESPACE}" +kubectl wait --for=delete "ns/${NAMESPACE}" --timeout=180s || true diff --git a/gpu-operator-devops-agent/scripts/install-gpu-operator-k3s-host-driver.sh b/gpu-operator-devops-agent/scripts/install-gpu-operator-k3s-host-driver.sh new file mode 100755 index 000000000..63334e945 --- /dev/null +++ b/gpu-operator-devops-agent/scripts/install-gpu-operator-k3s-host-driver.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Install GPU Operator on K3s with a host-managed driver and explicit K3s toolkit env. +# Status: validated-fixture +# Risk: high-mutation +# +# Use when: +# - K3s/containerd 2.3.x or newer-than-documented NRI range is present. +# - Host nvidia-smi works and driver lifecycle is platform-owned. +# - K3s config and socket paths exist. +# +# Do not use when: +# - The target is production/shared without approval. +# - Host GPU or K3s runtime paths are not proven. +# +# Expected success signals: +# - Helm release is deployed. +# - ClusterPolicy is ready. +# - Node advertises nvidia.com/gpu. +# +# Cleanup verification: +# helm list -n "${NAMESPACE}" +# kubectl get clusterpolicy +# kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +# +# Stop/escalate if: +# - nvidia-smi fails. +# - K3s config/socket paths are absent. +# - The same runtime/toolkit failure repeats twice. + +set -euo pipefail +GPU_OPERATOR_VERSION="${GPU_OPERATOR_VERSION:-v26.3.3}" +KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" +NAMESPACE="${NAMESPACE:-gpu-operator}" +RELEASE="${RELEASE:-gpu-operator}" +export KUBECONFIG +hostname +nvidia-smi +kubectl get nodes -o wide +test -f /var/lib/rancher/k3s/agent/etc/containerd/config.toml +test -S /run/k3s/containerd/containerd.sock +helm repo add nvidia https://helm.ngc.nvidia.com/nvidia >/dev/null 2>&1 || true +helm repo update nvidia +kubectl create ns "${NAMESPACE}" --dry-run=client -o yaml | kubectl apply -f - +kubectl label --overwrite ns "${NAMESPACE}" pod-security.kubernetes.io/enforce=privileged +helm upgrade --install "${RELEASE}" nvidia/gpu-operator -n "${NAMESPACE}" --version "${GPU_OPERATOR_VERSION}" --wait --timeout 15m --set driver.enabled=false --set cdi.nriPluginEnabled=false --set toolkit.enabled=true --set toolkit.env[0].name=CONTAINERD_CONFIG --set toolkit.env[0].value=/var/lib/rancher/k3s/agent/etc/containerd/config.toml --set toolkit.env[1].name=CONTAINERD_SOCKET --set toolkit.env[1].value=/run/k3s/containerd/containerd.sock --set toolkit.env[2].name=RUNTIME_CONFIG_SOURCE --set-string toolkit.env[2].value=file=/var/lib/rancher/k3s/agent/etc/containerd/config.toml +kubectl get clusterpolicy +kubectl get pods -n "${NAMESPACE}" -o wide +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' diff --git a/gpu-operator-devops-agent/scripts/inventory-gpu-operator-environment.sh b/gpu-operator-devops-agent/scripts/inventory-gpu-operator-environment.sh new file mode 100755 index 000000000..483c979c7 --- /dev/null +++ b/gpu-operator-devops-agent/scripts/inventory-gpu-operator-environment.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# Collect read-only GPU Operator deployment inventory before diagnosis. +# Status: validated-fixture +# Risk: read-only +# +# Use when: +# - Before install, maintenance, troubleshooting, or branch selection. +# +# Do not use when: +# - The shell is not the assigned target environment. +# +# Expected success signals: +# - An inventory directory is written. +# - Host, Kubernetes, runtime, GPU, and GPU Operator evidence files are present when available. +# +# Cleanup verification: +# test -d "${INVENTORY_DIR}" +# +# Stop/escalate if: +# - The shell or kube context is not the assigned target. +# - Required read access is missing. + +set -euo pipefail +KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" +INVENTORY_DIR="${INVENTORY_DIR:-./gpu-operator-inventory-$(date +%Y%m%d-%H%M%S)}" +mkdir -p "${INVENTORY_DIR}" +run() { + local name="$1" + shift + { echo "$ $*"; "$@"; } >"${INVENTORY_DIR}/${name}.txt" 2>&1 || true +} +run shell-context hostname +run id id +run pwd pwd +run os-release cat /etc/os-release +run kernel uname -a +run nvidia-smi nvidia-smi +run nvidia-smi-query nvidia-smi --query-gpu=name,driver_version,memory.total,mig.mode.current --format=csv,noheader +run lspci-nvidia bash -lc "lspci | grep -i nvidia" +run lsmod-nvidia bash -lc "lsmod | grep -E 'nouveau|nvidia'" +run dmesg-nvidia bash -lc "dmesg | grep -Ei 'NVRM|Xid|nouveau|nvidia' | tail -120" +run containerd-version bash -lc "containerd --version || true" +run crio-version bash -lc "crio --version || true" +run k3s-runtime-paths bash -lc "test -f /var/lib/rancher/k3s/agent/etc/containerd/config.toml && echo k3s-config-present || true; test -S /run/k3s/containerd/containerd.sock && echo k3s-socket-present || true; grep -i nvidia /var/lib/rancher/k3s/agent/etc/containerd/config.toml || true" +if test -f "${KUBECONFIG}"; then + export KUBECONFIG + run kube-context kubectl config current-context + run nodes-wide kubectl get nodes -o wide + run node-runtime kubectl get nodes -o json + run node-gpu-labels kubectl get nodes -L feature.node.kubernetes.io/pci-10de.present,nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu-driver-upgrade-state,nvidia.com/mig.config,nvidia.com/mig.config.state + run runtimeclasses kubectl get runtimeclass -o yaml + run namespaces kubectl get ns --show-labels + run gpu-operator-pods kubectl get pods -n gpu-operator -o wide + run gpu-operator-daemonsets kubectl get ds -n gpu-operator -o wide + run clusterpolicy kubectl get clusterpolicy -o yaml + run helm-list helm list -A + run helm-values bash -lc "helm get values gpu-operator -n gpu-operator -o yaml || true" + run events bash -lc "kubectl get events -A --sort-by='.lastTimestamp' | tail -200" + run gpu-workloads bash -lc "kubectl get pods -A -o json | jq -r '.items[] | select((.spec.containers // []) | any((.resources.limits // {})[\"nvidia.com/gpu\"] != null)) | [.metadata.namespace, .metadata.name, .status.phase] | @tsv'" +else + echo "KUBECONFIG not found: ${KUBECONFIG}" >"${INVENTORY_DIR}/kubeconfig-missing.txt" +fi +echo "Inventory written to ${INVENTORY_DIR}" diff --git a/gpu-operator-devops-agent/scripts/validate-cuda-vectoradd.sh b/gpu-operator-devops-agent/scripts/validate-cuda-vectoradd.sh new file mode 100755 index 000000000..764f6ee0d --- /dev/null +++ b/gpu-operator-devops-agent/scripts/validate-cuda-vectoradd.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Run and clean up the CUDA VectorAdd proof workload. +# Status: validated-live +# Risk: medium-mutation +# +# Use when: +# - GPU Operator is installed and node advertises allocatable nvidia.com/gpu. +# +# Do not use when: +# - Device-plugin logs or dmesg indicate unhealthy GPU/Xid. +# +# Expected success signals: +# - Pod phase reaches Succeeded. +# - Logs contain Test PASSED. +# - Pod is removed after cleanup. +# +# Cleanup verification: +# kubectl get pod cuda-vectoradd 2>&1 | grep -E 'NotFound|not found' +# +# Stop/escalate if: +# - VectorAdd fails after upstream health is ready. +# - GPU is marked unhealthy or Xid appears. + +set -euo pipefail +KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" +PACKAGE_ROOT="${PACKAGE_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +MANIFEST="${MANIFEST:-${PACKAGE_ROOT}/manifests/cuda-vectoradd.yaml}" +export KUBECONFIG +kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.'nvidia\.com/gpu' +kubectl delete pod cuda-vectoradd --ignore-not-found +kubectl apply -f "${MANIFEST}" +kubectl wait pod/cuda-vectoradd --for=jsonpath='{.status.phase}'=Succeeded --timeout=180s +kubectl logs cuda-vectoradd | grep 'Test PASSED' +kubectl delete pod cuda-vectoradd --ignore-not-found +kubectl get pod cuda-vectoradd 2>&1 | grep -E 'NotFound|not found' diff --git a/gpu-operator-devops-agent/skills/gpu-operator-evidence-bundle/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-evidence-bundle/SKILL.md new file mode 100644 index 000000000..6948aa627 --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-evidence-bundle/SKILL.md @@ -0,0 +1,93 @@ +--- +name: gpu-operator-evidence-bundle +description: Collect a safe GPU Operator evidence bundle for troubleshooting, escalation, or test reporting. +tags: [gpu-operator, evidence, must-gather, support] +--- + +# GPU Operator Evidence Bundle + +Use when troubleshooting is unclear, escalation is likely, or a test report +needs durable evidence. + +## Preconditions + +- Approval to collect logs/manifests. +- Sensitive data handling plan for node names, workload names, image pull + secrets, notebook tokens, and host logs. +- `kubectl` context proven. + +## Quick Bundle + +```bash +ARTIFACT_DIR="${ARTIFACT_DIR:-/tmp/nvidia-gpu-operator_$(date +%Y%m%d_%H%M)}" +mkdir -p "$ARTIFACT_DIR" +hostname > "$ARTIFACT_DIR/host.name" +echo "KUBECONFIG=${KUBECONFIG:-unset}" > "$ARTIFACT_DIR/kubeconfig.env" +kubectl config current-context > "$ARTIFACT_DIR/kube.context" +kubectl get nodes -o wide > "$ARTIFACT_DIR/nodes.status" +kubectl get nodes --show-labels > "$ARTIFACT_DIR/nodes.labels" +kubectl describe nodes -l nvidia.com/gpu.present=true > "$ARTIFACT_DIR/gpu_nodes.describe" || true +kubectl get runtimeclass > "$ARTIFACT_DIR/runtimeclasses.txt" || true +kubectl get pods -n gpu-operator -o wide > "$ARTIFACT_DIR/gpu_operator_pods.status" +kubectl get pods -n gpu-operator -o yaml > "$ARTIFACT_DIR/gpu_operator_pods.yaml" +kubectl get ds -n gpu-operator -o wide > "$ARTIFACT_DIR/gpu_operator_daemonsets.status" +kubectl get clusterpolicy -o yaml > "$ARTIFACT_DIR/clusterpolicy.yaml" +kubectl get nvidiadrivers.nvidia.com -A -o yaml > "$ARTIFACT_DIR/nvidiadrivers.yaml" || true +kubectl get events -A --sort-by='.lastTimestamp' > "$ARTIFACT_DIR/events.all" +helm list -n gpu-operator > "$ARTIFACT_DIR/helm.list" || true +helm get values gpu-operator -n gpu-operator -o yaml > "$ARTIFACT_DIR/helm.values.yaml" || true +``` + +Sandbox metadata when available: + +```bash +curl -fsS -H "Metadata-Flavor: Google" \ + http://metadata.google.internal/computeMetadata/v1/instance/machine-type \ + > "$ARTIFACT_DIR/cloud.machine_type" 2>/dev/null || true +curl -fsS -H "Metadata-Flavor: Google" \ + http://metadata.google.internal/computeMetadata/v1/instance/zone \ + > "$ARTIFACT_DIR/cloud.zone" 2>/dev/null || true +curl -fsS -H "Metadata-Flavor: Google" \ + http://metadata.google.internal/computeMetadata/v1/instance/id \ + > "$ARTIFACT_DIR/cloud.instance_id" 2>/dev/null || true +``` + +Pod logs: + +```bash +for pod in $(kubectl get pods -n gpu-operator -o name); do + name="${pod#pod/}" + kubectl describe "$pod" -n gpu-operator > "$ARTIFACT_DIR/pod_${name}.describe" || true + kubectl logs "$pod" -n gpu-operator --all-containers --prefix --timestamps > "$ARTIFACT_DIR/pod_${name}.log" || true + kubectl logs "$pod" -n gpu-operator --all-containers --prefix --timestamps --previous > "$ARTIFACT_DIR/pod_${name}.previous.log" || true +done +``` + +## Upstream Must-Gather + +Prefer the target tag: + +```bash +curl -o must-gather.sh -L https://raw.githubusercontent.com/NVIDIA/gpu-operator/v26.3.3/hack/must-gather.sh +chmod +x must-gather.sh +ARTIFACT_DIR="$ARTIFACT_DIR" ./must-gather.sh +``` + +Review before sharing; `nvidia-bug-report.sh` output can include detailed host +state. + +## Escalation Summary + +```text +Product/chart version: +Kubernetes/runtime/OS/kernel: +GPU model and driver: +Command context proof: +Install values: +First failing layer: +Primary symptom: +Commands run: +Risky action proposed: +Artifacts: +Known unknowns: +``` diff --git a/gpu-operator-devops-agent/skills/gpu-operator-improve/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-improve/SKILL.md new file mode 100644 index 000000000..2d23cc299 --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-improve/SKILL.md @@ -0,0 +1,70 @@ +--- +name: gpu-operator-improve +description: Capture durable GPU Operator DevOps package lessons, anti-patterns, discovered resources, operating parameters, and upstream issue drafts. +tags: [gpu-operator, improvement, memory, issues] +--- + +# GPU Operator Improve + +Use this skill at the end of every live run, failed attempt, troubleshooting +scenario, or source-discovery pass. + +## Inputs + +- Run transcript, evidence bundle, or scenario result. +- Target environment metadata. +- Any newly discovered public docs/source/issues/forums/blogs. +- Any observed stale command, missing branch, unsafe advice, or docs/product + improvement candidate. + +## Workflow + +1. Read `references/self-improvement.md`. +2. Classify each lesson: + - insight; + - anti-pattern; + - discovered resource; + - operating parameter; + - upstream issue draft; + - package source/reference update. +3. Append concise entries to the matching file in `runtime-memory/`. +4. If the lesson is validated and reusable, patch the package reference or + skill that future agents will load. +5. If the lesson belongs upstream, draft an issue in + `runtime-memory/upstream-issue-drafts.md`. +6. Ask for user approval before filing any external issue. + +## GitHub Issue Filing Boundary + +Creating or updating a GitHub issue is a mutating external action. Do it only +after the user approves: + +- repository; +- title/body; +- labels/milestone if any; +- account/credentials; +- whether the included evidence is safe to publish. + +Default target for public GPU Operator product/docs issues: + +```text +https://github.com/NVIDIA/gpu-operator/issues +``` + +If the issue concerns this package rather than the product/docs, +draft it locally and ask where the package issue should live. + +## Redaction Rules + +Never write secrets, tokens, private hostnames, unredacted customer data, +internal-only NVIDIA links, or private support artifacts into public issue +drafts. Keep internal or private context in a non-exportable authoring note +outside the public package. + +## Completion Criteria + +- Relevant `runtime-memory/` files updated. +- Package refs/skills patched when the lesson is validated and reusable. +- Upstream issue draft written when appropriate. +- External issue filing either approved and completed, or left as an explicit + pending draft. diff --git a/gpu-operator-devops-agent/skills/gpu-operator-install/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-install/SKILL.md new file mode 100644 index 000000000..14a5abe27 --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-install/SKILL.md @@ -0,0 +1,127 @@ +--- +name: gpu-operator-install +description: Install NVIDIA GPU Operator safely in a proven sandbox or cluster, including clean K3s bootstrap, staged branch decisions, and durable workload validation. +tags: [gpu-operator, install, helm, k3s, cdi, nri] +--- + +# GPU Operator Install + +Use this skill when planning or running a GPU Operator install. + +## Preconditions + +- Approved test-owned sandbox or explicit approval for target cluster mutation. +- `kubectl`, `helm`, and `jq`. +- Deployment inventory collected with `references/environment-inventory.md`. +- Command context proven with `references/bootstrap.md`. +- Target version selected. Default: `v26.3.3`. +- Driver ownership and runtime/toolkit branch not finalized until after + Kubernetes context and runtime evidence are collected. + +Before choosing Helm values, read `references/branch-matrix.md` and cite the +row that selected the branch. + +## Context Proof + +```bash +hostname +echo "KUBECONFIG=${KUBECONFIG:-unset}" +kubectl config current-context +kubectl get nodes -o wide +kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, os: .status.nodeInfo.osImage, kernel: .status.nodeInfo.kernelVersion, runtime: .status.nodeInfo.containerRuntimeVersion}' +helm version +``` + +For fresh K3s: + +```bash +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +kubectl get nodes -o wide +``` + +## Preflight Evidence And Inventory + +```bash +scripts/inventory-gpu-operator-environment.sh +``` + +Expected signals: intended sandbox, Ready node, GPU visible, no unexpected +`nouveau` conflict, runtime version known, and NFD ownership known. + +## Branch Logic + +| Branch | Choose when | Helm posture | Stop if | +|---|---|---|---| +| Operator-managed driver | no trusted host driver or disposable sandbox wants one owner | default `driver.enabled=true` | host kernel/OS unsupported, no registry access, `nouveau` conflict | +| Host-managed driver | host `nvidia-smi` works and platform owns driver lifecycle | `--set driver.enabled=false` | `nvidia-smi` fails | +| Chart NFD | no owned feature labels exist | default `nfd.enabled=true` | existing NFD owner requires integration | +| Existing NFD | owned NFD labels already present | `--set nfd.enabled=false` | labels stale/incomplete | +| CDI+NRI | K3s-like runtime and containerd in documented NRI range | `--set cdi.nriPluginEnabled=true` | CDI/toolkit disabled | +| Explicit K3s toolkit env | K3s/containerd `2.3.x` or newer-than-documented NRI range and K3s paths exist | `--set cdi.nriPluginEnabled=false` plus K3s `toolkit.env` paths | config/socket path unknown | +| Newer-than-known NRI experiment | user explicitly wants sandbox validation of newer-than-documented NRI | `--set cdi.nriPluginEnabled=true` plus caveat and full validation | production or no approval for uncertainty | +| Host-managed toolkit | K3s already has working NVIDIA handlers from host toolkit | `--set toolkit.enabled=false` | handler evidence absent | + +## Recommended K3s Sandbox Install + +Create/label namespace: + +```bash +kubectl create ns gpu-operator --dry-run=client -o yaml | kubectl apply -f - +kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged +``` + +For host-preinstalled driver plus GPU Operator-managed toolkit on +K3s/containerd `2.3.x`, use the explicit K3s toolkit-env script: + +```bash +scripts/install-gpu-operator-k3s-host-driver.sh +``` + +Only use CDI/NRI on newer-than-documented containerd when the user explicitly +approved a sandbox experiment. Record the support-matrix caveat and validate +with the full workload proof. + +For operator-managed driver, use the same branch matrix first and do not use +the K3s host-driver script. + +```bash +helm install gpu-operator nvidia/gpu-operator \ + -n gpu-operator \ + --version v26.3.3 \ + --set cdi.nriPluginEnabled=true +``` + +If using host-managed toolkit/runtime too, add `--set toolkit.enabled=false` +only after runtime handler evidence proves it. + +## Verification + +```bash +kubectl get pods -n gpu-operator -o wide +kubectl get ds -n gpu-operator +kubectl get clusterpolicy +kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu.deploy.driver,nvidia.com/mig.capable +kubectl get nodes -o json | jq '.items[] | select(.status.allocatable["nvidia.com/gpu"] != null) | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}' +``` + +Then run the durable core workload: + +```bash +scripts/validate-cuda-vectoradd.sh +``` + +Pass signal: pod phase is `Succeeded` and logs include `Test PASSED`. This +short-lived validation pod may complete before it ever stays `Ready`, so use +phase and logs rather than a `Ready` condition as the success gate. If only this +workload runs, report a core-install proof. + +## Recovery + +- Helm fails before pods: check kubeconfig, namespace PSA, chart values. +- Pods stuck in init: inspect driver and toolkit logs before retry. +- Runtime handler error: check CDI/NRI branch and runtime evidence. +- No allocatable GPU: inspect device-plugin logs and Xid/dmesg. +- Two same-shape failures: stop and collect escalation bundle. + +Sources: `references/bootstrap.md`, `references/operational-reasoning.md`, +`references/source-map.md`. diff --git a/gpu-operator-devops-agent/skills/gpu-operator-maintain/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-maintain/SKILL.md new file mode 100644 index 000000000..80e772e13 --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-maintain/SKILL.md @@ -0,0 +1,125 @@ +--- +name: gpu-operator-maintain +description: Plan GPU Operator upgrades, rollback, driver changes, MIG changes, and maintenance continuity checks. +tags: [gpu-operator, maintain, upgrade, rollback, driver, mig] +--- + +# GPU Operator Maintain + +Use for chart upgrades, rollback, driver changes, MIG changes, and config +maintenance. + +## Preconditions + +- Explicit approval for mutation. +- Maintenance window for driver, MIG, drain, reboot, or runtime changes. +- `kubectl`/`helm` context proven. +- Workload owner notified when GPU workloads may be interrupted. +- Declared workload suite selected before change. + +## Baseline And Continuity Capture + +```bash +helm list -n gpu-operator +helm get values gpu-operator -n gpu-operator -o yaml > gpu-operator-values-before.yaml +helm get manifest gpu-operator -n gpu-operator > gpu-operator-manifest-before.yaml +kubectl get clusterpolicy -o yaml > clusterpolicy-before.yaml +kubectl get crd | grep -E 'nvidia.com|nodefeaturerules' > crds-before.txt +kubectl get nvidiadrivers.nvidia.com -A -o yaml > nvidiadrivers-before.yaml || true +kubectl get nodes -L nvidia.com/gpu-driver-upgrade-state,nvidia.com/mig.config,nvidia.com/mig.config.state +kubectl get pods -A -o wide > pods-before.txt +kubectl get runtimeclass > runtimeclasses-before.txt || true +``` + +Run the declared workload suite before changing state: + +```bash +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl logs pod/cuda-vectoradd +kubectl delete -f manifests/cuda-vectoradd.yaml --ignore-not-found +``` + +## Chart Upgrade + +Use the CRD hook path for v24.9.0+ when image pulls and hook jobs are expected +to work: + +```bash +export RELEASE_TAG=v26.3.3 +helm repo update nvidia +helm show values nvidia/gpu-operator --version "$RELEASE_TAG" > "values-$RELEASE_TAG.yaml" +# Merge approved existing values into values-$RELEASE_TAG.yaml. +helm upgrade gpu-operator nvidia/gpu-operator \ + -n gpu-operator \ + --disable-openapi-validation \ + -f "values-$RELEASE_TAG.yaml" \ + --version "$RELEASE_TAG" +``` + +Use the manual CRD path only when hook behavior is unsuitable: + +```bash +kubectl apply -f "https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_clusterpolicies.yaml" +kubectl apply -f "https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/crds/nvidia.com_nvidiadrivers.yaml" +kubectl apply -f "https://raw.githubusercontent.com/NVIDIA/gpu-operator/refs/tags/$RELEASE_TAG/deployments/gpu-operator/charts/node-feature-discovery/crds/nfd-api-crds.yaml" +helm upgrade gpu-operator nvidia/gpu-operator -n gpu-operator -f "values-$RELEASE_TAG.yaml" --version "$RELEASE_TAG" +``` + +## Driver Upgrade + +Only for operator-managed drivers. If `driver.enabled=false`, use the +host/platform driver runbook instead. + +```bash +kubectl patch clusterpolicies.nvidia.com/cluster-policy \ + --type='json' \ + -p='[{"op":"replace","path":"/spec/driver/version","value":""}]' +kubectl get node -l nvidia.com/gpu.present \ + -ojsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}{end}' +kubectl get events -A --sort-by='.lastTimestamp' | grep GPUDriverUpgrade +kubectl logs -n gpu-operator deployment/gpu-operator | grep controllers.Upgrade +``` + +Do not enable `driver.upgradePolicy.drain.enable=true` until GPU pod deletion +controls are insufficient and workload owners approve broader eviction. + +## Post-Change Continuity + +```bash +kubectl get pods -n gpu-operator -o wide +kubectl get ds -n gpu-operator +kubectl get clusterpolicy +kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable_gpu: .status.allocatable["nvidia.com/gpu"]}' +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl logs pod/cuda-vectoradd +kubectl delete -f manifests/cuda-vectoradd.yaml --ignore-not-found +``` + +Pass requires the same suite to pass after change and no new persistent operand +errors during the declared soak interval. + +## Rollback + +```bash +helm history gpu-operator -n gpu-operator +helm rollback gpu-operator -n gpu-operator +``` + +For operator-managed driver rollback, patch the previous approved driver +version and monitor the upgrade controller. Do not roll back host drivers +through GPU Operator. + +## MIG Change + +MIG changes are disruptive. Stop until workload owners approve stopping or +migrating active GPU workloads. + +```bash +kubectl patch clusterpolicies.nvidia.com/cluster-policy \ + --type='json' \ + -p='[{"op":"replace","path":"/spec/mig/strategy","value":"single"}]' +kubectl label node nvidia.com/mig.config=all-1g.10gb --overwrite +kubectl get node -L nvidia.com/mig.config,nvidia.com/mig.config.state +``` + +Escalate if reboot/drain is needed and not approved. diff --git a/gpu-operator-devops-agent/skills/gpu-operator-operate/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-operate/SKILL.md new file mode 100644 index 000000000..aaeaaa2fe --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-operate/SKILL.md @@ -0,0 +1,97 @@ +--- +name: gpu-operator-operate +description: Run GPU Operator day-2 health checks, workload validation, drift detection, and optional example-suite checks. +tags: [gpu-operator, operate, health, validation, dcgm] +--- + +# GPU Operator Operate + +Use for health checks, post-install validation, resource inventory, and drift +detection. + +## Preconditions + +- `kubectl` context proven against the intended cluster. +- Read-only access for health checks. +- Approval to create/delete disposable validation workloads. + +## Health Ladder + +1. Context and nodes proven. +2. NVIDIA PCI/GPU labels present or NFD branch understood. +3. GPU Operator controller and operands healthy. +4. Driver is healthy or host-managed driver accepted. +5. Toolkit/CDI/NRI state matches install branch. +6. Device plugin advertises GPUs. +7. `ClusterPolicy` ready. +8. Durable workload manifest passes. +9. Optional examples selected for the environment pass. + +## Commands + +```bash +kubectl config current-context +kubectl get nodes -o wide +kubectl get nodes -L feature.node.kubernetes.io/pci-10de.present,nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu.product,nvidia.com/mig.config,nvidia.com/mig.config.state +kubectl get pods -n gpu-operator -o wide +kubectl get ds -n gpu-operator +kubectl get clusterpolicy +kubectl get events -n gpu-operator --sort-by='.lastTimestamp' | tail -80 +kubectl get runtimeclass || true +``` + +Resource advertisement: + +```bash +kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, allocatable_gpu: .status.allocatable["nvidia.com/gpu"], capacity_gpu: .status.capacity["nvidia.com/gpu"]}' +``` + +Core workload: + +```bash +kubectl apply -f manifests/cuda-vectoradd.yaml +kubectl wait --for=jsonpath='{.status.phase}'=Succeeded pod/cuda-vectoradd --timeout=120s || true +kubectl get pod cuda-vectoradd -o jsonpath='{.status.phase}{"\n"}' +kubectl logs pod/cuda-vectoradd +kubectl delete -f manifests/cuda-vectoradd.yaml --ignore-not-found +``` + +Expected: pod phase `Succeeded` and logs include `Test PASSED`. + +## Optional Example Suite + +- Jupyter notebook: use the docs `tf-notebook.yaml` only when exposing a + NodePort is acceptable. Capture pod/service status and redact tokens. +- Time slicing: apply documented time-slicing config, then + `manifests/time-slicing-verification.yaml`; pass requires every requested + verification replica to reach `Available` and at least one `Test PASSED` log + from each pod before cleanup. Align the ConfigMap replica count with the + verification manifest's replica count, or reduce the manifest replica count + to match the approved ConfigMap value. Restore the original + `ClusterPolicy.spec.devicePlugin.config` and verify node GPU labels/capacity + after the test. +- MIG: run only after approved MIG configuration; pass requires expected MIG + resources and matching workload success. + +When optional examples mutate the live `ClusterPolicy` directly instead of +Helm values, record that ownership distinction before maintenance actions. +Cleanup is not complete until the test namespace/config, live ClusterPolicy, +Helm values, node GPU labels/capacity, and a final CUDA smoke all show the +baseline state. + +## Drift Signals + +- Kube context changes unexpectedly. +- GPU labels disappear. +- Driver/device-plugin daemonset ready count falls below desired. +- `nvidia.com/gpu` allocatable drops or disappears. +- Device-plugin logs mark GPUs unhealthy. +- RuntimeClasses change unexpectedly after CDI/NRI changes. +- Upgrade labels remain non-terminal after the approved window. +- MIG state remains pending/rebooting/failure-like. + +## Escalation Package + +Include health ladder output, workload manifest/logs, node labels/resources, +`ClusterPolicy`, Helm values, recent events, device-plugin/toolkit/DCGM logs, +runtime version, and whether host/runtime changes occurred. diff --git a/gpu-operator-devops-agent/skills/gpu-operator-troubleshoot/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-troubleshoot/SKILL.md new file mode 100644 index 000000000..3c2e818c6 --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-troubleshoot/SKILL.md @@ -0,0 +1,126 @@ +--- +name: gpu-operator-troubleshoot +description: Diagnose GPU Operator failures by command context and earliest failing layer, with safe recovery and escalation. +tags: [gpu-operator, troubleshoot, runtime, driver, xid, mig] +--- + +# GPU Operator Troubleshoot + +Use for failed installs, not-ready `ClusterPolicy`, missing GPUs, runtime +handler errors, stuck upgrades, MIG issues, and DCGM failures. + +## First Rule + +Take inventory, prove command context, then find the earliest failed layer: + +1. Intended sandbox/cluster context. +2. Host GPU visible. +3. Kubernetes nodes ready. +4. NFD/GPU labels present. +5. Driver healthy or host driver accepted. +6. Toolkit/CDI/NRI runtime injection healthy. +7. Device plugin advertises resources. +8. Workload manifest succeeds. +9. Optional observability/MIG/example path succeeds. + +## Evidence Commands + +Preferred inventory path: + +```bash +scripts/inventory-gpu-operator-environment.sh +``` + +If the script is unavailable, collect the same facts manually: + +```bash +hostname +echo "KUBECONFIG=${KUBECONFIG:-unset}" +kubectl config current-context +kubectl get nodes -o wide +kubectl get pods -n gpu-operator -o wide +kubectl get ds -n gpu-operator +kubectl get clusterpolicy -o yaml +kubectl get nodes -L feature.node.kubernetes.io/pci-10de.present,nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu-driver-upgrade-state,nvidia.com/mig.config,nvidia.com/mig.config.state +kubectl get runtimeclass || true +kubectl get events -A --sort-by='.lastTimestamp' | tail -100 +``` + +For failing pods: + +```bash +kubectl describe pod -n gpu-operator +kubectl logs -n gpu-operator --all-containers --prefix --timestamps +kubectl logs -n gpu-operator --all-containers --prefix --timestamps --previous +``` + +## Branches + +### Helm Or Kubectl Cannot Reach Fresh K3s + +Likely context issue. Set `KUBECONFIG=/etc/rancher/k3s/k3s.yaml`, then rerun +read-only proof. Do not reinstall GPU Operator until context is proven. + +### Pods Blocked By Pod Security + +Events mention baseline/restricted PodSecurity or privileged container denial. +Label namespace privileged with approval: + +```bash +kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged +``` + +Then inspect pods/events before retrying Helm. + +### Runtime Handler Error + +For `no runtime for "nvidia" is configured`, classify as toolkit/runtime +injection. Load `references/branch-matrix.md`, then check: + +```bash +kubectl get clusterpolicy -o jsonpath='{.items[0].spec.cdi.enabled}{" "}{.items[0].spec.cdi.nriPluginEnabled}{"\n"}' 2>/dev/null || kubectl get clusterpolicy cluster-policy -o jsonpath='{.spec.cdi.enabled}{" "}{.spec.cdi.nriPluginEnabled}{"\n"}' +kubectl get runtimeclass || true +kubectl logs -n gpu-operator -l app=nvidia-container-toolkit-daemonset --all-containers --tail=200 +``` + +K3s branch: + +- If runtime is in the documented NRI range, CDI+NRI is a candidate. +- If K3s/containerd is `2.3.x` or otherwise newer than the documented NRI + range, prefer explicit K3s toolkit env when paths exist. +- Use newer-than-documented NRI only as an explicit sandbox experiment with + workload validation. + +### Missing `nvidia.com/gpu` + +Check device-plugin logs and Xid: + +```bash +kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset --all-containers --tail=200 +sudo dmesg | grep -Ei 'NVRM|Xid|nouveau' | tail -80 +``` + +Critical Xid means hardware/driver health; do not fix by restarting the plugin. + +### Driver Upgrade Stuck + +```bash +kubectl get node -l nvidia.com/gpu.present \ + -ojsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.nvidia\.com/gpu-driver-upgrade-state}{"\n"}{end}' +kubectl get events -A --sort-by='.lastTimestamp' | grep GPUDriverUpgrade +kubectl logs -n gpu-operator deployment/gpu-operator | grep controllers.Upgrade +``` + +If `upgrade-failed`, fix the recorded cause before relabeling +`upgrade-required`. + +### MIG Request With Active Workloads + +Stop until workload owners approve disruption. Record current clients, cordon +plan, desired profile, possible reboot, and rollback to `all-disabled`. + +## Escalation Package + +Use `gpu-operator-evidence-bundle`. Include context proof, first failing layer, +commands run, logs/events, host/runtime facts, Helm values, proposed next +mutation, risk, and rollback. From c524af0857e2db70713ac5d7e9ec607c9027ec8e Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Mon, 20 Jul 2026 10:58:13 -0700 Subject: [PATCH 2/3] Add gpu-operator-uninstall skill to the DevOps Agent package MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A doc-backed uninstall operator skill — Helm release removal, CRD cleanup (operator.cleanupCRD post-delete hook or manual deletion), host driver-module unload (rmmod), and the --no-hooks fallback — grounded in the GPU Operator uninstall documentation. Probe-first preconditions and gated destructive steps. Signed-off-by: Andrew Chen --- gpu-operator-devops-agent/AGENTS.md | 1 + .../skills/gpu-operator-uninstall/SKILL.md | 195 ++++++++++++++++++ 2 files changed, 196 insertions(+) create mode 100644 gpu-operator-devops-agent/skills/gpu-operator-uninstall/SKILL.md diff --git a/gpu-operator-devops-agent/AGENTS.md b/gpu-operator-devops-agent/AGENTS.md index 7b2fb45ef..bcb59913d 100644 --- a/gpu-operator-devops-agent/AGENTS.md +++ b/gpu-operator-devops-agent/AGENTS.md @@ -121,6 +121,7 @@ helm get values gpu-operator -n gpu-operator -o yaml | Fresh install or install plan | `skills/gpu-operator-install/SKILL.md` | | Health, workload validation, drift check | `skills/gpu-operator-operate/SKILL.md` | | Upgrade, rollback, driver/MIG/config maintenance | `skills/gpu-operator-maintain/SKILL.md` | +| Remove the product — uninstall the Helm release, clean up CRDs, unload host driver modules | `skills/gpu-operator-uninstall/SKILL.md` | | Failed install, missing GPU, runtime errors, Xid, stuck upgrade | `skills/gpu-operator-troubleshoot/SKILL.md` | | Support/debug handoff | `skills/gpu-operator-evidence-bundle/SKILL.md` | | Record a reusable lesson, anti-pattern, discovered resource, operating parameter, or upstream issue draft | `skills/gpu-operator-improve/SKILL.md` | diff --git a/gpu-operator-devops-agent/skills/gpu-operator-uninstall/SKILL.md b/gpu-operator-devops-agent/skills/gpu-operator-uninstall/SKILL.md new file mode 100644 index 000000000..cb91a35fc --- /dev/null +++ b/gpu-operator-devops-agent/skills/gpu-operator-uninstall/SKILL.md @@ -0,0 +1,195 @@ +--- +name: gpu-operator-uninstall +description: Cleanly uninstall the GPU Operator — remove the Helm release, clean up CRDs, unload host driver modules, and verify teardown. High-blast-radius, gated lifecycle operation; not a troubleshooting shortcut. +tags: [gpu-operator, uninstall, teardown, helm, crd, cleanup] +--- + +# GPU Operator Uninstall + +Remove the GPU Operator and its operands from a **test-owned** cluster and +verify the node is clean. Every mutating step here is high blast radius: it +deletes cluster-wide CRDs, tears down GPU operands, and can leave or unload +host kernel modules. Treat uninstall as a deliberate lifecycle action. + +> **Uninstall is not a troubleshooting shortcut.** A failed install or a broken +> operand is almost never fixed by uninstall/reinstall as a first response — +> that erases the evidence. Route failures through +> `skills/gpu-operator-troubleshoot/SKILL.md` first. Uninstall only when the +> goal is genuinely to remove the product. + +## Preconditions + +- **Run inventory first.** Load `references/environment-inventory.md` and run + `scripts/inventory-gpu-operator-environment.sh` (or the Step 0 evidence + below) before any deletion. Do not uninstall from assumption. +- **Explicit approval for teardown**, plus a maintenance window if GPU + workloads or the node itself may be interrupted. +- **`kubectl`/`helm` context proven** to point at the intended test-owned + cluster (`kubectl config current-context`), never production, shared, or + another team's cluster. +- **Blast radius + rollback named** before proceeding: uninstall removes the + `gpu-operator` release, the `ClusterPolicy`/`NVIDIADriver` CRDs (if CRD + cleanup is chosen), and — on operator-managed-driver nodes — requires + unloading kernel modules or a reboot to fully clear the driver. Rollback = + reinstall via `skills/gpu-operator-install/SKILL.md`. +- **Know the driver ownership.** If `driver.enabled=false`, the host/platform + owns the NVIDIA driver — do NOT unload host driver modules as part of this + uninstall; leave the host driver to its own lifecycle. + +## Command Environment + +```bash +export KUBECONFIG="${KUBECONFIG:-/etc/rancher/k3s/k3s.yaml}" +kubectl config current-context +helm version +``` + +## Step 0 — Capture pre-uninstall inventory (read-only) + +Status: docs-derived. Establish what exists before deleting anything. + +```bash +helm list -n gpu-operator +kubectl get clusterpolicy -o yaml > clusterpolicy-before-uninstall.yaml +kubectl get nvidiadrivers.nvidia.com -A -o yaml > nvidiadrivers-before-uninstall.yaml || true +kubectl get pods -n gpu-operator -o wide +kubectl get crd | grep -E 'nvidia.com|nodefeaturerules' +kubectl get nodes -L nvidia.com/gpu.present,nvidia.com/gpu.count,nvidia.com/gpu.deploy.driver +``` + +Record whether `driver.enabled` is true (operator-managed) or false +(host-managed) from `helm get values gpu-operator -n gpu-operator -o yaml`; it +decides Step 5. + +## Step 1 — (Optional) Delete NVIDIA driver custom resources — GATE + +Status: docs-derived (GPU Operator "Uninstalling the GPU Operator", step 1). +Only when the operator manages `NVIDIADriver` CRs and you intend to remove them. + +```bash +kubectl get nvidiadrivers +# for each returned driver CR: +kubectl delete nvidiadriver +# then remove the CRD itself (cluster-wide, GATE): +kubectl delete crd nvidiadrivers.nvidia.com +``` + +Deleting the CRD is cluster-wide and irreversible without reinstall. Confirm +before running. + +## Step 2 — Delete the Operator Helm release — GATE + +Status: docs-derived (GPU Operator "Uninstalling the GPU Operator", step 2). + +```bash +helm delete -n gpu-operator $(helm list -n gpu-operator | grep gpu-operator | awk '{print $1}') +``` + +This tears down all GPU operands (driver, toolkit, device-plugin, DCGM, +validators). If the release name is known, prefer it explicitly: +`helm uninstall gpu-operator -n gpu-operator`. + +## Step 3 — Confirm operands are terminating (read-only) + +Status: docs-derived. + +```bash +kubectl get pods -n gpu-operator +# expected: pods deleting, then "No resources found." +``` + +## Step 4 — CRD cleanup — GATE + +Status: docs-derived (GPU Operator uninstall docs, CRD-cleanup note). + +Helm does **not** delete CRDs on chart removal by default, so `clusterpolicy` +and `nvidiadrivers` CRDs remain: + +```bash +kubectl get crd clusterpolicies.nvidia.com +``` + +Two supported ways to clean them up: + +1. **Post-delete hook (set at install/upgrade time):** the chart parameter + `operator.cleanupCRD=true` enables a `post-delete` hook that removes the + CRDs automatically on chart deletion. It is disabled by default and must + have been set with `--set operator.cleanupCRD=true` during a prior install + or upgrade. The hook uses the Operator image (see Step 4a). +2. **Manual deletion (cluster-wide, GATE):** + ```bash + kubectl delete crd clusterpolicies.nvidia.com + ``` + Deleting a CRD deletes all its custom resources cluster-wide. Confirm no + other tenant relies on it before running. + +### Step 4a — Hook-failure fallback + +Status: docs-derived (GPU Operator uninstall docs, Helm-hooks note). + +The `cleanupCRD` hook runs the Operator image. If that image cannot be pulled +(network error, or an invalid NGC registry secret on NVAIE), the hook fails and +`helm delete` can hang. In that case, delete the chart with hooks disabled and +clean the CRDs manually: + +```bash +helm delete gpu-operator -n gpu-operator --no-hooks +kubectl delete crd clusterpolicies.nvidia.com nvidiadrivers.nvidia.com +``` + +## Step 5 — Unload host driver modules (operator-managed driver only) — GATE + +Status: docs-derived (GPU Operator uninstall docs, driver-modules note). + +Only when `driver.enabled=true` was in use. After uninstall, NVIDIA driver +modules may still be loaded on the node. Either reboot the node (with owner +approval) or unload the modules: + +```bash +sudo rmmod nvidia_modeset nvidia_uvm nvidia +``` + +If `rmmod` reports the module is in use, GPU workloads or processes still hold +it — stop and collect evidence rather than forcing. **If `driver.enabled=false` +(host-managed driver), skip this step entirely** — the host driver is not +yours to unload. + +## Step 6 — Verify teardown (read-only) + +Status: docs-derived + synthesized verification composition. + +```bash +kubectl get pods -n gpu-operator # No resources found +helm list -n gpu-operator # release gone +kubectl get crd | grep -E 'clusterpolicies.nvidia.com|nvidiadrivers.nvidia.com' || echo "CRDs removed" +kubectl get nodes -L nvidia.com/gpu.present # GPU labels cleared where operator-managed +nvidia-smi 2>/dev/null || echo "no NVIDIA runtime (expected on operator-managed-driver nodes after module unload)" +``` + +Uninstall is complete only when the release is gone, chosen CRDs are removed, +operand pods are gone, and (for operator-managed drivers) the modules are +unloaded or the node rebooted. + +## Rollback / Recovery + +To restore, reinstall via `skills/gpu-operator-install/SKILL.md` using the +values captured in Step 0. CRD deletion is not reversible without reinstall; +the operator re-creates its CRDs on the next install. + +## Stop / Escalation Conditions + +Stop and produce an escalation bundle +(`skills/gpu-operator-evidence-bundle/SKILL.md`) when: a `rmmod` reports the +module is still in use, `helm delete` hangs on a failing hook and `--no-hooks` +does not clear it, a CRD deletion would affect resources outside this +test-owned cluster, the node is host-managed-driver and a reboot/drain is +required but not approved, or any deletion targets shared/production +infrastructure. + +## Sources + +- NVIDIA GPU Operator documentation — "Uninstalling the GPU Operator" + (`gpu-operator/uninstall.rst`): Helm release removal, the + `operator.cleanupCRD` post-delete hook, manual CRD deletion, the + driver-modules-remain note (`rmmod`), and the `--no-hooks` fallback. +- Helm CRD lifecycle: Helm does not delete CRDs on chart removal by default. From 07e0e496145974537f9383f8f32771155c58b04b Mon Sep 17 00:00:00 2001 From: Andrew Chen Date: Mon, 20 Jul 2026 17:01:07 -0700 Subject: [PATCH 3/3] Update DevOps Agent README/AGENTS inventory for the uninstall skill The gpu-operator-uninstall skill was added to the package but the README 'What's in it' table and 'Skill coverage' feedback list still enumerated six skills and omitted it, and the purpose verb-lists dropped removal. Add gpu-operator-uninstall to the README inventory (now seven SOPs) + feedback list, and add 'uninstall' to the README and AGENTS purpose lines so the docs match the package + AGENTS skill-routing table. Signed-off-by: Andrew Chen --- gpu-operator-devops-agent/AGENTS.md | 4 ++-- gpu-operator-devops-agent/README.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/gpu-operator-devops-agent/AGENTS.md b/gpu-operator-devops-agent/AGENTS.md index bcb59913d..503a50a33 100644 --- a/gpu-operator-devops-agent/AGENTS.md +++ b/gpu-operator-devops-agent/AGENTS.md @@ -1,7 +1,7 @@ # NVIDIA GPU Operator DevOps Agent Package -Purpose: help an AI DevOps agent install, operate, maintain, and troubleshoot -NVIDIA GPU Operator from public docs/source and live environment evidence. This +Purpose: help an AI DevOps agent install, operate, maintain, uninstall, and +troubleshoot NVIDIA GPU Operator from public docs/source and live environment evidence. This package is operational guidance, not a substitute for change approval. ## Supported Scope diff --git a/gpu-operator-devops-agent/README.md b/gpu-operator-devops-agent/README.md index e3941b35c..e81217ac0 100644 --- a/gpu-operator-devops-agent/README.md +++ b/gpu-operator-devops-agent/README.md @@ -10,7 +10,7 @@ SPDX-License-Identifier: Apache-2.0 A prototype **DevOps agent package** for the NVIDIA GPU Operator: a curated `AGENTS.md` front door + skills + references that let an AI coding/ops agent (Claude, Cursor, etc.) help an operator **install / operate / maintain / -troubleshoot** the GPU Operator by reading and reasoning over the **public** +uninstall / troubleshoot** the GPU Operator by reading and reasoning over the **public** GPU Operator docs and source — instead of a fresh agent guessing from an unstructured search. @@ -23,7 +23,7 @@ routable operating charter with an explicit safety model. | Path | What it is | |---|---| | `AGENTS.md` | Agent front door / operating charter — supported scope, source precedence, bootstrap policy, safety policy, skill routing, operating invariants, stop/escalation conditions | -| `skills/` | Six operator SOPs: `gpu-operator-install`, `-operate`, `-maintain`, `-troubleshoot`, `-evidence-bundle`, `-improve` | +| `skills/` | Seven operator SOPs: `gpu-operator-install`, `-operate`, `-maintain`, `-uninstall`, `-troubleshoot`, `-evidence-bundle`, `-improve` | | `references/` | The reasoning substrate — `models.md` (the layered success ladder), `branch-matrix.md` (staged branch decisions), `command-cards.md` (validated recipes), `field-scenario-cards.md` + `failure-signatures.md` (troubleshooting), `environment-inventory.md`, `supported-scope.md`, `claim-ledger.md`, others | | `scripts/` | Read-only inventory + a few guarded mutation helpers (host-driver K3s install, CUDA VectorAdd validation, time-slicing) | | `runtime-memory/` | Seed files the agent appends to as it learns (insights, anti-patterns, discovered resources, operating parameters) | @@ -97,7 +97,7 @@ operational guidance, **not** a substitute for change approval. - **Layered model** — does `references/models.md` match how the GPU Operator actually fails and recovers (NFD → driver → toolkit → device-plugin → validator → workload)? -- **Skill coverage** — are install / operate / maintain / troubleshoot / +- **Skill coverage** — are install / operate / maintain / uninstall / troubleshoot / evidence-bundle / improve the right operator buckets? What's missing? ## Provenance