Skip to content

Latest commit

 

History

History
577 lines (403 loc) · 27.4 KB

File metadata and controls

577 lines (403 loc) · 27.4 KB

Container Image Inventory

This page lists every container image AICR can deploy across all registered components. It is the canonical reference for security review, air-gap planning, and any workflow that needs to know "what does AICR pull onto my cluster."

The image set below is regenerated from the live Helm chart catalog and the embedded manifests under recipes/components/*/manifests/. The auto-generated section is refreshed weekly by the bom-refresh GitHub Action, which opens a chore PR whenever upstream chart rerenders cause drift. Contributors changing recipes are expected to regenerate locally with make bom-docs and commit the result alongside their change.

A machine-readable CycloneDX 1.6 JSON companion to this page is produced by make bom and published as a release asset. Tooling that consumes SBOMs (Trivy, Grype, Cosign attestation, in-toto) should prefer the JSON; this Markdown is the human-readable view.

Summary

  • Components: 50
  • Unique images: 112
  • Distinct registries: 11

Registries: 602401143452.dkr.ecr.us-west-2.amazonaws.com, cr.agentgateway.dev, docker.io, gcr.io, ghcr.io, gke.gcr.io, nvcr.io, public.ecr.aws, quay.io, registry.k8s.io, us-docker.pkg.dev

Rendering fidelity: catalog-parity: charts are rendered with the shared recipes/components/<name>/values.yaml; per-recipe overlay overrides are not applied

Components

Component Type Chart Pinned Version Images
agentgateway helm agentgateway v1.5.0 1
agentgateway-crds helm agentgateway-crds v1.5.0 0
aws-ebs-csi-driver helm aws-ebs-csi-driver/aws-ebs-csi-driver 2.59.0 6
aws-efa helm aws-efa-k8s-device-plugin v0.5.29 1
cert-manager helm jetstack/cert-manager v1.20.2 4
cert-manager-ocp manifest 0
cert-manager-ocp-olm manifest 0
dra-node-labeler manifest 1
dranet manifest 1
dynamo-platform helm dynamo-platform 1.4.2 1
gatekeeper helm gatekeeper/gatekeeper 3.22.2 3
gcp-driver-installer manifest 3
gke-gb200-rdma manifest 2
gke-nccl-tcpxo manifest 4
gpu-operator helm nvidia/gpu-operator v26.7.0 15
gpu-operator-ocp manifest 0
gpu-operator-ocp-olm manifest 0
grove helm grove-charts v0.1.0-alpha.12 1
k8s-aibom helm k8s-aibom 1.3.0 1
k8s-ephemeral-storage-metrics helm k8s-ephemeral-storage-metrics/k8s-ephemeral-storage-metrics 1.19.2 1
k8s-nim-operator helm k8s-nim-operator 3.1.0 1
k8s-nim-operator-ocp helm k8s-nim-operator 3.1.0 1
kai-scheduler helm kai-scheduler v0.16.9 12
kube-prometheus-stack helm prometheus-community/kube-prometheus-stack 84.4.0 8
kubeflow-trainer helm kubeflow-trainer 2.2.0 4
kueue helm kueue 0.19.3 1
mariadb-operator helm mariadb-operator 26.6.0 1
mariadb-operator-crds helm mariadb-operator-crds 26.6.0 0
network-operator helm nvidia/network-operator 26.4.1 12
network-operator-ocp manifest 0
network-operator-ocp-olm manifest 0
nfd helm node-feature-discovery 0.19.0 1
nfd-ocp manifest 0
nfd-ocp-olm manifest 0
node-problem-detector helm node-problem-detector 2.4.1 1
nodewright-customizations manifest 6
nodewright-operator helm nodewright v0.19.0 2
nvcre helm cluster-readiness-engine v0.2.0 1
nvidia-dra-driver-gpu helm dra-driver-nvidia-gpu 0.5.0 1
nvidia-dra-driver-gpu-ocp helm dra-driver-nvidia-gpu 0.5.0 1
nvsentinel helm nvsentinel v1.20.0 6
prometheus-adapter helm prometheus-community/prometheus-adapter 5.3.0 1
prometheus-adapter-ocp helm prometheus-community/prometheus-adapter 5.3.0 1
prometheus-operator-crds helm prometheus-community/prometheus-operator-crds 28.0.1 0
rdma-netns-exclusive manifest 1
slinky-slurm helm slurm 1.2.0 5
slinky-slurm-operator helm slurm-operator 1.2.0 2
slinky-slurm-operator-crds helm slurm-operator-crds 1.2.0 0
slinky-topograph helm topograph/topograph 1.0.0 1
slurm-accounting-mariadb helm mariadb-cluster 26.6.0 0

Version variants

These versions are explicitly pinned by the listed sources and differ from the component's registry default above.

Component Variant Version Declared By Images
kube-prometheus-stack 83.7.0 aks 8

Images by component

agentgateway

  • cr.agentgateway.dev/controller:v1.5.0

agentgateway-crds

No images extracted.

aws-ebs-csi-driver

  • public.ecr.aws/csi-components/csi-attacher:v4.11.0-eksbuild.4
  • public.ecr.aws/csi-components/csi-node-driver-registrar:v2.16.0-eksbuild.4
  • public.ecr.aws/csi-components/csi-provisioner:v6.2.0-eksbuild.3
  • public.ecr.aws/csi-components/csi-resizer:v2.1.0-eksbuild.4
  • public.ecr.aws/csi-components/livenessprobe:v2.18.0-eksbuild.4
  • public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.59.0

aws-efa

  • 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efa-k8s-device-plugin:v0.5.20

cert-manager

  • quay.io/jetstack/cert-manager-cainjector:v1.20.2
  • quay.io/jetstack/cert-manager-controller:v1.20.2
  • quay.io/jetstack/cert-manager-startupapicheck:v1.20.2
  • quay.io/jetstack/cert-manager-webhook:v1.20.2

cert-manager-ocp

No images extracted.

cert-manager-ocp-olm

No images extracted.

dra-node-labeler

  • docker.io/alpine/kubectl:1.36.2@sha256:01d138ce994b684abc62d9cfdff44de42a4c8996dcc12626dd0193afc3fb5a95

dranet

  • registry.k8s.io/networking/dranet:stable@sha256:3248d8a520584100a5e87a2b92039591ea3b83a5492cc3e2d881b20d3b18ada6

dynamo-platform

  • nvcr.io/nvidia/ai-dynamo/kubernetes-operator:1.4.2

gatekeeper

  • curlimages/curl:8.12.0
  • openpolicyagent/gatekeeper-crds:v3.22.2
  • openpolicyagent/gatekeeper:v3.22.2

gcp-driver-installer

  • cos-nvidia-installer:fixed
  • gcr.io/gke-release/nvidia-partition-gpu@sha256:e226275da6c45816959fe43cde907ee9a85c6a2aa8a429418a4cadef8ecdb86a
  • gke.gcr.io/pause:3.8@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830

gke-gb200-rdma

  • gke.gcr.io/pause:3.8@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830
  • us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-arm64:v1.1.2@sha256:6b7950cac6e6833661d4206920f5633b6e361b18bfd5315b63f9bf4a4b84a80e

gke-nccl-tcpxo

  • gcr.io/gke-release/nri-device-injector:1.0.25-gke.6@sha256:7704e2bd74b8edbb76b6913c7904cc2362f1fa887c4d4aba7b19778ea353537c
  • gke.gcr.io/pause:3.8@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830
  • ubuntu:26.04@sha256:9559ceb7c21e528e233e8dff26a0fb2682f4094cce06176eeb075d87a22b31de
  • us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/nccl-plugin-gpudirecttcpx-dev:v1.0.15@sha256:4c9f0de3f39455a2ea35e844e0fc92564ca5629f6b03250fde40e8160719dae4

gpu-operator

  • docker.io/library/busybox:1.38.0@sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616
  • nvcr.io/nvidia/cloud-native/dcgm:4.6.0-1-ubuntu24.04
  • nvcr.io/nvidia/cloud-native/gdrdrv:v2.6
  • nvcr.io/nvidia/cloud-native/k8s-cc-manager:v0.4.3
  • nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.12.0
  • nvcr.io/nvidia/cloud-native/k8s-mig-manager:v0.15.0
  • nvcr.io/nvidia/cloud-native/nvidia-fs:2.29.4
  • nvcr.io/nvidia/cloud-native/nvidia-sandbox-device-plugin:v0.0.5
  • nvcr.io/nvidia/cloud-native/vgpu-device-manager:v0.5.0
  • nvcr.io/nvidia/driver:580.173.02
  • nvcr.io/nvidia/gpu-operator:v26.7.0
  • nvcr.io/nvidia/k8s-device-plugin:v0.20.0
  • nvcr.io/nvidia/k8s/container-toolkit:v1.20.0
  • nvcr.io/nvidia/k8s/dcgm-exporter:4.6.0-4.8.3-distroless
  • nvcr.io/nvidia/kubevirt-gpu-device-plugin:v1.6.0

gpu-operator-ocp

No images extracted.

gpu-operator-ocp-olm

No images extracted.

grove

  • ghcr.io/ai-dynamo/grove/grove-operator:v0.1.0-alpha.12

k8s-aibom

  • ghcr.io/googlecloudplatform/k8s-aibom@sha256:f8e48d4edc44e6ee8e40a2ac6c5f60b190aa18d411a75702dc5798a77a039e8d

k8s-ephemeral-storage-metrics

  • ghcr.io/jmcgrath207/k8s-ephemeral-storage-metrics:1.19.2

k8s-nim-operator

  • nvcr.io/nvidia/cloud-native/k8s-nim-operator:v3.1.0

k8s-nim-operator-ocp

  • nvcr.io/nvidia/cloud-native/k8s-nim-operator:v3.1.0

kai-scheduler

  • ghcr.io/kai-scheduler/kai-scheduler/admission:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/binder:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/crd-upgrader:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/nodescaleadjuster:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/numa-placement-exporter:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/operator:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/podgroupcontroller:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/podgrouper:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/queuecontroller:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/resourcereservation:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/scalingpod:v0.16.9
  • ghcr.io/kai-scheduler/kai-scheduler/scheduler:v0.16.9

kube-prometheus-stack

  • docker.io/grafana/grafana:13.0.1
  • ghcr.io/jkroepke/kube-webhook-certgen:1.8.2
  • quay.io/kiwigrid/k8s-sidecar:2.7.1
  • quay.io/prometheus-operator/prometheus-operator:v0.90.1
  • quay.io/prometheus/alertmanager:v0.32.0
  • quay.io/prometheus/node-exporter:v1.11.1
  • quay.io/prometheus/prometheus:v3.11.3
  • registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0

kubeflow-trainer

  • ghcr.io/kubeflow/trainer/trainer-controller-manager:v2.2.0
  • pytorch/pytorch:2.11.0-cuda12.8-cudnn9-runtime@sha256:eee11b3b3872a8c838e35ef48f08b2d5def2080902c7f666831310ca1a0ef2be
  • registry.k8s.io/jobset/jobset:v0.11.0
  • us-docker.pkg.dev/gce-ai-infra/gpudirect-tcpxo/tcpgpudmarxd-dev:v1.0.21@sha256:8d9e10fd589a34ab8a0aa64f7e70ad075c8f1c69bea176350f8d211367697e3d

kueue

  • registry.k8s.io/kueue/kueue:v0.19.3

mariadb-operator

  • ghcr.io/mariadb-operator/mariadb-operator:26.6.0

mariadb-operator-crds

No images extracted.

network-operator

  • docker.io/library/busybox:1.38.0@sha256:dc2d74b28e4cf8984fa52af1f39bc7c3d9c73760b41a74d629f5d11b1ab28616
  • ghcr.io/k8snetworkplumbingwg/multus-cni:v4.2.1
  • ghcr.io/k8snetworkplumbingwg/plugins:v1.6.2-update.1
  • ghcr.io/k8snetworkplumbingwg/sriov-network-device-plugin:v3.9.0
  • ghcr.io/mellanox/nic-configuration-operator-daemon:v1.3.1
  • ghcr.io/mellanox/nic-configuration-operator:v1.3.1
  • ghcr.io/mellanox/nvidia-k8s-ipam:v0.2.0
  • nvcr.io/nvidia/cloud-native/network-operator:v26.4.1
  • nvcr.io/nvidia/doca/doca_telemetry:1.22.5-doca3.1.0-host
  • nvcr.io/nvidia/mellanox/doca-driver:doca3.2.0-25.10-1.2.8.0-2
  • nvcr.io/nvidia/mellanox/doca-driver:doca3.4.0-26.04-0.8.6.0-0
  • nvcr.io/nvidia/mellanox/k8s-rdma-shared-dev-plugin:network-operator-v26.4.1

network-operator-ocp

No images extracted.

network-operator-ocp-olm

No images extracted.

nfd

  • registry.k8s.io/nfd/node-feature-discovery:v0.19.0

nfd-ocp

No images extracted.

nfd-ocp-olm

No images extracted.

node-problem-detector

  • registry.k8s.io/node-problem-detector/node-problem-detector:v1.35.1

nodewright-customizations

  • ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.3.0@sha256:f17c951d60b519d097c20a3d9f49668f043a996adb31b9bb4db24a112a8f60a2
  • ghcr.io/nvidia/nodewright-packages/nvidia-setup:0.8.0@sha256:1551abdd54476f6bddce863d383903a0a76f8a91bb4daa53c4f3c9490b980dae
  • ghcr.io/nvidia/nodewright-packages/nvidia-tuned:0.10.0@sha256:124f3bdedbb651125cec1bf20d3e2f7fcbbe10588c44f36d8e5680b38b9ef8f6
  • ghcr.io/nvidia/nodewright-packages/nvidia-tuned:0.3.2@sha256:a8bdca40dbe36de9d7a13e6afada49870714784fd9a3b9ce08717d675978c2b6
  • ghcr.io/nvidia/nodewright-packages/nvidia-tuning-gke:0.1.2@sha256:6671d49f006afdbeefd8858f1fa1216f7748205bc42edab3340210a2cc459a81
  • ghcr.io/nvidia/skyhook-packages/shellscript:1.1.1

nodewright-operator

  • docker.io/alpine/kubectl:1.36.2@sha256:01d138ce994b684abc62d9cfdff44de42a4c8996dcc12626dd0193afc3fb5a95
  • ghcr.io/nvidia/nodewright/operator:v0.19.0@sha256:38e9a79125633aa633f8e499306b2219c582c6f62a7bfe4083928212e70f74a7

nvcre

  • ghcr.io/nvidia/cluster-readiness-engine/manager:v0.2.0

nvidia-dra-driver-gpu

  • registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.5.0

nvidia-dra-driver-gpu-ocp

  • registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.5.0

nvsentinel

  • ghcr.io/nvidia/nvsentinel/gpu-health-monitor:v1.20.0-dcgm-3.x
  • ghcr.io/nvidia/nvsentinel/gpu-health-monitor:v1.20.0-dcgm-4.x
  • ghcr.io/nvidia/nvsentinel/labeler:v1.20.0
  • ghcr.io/nvidia/nvsentinel/metadata-collector:v1.20.0
  • ghcr.io/nvidia/nvsentinel/platform-connectors:v1.20.0
  • ghcr.io/nvidia/nvsentinel/syslog-health-monitor:v1.20.0

prometheus-adapter

  • registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0

prometheus-adapter-ocp

  • registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.12.0

prometheus-operator-crds

No images extracted.

rdma-netns-exclusive

  • ghcr.io/nvidia/skyhook-packages/shellscript:1.1.1@sha256:5414b06e52c090d0842704f2580798064362d771f12421ccc8888186b5f5a3cf

slinky-slurm

  • docker.io/library/alpine:3.23.3
  • ghcr.io/slinkyproject/login-pyxis@sha256:9e782d1a645aff1dedc498d7a3256733cde55a152659f44716e8a5f0dca02028
  • ghcr.io/slinkyproject/slurmctld:26.05-ubuntu26.04
  • ghcr.io/slinkyproject/slurmd-pyxis@sha256:0c03f87d5b5725df2d11392702fb647922b3060c076e9ce4b4f13c9a67c904b3
  • ghcr.io/slinkyproject/slurmrestd:26.05-ubuntu26.04

slinky-slurm-operator

  • ghcr.io/slinkyproject/slurm-operator-webhook:1.2.0
  • ghcr.io/slinkyproject/slurm-operator:1.2.0

slinky-slurm-operator-crds

No images extracted.

slinky-topograph

  • ghcr.io/nvidia/topograph:v1.0.0

slurm-accounting-mariadb

No images extracted.

kube-prometheus-stack@83.7.0 (variant)

  • docker.io/grafana/grafana:12.4.3
  • ghcr.io/jkroepke/kube-webhook-certgen:1.8.1
  • quay.io/kiwigrid/k8s-sidecar:2.6.0
  • quay.io/prometheus-operator/prometheus-operator:v0.90.1
  • quay.io/prometheus/alertmanager:v0.32.0
  • quay.io/prometheus/node-exporter:v1.11.1
  • quay.io/prometheus/prometheus:v3.11.2
  • registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.18.0

How to read this list

Explicit vs. implicit images

AICR pins some images directly in this repository — in recipes/components/<name>/values.yaml or in embedded Kubernetes manifests under recipes/components/<name>/manifests/. Those are the explicit images. Everything else comes from upstream Helm charts that AICR consumes without overriding their image references; those are the implicit images. The per-component image counts in the table above reflect the union of both.

OLM-managed components are a third, uninventoried category. cert-manager-ocp, cert-manager-ocp-olm, gpu-operator-ocp, gpu-operator-ocp-olm, network-operator-ocp, network-operator-ocp-olm, nfd-ocp, and nfd-ocp-olm install their operator and operand images by resolving a ClusterServiceVersion (CSV) through the Red Hat OperatorHub catalog at install time — not from a local values.yaml or vendored manifest. This BOM cannot enumerate those images: they aren't declared anywhere in this repository, and the actual image digests are pinned by whichever CSV version OLM resolves from the subscribed channel on the target cluster. The 0-image rows for these components in the table above reflect that gap, not an empty deployment.

Air-gapped OpenShift deployments must separately mirror the relevant Red Hat certified-operator catalog (redhat-operators) alongside the images this BOM does track. See the OpenShift documentation on mirroring Operator catalogs and this repo's air-gap mirroring guide for the OLM-specific mirroring workflow.

The trade-off is intentional. Pinning an image gives reproducibility; deferring to the upstream chart lets security patches flow without an AICR release. The split is policy, not oversight — see the supply chain epic for how each component's policy is being made explicit.

Opt-in values enabled by a leaf override or mixin are a fourth gap. A handful of images only appear once a component's values, not just its enablement, are overridden outside the shared recipes/components/<name>/values.yaml this BOM renders (tools/bom/main.go's renderHelmComponent resolves each component against only its base values file, so it cannot see leaf or mixin overrides). Four known cases, none counted in the nvsentinel row's image count above. Three set a global.* toggle:

  • The nvsentinel-observability mixin sets global.auditLogging.enabled: true, which conditionally adds a fix-audit-log-permissions init container (docker.io/bitnamilegacy/os-shell:12-debian-12-r30) to the platform-connectors DaemonSet and labeler Deployment. It is a third-party image AICR does not otherwise mirror.
  • The nvsentinel-object-monitor mixin sets global.kubernetesObjectMonitor.enabled, turning on the chart's kubernetes-object-monitor subchart and pulling in ghcr.io/nvidia/nvsentinel/kubernetes-object-monitor:v1.20.0. That image is in AICR's weekly image scan despite not being built here, since nothing else would surface a CVE in it.
  • The nvsentinel-nic-health-monitor mixin sets global.nicHealthMonitor.enabled, turning on the chart's nic-health-monitor subchart and pulling in ghcr.io/nvidia/nvsentinel/nic-health-monitor:v1.20.0. Its chown init container reuses docker.io/bitnamilegacy/os-shell:12-debian-12-r30 — the same third-party image the observability mixin above already pulls in, not a second one. Unlike the other two, this mixin is referenced by the shipped aks and oke-ol overlays, so every AKS and OKE recipe deploys these images; the other families do not.

A recipe composing any of these mixins with nvsentinel still enabled adds that mixin's images to what it deploys and mirrors; aicr bundle/aicr mirror on such a recipe surfaces them even though this static BOM cannot. A chain that disables nvsentinel (the OCP overlay, for example) can compose a mixin and ship none of them.

The nvsentinel-preflight mixin (see Preflight Checks) is the fourth case. Setting global.preflight.enabled: true on nvsentinel adds four images, all from ghcr.io/nvidia/nvsentinel/ at the chart's own version and therefore already covered by the NVIDIA mirroring path — but none of them appear in the nvsentinel row above:

Image Role
ghcr.io/nvidia/nvsentinel/preflight the admission webhook controller
ghcr.io/nvidia/nvsentinel/preflight-dcgm-diag injected init container: DCGM level-2 diagnostic
ghcr.io/nvidia/nvsentinel/preflight-nccl-loopback injected init container: NCCL loopback bandwidth test
ghcr.io/nvidia/nvsentinel/preflight-nccl-allreduce injected init container: NCCL all-reduce bandwidth test

TestNVSentinelPreflightChartRender pins all four against the rendered chart, so a bump that changes a repository fails there rather than silently diverging from this table.

Registries spanned

AICR pulls from a deliberately diverse set of registries:

  • nvcr.io — NVIDIA's primary container registry; GPU Operator, Network Operator, NIM Operator, Dynamo Platform.
  • ghcr.io — GitHub Container Registry; nvsentinel, nodewright, kai-scheduler, grove, kubeflow-trainer, k8s-ephemeral-storage-metrics.
  • quay.io — cert-manager and Prometheus components.
  • registry.k8s.io — Kubernetes SIG components (DRA driver, NFD, prometheus-adapter, kueue, csi-sidecars).
  • public.ecr.aws — AWS public artifacts (aws-ebs-csi-driver).
  • Regional ECR (<account>.dkr.ecr.<region>.amazonaws.com) — EKS-internal add-ons. The aws-efa entry below shows us-west-2 because that is the in-tree default; deployments in other regions override awsefa:image.repository at bundle or install time. See Regional registry overrides for the pattern.
  • gcr.io, gke.gcr.io, us-docker.pkg.dev — GCP/GKE add-ons (gke-nccl-tcpxo).
  • cr.agentgateway.dev — agentgateway (AI inference gateway).
  • docker.io — assorted upstream images (busybox, pytorch, etc.).

Customers running in air-gapped or private-registry environments need to mirror every registry above. A dedicated mirroring guide is tracked under #743.

Reproducibility

Two recipes rendered at the same chart version against the same registry should produce the same image set. Where charts are not yet pinned to a specific version, the upstream default determines the deployed images and the set can drift between renders — that's the drift the weekly refresh action surfaces. Tracking fully-deterministic deployments (chart-version pins, then digest pins for explicit refs) is the second stage of the supply chain epic; progress is tracked under issues #740, #748, and #749.

For chart-default sub-images that AICR cannot pin in-tree (e.g., the GPU Operator's ~15 sub-images, where the chart does not expose digest fields), the right answer is admission-time digest verification rather than per-image overrides — see #745.

Verifying supply-chain provenance

Presence is not trust. The commands below check whether any signature, SBOM, or in-toto attestation is attached to an image in its registry. They do not verify that the artifact was produced by the claimed publisher; that requires the publisher's public key or Sigstore certificate identity, which differs per upstream and is out of scope here. Treat a Y as "something is attached" — the strongest signal attainable without per-publisher trust roots.

The three checks are independent: an image may be signed without an SBOM, or carry an SBOM without an attestation, in any combination. Each subsection below shows the raw cosign invocation and how to interpret its output. The tools/s3c helper runs all three across every image in a component and prints a summary report.

Is it signed?

cosign tree <image>

A Signatures for an image tag: line in the output indicates a cosign signature is attached. Empty output (or no such line) means none is attached — the image is unsigned.

Does it have an SBOM?

cosign tree <image>

The same cosign tree output also reports SBOMs. An SBOMs for an image tag: line means an SBOM artifact is attached at the registry. Many publishers attach SBOMs as registry referrers rather than the legacy .sbom tag, but cosign tree surfaces both.

Does it have build provenance?

cosign download attestation <image> \
  | jq -r 'select(.payload != null) | .payload' \
  | base64 -d \
  | jq -r '.predicateType'

Any output line containing slsa or provenance (e.g., https://slsa.dev/provenance/v0.2) indicates an in-toto SLSA-style build provenance attestation is attached. A non-zero exit from the first cosign download attestation call means no attestation is attached.

Automated check

tools/s3c wraps the three commands above and emits a per-component report:

tools/s3c gpu-operator

Example output:

Component: nvidia-dra-driver-gpu (1 images)

Presence-only check: does NOT verify publisher trust/identity.
Y = artifact attached, - = artifact absent, ? = could not probe.

  Image                                                           Sig  SBOM  Prov  Notes
  --------------------------------------------------------------  ---  ----  ----  -----
  registry.k8s.io/dra-driver-nvidia/dra-driver-nvidia-gpu:v0.5.0  Y    -     -

Summary: 1/1 signed · 0/1 SBOM · 0/1 provenance

The script reads the per-component image list from this page, so the BOM inventory above is the source of truth — keep it in sync with make bom-docs before running. Requires cosign, jq, and awk on PATH.

Authentication and rate limits

cosign performs unauthenticated registry pulls by default. Both nvcr.io and ghcr.io rate-limit anonymous traffic and may return 429 when many images are probed in quick succession. cosign authenticates through the Docker credential chain (~/.docker/config.json), so a single docker login per registry raises the limits for every subsequent run:

# nvcr.io — use an NGC API key as the password.
echo "$NGC_API_KEY" | docker login nvcr.io -u '$oauthtoken' --password-stdin

# ghcr.io — use a personal access token with `read:packages` scope.
echo "$GH_TOKEN" | docker login ghcr.io -u "$GITHUB_USER" --password-stdin

Unreachable registries

Some registries cannot be probed from arbitrary networks; the script reports those images as ? and labels the reason in the Notes column rather than reporting them as absent. The most common cases:

  • Regional ECR (e.g., <account>.dkr.ecr.<region>.amazonaws.com) requires AWS credentials for that account/region. Reported as auth required. The aws-efa entry above is the canonical example; deployments override the registry per region at install time.
  • Authenticated mirrors (private mirrors fronting a public registry) require credentials that the local environment may not carry. Reported as auth required.
  • Transient network errors (DNS, TLS, timeouts) are reported as network unreachable and are typically resolved by re-running.

Distinguishing ? (could not probe) from - (probed and absent) keeps the report honest: an image that we could not reach is not the same as an image we know to be unsigned.

Regenerating locally

# Full BOM (CycloneDX JSON + Markdown) into dist/bom/
make bom

# Just regenerate this doc page from the live registry
make bom-docs

# Verify the committed page is in sync with the live registry
make bom-check

Both targets shell out to helm template for every chart, so an internet connection is required.

Related