docs: CNT 1.20 updates

container-toolkit/cdi-support.md

@@ -161,6 +161,59 @@ directly:
 $ sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml
 ```
 
+## JIT-CDI Mode
+
+Just-in-time CDI (JIT-CDI) mode generates an in-memory CDI specification for the
+NVIDIA devices that a container requests. The NVIDIA Container Runtime uses this
+specification to update the container configuration. JIT-CDI mode does not write
+a persistent specification to `/var/run/cdi` or `/etc/cdi`.
+
+When `nvidia-container-runtime.mode` is set to `auto`, the runtime selects JIT-CDI
+mode on systems that use the NVIDIA Management Library (NVML) or Windows
+Subsystem for Linux 2 (WSL2). Native CDI-enabled runtimes still use the
+persistent specifications described in
+[Automatic CDI Specification Generation](#automatic-cdi-specification-generation).
+
+### Disabling Hooks in JIT-CDI Mode
+
+You can prevent JIT-CDI mode from adding specific hooks to an in-memory
+specification. The supported hook names are `chmod`, `create-symlinks`,
+`disable-device-node-modification`, `enable-cuda-compat`,
+`update-application-profile`, and `update-ldcache`. The deprecated `chmod` hook
+is disabled by default. Use `all` to disable every hook.
+
+The following command disables the application-profile and dynamic-linker cache
+hooks. Separate multiple values with a colon:
+
+```console
+$ sudo nvidia-ctk config --in-place \
+    --set nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks=update-application-profile:update-ldcache
+```
+
+The resulting configuration contains the following settings:
+
+```toml
+[nvidia-container-runtime.modes.jit-cdi]
+nvcdi-disable-hooks = ["update-application-profile", "update-ldcache"]
+```
+
+```{warning}
+Disable only the hook that conflicts with your environment. Hooks configure
+libraries, links, device behavior, and GPU visibility in the container. Disabling
+a required hook can prevent an application from starting or can expose more GPUs
+to EGL and Vulkan applications than the container requested.
+```
+
+For a persistent specification, pass `--disable-hook` once for each hook when you
+[generate the CDI specification manually](#manual-cdi-specification-generation):
+
+```console
+$ sudo nvidia-ctk cdi generate \
+    --disable-hook update-application-profile \
+    --disable-hook update-ldcache \
+    --output=/var/run/cdi/nvidia.yaml
+```
+
 ## Running a Workload with CDI
 
 Using CDI to inject NVIDIA devices can conflict with using the NVIDIA Container Runtime hook.
@@ -191,6 +244,32 @@ $ podman run --rm \
 The preceding sample command requests the full GPU with index 0 and the first MIG device on GPU 1.
 The output should show only the UUIDs of the requested devices.
 
+### Graphics and OpenCL Workloads
+
+CDI specifications include the driver libraries and configuration files that
+graphics and OpenCL applications require. When the NVIDIA OpenCL installable
+client driver (ICD) file is present on the host, the specification mounts it at
+`/etc/OpenCL/vendors/nvidia.icd` in the container.
+
+CDI specifications also include the `update-application-profile` hook. At
+container creation, this hook writes
+`/etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf`. The profile
+limits EGL and Vulkan visibility to the physical GPUs that are available in the
+container. As a result, applications do not enumerate unassigned physical GPUs.
+
+If the application-profile hook conflicts with an application, disable it only
+for that workload or environment. For JIT-CDI mode, refer to
+[Disabling Hooks in JIT-CDI Mode](#disabling-hooks-in-jit-cdi-mode). For a
+persistent specification, use the `--disable-hook update-application-profile`
+option described in [Manual CDI Specification Generation](#manual-cdi-specification-generation).
+
+### IMEX Channels
+
+Containers can request specific NVIDIA IMEX channels with the
+`NVIDIA_IMEX_CHANNELS` environment variable. For supported values, an example,
+and validation errors, refer to
+[Requesting IMEX Channels](docker-specialized.md#requesting-imex-channels).
+
 ## Using CDI with Non-CDI-Enabled Runtimes
 
 To support runtimes that do not natively support CDI, you can configure the NVIDIA Container Runtime in a `cdi` mode.
@@ -237,4 +316,4 @@ $ docker run --rm -ti --runtime=nvidia \
 - [Container Device Interface](https://github.com/cncf-tags/container-device-interface) (CDI) specification from the Container Device Interface repository on GitHub.
 - [How to configure CDI](https://github.com/cncf-tags/container-device-interface#how-to-configure-cdi) from the GitHub repository provides an overview
   of manual configuration for CRI-O, containerd, and Podman.
-  The NVIDIA Container Toolkit performs the configuration for you.
+  The NVIDIA Container Toolkit performs the configuration for you.

container-toolkit/docker-specialized.md

@@ -111,6 +111,34 @@ The following examples show common usage:
         nvidia/cuda nvidia-smi
    ```
 
+(requesting-imex-channels)=
+
+### Requesting IMEX Channels
+
+Use the `NVIDIA_IMEX_CHANNELS` environment variable to request NVIDIA IMEX
+channels for a container. Specify one or more numeric channel IDs as a
+comma-separated list.
+
+The following command requests channels 0 and 1:
+
+```console
+$ docker run --rm --runtime=nvidia \
+    -e NVIDIA_VISIBLE_DEVICES=all \
+    -e NVIDIA_IMEX_CHANNELS=0,1 \
+    <image> <command>
+```
+
+In CDI and JIT-CDI mode, each channel ID must meet both requirements:
+
+- The ID is in the range from 0 through 1,048,575.
+- The corresponding `/dev/nvidia-caps-imex-channels/channel<ID>` device exists
+  on the host.
+
+If either requirement is not met, container creation fails with an error that
+identifies the invalid or missing channel. Inspect
+`/dev/nvidia-caps-imex-channels/` on the host and request only the channel IDs
+that are present.
+
 ### Driver Capabilities
 
 The `NVIDIA_DRIVER_CAPABILITIES` variable controls which driver libraries and binaries are mounted inside the container.
@@ -146,13 +174,14 @@ The following table describes the supported driver capabilities:
       - Description
 
     * - ``compute``
-      - required for CUDA and OpenCL applications.
+      - Required for CUDA and OpenCL applications. When present on the host,
+        the NVIDIA OpenCL ICD file is also available in the container.
 
     * - ``compat32``
       - required for running 32-bit applications.
 
     * - ``graphics``
-      - required for running OpenGL and Vulkan applications.
+      - Required for running OpenGL, EGL, and Vulkan applications.
 
     * - ``utility``
       - required for using ``nvidia-smi`` and NVML.
@@ -178,6 +207,22 @@ For example, to allow usage of CUDA and NVML, specify the `compute` and `utility
 >     nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi
 > ```
 
+To run an OpenGL, EGL, or Vulkan application on a selected GPU, include the
+`graphics` capability. The following command makes GPU 0 and the graphics and
+utility driver components available to the container:
+
+```console
+$ docker run --rm --runtime=nvidia \
+    -e NVIDIA_VISIBLE_DEVICES=0 \
+    -e NVIDIA_DRIVER_CAPABILITIES=graphics,utility \
+    <image> <command>
+```
+
+When CDI or JIT-CDI injects the GPU, the toolkit also limits EGL and Vulkan
+visibility to the physical GPUs assigned to the container. For more information
+and exceptional compatibility settings, refer to
+[Graphics and OpenCL Workloads](cdi-support.md#graphics-and-opencl-workloads).
+
 ### Constraints
 
 The NVIDIA runtime also lets you define constraints on the configurations that the container supports.

container-toolkit/index.md

@@ -20,6 +20,7 @@ release-notes.md
 
 arch-overview.md
 Container Device Interface <cdi-support.md>
+Node Resource Interface <nri-support.md>
 docker-specialized.md
 ```
 

container-toolkit/install-guide.md

@@ -149,6 +149,49 @@ where `systemd` cgroup drivers are used that cause containers to lose access to
 * You installed a supported container engine (Docker, Containerd, CRI-O, Podman).
 * You installed the NVIDIA Container Toolkit.
 
+### Managing `config.toml`
+
+The `nvidia-ctk config` command reads
+`/etc/nvidia-container-runtime/config.toml` by default. Without `--in-place` or
+`--output`, the command writes the updated configuration to standard output and
+does not change the source file.
+
+Preview a change before you write it to the host:
+
+```console
+$ sudo nvidia-ctk config \
+    --set nvidia-container-runtime.log-level=debug
+```
+
+Review the output. Then, add `--in-place` to update the source file:
+
+```console
+$ sudo nvidia-ctk config --in-place \
+    --set nvidia-container-runtime.log-level=debug
+```
+
+You can specify `--set` more than once. The following command configures a log
+level and a log file in one update:
+
+```console
+$ sudo nvidia-ctk config --in-place \
+    --set nvidia-container-runtime.log-level=debug \
+    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log
+```
+
+For a list setting, separate elements with a colon. The following command makes
+`crun` the first low-level runtime candidate and retains `runc` as a fallback:
+
+```console
+$ sudo nvidia-ctk config --in-place \
+    --set nvidia-container-runtime.runtimes=crun:runc
+```
+
+When the same key appears more than once, the last value takes effect. The
+command preserves existing settings when you change a different setting. Review
+the preview before every production update, especially when you use a custom
+file with `--config-file`.
+
 (setting-up-docker)=
 
 ### Configuring Docker

container-toolkit/nri-support.md

@@ -0,0 +1,81 @@
+(nri-support)=
+
+# Support for Node Resource Interface
+
+## About the NRI Plugin
+
+The NVIDIA Container Toolkit container can run a Node Resource Interface (NRI)
+plugin. The plugin connects to an NRI-enabled container runtime and adds
+Container Device Interface (CDI) device requests to container configurations.
+This path is useful when the runtime cannot process the original CDI request
+directly.
+
+The plugin enforces different namespace rules for workload and management
+devices:
+
+- The plugin can inject ordinary workload devices, such as `nvidia.com/gpu`,
+  into containers in any namespace.
+- The plugin injects `management.nvidia.com/gpu` devices only into the toolkit
+  namespace or an additional namespace that an administrator explicitly
+  authorizes.
+
+## Enabling the NRI Plugin
+
+The container runtime must have NRI enabled, and the toolkit container must have
+access to the runtime's NRI socket. Keep the toolkit installer running as a
+daemon. The `--enable-nri-plugin` option is ignored when `--no-daemon` is set.
+
+Add the following options to the toolkit container entry point:
+
+```console
+$ nvidia-ctk-installer \
+    --enable-nri-plugin \
+    --nri-namespace=nvidia-gpu-operator
+```
+
+The `--nri-namespace` value must match the Kubernetes namespace that contains
+the toolkit pod. Use `--nri-socket` if the runtime exposes the NRI socket at a
+nondefault path.
+
+You can configure the same settings with environment variables:
+
+```yaml
+env:
+  - name: ENABLE_NRI_PLUGIN
+    value: "true"
+  - name: NRI_NAMESPACE
+    valueFrom:
+      fieldRef:
+        fieldPath: metadata.namespace
+```
+
+## Authorizing Management Devices in Additional Namespaces
+
+Use `--nri-management-cdi-device-namespaces` to allow a namespace to receive
+management CDI devices. You can repeat the option. The toolkit namespace is
+always allowed and does not need to appear in the list.
+
+The following example authorizes management devices in two additional
+namespaces:
+
+```console
+$ nvidia-ctk-installer \
+    --enable-nri-plugin \
+    --nri-namespace=nvidia-gpu-operator \
+    --nri-management-cdi-device-namespaces=gpu-monitoring \
+    --nri-management-cdi-device-namespaces=gpu-diagnostics
+```
+
+For a container deployment, the equivalent environment variable accepts a
+comma-separated list:
+
+```yaml
+env:
+  - name: NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES
+    value: "gpu-monitoring,gpu-diagnostics"
+```
+
+Namespace matching is exact. Wildcards are not supported. Allow only namespaces
+that contain trusted management workloads. A management CDI device can provide
+broader access than an ordinary workload device, so follow the principle of
+least privilege.

container-toolkit/release-notes.md

@@ -8,6 +8,62 @@
 
 This document describes the new features, improvements, fixes and known issues for the NVIDIA Container Toolkit.
 
+## NVIDIA Container Toolkit 1.20.0
+
+This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release.
+
+### Fixes and Features
+
+- CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container.
+  Graphics applications no longer see unassigned host GPUs through these APIs.
+- CUDA compatibility handling now uses `libcuda.so` ELF metadata whenever it is available.
+  This improves CUDA minor-version compatibility by selecting the container's compatibility libraries only when they are appropriate for the installed driver.
+- Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version.
+  This fix avoids both missing required libraries and injecting libraries from another installed driver version.
+- CDI specifications now include the NVIDIA OpenCL ICD file and the legacy `libnvidia-nvvm70.so` library when present.
+  OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically.
+- On WSL2, CDI discovery now includes additional `.so`, `.bin`, and `.dll` files from the NVIDIA driver store instead of relying only on a fixed file list.
+  This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update.
+- IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device.
+  Invalid requests fail with a clear error.
+- JIT-CDI mode now honors the `nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks` configuration option.
+  You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible.
+- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace.
+  For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide.
+- Updating `config.toml` no longer reverts previously modified options to their defaults.
+- CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries.
+  This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries.
+- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script.
+  NRI-based deployments can invoke the hook on hosts that do not provide a shell.
+- The `nvidia-cdi-refresh` systemd units are now installed and enabled using distribution-native package mechanisms, including migration of stale RPM enablement links during upgrades.
+  Automatic CDI specification refresh remains available more reliably after package installation and upgrades.
+
+### Packaging Changes
+
+- RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd.
+  The resulting packages can be installed on Amazon Linux 2.
+- Source package builds can use Podman by setting `DOCKER=podman`.
+  The build handles Podman's local image naming, SELinux volume labeling, and artifact-directory creation automatically.
+
+#### Enhancements to container-toolkit Container Images
+
+- The `container-toolkit` image now uses the non-development distroless base and includes a static BusyBox shell.
+  Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image.
+
+### Included Packages
+
+The following packages are included:
+
+- `nvidia-container-toolkit 1.20.0`
+- `nvidia-container-toolkit-base 1.20.0`
+- `libnvidia-container-tools 1.20.0`
+- `libnvidia-container1 1.20.0`
+
+The following `container-toolkit` containers are included:
+
+- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0`
+- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0-packaging`
+
 ## NVIDIA Container Toolkit 1.19.1
 
 This release of the NVIDIA Container Toolkit `v1.19.1` is a bugfix release.