diff --git a/container-toolkit/cdi-support.md b/container-toolkit/cdi-support.md index 1e9dd2273..ccb278ae2 100644 --- a/container-toolkit/cdi-support.md +++ b/container-toolkit/cdi-support.md @@ -161,6 +161,59 @@ directly: $ sudo nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml ``` +## JIT-CDI Mode + +Just-in-time CDI (JIT-CDI) mode generates an in-memory CDI specification for the +NVIDIA devices that a container requests. The NVIDIA Container Runtime uses this +specification to update the container configuration. JIT-CDI mode does not write +a persistent specification to `/var/run/cdi` or `/etc/cdi`. + +When `nvidia-container-runtime.mode` is set to `auto`, the runtime selects JIT-CDI +mode on systems that use the NVIDIA Management Library (NVML) or Windows +Subsystem for Linux 2 (WSL2). Native CDI-enabled runtimes still use the +persistent specifications described in +[Automatic CDI Specification Generation](#automatic-cdi-specification-generation). + +### Disabling Hooks in JIT-CDI Mode + +You can prevent JIT-CDI mode from adding specific hooks to an in-memory +specification. The supported hook names are `chmod`, `create-symlinks`, +`disable-device-node-modification`, `enable-cuda-compat`, +`update-application-profile`, and `update-ldcache`. The deprecated `chmod` hook +is disabled by default. Use `all` to disable every hook. + +The following command disables the application-profile and dynamic-linker cache +hooks. Separate multiple values with a colon: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks=update-application-profile:update-ldcache +``` + +The resulting configuration contains the following settings: + +```toml +[nvidia-container-runtime.modes.jit-cdi] +nvcdi-disable-hooks = ["update-application-profile", "update-ldcache"] +``` + +```{warning} +Disable only the hook that conflicts with your environment. Hooks configure +libraries, links, device behavior, and GPU visibility in the container. Disabling +a required hook can prevent an application from starting or can expose more GPUs +to EGL and Vulkan applications than the container requested. +``` + +For a persistent specification, pass `--disable-hook` once for each hook when you +[generate the CDI specification manually](#manual-cdi-specification-generation): + +```console +$ sudo nvidia-ctk cdi generate \ + --disable-hook update-application-profile \ + --disable-hook update-ldcache \ + --output=/var/run/cdi/nvidia.yaml +``` + ## Running a Workload with CDI Using CDI to inject NVIDIA devices can conflict with using the NVIDIA Container Runtime hook. @@ -191,6 +244,32 @@ $ podman run --rm \ The preceding sample command requests the full GPU with index 0 and the first MIG device on GPU 1. The output should show only the UUIDs of the requested devices. +### Graphics and OpenCL Workloads + +CDI specifications include the driver libraries and configuration files that +graphics and OpenCL applications require. When the NVIDIA OpenCL installable +client driver (ICD) file is present on the host, the specification mounts it at +`/etc/OpenCL/vendors/nvidia.icd` in the container. + +CDI specifications also include the `update-application-profile` hook. At +container creation, this hook writes +`/etc/nvidia/nvidia-application-profiles-rc.d/10-container.conf`. The profile +limits EGL and Vulkan visibility to the physical GPUs that are available in the +container. As a result, applications do not enumerate unassigned physical GPUs. + +If the application-profile hook conflicts with an application, disable it only +for that workload or environment. For JIT-CDI mode, refer to +[Disabling Hooks in JIT-CDI Mode](#disabling-hooks-in-jit-cdi-mode). For a +persistent specification, use the `--disable-hook update-application-profile` +option described in [Manual CDI Specification Generation](#manual-cdi-specification-generation). + +### IMEX Channels + +Containers can request specific NVIDIA IMEX channels with the +`NVIDIA_IMEX_CHANNELS` environment variable. For supported values, an example, +and validation errors, refer to +[Requesting IMEX Channels](docker-specialized.md#requesting-imex-channels). + ## Using CDI with Non-CDI-Enabled Runtimes To support runtimes that do not natively support CDI, you can configure the NVIDIA Container Runtime in a `cdi` mode. @@ -237,4 +316,4 @@ $ docker run --rm -ti --runtime=nvidia \ - [Container Device Interface](https://github.com/cncf-tags/container-device-interface) (CDI) specification from the Container Device Interface repository on GitHub. - [How to configure CDI](https://github.com/cncf-tags/container-device-interface#how-to-configure-cdi) from the GitHub repository provides an overview of manual configuration for CRI-O, containerd, and Podman. - The NVIDIA Container Toolkit performs the configuration for you. \ No newline at end of file + The NVIDIA Container Toolkit performs the configuration for you. diff --git a/container-toolkit/docker-specialized.md b/container-toolkit/docker-specialized.md index 6b88c1c5e..465308ab9 100644 --- a/container-toolkit/docker-specialized.md +++ b/container-toolkit/docker-specialized.md @@ -111,6 +111,34 @@ The following examples show common usage: nvidia/cuda nvidia-smi ``` +(requesting-imex-channels)= + +### Requesting IMEX Channels + +Use the `NVIDIA_IMEX_CHANNELS` environment variable to request NVIDIA IMEX +channels for a container. Specify one or more numeric channel IDs as a +comma-separated list. + +The following command requests channels 0 and 1: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=all \ + -e NVIDIA_IMEX_CHANNELS=0,1 \ + +``` + +In CDI and JIT-CDI mode, each channel ID must meet both requirements: + +- The ID is in the range from 0 through 1,048,575. +- The corresponding `/dev/nvidia-caps-imex-channels/channel` device exists + on the host. + +If either requirement is not met, container creation fails with an error that +identifies the invalid or missing channel. Inspect +`/dev/nvidia-caps-imex-channels/` on the host and request only the channel IDs +that are present. + ### Driver Capabilities The `NVIDIA_DRIVER_CAPABILITIES` variable controls which driver libraries and binaries are mounted inside the container. @@ -146,13 +174,14 @@ The following table describes the supported driver capabilities: - Description * - ``compute`` - - required for CUDA and OpenCL applications. + - Required for CUDA and OpenCL applications. When present on the host, + the NVIDIA OpenCL ICD file is also available in the container. * - ``compat32`` - required for running 32-bit applications. * - ``graphics`` - - required for running OpenGL and Vulkan applications. + - Required for running OpenGL, EGL, and Vulkan applications. * - ``utility`` - required for using ``nvidia-smi`` and NVML. @@ -178,6 +207,22 @@ For example, to allow usage of CUDA and NVML, specify the `compute` and `utility > nvidia/cuda:12.5.0-base-ubuntu22.04 nvidia-smi > ``` +To run an OpenGL, EGL, or Vulkan application on a selected GPU, include the +`graphics` capability. The following command makes GPU 0 and the graphics and +utility driver components available to the container: + +```console +$ docker run --rm --runtime=nvidia \ + -e NVIDIA_VISIBLE_DEVICES=0 \ + -e NVIDIA_DRIVER_CAPABILITIES=graphics,utility \ + +``` + +When CDI or JIT-CDI injects the GPU, the toolkit also limits EGL and Vulkan +visibility to the physical GPUs assigned to the container. For more information +and exceptional compatibility settings, refer to +[Graphics and OpenCL Workloads](cdi-support.md#graphics-and-opencl-workloads). + ### Constraints The NVIDIA runtime also lets you define constraints on the configurations that the container supports. diff --git a/container-toolkit/index.md b/container-toolkit/index.md index 3cab94725..45253d4f2 100644 --- a/container-toolkit/index.md +++ b/container-toolkit/index.md @@ -20,6 +20,7 @@ release-notes.md arch-overview.md Container Device Interface +Node Resource Interface docker-specialized.md ``` diff --git a/container-toolkit/install-guide.md b/container-toolkit/install-guide.md index 56282dab4..81480f170 100644 --- a/container-toolkit/install-guide.md +++ b/container-toolkit/install-guide.md @@ -149,6 +149,49 @@ where `systemd` cgroup drivers are used that cause containers to lose access to * You installed a supported container engine (Docker, Containerd, CRI-O, Podman). * You installed the NVIDIA Container Toolkit. +### Managing `config.toml` + +The `nvidia-ctk config` command reads +`/etc/nvidia-container-runtime/config.toml` by default. Without `--in-place` or +`--output`, the command writes the updated configuration to standard output and +does not change the source file. + +Preview a change before you write it to the host: + +```console +$ sudo nvidia-ctk config \ + --set nvidia-container-runtime.log-level=debug +``` + +Review the output. Then, add `--in-place` to update the source file: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug +``` + +You can specify `--set` more than once. The following command configures a log +level and a log file in one update: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.log-level=debug \ + --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log +``` + +For a list setting, separate elements with a colon. The following command makes +`crun` the first low-level runtime candidate and retains `runc` as a fallback: + +```console +$ sudo nvidia-ctk config --in-place \ + --set nvidia-container-runtime.runtimes=crun:runc +``` + +When the same key appears more than once, the last value takes effect. The +command preserves existing settings when you change a different setting. Review +the preview before every production update, especially when you use a custom +file with `--config-file`. + (setting-up-docker)= ### Configuring Docker diff --git a/container-toolkit/nri-support.md b/container-toolkit/nri-support.md new file mode 100644 index 000000000..d767d71b0 --- /dev/null +++ b/container-toolkit/nri-support.md @@ -0,0 +1,81 @@ +(nri-support)= + +# Support for Node Resource Interface + +## About the NRI Plugin + +The NVIDIA Container Toolkit container can run a Node Resource Interface (NRI) +plugin. The plugin connects to an NRI-enabled container runtime and adds +Container Device Interface (CDI) device requests to container configurations. +This path is useful when the runtime cannot process the original CDI request +directly. + +The plugin enforces different namespace rules for workload and management +devices: + +- The plugin can inject ordinary workload devices, such as `nvidia.com/gpu`, + into containers in any namespace. +- The plugin injects `management.nvidia.com/gpu` devices only into the toolkit + namespace or an additional namespace that an administrator explicitly + authorizes. + +## Enabling the NRI Plugin + +The container runtime must have NRI enabled, and the toolkit container must have +access to the runtime's NRI socket. Keep the toolkit installer running as a +daemon. The `--enable-nri-plugin` option is ignored when `--no-daemon` is set. + +Add the following options to the toolkit container entry point: + +```console +$ nvidia-ctk-installer \ + --enable-nri-plugin \ + --nri-namespace=nvidia-gpu-operator +``` + +The `--nri-namespace` value must match the Kubernetes namespace that contains +the toolkit pod. Use `--nri-socket` if the runtime exposes the NRI socket at a +nondefault path. + +You can configure the same settings with environment variables: + +```yaml +env: + - name: ENABLE_NRI_PLUGIN + value: "true" + - name: NRI_NAMESPACE + valueFrom: + fieldRef: + fieldPath: metadata.namespace +``` + +## Authorizing Management Devices in Additional Namespaces + +Use `--nri-management-cdi-device-namespaces` to allow a namespace to receive +management CDI devices. You can repeat the option. The toolkit namespace is +always allowed and does not need to appear in the list. + +The following example authorizes management devices in two additional +namespaces: + +```console +$ nvidia-ctk-installer \ + --enable-nri-plugin \ + --nri-namespace=nvidia-gpu-operator \ + --nri-management-cdi-device-namespaces=gpu-monitoring \ + --nri-management-cdi-device-namespaces=gpu-diagnostics +``` + +For a container deployment, the equivalent environment variable accepts a +comma-separated list: + +```yaml +env: + - name: NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES + value: "gpu-monitoring,gpu-diagnostics" +``` + +Namespace matching is exact. Wildcards are not supported. Allow only namespaces +that contain trusted management workloads. A management CDI device can provide +broader access than an ordinary workload device, so follow the principle of +least privilege. diff --git a/container-toolkit/release-notes.md b/container-toolkit/release-notes.md index 29521bfe9..394207e85 100644 --- a/container-toolkit/release-notes.md +++ b/container-toolkit/release-notes.md @@ -8,6 +8,62 @@ This document describes the new features, improvements, fixes and known issues for the NVIDIA Container Toolkit. +## NVIDIA Container Toolkit 1.20.0 + +This release of the NVIDIA Container Toolkit `v1.20.0` is a feature release. + +### Fixes and Features + +- CDI specifications can now include an application-profile hook that limits EGL and Vulkan visibility to the GPUs assigned to the container. + Graphics applications no longer see unassigned host GPUs through these APIs. +- CUDA compatibility handling now uses `libcuda.so` ELF metadata whenever it is available. + This improves CUDA minor-version compatibility by selecting the container's compatibility libraries only when they are appropriate for the installed driver. +- Driver file discovery now supports libraries spread across multiple directories, as occurs on distributions such as Debian, and matches graphics libraries against the exact installed driver version. + This fix avoids both missing required libraries and injecting libraries from another installed driver version. +- CDI specifications now include the NVIDIA OpenCL ICD file and the legacy `libnvidia-nvvm70.so` library when present. + OpenCL loaders can locate the NVIDIA implementation, and workloads that depend on the legacy NVVM library receive it automatically. +- On WSL2, CDI discovery now includes additional `.so`, `.bin`, and `.dll` files from the NVIDIA driver store instead of relying only on a fixed file list. + This fix enables containers to receive driver components introduced by newer Windows driver releases without waiting for a toolkit-specific allowlist update. +- IMEX channel requests in CDI and JIT-CDI mode are now validated for both the supported ID range and the presence of the corresponding host device. + Invalid requests fail with a clear error. +- JIT-CDI mode now honors the `nvidia-container-runtime.modes.jit-cdi.nvcdi-disable-hooks` configuration option. + You can disable individual CDI hooks for environments where a generated hook is unnecessary or incompatible. +- The NRI plugin can inject ordinary workload CDI devices outside the toolkit namespace. + For management devices, you can authorize additional namespaces with the `--nri-management-cdi-device-namespaces` option or the `NRI_MANAGEMENT_CDI_DEVICE_NAMESPACES` environment variable, enabling centralized management workloads without granting access cluster-wide. +- Updating `config.toml` no longer reverts previously modified options to their defaults. +- CDI generation no longer adds an `update-ldcache` hook when it discovers no driver libraries. + This fix prevents containers from running an unnecessary hook and avoids failures on systems or modes that do not inject libraries. +- The toolkit installer now installs `nvidia-cdi-hook` instead of wrapping it with a shell script. + NRI-based deployments can invoke the hook on hosts that do not provide a shell. +- The `nvidia-cdi-refresh` systemd units are now installed and enabled using distribution-native package mechanisms, including migration of stale RPM enablement links during upgrades. + Automatic CDI specification refresh remains available more reliably after package installation and upgrades. + +### Packaging Changes + +- RPMs rebuilt by the toolkit packaging image now use XZ payload compression instead of zstd. + The resulting packages can be installed on Amazon Linux 2. +- Source package builds can use Podman by setting `DOCKER=podman`. + The build handles Podman's local image naming, SELinux volume labeling, and artifact-directory creation automatically. + +#### Enhancements to container-toolkit Container Images + +- The `container-toolkit` image now uses the non-development distroless base and includes a static BusyBox shell. + Init-container wrappers and lifecycle hooks retain the shell commands they need without depending on the development image. + +### Included Packages + +The following packages are included: + +- `nvidia-container-toolkit 1.20.0` +- `nvidia-container-toolkit-base 1.20.0` +- `libnvidia-container-tools 1.20.0` +- `libnvidia-container1 1.20.0` + +The following `container-toolkit` containers are included: + +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0` +- `nvcr.io/nvidia/k8s/container-toolkit:v1.20.0-packaging` + ## NVIDIA Container Toolkit 1.19.1 This release of the NVIDIA Container Toolkit `v1.19.1` is a bugfix release. diff --git a/container-toolkit/versions1.json b/container-toolkit/versions1.json index 24b46d827..b1f6d6c16 100644 --- a/container-toolkit/versions1.json +++ b/container-toolkit/versions1.json @@ -1,6 +1,10 @@ [ { "preferred": "true", + "url": "../1.20/", + "version": "1.20" + }, + { "url": "../1.19.1/", "version": "1.19.1" }, @@ -23,49 +27,5 @@ { "url": "../1.17.8/", "version": "1.17.8" - }, - { - "url": "../1.17.7/", - "version": "1.17.7" - }, - { - "url": "../1.17.6/", - "version": "1.17.6" - }, - { - "url": "../1.17.5/", - "version": "1.17.5" - }, - { - "url": "../1.17.4/", - "version": "1.17.4" - }, - { - "url": "../1.17.3/", - "version": "1.17.3" - }, - { - "url": "../1.17.2/", - "version": "1.17.2" - }, - { - "url": "../1.17.1/", - "version": "1.17.1" - }, - { - "url": "../1.17.0/", - "version": "1.17.0" - }, - { - "url": "../1.16.2/", - "version": "1.16.2" - }, - { - "url": "../1.16.1/", - "version": "1.16.1" - }, - { - "url": "../1.16.0/", - "version": "1.16.0" } ] diff --git a/repo.toml b/repo.toml index 374e1d164..458924fd9 100644 --- a/repo.toml +++ b/repo.toml @@ -110,8 +110,8 @@ project_build_order = [ docs_root = "${root}/container-toolkit" project = "container-toolkit" name = "NVIDIA Container Toolkit" -version = "1.19.1" -source_substitutions = {version = "1.19.1"} +version = "1.20" +source_substitutions = {version = "1.20"} copyright_start = 2020 redirects = [ { path="concepts.html", target="index.html" },