Multi-versioning fails for CUDA toolkit

[gkouretas-maven]

For the autogenerated NVCC toolkit BUILD stubs, there is inconsistency between the declared and actual version of the toolkit components. This is due to the highest version being declared for the toolkit info, while all the other variables are aliases that get selected as a function of version.

I do see that this was done intentionally, as mentioned in this snippet here:
cuda/extensions.bzl:

    for _, toolkit in registrations.items():
        if components_mapping != None:
            # Always use the maximum version so the toolkit includes all components.
            # Components that don't exist in older versions will fall back to dummy.
            toolkit_version = sorted(redist_versions, key = _version_sort_key)[-1]
            cuda_toolkit(name = toolkit.name, components_mapping = components_mapping, version = toolkit_version)
        else:
            cuda_toolkit(**_module_tag_to_dict(toolkit))

However, issues arise when using a 12.X and 13.X version due to false positives being set for certain compiler flags. In particular, I was seeing a failure when using CUDA 12.X with version due to this random seed feature flag in particular (which, as stated in the comment, fails on CUDA <= 12.9):

cuda/private/toolchain_configs/nvcc.bzl:

    # NOTE: this only works on compiler newer than 12.9
    nvcc_fixed_random_seed_feature = feature(
        name = "nvcc_fixed_random_seed",
        enabled = True,
        flag_sets = [
            flag_set(
                actions = [
                    ACTION_NAMES.cuda_compile,
                    ACTION_NAMES.device_link,
                ],
                flag_groups = [flag_group(flags = ["--frandom-seed=%{output_file}"])],
            ),
        ] if nvcc_version_ge(ctx, 12, 9) else [],
    )

A minimal setup to see the issue is to declare a 12.X and 13.X distribution, and attempt to build with the 12.X version:

MODULE.bazel:

bazel_dep(name = "rules_cuda")

git_override(
    module_name = "rules_cuda",
    commit = "ccd93bdfb002172e570993846db16fb6b8670136",  # < This is the commit I've been working off as I said - the issue should still exist on main from what I've seen
    remote = "https://github.com/bazel-contrib/rules_cuda.git",
)

cuda = use_extension("@rules_cuda//cuda:extensions.bzl", "toolchain")
cuda.redist_json(
    name = "cuda_12_6_0",
    platforms = [
        "linux-aarch64",
        "linux-sbsa",
        "linux-x86_64",
    ],
    version = "12.6.0",
)
cuda.redist_json(
    name = "cuda_13_0_1",
    platforms = [
        "linux-sbsa",
        "linux-x86_64",
    ],
    version = "13.0.1",
)
cuda.toolkit(
    name = "cuda",
)
use_repo(cuda, "cuda")

BUILD.bazel:

load("@bazel_skylib//rules:write_file.bzl", "write_file")
load("@rules_cuda//cuda:defs.bzl", "cuda_library")

write_file(
    name = "dummy",
    out = "dummy.cu",
    content = ["#include <cuda_runtime.h>"],
)

cuda_library(
    name = "dummy_lib",
    srcs = [":dummy"],
    deps = ["@cuda//:cuda_runtime"],
)

Build command:

bazel build --@rules_cuda//cuda:enable=true --@rules_cuda//cuda:version=12.6.0 //:dummy_lib

Error:

INFO: Analyzed target //:dummy_lib (84 packages loaded, 5855 targets configured).
ERROR: /<>/BUILD.bazel:55:13: Compiling bazel-out/aarch64-fastbuild/bin/dummy.cu failed: (Exit 1): nvcc failed: error executing CudaCompile command (from cuda_library rule target //:dummy_lib) external/rules_cuda++toolchain+cuda_nvcc_linux_sbsa_12_6_0/nvcc/bin/nvcc -x cu -Xcompiler -fPIC -ccbin /usr/bin/clang -I . -I bazel-out/aarch64-fastbuild/bin -I ... (remaining 37 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
nvcc fatal   : Unknown option '--frandom-seed=bazel-out/aarch64-fastbuild/bin/_objs/dummy_lib/dummy.pic.o'
Target //:dummy_lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5.339s, Critical Path: 0.08s
INFO: 2 processes: 3 action cache hit, 2 internal.
ERROR: Build did NOT complete successfully

The natural fix would be to add the version as a select statement as done for the other versions. As constructed, this does technically need to be declared in the toolchain BUILD file itself since the version attribute(s) accepts a string or integer depending on the rule. The attached patch addresses the issue (based off commit ccd93bdfb002172e570993846db16fb6b8670136): 0000_toolchain_build_version_select.patch

Note

This patch was AI generated so there's a bit of slop / unnecessary stuff, but I think it mostly illustrates the high level changes that are required.

[cloudhan]

I don't know what you are talking abount exactly. You should list you error message first so that I can say for sure I am reproducing it. And I am not sure what the run command runs.

[gkouretas-maven]

@cloudhan Here's a comprehensive example to reproduce (I've updated the issue description to include this):

MODULE.bazel:

bazel_dep(name = "rules_cuda")

git_override(
    module_name = "rules_cuda",
    commit = "ccd93bdfb002172e570993846db16fb6b8670136",  # < This is the commit I've been working off as I said - the issue should still exist on main from what I've seen
    remote = "https://github.com/bazel-contrib/rules_cuda.git",
)

cuda = use_extension("@rules_cuda//cuda:extensions.bzl", "toolchain")
cuda.redist_json(
    name = "cuda_12_6_0",
    platforms = [
        "linux-aarch64",
        "linux-sbsa",
        "linux-x86_64",
    ],
    version = "12.6.0",
)
cuda.redist_json(
    name = "cuda_13_0_1",
    platforms = [
        "linux-sbsa",
        "linux-x86_64",
    ],
    version = "13.0.1",
)
cuda.toolkit(
    name = "cuda",
)
use_repo(cuda, "cuda")

BUILD.bazel:

load("@bazel_skylib//rules:write_file.bzl", "write_file")
load("@rules_cuda//cuda:defs.bzl", "cuda_library")

write_file(
    name = "dummy",
    out = "dummy.cu",
    content = ["#include <cuda_runtime.h>"],
)

cuda_library(
    name = "dummy_lib",
    srcs = [":dummy"],
    deps = ["@cuda//:cuda_runtime"],
)

Build command:

bazel build --@rules_cuda//cuda:enable=true --@rules_cuda//cuda:version=12.6.0 //:dummy_lib

Error:

INFO: Analyzed target //:dummy_lib (84 packages loaded, 5855 targets configured).
ERROR: /<>/BUILD.bazel:55:13: Compiling bazel-out/aarch64-fastbuild/bin/dummy.cu failed: (Exit 1): nvcc failed: error executing CudaCompile command (from cuda_library rule target //:dummy_lib) external/rules_cuda++toolchain+cuda_nvcc_linux_sbsa_12_6_0/nvcc/bin/nvcc -x cu -Xcompiler -fPIC -ccbin /usr/bin/clang -I . -I bazel-out/aarch64-fastbuild/bin -I ... (remaining 37 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
nvcc fatal   : Unknown option '--frandom-seed=bazel-out/aarch64-fastbuild/bin/_objs/dummy_lib/dummy.pic.o'
Target //:dummy_lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 5.339s, Critical Path: 0.08s
INFO: 2 processes: 3 action cache hit, 2 internal.
ERROR: Build did NOT complete successfully

[gkouretas-maven]

And with the patch linked in the description applied, the example builds successfully since the NVCC version is correctly assigned.

[cloudhan]

After inspection, your patch is pretty valid. Basically, our alias system only properly handled repo alias, but the toolchain config always live at top level @cuda namespace. Thus the cuda tookit itself is not properly handled. Adding a alisa for version is a correct fix for this problem.

---

Following are MEMO only

Currently:

---
title: Version & Arch Resolution
---
flowchart TD
    cudanvcc["@cuda:nvcc<br>alias(name='nvcc',actual='@cuda_nvcc//:nvcc')"]
    cudanvcc_nvcc["@cuda_nvcc:nvcc<br>arche&version alias repo"]
    x64["@cuda_nvcc//:linux_x86_64_nvcc"]
    arm["@cuda_nvcc//:linux_aarch64_nvcc"]
    x64_win["@rules_cuda//cuda/dummy:dummy<br>as not on windows"]
    x64_12["@cuda_nvcc_linux<br>_x86_64_12_6_0//<br>:nvcc"]
    x64_13["@cuda_nvcc_linux<br>_x86_64_13_0_0//<br>:nvcc"]
    arm_12["@cuda_nvcc_linux<br>_sbsa_12_6_0//<br>:nvcc"]
    arm_13["@cuda_nvcc_linux<br>_sbsa_13_0_0//<br>:nvcc"]

    cudanvcc --> cudanvcc_nvcc
    cudanvcc_nvcc --"platform_is_linux_x86_64"--> x64
    cudanvcc_nvcc --"platform_is_linux_aarch64"--> arm
    cudanvcc_nvcc --"platform_is_windows_x86_64"--> x64_win

    x64 --"version_is_nvcc_12_6_0"--> x64_12
    x64 --"version_is_nvcc_13_0_1"--> x64_13

    arm --"version_is_nvcc_12_6_0"--> arm_12
    arm --"version_is_nvcc_13_0_1"--> arm_13

Loading

I think all @cuda_<component>s and its downstream repos have been correctly handled but the @cuda is not.
The root cause is

rules_cuda/cuda/extensions.bzl

Lines 249 to 252 in ac1f726

	# Always use the maximum version so the toolkit includes all components.
	# Components that don't exist in older versions will fall back to dummy.
	toolkit_version = sorted(redist_versions, key = _version_sort_key)[-1]
	cuda_toolkit(name = toolkit.name, components_mapping = components_mapping, version = toolkit_version)

cannot properly access flags to determine which cuda ctk to setup, so a grand unified latest ctk is picked.

@charleysaintNV Do you think we have other options? Somehow @cuda//toolchain:<target>s need to be aliased.