fix(vllm-cpp): build every CUDA architecture the platform can host - #11512
Merged
Conversation
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64. vllm.cpp's own release archive builds ten architectures, so LocalAI shipped one or two of them. The failure mode is the problem. An unlisted card is not slower, it dies at the first request with "no kernel image is available for execution on the device", long after `backends install` reported success. That covers A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor, and it is how a Jetson Thor node was found serving nothing at all. amd64 now builds 80;86;89;90a;100a;103a;120a;121a and arm64 builds 87;90a;100a;110;121a, split by where the silicon exists: Jetson is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of GH200/GB200. Triton-AOT stays ON for both, which the old comment said was impossible. It is not, at the version we pin: only maintainer REGEN needs a single arch, while the BUILDER path embeds every vendored cubin tree and selects by exact SM, so 87/103a/110/120a take the portable CUDA kernels and can never load a neighbouring cubin. Upstream ships its ten-SM archive that way. The CUDA 13 guard now covers both branches rather than amd64 alone. arm64 needs compute_121a just as much, and CI already builds it with 13. Cost is smaller than the arch count suggests, because gencode is per-source: fp4-mma still resolves to 120a;121a, and the CUTLASS scaled-mm kernels to one arch each, so the added architectures do not multiply the expensive translation units. Verified: flag generation checked for both branches, CUDA 12 still refused, CPU build untouched; both arch lists expanded through vllm.cpp's own vt_cuda_gencode_options and per-feature arch gating, and all six vendored Triton trees confirmed intact, at the exact pinned commit. A real compile is CI-only: there is no CUDA toolchain on the dev box. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
vllm-cppCUDA images are built for Blackwell only:120a;121aon amd64 and121aalone on arm64. vllm.cpp's own release archive builds ten architectures, so LocalAI has been shipping one or two of them.The failure mode is what makes this worth fixing. An unlisted card is not slower, it dies at the first request with
no kernel image is available for execution on the device, long afterlocal-ai backends install vllm-cppreported success. This was found on a Jetson Thor node that had the backend installed and could not serve anything.Change
120a;121a80;86;89;90a;100a;103a;120a;121a121a87;90a;100a;110;121aSplit by where the silicon exists: Jetson (
87Orin,110Thor) is arm64-only, desktop120ais amd64-only, and90a/100aare on both because of the SBSA parts.Gained: A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin, Jetson Thor.
The CUDA 13 guard now covers both branches instead of amd64 alone. arm64 needs
compute_121ajust as much, and the matrix already builds it with 13.Triton-AOT stays ON, contrary to the old comment
The removed comment claimed a fat build could not keep Triton-AOT because the vendored cubin trees are per-arch. That is no longer true at the commit we pin. In
cmake/TritonAOT.cmake, theFATAL_ERRORabout multi-arch is reached only from the maintainer REGEN flow; the builder path goes through_triton_aot_arch_names, which:The same file notes "the shipped ten-SM release archive is exactly that". So the added architectures get correct kernels either way, and the ones without a vendored tree fall back rather than misloading.
Cost
Smaller than the architecture count suggests, because gencode is per-source (
CUDA_ARCHITECTURES OFFplusvt_cuda_set_source_gencode). Resolved through vllm.cpp's own helpers with the new lists:fp4-mma->120a;121a(amd64),121a(arm64)scaledmm-c3x-sm90->90a,scaledmm-c3x-sm100->100afa2->80;86;89;120a;121a(amd64),87;121a(arm64)The expensive NVFP4 and CUTLASS translation units are therefore not multiplied by the added architectures.
Verification
CMAKE_ARGSgeneration checked for both branches; CUDA 12 still refused; the CPU build is untouched.vt_cuda_gencode_optionsand per-feature arch gating at the exact pinned commit (9fd9e8f3), whoseTritonAOT.cmake,TritonAOTMultiArch.cmakeandCudaSourceGencodeTest.cmakeare byte-identical to upstream.sm_80/86/89/90a/100a/121a) verified present and intact viavt_triton_aot_tree_defect.A real compile is CI-only: there is no CUDA toolchain on the dev box, so build wall-time and image size are unmeasured here. If a job runs long, the arch lists are the single knob to trim.
Follow-up to #11511, which documents the current Blackwell-only limitation; that page should be updated once this lands.
🤖 Generated with Claude Code