feat: enable CUDA backend support on Linux#200
Open
DjagbleyEmmanuel wants to merge 3 commits into
Open
Conversation
- Preserve user-imported ubuntu-cuda-* archives as linux-cuda-* in map_old_backend_to_new, installBackend, and backendTypeEquivalents - Add linux-cuda-12-x64 and linux-cuda-13-x64 to the Linux supported backends matrix - Add generic cuda-12/13/11 family matching in get_backend_category - Update tests to expect CUDA in the Linux backend matrix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables CUDA backend support on Linux by preserving user-imported CUDA builds instead of silently collapsing them to CPU. Previously, any CUDA archive imported via "Install Backend from File" on Linux would be downgraded to the CPU backend. Now CUDA backends remain selectable and runnable when the host driver supports them.
What changed
Rust plugin (tauri-plugin-llamacpp-upstream/src/backend.rs)
map_old_backend_to_newnow passes throughubuntu-cuda-*names aslinux-cuda-*instead of collapsing them to CPU. User-imported CUDA builds (linux-cuda-*,cu12,cu13,cu11) are preserved as-is.determine_supported_backendsaddslinux-cuda-12-x64andlinux-cuda-13-x64to the Linux matrix when the host driver supports CUDA 12 or 13. Previously only CPU and Vulkan were listed for Linux.get_backend_categorygained genericcuda-13/cuda-12/cuda-11family matching so locally imported Linux CUDA backends are correctly categorized for priority ranking instead of falling through to a plain "x64" bucket.TypeScript extension (backend.ts, index.ts)
backendTypeEquivalentsnow generates genericlinux-*toubuntu-*aliases (and vice versa) so custom CUDA archives installed from file are discoverable regardless of prefix.installBackendpreservesubuntu-cuda-*archive names aslinux-cuda-*on Linux instead of collapsing them to CPU.Why
ggml-org does not publish prebuilt CUDA-Linux binaries. The only way to get CUDA on Linux is to compile llama.cpp from source and import the archive manually. Before this change that workflow was broken: the imported backend would silently become a CPU backend. Now it stays as CUDA and shows up in the backend list.
I tested this on a GTX 1050 with driver 535.309.01 (supports CUDA 12). Users with similar hardware can now use their GPU for inference on Linux.
Testing
determine_supported_backendsRust tests passubuntu-cuda-12.0-x64archive name correctly maps tolinux-cuda-12.0-x64