Managed llama.cpp offloads to integrated GPU instead of discrete NVIDIA card

[sachin-detrax]

Summary

On a Windows PC with both a discrete NVIDIA GPU and an AMD APU integrated GPU, the managed llama.cpp daemon offloads to the integrated GPU instead of the NVIDIA card.

Reported on a machine with an RTX 5070 Ti + AMD Radeon(TM) Graphics iGPU:

> atomic-agent models devices
configured device: auto
effective device:  auto → Vulkan1

ID         | VRAM        | DEVICE
  Vulkan0  | 15995 MiB   | NVIDIA GeForce RTX 5070 Ti
* Vulkan1  | 16180 MiB   | AMD Radeon(TM) Graphics

Investigation turned up three independent defects that stack on this hardware.

1. pickBestDevice only recognises Intel iGPUs

src/local-llm/gpu-devices.ts

const INTEGRATED_DEVICE_RE = /intel|integrated|\buhd\b|\biris\b/i;

AMD APU iGPUs report as AMD Radeon(TM) Graphics, AMD Radeon 780M Graphics, or Radeon(TM) Vega 8 Graphics — none of which match. They are therefore scored as discrete, and the ranking falls through to the totalMemMiB tiebreak. Vulkan reports an iGPU's heap as a slice of system RAM, so 16180 MiB (shared) beats the RTX's 15995 MiB of real VRAM and the iGPU wins.

The same regex has a false positive in the other direction: a discrete Intel Arc is matched by intel and demoted to integrated.

2. The installed Windows backend variant is never re-checked

src/local-llm/backend-installer.ts, src/local-llm/windows-backend-variant.ts

All four Windows release zips contain an identically-named llama-server.exe. isBackendDownloaded() only checks for that filename, and detectWindowsBackendAsset() (the nvidia-smi probe) runs only inside downloadBackend(). backend-version.json records the release tag but not which asset was installed.

Consequence: a machine that installed the Vulkan build once — because nvidia-smi was unavailable at that moment, or the install predates CUDA variant support — keeps the Vulkan build forever. checkForBackendUpdate sees a matching tag and reports no update.

This is what puts an NVIDIA machine on the Vulkan backend, which is in turn the only reason the iGPU is enumerated as a candidate at all — the CUDA backend enumerates NVIDIA devices only, so defect 1 cannot fire there.

3. CUDA build gated on driver >= 13.3, sending 13.0 drivers to a toolkit that predates their GPU

src/local-llm/windows-backend-variant.ts

if (isAtLeast(cuda, 13, 3)) return WINDOWS_BACKEND_ASSETS.cuda133;
if (isAtLeast(cuda, 12, 4)) return WINDOWS_BACKEND_ASSETS.cuda124;

CUDA minor-version compatibility means a binary built against toolkit 13.3 runs on any 13.x driver, so gating on the exact minor is unnecessary. It is also actively harmful: a driver reporting CUDA Version: 13.0 is routed to the 12.4 build, which contains no sm_120 code because the Blackwell architecture postdates that toolkit entirely. RTX 50-series cards then either JIT from PTX or fail to get a usable device.

Notes

• Linux ships no CUDA asset at all (platform-assets.ts always resolves llama-turboquant-linux-x64-vulkan.zip), so Linux + NVIDIA is permanently on Vulkan and permanently exposed to defect 1.
• Workaround available today: atomic-agent models use-device Vulkan0 pins the discrete card and survives a restart.

Expected

auto selects the discrete NVIDIA GPU; an NVIDIA machine ends up on the CUDA backend appropriate to its driver.