GGUF completion returns garbage logits under default device config on hosts without a usable GPU (no error raised)

[ajanaku1]

Summary

On a machine with no usable GPU offload (Intel MacBook, macOS 14, @qvac/sdk@0.13.5), loadModel + completion with the default modelConfig loads the model and streams tokens, but the tokens are non-language garbage, e.g. " red@@@@@@@@@@ @@@@ …". Nothing throws and the run finishes with populated stats. Setting modelConfig.device = "cpu" (with gpu_layers: 0) fixes it on the same machine.

The silent part is what bit us. The caller gets no signal that the output is wrong. We only noticed because we bill per request, so garbage that looks like a successful run is worse for us than an outright error would have been.

Environment

• @qvac/sdk 0.13.5
• Node.js 22+ (also seen on 25)
• macOS 14 (Darwin 24.6.0), Intel CPU, no discrete/usable GPU
• Model: LLAMA_3_2_1B_INST_Q4_0 (registry)

Steps to reproduce

Save this as repro-gpu-default-garbage.mjs:

import { completion, loadModel, close, LLAMA_3_2_1B_INST_Q4_0 } from "@qvac/sdk";

const FORCE_CPU = process.env.FORCE_CPU === "1";

const modelId = await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  // Default (no device set) reproduces the bug on non-GPU hosts.
  // FIX: modelConfig: { device: "cpu", gpu_layers: 0 }
  modelConfig: FORCE_CPU ? { ctx_size: 2048, device: "cpu", gpu_layers: 0 } : { ctx_size: 2048 },
  onProgress: (p) => process.stdout.write(`\rloading ${p.percentage.toFixed(0)}%`),
});
console.log(`\nloaded ${modelId} (FORCE_CPU=${FORCE_CPU})`);

const run = completion({
  modelId,
  history: [{ role: "user", content: "In one sentence, what is peer-to-peer compute?" }],
  stream: true,
});

let out = "";
for await (const tok of run.tokenStream) { out += tok; process.stdout.write(tok); }
const stats = (await run.stats) ?? {};

const letters = (out.match(/[a-zA-Z ]/g) ?? []).length;
const ratio = out.length ? letters / out.length : 0;
console.log(`\n\nbackendDevice: ${stats.backendDevice} · letter-ratio: ${ratio.toFixed(2)}`);
console.log(ratio < 0.6 ? "→ GARBAGE OUTPUT (bug reproduced)" : "→ coherent output");

void close();

Then:

# garbage (default device config)
node repro-gpu-default-garbage.mjs

# coherent (forced CPU)
FORCE_CPU=1 node repro-gpu-default-garbage.mjs

The script prints a crude letter-ratio and flags garbage output automatically.

Expected

Either a coherent completion, or, if the selected GPU backend can't actually execute the model, a raised error, so the caller knows the run failed rather than silently receiving corrupt tokens.

Actual

loadModel picks the GGUF default (device: "gpu", gpu_layers: 99) and the completion streams garbage logits. No error. stats is populated as if the run succeeded.

Workaround

await loadModel({
  modelSrc: LLAMA_3_2_1B_INST_Q4_0,
  modelConfig: { device: "cpu", gpu_layers: 0 },
});

Suggested fixes

Any one of these would have saved us the debugging time:

1. If the GPU backend can't actually run the model, fall back to CPU with a warning instead of emitting corrupt output.
2. Run a backend sanity check at load and throw if it fails, so the caller gets an error instead of garbage tokens.
3. If neither is practical, document in loadModel that non-GPU hosts should pass device: "cpu" and that the GPU default can silently corrupt output.

A smaller, separate finding

While extending delegation to transcribe, we hit a second issue worth mentioning. A malformed Whisper audio buffer (byte length not a multiple of 4 for audio_format: "f32le") doesn't just fail that one request. It makes the model's processing queue reject every later request on the same instance with "Failed to append data to processing queue, error: f32le buffer length must be a multiple of 4", until the provider restarts. A per-request validation error that left the queue intact would be easier to recover from. Filed separately as #3221.

[victorchimakanu]

The silent part is the defect that matters here. A completion that streams corrupt tokens and then returns populated stats gives the caller no signal to branch on, and your billing case shows why a hard error would have been cheaper than a run that looks successful.

Two details will help the team pin down the backend selection. First, the backendDevice value your script prints from stats on a garbage run. Second, whether it still reproduces on @qvac/sdk 0.14.1, since 0.13.5 predates several backend changes.

Forcing device: "cpu" with gpu_layers: 0 is the right call today on a host without a usable GPU. Whether loadModel should sanity check the selected backend, fall back to CPU with a warning, or document the constraint is a product decision, so I am flagging the report with your three suggested fixes to the team. It also connects to #1823, which tracks the related gap that GPU dependencies are bundled even for CPU-only use.

[ajanaku1]

Retested. Both answers you asked for:

backendDevice on a garbage run: "gpu". On the same host (Intel MacBook, Iris Plus 645, no discrete GPU), default config, sdk 0.13.5:

stats: {"timeToFirstToken":1728.9,"tokensPerSecond":15.28,"cacheTokens":0,"promptTokens":46,"generatedTokens":80,"backendDevice":"gpu"}
output head: "zero@@@@@@@@@@@@@@@@@@@@..."

One note on capturing that: I had to load with modelConfig: { predict: 80 } to get stats at all. Without a cap, the garbage generation now runs until it fills the context and the run dies with CONTEXT_OVERFLOW: prompt exceeds the model's context window, so run.stats rejects and there is nothing to read. The "looks successful with populated stats" behavior from my original report has effectively degraded into a crash on longer runs.

Still reproduces on 0.14.1. Same repro script, fresh install, same host: garbage token loops (thonthonthon..., HerHerHer...) until the same processPromptImpl: context overflow. Full log: https://github.com/ajanaku1/InferMart/blob/main/docs/upstream/retest-0.14.1/gpu-garbage-0141.log

device: "cpu" with gpu_layers: 0 still fixes it on both versions. Thanks for the #1823 pointer, that connection makes sense.

[victorchimakanu]

backendDevice: "gpu" on an Iris Plus 645 confirms the issue. The backend selection is picking GPU despite the hardware not supporting LLM offload, and the completion silently succeeds with corrupt output instead of falling back to CPU or raising an error. That is the information the team needs to pin the backend detection path. Flagged.