Skip to content

Eval bug: Segfault at slot initialization with CUDA on SM75 (Turing) — ngl > 0 #41

Closed
Closed
Eval bug: Segfault at slot initialization with CUDA on SM75 (Turing) — ngl > 0#41

Description

@Bino5150
Bino5150
opened on May 26, 2026

Name and Version

./llama-server --version
version: 9867 (e727109)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

Problem description & steps to reproduce

CPU: Intel i7
GPU: NVIDIA Quadro T1000 Max-Q (SM75 / Turing 4GB)
CUDA: 12.8

Server segfaults immediately after initializing slots when any CUDA layers are offloaded (-ngl > 0) on SM75 (Turing) hardware. CPU-only (-ngl 0) works correctly. Reproducible on both MTP and non-MTP model.

Segfaults at initializing slots with -ngl > 0
Warning: fused Gated Delta Net (chunked) not supported, set to disabled
CPU-only (-ngl 0) works fine
Reproducible on both MTP and non-MTP model
Occurs with and without --spec-type mtp
Occurs with and without --no-warmup
fused Gated Delta Net disabled warning appears consistently before crash
CPU-only run (-ngl 0) loads and serves correctly
Model: Qwopus3.5-4B-v3-MTP Q5_K_M GGUF

First Bad Commit

No response

Relevant log output

Logs 
Here's the last log lines before the SegFault:
W sched_reserve: layer 0 is assigned to device CPU but the fused Gated Delta Net tensor is assigned to device CUDA0
W sched_reserve: fused Gated Delta Net (chunked) not supported, set to disabled
I srv load_model: initializing slots, n_slots = 4
[segfault]

Activity

  1. Bino5150
    changed the title [-]Misc. bug: Segfault at slot initialization with CUDA on SM75 (Turing) — ngl > 0[/-] [+]Eval bug: Segfault at slot initialization with CUDA on SM75 (Turing) — ngl > 0[/+] on May 26, 2026

  2. Anbeeld commented on May 26, 2026

    @Anbeeld
    Anbeeld
    on May 26, 2026
    Owner

    Could you please try these variants and report which ones crash?

    # 1. CUDA offload, but no turbo KV / forced FA
./llama-server --model model.gguf --port 8081 -ngl 1 -c 8192 --flash-attn off -t 6

# 2. Same, but one slot only
./llama-server --model model.gguf --port 8081 -ngl 1 -c 8192 --flash-attn off --parallel 1 -t 6

# 3. Original-style settings, but one slot
./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --parallel 1 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

# 4. Try full offload instead of mixed CPU/CUDA placement
./llama-server --model model.gguf --port 8081 -ngl 999 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6

    Also attach a backtrace from:

    CUDA_LAUNCH_BLOCKING=1 gdb --args ./llama-server --model model.gguf --port 8081 -ngl 32 -c 8192 --flash-attn on -ctk turbo4 -ctv turbo3 -t 6
run
bt full

  3. Bino5150 commented on May 27, 2026

    @Bino5150
    Bino5150
    on May 27, 2026
    Author

    #1 (-ngl 1, no flash attn, no turbo KV) and #2 (-ngl 1, no flash attn, --parallel 1) loaded. #3 SegFault at warmup. Backtrace: null ptr in ggml_compute_forward_flash_attn_ext called from ggml_graph_compute_thread
    The sched_reserve warnings about Gated Delta Net tensor CPU/CUDA conflict appearing just before the crash. Skipped #4 since #3 failed.

    Backtrace:
    #0 0x0000000000000000 in ?? ()
    #1 ggml_compute_forward_flash_attn_ext ← crash here
    #2 ggml_graph_compute_thread

    Right before the crash:
    W sched_reserve: layer 0 is assigned to device CPU but the fused Gated Delta Net tensor is assigned to device CUDA0
    W sched_reserve: fused Gated Delta Net (chunked) not supported, set to disabled

  4. Anbeeld commented on May 31, 2026

    @Anbeeld
    Anbeeld
    on May 31, 2026
    Owner

    Thanks, the follow-up results narrowed this down.

    -ngl 1 without forced FA / TurboQuant KV loads, while the original-style command with --flash-attn on -ctk turbo4 -ctv turbo3 --parallel 1 crashes in ggml_compute_forward_flash_attn_ext. That points away from slot count and toward mixed CPU/CUDA placement plus TurboQuant FA.

    Root cause: the CPU backend was reporting support for flash_attn_ext by default, even for TurboQuant K types. Those CPU TurboQuant traits do not provide a vec_dot implementation, so when a mixed -ngl graph left a layer on CPU, the scheduler could route a TurboQuant FA op to CPU and crash through a null function pointer.

    Fixed in 9f268fb03: CPU now rejects unsupported FA K/V types in supports_op, and the CPU FA kernel asserts the missing K vec_dot invariant directly. Please try it.

  5. Bino5150 commented on May 31, 2026

    @Bino5150
    Bino5150
    on May 31, 2026
    Author

    Thanks again for the active support. I know you've been slammed lately. I'll give it another test run and let you know how it goes.

  6. Anbeeld commented on May 31, 2026

    @Anbeeld
    Anbeeld
    on May 31, 2026
    Owner

    I want to ensure v0.3.0 will be a somewhat stable release so I welcome all kind of reports.

  7. Bino5150 commented on Jun 2, 2026

    @Bino5150
    Bino5150
    on Jun 2, 2026
    Author

    Ok, I patched the build and got it up and running... kinda... lol. The MTP patch worked great. Then I ran into some other issues (ggml_backend_sched_split_graph).

    Running Qwopus 3.5 v3 Q5km (hybrid linear_attention + full_attention architecture)
    TheTom TurboQuant fork of llama.cpp (baseline) t4K/t3V - 21.7 tok/s
    beellama.cpp commit 9f268fb q8K/t3V - 27.3 tok/sec (you're faster right out of the box)

    Any attempts to run turbo K caused the model to crash out on inference. It would load, but dump as soon as I said hello.
    Any attempt to run turbo K and/or turbo V with partial offload crashed out and wouldn't load at all. I'd get a "turbo KV cache falling back to q8_0 for CPU-bound layers" warning followed by the scheduler crash.

    MTP works now. Using the MTP version of that same model. Ran mtp --spec-draft-n-max 1 However, with the MTP/context overhead and having to run q8KV, I had to run -ngl20 to avoid OOM. This of course killed my tok/s (12.6) but I had an 86% acceptance rate.

    So turbo K is not working at all in any configuration for me, and turbo V only works if all layers are on the gpu.

  8. Bino5150 commented on Jun 4, 2026

    @Bino5150
    Bino5150
    on Jun 4, 2026
    Author

    I emailed you a very detailed report with all of my findings so far. Let me know if there's anything else I can do to help out bro.

  9. Anbeeld commented on Jun 4, 2026

    @Anbeeld
    Anbeeld
    on Jun 4, 2026
    Owner

    Yeah, I saw that, thanks. I'll try to fix the issues you encountered.

  10. Anbeeld
    added a commit that references this issue on Jun 15, 2026

    Fix issue 41 TurboQuant CUDA paths

    0776d1d

  11. Anbeeld commented on Jun 15, 2026

    @Anbeeld
    Anbeeld
    on Jun 15, 2026
    Owner

    I went back through the original repro and the follow-up report. Current status:

    The original SM75/Turing crash at slot initialization was already fixed in 9f268fb03. The fused Gated Delta Net warning in the log was not the root cause; the bad path was CPU fallback for flash_attn_ext with unsupported TurboQuant K/V types, which could crash through a null function pointer. CPU now rejects those unsupported FA types instead of advertising support.

    The later TurboQuant CUDA issues from your follow-up are addressed in 0776d1da9. CUDA FA execution now validates the route plan's effective K/V types instead of aborting on the raw TurboQuant pair before decode/dequant routing can run. I also hardened the TCQ set_rows path: the backtrace allocation is checked before publishing the pointer, diagnostic dump allocations are checked, and Turbo/TCQ set_rows launches go through the checked kernel-launch wrapper. That should turn the large-prefill/TCQ failure class into a clean CUDA failure instead of possible corruption or repeated-token garbage.

    For partial offload: Turbo KV on CPU-bound KV layers is still not something I want to silently fake by falling back to another cache type. The intended current behavior is to fail early with a clear placement/cache-type diagnostic rather than crash in ggml_backend_sched_split_graph.

    The converter side was also cleaned up in 0776d1da9. Text conversion now warns when the config contains a supported multimodal companion and the current run is only converting the text model, with guidance to use --mmproj. MTP is still included by default for the supported Qwen3.5/3.6 text path unless --no-mtp is used; if a specific community merge still drops MTP, I would need the actual tensor-name list because that would likely be a naming/remapping issue rather than the normal Qwen3.5 path.

    Please retest on current v0.3.2 on the T1000. For TurboQuant/TCQ native FA pair testing, build with GGML_CUDA_FA_HALF_QUANTS=ON or GGML_CUDA_FA_ALL_QUANTS=ON.

  12. Bino5150 commented on Jun 19, 2026

    @Bino5150
    Bino5150
    on Jun 19, 2026
    Author

    Retested on v0.3.2 with GGML_CUDA_FA_ALL_QUANTS=ON, on the T1000 (SM75). Results:

    Bug 1 (turbo K crash) — confirmed fixed.
    Ran the exact previously-crashing config: -ctk turbo4 -ctv turbo3, full offload, 16k ctx. Loads clean, serves clean. 27.3 tok/s — actually faster than my TurboQuant baseline at the identical cache config (21.7 tok/s). No crash, no garbage output.

    Bug 2 (TCQ + large system prompt corruption) — confirmed fixed.
    Tested in stages:

    1. Bare curl, turbo3_tcq K+V, no system prompt — clean, as expected (matches original report).
    2. Bare curl, same TCQ config, ~4068 prompt tokens (synthetic filler standing in for a real system prompt) — clean coherent reasoning_content, no question-mark garbage. This is right in the trigger zone from my original report.
    3. Full integration test — pointed my actual app (real persona system prompt, memory, tool schemas, the works) at the beellama server with turbo3_tcq K+V active. Multi-turn conversation, tool calls, browser navigation, screenshots, memory writes, codebase reindex, skill creation — sustained tool use and accumulated context the whole way through. Zero corruption, zero garbage tokens, coherent output start to finish.

    The set_rows hardening looks like it actually fixed the corruption, not just converted it to a clean failure — every test came back coherent, not erroring.

    Bonus finding: VRAM usage on the TCQ config is running ~10% lower than my TurboQuant daily driver at equivalent settings (79% vs ~89% load). Wasn't expecting that, but I'll take it.

    Still open:

    • Bug 3 (MTP stripped on the Jackrong Qwopus3.5 merge) — I'll pull the actual tensor names from the conversion and get those to you separately, since you're right that it's probably merge-specific rather than a general beellama issue.
    • The mismatched K/V type crash I noted (turbo3_tcq K + turbo3 V crashing the scheduler) — didn't see this addressed in your last message. Still happening on v0.3.2, wanted to flag it again in case it's not intended behavior.

    Great work on this round — two confirmed fixes and a free VRAM improvement is a hell of a release. Thanks for digging into it.

  13. Anbeeld commented on Jul 19, 2026

    @Anbeeld
    Anbeeld
    on Jul 19, 2026
    Owner

    Starting with v0.4.0, BeeLlama uses upstream implementation for DFlash, with some adjustments on top of it.
    TurboQuant is not supported anymore as well.
    Closing this issue as it's relevant only for pre-v0.4.0.

  14. Anbeeld
    closed this as completedon Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions