Repository navigation
Eval bug: HIP + turbo3_tcq = crash #14
Description
Name and Version
latest
Operating systems
Linux
GGML backends
HIP
Hardware
7900XTX GFX1100
Models
Qwen 3.6 27B Q4KM
Problem description & steps to reproduce
Normal llama run with 128k context size, turbo3_tcp modes
First Bad Commit
No response
Relevant log output
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f8f98280e32 in __syscall_cancel_arch () from /lib64/libc.so.6
#0 0x00007f8f98280e32 in __syscall_cancel_arch () from /lib64/libc.so.6
#1 0x00007f8f9827506c in __internal_syscall_cancel () from /lib64/libc.so.6
#2 0x00007f8f982750b4 in __syscall_cancel () from /lib64/libc.so.6
#3 0x00007f8f982e552f in wait4 () from /lib64/libc.so.6
#4 0x00007f8f98f1dd9b in ggml_print_backtrace () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#5 0x00007f8f98f1df0d in ggml_abort () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#6 0x00007f8f9e39b582 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#7 0x00007f8f9e56d127 in void launch_fattn<256, 2, 2>(ggml_backend_cuda_context&, ggml_tensor*, void ()(char const, char const*, char const*, char const*, char const*, int const*, float*, HIP_vector_type<float, 2u>, float, float, float, float, unsigned int, float, int, HIP_vector_type<unsigned int, 3u>, int, int, int, int, int, int, int, int, int, int, int, long, int, int, long, int, int, int, int, int, long), int, unsigned long, int, bool, bool, bool, int) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#8 0x00007f8f9e557b55 in void ggml_cuda_flash_attn_ext_tile_case<256, 256>(ggml_backend_cuda_context&, ggml_tensor) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#9 0x00007f8f9e37b70d in ggml_cuda_flash_attn_ext(ggml_backend_cuda_context&, ggml_tensor*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#10 0x00007f8f9e3a5afe in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#11 0x00007f8f9e3a1fdc in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-hip.so.0
#12 0x00007f8f98f3a83b in ggml_backend_sched_graph_compute_async () from /home/ezzz/Desktop/beellama.cpp/build/bin/libggml-base.so.0
#13 0x00007f8f9ee4cfb0 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#14 0x00007f8f9ee5119b in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#15 0x00007f8f9ee59d9f in llama_context::decode(llama_batch const&) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#16 0x00007f8f9ee5c85e in llama_decode () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama.so.0
#17 0x00007f8f9f5494a2 in common_context_can_seq_rm(llama_context*) () from /home/ezzz/Desktop/beellama.cpp/build/bin/libllama-common.so.0
#18 0x00000000004f861e in server_context_impl::load_model(common_params&) ()
#19 0x000000000043a0a7 in main ()
[Inferior 1 (process 19251) detached]
Activity
Anbeeld commentedon May 15, 2026
I pushed a diagnostic branch for this issue:
hip-turbo3-tcq-crash.This branch does not try to fix the crash yet. It adds logging around the HIP flash-attention path so we can tell whether the failure is coming from turbo K/V dequant, Q rotation, or the final TILE dispatch.
Could you please build this branch and run the same command that currently crashes?
First, run the baseline from the branch:
Then run with diagnostics:
Please paste the last 30–50 lines of output around the failure. The most useful line will look like one of these:
After that, please try forcing VEC:
And forcing TILE:
One more useful comparison is native turbo decode, which skips the fp16 dequant path:
If you can, also test plain fp16 KV cache with forced TILE:
That last run helps separate "TILE is broken for D=256 on HIP in general" from "TILE only fails after turbo3_tcq dequant creates the temporary fp16 K/V layout."
Please include:
If
GGML_FATTN_DEBUG_SYNC=1conflicts with HIP graph capture, rebuild with graphs disabled, for example with-DGGML_HIP_GRAPHS=OFF, then rerun the diagnostic command.Once we have those logs, I should be able to tell whether the real fix belongs in the dequant kernels, the Q rotation step, the TILE selector/config, or the TILE handling of the temporary fp16 K/V strides.
cryptopsy0 commentedon May 19, 2026
I also got this error, and i'm using turbo4 as a workaround because I am not seeing more than 20% increase on dflash (my main problem!)
Anbeeld commentedon May 24, 2026
Just pushed pre-release WIP version of v0.3.0: https://github.com/Anbeeld/beellama.cpp/tree/v0.3.0
The branch contains a full rebase to the latest llama.cpp (with MTP and upstream spec architecture), but also a lot of DFlash fixes (multi-slot, multi-GPU, speculation in tool calls, etc.)
Please build and try that branch, and report back on what's the state of the issue there. All the follow up fixes will be on v0.3.0 too, as it's a bit pointless to do patches within v0.2.0 because of upstream rebasing.
Anbeeld commentedon May 26, 2026
Closing due to no follow up, but also AMD profiling being covered by other issues.