Repository navigation
Eval bug: Sm tensor doesn't work #60
Description
Name and Version
compiled to use --sm tensor, but with qwen3.6 27b crashes.
Operating systems
Linux
GGML backends
CUDA
Hardware
2xXeon8628 + 4x3090
Models
No response
Problem description & steps to reproduce
compile with cuda_nccl on, and use --sm tensor
First Bad Commit
No response
Relevant log output
Logs
0.30.664.461 I srv llama_server: model loaded 0.30.664.474 I srv llama_server: server is listening on http://0.0.0.0:8010 0.30.664.495 I srv update_slots: all slots are idle 0.39.673.814 I srv params_from_: Chat format: peg-native 0.39.685.962 I slot get_availabl: id 0 | task -1 | selected slot by LRU, t_last = -1 0.40.331.044 I srv recurrent_sh: shrunk recurrent state to 1 cells for prompt cache (before prompt cache save/load, removed 1 backup cells) 0.40.331.049 I srv get_availabl: updating prompt cache 0.40.331.057 I srv load: - looking for better prompt, base f_keep = -1.000, sim = 0.000 0.40.331.063 I srv update: - cache state: 0 prompts, 0.000 MiB (limits: 262144.000 MiB, 256000 tokens, 274877906944 est) 0.40.928.309 I srv recurrent_ex: expanded recurrent state to 2 cells after prompt cache (after prompt cache save/load) 0.40.928.313 I srv get_availabl: prompt cache update took 597.26 ms 0.40.929.138 I reasoning-budget: activated, budget=2147483647 tokens 0.40.929.166 I slot launch_slot_: id 0 | task 0 | processing task, is_child = 0 /app/ggml/src/ggml-backend-meta.cpp:814: GGML_ASSERT(src_ss[i].axis != GGML_BACKEND_SPLIT_AXIS_UNKNOWN) failed libggml-base.so.0(+0x1c9bb)[0x7fdf367789bb] libggml-base.so.0(ggml_print_backtrace+0x21f)[0x7fdf36778e3f] libggml-base.so.0(ggml_abort+0x152)[0x7fdf36779012] libggml-base.so.0(+0x49c49)[0x7fdf367a5c49] libggml-base.so.0(+0x40a99)[0x7fdf3679ca99] libggml-base.so.0(+0x49dbf)[0x7fdf367a5dbf] libggml-base.so.0(ggml_gallocr_alloc_graph+0x474)[0x7fdf367902e4] libggml-base.so.0(ggml_backend_sched_alloc_graph+0x111)[0x7fdf367966e1] libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xd4)[0x7fdf36942344] libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0xb19)[0x7fdf36959389] libllama.so.0(llama_decode+0x10)[0x7fdf3695e1f0] libllama-common.so.0(_ZN33common_speculative_impl_draft_mtp7processERK11llama_batch+0x2dd)[0x7fdf36fce83d] libllama-common.so.0(_Z26common_speculative_processP18common_speculativeRK11llama_batch+0x2d)[0x7fdf36fbd18d] libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x3d98)[0x7fdf37796208] libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x221)[0x7fdf37824641] libllama-server-impl.so(_Z12llama_serveriPPc+0x29c7)[0x7fdf376d52b7] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fdf371bbd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fdf371bbe40] ./llama-server(+0x12b5)[0x5b6bd46a92b5]
Activity
Anbeeld commentedon Jun 7, 2026
Will investigate.
efschu commentedon Jun 7, 2026
It would be awesome to see the following at the same time:
-sm tensor
-cram
-slot save/load
-parallel >1
-mtp dflash
and quantized KV
and sleep mode
Would kickass with every other llm engine.
Anbeeld commentedon Jun 14, 2026
Investigated. This is a crash in llama.cpp's tensor-parallel backend, and it isn't specific to this fork.
With
--sm tensor, every tensor in the compute graph gets a shard layout assigned across your GPUs. When the MTP (nextn) speculative draft runs its decode, one op in that graph ends up with an empty, zero-element tensor as an input, and the layout pass has no rule for sharding an empty input, so it aborts at theGGML_ASSERTinggml/src/ggml-backend-meta.cppfrom your traceback. It only triggers at the intersection of two features, tensor parallelism and the MTP draft. Single-GPU and--sm layernever run that layout pass, which is why the server loads and sits idle fine and only dies on the first draft.Both halves of that intersection are stock llama.cpp. The tensor-parallel meta backend is
ggml-org/llama.cpp#19378, MTP speculative decoding isggml-org/llama.cpp#22673, and the gate that decides whether--sm tensoris allowed for an architecture (llm_arch_supports_sm_tensor) is upstream too and doesn't exclude Qwen3.6. None of this fork's own work (DFlash, TurboQuant, KVarN) is on that path. A stock llama.cpp built with NCCL would hit the same assert running an MTP draft.To keep serving, avoid the intersection: either keep the MTP draft and use
--sm layerinstead of--sm tensor, or keep--sm tensorand disable the MTP draft. Layer split runs the 27B across your 4x3090s without the crash.The real fix has to teach upstream's tensor-parallel backend to accept an empty tensor as an op input, which it already accepts as an op output; that asymmetry is the bug. That change belongs in
ggml-org/llama.cpp, so it's better tracked there than patched around in this fork.efschu commentedon Jun 17, 2026
@Anbeeld
I'm using -sm tensor with mtp on llama.cpp for a while now. So what you say can not be true. The think is, you only can use F16 or Q8_0 kv cache while using sm tensor and mtp. So it's a quant thing, not a sm tensor/mtp thing.
fix: add KVarN ops to tensor-parallel meta backend + empty tensor guards
antdea commentedon Jun 22, 2026
@Anbeeld I can also confirm MTP + tensor split works on llama.cpp but doesn't work on beellama.cpp
When using kvarn6/kvarn5 I get the following error
When running with the exact same script that works when pointed at llama.cpp. Using q8_0
beellama.cpp compiled against branch 0.3.2