fix: clear prompt for recurrent / hybrid models when only a partial prefix matches

[avion23]

Summary

• Cache recurrent and hybrid model detection separately in Llama.
• Clear llama.cpp memory when resetting recurrent or hybrid models.
• Force full prompt re-processing for recurrent or hybrid models when prompt history is edited instead of attempting partial prefix reuse.
• Add repeated-prompt, recurrent, and hybrid regression coverage for prompt-cache behavior.

Tests

• python -m compileall llama_cpp/llama.py tests/test_llama.py
• python -m pytest tests/test_llama.py::test_real_llama_repeated_prompt_cache tests/test_llama.py::test_recurrent_model_prompt_cache_reset tests/test_llama.py::test_hybrid_model_prompt_cache_reset -q

[avion23]

Tested on macos using CMAKE_ARGS="-DGGML_METAL=on" pip3.14 install --force-reinstall --no-cache-dir "llama-cpp-python @ git+https://github.com/avion23/llama-cpp-python.git@update-llama-cpp-2026-01" --break-system-packages

[dhdaines]

This will need at least one more (very important) change as the layout of mtmd_context_params has changed. It should be updated in mtmd_cpp.py to this:

class mtmd_context_params(Structure):
    _fields_ = [
        ("use_gpu", c_bool),
        ("print_timings", c_bool),
        ("n_threads", c_int),
        ("image_marker", c_char_p),
        ("media_marker", c_char_p),
        ("flash_attn_type", c_int),
        ("warmup", c_bool),
        ("image_min_tokens", c_int),
        ("image_max_tokens", c_int),
    ]

[dhdaines]

More changes needed as the layout of llama_context_params has also changed... a new field flash_attn_type has been added after attention_type.

[dhdaines]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

[avion23]

@dhdaines thanks for the review, I need some time to incorporate your comments, setting to draft in the meantime

[avion23]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

I think I have fixed this, could you check?

[dhdaines]

Also the flash_attn parameter no longer exists, has been replaced by flash_attn_type... the default is now to determine automatically whether to use it (as some models require it). This is unfortunately a breaking change, not sure if you want to preserve the flash_attn parameter in the higher-level Python API.

I think I have fixed this, could you check?

Yes, this looks to me like a good way to handle it! We can see what the maintainer @abetlen thinks though...

[avion23]

My intention was to sweep in like a hero and save the day. Didn't work as planned :/

I've rewritten the PR, much less whitespace noise, and cleaner. All review comments are incorporated.

[oss-roettger]

Thank you so much avion23 for your efforts to update the python bindings to a recent llama-cpp version!

I'm trying to use them in a Jupyter notebook (in Docker) on a Nvidia 5090 GPU. Although the latest locally build llama-cli version is running in that same environment (see attached llama-cli.txt) and the above considered problems are gone, the freshly build bindings produce a kernel crash when loading models to GPU (after loading weights to GPU, maybe a context issue, see attached build.txt).

I'm pretty sure, it can be my mistake when installing your branch for GPU support:
!CMAKE_ARGS="-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+https://github.com/avion23/llama-cpp-python@update-llama-cpp-2026-01

Any ideas what I did wrong?!

Edit (New Findings): The above GPU build works with n_gpu_layers=0 (CPU only). This narrows down the problem to context handling in the GPU context code path.
Edit2: Very,very strange: after switching back to n_gpu_layers=100 (from n_gpu_layers=0) I was able to load and successfully run the new Nemotron-Nano-3-30B-A3B-Q4_K_M.gguf and Ling-mini-2.0.Q4_K_M.gguf models on GPU (on the same build that crashed the kernel always(!) while loading the models before. Could it be, that there is any context initialization code which is run in CPU mode only but also important for GPU mode?!

[oss-roettger]

@abetlen Thank you for your work! Please keep this repo alive and merge avion23's updates into the main branch.
These are working now on my RTX 5090 Cuda environment. It's not only that recent llama.cpp supports additional models, but it also offers significant performance gains (e.g. GPT OSS 20 NVFP4 +50% more tokens/s compared to the Oktober versions in your repo wheels).

[avion23]

@oss-roettger thank you for all the testing. I found a bug with flash_attn which caused your error. Could you retest?

[oss-roettger]

@avion23 once again respect for your dedication. I have tested the new version after building it with

!CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+https://github.com/avion23/llama-cpp-python@update-llama-cpp-2026-01

Good news first:
The update runs out of the box (with all values for flash_attn in the constructor flash_attn=None/True/False/not present)

But: I think I have discovered an additional issue (with & without your latest update; on GPU & CPU):
https://huggingface.co/ggml-org/Nemotron-Nano-3-30B-A3B-GGUF produces a KV cache issue on the second dialog turn. I guess there is a cache initialization parameter missing in the python bindings, since the llama-cli command of the same llama build (=same libllama.so) works on multi-turn dialogs with the same Nemotron model.
(see Llama_test.txt for minimum code to reproduce the error)

Edit: Log added:
log.txt

[avion23]

Thank you, Could You attach the log as before? Then it'll be quicker for me and you get a nice solution. Let's just pray this gets merged, i don't want to maintain my fork.

…

On Mon 12. Jan 2026 at 18:29, oss-roettger ***@***.***> wrote: *oss-roettger* left a comment (abetlen/llama-cpp-python#2108) <#2108 (comment)> @avion23 <https://github.com/avion23> once again respect for your dedication. I have tested the new version after building it with !CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86";pip install --force-reinstall --upgrade git+ ***@***.*** Good news first: The update runs out of the box (with all values for *flash_attn* in the constructor flash_attn=None/True/False/not present) But: I think I have discovered an *additional issue* (with & without your latest update; on GPU & CPU): https://huggingface.co/ggml-org/Nemotron-Nano-3-30B-A3B-GGUF produces a KV cache issue on the second dialog turn. I guess there is a cache initialization parameter missing in the python bindings, since the llama-cli command of the same llama build (=same libllama.so) works on multi-turn dialogs with the same Nemotron model. (see Llama_test.txt <https://github.com/user-attachments/files/24562603/Llama_test.txt> for minimum code to reproduce the error) — Reply to this email directly, view it on GitHub <#2108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTGWLHPMSGS7UVCWIRBTY34GOAP5AVCNFSM6AAAAACQO5EZGKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTOMZYGEYDCNRRGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[avion23]

I've implemented a fix for recurrent/hybrid models (Nemotron-A3B, Mamba, RWKV, Jamba)
that prevents "sequence positions not consecutive" crashes during multi-turn conversations.
The fix preserves full speed for normal append-only chat and only triggers reset when
history is edited. Verified compatible with #2109.

It's a bit of scope creep though. Diff is becoming huge even though I only adapted some c bindings.

[oss-roettger]

👍 @avion23: Thank you very much on behalf of the community!

Tests successfully passed with Nemotron-3-Nano-30B-A3B on RTX5090 (CUDA).

✅ https://huggingface.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF/blob/main/Nemotron-3-Nano-30B-A3B-Q4_K_M.gguf
✅ https://huggingface.co/ggml-org/Nemotron-Nano-3-30B-A3B-GGUF/blob/main/Nemotron-Nano-3-30B-A3B-Q4_K_M.gguf

Also successfully retested on RTX5090 (CUDA):

✅ https://huggingface.co/bartowski/openai_gpt-oss-20b-GGUF?show_file_info=openai_gpt-oss-20b-Q4_K_M.gguf
✅ https://huggingface.co/Face314/Qwen3-30B-A3B-Instruct-2507-MXFP4_MOE/blob/main/Qwen3-30B-A3B-Instruct-2507-MXFP4_MOE.gguf
✅ https://huggingface.co/bartowski/google_gemma-3-27b-it-GGUF?show_file_info=google_gemma-3-27b-it-Q4_K_M.gguf
✅ https://huggingface.co/mradermacher/Ling-mini-2.0-GGUF?show_file_info=Ling-mini-2.0.Q4_K_M.gguf

🙋 @abetlen: Please appreciate the extensive work of @avion23 and merge it into the main branch. Thank you!

[avion23]

Thank you for retesting this. I am using it daily on Apple M4 Max and it's working good enough.

@abetlen Is there something I can improve so you can merge this with a good conscience?

[antheas]

I vouch for this, thanks @avion23. Saved my bacon on dottxt-ai/outlines#1812. I will test a few more models and if anything pops up will report back.

[bartwesthoff-fyrm]

@avion23 Do you have a notebook for testing? Can't seem to run nemotron yet but it most probably is a mistake on my end.

[avion23]

@bartwesthoff-fyrm The PR is stable, but Nemotron is a tricky model (Hybrid architecture) that requires specific initialization parameters to run correctly. n_batch=512, n_ubatch=512, flash_attn=True. Vibe coded snippet attached, test_pr_2108.py

This pr might never be merged, project seems abandonware. Have a look at https://github.com/TheBigEye/guanaco-py

[bartwesthoff-fyrm]

@avion23 worked perfectly with the new repository. Thank you for your helpfull responses in this PR.

[mxbi]

For anyone else in this boat, I can confirm https://github.com/TheBigEye/guanaco-py v0.5.0 works great. Thanks @TheBigEye!

[avion23]

Upstream was updated and the submodule is updated. Most changes are upstream. This MR is not strictly necessary.

I rebased and only kept the functionality which is not in upstream yet.

[abetlen]

@avion23 thank you for the contribution, i extracted just the hybrid / recurrent prompt re-use fix and added some tests for those types of models.