Aug 7, 2026 — a new optional native engine, and a KV-cache gotcha every local Gemma agent hits

[nicedreamzapp]

Hey everyone — this repo got way more attention than I ever expected, so before landing a bigger change I wanted to write down what's coming and why, especially for everyone who's forked it.

TL;DR: nothing you use today changes. Every Claude Code launcher, the proxy, the installer — all untouched. What's landing is a second, optional way to run the same local models: a small native engine that starts answering in ~0.36 s where the old path took ~7 s, deep into long sessions.

Why

Claude Code is a great harness, but it was built for cloud models. Its system prompt is tens of thousands of tokens, and parts of the prompt change every turn — so a local model re-prefills a huge prompt on every single message, and the KV cache never gets to help. We kept optimizing around that (Gen 1 → 3 in the README) and eventually asked the obvious question: what if the harness itself were designed for local inference?

The native engine (agent/agent.py) is ~900 lines of Python on mlx-lm. Fixed ~550-token system prompt, the same tools (Bash/Read/Write/Edit/Glob/Grep), and a KV cache trimmed to the shared prefix each turn so only the new tokens are ever prefilled.

The part worth reading even if you never run it

While benchmarking we found that Gemma-family models give their sliding-window layers a RotatingKVCache capped at the window (1024 tokens on Gemma 4). Once your conversation outgrows that, those caches report untrimmable, and mlx-lm prompt-cache reuse silently dies — every turn re-prefills the entire transcript, precisely in the long sessions where caching matters most. If you're building any Gemma-based agent on mlx-lm, you probably have this bug right now.

The fix: plain KVCache on every layer. The sliding-window attention mask still enforces the window — the cache type only decides what's stored — so outputs are byte-identical (verified with a greedy A/B comparison). The trade is that KV memory now grows with the transcript instead of capping at the window; AGENT_ROLLING_KV=1 restores stock behavior.

Numbers (Gemma 4 31B 4-bit, 4.5k-token conversation, Apple Silicon)

	time to first token
stock rotating cache (full re-prefill every turn)	6.5–7.2 s
native engine with the fix	0.36 s

Generation stays at 26–28 tok/s. Reproduce it on your own machine: python3 bench/agent_bench.py.

Credit

This builds directly on the prompt-cache trim fix contributed in #46 — that work is what made the deeper rotating-cache problem visible. Thank you to everyone who's contributed PRs and issues; this repo is much better for it.

Feedback, counter-benchmarks, and PRs very welcome — especially if you can test on models beyond Gemma and Qwen.