Why
TurboQuant (Google Research, ICLR 2026, arxiv:2504.19874) compresses LLM KV cache to 3-bit keys / 2-bit values with 99.5% attention fidelity — 6× KV cache memory reduction vs standard FP16, and up to 8× speedup on H100-class GPUs.
For LLMKube workloads on consumer hardware (2× 5060 Ti, M-series Macs), this is a significant unlock: push usable context from ~32K → ~180K at the same VRAM budget, or fit longer agentic-coding sessions without reaching for data-center GPUs.
What it doesn't do (scope clarification)
TurboQuant compresses the KV cache, not model weights. It's orthogonal to quantizations like FP8 / Q4_K_M / AWQ. Memory-budget impact:
| Knob |
What it reduces |
| Weight quantization (FP8, Q4_K_M, NVFP4) |
Model weight footprint |
| TurboQuant |
KV cache footprint (during generation) |
| `cpu-offload-gb` (see #307) |
Moves weights to CPU RAM |
| `gpu-memory-utilization` (see #307) |
vLLM's claim on VRAM |
The lever this issue unlocks is "longer context windows on the same GPU." It does NOT fit models that don't fit VRAM in the first place — those need weight quantization or offload.
Integration paths
Two backends, two paths:
vLLM path
- `0xSero/turboquant` ships Triton kernels + vLLM integration
- Not yet upstream in vLLM; would require a patched image or a dedicated TurboQuant-enabled sidecar runtime
- Would add a new VLLMConfig field: `kvCacheQuantization: turboquant | fp8_e4m3 | fp8_e5m2 | auto`
llama.cpp path
- Community discussion at ggml-org/llama.cpp#20969
- GGUF quants already published by community (`majentik/Nemotron-3-Super-120B-A12B-TurboQuant-GGUF-Q6_K`, several Gemma 4 variants)
- Waiting on upstream llama.cpp to land runtime support for the TurboQuant KV block format
Proposed CRD changes (once runtime support exists)
```yaml
spec:
runtime:
vllm:
# Extends existing kvCacheDtype enum with turboquant values:
kvCacheDtype: turboquant_k3v2 # new
# Or a separate opt-in field so we don't confuse with FP8 dtypes:
kvCacheQuantization:
algorithm: turboquant
keyBits: 3
valueBits: 2
```
Acceptance (when the time comes)
Not-now
This issue is tracking-only. Neither vLLM upstream nor llama.cpp upstream has landed TurboQuant support yet — the community implementations are forks/discussions. Realistic timeline is 2026 Q3 once either path matures. Filing now so the design notes don't get lost.
Related
Discovered during #306 validation on 2x RTX 5060 Ti Shadowstack cluster.
Why
TurboQuant (Google Research, ICLR 2026, arxiv:2504.19874) compresses LLM KV cache to 3-bit keys / 2-bit values with 99.5% attention fidelity — 6× KV cache memory reduction vs standard FP16, and up to 8× speedup on H100-class GPUs.
For LLMKube workloads on consumer hardware (2× 5060 Ti, M-series Macs), this is a significant unlock: push usable context from ~32K → ~180K at the same VRAM budget, or fit longer agentic-coding sessions without reaching for data-center GPUs.
What it doesn't do (scope clarification)
TurboQuant compresses the KV cache, not model weights. It's orthogonal to quantizations like FP8 / Q4_K_M / AWQ. Memory-budget impact:
The lever this issue unlocks is "longer context windows on the same GPU." It does NOT fit models that don't fit VRAM in the first place — those need weight quantization or offload.
Integration paths
Two backends, two paths:
vLLM path
llama.cpp path
Proposed CRD changes (once runtime support exists)
```yaml
spec:
runtime:
vllm:
# Extends existing kvCacheDtype enum with turboquant values:
kvCacheDtype: turboquant_k3v2 # new
# Or a separate opt-in field so we don't confuse with FP8 dtypes:
kvCacheQuantization:
algorithm: turboquant
keyBits: 3
valueBits: 2
```
Acceptance (when the time comes)
Not-now
This issue is tracking-only. Neither vLLM upstream nor llama.cpp upstream has landed TurboQuant support yet — the community implementations are forks/discussions. Realistic timeline is 2026 Q3 once either path matures. Filing now so the design notes don't get lost.
Related
Discovered during #306 validation on 2x RTX 5060 Ti Shadowstack cluster.