Add TurboQuant KV cache compression to serving pipeline#33

Open

sunghajung6688 wants to merge 19 commits into

hw-native-sys:mainfrom

sunghajung6688:turboquant

Commits on May 27, 2026

[Feature] Add TurboQuant
huangzhuo
committed

Commits on Jun 30, 2026

[Feature] Add TurboQuant
sunghajung6688
committed

Commits on Jul 7, 2026

Merge origin/main (TurboQuant) into kv_malloc
sunghajung6688
committed
Merge hw-native-sys/main (L3 serving + device sampling) into turboquant
sunghajung6688
committed
Merge hw-native-sys/main: L3 serving + device sampling + TurboQuant
sunghajung6688
committed
kv_cache: remove TQ hardcoded num_pages, use same dynamic sizing as non-TQ
sunghajung6688
committed
fix: resolve duplicate kv_dtype/weight_dtype in npu_generate.py
sunghajung6688
committed
fix: remove hardcoded total_kv_pages=200 from npu_generate.py
sunghajung6688
committed
kv_cache: restore TQ full-page-count override in register_model

sunghajung6688
and
claude
committed
turboquant: fix KVCompressor construction hang (CPU thread contention)

sunghajung6688
and
claude
committed
npu_executor: gate REAL_VOCAB/sampling validation under non-TQ only

sunghajung6688
and
claude
committed
qwen3_l3_dispatch: remove duplicate undecorated TQ host wrappers

sunghajung6688
and
claude
committed
npu_executor: fix TQ prefill/decode dummy_args to match host wrapper sig

sunghajung6688
and
claude
committed
npu_runner: skip device sampling kernels in TQ worker; gate supports_device_*

sunghajung6688
and
claude
committed
npu_runner: route TQ kernel args through quant pool; add rot/codebook statics

sunghajung6688
and
claude
committed
npu_runner: TQ-aware KV cache bytes_per_page (uint8 + fp32 scales)

sunghajung6688
and
claude
committed

Commits on Jul 8, 2026