feat: hardware-aware automatic KV quantization configuration (#253) - #262
Open
rajveer43 wants to merge 1 commit into
Open
feat: hardware-aware automatic KV quantization configuration (#253)#262rajveer43 wants to merge 1 commit into
rajveer43 wants to merge 1 commit into
Conversation
Add a lightweight, calibration-free config selector that picks a KV-cache quantization method, bit-width, group size, and packing strategy directly from workload (context length, head dimension) and hardware (unified memory, Metal availability) characteristics, so callers don't have to hand-tune every KVCacheConfig field themselves. - veloxquant_mlx/tools/auto_kv_config.py: pure-heuristic selector (WorkloadProfile, HardwareProfile, select_kv_config, to_kv_cache_config). Short contexts get higher precision (kivi, 4-bit), long contexts get aggressive compression (turboquant_rvq, 1-bit), large head dims get a coarser group size, and memory pressure lowers bit-width or falls back to a bounded-memory eviction method (streaming_llm) when no bit-width fits the budget. No MLX import at module scope, mirroring tools/mac_recommender.py. - veloxquant_mlx/cli/autoconfig.py + __main__.py: `veloxquant autoconfig` CLI entry point. - tests/non_metal/test_auto_kv_config.py: pure-Python unit tests (wired into non-metal-unit.yml, no MLX required). - veloxquant_mlx/tests/tools/test_auto_kv_config_integration.py: builds real KVCacheFactory caches from the selector's output and round-trips tensors through update_and_fetch for every method it can choose.
✅ Deploy Preview for veloxquant-mlx ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a lightweight, calibration-free config selector that picks a KV-cache
quantization method, bit-width, group size, and packing strategy directly
from workload (context length, head dimension) and hardware (unified
memory, Metal availability) characteristics, so callers don't have to
hand-tune every KVCacheConfig field themselves.
(WorkloadProfile, HardwareProfile, select_kv_config, to_kv_cache_config).
Short contexts get higher precision (kivi, 4-bit), long contexts get
aggressive compression (turboquant_rvq, 1-bit), large head dims get a
coarser group size, and memory pressure lowers bit-width or falls back
to a bounded-memory eviction method (streaming_llm) when no bit-width
fits the budget. No MLX import at module scope, mirroring
tools/mac_recommender.py.
veloxquant autoconfigCLI entry point.
into non-metal-unit.yml, no MLX required).
real KVCacheFactory caches from the selector's output and round-trips
tensors through update_and_fetch for every method it can choose.