馃殌 The feature, motivation and pitch
EXL3 has strong potential to become the go-to format for small BPW quantization, offering excellent perplexity-to-size performance. The authors note that changes in the format make integration with VLLM more feasible.
See: https://github.com/turboderp-org/exllamav3?tab=readme-ov-file#exl3-quantization
Adding EXL3 support to VLLM would be a major win, as the native EXLlama engine struggles with large-scale serving, where VLLM excels.
Alternatives
Using EXLLama for inference - bad performance, bad concurrency
Using different quantization formats - more vram, or worse model performance
Additional context
No response
Before submitting a new issue...
馃殌 The feature, motivation and pitch
EXL3 has strong potential to become the go-to format for small BPW quantization, offering excellent perplexity-to-size performance. The authors note that changes in the format make integration with VLLM more feasible.
See: https://github.com/turboderp-org/exllamav3?tab=readme-ov-file#exl3-quantization
Adding EXL3 support to VLLM would be a major win, as the native EXLlama engine struggles with large-scale serving, where VLLM excels.
Alternatives
Using EXLLama for inference - bad performance, bad concurrency
Using different quantization formats - more vram, or worse model performance
Additional context
No response
Before submitting a new issue...