[Blog] Add "Toward a Cleaner Quantization Stack in SGLang"#366
[Blog] Add "Toward a Cleaner Quantization Stack in SGLang"#366ping1jing2 wants to merge 14 commits into
Conversation
Co-authored-by: fzyzcjy <ch271828n@outlook.com>
ping1jing2 add images
|
|
||
| * **Memory capacity** determines how large a model can be, how much KV cache remains available, and how many requests can run concurrently. The impact is especially visible in large checkpoints. For example, the original Qwen3.5-397B-A17B checkpoint occupies approximately 807 GB, while its FP8 version requires about 406 GB. | ||
|
|
||
| * **Memory bandwidth** strongly affects inference latency. Lower-bit weights reduce the amount of data transferred during each operation, allowing optimized kernels to improve time to first token, inter-token latency, and throughput. |
There was a problem hiding this comment.
Lower-bit weights can reduce memory traffic in weight-bandwidth-bound workloads, especially during decode. The end-to-end benefit depends on kernel efficiency, quantization overhead, and the prefill/decode workload mix.
BBuf
left a comment
BBuf
left a comment
There was a problem hiding this comment.
A few spots I think we should tighten before publishing.
| <div align="center"> | ||
| <img src="/images/blog/2026-07-24-cleaner-quantization-stack/02-diff-diagram.png" alt="Before and after quantization architecture" /> | ||
| <br> | ||
| <em>Figure 1: The refactor separates format-specific weight handling from platform-specific processing and kernel execution.</em> |
There was a problem hiding this comment.
Can we call this the target architecture instead of the “after” state? The linked roadmap still tracks the Scheme/Kernel split and platform-based selection as ongoing work.
|
|
||
| * **Memory capacity** determines how large a model can be, how much KV cache remains available, and how many requests can run concurrently. The impact is especially visible in large checkpoints. For example, the original Qwen3.5-397B-A17B checkpoint occupies approximately 807 GB, while its FP8 version requires about 406 GB. | ||
|
|
||
| * **Memory bandwidth** strongly affects inference latency. Lower-bit weights reduce the amount of data transferred during each operation, allowing optimized kernels to improve time to first token, inter-token latency, and throughput. |
There was a problem hiding this comment.
The table below actually shows TTFT regressing by ~22% for W8A8 and ~10% for W4A4/W8A8. I’d qualify this: quantization helps bandwidth-bound decode here, but not necessarily prefill or TTFT.
| <div align="center"> | ||
| <img src="/images/blog/2026-07-24-cleaner-quantization-stack/03-table.jpg" alt="Quantization performance and accuracy comparison" /> | ||
| <br> | ||
| <em>Figure 2: Performance and accuracy comparison for Qwen3-30B-A3B with tensor parallelism 4 on Ascend A3, using 64 prompts, concurrency 64, and 2048-token input and output lengths.</em> |
There was a problem hiding this comment.
Please add the accuracy dataset/metric, exact checkpoint, SGLang commit, launch command, software stack, and number of runs. Otherwise this table is hard to interpret or reproduce.
| <div align="center"> | ||
| <img src="/images/blog/2026-07-24-cleaner-quantization-stack/05-kernel_reusing_Diagram.png" alt="Multiple quantization schemes reusing shared hardware kernels" /> | ||
| <br> | ||
| <em>Figure 4: Multiple checkpoint formats can share the same backend kernel.</em> |
There was a problem hiding this comment.
This reads as if AWQ, GPTQ, and Compressed-Tensors already share one WnA16 kernel. If this is the target state, please label it that way and mention that reuse depends on compatible layouts, scales, zero points, and group sizes.
|
|
||
| We are grateful to the broader SGLang community, including quantization codeowners Cheng Wan (@ch-wan), Xiaoyu Zhang (@BBuf), Zhiyu Cheng (@Edwardf0t1), Fan Yin (@FlamingoPg), and Peng Zhang (@AniZpZ), as well as all contributors who have implemented formats, developed kernels, reviewed architectural changes, and validated quantization paths across CUDA, ROCm, Ascend NPU, XPU, CPU, and other backends. | ||
|
|
||
| Special thanks to the Intel Neural Compressor and AutoRound contributors for their collaboration on AutoRound integration, and to the NVIDIA, AMD, ModelOpt, Compressed-Tensors, Quark, GGUF, and hardware-backend communities whose work continues to expand the range of efficient inference formats available to SGLang. |
There was a problem hiding this comment.
Compressed-Tensors in SGLang is explicitly adapted from vLLM. We should acknowledge the vLLM contributors here as well.
There was a problem hiding this comment.
The cover skips the Method and Kernel layers and uses “Backend Dispatch” instead, so it does not quite match the four-layer architecture in the post. Can we align the labels?
There was a problem hiding this comment.
I’d change “After” to “Target Architecture” since this migration is still in progress.
There was a problem hiding this comment.
NPUMXFP8LinearMethod is shown as a Kernel, while the layer above is also called Method. Could we clarify that this box is a kernel runner/backend adapter?
There was a problem hiding this comment.
Could this be a Markdown/HTML table instead of a JPEG? It would read better on mobile and be easier to update.
There was a problem hiding this comment.
Maybe mark the shared WnA16 kernel as “target” and note that reuse requires compatible layouts and quantization semantics.
3 participants
No description provided.