MindSpore Golden Stick is a model compression tool for the MindSpore open source community, supporting quantization of Hugging Face weights on Ascend hardware and deployment on vLLM-MindSpore Plugin or MindSpore Transformers.
Architecture | Workflow | Documentation | Issue Feedback
MindSpore Golden Stick is a model compression tool jointly designed and developed by MindSpore team and Huawei Noah's Ark Lab. We have two main goals: first, to be a model compression tool that provides concise interfaces and rich algorithm libraries to improve the deployment efficiency of MindSpore networks; second, to be an algorithm research platform with flexible configuration interfaces, modular algorithm libraries, and a framework that supports rapid customization, facilitating algorithm researchers to quickly practice innovation. Specifically:
- Multi-level APIs: Provides different level APIs, offering both ease of use and flexibility, lowering the barrier to entry while retaining algorithm customization capabilities;
- Rich and Modular Algorithm Library: Provides rich SoTA compression algorithms and supports flexible modular combinations;
- Highly Extensible Framework Architecture: Layered decoupling that shields the complexity of hardware and frameworks, while supporting integration of custom algorithm components to build customized compression pipelines with flexible APIs.
- [2025/12] v1.4.0 Release: Completed framework pluginization refactoring with MindONE backend support, integrating mainstream quantization algorithms including OSL, SmoothQuant, AWQ, GPTQ, A16W8, A8dynW8, and A8W4, validated on models such as glm4v and qwen3.
- [2025/12] Multimodal Model Quantization: Added support for quantization of multimodal understanding models, successfully validating the OSL-A8W8 quantization scheme for the qwen3-vl network under MindONE framework.
- [2025/09] OutlierSuppressionLite provides higher precision A8W8 quantization capabilities.
- [2025/09] Combined OutlierSuppressionLite and GPTQ algorithms to achieve A8W4 quantization for DeepSeekV3/R1 networks, further lowering the deployment threshold for full-featured DeepSeek. Quantized weights can be found at Modelers.
- [2025/09] Support for Transformers-Like-API and support for saving weights in Hugging Face format, see BaseQuantForCausalLM interface for details.
- [2025/06] Support for SmoothQuant-8bit and GPTQ-4bit quantization of DeepSeekV3/R1 networks.
Please refer to Installation Tutorials.
Take Simulated Quantization (SimQAT) as an example for demonstrating how to use MindSpore Golden Stick.
Overview |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Architecture |
Workflow |
Examples | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Transformers like APIs🔥 | APIs | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Post-Training Quantization |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| RoundToNearest-A16W8 | SmoothQuant-A8W8 | AWQ-A16W4 | GPTQ-A16W4 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| QoQ-A8W4🔥 | FAQuant(demo) | Dynamic Quantization | KVCacheInt8(demo) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| OutlierSuppressionLite🔥 | OutlierSuppressionPlus(demo) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Others |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Auto Quantization Strategy | Fake Quant Evaluation | Ascend Hardware Adapter layer | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
End Of Life |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| QAT-SimQAT | QAT-SLB | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| pruner-SCOP | pruner-uni_pruning(demo) | pruner-LRP(demo) | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Ghost | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
The model compression results from Golden Stick are weights in Hugging Face format. It is recommended to deploy them on vLLM-MindSpore Plugin or MindSpore Transformers. You can also try deploying them on mainstream frameworks such as PyTorch, ONNX Runtime, TensorRT, etc.
🎯Video Conference：https://meeting.tencent.com/dm/U5EJCKl1FP8z
📬SIG：https://www.mindspore.cn/sig/LLM%20Inference%20Serving
📍WeChat Group：https://atomgit.com/mindspore/golden-stick/issues/ID2UGQ
Please read CONTRIBUTING for details on setting up development environments, testing functions, and submitting PR.
We welcome and value any form of contribution and cooperation. Please use Issue to inform us of any bugs you encounter, or to submit your feature requests, improvement suggestions, and technical solutions.