Update docs for LLM Compressor v0.13.0 - #3036
Conversation
Update what's new section Signed-off-by: Dipika Sikka <dipikasikka1@gmail.com>
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the release notes in docs/index.md for LLM Compressor v0.13.0, highlighting new features such as REAP Expert Pruning, Arbitrary Bit-Width Quantization, Observer Fusion and Deletion, Expanded MoE Architecture Support, and Improved XPU Compatibility. Feedback on the documentation includes correcting a grammatical awkwardness in the REAP paper reference and fixing a factual inaccuracy regarding how observer statistics deletion reduces memory usage.
| Review the [LLM Compressor v0.13.0 release notes](https://github.com/vllm-project/llm-compressor/releases/tag/0.13.0) for details about new features. New features to be aware of include: | ||
|
|
||
| - **Transformers v5 Upgrade**: Full integration with Transformers v5, including refactored MoE linearization with `load_context` for efficient loading, updated model structure handling, and improved tied embeddings support. LM eval performance is maintained across the transition. Note that LLM Compressor no longer supports installation with `transformers<5.0.0` | ||
| - **REAP Expert Pruning**: New modifier for structurally pruning Mixture-of-Experts (MoE) models by removing individual experts based on calibration-based saliency scores. Based on the REAP the Experts paper. |
There was a problem hiding this comment.
There is an extra space after the colon, and the phrasing "the REAP the Experts paper" is grammatically awkward. It should be simplified to "the REAP paper" or "the REAP: Router-weighted Expert Activation Pruning paper".
| - **REAP Expert Pruning**: New modifier for structurally pruning Mixture-of-Experts (MoE) models by removing individual experts based on calibration-based saliency scores. Based on the REAP the Experts paper. | |
| - **REAP Expert Pruning**: New modifier for structurally pruning Mixture-of-Experts (MoE) models by removing individual experts based on calibration-based saliency scores. Based on the REAP paper. |
| - **Arbitrary Bit-Width Quantization (Humming)**: Dense packing for non-power-of-2 bit widths (3, 5, 6, 7) with no wasted bits, plus 16 new WxAy presets covering W2–W8 weights with A4, A8, or A16 activations. | ||
|
|
||
| - **Multi-GPU Model-Free PTQ**: `model_free_ptq` jobs can now be distributed across multiple GPUs when available, automatically parallelizing the quantization workflow for significant speedups on large models | ||
| - **Observer Fusion and Deletion**: Refactored observer lifecycle and significantly reduced memory usage for large models due to observer statistics persisting after calibration. |
There was a problem hiding this comment.
The description states that memory usage is reduced "due to observer statistics persisting after calibration." However, persisting statistics would increase memory usage. Based on the implementation of maybe_delete_statistics in fusion.py, the memory reduction is achieved by deleting/cleaning up these statistics after calibration rather than letting them persist.
| - **Observer Fusion and Deletion**: Refactored observer lifecycle and significantly reduced memory usage for large models due to observer statistics persisting after calibration. | |
| - **Observer Fusion and Deletion**: Refactored observer lifecycle and significantly reduced memory usage for large models by deleting observer statistics after calibration. |
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
|
1 participant
Update what's new section