disable moe_calibrate_all_experts for Autoround example. - #2877
disable moe_calibrate_all_experts for Autoround example.#2877changwangss wants to merge 6 commits into
Conversation
… linearize_moe Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR adds the ChangesAutoRound Example MoE Calibration Flag
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsLinked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates several AutoRound quantization examples by explicitly setting the moe_calibrate_all_experts parameter to False. This includes the LLaMA 4 dynamic and static quantization examples, as well as the MXFP8 example where a comment is added explaining that disabling this option prevents out-of-memory (OOM) errors on a single 80GB GPU. There are no review comments, and I have no feedback to provide.
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
Merge Protections🟢 All 2 merge protections satisfied — ready to merge. Show 2 satisfied protections🟢 Require one maintainer reviewAll PRs must have at least one approving review from a maintainer before merging.
🟢 Require two reviewsPRs labelled "two-reviews" must have at least two approving reviews before merging.
|
coderabbitai
Bot
added
autoround
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
examples/autoround/quantization_w8a8_fp8/llama4_dynamic_quant_example.py (1)
51-51: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider adding a rationale comment, matching the mxfp8 example.
The PR objective explains this flag avoids OOM when
llmcompressorhas already linearized MoE experts, while 128 samples/200 iterations still give sufficient expert coverage. The mxfp8 example documents this reasoning inline, but this file doesn't. Adding a short comment here would help readers understand why the flag is set.📝 Suggested comment
+ # Disable calibrating all experts to avoid OOM when llmcompressor's + # linearize_moe has already run before AutoRound; 128 samples / 200 iters + # still provide sufficient expert routing coverage. moe_calibrate_all_experts=False,As per path instructions, "Review for clarity, correctness, and educational value... comments and documentation are helpful for users learning the library."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/autoround/quantization_w8a8_fp8/llama4_dynamic_quant_example.py` at line 51, Add a short rationale comment near the moe_calibrate_all_experts setting in the llama4_dynamic_quant_example example, matching the inline explanation used in the mxfp8 example. Mention that disabling expert-wide calibration avoids OOM when llmcompressor has already linearized MoE experts, while the 128 samples/200 iterations still provide sufficient expert coverage.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@examples/autoround/quantization_w8a8_fp8/llama4_dynamic_quant_example.py`:
- Line 51: Add a short rationale comment near the moe_calibrate_all_experts
setting in the llama4_dynamic_quant_example example, matching the inline
explanation used in the mxfp8 example. Mention that disabling expert-wide
calibration avoids OOM when llmcompressor has already linearized MoE experts,
while the 128 samples/200 iterations still provide sufficient expert coverage.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 228b27af-3a56-4edd-91b1-c0fbc1399834
📒 Files selected for processing (3)
examples/autoround/quantization_w8a8_fp8/llama4_dynamic_quant_example.pyexamples/autoround/quantization_w8a8_fp8/llama4_static_quant_example.pyexamples/autoround/quantization_w8a8_mxfp8/autoround_example.py
Signed-off-by: changwangss <chang1.wang@intel.com>
In my understanding, AutoRound's memory usage is purely a function of (number of gradients which is proportional to number of activated experts), NOT the number of samples The samples are reduced into gradients, so as long as each expert gets at least one sample, then adding more samples via calibrate_all_experts shouldn't increase memory usage, right? As long as each expert gets at least one sample, I don't see a reason why |
|
I think the things I need to understand are
|
|
sorry for the late response. Let me answer the questions.
The main memory usage is not from the final gradient tensors. With moe_calibrate_all_experts=True, every expert processes all tokens, producing much larger QDQ inputs and intermediate activations. With False, each token is processed only by its top-8 routed experts, significantly reducing peak memory.
With 128 calibration samples, a sequence length of 2048, and top-8 routing, each MoE layer receives approximately 2.1 million expert-token assignments. In our DeepSeek-R1 test with the same calibration configuration, every expert was activated. The distribution was not perfectly uniform, as expected for MoE routing, but no expert was extremely underrepresented. Therefore, we expect this configuration to provide sufficient coverage in practice, although the exact distribution remains model- and dataset-dependent.
The current MXFP8 example uses dynamic activation quantization, where activation scales are computed at runtime, See the MXFP8 configuration |
|
@yiliu30 are you satisfied with the changes and explanation for us to merge this in? |
kylesayrs
left a comment
kylesayrs
left a comment
There was a problem hiding this comment.
@changwangss Good to merge from my side, just adding my comments here
- With moe_calibrate_all_experts=True, every expert processes all tokens, producing much larger QDQ inputs and intermediate activations
Note that LLM Compressor linearizes experts. This means that experts are executed sequentially, and therefore processing more experts does not increase peak memory usage, only increases runtime.
Even when sequential_targets="ExpertWithGate" and you care about offload memory, the sequential pipeline's tracing will guarantee that the expert outputs are added to the fixed size summing tensor, not accumulated before adding.
- In our DeepSeek-R1 test with the same calibration configuration, every expert was activated.
That's interesting to hear. We initially added this feature for DSV3 and found that many experts received insufficient samples. This could be reflective of either models having better expert distribution, or autoround being more resilient to low calibration data than GPTQ
- If an expert is not activated at all, AutoRound fills its missing amax using the maximum value from calibrated experts
Sounds good
|
The quality checks have failed. Please run |
4 participants
SUMMARY:
llmcompressornow applieslinearize_moein oneshot, so MoE experts may already be linearized beforeAutoRoundruns.Because of this, we added a skip in
AutoRound(Skip fused_moe replacement when llm-compressor oneshot already linearizes experts intel/auto-round#1966) to avoid redundant fused_moe replacement when linearized experts are detected.For Llama4, the default linearized flow calibrates all experts at once, which can OOM on an 80GB GPU.
Setting
moe_calibrate_all_experts=Falseavoids this, and with 128 samples + 200 iterations we still get sufficient expert coverage during calibration.moe_calibrate_all_experts=FalseforQwen-30B-A3B mxfp8was already introduced in an earlier PR ([Fix AutoRound ignore-layer metadata handling and add Qwen3-30B to mxfp8 example. #2695]. It was likely lost after multiple example updates, so this change restores it.TEST PLAN:
"please outline how the changes were tested"