feat: add Kimi-K3 model definition and quantization example - #2994
feat: add Kimi-K3 model definition and quantization example#2994kylesayrs wants to merge 7 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
|
The quality checks have failed. Please run |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Kimi-K3 model, adding its configuration, XTML encoding helpers, multimodal and vision processors, model architectures, and a tokenizer, along with a quantization example. The review feedback identifies a critical bug in modeling_kimi_k3_linear.py where the optimized inference path is hardcoded to False. Additionally, it highlights several improvement opportunities, such as replacing unsafe assert statements used for runtime input, file, and dependency validation with standard if checks that raise appropriate exceptions, and defensively using .get() and type checks to prevent potential KeyError or TypeError crashes.
brian-dellabetta
left a comment
brian-dellabetta
left a comment
There was a problem hiding this comment.
overall lgtm, couple questions. when you have empty checkboxes in your PR summary, does that mean PR is still WIP or that you've added each of those?
There was a problem hiding this comment.
do you have a record at all of what you had to change to get this to work? In case we need to modify later?
There was a problem hiding this comment.
Added a diff record!
There was a problem hiding this comment.
sorry, where is the diff? is it a commit?
There was a problem hiding this comment.
There is no diff for this file, but other files have these comments
https://github.com/vllm-project/llm-compressor/pull/2994/changes#diff-31d1b29fad82afccfeab91bce7bbcb974598b53cff9089a22ee1d0dbcac01220R23
There was a problem hiding this comment.
Hmm, it's not loading correctly for me. can you point to the file name / line number? This is also for personal edification, but a README outlining changes from what was copy-pasted from the model checkpoint itself would be handy
|
The quality checks have failed. Please run |
f849c0d to
566f0e1
Compare
|
The quality checks have failed. Please run |
|
The quality checks have failed. Please run |
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews
🔴 Require one maintainer reviewWaiting for any of
This rule is failing.All PRs must have at least one approving review from a maintainer before merging.
|
Add vendored Kimi-K3 model implementation (KimiK3ForConditionalGeneration) with configuration, vision processing, tokenization, and encoding support. Add quantization example that loads a quantization config from pretrained, adds ignore patterns for residual projections and routed experts, and passes the config as a loading argument. Also adds _apply_attn_res to default tracing ignore list for Kimi-K3 compatibility with the sequential pipeline. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
…files Each kimi_k3 file now has a comment at the top documenting implementation differences from the original files at moonshotai/Kimi-K3 on Hugging Face. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…, remove formatting-only comments - Rename modeling_kimi_k3_linear.py back to modeling_kimi_linear.py to match the upstream HuggingFace filename - Remove the re-export shim that was in modeling_kimi_linear.py - Remove "Differences: formatting only" comments from files with no substantive changes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Kimi-K3 key models documentation page with NVFP4 example, update nav and index. Exclude kimi_k3 modeling files from ruff since they are vendored upstream files. Fix import ordering in kimi_k3_example.py. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
47b0f68 to
15e99ed
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 participants
Currently dequantizes the entire model up-front, but we can avoid upfront quantization using layerwise decompression