Kimi k3 - #481
Closed
baoqian426 wants to merge 53 commits into
Closed
Conversation
…ng on P800 - fused_moe_ct_w4a16: moe_sigmoid_group_topk_norm and moe_pre_sorted only support expert_num <= 512 on P800 (the former returns ret=1 and leaves its outputs untouched, so the uninitialized topk ids made the pre-sort kernels read out of bounds). Add torch fallbacks for >512 experts, verified bit-exact against the kernels at expert_num=512. - compressed_tensors_moe: probe the expert quant scheme with the checkpoint's projection names (Kimi-K3 uses w1/w3/w2) as well, otherwise the scheme lookup misses and the MoE silently falls back to UnquantizedFusedMoEMethod. - kunlun_communicator: stop forwarding use_all2all to DeviceCommunicatorBase, which derives it itself and rejects the kwarg. Co-authored-by: Kiro
…me probing on P800" This reverts commit 00db81a.
…rt more than 512 experts
use xspeedgate_ops.situ_and_mul_kernel
This reverts commit 560809a.
… + remove cocopod)
…nto kimi-k3 # Conflicts: # vllm_kunlun/ops/kda.py
…ty_context" This reverts commit 87eb39a.
Replace the temporary torch implementations of causal_conv1d_fn and causal_conv1d_update with kunlun_ops.causal_conv1d_fwd / causal_conv1d_update. Both kernels write in place, so the prefill path hands them a private contiguous buffer (callers pass strided views into the packed QKV tensor and still expect their input to survive) and the decode path runs on the caller's `out` buffer. K3 stores conv1d weights in fp32 (ColumnParallelLinear(params_dtype= torch.float32)) against fp16 activations. The 20260818 kunlun_ops build accepts that combination and accumulates in fp32, which matches an fp64 reference to 5.99e-04 on prefill and 9.21e-08 on decode -- an order of magnitude tighter than casting the weight down to fp16 first. Padded cuda-graph lanes carry conv_state_indices == -1; pad_slot_id=-1 makes the kernel skip them instead of writing a real conv-state row, which is what the previous clamp-to-slot-0 workaround had to do. _conv1d_query_start_loc_cpu prefers the CPU mirror the GDN metadata already keeps over .tolist(), to avoid a device sync on every layer.
github-actions
Bot
added
needs review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Description
kimi-k3 w4a16/w4a8_int8/pd disaggregate/dp/ep/pp/tp/mooncake
Checklist (Required)
Before submitting this PR, please ensure that all the following items are completed:
pre-commitchecks.git commit -s.PR Type
Please prefix the PR title with one or more of the following labels to help reviewers quickly understand the nature of the change:
[Feature]– New features or enhancements (e.g. Attention, Communicator, Kernel, Worker, etc.)[Bugfix]– Bug fixes[CI/Build]– CI, build system, or infrastructure improvements[Doc]– Documentation updates or fixes[Misc]– Other changes that do not fit the above categories (use sparingly)Detailed Checklist (Click to Expand)
Thank you for contributing to vLLM Kunlun! To help us maintain high code quality and streamline the review process, please ensure your PR meets the following requirements.
1. Code Quality
pre-commit).2. Testing
3. DCO Compliance
This project follows the Developer Certificate of Origin (DCO).
Signed-off-by:line.git commit -sto automatically add the sign-off.4. Review Expectations
During the review process, maintainers may:
We appreciate your patience and collaboration throughout the review process!