perf(ggml/hip): add batched ROCmFPX MMQ#513
Draft
weicj wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add gfx1151 batch MMQ support for ROCmFP2, ROCmFP3, and ROCmFP4_FAST weights. This keeps DeepSeek V4 Flash batch prefill on the GPU for both dense projections and routed MoE matrices instead of falling through to per-expert dequantization and hipBLAS calls.
Stack:
#502 -> #503 -> this PR.Changes
Validation
6c7ee19).dflash_serverwith ROCm 7.2.4 for gfx1151. The ROCmFP2, ROCmFP3, and ROCmFP4_FAST MMQ instances compile and link successfully.DeepSeek-V4-Flash-ROCMFP2-STRIX.gguf: its 660 ROCmFP4_FAST dense matrices, 86 ROCmFP2 MoE gate/up matrices, and 43 ROCmFP3 MoE down matrices are covered by this dispatch.