test(deepseek4): CPU-parity test for HC pre-mix kernels#510
Merged
Conversation
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 1 file (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
deepseek4_cuda_hc_pre_mix is called on the split mix plus CPU-finish decode path, so the 'currently uncalled anywhere in server/' note was wrong. Describe it as a separate mix-only entry instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a CPU-parity unit test for the DeepSeek-V4 Hierarchical-Controller pre-mix kernels (
server/src/deepseek4/deepseek4_hc_cuda.cu), registered as thedeepseek4_hc_cudactest. HIP-only, no runtime code touched.Why
The HC pre step is hand-written CUDA/HIP with block reductions whose width follows the wavefront, and it had no correctness backstop — the sibling
test_rms_norm_hipcovers only RMSNorm. This guards the HC path (mix matvec + NHC=4 Sinkhorn + gated combine) against reduction and wave-width regressions on AMD the same way the RMSNorm test does: it compares the GPU kernels against a CPU reference on the device's actual wavefront width, so wave32 cards (gfx1100 / gfx1151) and any future wave64 card each validate the mode they run.How
server/test/test_deepseek4_hc_cuda.cppdrivesdeepseek4_cuda_hc_prewith the DS4-Flash config (n_hc = 4, somix_dim = 24;n_embd = 512sohc_dim = 2048spans several strided load iterations in both reductions).hc_mix_norm_kernel— per-row RMS-normalized dotfn[row] . state.hc_finish_kernel— sigmoid pre/post gates, per-dst row-softmax followed by thesinkhorn_itersnormalization sweeps, then the gated combineworking[d] = sum_h pre[h] * state[h][d].fnis round-tripped through__halfon the host so both sides consume identical weights; the check covers all three outputs (working,post,comb).test_rms_norm_hipblock (LANGUAGE HIP, includessrc+hip_compat, linksdflash_common) and prints the devicewarpSizeso the run doubles as a wave-width report.Verification
Compiled the test together with its kernel translation unit directly (
hipcc --offload-arch=gfx1151) and ran on gfx1151 (Radeon 8060S, wave32):Tolerance is
1e-4(matchingtest_rms_norm_hip), ~3 orders of magnitude above the observed margin, leaving headroom for wave64 reduction ordering. Thedeepseek4_hc_cudaCMake target is wired verbatim from the mergedtest_rms_norm_hiptarget; a full CMake build of the test was not run in this pass.