[CuTe, Bwd] Return zero gradients for empty Q/K workloads - #2776
Draft
guoriyue wants to merge 2 commits into
Draft
[CuTe, Bwd] Return zero gradients for empty Q/K workloads#2776guoriyue wants to merge 2 commits into
guoriyue wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dQ/dK/dVbefore scratch allocation, preprocessing, JIT selection, or attention-kernel launch;dSinkbehavior.Problem
CuTe forward already defines empty workloads: empty Q returns an empty output, while empty K returns zero output with
-infLSE. Standard backward did not share that contract.On RTX 5090 / SM120 with BF16, unmodified main gives:
Sq=0: backward preprocessing rejects an inferred empty-tensor stride;Sk=0: the main backward path reaches a zero-work CUDA launch and returns runtime error 9;batch=0: the same dense preprocessing/launch failures;The six passing baseline cases are the varlen combinations: their current autograd route already happens to return successfully. They remain regression coverage for the shared mathematical contract; this PR does not claim those six were broken.
Why the early return is correct
When there are no query rows, no loss term contributes to K or V, so
dK=dV=0anddQis empty. When there are no K/V rows, the output is independent of Q, sodQ=0anddK/dVare empty.The existing zero-gradient logic was incorrectly nested under
learnable_sink. Hoisting it to the common Q/K/V dispatch boundary applies the same rule to ordinary attention before any backward implementation detail sees an empty tensor.This is useful for ragged or distributed training where one microbatch, sequence, or rank can legitimately receive zero tokens.
Validation
RTX 5090 / SM120:
dQ/dK/dVbuffers are returned by identity and zeroed;git diff --checkpasses.Scope
zeros_likeor in-place zeroing may still perform the device work required to materialize returned gradients; the claim is specifically that no attention scratch space is allocated and no backward attention/preprocess/postprocess kernel is compiled or launched. Sparse MLA/QV backward uses a separate implementation and remains out of scope.