Optimize flashmask sm90#165
Open
baoqiwen wants to merge 11 commits into
Open
Conversation
…reduce_block_count
baoqiwen
commented
Jul 24, 2026
| # the KV block index and forward it so mma skips the per-element | ||
| # flashmask apply on causal/seqlen-only mask blocks. | ||
| mask_raw = curr_mask_block_idx[curr_mask_block_cnt - 1] | ||
| mask_n_block = mask_raw & 0x3FFFFFFF |
Author
There was a problem hiding this comment.
cpp 描述:单开一个 bool 数组存 partial flag,放在 smem,会引发 4-way bank conflict。把 partial flag 编进 n_block 这个 int 里,既省 smem 又消除 bank conflict。
cutedsl 设计:
bit 0–29(30 位): block 下标, 范围 0 ~ 2³⁰-1。
bit 30(1 位):fm_partial 标志(1=部分被 mask, 需逐元素 apply; 0=全可见, skip)。
bit 31(符号位):特意不用,保持为 0,让整个值恒为非负。
cpp 用符号位存 flag,(索引占 31 位);
DSL 用 bit30 (索引占 30 位, 符号位留空换取全程非负、无移位陷阱,更加安全)。
block idx 不可能超过 2**30 的,不然早 oom 了。用 30 bit 也十分安全。
baoqiwen
commented
Jul 24, 2026
| _postprocess_run( | ||
| dq_accum_tensor, dq_tensor, softmax_scale, | ||
| head_dim_rounded, m_block_size, AtomLayoutMdQ, dQ_swapAB, | ||
| head_dim, m_block_size, AtomLayoutMdQ, dQ_swapAB, |
Author
There was a problem hiding this comment.
这地方其实无所谓,反正如果这里是非 pad,进了 bwd post 之后也会统一 pad。
看 overlap pr 合入后,统一一下写法。保持外部全部非 pad,由 bwd post pad。
baoqiwen
force-pushed
the
bqw_perf
branch
2 times, most recently
from
July 24, 2026 19:05
3508785 to
0426f1d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
针对 flashmask v4 cutedsl sm90 做优化
27661bd:Optimize flashmask bwd: O1 m-block segments + full/partial mask split
8c2e847:Optimize flashmask fwd: build block lists in a cute kernel instead of paddle ops
d4fa1ab:Optimize flashmask fwd: drop per-call output zero-fill and skip SM90 reduce_block_count
f09d963:Encode flashmask partial flag into mask index for SM90 fwd
122faf4:Optimize flashmask cutedsl d256/dv256 bwd via atomicAdd dQaccum
631ba7f:Optimize flashmask cutedsl d256/dv256 bwd: slice dQKV MMA + larger KV tile
24e5363:skip causal mask on fully-visible blocks in SM90 fwd
SM90 前向 causal 注意力里,对角线以下那些"全部可见、不需要掩码"的 K/V 块之前还在逐元素跑一遍无用的 causal mask,现在直接跳过(仅在有 FlexAttention mask_mod 时才保留),从而砍掉约一半块的冗余整型比较,把 d128 causal 前向从 15.7ms 降到 14.2ms。(对齐cpp实现)