Skip to content

[Optim] SWA related performance optimization (part 3/3)#163

Open
Enigmatisms wants to merge 1 commit into
PaddlePaddle:mainfrom
Enigmatisms:swa_optim_p2
Open

[Optim] SWA related performance optimization (part 3/3)#163
Enigmatisms wants to merge 1 commit into
PaddlePaddle:mainfrom
Enigmatisms:swa_optim_p2

Conversation

@Enigmatisms

Copy link
Copy Markdown

本 PR 主要实现了 analytical 的 local 以及 causal 操作:

  • FM4 不支持 local,在 bwd 中,我们希望通过 local 的 window_size 信息快速地跳过一些无效块。目前主要支持的 window_size_left,right(non-causal window, 如 GSWA 不支持)
  • FM4 在 CP 下不支持 causal,但 contiguous 切分模式的 causality 不会受切分影响,只要传入一个 Q 的 rank offset 即可。

这是三个拆分优化的第三个:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant