AllReduce LL128: split into flag-synced + warpsync sub-tiers (gfx1250)#8915
Open
srirajpaul wants to merge 2 commits into
Open
AllReduce LL128: split into flag-synced + warpsync sub-tiers (gfx1250)#8915srirajpaul wants to merge 2 commits into
srirajpaul wants to merge 2 commits into
Conversation
Add a 128B-line LL128 all-reduce variant that carries payload in all 16 words of each line (no in-line flag word) and synchronizes phase 1/2 with a separate all-to-all, per-global-warp arrival barrier staged in a reserved 1 MiB front region of each peer's scratch. Payload efficiency improves from 15/16 to 16/16. The flag region is not double-buffered, so the wait uses a monotonic ">= flag" compare to tolerate a peer racing one call ahead. New codepath-agnostic kernel ddaAllReduceFlatLL128_1 in all_reduce_dda_fabric_ll128_1.h. The fabric launcher selects between it and the base flagged kernel via a useFullLine toggle (default full-line); numLines / slot stride / scratch sizing follow the selection, and the scratch bound now accounts for the reserved 1 MiB. Ordering note: like the base LL128 kernel this is unfenced, but the flag now lives in a separate scratch region from the payload -- a stronger visibility assumption to validate on hardware (or guard with __threadfence_system()). Co-authored-by: Cursor <cursoragent@cursor.com>
Rename the full-line + barrier LL128 all-reduce header/kernel from the _1 suffix to _warpsync, and move its host launcher into a dedicated dda_all_reduce_fabric_ll128_warpsync.cu (the flagged kernel stays in dda_all_reduce_fabric_ll128.cu, now flagged-only). Introduce AllReduce-specific LL128 sub-tiering so the shared DDA_LL128_THRESHOLD (32 MiB) still governs the other collectives: AllReduce uses DDA_LL128_AR_THRESHOLD (2 MiB) for the flag-synced kernel and DDA_LL128_AR_WARPSYNC_THRESHOLD (32 MiB) for the warpsync variant. Wire the new translation unit through CMake and the device linker. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Improve performance of Allreduce for mid sizes from 4MB to 32MB
Technical Details
Summary
DDA_LL128_THRESHOLD(32 MiB) untouched for the other collectives (AllGather/AllToAll/ReduceScatter):size <= DDA_LL128_AR_THRESHOLD(2 MiB) -> LL128 flag-synced kernel (in-line trailing flag word, no barrier).size <= DDA_LL128_AR_WARPSYNC_THRESHOLD(32 MiB) -> LL128 warpsync kernel (full 128B-line payload synced by a per-warp arrival barrier)._1suffix to_warpsync(all_reduce_dda_fabric_ll128_warpsync.h, kernelddaAllReduceFlatLL128_warpsync).dda_all_reduce_fabric_ll128_warpsync.cu;dda_all_reduce_fabric_ll128.cuis now flagged-only (drops theRCCL_DDA_LL128_AR_FULL_LINEruntime toggle).src/CMakeLists.txtand the device linker (cmake/DeviceLinker.cmake).New env knobs (AllReduce-only):
RCCL_DDA_LL128_AR_THRESHOLD,RCCL_DDA_LL128_AR_WARPSYNC,RCCL_DDA_LL128_AR_WARPSYNC_THRESHOLD.Issue Tracking
Test Plan
Test Result
Performance improvement from size 4MB to 32MB
Submission Checklist