[rocprofiler-sdk] Add RCCL function args serialization to buffer records#8271
Closed
prasanna-amd wants to merge 5 commits into
Closed
[rocprofiler-sdk] Add RCCL function args serialization to buffer records#8271prasanna-amd wants to merge 5 commits into
prasanna-amd wants to merge 5 commits into
Conversation
github-actions
Bot
added
documentation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new RCCL_API_EXT buffered tracing kind to rocprofiler-sdk to serialize RCCL function arguments/return values (similar to HIP_API_EXT), and wires that through tool output + new integration tests (single-process multi-GPU RCCL) with JSON/CSV/Perfetto validation.
Changes:
- Introduces
ROCPROFILER_BUFFER_TRACING_RCCL_API_EXTand a newrocprofiler_buffer_tracing_rccl_api_ext_record_tcarrying args/retval. - Updates SDK/tool/output pipelines (JSON/CSV/Perfetto/OTF2/rocpd/stats) to consume the RCCL EXT record type.
- Adds RCCL test binary + integration tests + pytest validators for rocprofv3 and json-tool paths.
Reviewed changes
Copilot reviewed 42 out of 46 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| projects/rocprofiler-sdk/tests/tools/json-tool.cpp | Switches RCCL buffered tracing to EXT + adds output filename helper. |
| projects/rocprofiler-sdk/tests/rocprofv3/rccl-trace/validate-single-process.py | New rocprofv3 RCCL tracing validator (JSON/CSV). |
| projects/rocprofiler-sdk/tests/rocprofv3/rccl-trace/pytest.ini | Pytest config for rocprofv3 RCCL tracing test. |
| projects/rocprofiler-sdk/tests/rocprofv3/rccl-trace/conftest.py | Pytest fixtures for reading JSON/CSV outputs. |
| projects/rocprofiler-sdk/tests/rocprofv3/rccl-trace/CMakeLists.txt | Adds rocprofv3 integration execute/validate tests for RCCL tracing. |
| projects/rocprofiler-sdk/tests/rocprofv3/CMakeLists.txt | Adds rocprofv3/rccl-trace subdirectory. |
| projects/rocprofiler-sdk/tests/rccl/validate-single-process.py | New json-tool-based RCCL tracing validator. |
| projects/rocprofiler-sdk/tests/rccl/pytest.ini | Pytest config for json-tool RCCL tracing test. |
| projects/rocprofiler-sdk/tests/rccl/conftest.py | Pytest fixtures for json-tool output. |
| projects/rocprofiler-sdk/tests/rccl/CMakeLists.txt | Adds json-tool integration execute/validate tests for RCCL tracing. |
| projects/rocprofiler-sdk/tests/CMakeLists.txt | Adds tests/rccl subdirectory. |
| projects/rocprofiler-sdk/tests/bin/rccl/single-process/main.cpp | RCCL single-process multi-GPU test binary. |
| projects/rocprofiler-sdk/tests/bin/rccl/single-process/CMakeLists.txt | Builds RCCL single-process test binary with HIP. |
| projects/rocprofiler-sdk/tests/bin/rccl/CMakeLists.txt | Adds RCCL test binaries when rccl is found. |
| projects/rocprofiler-sdk/tests/bin/CMakeLists.txt | Adds tests/bin/rccl subdirectory. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/utils.hpp | Adds stringization helpers for RCCL arg serialization. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/rccl.hpp | Extends RCCL arg iteration to support buffered EXT callbacks. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/rccl.def.cpp | Adds RCCL buffered EXT domain metadata/type wiring. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/rccl.cpp | Emits RCCL EXT buffered records with args/retval + arg iteration changes. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/details/fmt.hpp | Adds fmt formatters for RCCL-related types. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/details/CMakeLists.txt | Adds RCCL details headers to object library. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/rccl/defines.hpp | Updates RCCL API wrapper macros to support args list/stringization. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/callback_tracing.cpp | Enables iterate-args support for RCCL callback tracing. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/buffer_tracing.cpp | Adds RCCL_API_EXT kind support + iterate buffer record args for RCCL EXT. |
| projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp | Switches tool buffered callback/config to RCCL_API_EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateStats.hpp | Stats generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateStats.cpp | Stats generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateRocpd.hpp | rocpd generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateRocpd.cpp | rocpd generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generatePerfetto.hpp | Perfetto generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generatePerfetto.cpp | Adds perfetto annotations for RCCL args from EXT record. |
| projects/rocprofiler-sdk/source/lib/output/generateOTF2.hpp | OTF2 generator signature updated to RCCL EXT. |
| projects/rocprofiler-sdk/source/lib/output/generateOTF2.cpp | OTF2 generator updated to emit RCCL events from EXT records. |
| projects/rocprofiler-sdk/source/lib/output/generateJSON.hpp | JSON generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateJSON.cpp | JSON generator now serializes RCCL EXT records under rccl_api. |
| projects/rocprofiler-sdk/source/lib/output/generateCSV.hpp | CSV generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/generateCSV.cpp | CSV generator now consumes RCCL EXT record type. |
| projects/rocprofiler-sdk/source/lib/output/buffered_output.hpp | Buffered output typedef switched to RCCL EXT record. |
| projects/rocprofiler-sdk/source/include/rocprofiler-sdk/fwd.h | Adds new buffer tracing kind enum: RCCL_API_EXT. |
| projects/rocprofiler-sdk/source/include/rocprofiler-sdk/cxx/serialization/save.hpp | Adds serialization for RCCL EXT record (args + retval). |
| projects/rocprofiler-sdk/source/include/rocprofiler-sdk/cxx/perfetto.hpp | Adds perfetto category mapping for RCCL_API_EXT. |
| projects/rocprofiler-sdk/source/include/rocprofiler-sdk/cxx/enum_string.hpp | Updates enum labels/static_assert for new kind. |
| projects/rocprofiler-sdk/source/include/rocprofiler-sdk/buffer_tracing.h | Defines rocprofiler_buffer_tracing_rccl_api_ext_record_t. |
| projects/rocprofiler-sdk/source/docs/rocprofv3-schema.json | Updates schema to include RCCL buffer record shape. |
| projects/rocprofiler-sdk/samples/common/CMakeLists.txt | Expands default fail regex for tests. |
| projects/rocprofiler-sdk/cmake/rocprofiler_options.cmake | Expands default fail regex for tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add RCCL_API_EXT buffer tracing kind that captures RCCL function arguments and return values alongside existing trace metadata. This mirrors the existing HIP_API_EXT pattern. Includes test infrastructure with single-process multi-GPU RCCL test and pytest validators for JSON, CSV, and Perfetto output formats. Rebased onto current develop, resolving conflicts with refactored test infrastructure (rocprofiler_add_integration_execute_test / rocprofiler_add_integration_validate_test). Originally from ROCm/rocprofiler-sdk#129 by Mythreya Kuricheti. Ref: SWDEV-528449, SWDEV-527517
prasanna-amd
force-pushed
the
users/prasanna-amd/rocprof-rccl-func-args-serialization
branch
from
July 8, 2026 19:30
5de161b to
b99378d
Compare
added 4 commits
July 8, 2026 16:38
Remove -vv from pytest.ini addopts (breaks test discovery in rocprofiler_add_integration_validate_test) and fix extra blank line in json-tool.cpp flagged by clang-format.
…name bugs Add fmt::formatter for ncclWaitSignalDesc_t which was added to RCCL after the original serialization code was written, causing a compile error in stringize_impl's ostream fallback. Make stringize_impl resilient to future unstreamable types by adding SFINAE detection for operator<< before attempting stream insertion. Fix get_filename() when ROCPROFILER_TOOL_OUTPUT_FILE_APPEND_PID is set: preserve parent directory, avoid double-dot from extension(), use the caller-provided extension as fallback instead of hardcoding .json, and return a properly sized string instead of a 512-byte null-padded buffer.
Fix fmt::formatter for ncclUniqueId: cast char[] to unsigned char[] for proper hex encoding and use :02x for zero-padded 2-digit hex per byte. Fix fmt::formatter for ncclComm_t: use fmt::ptr() instead of integer hex format specifier on an opaque pointer type. Fix pytest.ini files: remove bracket syntax from testpaths to match repo convention, fix rocprofv3 rccl-trace testpaths to reference the actual validate-single-process.py instead of nonexistent validate-sp.py and validate-mp.py. Fix rocprofv3 validate script: use RCCL_API_EXT instead of RCCL_API for buffer record operation lookup, matching the actual buffer tracing kind string. Fix rocprofv3-schema.json: rename rccl_api_traces to rccl_api to match the key emitted by generateJSON.cpp, add missing retval field. Fix typo: cakkback -> callback in test comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RCCL_API_EXTbuffer tracing kind to rocprofiler-sdk that captures RCCL function arguments and return values alongside existing trace metadata (mirrors the HIP_API_EXT pattern)rocprofiler_add_integration_execute_test/rocprofiler_add_integration_validate_test)Originally from ROCm/rocprofiler-sdk#129 by Mythreya Kuricheti (PR #184, closed for rebase).
Ref: SWDEV-528449, SWDEV-527517
Associated Jira Ticket Number/Link
SWDEV-528449
SWDEV-527517
What type of PR is this?
Test plan
Note: Previous PR #184 was blocked citing "IOMMU support on CI" but this was a misdiagnosis. The RCCL IOMMU check at
init.cc:301is only a warning (WARN), not an error — it does not prevent initialization or cause test failure. The actual CI issue was likely missing multi-GPU support or environment configuration in the CI runner, not IOMMU.Generated with Claude Code