Skip to content

Add SQTT marker support to rocSHMEM to improve device-level performance tracing#8176

Open
mberenjk wants to merge 1 commit into
developfrom
users/mberenjk/add_sqtt_marker
Open

Add SQTT marker support to rocSHMEM to improve device-level performance tracing#8176
mberenjk wants to merge 1 commit into
developfrom
users/mberenjk/add_sqtt_marker

Conversation

@mberenjk

@mberenjk mberenjk commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Motivation

Adding sqtt_marker to device functions and critical sections of the code to mprove device-level performance tracing

Technical Details

Add SQTT marker support to rocSHMEM to enable device-side performance tracing. Similar to ROCTX, this provides annotations for performance analysis, but unlike ROCTX, it works on the device side, allowing correlation of rocSHMEM operations directly with GPU execution. This will improve trace visibility and help identify kernel-level bottlenecks more effectively.

JIRA ID

AIROCSHMEM-431

Test Plan

A complete test will be added in the CI with rocprofiler-sdk

Test Result

Submission Checklist

adding sqtt_enabled flag to hipcc
Adding sqtt_marker to different section of the code
@mberenjk
mberenjk requested a review from a team as a code owner July 7, 2026 00:54
@therock-pr-bot

therock-pr-bot Bot commented Jul 7, 2026

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
🌿 Branch Name ✅ Pass
📝 PR Title/Description ❌ Fail Error: Title does not follow Conventional Commits style.
Expected: start with a valid type (feat, fix, docs, …).
Desired format: type(optional-scope): short description
───
Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword.
Expected: include a JIRA ID / ISSUE ID line (separator : or -, or omitted; value may be a JIRA key, a number with/without #, or a link), OR a closing keyword + issue reference. Accepted examples:
JIRA ID : TESTAUTO-6039
JIRA ID - #330
JIRA ID #330
ISSUE ID : TESTUTO-3334
ISSUE ID #3334
ISSUE ID - TESTAUTO-3433
ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234
Closes #10
Fixes octo-org/octo-repo#100
Resolves: #123
#123
https://github.com/<org_name>/<repo_name>/issues/123
Current: no valid JIRA/ISSUE/closing-keyword reference found
Forbidden Files ✅ Pass
🧪 Unit Test ❌ Fail Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocshmem/src/backend_type.hpp, projects/rocshmem/src/gda/bnxt/queue_pair_bnxt.cpp, projects/rocshmem/src/gda/context_gda_device.cpp, projects/rocshmem/src/gda/context_gda_tmpl_device.hpp, projects/rocshmem/src/gda/ionic/queue_pair_ionic.cpp (+3 more); no test file found
🔎 pre-commit ⏳ Pending ⏳ Still running…
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

⚠️ 2 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

therock-pr-bot Bot commented Jul 7, 2026

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

@omor1 omor1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please hold until #7217 is merged to prevent conflicts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the switch case itself a performance concern?
If not, I'd recommend to have just enter post_wqe_rma here
(it can be if gda_provider_ can be slow to load)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be useful if we provided a way for enter/exit to contain extra information? E.g.

sqtt_marker_enter("mymarker")

if (a)
sqtt_marker_exit("mymarker", "A")
else
sqtt_marker_exit("mymarker", "B")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need a flag here to switch these off when needed, like you do in #ifdef ENABLE_ROCTX

Imagine we changed the path, or sqtt_trace.hpp wasn't available for some reason

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan on having this available globally in rocm soon. I dont think it hurts for now, though.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't hurt, but it's off by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants