Skip to content

feat(clr): any-order overlap for oversubscribed shared HW queues.#8916

Draft
jaydeeppatel1111 wants to merge 1 commit into
developfrom
users/jaypatel/clr-shared-queue-anyorder-overlap
Draft

feat(clr): any-order overlap for oversubscribed shared HW queues.#8916
jaydeeppatel1111 wants to merge 1 commit into
developfrom
users/jaypatel/clr-shared-queue-anyorder-overlap

Conversation

@jaydeeppatel1111

@jaydeeppatel1111 jaydeeppatel1111 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Motivation

When the number of HIP streams exceeds the hardware queue pool (GPU_MAX_HW_QUEUES, default 4), extra streams are recycled onto shared HW rings. Today every kernel keeps its head barrier bit set, so streams that merely share a ring serialize against each other even when they are logically independent — throughput collapses linearly with stream count. This adds an opt-in path that lets independent streams overlap on a shared ring, recovering most of the lost concurrency.

Technical Details

  • New release flag DEBUG_HIP_SHARED_QUEUE_ANYORDER (default 0, opt-in). App must guarantee the sharing streams are independent (no cross-stream data hazard).
  • In VirtualGPU::submitKernelInternal, clear the AQL head barrier bit for a stream's first packet since queue acquire on an oversubscribed ring. It reuses the existing any-order path (same bit-mask as hipExtAnyOrderLaunch).
  • Gated by a conservative guard set so we only drop incidental same-ring serialization, never a real dependency:
  1. last_write_index_ == kInvalidQueueIndex (stream's first packet since acquire),
  2. eventWaitList().empty() (no explicit cross-stream dependency),
  3. !cooperativeGroups()
  4. !dedicated_queue_,
  5. SharedHwQueueRefCount(queue, priority) > 1 (ring actually shared).
  • Device::SharedHwQueueRefCount returns the pool share count for a HW queue, searching only the stream's priority pool (mirrors acquireQueue, honors DEBUG_HIP_IGNORE_STREAM_PRIORITY).
  • Graph-capture exclusion (!isGraphCapture): graph replay encodes producer→consumer ordering as same-HW-queue in-order execution with an empty waitlist; clearing the barrier there would let a consumer overlap its producer. This guard keeps the graph executor's in-order invariant intact.

Issue Tracking

JIRA ID : AIRUNTIME-74

Test Plan

New correctness test unit/stream/hipSharedQueueAnyOrderOverlap.cc: independent oversubscribed streams produce correct results; explicit hipStreamWaitEvent dependencies are honored (must pass flag ON and OFF).
New perf test performance/scenarios/stream/hipPerfSharedQueueAnyOrderOverlap.cc: sweeps 2–32 streams, reports µs/iter.

Test Result

Full suite parity: ON and OFF on NV48
Performance (Performance_hipPerfSharedQueueAnyOrderOverlap, µs/iter) with streams 8, 16, 32 shows speedup 1.9x, 3.7x, 7.2x respectively.

Submission Checklist

@therock-pr-bot

therock-pr-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
🌿 Branch Name ✅ Pass
📝 PR Title/Description ✅ Pass
Forbidden Files ✅ Pass
🧪 Unit Test ❌ Fail Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/clr/rocclr/device/rocm/rocdevice.cpp, projects/clr/rocclr/device/rocm/rocdevice.hpp, projects/clr/rocclr/device/rocm/rocvirtual.cpp, projects/clr/rocclr/utils/flags.hpp, projects/hip-tests/catch/performance/scenarios/stream/hipPerfSharedQueueAnyOrderOverlap.cc (+1 more); no test file found
🔎 pre-commit ⏳ Pending ⏳ Still running…
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

therock-pr-bot Bot commented Jul 21, 2026

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

@jaydeeppatel1111
jaydeeppatel1111 force-pushed the users/jaypatel/clr-shared-queue-anyorder-overlap branch from c2b3c22 to 89c5c5d Compare July 21, 2026 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant