Skip to content

Add: SDMA workspace overlay + async completion demo on a5 onboard#1179

Open
jvjhfhg wants to merge 2 commits into
hw-native-sys:mainfrom
jvjhfhg:feat/comm-a5-sdma
Open

Add: SDMA workspace overlay + async completion demo on a5 onboard#1179
jvjhfhg wants to merge 2 commits into
hw-native-sys:mainfrom
jvjhfhg:feat/comm-a5-sdma

Conversation

@jvjhfhg

@jvjhfhg jvjhfhg commented Jun 27, 2026

Copy link
Copy Markdown
Collaborator

Layers the host-side SDMA workspace allocation on top of the comm backend from the previous commit. Until CANN exposes the missing SDMA primitives on a5, this overlay is the only piece of comm work that fails on real a5 silicon -- aclnnShmemSdmaStarsQuery raises an AICPU exception (InnerCode=0x715002a) that aborts the entire ACL thread context. Dropping this commit therefore unblocks the non-SDMA comm demos (async_notify_demo etc.) without touching the deferred-completion runtime, which is already SDMA-aware on the kernel side (dormant until a kernel registers an SDMA condition).

  • Wire SdmaWorkspaceManager into comm_alloc_windows under SIMPLER_ENABLE_PTO_SDMA_WORKSPACE: pre-allocates the per-rank workspace via aclnnShmemSdmaStarsQuery and overlays the result into CommContext.workSpace/.workSpaceSize. On CANN 8.5 the dlsym fails by design and we demote to "no workspace" rather than failing comm_init.
  • a5 onboard CMakeLists forces SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ON, requires PTO_ISA_ROOT (with FATAL_ERROR message pointing to the workspace coupling), adds pto-isa headers to the include path, and links libnnopbase.
  • runtime_compiler._init_a5 enforces the same PTO_ISA_ROOT env contract as _init_a2a3.
  • Migrate sdma_async_completion_demo to examples/a5/ (kernels + orch byte-identical with the a2a3 version; test.py platform- renamed).

@jvjhfhg jvjhfhg changed the title [WIP] Add: SDMA workspace overlay + async completion demo on a5 onboard Add: SDMA workspace overlay + async completion demo on a5 onboard Jun 27, 2026
@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3e90bcf6-9a49-4d6d-bf3a-5dd645a40b6d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds an A5 SDMA async-completion demo: new device kernels and orchestration code, host-side workspace support, updated A5 build/runtime checks for PTO_ISA_ROOT, and a smoke test that builds, runs, and validates the two-device flow.

Changes

SDMA async completion demo

Layer / File(s) Summary
A5 build and runtime contract
simpler_setup/runtime_compiler.py, src/a5/platform/onboard/host/CMakeLists.txt
A5 host setup requires PTO_ISA_ROOT, adds its include path, and forces SIMPLER_ENABLE_PTO_SDMA_WORKSPACE into host_runtime compile and link settings.
HCCL workspace ownership
src/a5/platform/onboard/host/comm_hccl.cpp
The HCCL host handle conditionally includes and owns SdmaWorkspaceManager under SIMPLER_ENABLE_PTO_SDMA_WORKSPACE, and the nearby window-allocation comment is updated.
Producer and consumer kernels
examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/aiv/kernel_sdma_tget_async.cpp, examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/aiv/kernel_consumer.cpp
Adds the peer-window SDMA kernel_entry and the tile-processing consumer kernel entrypoint used by the demo.
Orchestration and smoke test
examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/orchestration/sdma_async_completion_orch.cpp, examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py
Adds orchestration entrypoints with four-argument validation and producer/consumer task submission, plus the Python smoke test that builds the chip callable, runs on two devices, and checks out and result.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • hw-native-sys/simpler#823 — Adds the a5 comm port and deferred-completion/SDMA backend that this demo and workspace plumbing build on.
  • hw-native-sys/simpler#1166 — Updates SDMA/PTO-ISA header and include-path handling that matches the new workspace-gated host and kernel code here.

Poem

A bunny hopped through tiles so neat,
With SDMA drums beneath my feet.
One peer-window carrot, bright and plain,
Made peer_input + 1 shine again.
Thump-thump — the test passed cleanly! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the SDMA workspace overlay and a5 async completion demo migration.
Description check ✅ Passed The description is on-topic and matches the comm backend overlay, a5 build updates, and demo migration.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an SDMA deferred completion demo for the onboard a5 platform, adding consumer and async SDMA TGET kernels, orchestration logic, and a Python smoke test. It also enables the PTO-ISA async SDMA workspace pre-allocation by default, making PTO_ISA_ROOT a hard requirement for the a5 onboard host runtime. The review feedback suggests tightening argument validation in the orchestration code to prevent potential out-of-bounds access and robustly checking for empty PTO_ISA_ROOT environment variables in CMake.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread src/a5/platform/onboard/host/CMakeLists.txt Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/orchestration/sdma_async_completion_orch.cpp`:
- Around line 31-39: The current guard in sdma_async_completion_orch.cpp only
checks the total argument count, so a bad tensor/scalar mix can still reach
tensor(2) and scalar(0). Update the validation around the orchestration argument
parsing to verify the exact split expected by Tensor accessors and the comm_ctx
scalar, not just orch_args.tensor_count() + orch_args.scalar_count(). Keep the
existing error handling in the same flow so invalid inputs are rejected before
from_tensor_arg() and reinterpret_cast<CommContext *> are used.

In
`@examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py`:
- Around line 27-37: The test is still importing and using the deprecated
task-interface alias ContinuousTensor instead of the renamed Tensor type, so
update the import list in the sdma_async_completion demo test to use Tensor and
replace any ContinuousTensor references in the test setup with Tensor. Keep the
rest of the task-interface imports unchanged and ensure the test only depends on
the current public symbol from simpler.task_interface.
- Around line 66-86: The child callables built in the loop around
CoreCallable.build are advertising the wrong ABI because both entries reuse the
parent’s 4-arg signature. Update each child metadata entry to match the actual
kernel interface for kernel_sdma_tget_async.cpp and kernel_consumer.cpp, so the
producer and consumer callables each expose their real argument
directions/counts instead of the parent signature.

In `@src/a5/platform/onboard/host/CMakeLists.txt`:
- Around line 44-49: The current PTO_ISA_ROOT check in the host CMake logic only
verifies that the environment variable is defined, so an empty or nonexistent
path still reaches the include path append and fails later. Update the
validation near the CMakeLists.txt guard around the `PTO_ISA_ROOT` handling to
also reject empty values and paths that do not exist before `list(APPEND
CMAKE_CUSTOM_INCLUDE_DIRS ...)`, and keep the fatal error in the same
`host_runtime` setup path so configuration fails immediately with a clear
message.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 120b4b64-2bb8-4a14-88f8-98090fe3ab51

📥 Commits

Reviewing files that changed from the base of the PR and between 47a411c and 781c4e2.

📒 Files selected for processing (7)
  • examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/aiv/kernel_consumer.cpp
  • examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/aiv/kernel_sdma_tget_async.cpp
  • examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/kernels/orchestration/sdma_async_completion_orch.cpp
  • examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py
  • simpler_setup/runtime_compiler.py
  • src/a5/platform/onboard/host/CMakeLists.txt
  • src/a5/platform/onboard/host/comm_hccl.cpp

Comment thread src/a5/platform/onboard/host/CMakeLists.txt Outdated
@jvjhfhg jvjhfhg force-pushed the feat/comm-a5-sdma branch 6 times, most recently from 4547a68 to a7ac4ca Compare July 3, 2026 03:44
@jvjhfhg jvjhfhg force-pushed the feat/comm-a5-sdma branch 2 times, most recently from f895e40 to 4a409f2 Compare July 9, 2026 10:16
Layers the host-side SDMA workspace allocation on top of the comm
backend from the previous commit. Until CANN exposes the missing
SDMA primitives on a5, this overlay is the only piece of comm work
that fails on real a5 silicon -- aclnnShmemSdmaStarsQuery raises
an AICPU exception (InnerCode=0x715002a) that aborts the entire
ACL thread context. Dropping this commit therefore unblocks the
non-SDMA comm demos (async_notify_demo etc.) without touching the
deferred-completion runtime, which is already SDMA-aware on the
kernel side (dormant until a kernel registers an SDMA condition).

- Wire SdmaWorkspaceManager into comm_alloc_windows under
  SIMPLER_ENABLE_PTO_SDMA_WORKSPACE: pre-allocates the per-rank
  workspace via aclnnShmemSdmaStarsQuery and overlays the result
  into CommContext.workSpace/.workSpaceSize. On CANN 8.5 the
  dlsym fails by design and we demote to "no workspace" rather
  than failing comm_init.
- a5 onboard CMakeLists forces SIMPLER_ENABLE_PTO_SDMA_WORKSPACE
  ON, requires PTO_ISA_ROOT (with FATAL_ERROR message pointing
  to the workspace coupling), adds pto-isa headers to the include
  path, and links libnnopbase.
- runtime_compiler._init_a5 enforces the same PTO_ISA_ROOT env
  contract as _init_a2a3.
- Migrate sdma_async_completion_demo to examples/a5/ (kernels +
  orch byte-identical with the a2a3 version; test.py platform-
  renamed).
@jvjhfhg jvjhfhg force-pushed the feat/comm-a5-sdma branch from 4a409f2 to efe107d Compare July 10, 2026 01:53
The SDMA workspace init (ensure_sdma_workspace -> aclnnShmemSdmaStarsQuery)
breaks all a5 communication cases on CANN drops that lack working a5 SDMA
primitives (T500: AICPU exception 0x715002a -> 507018; 9.1.0:
HCCL_E_INTERNAL). Gating it behind an opt-in macro lets the branch merge
now and re-enable once the a5 environment exposes the primitives.

- CMakeLists: set(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ON) -> option(... OFF);
  PTO_ISA_ROOT check + pto-isa include moved inside the if-guard.
- runtime_compiler._init_a5: PTO_ISA_ROOT ensure now conditional on the
  SIMPLER_ENABLE_PTO_SDMA_WORKSPACE env var (mirrors the CMake option).
- runtime_builder: forward the env var to the host CMake define so
  SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON pip install -e . opts in.
- sdma_async_completion_demo: skip when CommContext.workSpace == 0
  (overlay not built) instead of silently producing all-zero output.

Default (env var unset): overlay dormant, ensure_sdma_workspace is a no-op,
SDMA demo self-skips, all comm cases run normally.
@jvjhfhg jvjhfhg force-pushed the feat/comm-a5-sdma branch from efe107d to d4aa2dd Compare July 10, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant