[Code Health] Require upgraded a5 environment supporting sdma

[jvjhfhg]

Category

Technical Debt (cleanup, refactor)

Component

Host Runtime

Description

PR #1179 landed the a5 SDMA workspace overlay (ensure_sdma_workspace → aclnnShmemSdmaStarsQuery) but gated it behind an opt-in macro because the available a5 CANN drops do not expose working SDMA primitives:

• CANN 9.1.T500: aclnnShmemSdmaStarsQuery creates STARS streams but aclrtSynchronizeStream fails with AICPU exception 0x715002a, which poisons the AICPU context and turns every subsequent kernel launch into 507018. This breaks all a5 communication cases, not just the SDMA demo.
• CANN 9.1.0 (timestamp=20260625): HcclCommInitRootInfo itself returns HCCL_E_INTERNAL (4) — base HCCL never comes up.

The overlay was verified working on a separate a5 box whose CANN does expose the primitive (sdma_async_completion_demo passes). So this is purely an environment gating issue, not a code defect.

Currently the overlay defaults OFF via option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ... OFF) and the SIMPLER_ENABLE_PTO_SDMA_WORKSPACE env var. With it OFF: ensure_sdma_workspace is a no-op, CommContext.workSpace stays 0, the SDMA demo self-skips, and all other comm cases run normally.

Location

- `src/a5/platform/onboard/host/CMakeLists.txt` — `option(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ... OFF)` + conditional `PTO_ISA_ROOT` guard
- `simpler_setup/runtime_compiler.py` — `_init_a5` conditional `PTO_ISA_ROOT` ensure + `_sdma_workspace_enabled()`
- `simpler_setup/runtime_builder.py` — `_compile_target` env-var→CMake-define forwarding
- `examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py` — `pytest.skip` when env var unset

Proposed Fix

Once the st-onboard-a5 CI CANN exposes a working aclnnShmemSdmaStarsQuery (verify with nm -D $ASCEND_HOME_PATH/lib64/libascendcl.so | grep -ic SdmaStars), re-enable the overlay. Two options:

Option A — CI env var only (no code change): set SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON (plus PTO_ISA_ROOT) in the st-onboard-a5 job. The existing env-var→CMake define forwarding and test skip gate pick it up automatically.

Option B — make SDMA the a5 default (revert the gating, 4 spots):

1. CMakeLists.txt: option(... OFF) → set(SIMPLER_ENABLE_PTO_SDMA_WORKSPACE ON); move PTO_ISA_ROOT check + pto-isa include back out of the if-guard.
2. runtime_compiler.py _init_a5: make env_manager.ensure("PTO_ISA_ROOT") unconditional again (mirror _init_a2a3).
3. runtime_builder.py _compile_target: the env-var→define forwarding can stay (harmless) or be removed.
4. test_sdma_async_completion_demo.py: remove the env-var pytest.skip gate.

Optionally also have build_runtimes.py auto-resolve PTO_ISA_ROOT for a5 the same way it does for a2a3 (lines 120–125), so callers do not need to set it manually.

Verification after re-enabling:

python -m pytest examples/a5/tensormap_and_ringbuffer/sdma_async_completion_demo/test_sdma_async_completion_demo.py -v --platform a5 --device <ids> -s
python -m pytest examples/workers/l3/allreduce_distributed/test_allreduce.py -v --platform a5 --device <ids> -k onephase

The SDMA demo must PASS (not skip) and the allreduce regression must stay green.

Priority

Low (no impact today, good to fix eventually)

[jvjhfhg]

Added docs/a5-sdma-overlay.md as the single source of truth for the isolation status, with cross-referenced comments at every gating point. Updates to this issue:
Location — add:

• docs/a5-sdma-overlay.md — central doc (status, gating points, dev-under-isolation, re-enable steps)
• examples/a5/.../kernels/aiv/kernel_sdma_tget_async.cpp — workSpace == 0 self-skip guard now references the doc
  Proposed Fix — both Option A and Option B are unchanged, but after re-enabling also update docs/a5-sdma-overlay.md (the "Current state (default OFF)" section + re-enable verification) so the doc reflects the new always-on reality. The code comments at the 5 gating points already point to this doc, so no scattered comment edits are needed beyond flipping the gate itself.
  The doc covers: what the overlay is, why it's gated (CANN 9.1.T500 → AICPU exception 0x715002a → 507018; 9.1.0 → HCCL_E_INTERNAL), current default-OFF behavior, the 5 gating-point file table, how to develop non-SDMA / new-SDMA features under isolation, and the Option A/B re-enable steps with verification commands.

[ChaoZheng109]

⚠️ Required precondition before flipping SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON (both Option A and Option B): resolve #1351 first.

Turning the overlay ON makes a5 host_runtime.so compile pto-isa headers into itself, but the PTO-ISA build/run version guard (#1096 + #1194) is still scoped to arch == "a2a3" — it was never extended to a5 when #1179 added this dependency (see #1351). So re-enabling via either option here would embed pto-isa on a5 with no pin verification, no build-metadata recording, no load-time staleness rejection, and no cmake-cache / ccache invalidation on a pin bump — silently re-arming the 507018/507899-class stale-binary failure #1194 was written to prevent.

Please add '#1351 resolved' as an explicit checklist item before the re-enable steps (Option A and Option B).