[Bug] Multi-rank comm_alloc_domain_windows blocked by driver support_shmem_map_exbus=0 (cross-card aclrtIpcMemImportByKey returns 507899)

[csy0225]

[Bug] Multi-rank comm_alloc_domain_windows blocked by driver support_shmem_map_exbus=0 — cross-card aclrtIpcMemImportByKey returns 507899

Repository: hw-native-sys/simpler (for the backend-choice fix path);
secondary CANN driver / Ascend platform team (for the underlying capability
gap). Refile to the right component on triage.

Filed as: simpler#1037 (2026-06-12)

Severity: blocking multi-rank L3 deployment on driver 25.5.1 + CANN
9.0.0-beta.1; we cannot reach Phase 16 (TP=8 / EP=8) of step3p5 bring-up.

Filed jointly with step3p5-507018-vec-ub-align.md (Phase 15 single-rank
blocker). The two problems are logically independent — fixing this one
unblocks Phase 16 multi-card deployment but does not fix the 507018
single-rank VEC UB align fault, and vice versa.

Summary

After simpler#1018
(comm_init segfault) was resolved via the --no-as-needed link patch on
src/{a2a3,a5}/platform/onboard/host/CMakeLists.txt (this session, verified
working with nm -D libhost_runtime.so | grep NEEDED showing libhcomm.so
present), the next failure on the multi-rank path is now:

[ERROR] domain_alloc_via_ipc: aclrtIpcMemImportByKey failed with 507899
[ERROR] comm_alloc_domain_windows: domain alloc failed for chip N

Tracing through CANN runtime, the underlying driver query is:

halShmemOpenHandleByDevId(chip_handle, peer_dev_id, &remote_handle);
// returns DRV_ERROR_PARA_ERROR(8) on this driver, because:
//   driver_query(chip_handle, /*query=*/SUPPORT_SHMEM_MAP_EXBUS) == 0

That is, the driver firmware on this machine reports
support_shmem_map_exbus = 0, which means cross-die / cross-card shmem
mapping over Ex-Bus is not supported on this driver+silicon
combination. simpler's Path-D DIY-IPC comm backend depends on this
capability for cross-card window mapping, so it cannot complete
comm_alloc_domain_windows.

Reproducer

After applying the --no-as-needed link patch from simpler#1018 and the
CANN-9.0.0-beta.1 install, run any multi-rank simpler L3 example that calls
comm_alloc_domain_windows:

export PTO_ISA_ROOT=/path/to/pto-isa
export ASCEND_HOME_PATH=/usr/local/Ascend/cann-9.0.0-beta.1
cd /path/to/simpler/examples/workers/l3/allreduce_distributed
PYTHONFAULTHANDLER=1 python -X faulthandler main.py -p a2a3 -d 0-1

After comm_init succeeds (no longer segfaults), the next stage fails:

[chip 0] comm_init done
[chip 1] comm_init done
[ERROR] domain_alloc_via_ipc: aclrtIpcMemImportByKey failed with 507899
        flags=0 (also tested ACL_RT_IPC_MEM_IMPORT_FLAG_ENABLE_PEER_ACCESS)
[ERROR] comm_alloc_domain_windows: domain alloc failed for chip 1

Direct driver query

The capability gap is observable at the driver layer, independent of
simpler:

# Use Ascend's driver query interface (or equivalent CANN-side probe)
# to read the SUPPORT_SHMEM_MAP_EXBUS bit for chip 0:
$ ascend-driver-query --chip 0 --capability support_shmem_map_exbus
support_shmem_map_exbus = 0   # ← false on this driver/firmware

A driver that reports support_shmem_map_exbus = 1 should accept the same
halShmemOpenHandleByDevId(...) call and return a valid remote handle,
unblocking aclrtIpcMemImportByKey.

Verification that this is the driver and not simpler

Two independent control experiments confirm the limitation is in the driver
firmware, not in simpler's IPC bring-up code:

1. vLLM-Ascend TP=8 on the same pod, same 8 cards, same CANN 9.0.0-beta.1,
   same driver 25.5.1 — passes cleanly:

   vllm serve <model> --tensor-parallel-size 8
   # [OK] Worker 7 ready, all 8 ranks serving traffic

   vLLM-Ascend uses CANN's HCCL collective communication primitives
   (torch.distributed.init_process_group("hccl") with spawn-style worker
   bring-up). It does not touch aclrtIpcMemImportByKey or
   halShmemOpenHandleByDevId because HCCL collectives bypass user-space
   cross-card device-memory IPC entirely.

2. simpler's own aclrtDeviceEnablePeerAccess succeeds (returns 0) on
   the same chip pair where aclrtIpcMemImportByKey returns 507899. Peer
   access is enabled at the device level; what's missing is the
   shmem-over-exbus path that aclrtIpcMemImportByKey requires.

So the runtime stack (driver / HCCL / firmware) is healthy for collective
communication, and the specific gap is "user-space cross-card IPC via Ex-Bus
shmem mapping" which simpler's current DIY-IPC backend relies on.

Version pin table

Component	HEAD	Notes
Chip	8 × Ascend 910B2C, Short_SoC_version=Ascend910B	shared 64 GB HBM per die, PCIe Gen4 x16 between cards
OS	Ubuntu 22.04, x86_64 host
Driver	npu-smi 25.5.1, firmware 7.8.0.6.201	the driver where support_shmem_map_exbus=0
CANN	9.0.0-beta.1 (matches driver)	Also reproduces on 8.5.1 after similar patches
simpler	branch fix/tensor-zero-size-view-bounds:0cd317e7 (= PR #1023 + host --no-as-needed + comm_hccl P2P best-effort + ImportByKey ENABLE_PEER_ACCESS)	comm_init segfault simpler#1018 fix verified working
pto-isa	main:109c9f72
PTOAS	binary v0.44

What we tried (none fix it)

1. Enable peer access first via aclrtDeviceEnablePeerAccess before
   aclrtIpcMemImportByKey → still 507899.
2. Pass ACL_RT_IPC_MEM_IMPORT_FLAG_ENABLE_PEER_ACCESS flag to
   aclrtIpcMemImportByKey (instead of flags=0) → still 507899.
3. Retry with ACL_RT_PEER_ACCESS_BLOCK_MODE_AUTO and other flag
   combinations from the CANN 9.0 ACL reference → still 507899.
4. Hit each pair of chip IDs (0↔1, 0↔7, 6↔7) — all fail identically.
   Not a chip-pair-specific routing problem.

Per the comments at
simpler/src/a2a3/platform/onboard/host/comm_hccl.cpp:652-660, the SDMA
workspace path that depends on the same capability is also gated; we have
that path compiled out (SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=OFF) as a
side-effect of our comm_init workaround, but the underlying capability
remains absent.

Two possible fix paths (asking maintainers to choose)

Path A — driver / firmware update enables support_shmem_map_exbus

If the underlying silicon (910B2C) actually supports cross-card Ex-Bus shmem
mapping, the right fix is a driver / firmware update that turns the
capability bit on. Cleanest, requires zero simpler-side change. The
infrastructure team has to deploy this — outside our control, hence this
issue.

Path B — simpler swaps DIY-IPC for HCCL collective comm

If the capability is not supported on this silicon and never will be,
simpler's L3 comm backend should fall back to CANN's HCCL collectives
(HcclAllReduce, HcclAllToAll, etc.) the same way vLLM-Ascend does.
That's a larger refactor but it would unblock all current driver/firmware
combinations. Requires:

• Add a new comm_backend selector in simpler init (we'd default to
  hccl_collectives on driver-without-exbus, keep diy_ipc available for
  driver-with-exbus).
• Map simpler's per-call pld.tensor.put / pld.tensor.get primitives onto
  HCCL collective calls (or HCCL P2P send/recv where the semantics fit).

We can pilot Path B in our fork if that's the preferred direction; happy to
contribute the patch.

What we are asking for

1. Decide which path (A driver-fix vs B backend-swap) is the intended
   long-term fix.
2. If A: tell us what driver/firmware version turns support_shmem_map_exbus
   on for 910B2C, so we can request the install.
3. If B: confirm direction and we'll prepare a draft PR adding the
   HCCL-collective backend behind a comm-backend selector in simpler init.

Related local artifacts (available on request)

• Plog from a failing 2-rank allreduce_distributed run with the
  comm_init segfault fix applied
• npu-smi info -t topo output showing PCIe inter-die routing on this pod
• Output of the SDMA workspace probe (SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON
  reproduces the original AICPU ShmemSdmaStarsQuery 0x2a → 507018 cascade
  the CMake comment at comm_hccl.cpp:36-50 describes; with OFF, that
  particular cascade is gone and we land cleanly at the
  aclrtIpcMemImportByKey 507899 described here)

See also

• docs/upstream-issues/simpler-comm-init-segfault.md — the prerequisite
  fix (simpler#1018)
  for comm_init segfault; verified working this session via
  --no-as-needed link patch
• docs/upstream-issues/step3p5-507018-vec-ub-align.md — Phase 15
  single-rank VEC UB align (filed jointly with this issue; logically
  independent)
• docs/step3p5/phases/16-multirank-npu.md — local Phase 16 status,
  including the working comm_init verification and the
  aclrtIpcMemImportByKey 507899 block detail

---

中文说明

一句话总结

step3p5 多卡 deployment 在 simpler#1018
（comm_init 段错）通过 host CMakeLists --no-as-needed 链接补丁解决之后，
下一个卡点变成了 driver firmware 的能力缺口：910B2C 在当前 driver/CANN
组合下 support_shmem_map_exbus = 0，导致 simpler 的 DIY-IPC 通信后端做
跨卡 window mapping 时 aclrtIpcMemImportByKey 一律返回 507899。

复现

修了 simpler#1018 的 --no-as-needed 之后，跑 simpler 自带的多卡 L3 例子：

export PTO_ISA_ROOT=/path/to/pto-isa
export ASCEND_HOME_PATH=/usr/local/Ascend/cann-9.0.0-beta.1
cd /path/to/simpler/examples/workers/l3/allreduce_distributed
PYTHONFAULTHANDLER=1 python -X faulthandler main.py -p a2a3 -d 0-1
# 期望: comm_init 已不再段错；下一阶段崩在
# [ERROR] domain_alloc_via_ipc: aclrtIpcMemImportByKey failed with 507899
# [ERROR] comm_alloc_domain_windows: domain alloc failed for chip 1

CANN runtime 内部对应：

halShmemOpenHandleByDevId(chip_handle, peer_dev_id, &remote_handle);
// 返回 DRV_ERROR_PARA_ERROR(8)，因为
//   driver_query(chip_handle, /*query=*/SUPPORT_SHMEM_MAP_EXBUS) == 0

反证：vLLM-Ascend 同环境通过

同 pod / 同 8 卡 / 同 driver 25.5.1 / 同 CANN 9.0.0-beta.1：

vllm serve <model> --tensor-parallel-size 8
# [OK] Worker 7 ready, all 8 ranks serving traffic

vLLM-Ascend 用的是 CANN 的 HCCL 集合通信原语
（torch.distributed.init_process_group("hccl") + spawn 起 worker），
完全不走 aclrtIpcMemImportByKey 或 halShmemOpenHandleByDevId，
因为 HCCL 集合通信绕开了 user-space 跨卡 device-memory IPC。所以 chip

• runtime + HCCL 没毛病；缺的是 simpler DIY-IPC 后端依赖的"跨卡 Ex-Bus
  shmem mapping"这个特定能力。

我们的两条 fix path（请上游选）

Path A — 驱动/固件升级，开 support_shmem_map_exbus

如果 910B2C 硬件确实支持跨卡 Ex-Bus shmem mapping，正确的修法是出一个
开了这个 capability bit 的 driver/firmware。simpler 侧零改动，最干净。
但需要基础设施团队部署 — 不在我们的控制范围，所以才发这个 issue。

Path B — simpler 把 DIY-IPC 换成 HCCL 集合通信

如果 silicon 不支持也不会再支持 Ex-Bus shmem mapping，那 simpler 的 L3
通信后端应该 fallback 到 CANN HCCL collectives（HcclAllReduce、
HcclAllToAll 等），跟 vLLM-Ascend 一样。这个改动大一些但能解锁所有当前
driver/firmware 组合。需要：

• simpler init 加一个 comm_backend 选项（默认在 driver-without-exbus
  上选 hccl_collectives，driver-with-exbus 时仍可选 diy_ipc）
• simpler 的每一次 pld.tensor.put / pld.tensor.get 原语映射到 HCCL
  collective 调用（或者 HCCL P2P send/recv，按语义合适的）

如果上游选 Path B，我们这边可以试做 patch 在 fork 上，欢迎指导方向。

我们已经试过没用的

尝试	结果
aclrtIpcMemImportByKey 之前先 aclrtDeviceEnablePeerAccess	还是 507899
给 aclrtIpcMemImportByKey 传 ACL_RT_IPC_MEM_IMPORT_FLAG_ENABLE_PEER_ACCESS flag	还是 507899
试 ACL_RT_PEER_ACCESS_BLOCK_MODE_AUTO 等其他 CANN 9.0 ACL 文档里的 flag 组合	还是 507899
换不同 chip pair (0↔1, 0↔7, 6↔7)	都崩，不是 chip-pair-specific 路由问题

注：SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=OFF 在我们这边已经强制关闭
（comm_hccl.cpp:36-50 的注释说明这条路径会触发 aclnnShmemSdmaStarsQuery
AICPU 0x2a → 507018 cascade）。关掉之后那条 cascade 不再触发，但底层能力
缺口还在 = 卡在 aclrtIpcMemImportByKey 507899。

可按需要提供的本地工件

• 加了 [Bug] Multi-rank comm_init fails in os.fork()'d chip worker — rank 0 dies during HCCL bring-up, version-invariant #1018 修复后跑 2-rank allreduce_distributed 失败的完整 plog
• 本 pod 的 npu-smi info -t topo 输出（PCIe inter-die routing）
• 把 SIMPLER_ENABLE_PTO_SDMA_WORKSPACE=ON 时复现的 SDMA workspace AICPU
  0x2a → 507018 cascade 完整链（comm_hccl.cpp:36-50 注释里描述的那一条）

[csy0225]

Companion issue (filed jointly): #1036 — Phase 15 single-rank AICore VEC UB-not-aligned (507018). Logically independent but both gate the same step3p5 bring-up effort.