perf(runtime): overlap AICore handshake wakeups; batch the release barrier

docs/aicpu-kernel-launch-mechanisms.md

@@ -192,17 +192,17 @@ dispatched the `Dyn*` exports to the cust subprocess, the inner
 `SchedulerContext::handshake_all_cores` step 1 successfully wrote
 `complete=1` to all 9 cores' `Handshake` slots in HBM. **The host's D2H
 readback confirmed the AICPU's writes landed in HBM**, AICore picked
-them up, ran past its phase 1, and wrote `aicore_regs_ready=1` back
-into the same `Handshake` slots.
+them up, ran past its phase 1, and wrote its report (`aicore_done` +
+`physical_core_id` + `core_type`) back into the same `Handshake` slots.
 
 Step 2 of `handshake_all_cores` is then a spin-wait:
 
 ```cpp
 // src/{arch}/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp
-while (hank->aicore_regs_ready == 0) {}   // ← cust AICPU stuck here forever
+while (hank->aicore_done == 0) {}   // ← cust AICPU stuck here forever
 ```
 
-The cust AICPU's L1 cache holds a stale `0` for `aicore_regs_ready`.
+The cust AICPU's L1 cache holds a stale `0` for `aicore_done`.
 HBM has `1`, the host's D2H readback sees `1`, but the cust AICPU never
 observes the change. After 2 s,
 `aclrtSynchronizeStreamWithTimeout(stream_aicpu_)` reports
@@ -228,7 +228,7 @@ saves a future session from running the same experiment.
 | ------- | ------ | ------------------- |
 | `volatile uint32_t` field qualifier | No effect | Prevents the compiler from caching the value in a register / reusing it across statements. The CPU still reads from L1 (or whatever level holds the stale line). Cache coherency is an architectural issue below the C abstract machine. |
 | `__atomic_load_n(..., __ATOMIC_ACQUIRE)` (compiles to `ldar`) | No effect | `ldar` is an ordering instruction — it orders this load with respect to later loads/stores. It does **not** force a coherent reload from memory or invalidate the L1 line. |
-| `dc civac` (clean + invalidate by VA to Point of Coherency) in spin loop | Worse — corrupts AICore data | The same cache line holds AICPU-written fields (`aicpu_ready`, `task`) and the AICore-written field (`aicore_regs_ready`). `civac` writes back the AICPU's dirty stale view of the line, **clobbering AICore's HBM writes** for the AICPU-owned fields. |
+| `dc civac` (clean + invalidate by VA to Point of Coherency) in spin loop | Worse — corrupts AICore data | The same cache line holds AICPU-written fields (`aicpu_ready`, `task`) and the AICore-written field (`aicore_done`). `civac` writes back the AICPU's dirty stale view of the line, **clobbering AICore's HBM writes** for the AICPU-owned fields. |
 | `dc ivac` (invalidate-only by VA to Point of Coherency) in spin loop | Silently NOP'd | EL0 access to `dc ivac` is gated by `SCTLR_EL1.UCI`. The Linux kernel inside the cust subprocess has `UCI=0`, so the instruction traps and the kernel handler silently turns it into a NOP rather than emulating it. From userspace it looks like the instruction ran with no effect. |
 
 The fix has to live somewhere with the privilege to either change the

docs/hardware/mmio-performance.md

@@ -156,7 +156,7 @@ and the matching producer code in
 LOG_WARN lines like `[P12-A] ... per=5.29 ticks (~105 ns/LDR)` land in
 the AICPU device log at `~/ascend/log/debug/device-<id>/`. Each phase
 is gated by a `hammer_go` value so that the AICore-side handshake is
-explicit; the AICPU-side `scheduler_cold_path::handshake_all_cores` is
+explicit; the AICPU-side `scheduler_cold_path::handshake_partition` is
 the host of the test. Phase 10 and Phase 11 are kept under `#if 0`
 because they hang the chip on entry — see
 `docs/investigations/2026-06-aicore-mmio-to-spr.md` for the verdict.

docs/investigations/2026-06-pa-unroll-207001-optimeout-window.md

@@ -57,7 +57,7 @@ sizing/leak/OOM bug.
   device-side binary load inside `rtKernelLaunchWithHandleV2`
   (`Module::Load` = HBM `DevMemAlloc` + H2D `MemCopySync`).
 - **Spin poll-frequency is NOT the cause** (throttle experiment, inconclusive):
-  the handshake spin on `aicore_regs_ready` is a no-op `SPIN_WAIT_HINT()` =
+  the handshake spin on `aicore_done` is a no-op `SPIN_WAIT_HINT()` =
   a tight GM-hammer loop. Inserting a busy delay between polls gave a
   non-monotonic, within-noise result — median first-launch submit 1.46 s (no
   throttle) → 1.10 s (200k-iter delay) → 1.40 s (4M-iter delay), n=8 each with

docs/troubleshooting/sim-oversubscription-hang.md

@@ -51,7 +51,7 @@ Under that pressure two HW-correct patterns misbehave:
 
 - **Init handshake (livelock).** `SchedulerContext::handshake_all_cores` and
   `aicore_execute` synchronize startup with **bare busy-spins**
-  (`while (hank->aicore_done == 0) {}`, `while (my_hank->aicpu_regs_ready == 0) {}`).
+  (`while (hank->aicore_done == 0) {}`, `while (read_reg(RegId::DATA_MAIN_BASE) == 0) {}`).
   With no yield, the spinning threads monopolize the CPUs and starve the very
   threads they wait on, so the handshake never completes. A `gdb` snapshot of a
   wedged chip subprocess (8× `domain_rank_map` on 2 CPUs) showed the AICPU stuck

src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp

@@ -75,25 +75,29 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in
         SPIN_WAIT_HINT();
     }
 
-    // Phase 2: Report physical core ID, signal ready
+    // Phase 2: Report physical core ID + core type and signal done in one write.
+    // The AICPU opens this core's register window (platform_init_aicore_regs:
+    // FAST_PATH + DATA_MAIN_BASE=IDLE) only after it observes aicore_done, so a
+    // single report suffices — there is no separate aicpu_regs_ready round-trip.
     my_hank->physical_core_id = get_physical_core_id();
-    OUT_OF_ORDER_STORE_BARRIER();
-    my_hank->aicore_regs_ready = 1;
-    dcci(&my_hank->aicore_regs_ready, SINGLE_CACHE_LINE, CACHELINE_OUT);
-    while (my_hank->aicpu_regs_ready == 0) {
-        dcci(&my_hank->aicpu_regs_ready, SINGLE_CACHE_LINE);
-        SPIN_WAIT_HINT();
-    }
-    // Report initial idle status via register
-    write_reg(RegId::COND, AICORE_IDLE_VALUE);
-
-    // Phase 3: Report core type, signal ready
     my_hank->core_type = core_type;
     OUT_OF_ORDER_STORE_BARRIER();
     my_hank->aicore_done = block_idx + 1;  // Signal ready (use block_idx + 1 to avoid 0)
-
     dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT);
 
+    // Phase 3: Wait for the AICPU to open our register window. A kernel launch
+    // resets DATA_MAIN_BASE to 0 (verified on a2a3 silicon); the AICPU writes
+    // DATA_MAIN_BASE = AICPU_IDLE_TASK_ID (non-zero) as it opens FAST_PATH, so a
+    // non-zero read means the window is open and reads/writes are valid. The
+    // AICPU runs assign_cores_to_threads (µs) between opening the window and the
+    // first dispatch, so this IDLE is observed long before any task_id lands —
+    // the poll cannot miss it and mistake a later task for the reset value.
+    while (read_reg(RegId::DATA_MAIN_BASE) == 0) {
+        SPIN_WAIT_HINT();
+    }
+    // Report initial idle status via register (FAST_PATH is now open).
+    write_reg(RegId::COND, AICORE_IDLE_VALUE);
+
     // Cache per-core dispatch payload pointer (set by AICPU before aicpu_ready)
     __gm__ PTO2DispatchPayload *payload = reinterpret_cast<__gm__ PTO2DispatchPayload *>(my_hank->task);
 

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime.h

@@ -89,18 +89,18 @@ constexpr int RUNTIME_DEFAULT_READY_QUEUE_SHARDS = PLATFORM_MAX_AICPU_THREADS -
  *
  * Field Access Patterns:
  * - aicpu_ready: Written by AICPU, read by AICore
- * - aicore_done: Written by AICore, read by AICPU
+ * - aicore_done: Written by AICore, read by AICPU (final report; physical_core_id
+ *   and core_type are published alongside it in the same write)
  * - task: Written by AICPU, read by AICore (0 = not ready, non-zero = PTO2DispatchPayload*)
- * - core_type: Written by AICPU, read by AICore (CoreType::AIC or CoreType::AIV)
+ * - core_type: Written by AICore (with aicore_done), read by AICPU (CoreType::AIC or CoreType::AIV)
+ * - physical_core_id: Written by AICore (with aicore_done), read by AICPU
  */
 struct Handshake {
-    volatile uint32_t aicpu_ready;        // AICPU ready signal: 0=not ready, 1=ready
-    volatile uint32_t aicore_done;        // AICore ready signal: 0=not ready, core_id+1=ready
-    volatile uint64_t task;               // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused
-    volatile CoreType core_type;          // Core type: CoreType::AIC or CoreType::AIV
-    volatile uint32_t physical_core_id;   // Physical core ID
-    volatile uint32_t aicpu_regs_ready;   // AICPU register init done: 0=pending, 1=done
-    volatile uint32_t aicore_regs_ready;  // AICore ID reported: 0=pending, 1=done
+    volatile uint32_t aicpu_ready;  // AICPU ready signal: 0=not ready, 1=ready
+    volatile uint32_t aicore_done;  // AICore ready signal: 0=not ready, core_id+1=ready
+    volatile uint64_t task;         // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused
+    volatile CoreType core_type;    // Core type: CoreType::AIC or CoreType::AIV (reported by AICore with aicore_done)
+    volatile uint32_t physical_core_id;  // Physical core ID (reported by AICore with aicore_done)
 } __attribute__((aligned(64)));
 
 enum class TensorReleaseKind {

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp

@@ -672,76 +672,97 @@ int32_t SchedulerContext::shutdown(int32_t thread_idx) {
 // Handshake a contiguous slice of AICore workers. Runs on every AICPU thread in
 // parallel (partitioned by tidx/nthreads); the leader's pre_handshake_init has
 // already zeroed state, set cores_total_num_, and reset the counts/flag. The
-// serial per-core MMIO here is what dominates preamble, so splitting the slice
-// across threads is the whole point. Worker-id lists are built serially in
-// post_handshake_init (core-index order) once every slice has landed.
+// per-core work here — releasing the core, then opening its register window over
+// serial MMIO — is what dominates preamble, so splitting the slice across
+// threads is the whole point. Within a slice we still sweep (poll every
+// outstanding core per pass, service whichever reported) so one slow core's
+// wakeup overlaps its neighbours' instead of blocking them. Worker-id lists are
+// built serially in post_handshake_init (core-index order) once every slice has
+// landed, so the shared aic_count_/aiv_count_ are written by one thread only.
 // =============================================================================
 void SchedulerContext::handshake_partition(Runtime *runtime, int32_t tidx, int32_t nthreads) {
     Handshake *all_handshakes = reinterpret_cast<Handshake *>(runtime->dev.workers);
     const int32_t total = cores_total_num_;
     const int32_t lo = static_cast<int32_t>((static_cast<int64_t>(tidx) * total) / nthreads);
     const int32_t hi = static_cast<int32_t>((static_cast<int64_t>(tidx + 1) * total) / nthreads);
 
-    // Step 1: Write per-core payload addresses and send handshake signal.
-    // OUT_OF_ORDER_STORE_BARRIER() ensures task is globally visible before
-    // aicpu_ready=1, so AICore reads the correct payload pointer after waking up.
+    // Step 1: publish this slice's payload pointers, then release its cores. One
+    // barrier (not one per core) suffices: it guarantees every task store is
+    // globally visible before any aicpu_ready store, which is the only ordering
+    // AICore relies on (it reads task only after observing aicpu_ready==1). Each
+    // core is written by exactly one thread, so the slices never overlap.
     for (int32_t i = lo; i < hi; i++) {
         all_handshakes[i].task = reinterpret_cast<uint64_t>(&payload_per_core_[i][0]);
-        OUT_OF_ORDER_STORE_BARRIER();
+    }
+    OUT_OF_ORDER_STORE_BARRIER();
+    for (int32_t i = lo; i < hi; i++) {
         all_handshakes[i].aicpu_ready = 1;
     }
     OUT_OF_ORDER_STORE_BARRIER();
 
     // Get platform physical cores count for validation
     uint32_t max_physical_cores_count = platform_get_physical_cores_count();
 
-    // Step 2: Wait for this slice's cores, init registers, collect state.
-    for (int32_t i = lo; i < hi; i++) {
-        Handshake *hank = &all_handshakes[i];
-
-        while (hank->aicore_regs_ready == 0) {
-            SPIN_WAIT_HINT();
-        }
-
-        uint32_t physical_core_id = hank->physical_core_id;
-
-        if (physical_core_id >= max_physical_cores_count) {
-            LOG_ERROR(
-                "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id,
-                max_physical_cores_count
-            );
-            handshake_failed_.store(true, std::memory_order_release);
-            continue;
-        }
-
-        uint64_t *regs = reinterpret_cast<uint64_t *>(regs_);
-        uint64_t reg_addr = regs[physical_core_id];
-
-        // Initialize AICore registers after discovery (first round)
-        platform_init_aicore_regs(reg_addr);
-        OUT_OF_ORDER_STORE_BARRIER();
-        hank->aicpu_regs_ready = 1;
-
-        OUT_OF_ORDER_STORE_BARRIER();
+    // Step 2: collect responses from this slice. Each core reports
+    // {physical_core_id, core_type, aicore_done} in one write, then waits — by
+    // polling its own DATA_MAIN_BASE SPR — for us to open its register window.
+    // We sweep the slice: poll every outstanding core per pass and service
+    // whichever have reported, rather than blocking on core i before looking at
+    // core i+1, so per-core wakeups overlap (≈ max, not Σ). aicore_done is a GM
+    // read (not the nGnRE MMIO reg window), so sweeping is not forced serial the
+    // way RegId::COND polling is.
+    //
+    // Servicing a core = validate its physical_core_id, then open its register
+    // window (platform_init_aicore_regs: FAST_PATH + DATA_MAIN_BASE=IDLE). That
+    // IDLE write is *also* the signal the core polls for to leave its
+    // post-report wait — so opening the window IS the acknowledgement. There is
+    // no separate aicpu_regs_ready ack and no second round-trip. AIC/AIV
+    // classification is deferred to post_handshake_init (serial) so aic_count_/
+    // aiv_count_ are never incremented from more than one thread.
+    uint64_t *regs = reinterpret_cast<uint64_t *>(regs_);
+    bool core_serviced[RUNTIME_MAX_WORKER] = {false};
+
+    for (int32_t remaining = hi - lo; remaining > 0;) {
+        for (int32_t i = lo; i < hi; i++) {
+            if (core_serviced[i]) continue;
+            Handshake *hank = &all_handshakes[i];
+            if (hank->aicore_done == 0) {
+                SPIN_WAIT_HINT();
+                continue;
+            }
 
-        while (hank->aicore_done == 0) {
-            SPIN_WAIT_HINT();
-        }
+            uint32_t physical_core_id = hank->physical_core_id;
+            if (physical_core_id >= max_physical_cores_count) {
+                LOG_ERROR(
+                    "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id,
+                    max_physical_cores_count
+                );
+                handshake_failed_.store(true, std::memory_order_release);
+                core_serviced[i] = true;
+                remaining--;
+                continue;
+            }
 
-        core_exec_states_[i].reg_addr = reg_addr;
-        core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND);
+            // Open the window; the IDLE write releases the core's DATA_MAIN_BASE poll.
+            uint64_t reg_addr = regs[physical_core_id];
+            platform_init_aicore_regs(reg_addr);
+            OUT_OF_ORDER_STORE_BARRIER();
 
+            core_exec_states_[i].reg_addr = reg_addr;
+            core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND);
 #if PTO2_PROFILING
-        // Record physical_core_id for PMU init later (CoreExecState has no room
-        // for this field under PTO2_PROFILING).
-        physical_core_ids_[i] = physical_core_id;
+            physical_core_ids_[i] = physical_core_id;
 #endif
 #if !PTO2_PROFILING
-        core_exec_states_[i].worker_id = i;
-        core_exec_states_[i].physical_core_id = physical_core_id;
-        core_exec_states_[i].core_type = hank->core_type;
+            core_exec_states_[i].worker_id = i;
+            core_exec_states_[i].physical_core_id = physical_core_id;
+            core_exec_states_[i].core_type = hank->core_type;
 #endif
+            core_serviced[i] = true;
+            remaining--;
+        }
     }
+    OUT_OF_ORDER_STORE_BARRIER();
 }
 
 // =============================================================================
@@ -813,13 +834,14 @@ bool SchedulerContext::assign_cores_to_threads() {
 // deinit their AICore register blocks. Idempotent.
 // =============================================================================
 void SchedulerContext::emergency_shutdown(Runtime *runtime) {
+    (void)runtime;  // exit is now delivered via each core's register block, not GM
     LOG_WARN("Emergency shutdown: sending exit signal to all initialized cores");
-    Handshake *all_handshakes = reinterpret_cast<Handshake *>(runtime->dev.workers);
     int32_t timeout_count = 0;
     for (int32_t i = 0; i < cores_total_num_; i++) {
-        Handshake *hank = &all_handshakes[i];
-        OUT_OF_ORDER_STORE_BARRIER();
-        hank->aicpu_regs_ready = 1;
+        // platform_deinit_aicore_regs writes DATA_MAIN_BASE=EXIT, which both
+        // releases a core still polling for its window to open and signals it to
+        // exit. Cores never opened (reg_addr==0) are reaped by the host device
+        // reset that follows a handshake failure.
         if (core_exec_states_[i].reg_addr != 0) {
             if (platform_deinit_aicore_regs(core_exec_states_[i].reg_addr) != 0) {
                 timeout_count++;

src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp

@@ -71,25 +71,30 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in
         SPIN_WAIT_HINT();
     }
 
-    // Phase 2: Report physical core ID, signal ready
+    // Phase 2: Report physical core ID + core type and signal done in one write.
+    // The AICPU opens this core's register window (platform_init_aicore_regs:
+    // FAST_PATH + DATA_MAIN_BASE=IDLE) only after it observes aicore_done, so a
+    // single report suffices — there is no separate aicpu_regs_ready round-trip.
     my_hank->physical_core_id = get_physical_core_id();
-    OUT_OF_ORDER_STORE_BARRIER();
-    my_hank->aicore_regs_ready = 1;
-    dcci(&my_hank->aicore_regs_ready, SINGLE_CACHE_LINE, CACHELINE_OUT);
-    while (my_hank->aicpu_regs_ready == 0) {
-        dcci(&my_hank->aicpu_regs_ready, SINGLE_CACHE_LINE);
-        SPIN_WAIT_HINT();
-    }
-    // Report initial idle status via register
-    write_reg(RegId::COND, AICORE_IDLE_VALUE);
-
-    // Phase 3: Report core type, signal ready
     my_hank->core_type = core_type;
     OUT_OF_ORDER_STORE_BARRIER();
     my_hank->aicore_done = s_block_idx + 1;  // Signal ready (use s_block_idx + 1 to avoid 0)
-
     dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT);
 
+    // Phase 3: Wait for the AICPU to open our register window. A kernel launch
+    // resets DATA_MAIN_BASE to 0 (verified on a2a3 silicon; a5 shares this
+    // register protocol and relies on CI); the AICPU writes DATA_MAIN_BASE =
+    // AICPU_IDLE_TASK_ID (non-zero) as it opens FAST_PATH, so a non-zero read
+    // means the window is open and reads/writes are valid. The AICPU runs
+    // assign_cores_to_threads (µs) between opening the window and the first
+    // dispatch, so this IDLE is observed long before any task_id lands — the
+    // poll cannot miss it and mistake a later task for the reset value.
+    while (read_reg(RegId::DATA_MAIN_BASE) == 0) {
+        SPIN_WAIT_HINT();
+    }
+    // Report initial idle status via register (FAST_PATH is now open).
+    write_reg(RegId::COND, AICORE_IDLE_VALUE);
+
     // Cache per-core dispatch payload pointer (set by AICPU before aicpu_ready)
     __gm__ PTO2DispatchPayload *payload = reinterpret_cast<__gm__ PTO2DispatchPayload *>(my_hank->task);