From 8271c311249bc8c6a132763d581f7a245ff5f9ae Mon Sep 17 00:00:00 2001 From: ChaoZheng109 Date: Thu, 9 Jul 2026 01:36:34 -0700 Subject: [PATCH] perf(runtime): single-round-trip AICore handshake; parallelize a5 Collapse the per-core AICPU<->AICore handshake to a single round-trip on a2a3 and a5, and bring a5's handshake up to the parallel multi-thread structure a2a3 already carries from #1279. Single round-trip (both arches): - AICore reports {physical_core_id, core_type, aicore_done} in one write, then polls its own DATA_MAIN_BASE SPR for the AICPU to open its register window. A kernel launch resets that SPR to 0 (verified on a2a3 silicon); the AICPU writes IDLE (non-zero) as it opens FAST_PATH, so a non-zero read means the window is open. Opening the window IS the acknowledgement -- the aicpu_regs_ready / aicore_regs_ready Handshake fields and the second round-trip are removed. - The AICPU sweeps its core slice (poll aicore_done, service whoever has reported) instead of blocking core-by-core, so per-core wakeups overlap. Parallelize a5: - a5 previously handshaked all cores serially on one AICPU thread while the others spun idle. Split SchedulerContext::init into pre_handshake_init (leader) / handshake_partition (all threads, each a disjoint core slice) / post_handshake_init (leader, after a barrier), mirroring a2a3. AicpuExecutor::init orchestrates the barrier and hands sim threads a distinct index when the affinity gate is inactive. - Worker-id list building is deferred to post_handshake_init (serial) so aic_count_/aiv_count_ are never incremented from more than one thread. handshake_partition and AicpuExecutor::init are now identical across a2a3 and a5. Builds clean on a2a3/a5 (onboard + sim); a2a3sim/a5sim scene tests pass. a2a3/a5 onboard validation relies on CI (no NPU on this box). --- docs/aicpu-kernel-launch-mechanisms.md | 10 +- docs/hardware/mmio-performance.md | 2 +- ...26-06-pa-unroll-207001-optimeout-window.md | 2 +- .../sim-oversubscription-hang.md | 2 +- .../aicore/aicore_executor.cpp | 30 +-- .../runtime/runtime.h | 18 +- .../runtime/scheduler/scheduler_cold_path.cpp | 122 ++++++---- .../aicore/aicore_executor.cpp | 31 +-- .../aicpu/aicpu_executor.cpp | 128 +++++++--- .../docs/RUNTIME_LOGIC.md | 2 +- .../runtime/runtime.h | 20 +- .../runtime/scheduler/scheduler_cold_path.cpp | 221 ++++++++++-------- .../runtime/scheduler/scheduler_context.h | 36 ++- 13 files changed, 381 insertions(+), 243 deletions(-) diff --git a/docs/aicpu-kernel-launch-mechanisms.md b/docs/aicpu-kernel-launch-mechanisms.md index ca407e94c..275ace90f 100644 --- a/docs/aicpu-kernel-launch-mechanisms.md +++ b/docs/aicpu-kernel-launch-mechanisms.md @@ -192,17 +192,17 @@ dispatched the `Dyn*` exports to the cust subprocess, the inner `SchedulerContext::handshake_all_cores` step 1 successfully wrote `complete=1` to all 9 cores' `Handshake` slots in HBM. **The host's D2H readback confirmed the AICPU's writes landed in HBM**, AICore picked -them up, ran past its phase 1, and wrote `aicore_regs_ready=1` back -into the same `Handshake` slots. +them up, ran past its phase 1, and wrote its report (`aicore_done` + +`physical_core_id` + `core_type`) back into the same `Handshake` slots. Step 2 of `handshake_all_cores` is then a spin-wait: ```cpp // src/{arch}/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp -while (hank->aicore_regs_ready == 0) {} // ← cust AICPU stuck here forever +while (hank->aicore_done == 0) {} // ← cust AICPU stuck here forever ``` -The cust AICPU's L1 cache holds a stale `0` for `aicore_regs_ready`. +The cust AICPU's L1 cache holds a stale `0` for `aicore_done`. HBM has `1`, the host's D2H readback sees `1`, but the cust AICPU never observes the change. After 2 s, `aclrtSynchronizeStreamWithTimeout(stream_aicpu_)` reports @@ -228,7 +228,7 @@ saves a future session from running the same experiment. | ------- | ------ | ------------------- | | `volatile uint32_t` field qualifier | No effect | Prevents the compiler from caching the value in a register / reusing it across statements. The CPU still reads from L1 (or whatever level holds the stale line). Cache coherency is an architectural issue below the C abstract machine. | | `__atomic_load_n(..., __ATOMIC_ACQUIRE)` (compiles to `ldar`) | No effect | `ldar` is an ordering instruction — it orders this load with respect to later loads/stores. It does **not** force a coherent reload from memory or invalidate the L1 line. | -| `dc civac` (clean + invalidate by VA to Point of Coherency) in spin loop | Worse — corrupts AICore data | The same cache line holds AICPU-written fields (`aicpu_ready`, `task`) and the AICore-written field (`aicore_regs_ready`). `civac` writes back the AICPU's dirty stale view of the line, **clobbering AICore's HBM writes** for the AICPU-owned fields. | +| `dc civac` (clean + invalidate by VA to Point of Coherency) in spin loop | Worse — corrupts AICore data | The same cache line holds AICPU-written fields (`aicpu_ready`, `task`) and the AICore-written field (`aicore_done`). `civac` writes back the AICPU's dirty stale view of the line, **clobbering AICore's HBM writes** for the AICPU-owned fields. | | `dc ivac` (invalidate-only by VA to Point of Coherency) in spin loop | Silently NOP'd | EL0 access to `dc ivac` is gated by `SCTLR_EL1.UCI`. The Linux kernel inside the cust subprocess has `UCI=0`, so the instruction traps and the kernel handler silently turns it into a NOP rather than emulating it. From userspace it looks like the instruction ran with no effect. | The fix has to live somewhere with the privilege to either change the diff --git a/docs/hardware/mmio-performance.md b/docs/hardware/mmio-performance.md index 903e09f56..4b3474feb 100644 --- a/docs/hardware/mmio-performance.md +++ b/docs/hardware/mmio-performance.md @@ -156,7 +156,7 @@ and the matching producer code in LOG_WARN lines like `[P12-A] ... per=5.29 ticks (~105 ns/LDR)` land in the AICPU device log at `~/ascend/log/debug/device-/`. Each phase is gated by a `hammer_go` value so that the AICore-side handshake is -explicit; the AICPU-side `scheduler_cold_path::handshake_all_cores` is +explicit; the AICPU-side `scheduler_cold_path::handshake_partition` is the host of the test. Phase 10 and Phase 11 are kept under `#if 0` because they hang the chip on entry — see `docs/investigations/2026-06-aicore-mmio-to-spr.md` for the verdict. diff --git a/docs/investigations/2026-06-pa-unroll-207001-optimeout-window.md b/docs/investigations/2026-06-pa-unroll-207001-optimeout-window.md index 29810db47..8a9905f5c 100644 --- a/docs/investigations/2026-06-pa-unroll-207001-optimeout-window.md +++ b/docs/investigations/2026-06-pa-unroll-207001-optimeout-window.md @@ -57,7 +57,7 @@ sizing/leak/OOM bug. device-side binary load inside `rtKernelLaunchWithHandleV2` (`Module::Load` = HBM `DevMemAlloc` + H2D `MemCopySync`). - **Spin poll-frequency is NOT the cause** (throttle experiment, inconclusive): - the handshake spin on `aicore_regs_ready` is a no-op `SPIN_WAIT_HINT()` = + the handshake spin on `aicore_done` is a no-op `SPIN_WAIT_HINT()` = a tight GM-hammer loop. Inserting a busy delay between polls gave a non-monotonic, within-noise result — median first-launch submit 1.46 s (no throttle) → 1.10 s (200k-iter delay) → 1.40 s (4M-iter delay), n=8 each with diff --git a/docs/troubleshooting/sim-oversubscription-hang.md b/docs/troubleshooting/sim-oversubscription-hang.md index 7598ee95d..b11cb1393 100644 --- a/docs/troubleshooting/sim-oversubscription-hang.md +++ b/docs/troubleshooting/sim-oversubscription-hang.md @@ -51,7 +51,7 @@ Under that pressure two HW-correct patterns misbehave: - **Init handshake (livelock).** `SchedulerContext::handshake_all_cores` and `aicore_execute` synchronize startup with **bare busy-spins** - (`while (hank->aicore_done == 0) {}`, `while (my_hank->aicpu_regs_ready == 0) {}`). + (`while (hank->aicore_done == 0) {}`, `while (read_reg(RegId::DATA_MAIN_BASE) == 0) {}`). With no yield, the spinning threads monopolize the CPUs and starve the very threads they wait on, so the handshake never completes. A `gdb` snapshot of a wedged chip subprocess (8× `domain_rank_map` on 2 CPUs) showed the AICPU stuck diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp index 7f0cce021..cae2a17b2 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp @@ -75,25 +75,29 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in SPIN_WAIT_HINT(); } - // Phase 2: Report physical core ID, signal ready + // Phase 2: Report physical core ID + core type and signal done in one write. + // The AICPU opens this core's register window (platform_init_aicore_regs: + // FAST_PATH + DATA_MAIN_BASE=IDLE) only after it observes aicore_done, so a + // single report suffices — there is no separate aicpu_regs_ready round-trip. my_hank->physical_core_id = get_physical_core_id(); - OUT_OF_ORDER_STORE_BARRIER(); - my_hank->aicore_regs_ready = 1; - dcci(&my_hank->aicore_regs_ready, SINGLE_CACHE_LINE, CACHELINE_OUT); - while (my_hank->aicpu_regs_ready == 0) { - dcci(&my_hank->aicpu_regs_ready, SINGLE_CACHE_LINE); - SPIN_WAIT_HINT(); - } - // Report initial idle status via register - write_reg(RegId::COND, AICORE_IDLE_VALUE); - - // Phase 3: Report core type, signal ready my_hank->core_type = core_type; OUT_OF_ORDER_STORE_BARRIER(); my_hank->aicore_done = block_idx + 1; // Signal ready (use block_idx + 1 to avoid 0) - dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT); + // Phase 3: Wait for the AICPU to open our register window. A kernel launch + // resets DATA_MAIN_BASE to 0 (verified on a2a3 silicon); the AICPU writes + // DATA_MAIN_BASE = AICPU_IDLE_TASK_ID (non-zero) as it opens FAST_PATH, so a + // non-zero read means the window is open and reads/writes are valid. The + // AICPU runs assign_cores_to_threads (µs) between opening the window and the + // first dispatch, so this IDLE is observed long before any task_id lands — + // the poll cannot miss it and mistake a later task for the reset value. + while (read_reg(RegId::DATA_MAIN_BASE) == 0) { + SPIN_WAIT_HINT(); + } + // Report initial idle status via register (FAST_PATH is now open). + write_reg(RegId::COND, AICORE_IDLE_VALUE); + // Cache per-core dispatch payload pointer (set by AICPU before aicpu_ready) __gm__ PTO2DispatchPayload *payload = reinterpret_cast<__gm__ PTO2DispatchPayload *>(my_hank->task); diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime.h index 3f6b6dee0..76742b3c6 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime.h @@ -89,18 +89,18 @@ constexpr int RUNTIME_DEFAULT_READY_QUEUE_SHARDS = PLATFORM_MAX_AICPU_THREADS - * * Field Access Patterns: * - aicpu_ready: Written by AICPU, read by AICore - * - aicore_done: Written by AICore, read by AICPU + * - aicore_done: Written by AICore, read by AICPU (final report; physical_core_id + * and core_type are published alongside it in the same write) * - task: Written by AICPU, read by AICore (0 = not ready, non-zero = PTO2DispatchPayload*) - * - core_type: Written by AICPU, read by AICore (CoreType::AIC or CoreType::AIV) + * - core_type: Written by AICore (with aicore_done), read by AICPU (CoreType::AIC or CoreType::AIV) + * - physical_core_id: Written by AICore (with aicore_done), read by AICPU */ struct Handshake { - volatile uint32_t aicpu_ready; // AICPU ready signal: 0=not ready, 1=ready - volatile uint32_t aicore_done; // AICore ready signal: 0=not ready, core_id+1=ready - volatile uint64_t task; // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused - volatile CoreType core_type; // Core type: CoreType::AIC or CoreType::AIV - volatile uint32_t physical_core_id; // Physical core ID - volatile uint32_t aicpu_regs_ready; // AICPU register init done: 0=pending, 1=done - volatile uint32_t aicore_regs_ready; // AICore ID reported: 0=pending, 1=done + volatile uint32_t aicpu_ready; // AICPU ready signal: 0=not ready, 1=ready + volatile uint32_t aicore_done; // AICore ready signal: 0=not ready, core_id+1=ready + volatile uint64_t task; // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused + volatile CoreType core_type; // Core type: CoreType::AIC or CoreType::AIV (reported by AICore with aicore_done) + volatile uint32_t physical_core_id; // Physical core ID (reported by AICore with aicore_done) } __attribute__((aligned(64))); enum class TensorReleaseKind { diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp index 8891b9075..47ea6dad8 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp @@ -672,9 +672,13 @@ int32_t SchedulerContext::shutdown(int32_t thread_idx) { // Handshake a contiguous slice of AICore workers. Runs on every AICPU thread in // parallel (partitioned by tidx/nthreads); the leader's pre_handshake_init has // already zeroed state, set cores_total_num_, and reset the counts/flag. The -// serial per-core MMIO here is what dominates preamble, so splitting the slice -// across threads is the whole point. Worker-id lists are built serially in -// post_handshake_init (core-index order) once every slice has landed. +// per-core work here — releasing the core, then opening its register window over +// serial MMIO — is what dominates preamble, so splitting the slice across +// threads is the whole point. Within a slice we still sweep (poll every +// outstanding core per pass, service whichever reported) so one slow core's +// wakeup overlaps its neighbours' instead of blocking them. Worker-id lists are +// built serially in post_handshake_init (core-index order) once every slice has +// landed, so the shared aic_count_/aiv_count_ are written by one thread only. // ============================================================================= void SchedulerContext::handshake_partition(Runtime *runtime, int32_t tidx, int32_t nthreads) { Handshake *all_handshakes = reinterpret_cast(runtime->dev.workers); @@ -682,12 +686,16 @@ void SchedulerContext::handshake_partition(Runtime *runtime, int32_t tidx, int32 const int32_t lo = static_cast((static_cast(tidx) * total) / nthreads); const int32_t hi = static_cast((static_cast(tidx + 1) * total) / nthreads); - // Step 1: Write per-core payload addresses and send handshake signal. - // OUT_OF_ORDER_STORE_BARRIER() ensures task is globally visible before - // aicpu_ready=1, so AICore reads the correct payload pointer after waking up. + // Step 1: publish this slice's payload pointers, then release its cores. One + // barrier (not one per core) suffices: it guarantees every task store is + // globally visible before any aicpu_ready store, which is the only ordering + // AICore relies on (it reads task only after observing aicpu_ready==1). Each + // core is written by exactly one thread, so the slices never overlap. for (int32_t i = lo; i < hi; i++) { all_handshakes[i].task = reinterpret_cast(&payload_per_core_[i][0]); - OUT_OF_ORDER_STORE_BARRIER(); + } + OUT_OF_ORDER_STORE_BARRIER(); + for (int32_t i = lo; i < hi; i++) { all_handshakes[i].aicpu_ready = 1; } OUT_OF_ORDER_STORE_BARRIER(); @@ -695,53 +703,66 @@ void SchedulerContext::handshake_partition(Runtime *runtime, int32_t tidx, int32 // Get platform physical cores count for validation uint32_t max_physical_cores_count = platform_get_physical_cores_count(); - // Step 2: Wait for this slice's cores, init registers, collect state. - for (int32_t i = lo; i < hi; i++) { - Handshake *hank = &all_handshakes[i]; - - while (hank->aicore_regs_ready == 0) { - SPIN_WAIT_HINT(); - } - - uint32_t physical_core_id = hank->physical_core_id; - - if (physical_core_id >= max_physical_cores_count) { - LOG_ERROR( - "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id, - max_physical_cores_count - ); - handshake_failed_.store(true, std::memory_order_release); - continue; - } - - uint64_t *regs = reinterpret_cast(regs_); - uint64_t reg_addr = regs[physical_core_id]; - - // Initialize AICore registers after discovery (first round) - platform_init_aicore_regs(reg_addr); - OUT_OF_ORDER_STORE_BARRIER(); - hank->aicpu_regs_ready = 1; - - OUT_OF_ORDER_STORE_BARRIER(); + // Step 2: collect responses from this slice. Each core reports + // {physical_core_id, core_type, aicore_done} in one write, then waits — by + // polling its own DATA_MAIN_BASE SPR — for us to open its register window. + // We sweep the slice: poll every outstanding core per pass and service + // whichever have reported, rather than blocking on core i before looking at + // core i+1, so per-core wakeups overlap (≈ max, not Σ). aicore_done is a GM + // read (not the nGnRE MMIO reg window), so sweeping is not forced serial the + // way RegId::COND polling is. + // + // Servicing a core = validate its physical_core_id, then open its register + // window (platform_init_aicore_regs: FAST_PATH + DATA_MAIN_BASE=IDLE). That + // IDLE write is *also* the signal the core polls for to leave its + // post-report wait — so opening the window IS the acknowledgement. There is + // no separate aicpu_regs_ready ack and no second round-trip. AIC/AIV + // classification is deferred to post_handshake_init (serial) so aic_count_/ + // aiv_count_ are never incremented from more than one thread. + uint64_t *regs = reinterpret_cast(regs_); + bool core_serviced[RUNTIME_MAX_WORKER] = {false}; + + for (int32_t remaining = hi - lo; remaining > 0;) { + for (int32_t i = lo; i < hi; i++) { + if (core_serviced[i]) continue; + Handshake *hank = &all_handshakes[i]; + if (hank->aicore_done == 0) { + SPIN_WAIT_HINT(); + continue; + } - while (hank->aicore_done == 0) { - SPIN_WAIT_HINT(); - } + uint32_t physical_core_id = hank->physical_core_id; + if (physical_core_id >= max_physical_cores_count) { + LOG_ERROR( + "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id, + max_physical_cores_count + ); + handshake_failed_.store(true, std::memory_order_release); + core_serviced[i] = true; + remaining--; + continue; + } - core_exec_states_[i].reg_addr = reg_addr; - core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND); + // Open the window; the IDLE write releases the core's DATA_MAIN_BASE poll. + uint64_t reg_addr = regs[physical_core_id]; + platform_init_aicore_regs(reg_addr); + OUT_OF_ORDER_STORE_BARRIER(); + core_exec_states_[i].reg_addr = reg_addr; + core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND); #if PTO2_PROFILING - // Record physical_core_id for PMU init later (CoreExecState has no room - // for this field under PTO2_PROFILING). - physical_core_ids_[i] = physical_core_id; + physical_core_ids_[i] = physical_core_id; #endif #if !PTO2_PROFILING - core_exec_states_[i].worker_id = i; - core_exec_states_[i].physical_core_id = physical_core_id; - core_exec_states_[i].core_type = hank->core_type; + core_exec_states_[i].worker_id = i; + core_exec_states_[i].physical_core_id = physical_core_id; + core_exec_states_[i].core_type = hank->core_type; #endif + core_serviced[i] = true; + remaining--; + } } + OUT_OF_ORDER_STORE_BARRIER(); } // ============================================================================= @@ -813,13 +834,14 @@ bool SchedulerContext::assign_cores_to_threads() { // deinit their AICore register blocks. Idempotent. // ============================================================================= void SchedulerContext::emergency_shutdown(Runtime *runtime) { + (void)runtime; // exit is now delivered via each core's register block, not GM LOG_WARN("Emergency shutdown: sending exit signal to all initialized cores"); - Handshake *all_handshakes = reinterpret_cast(runtime->dev.workers); int32_t timeout_count = 0; for (int32_t i = 0; i < cores_total_num_; i++) { - Handshake *hank = &all_handshakes[i]; - OUT_OF_ORDER_STORE_BARRIER(); - hank->aicpu_regs_ready = 1; + // platform_deinit_aicore_regs writes DATA_MAIN_BASE=EXIT, which both + // releases a core still polling for its window to open and signals it to + // exit. Cores never opened (reg_addr==0) are reaped by the host device + // reset that follows a handshake failure. if (core_exec_states_[i].reg_addr != 0) { if (platform_deinit_aicore_regs(core_exec_states_[i].reg_addr) != 0) { timeout_count++; diff --git a/src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp b/src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp index eefca4fde..5eb68ddaf 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp @@ -71,25 +71,30 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in SPIN_WAIT_HINT(); } - // Phase 2: Report physical core ID, signal ready + // Phase 2: Report physical core ID + core type and signal done in one write. + // The AICPU opens this core's register window (platform_init_aicore_regs: + // FAST_PATH + DATA_MAIN_BASE=IDLE) only after it observes aicore_done, so a + // single report suffices — there is no separate aicpu_regs_ready round-trip. my_hank->physical_core_id = get_physical_core_id(); - OUT_OF_ORDER_STORE_BARRIER(); - my_hank->aicore_regs_ready = 1; - dcci(&my_hank->aicore_regs_ready, SINGLE_CACHE_LINE, CACHELINE_OUT); - while (my_hank->aicpu_regs_ready == 0) { - dcci(&my_hank->aicpu_regs_ready, SINGLE_CACHE_LINE); - SPIN_WAIT_HINT(); - } - // Report initial idle status via register - write_reg(RegId::COND, AICORE_IDLE_VALUE); - - // Phase 3: Report core type, signal ready my_hank->core_type = core_type; OUT_OF_ORDER_STORE_BARRIER(); my_hank->aicore_done = s_block_idx + 1; // Signal ready (use s_block_idx + 1 to avoid 0) - dcci(my_hank, SINGLE_CACHE_LINE, CACHELINE_OUT); + // Phase 3: Wait for the AICPU to open our register window. A kernel launch + // resets DATA_MAIN_BASE to 0 (verified on a2a3 silicon; a5 shares this + // register protocol and relies on CI); the AICPU writes DATA_MAIN_BASE = + // AICPU_IDLE_TASK_ID (non-zero) as it opens FAST_PATH, so a non-zero read + // means the window is open and reads/writes are valid. The AICPU runs + // assign_cores_to_threads (µs) between opening the window and the first + // dispatch, so this IDLE is observed long before any task_id lands — the + // poll cannot miss it and mistake a later task for the reset value. + while (read_reg(RegId::DATA_MAIN_BASE) == 0) { + SPIN_WAIT_HINT(); + } + // Report initial idle status via register (FAST_PATH is now open). + write_reg(RegId::COND, AICORE_IDLE_VALUE); + // Cache per-core dispatch payload pointer (set by AICPU before aicpu_ready) __gm__ PTO2DispatchPayload *payload = reinterpret_cast<__gm__ PTO2DispatchPayload *>(my_hank->task); diff --git a/src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp b/src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp index 007790be2..aa7198496 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp @@ -119,11 +119,20 @@ struct AicpuExecutor { // ===== Thread management state ===== std::atomic thread_idx_{0}; - std::atomic initialized_{false}; std::atomic init_done_{false}; std::atomic init_failed_{false}; std::atomic finished_{false}; + // Parallel-handshake coordination (see AicpuExecutor::init). hs_setup_done_ + // is published by the leader once the shared pre-handshake setup is visible; + // hs_arrived_ is the barrier counting threads that finished their core slice. + // hs_thread_seq_ hands out a distinct [0, nthreads) index when the platform + // exposes no affinity idx (sim, where platform_aicpu_affinity_thread_idx() + // is -1 during init) so the threads don't all collapse to leader 0. + std::atomic hs_setup_done_{false}; + std::atomic hs_arrived_{0}; + std::atomic hs_thread_seq_{0}; + int32_t aicpu_thread_num_{0}; // ===== Task queue state (managed by scheduler ready queues) ===== @@ -189,42 +198,88 @@ static_assert( // ===== AicpuExecutor Method Implementations ===== int32_t AicpuExecutor::init(Runtime *runtime) { - bool expected = false; - if (!initialized_.compare_exchange_strong(expected, true, std::memory_order_acq_rel, std::memory_order_acquire)) { - return 0; - } - - LOG_INFO_V0("AicpuExecutor: Initializing"); - if (runtime == nullptr) { LOG_ERROR("runtime is nullptr"); init_failed_.store(true, std::memory_order_release); return -1; } - // Read execution parameters from runtime. The 0 → 1 fixup runs before the - // sched_thread_num_ derivation so a zero input doesn't leave the scheduler - // count at -1. - aicpu_thread_num_ = runtime->dev.aicpu_thread_num; - if (aicpu_thread_num_ == 0) aicpu_thread_num_ = 1; - sched_thread_num_ = aicpu_thread_num_ - 1; - serial_orch_sched_ = runtime->dev.serial_orch_sched; - - if (aicpu_thread_num_ < 1 || aicpu_thread_num_ > MAX_AICPU_THREADS) { - LOG_ERROR("Invalid aicpu_thread_num: %d", aicpu_thread_num_); + // All AICPU threads enter init. The per-core AICore handshake is the + // dominant preamble cost (serial MMIO, ~217 µs of ~283 µs for 72 cores), so + // it is parallelized: the leader (tidx 0) does the shared setup, every + // thread handshakes a disjoint slice of cores, then the leader finishes init + // after a barrier. Non-leaders spin on init_done_. + int32_t nthreads = runtime->dev.aicpu_thread_num; + if (nthreads == 0) nthreads = 1; + if (nthreads < 1 || nthreads > MAX_AICPU_THREADS) { + LOG_ERROR("Invalid aicpu_thread_num: %d", nthreads); init_failed_.store(true, std::memory_order_release); return -1; } - - if (sched_ctx_.init(runtime, aicpu_thread_num_, sched_thread_num_, get_platform_regs()) != 0) { - init_failed_.store(true, std::memory_order_release); + // Each thread needs a distinct index in [0, nthreads) to pick the leader and + // partition the cores. Onboard the gate filter assigns it (exec_idx); sim's + // gate does not, so platform_aicpu_affinity_thread_idx() is -1 here for every + // thread — hand those a distinct index from a counter (mirrors run()'s + // thread_idx_++ fallback) instead of collapsing them all to leader 0, which + // would run pre_/post_handshake_init on every thread and race the shared + // scheduler state. Exactly nthreads threads reach init (the gate drops the + // rest), so the counter yields a gap-free [0, nthreads). + int32_t tidx = platform_aicpu_affinity_thread_idx(); + if (tidx < 0) tidx = hs_thread_seq_.fetch_add(1, std::memory_order_acq_rel); + // A thread whose index still falls outside [0, nthreads) owns no core slice: + // handshake_partition would compute lo/hi past cores_total_num_ and index + // all_handshakes[]/core_exec_states_ out of bounds. Reject it here (mirrors + // the bounds guard already in run()). Fail only this thread and do NOT set + // init_failed_ — that would make the valid peers abort before their + // hs_arrived_ increment and hang the leader at the barrier below. + if (tidx >= nthreads) { + LOG_ERROR("AICPU affinity thread idx %d out of range [0,%d) in init", tidx, nthreads); return -1; } + const bool is_leader = (tidx == 0); - finished_count_.store(0, std::memory_order_release); + if (is_leader) { + LOG_INFO_V0("AicpuExecutor: Initializing"); + // The 0 → 1 fixup already applied above; derive scheduler count from it. + aicpu_thread_num_ = nthreads; + sched_thread_num_ = nthreads - 1; + serial_orch_sched_ = runtime->dev.serial_orch_sched; - init_done_.store(true, std::memory_order_release); - LOG_INFO_V0("AicpuExecutor: Init complete"); + hs_arrived_.store(0, std::memory_order_relaxed); + if (sched_ctx_.pre_handshake_init(runtime, aicpu_thread_num_, sched_thread_num_, get_platform_regs()) != 0) { + init_failed_.store(true, std::memory_order_release); + hs_setup_done_.store(true, std::memory_order_release); + return -1; + } + hs_setup_done_.store(true, std::memory_order_release); + } else { + while (!hs_setup_done_.load(std::memory_order_acquire)) { + if (init_failed_.load(std::memory_order_acquire)) return -1; + } + if (init_failed_.load(std::memory_order_acquire)) return -1; + } + + // All threads: handshake this thread's slice of cores in parallel. + sched_ctx_.handshake_partition(runtime, tidx, nthreads); + + // Barrier: leader waits for every slice to finish, then completes init. + hs_arrived_.fetch_add(1, std::memory_order_acq_rel); + if (is_leader) { + while (hs_arrived_.load(std::memory_order_acquire) < nthreads) {} + finished_count_.store(0, std::memory_order_release); + if (sched_ctx_.post_handshake_init(runtime) != 0) { + init_failed_.store(true, std::memory_order_release); + init_done_.store(true, std::memory_order_release); + return -1; + } + init_done_.store(true, std::memory_order_release); + LOG_INFO_V0("AicpuExecutor: Init complete"); + } else { + while (!init_done_.load(std::memory_order_acquire)) { + if (init_failed_.load(std::memory_order_acquire)) return -1; + } + if (init_failed_.load(std::memory_order_acquire)) return -1; + } return 0; } @@ -803,9 +858,11 @@ void AicpuExecutor::deinit(Runtime * /*runtime*/) { LOG_INFO_V0("DeInit: Runtime execution state reset"); - initialized_.store(false, std::memory_order_release); init_done_.store(false, std::memory_order_release); init_failed_.store(false, std::memory_order_release); + hs_setup_done_.store(false, std::memory_order_release); + hs_arrived_.store(0, std::memory_order_release); + hs_thread_seq_.store(0, std::memory_order_release); thread_idx_.store(0, std::memory_order_release); finished_.store(false, std::memory_order_release); @@ -844,10 +901,11 @@ extern "C" __attribute__((visibility("default"))) int simpler_aicpu_register_cal * * This is called by DynTileFwkBackendKernelServer in kernel.cpp. * Orchestrates the complete task runtime execution: - * 1. Initialize executor (thread-safe, first thread only) - * 2. Wait for initialization to complete - * 3. Execute tasks on managed cores - * 4. Cleanup when last thread finishes + * 1. Initialize executor: all threads enter init(), which handshakes the cores + * in parallel and barriers internally until init is complete (or a thread + * failed); its return value is authoritative on every thread. + * 2. Execute tasks on managed cores + * 3. Cleanup when last thread finishes * * @param runtime Pointer to Runtime structure * @return 0 on success, non-zero on error @@ -865,12 +923,12 @@ extern "C" int32_t aicpu_execute(Runtime *runtime) { // rc / runtime_rc are declared out here because they outlive their phase. { AicpuPhaseScope preamble(AicpuPhase::Preamble); - g_aicpu_executor.init(runtime); - while (!g_aicpu_executor.init_done_.load(std::memory_order_acquire)) { - if (g_aicpu_executor.init_failed_.load(std::memory_order_acquire)) { - LOG_ERROR("%s", "aicpu_execute: Initialization failed, aborting execution"); - return -1; - } + // init() barriers every thread internally until init is complete on the + // leader (or a thread failed), then returns the status — so a non-zero + // return is authoritative on all threads and no extra spin is needed. + if (g_aicpu_executor.init(runtime) != 0) { + LOG_ERROR("%s", "aicpu_execute: Initialization failed, aborting execution"); + return -1; } } diff --git a/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md b/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md index 363a8d1bb..f4815bdcb 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md +++ b/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md @@ -599,7 +599,7 @@ Private internals are split across three .cpp files by responsibility: - `scheduler_completion.cpp` — completion polling, drain protocol - `scheduler_dispatch.cpp` — task dispatch loop and helpers -- `scheduler_cold_path.cpp` — exit checks, stall diagnostics, profiling, lifecycle (`init/deinit`), core management (`handshake_all_cores` / `assign_cores_to_threads` / `emergency_shutdown`), and `on_orchestration_done` +- `scheduler_cold_path.cpp` — exit checks, stall diagnostics, profiling, lifecycle (`pre_handshake_init` / `handshake_partition` / `post_handshake_init` / `deinit`), core management (`assign_cores_to_threads` / `emergency_shutdown`), and `on_orchestration_done` `AicpuExecutor` calls neither `handshake_*`, `assign_*`, `reassign_*`, nor `emergency_shutdown` directly — they are private, invoked only by `init` and `on_orchestration_done`. diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime.h b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime.h index 191a3f12c..b81d97a8b 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime.h +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime.h @@ -95,20 +95,18 @@ constexpr int RUNTIME_DEFAULT_READY_QUEUE_SHARDS = PLATFORM_MAX_AICPU_THREADS - * * Field Access Patterns: * - aicpu_ready: Written by AICPU, read by AICore - * - aicore_done: Written by AICore, read by AICPU + * - aicore_done: Written by AICore, read by AICPU (final report; physical_core_id + * and core_type are published alongside it in the same write) * - task: Written by AICPU, read by AICore (Init: PTO2DispatchPayload*; runtime: unused) - * - core_type: Written by AICPU, read by AICore (CoreType::AIC or CoreType::AIV) - * - physical_core_id: Written by AICore (Phase 2), read by AICPU - * - aicpu_regs_ready / aicore_regs_ready: handshake sequence flags + * - core_type: Written by AICore (with aicore_done), read by AICPU + * - physical_core_id: Written by AICore (with aicore_done), read by AICPU */ struct Handshake { - volatile uint32_t aicpu_ready; // AICPU ready signal: 0=not ready, 1=ready - volatile uint32_t aicore_done; // AICore ready signal: 0=not ready, core_id+1=ready - volatile uint64_t task; // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused - volatile CoreType core_type; // Core type: CoreType::AIC or CoreType::AIV - volatile uint32_t physical_core_id; // Physical core ID - volatile uint32_t aicpu_regs_ready; // AICPU register init done: 0=pending, 1=done - volatile uint32_t aicore_regs_ready; // AICore ID reported: 0=pending, 1=done + volatile uint32_t aicpu_ready; // AICPU ready signal: 0=not ready, 1=ready + volatile uint32_t aicore_done; // AICore ready signal: 0=not ready, core_id+1=ready + volatile uint64_t task; // Init: PTO2DispatchPayload* (set before aicpu_ready); runtime: unused + volatile CoreType core_type; // Core type: CoreType::AIC or CoreType::AIV (reported by AICore with aicore_done) + volatile uint32_t physical_core_id; // Physical core ID (reported by AICore with aicore_done) } __attribute__((aligned(64))); enum class TensorReleaseKind { diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp index 337cc5df4..4dbd1589f 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp @@ -673,29 +673,33 @@ int32_t SchedulerContext::shutdown(int32_t thread_idx) { } // ============================================================================= -// Handshake with all AICore workers; discover core type and reg address. +// Handshake a contiguous slice of AICore workers. Runs on every AICPU thread in +// parallel (partitioned by tidx/nthreads); the leader's pre_handshake_init has +// already zeroed state, set cores_total_num_, and reset the counts/flag. The +// per-core work here — releasing the core, then opening its register window over +// serial MMIO — is what dominates preamble, so splitting the slice across +// threads is the whole point. Within a slice we still sweep (poll every +// outstanding core per pass, service whichever reported) so one slow core's +// wakeup overlaps its neighbours' instead of blocking them. Worker-id lists are +// built serially in post_handshake_init (core-index order) once every slice has +// landed, so the shared aic_count_/aiv_count_ are written by one thread only. // ============================================================================= -int32_t SchedulerContext::handshake_all_cores(Runtime *runtime) { +void SchedulerContext::handshake_partition(Runtime *runtime, int32_t tidx, int32_t nthreads) { Handshake *all_handshakes = reinterpret_cast(runtime->dev.workers); - cores_total_num_ = runtime->dev.worker_count; - - // Validate cores_total_num_ before using as array index - if (cores_total_num_ == 0 || cores_total_num_ > RUNTIME_MAX_WORKER) { - LOG_ERROR("Invalid cores_total_num %d (expected 1-%d)", cores_total_num_, RUNTIME_MAX_WORKER); - return -1; - } - - aic_count_ = 0; - aiv_count_ = 0; - - LOG_INFO_V0("Handshaking with %d cores", cores_total_num_); - - // Step 1: Write per-core payload addresses and send handshake signal. - // OUT_OF_ORDER_STORE_BARRIER() ensures task is globally visible before - // aicpu_ready=1, so AICore reads the correct payload pointer after waking up. - for (int32_t i = 0; i < cores_total_num_; i++) { + const int32_t total = cores_total_num_; + const int32_t lo = static_cast((static_cast(tidx) * total) / nthreads); + const int32_t hi = static_cast((static_cast(tidx + 1) * total) / nthreads); + + // Step 1: publish this slice's payload pointers, then release its cores. One + // barrier (not one per core) suffices: it guarantees every task store is + // globally visible before any aicpu_ready store, which is the only ordering + // AICore relies on (it reads task only after observing aicpu_ready==1). Each + // core is written by exactly one thread, so the slices never overlap. + for (int32_t i = lo; i < hi; i++) { all_handshakes[i].task = reinterpret_cast(&payload_per_core_[i][0]); - OUT_OF_ORDER_STORE_BARRIER(); + } + OUT_OF_ORDER_STORE_BARRIER(); + for (int32_t i = lo; i < hi; i++) { all_handshakes[i].aicpu_ready = 1; } OUT_OF_ORDER_STORE_BARRIER(); @@ -703,71 +707,66 @@ int32_t SchedulerContext::handshake_all_cores(Runtime *runtime) { // Get platform physical cores count for validation uint32_t max_physical_cores_count = platform_get_physical_cores_count(); - // Step 2: Wait for all cores to respond, collect core type and register addresses - bool handshake_failed = false; - for (int32_t i = 0; i < cores_total_num_; i++) { - Handshake *hank = &all_handshakes[i]; - - while (hank->aicore_regs_ready == 0) { - SPIN_WAIT_HINT(); - } - - uint32_t physical_core_id = hank->physical_core_id; - - if (physical_core_id >= max_physical_cores_count) { - LOG_ERROR( - "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id, - max_physical_cores_count - ); - handshake_failed = true; - continue; - } - - uint64_t *regs = reinterpret_cast(regs_); - uint64_t reg_addr = regs[physical_core_id]; - - // Initialize AICore registers after discovery (first round) - platform_init_aicore_regs(reg_addr); - OUT_OF_ORDER_STORE_BARRIER(); - hank->aicpu_regs_ready = 1; - - OUT_OF_ORDER_STORE_BARRIER(); - - while (hank->aicore_done == 0) { - SPIN_WAIT_HINT(); - } + // Step 2: collect responses from this slice. Each core reports + // {physical_core_id, core_type, aicore_done} in one write, then waits — by + // polling its own DATA_MAIN_BASE SPR — for us to open its register window. + // We sweep the slice: poll every outstanding core per pass and service + // whichever have reported, rather than blocking on core i before looking at + // core i+1, so per-core wakeups overlap (≈ max, not Σ). aicore_done is a GM + // read (not the nGnRE MMIO reg window), so sweeping is not forced serial the + // way RegId::COND polling is. + // + // Servicing a core = validate its physical_core_id, then open its register + // window (platform_init_aicore_regs: FAST_PATH + DATA_MAIN_BASE=IDLE). That + // IDLE write is *also* the signal the core polls for to leave its + // post-report wait — so opening the window IS the acknowledgement. There is + // no separate aicpu_regs_ready ack and no second round-trip. AIC/AIV + // classification is deferred to post_handshake_init (serial) so aic_count_/ + // aiv_count_ are never incremented from more than one thread. + uint64_t *regs = reinterpret_cast(regs_); + bool core_serviced[RUNTIME_MAX_WORKER] = {false}; + + for (int32_t remaining = hi - lo; remaining > 0;) { + for (int32_t i = lo; i < hi; i++) { + if (core_serviced[i]) continue; + Handshake *hank = &all_handshakes[i]; + if (hank->aicore_done == 0) { + SPIN_WAIT_HINT(); + continue; + } - CoreType type = hank->core_type; + uint32_t physical_core_id = hank->physical_core_id; + if (physical_core_id >= max_physical_cores_count) { + LOG_ERROR( + "Core %d reported invalid physical_core_id=%u (platform max=%u)", i, physical_core_id, + max_physical_cores_count + ); + handshake_failed_.store(true, std::memory_order_release); + core_serviced[i] = true; + remaining--; + continue; + } - core_exec_states_[i].reg_addr = reg_addr; - core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND); + // Open the window; the IDLE write releases the core's DATA_MAIN_BASE poll. + uint64_t reg_addr = regs[physical_core_id]; + platform_init_aicore_regs(reg_addr); + OUT_OF_ORDER_STORE_BARRIER(); + core_exec_states_[i].reg_addr = reg_addr; + core_exec_states_[i].cond_ptr = get_reg_ptr(reg_addr, RegId::COND); #if PTO2_PROFILING - physical_core_ids_[i] = physical_core_id; + physical_core_ids_[i] = physical_core_id; #endif - #if !PTO2_PROFILING - core_exec_states_[i].worker_id = i; - core_exec_states_[i].physical_core_id = physical_core_id; - core_exec_states_[i].core_type = type; + core_exec_states_[i].worker_id = i; + core_exec_states_[i].physical_core_id = physical_core_id; + core_exec_states_[i].core_type = hank->core_type; #endif - - if (type == CoreType::AIC) { - aic_worker_ids_[aic_count_++] = i; - LOG_INFO_V0("Core %d: AIC, physical_id=%u, reg_addr=0x%lx", i, physical_core_id, reg_addr); - } else { - aiv_worker_ids_[aiv_count_++] = i; - LOG_INFO_V0("Core %d: AIV, physical_id=%u, reg_addr=0x%lx", i, physical_core_id, reg_addr); + core_serviced[i] = true; + remaining--; } } - - if (handshake_failed) { - emergency_shutdown(runtime); - return -1; - } - - LOG_INFO_V0("Core discovery complete: %d AIC, %d AIV", aic_count_, aiv_count_); - return 0; + OUT_OF_ORDER_STORE_BARRIER(); } // ============================================================================= @@ -839,13 +838,14 @@ bool SchedulerContext::assign_cores_to_threads() { // deinit their AICore register blocks. Idempotent. // ============================================================================= void SchedulerContext::emergency_shutdown(Runtime *runtime) { + (void)runtime; // exit is now delivered via each core's register block, not GM LOG_WARN("Emergency shutdown: sending exit signal to all initialized cores"); - Handshake *all_handshakes = reinterpret_cast(runtime->dev.workers); int32_t timeout_count = 0; for (int32_t i = 0; i < cores_total_num_; i++) { - Handshake *hank = &all_handshakes[i]; - OUT_OF_ORDER_STORE_BARRIER(); - hank->aicpu_regs_ready = 1; + // platform_deinit_aicore_regs writes DATA_MAIN_BASE=EXIT, which both + // releases a core still polling for its window to open and signals it to + // exit. Cores never opened (reg_addr==0) are reaped by the host device + // reset that follows a handshake failure. if (core_exec_states_[i].reg_addr != 0) { if (platform_deinit_aicore_regs(core_exec_states_[i].reg_addr) != 0) { timeout_count++; @@ -861,8 +861,9 @@ void SchedulerContext::emergency_shutdown(Runtime *runtime) { // ============================================================================= // Lifecycle: init / deinit // ============================================================================= -int32_t -SchedulerContext::init(Runtime *runtime, int32_t aicpu_thread_num, int32_t sched_thread_num, uint64_t regs_base) { +int32_t SchedulerContext::pre_handshake_init( + Runtime *runtime, int32_t aicpu_thread_num, int32_t sched_thread_num, uint64_t regs_base +) { always_assert(runtime != nullptr); // Zero all per-core execution state before handshake @@ -882,7 +883,9 @@ SchedulerContext::init(Runtime *runtime, int32_t aicpu_thread_num, int32_t sched // and the CYCLE_COUNT_START() gate in pto_orchestrator.cpp would suppress // all ORCH_PHASES records. Reset the cached level on disabled runs so a // prior enabled launch's level can't leak into the phase-record gates in - // scheduler_dispatch (`>= SCHED_PHASES`). + // scheduler_dispatch (`>= SCHED_PHASES`). This runs on the leader before it + // publishes hs_setup_done_, so it happens-before every thread's + // handshake_partition (and therefore before any aicpu_ready=1 write). if (is_l2_swimlane_enabled()) { l2_swimlane_aicpu_init(runtime->dev.worker_count); l2_swimlane_level_ = get_l2_swimlane_level(); @@ -906,19 +909,55 @@ SchedulerContext::init(Runtime *runtime, int32_t aicpu_thread_num, int32_t sched } #endif - // Discover cores and assign to scheduler threads. - int32_t rc = handshake_all_cores(runtime); - if (rc != 0) { - LOG_ERROR("handshake_all_cores failed"); - return rc; + // Core count is needed by every thread to compute its handshake slice. + cores_total_num_ = runtime->dev.worker_count; + if (cores_total_num_ == 0 || cores_total_num_ > RUNTIME_MAX_WORKER) { + LOG_ERROR("Invalid cores_total_num %d (expected 1-%d)", cores_total_num_, RUNTIME_MAX_WORKER); + return -1; + } + aic_count_ = 0; + aiv_count_ = 0; + handshake_failed_.store(false, std::memory_order_release); + + LOG_INFO_V0("Handshaking with %d cores", cores_total_num_); + return 0; +} + +int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { + if (handshake_failed_.load(std::memory_order_acquire)) { + emergency_shutdown(runtime); + return -1; } + + // Build cluster-ordered AIC/AIV worker-id lists from the discovered cores. + // Serial and MMIO-free — the expensive per-core handshake already ran in + // parallel across threads. Core-index order matches the original + // single-thread handshake so assign_cores_to_threads forms identical + // clusters. core_type / physical_core_id are read from the Handshake struct + // (stable after the handshake) so this stays correct under PTO2_PROFILING, + // where they are not mirrored into CoreExecState. + Handshake *all_handshakes = reinterpret_cast(runtime->dev.workers); + for (int32_t i = 0; i < cores_total_num_; i++) { + CoreType type = all_handshakes[i].core_type; + uint32_t physical_core_id = all_handshakes[i].physical_core_id; + uint64_t reg_addr = core_exec_states_[i].reg_addr; + if (type == CoreType::AIC) { + aic_worker_ids_[aic_count_++] = i; + LOG_INFO_V0("Core %d: AIC, physical_id=%u, reg_addr=0x%lx", i, physical_core_id, reg_addr); + } else { + aiv_worker_ids_[aiv_count_++] = i; + LOG_INFO_V0("Core %d: AIV, physical_id=%u, reg_addr=0x%lx", i, physical_core_id, reg_addr); + } + } + LOG_INFO_V0("Core discovery complete: %d AIC, %d AIV", aic_count_, aiv_count_); + if (!assign_cores_to_threads()) { return -1; } - // Profiling-subsystem buffer/state init: single-threaded cold path, so the - // "do it once" guarantee is structural (no CAS needed). Runs after - // handshake_all_cores / assign_cores_to_threads because pmu_aicpu_init needs + // Profiling-subsystem buffer/state init: single-threaded cold path (leader + // only), so the "do it once" guarantee is structural (no CAS needed). Runs + // after the handshake / assign_cores_to_threads because pmu_aicpu_init needs // physical_core_ids_ / cores_total_num_. Mirrors the l2_swimlane_aicpu_init // convention above; the per-thread *_set_orch_thread_idx setters stay on the // orchestrator thread (see aicpu_executor.cpp). diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h index 2ec464587..f8085e02b 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h @@ -52,14 +52,25 @@ class SchedulerContext { // Lifecycle // ========================================================================= - // Initialize scheduler state from the given runtime and thread layout. - // - Discovers cores via handshake_all_cores() - // - Assigns cores to scheduler threads - // - Resets task counters, payloads, per-core GlobalContext - // - Binds func_id_to_addr_ / initial sched_ (if rt is already known) - // - Captures AICore-register base (consumed by handshake_all_cores()) - // Returns 0 on success, negative on failure (handshake / assignment error). - int32_t init(Runtime *runtime, int32_t aicpu_thread_num, int32_t sched_thread_num, uint64_t regs_base); + // Initialize scheduler state from the given runtime and thread layout. Split + // into three parts so the per-core AICore handshake — a serial, MMIO-bound + // loop that dominates preamble (~217 µs of ~283 µs for 72 cores) — can run in + // parallel across all AICPU threads. Orchestrated by AicpuExecutor::init: + // the leader runs pre_handshake_init, every thread handshakes a disjoint + // slice of cores via handshake_partition, then the leader runs + // post_handshake_init after a barrier. + // + // Leader-only: per-core state + config + swimlane buffers + core count. Must + // be published before any thread enters handshake_partition. Returns 0 on + // success, negative on failure. + int32_t + pre_handshake_init(Runtime *runtime, int32_t aicpu_thread_num, int32_t sched_thread_num, uint64_t regs_base); + // All threads: handshake this thread's contiguous slice [lo, hi) of cores + // (partitioned by tidx/nthreads). Each core is touched by exactly one thread. + void handshake_partition(Runtime *runtime, int32_t tidx, int32_t nthreads); + // Leader-only, after the handshake barrier: build worker-id lists, assign + // cores, init profiling subsystems, read task counts, init payloads. + int32_t post_handshake_init(Runtime *runtime); // Reset all SchedulerContext-owned state to its post-construction defaults. // Called by AicpuExecutor::deinit() during per-run teardown. @@ -155,12 +166,16 @@ class SchedulerContext { int32_t aicpu_thread_num_{0}; int32_t cores_total_num_{0}; - // Cluster-ordered worker_id lists, populated by handshake_all_cores(). + // Cluster-ordered worker_id lists, populated by post_handshake_init(). int32_t aic_worker_ids_[RUNTIME_MAX_WORKER]{}; int32_t aiv_worker_ids_[RUNTIME_MAX_WORKER]{}; int32_t aic_count_{0}; int32_t aiv_count_{0}; + // Set by any thread whose slice hits an invalid physical_core_id in + // handshake_partition; checked by the leader in post_handshake_init. + std::atomic handshake_failed_{false}; + #if PTO2_PROFILING // Physical core ids keyed by logical worker id. Populated by // handshake_all_cores() and handed to pmu_aicpu_init() so the platform @@ -177,9 +192,6 @@ class SchedulerContext { // Core management (scheduler_cold_path.cpp) // ========================================================================= - // Handshake with all AICore workers; populates core_exec_states_, worker id lists. - int32_t handshake_all_cores(Runtime *runtime); - // Assign discovered cores (cluster = 1 AIC + 2 AIV) round-robin across scheduler threads. bool assign_cores_to_threads();