diff --git a/docs/investigations/2026-07-aicore-fills-all-args-ready-path.md b/docs/investigations/2026-07-aicore-fills-all-args-ready-path.md new file mode 100644 index 000000000..2ed9e5fe3 --- /dev/null +++ b/docs/investigations/2026-07-aicore-fills-all-args-ready-path.md @@ -0,0 +1,91 @@ +# AICore-side arg materialization for ALL dispatches (not just the gated path) + +**Date**: 2026-07-11 +**Verdict**: measured and rejected for the ready path — making the AICore +fill its own `args[]` for **every** task adds **~1.0 µs to each task's +setup** (`receive → start`), on the critical path, because a ready task has +no idle doorbell gate to hide the fill behind. Offloading is only a win on +the `not_ready` (early-dispatch) path, where the AICore is already spinning +at the gate and absorbs the fill for free. The shipped design keeps the +AICPU filling `args[]` for ready tasks and offloads only gated ones. + +## Question + +The tmr dispatch handoff writes the kernel `args[]` (tensor GM pointers + +scalar values) into the per-core `PTO2DispatchPayload` on the AICPU. For +gated (`not_ready` / early-dispatch) tasks we already offload that fill to +the AICore — it does it during its doorbell wait, off the AICPU dispatch +path (see the folded-gate `src_payload` design in the dispatch cold-write +PR). + +Natural follow-on: why not offload the fill for **all** tasks and take the +`args[]` write off the AICPU entirely? What does that cost the AICore? + +## What was tried + +An experiment build (not merged) that: + +- Adds a separate `gated` flag to `PTO2DispatchPayload` — with all tasks + filling from source, `src_payload` is always non-zero and can no longer + double as the gate flag (the shipped design folds `not_ready` into + `src_payload == 0`). +- AICPU `build_payload` writes only `src_payload = &PTO2TaskPayload` and + `gated`, and **never** fills `args[]`. +- AICore `aicore_executor` fills `args[0..num_args)` from `src_payload` for + **every** task (moved out of the gate branch), then gates only when + `gated`. + +Measured `paged_attention_unroll` (`--enable-l2-swimlane`, 1024 AICore +tasks, a2a3 silicon) and read each task's AICore setup — +`receive_to_start_cycles` from `l2_swimlane_records.json::aicore_tasks` +(50 MHz → 20 ns/cycle). + +## Result + +| build | AICore setup (`receive → start`) | +| ----- | -------------------------------- | +| baseline (AICPU fills args) | mean **349 ns** / p50 320 ns | +| experiment (AICore fills **all**) | mean **1356 ns** / p50 1340 ns | +| **delta** | **+~1.0 µs/task (+50 cycles, ~4×)** | + +Correctness held (`paged_attention_unroll` passes), so the AICore fill is +functionally equivalent — it just costs ~1 µs. The fill is AICore GM work: +after the per-task whole-cache `dcci(ENTIRE_DATA_CACHE)`, the first reads of +the source `PTO2TaskPayload` (counts, scalars) miss to HBM, then the loop +computes `&tensors[i]` and writes `args[]` — all on a core whose GM-access +latency is high. On the **ready** path there is no doorbell wait to overlap +it with, so the whole ~1 µs lands on the `receive → start` setup. + +## Why the shipped design offloads only the gated path + +- **Gated (`not_ready`) tasks are staged before their producer completes**; + the AICore then spins at the doorbell gate. The ~1 µs fill runs inside + that otherwise-idle wait → net ~zero, and it takes the `args[]` write off + the AICPU dispatch path (the win we wanted). +- **Ready tasks execute on pickup** — no gate, no idle window. Offloading + the fill there is a straight +~1 µs/task regression on the AICore + critical path. For an AICore-bound workload (e.g. `paged_attention`, + 1024 tasks) that is a large, unhideable cost. + +So the asymmetry is intrinsic: offload where there is idle time to spend +(gated), keep it on the AICPU where the AICore would otherwise stall +(ready). + +## When to reconsider + +- If a future protocol gives ready tasks an idle window between pickup and + the point their inputs are guaranteed visible (today the kernel's input + `dcci` runs right after ACK), the fill could hide there — re-measure. +- If the AICPU dispatch path becomes the proven end-to-end bottleneck for a + ready-dominated workload AND the AICore has spare setup budget (setup ≪ + the WAIT/exec time), the trade could flip — but weigh against the ~1 µs + measured here, and note it scales with per-task arg count. + +## References + +- Dispatch cold-write PR (folded-gate `src_payload`, AICore fill on the + gated path, init prefill, deferred-slab-at-completion, prefetch): + hw-native-sys/simpler#1328. +- [2026-07 — L2 swimlane AICore switch-overhead](2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md) + — the baseline ~0.28–0.35 µs AICore setup (`payload dcci + ACK`) this + experiment adds onto. diff --git a/docs/investigations/README.md b/docs/investigations/README.md index 3742f4323..061c8edf2 100644 --- a/docs/investigations/README.md +++ b/docs/investigations/README.md @@ -84,6 +84,7 @@ that ...". Newest first. +- [2026-07 — AICore-side arg fill for ALL dispatches (not just the gated path)](2026-07-aicore-fills-all-args-ready-path.md) — measured & rejected for the ready path: making the AICore fill its own `args[]` for every task adds ~1.0 µs to each task's `receive→start` setup (paged_attention_unroll: 349 ns → 1356 ns), because a ready task has no idle doorbell gate to hide the fill. Offload is a win only on `not_ready` (early-dispatch), where the AICore already spins at the gate; shipped design keeps the AICPU filling ready tasks (#1328) - [2026-07 — L2 swimlane AICore: switch-overhead source + FIN-early reorder & ACK-gate](2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md) — measured: the ~0.8 µs inter-task switch is the record write-back `dcci(record,OUT)+dsb` (~0.5 µs) + payload setup (~0.28 µs), inherent and not reducible by moving FIN; the WAIT gap (p99 ~700 µs) dominates decode. Shipped: sample `end_time` after an early FIN, and an AICPU ACK-gate on buffer rotation (release the old buffer only when AICore ACKs the new buffer's first task) to close the FIN-before-record boundary race the reorder introduced - [2026-07 — Removing PTO2LocalReadyBuffer exposed a missing dcci in EP dispatch](2026-07-local-buffer-removal-ep-combine-regression.md) — RESOLVED in #1245: local-buffer removal changed dispatch timing and unmasked a latent kernel bug (dispatch never dcci'd `recv_count_out` to HBM → local_expert read count=0 → all-zero output); fixed with a one-line dcci in the example kernel - [2026-06 — Gating the two residual profiling enable() calls on the orch/scheduler hot path](2026-06-orch-profiling-enable-gates-hot-path.md) — gated under existing `SIMPLER_DFX`; magnitude unmeasured, no new macro diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp index 4ede4a07e..343c5a8f3 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp @@ -144,9 +144,9 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in // (receive_time → start_time). Stored as a 32-bit delta // `start_time - receive_time`. // - // Common path (not_ready == 0): the new task_id is itself the ready + // Common path (src_payload == 0): the new task_id is itself the ready // signal, so receive_time is the true ready moment. Early-dispatch path - // (not_ready == 1): receive_time stays at pickup — before the + // (src_payload != 0): receive_time stays at pickup — before the // doorbell wait — so it precedes the producer's end_time; the host // folds it to start_time for those tasks (detected when receive // precedes the producer task's end_time). @@ -160,16 +160,39 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in // Invalidate payload buffer (AICPU updates its content each dispatch) dcci(exec_payload, ENTIRE_DATA_CACHE); - // Early-dispatch gate. A not-ready task was staged on - // this core before its dependencies resolved; wait until AICPU rings - // the doorbell (DATA_MAIN_BASE high 32 == task_id) before executing. - // The ACK is deferred until AFTER the gate so the scheduler keeps the - // core off-limits (pending_occupied stays set, no ACK->pending_freed) - // while the task is gated — preventing a real task from being - // dual-issued behind it. The kernel's own input dcci runs inside - // execute_task() below — strictly AFTER this gate — so predecessor - // outputs are visible. not_ready == 0 (the common path) skips this. - if (exec_payload->not_ready) { + // Early-dispatch gate. A gated task was staged on this core before its + // dependencies resolved; wait until AICPU rings the doorbell + // (DATA_MAIN_BASE high 32 == task_id) before executing. The ACK is + // deferred until AFTER the gate so the scheduler keeps the core + // off-limits (pending_occupied stays set, no ACK->pending_freed) while + // the task is gated — preventing a real task from being dual-issued + // behind it. The kernel's own input dcci runs inside execute_task() + // below — strictly AFTER this gate — so predecessor outputs are visible. + // src_payload == 0 (the common ready path) skips this; a non-zero + // src_payload is both the gate flag and the source PTO2TaskPayload. + if (exec_payload->src_payload != 0) { + // AICPU staged only src_payload, not the arg vector — fill + // args[0..num_args) ourselves now, while we are idle waiting for + // the doorbell. The whole-cache dcci(ENTIRE_DATA_CACHE) above + // already invalidated src's lines, so tensor_count/scalar_count/ + // scalars read coherently with the orchestrator's submit writes. + // args[SPMD_LOCAL_CONTEXT_INDEX]/[SPMD_GLOBAL_CONTEXT_INDEX] are + // still written by the AICPU (num_args <= 48 never reaches them). + __gm__ char *src = reinterpret_cast<__gm__ char *>(exec_payload->src_payload); + int32_t tensor_count = *reinterpret_cast<__gm__ int32_t *>(src + PTO2_TASKPAYLOAD_TENSOR_COUNT_OFFSET); + int32_t scalar_count = *reinterpret_cast<__gm__ int32_t *>(src + PTO2_TASKPAYLOAD_SCALAR_COUNT_OFFSET); + __gm__ uint64_t *src_scalars = + reinterpret_cast<__gm__ uint64_t *>(src + PTO2_TASKPAYLOAD_SCALARS_OFFSET); + int n = 0; + for (int32_t i = 0; i < tensor_count; i++) { + exec_payload->args[n++] = reinterpret_cast( + src + PTO2_TASKPAYLOAD_TENSORS_OFFSET + i * PTO2_TASKPAYLOAD_TENSOR_STRIDE + ); + } + for (int32_t i = 0; i < scalar_count; i++) { + exec_payload->args[n++] = src_scalars[i]; + } + OUT_OF_ORDER_STORE_BARRIER(); while (true) { // Honor teardown: shutdown overwrites the low half with EXIT. // Check it on the doorbell-match iteration too, so an EXIT that diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto2_dispatch_payload.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto2_dispatch_payload.h index aa9efa5f9..899de63f0 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto2_dispatch_payload.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto2_dispatch_payload.h @@ -34,6 +34,7 @@ #pragma once +#include #include #include "arg_direction.h" @@ -57,6 +58,18 @@ static_assert( "GLOBAL_CONTEXT_INDEX out of sync with intrinsic.h" ); +// Byte offsets into PTO2TaskPayload used by AICore to materialize args[] on the +// not_ready (early-dispatch) path. The AICore .o does not include +// pto_runtime2_types.h, so it reads tensor_count / scalar_count / tensors[] / +// scalars[] through these constants. The AICPU side (scheduler_dispatch.cpp) +// static_asserts them against offsetof where the full struct is visible, so any +// PTO2TaskPayload layout drift fails the build rather than corrupting args[]. +constexpr uint32_t PTO2_TASKPAYLOAD_TENSOR_COUNT_OFFSET = 0; +constexpr uint32_t PTO2_TASKPAYLOAD_SCALAR_COUNT_OFFSET = 4; +constexpr uint32_t PTO2_TASKPAYLOAD_TENSORS_OFFSET = 576; +constexpr uint32_t PTO2_TASKPAYLOAD_SCALARS_OFFSET = 4672; +constexpr uint32_t PTO2_TASKPAYLOAD_TENSOR_STRIDE = 128; // sizeof(Tensor) + /** * Per-core dispatch payload: function address + args[] + SPMD context. * @@ -68,24 +81,40 @@ static_assert( * concurrently dispatched cores. */ struct alignas(64) PTO2DispatchPayload { - uint64_t function_bin_addr; /**< Kernel entry address in GM (set by Scheduler) */ - uint64_t args[PTO2_DISPATCH_MAX_ARGS]; /**< Kernel arguments (GM pointers + scalars + ext params) */ + // === Cache line 0 (64B): control block, the only line written per dispatch === + // function_bin_addr, local_context.{block_idx,block_num,async_ctx.task_token} + // and src_payload are the per-dispatch writes; async_ctx's slab pointers + + // capacity are cold (prefilled once at init) but ride this hot line for free. + // Sized to exactly 64B so both dispatch paths write one control line: the + // ready path (src_payload = 0) then also fills args[0..num_args); the gated + // path (src_payload = &PTO2TaskPayload) leaves args[] to the idle AICore. + uint64_t function_bin_addr; /**< Kernel entry address in GM (set by Scheduler). */ - /** Per-dispatch context: block_idx and block_num. - * Written by build_payload() before each dispatch. - * args[SPMD_LOCAL_CONTEXT_INDEX] points here. */ + /** Per-dispatch context: block_idx/block_num (hot) + async_ctx (task_token hot, + * slab pointers + capacity prefilled once at init). args[SPMD_LOCAL_CONTEXT_INDEX] + * points here. */ LocalContext local_context; - /** Per-core global context: sub_block_id (AIV lane identity). - * Initialized once by init_global_context() at runtime startup. - * args[SPMD_GLOBAL_CONTEXT_INDEX] points here. */ - GlobalContext global_context; + /** Early-dispatch gate AND source pointer, folded into one field. 0 = ready: + * AICore executes on pickup (args[] already filled by the AICPU). Non-zero = + * gated: the value is the source PTO2TaskPayload address; the AICore fills + * args[0..num_args) from it, then waits for the doorbell (DATA_MAIN_BASE + * high 32 == this dispatch's reg_task_id) before executing. A payload address + * is never 0, so the gate flag is lossless. */ + volatile uint64_t src_payload; - /** Early-dispatch gate. 0 = ready: AICore executes on pickup. - * 1 = not-ready: AICore waits until AICPU rings the doorbell - * (DATA_MAIN_BASE high 32 == this dispatch's reg_task_id) before executing. */ - volatile uint32_t not_ready; - uint8_t reserved_payload_abi_pad[4]; + // === Cache lines 1..7: kernel argument vector === + /** [0..num_args) = GM tensor pointers + scalar values; [SPMD_LOCAL_CONTEXT_INDEX] + * = &local_context, [SPMD_GLOBAL_CONTEXT_INDEX] = &global_context (both prefilled + * once at init). On the gated path the AICPU leaves [0..num_args) unwritten and + * the idle AICore fills them from src_payload during its gate wait. */ + uint64_t args[PTO2_DISPATCH_MAX_ARGS]; + + /** Per-core global context: sub_block_id (AIV lane identity). Cold: written once + * at init, never per dispatch, so it lives in the tail (not the CL0 control + * block). args[SPMD_GLOBAL_CONTEXT_INDEX] points here. */ + GlobalContext global_context; + // No explicit tail padding: alignas(64) rounds sizeof up to 512 (8 cache lines). static_assert(sizeof(args[0]) == 8); static_assert( @@ -95,3 +124,5 @@ struct alignas(64) PTO2DispatchPayload { }; static_assert(sizeof(PTO2DispatchPayload) == 512, "PTO2DispatchPayload hardware ABI size drift"); +static_assert(offsetof(PTO2DispatchPayload, args) == 64, "args[] must start at cache line 1 (control block = CL0)"); +static_assert(offsetof(PTO2DispatchPayload, src_payload) < 64, "src_payload (gate) must live on the CL0 control block"); diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp index 0b87b4003..e00144d43 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp @@ -1033,6 +1033,35 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { } } + // Prefill the per-dispatch AsyncCtx constant fields once. Of AsyncCtx's five + // fields, four are constant for a given (core, buf_idx): the three pointers + // target the fixed deferred_slab_per_core_[core][buf] members, and capacity is + // MAX_COMPLETIONS_PER_TASK. Only task_token varies per dispatch, so build_payload + // writes just that; these constants survive across dispatches because the + // payload buffer is never zeroed between them. + // The two context-pointer args are also per-(core, buf_idx) constants — they + // target this buffer's own local_context / global_context — so prefill them + // here too and drop them from the per-dispatch build_payload writes. This keeps + // the per-dispatch write footprint on the CL0 control block only (args[48]/[49] + // live on a later line). + for (int32_t core_id = 0; core_id < RUNTIME_MAX_WORKER; core_id++) { + for (int32_t buf = 0; buf < 2; buf++) { + PTO2DispatchPayload &dp = payload_per_core_[core_id][buf]; + AsyncCtx &ac = dp.local_context.async_ctx; + volatile DeferredCompletionSlab *slab = &deferred_slab_per_core_[core_id][buf]; + ac.completion_count = &slab->count; + ac.completion_error_code = &slab->error_code; + ac.completion_entries = &slab->entries[0]; + ac.completion_capacity = MAX_COMPLETIONS_PER_TASK; + // Clear the slab once here; thereafter only the completion path re-clears + // count (and only when a deferred task dirtied it), never per dispatch. + slab->count = 0; + slab->error_code = PTO2_ERROR_NONE; + dp.args[PAYLOAD_LOCAL_CONTEXT_INDEX] = reinterpret_cast(&dp.local_context); + dp.args[PAYLOAD_GLOBAL_CONTEXT_INDEX] = reinterpret_cast(&dp.global_context); + } + } + func_id_to_addr_ = runtime->dev.func_id_to_addr_; return 0; @@ -1047,11 +1076,14 @@ void SchedulerContext::deinit() { } // No per-core memset of payload_per_core_ / deferred_slab_per_core_ here - // (~300 KB across all cores). Both are fully re-initialized at dispatch - // before they can be read: dispatch_task sets deferred_slab->count = 0 / - // error_code = NONE and build_payload() overwrites every payload field - // (function addr, args[], contexts, not_ready) on the exact [core][buf_idx] - // about to run. The consumer side cannot reach a stale slot either: the + // (~300 KB across all cores). They are re-initialized before they can be read: + // build_payload() overwrites the per-dispatch payload fields (function addr, + // args[0..num_args) or src_payload, block_idx/block_num, async_ctx.task_token) + // on the exact [core][buf_idx] about to run; the async_ctx slab pointers + + // capacity, the two context-pointer args, and the deferred slab (count = 0 / + // error_code = NONE) are all cleared once per run in init() — the slab is + // thereafter re-cleared only by the completion path after a deferred task. + // The consumer side cannot reach a stale slot either: the // drain only services a core's running_reg_task_id, and the loop above // already reset every core_exec_states_[].running/pending_reg_task_id to // AICPU_TASK_INVALID — so no FIN for an undispatched slot is processed, and diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp index ca37f2801..294b39092 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_completion.cpp @@ -144,6 +144,11 @@ void SchedulerContext::complete_slot_task( SPIN_WAIT_HINT(); } } + // Re-clear for the next reuse of this (core, buf) slot. Done here — on + // the hot cache line we just read — instead of on every dispatch, since + // only a deferred task (count > 0) ever dirties it. error_code needs no + // reset: a non-NONE code aborted the run above. + deferred_slab->count = 0; } } @@ -505,11 +510,18 @@ void SchedulerContext::drain_worker_dispatch(int32_t block_num) { slot_state->next_block_idx.store(static_cast(start + claim), std::memory_order_relaxed); PublishHandle handles[CoreTracker::MAX_CLUSTERS * 3]; int handle_count = 0; + // Gather the claimed offsets so the per-core destination structures of block + // b+1 can be prefetched while block b's prepare runs (software pipeline — + // this drain launches every block on a single elected thread). + int32_t claimed[CoreTracker::MAX_CLUSTERS * 3]; + for (int32_t b = 0; b < claim; b++) + claimed[b] = valid.pop_first(); + bool is_mix = (shape == PTO2ResourceShape::MIX); + if (claim > 0) prefetch_block_dst(t, claimed[0], is_mix); for (int32_t b = 0; b < claim; b++) { - auto core_offset = valid.pop_first(); - handle_count += prepare_block_for_dispatch( - t, core_offset, *slot_state, shape, false, start + b, &handles[handle_count] - ); + if (b + 1 < claim) prefetch_block_dst(t, claimed[b + 1], is_mix); + handle_count += + prepare_block_for_dispatch(t, claimed[b], *slot_state, shape, false, start + b, &handles[handle_count]); } wmb(); uint64_t dispatch_ts = 0; diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h index 347126cff..4143242db 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_context.h @@ -230,8 +230,7 @@ class SchedulerContext { ); void build_payload( - PTO2DispatchPayload &dispatch_payload, PTO2TaskSlotState &slot_state, PTO2SubtaskSlot subslot, - const AsyncCtx &async_ctx, int32_t block_idx + PTO2DispatchPayload &dispatch_payload, PTO2TaskSlotState &slot_state, PTO2SubtaskSlot subslot, int32_t block_idx ); // Batched-dispatch primitives. prepare_* builds the payload and per-core @@ -262,6 +261,39 @@ class SchedulerContext { write_reg(h.reg_addr, RegId::DATA_MAIN_BASE, static_cast(h.reg_task_id)); } + // Prefetch the cold per-core structures the next block's prepare touches. + // Ordering is load-bearing: issue the STALLING LOAD first — CoreExecState, + // read by dispatch_seq++ whose value feeds reg_task_id -> buf_idx -> the whole + // dispatch — for every core of the block, BEFORE the store-target prefetches. + // MSHRs saturate (a MIX block warms 3 cores); issuing the read prefetches + // first keeps them from being the ones dropped. The dispatch-buffer writes + // still get prefetched (measured to help ~30% on this shallow-store-buffer + // control core), just after the reads. rw=1 on CoreExecState (read AND + // written) gives Exclusive, serving both without a Shared->Exclusive upgrade. + inline void prefetch_block_dst(int32_t thread_idx, int32_t core_offset, bool is_mix) { + CoreTracker &tracker = core_trackers_[thread_idx]; + int32_t cids[3]; + int32_t nc = 0; + if (is_mix) { + cids[nc++] = tracker.get_core_id_by_offset(tracker.get_aic_core_offset(core_offset)); + cids[nc++] = tracker.get_core_id_by_offset(tracker.get_aiv0_core_offset(core_offset)); + cids[nc++] = tracker.get_core_id_by_offset(tracker.get_aiv1_core_offset(core_offset)); + } else { + cids[nc++] = tracker.get_core_id_by_offset(core_offset); + } + // Stalling loads first. + for (int32_t i = 0; i < nc; i++) + __builtin_prefetch(&core_exec_states_[cids[i]], 1, 3); + // Store targets after (dispatch buffer CL0 control + CL1 args, both bufs). + for (int32_t i = 0; i < nc; i++) { + for (int32_t buf = 0; buf < 2; buf++) { + const char *dp = reinterpret_cast(&payload_per_core_[cids[i]][buf]); + __builtin_prefetch(dp, 1, 3); + __builtin_prefetch(dp + 64, 1, 3); + } + } + } + // Fan out one block's subtasks (1 for AIC/AIV, 1-3 for MIX) into the // caller-supplied handles buffer. Returns the number of handles written. int prepare_block_for_dispatch( diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp index 490d1648b..46767d9df 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_dispatch.cpp @@ -37,6 +37,15 @@ #define unlikely(x) __builtin_expect(!!(x), 0) #endif +// AICore materializes args[] from src_payload on the not_ready path using the +// byte offsets in pto2_dispatch_payload.h (the AICore .o cannot see PTO2TaskPayload). +// Pin those constants to the real layout here, where the struct is fully visible. +static_assert(offsetof(PTO2TaskPayload, tensor_count) == PTO2_TASKPAYLOAD_TENSOR_COUNT_OFFSET); +static_assert(offsetof(PTO2TaskPayload, scalar_count) == PTO2_TASKPAYLOAD_SCALAR_COUNT_OFFSET); +static_assert(offsetof(PTO2TaskPayload, tensors) == PTO2_TASKPAYLOAD_TENSORS_OFFSET); +static_assert(offsetof(PTO2TaskPayload, scalars) == PTO2_TASKPAYLOAD_SCALARS_OFFSET); +static_assert(sizeof(Tensor) == PTO2_TASKPAYLOAD_TENSOR_STRIDE); + // ============================================================================= // Dispatch helpers // ============================================================================= @@ -114,32 +123,41 @@ int SchedulerContext::pop_ready_tasks_batch( } void SchedulerContext::build_payload( - PTO2DispatchPayload &dispatch_payload, PTO2TaskSlotState &slot_state, PTO2SubtaskSlot subslot, - const AsyncCtx &async_ctx, int32_t block_idx + PTO2DispatchPayload &dispatch_payload, PTO2TaskSlotState &slot_state, PTO2SubtaskSlot subslot, int32_t block_idx ) { int32_t slot_idx = static_cast(subslot); uint64_t callable_addr = get_function_bin_addr(slot_state.task->kernel_id[slot_idx]); const CoreCallable *callable = reinterpret_cast(callable_addr); dispatch_payload.function_bin_addr = callable->resolved_addr(); auto &payload = *slot_state.payload; - int n = 0; - for (int32_t i = 0; i < payload.tensor_count; i++) { - dispatch_payload.args[n++] = reinterpret_cast(&payload.tensors[i]); - } - for (int32_t i = 0; i < payload.scalar_count; i++) { - dispatch_payload.args[n++] = payload.scalars[i]; - } - dispatch_payload.local_context.block_idx = block_idx; - dispatch_payload.local_context.block_num = slot_state.logical_block_num; - dispatch_payload.local_context.async_ctx = async_ctx; - dispatch_payload.args[PAYLOAD_LOCAL_CONTEXT_INDEX] = reinterpret_cast(&dispatch_payload.local_context); - dispatch_payload.args[PAYLOAD_GLOBAL_CONTEXT_INDEX] = reinterpret_cast(&dispatch_payload.global_context); // Early-dispatch: a task being staged (Hook 1 set early_dispatch_state to // STAGING before this call) is gated — the AICore must wait for the // DATA_MAIN_BASE high-32 doorbell. All other dispatches run on pickup. - dispatch_payload.not_ready = - (slot_state.payload->early_dispatch_state.load(std::memory_order_relaxed) == PTO2_EARLY_DISPATCH_STAGING) ? 1 : - 0; + if (payload.early_dispatch_state.load(std::memory_order_relaxed) == PTO2_EARLY_DISPATCH_STAGING) { + // Gated task: hand the idle AICore the source payload (non-zero = gate) and + // let it fill args[0..num_args) itself during its doorbell wait, instead of + // paying the arg-vector write on this scheduler thread. + dispatch_payload.src_payload = reinterpret_cast(&payload); + } else { + // Ready task: fill args here; src_payload = 0 signals AICore to run on pickup. + dispatch_payload.src_payload = 0; + int n = 0; + for (int32_t i = 0; i < payload.tensor_count; i++) { + dispatch_payload.args[n++] = reinterpret_cast(&payload.tensors[i]); + } + for (int32_t i = 0; i < payload.scalar_count; i++) { + dispatch_payload.args[n++] = payload.scalars[i]; + } + } + dispatch_payload.local_context.block_idx = block_idx; + dispatch_payload.local_context.block_num = slot_state.logical_block_num; + // AsyncCtx's slab pointers + capacity are prefilled once per (core, buf_idx) + // in init(); only task_token varies per dispatch. deferred_slab is never null + // on this path, so task_token is always the live task id (matches the old + // AsyncCtx::make(non-null) result). args[PAYLOAD_LOCAL_CONTEXT_INDEX] / + // [PAYLOAD_GLOBAL_CONTEXT_INDEX] are per-(core, buf_idx) constants, also + // prefilled in init(). + dispatch_payload.local_context.async_ctx.task_token = slot_state.task->task_id; } SchedulerContext::PublishHandle SchedulerContext::prepare_subtask_to_core( @@ -162,11 +180,12 @@ SchedulerContext::PublishHandle SchedulerContext::prepare_subtask_to_core( uint32_t buf_idx = reg_task_id & 1u; PTO2DispatchPayload &payload = payload_per_core_[core_id][buf_idx]; - DeferredCompletionSlab *deferred_slab = &deferred_slab_per_core_[core_id][buf_idx]; - deferred_slab->count = 0; - deferred_slab->error_code = PTO2_ERROR_NONE; - AsyncCtx async_ctx = AsyncCtx::make(slot_state.task->task_id, deferred_slab); - build_payload(payload, slot_state, subslot, async_ctx, block_idx); + // The deferred_slab is NOT reset here: it is init-cleared once and the + // completion path re-clears count only after a task actually recorded a + // deferred completion (count > 0), which is rare. A non-deferred task never + // touches count/error_code, so the slab stays clean without a per-dispatch + // write — keeping this cold per-core line off the dispatch path. + build_payload(payload, slot_state, subslot, block_idx); if (to_pending) { core_exec_state.pending_subslot = subslot;