perf(a2a3/tmr): shrink dispatch cold-write cost on the sync_start drain

docs/investigations/2026-07-aicore-fills-all-args-ready-path.md

@@ -0,0 +1,91 @@
+# AICore-side arg materialization for ALL dispatches (not just the gated path)
+
+**Date**: 2026-07-11
+**Verdict**: measured and rejected for the ready path — making the AICore
+fill its own `args[]` for **every** task adds **~1.0 µs to each task's
+setup** (`receive → start`), on the critical path, because a ready task has
+no idle doorbell gate to hide the fill behind. Offloading is only a win on
+the `not_ready` (early-dispatch) path, where the AICore is already spinning
+at the gate and absorbs the fill for free. The shipped design keeps the
+AICPU filling `args[]` for ready tasks and offloads only gated ones.
+
+## Question
+
+The tmr dispatch handoff writes the kernel `args[]` (tensor GM pointers +
+scalar values) into the per-core `PTO2DispatchPayload` on the AICPU. For
+gated (`not_ready` / early-dispatch) tasks we already offload that fill to
+the AICore — it does it during its doorbell wait, off the AICPU dispatch
+path (see the folded-gate `src_payload` design in the dispatch cold-write
+PR).
+
+Natural follow-on: why not offload the fill for **all** tasks and take the
+`args[]` write off the AICPU entirely? What does that cost the AICore?
+
+## What was tried
+
+An experiment build (not merged) that:
+
+- Adds a separate `gated` flag to `PTO2DispatchPayload` — with all tasks
+  filling from source, `src_payload` is always non-zero and can no longer
+  double as the gate flag (the shipped design folds `not_ready` into
+  `src_payload == 0`).
+- AICPU `build_payload` writes only `src_payload = &PTO2TaskPayload` and
+  `gated`, and **never** fills `args[]`.
+- AICore `aicore_executor` fills `args[0..num_args)` from `src_payload` for
+  **every** task (moved out of the gate branch), then gates only when
+  `gated`.
+
+Measured `paged_attention_unroll` (`--enable-l2-swimlane`, 1024 AICore
+tasks, a2a3 silicon) and read each task's AICore setup —
+`receive_to_start_cycles` from `l2_swimlane_records.json::aicore_tasks`
+(50 MHz → 20 ns/cycle).
+
+## Result
+
+| build | AICore setup (`receive → start`) |
+| ----- | -------------------------------- |
+| baseline (AICPU fills args) | mean **349 ns** / p50 320 ns |
+| experiment (AICore fills **all**) | mean **1356 ns** / p50 1340 ns |
+| **delta** | **+~1.0 µs/task (+50 cycles, ~4×)** |
+
+Correctness held (`paged_attention_unroll` passes), so the AICore fill is
+functionally equivalent — it just costs ~1 µs. The fill is AICore GM work:
+after the per-task whole-cache `dcci(ENTIRE_DATA_CACHE)`, the first reads of
+the source `PTO2TaskPayload` (counts, scalars) miss to HBM, then the loop
+computes `&tensors[i]` and writes `args[]` — all on a core whose GM-access
+latency is high. On the **ready** path there is no doorbell wait to overlap
+it with, so the whole ~1 µs lands on the `receive → start` setup.
+
+## Why the shipped design offloads only the gated path
+
+- **Gated (`not_ready`) tasks are staged before their producer completes**;
+  the AICore then spins at the doorbell gate. The ~1 µs fill runs inside
+  that otherwise-idle wait → net ~zero, and it takes the `args[]` write off
+  the AICPU dispatch path (the win we wanted).
+- **Ready tasks execute on pickup** — no gate, no idle window. Offloading
+  the fill there is a straight +~1 µs/task regression on the AICore
+  critical path. For an AICore-bound workload (e.g. `paged_attention`,
+  1024 tasks) that is a large, unhideable cost.
+
+So the asymmetry is intrinsic: offload where there is idle time to spend
+(gated), keep it on the AICPU where the AICore would otherwise stall
+(ready).
+
+## When to reconsider
+
+- If a future protocol gives ready tasks an idle window between pickup and
+  the point their inputs are guaranteed visible (today the kernel's input
+  `dcci` runs right after ACK), the fill could hide there — re-measure.
+- If the AICPU dispatch path becomes the proven end-to-end bottleneck for a
+  ready-dominated workload AND the AICore has spare setup budget (setup ≪
+  the WAIT/exec time), the trade could flip — but weigh against the ~1 µs
+  measured here, and note it scales with per-task arg count.
+
+## References
+
+- Dispatch cold-write PR (folded-gate `src_payload`, AICore fill on the
+  gated path, init prefill, deferred-slab-at-completion, prefetch):
+  hw-native-sys/simpler#1328.
+- [2026-07 — L2 swimlane AICore switch-overhead](2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md)
+  — the baseline ~0.28–0.35 µs AICore setup (`payload dcci + ACK`) this
+  experiment adds onto.

docs/investigations/README.md

@@ -84,6 +84,7 @@ that ...".
 
 Newest first.
 
+- [2026-07 — AICore-side arg fill for ALL dispatches (not just the gated path)](2026-07-aicore-fills-all-args-ready-path.md) — measured & rejected for the ready path: making the AICore fill its own `args[]` for every task adds ~1.0 µs to each task's `receive→start` setup (paged_attention_unroll: 349 ns → 1356 ns), because a ready task has no idle doorbell gate to hide the fill. Offload is a win only on `not_ready` (early-dispatch), where the AICore already spins at the gate; shipped design keeps the AICPU filling ready tasks (#1328)
 - [2026-07 — L2 swimlane AICore: switch-overhead source + FIN-early reorder & ACK-gate](2026-07-aicore-swimlane-switch-overhead-and-ack-gate.md) — measured: the ~0.8 µs inter-task switch is the record write-back `dcci(record,OUT)+dsb` (~0.5 µs) + payload setup (~0.28 µs), inherent and not reducible by moving FIN; the WAIT gap (p99 ~700 µs) dominates decode. Shipped: sample `end_time` after an early FIN, and an AICPU ACK-gate on buffer rotation (release the old buffer only when AICore ACKs the new buffer's first task) to close the FIN-before-record boundary race the reorder introduced
 - [2026-07 — Removing PTO2LocalReadyBuffer exposed a missing dcci in EP dispatch](2026-07-local-buffer-removal-ep-combine-regression.md) — RESOLVED in #1245: local-buffer removal changed dispatch timing and unmasked a latent kernel bug (dispatch never dcci'd `recv_count_out` to HBM → local_expert read count=0 → all-zero output); fixed with a one-line dcci in the example kernel
 - [2026-06 — Gating the two residual profiling enable() calls on the orch/scheduler hot path](2026-06-orch-profiling-enable-gates-hot-path.md) — gated under existing `SIMPLER_DFX`; magnitude unmeasured, no new macro

src/a2a3/runtime/tensormap_and_ringbuffer/aicore/aicore_executor.cpp

@@ -144,9 +144,9 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in
             // (receive_time → start_time). Stored as a 32-bit delta
             // `start_time - receive_time`.
             //
-            // Common path (not_ready == 0): the new task_id is itself the ready
+            // Common path (src_payload == 0): the new task_id is itself the ready
             // signal, so receive_time is the true ready moment. Early-dispatch path
-            // (not_ready == 1): receive_time stays at pickup — before the
+            // (src_payload != 0): receive_time stays at pickup — before the
             // doorbell wait — so it precedes the producer's end_time; the host
             // folds it to start_time for those tasks (detected when receive
             // precedes the producer task's end_time).
@@ -160,16 +160,39 @@ __aicore__ __attribute__((weak)) void aicore_execute(__gm__ Runtime *runtime, in
             // Invalidate payload buffer (AICPU updates its content each dispatch)
             dcci(exec_payload, ENTIRE_DATA_CACHE);
 
-            // Early-dispatch gate. A not-ready task was staged on
-            // this core before its dependencies resolved; wait until AICPU rings
-            // the doorbell (DATA_MAIN_BASE high 32 == task_id) before executing.
-            // The ACK is deferred until AFTER the gate so the scheduler keeps the
-            // core off-limits (pending_occupied stays set, no ACK->pending_freed)
-            // while the task is gated — preventing a real task from being
-            // dual-issued behind it. The kernel's own input dcci runs inside
-            // execute_task() below — strictly AFTER this gate — so predecessor
-            // outputs are visible. not_ready == 0 (the common path) skips this.
-            if (exec_payload->not_ready) {
+            // Early-dispatch gate. A gated task was staged on this core before its
+            // dependencies resolved; wait until AICPU rings the doorbell
+            // (DATA_MAIN_BASE high 32 == task_id) before executing. The ACK is
+            // deferred until AFTER the gate so the scheduler keeps the core
+            // off-limits (pending_occupied stays set, no ACK->pending_freed) while
+            // the task is gated — preventing a real task from being dual-issued
+            // behind it. The kernel's own input dcci runs inside execute_task()
+            // below — strictly AFTER this gate — so predecessor outputs are visible.
+            // src_payload == 0 (the common ready path) skips this; a non-zero
+            // src_payload is both the gate flag and the source PTO2TaskPayload.
+            if (exec_payload->src_payload != 0) {
+                // AICPU staged only src_payload, not the arg vector — fill
+                // args[0..num_args) ourselves now, while we are idle waiting for
+                // the doorbell. The whole-cache dcci(ENTIRE_DATA_CACHE) above
+                // already invalidated src's lines, so tensor_count/scalar_count/
+                // scalars read coherently with the orchestrator's submit writes.
+                // args[SPMD_LOCAL_CONTEXT_INDEX]/[SPMD_GLOBAL_CONTEXT_INDEX] are
+                // still written by the AICPU (num_args <= 48 never reaches them).
+                __gm__ char *src = reinterpret_cast<__gm__ char *>(exec_payload->src_payload);
+                int32_t tensor_count = *reinterpret_cast<__gm__ int32_t *>(src + PTO2_TASKPAYLOAD_TENSOR_COUNT_OFFSET);
+                int32_t scalar_count = *reinterpret_cast<__gm__ int32_t *>(src + PTO2_TASKPAYLOAD_SCALAR_COUNT_OFFSET);
+                __gm__ uint64_t *src_scalars =
+                    reinterpret_cast<__gm__ uint64_t *>(src + PTO2_TASKPAYLOAD_SCALARS_OFFSET);
+                int n = 0;
+                for (int32_t i = 0; i < tensor_count; i++) {
+                    exec_payload->args[n++] = reinterpret_cast<uint64_t>(
+                        src + PTO2_TASKPAYLOAD_TENSORS_OFFSET + i * PTO2_TASKPAYLOAD_TENSOR_STRIDE
+                    );
+                }
+                for (int32_t i = 0; i < scalar_count; i++) {
+                    exec_payload->args[n++] = src_scalars[i];
+                }
+                OUT_OF_ORDER_STORE_BARRIER();
                 while (true) {
                     // Honor teardown: shutdown overwrites the low half with EXIT.
                     // Check it on the doorbell-match iteration too, so an EXIT that

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/pto2_dispatch_payload.h

@@ -34,6 +34,7 @@
 
 #pragma once
 
+#include <stddef.h>
 #include <stdint.h>
 
 #include "arg_direction.h"
@@ -57,6 +58,18 @@ static_assert(
     "GLOBAL_CONTEXT_INDEX out of sync with intrinsic.h"
 );
 
+// Byte offsets into PTO2TaskPayload used by AICore to materialize args[] on the
+// not_ready (early-dispatch) path. The AICore .o does not include
+// pto_runtime2_types.h, so it reads tensor_count / scalar_count / tensors[] /
+// scalars[] through these constants. The AICPU side (scheduler_dispatch.cpp)
+// static_asserts them against offsetof where the full struct is visible, so any
+// PTO2TaskPayload layout drift fails the build rather than corrupting args[].
+constexpr uint32_t PTO2_TASKPAYLOAD_TENSOR_COUNT_OFFSET = 0;
+constexpr uint32_t PTO2_TASKPAYLOAD_SCALAR_COUNT_OFFSET = 4;
+constexpr uint32_t PTO2_TASKPAYLOAD_TENSORS_OFFSET = 576;
+constexpr uint32_t PTO2_TASKPAYLOAD_SCALARS_OFFSET = 4672;
+constexpr uint32_t PTO2_TASKPAYLOAD_TENSOR_STRIDE = 128;  // sizeof(Tensor)
+
 /**
  * Per-core dispatch payload: function address + args[] + SPMD context.
  *
@@ -68,24 +81,40 @@ static_assert(
  * concurrently dispatched cores.
  */
 struct alignas(64) PTO2DispatchPayload {
-    uint64_t function_bin_addr;            /**< Kernel entry address in GM (set by Scheduler) */
-    uint64_t args[PTO2_DISPATCH_MAX_ARGS]; /**< Kernel arguments (GM pointers + scalars + ext params) */
+    // === Cache line 0 (64B): control block, the only line written per dispatch ===
+    // function_bin_addr, local_context.{block_idx,block_num,async_ctx.task_token}
+    // and src_payload are the per-dispatch writes; async_ctx's slab pointers +
+    // capacity are cold (prefilled once at init) but ride this hot line for free.
+    // Sized to exactly 64B so both dispatch paths write one control line: the
+    // ready path (src_payload = 0) then also fills args[0..num_args); the gated
+    // path (src_payload = &PTO2TaskPayload) leaves args[] to the idle AICore.
+    uint64_t function_bin_addr; /**< Kernel entry address in GM (set by Scheduler). */
 
-    /** Per-dispatch context: block_idx and block_num.
-     *  Written by build_payload() before each dispatch.
-     *  args[SPMD_LOCAL_CONTEXT_INDEX] points here. */
+    /** Per-dispatch context: block_idx/block_num (hot) + async_ctx (task_token hot,
+     *  slab pointers + capacity prefilled once at init). args[SPMD_LOCAL_CONTEXT_INDEX]
+     *  points here. */
     LocalContext local_context;
 
-    /** Per-core global context: sub_block_id (AIV lane identity).
-     *  Initialized once by init_global_context() at runtime startup.
-     *  args[SPMD_GLOBAL_CONTEXT_INDEX] points here. */
-    GlobalContext global_context;
+    /** Early-dispatch gate AND source pointer, folded into one field. 0 = ready:
+     *  AICore executes on pickup (args[] already filled by the AICPU). Non-zero =
+     *  gated: the value is the source PTO2TaskPayload address; the AICore fills
+     *  args[0..num_args) from it, then waits for the doorbell (DATA_MAIN_BASE
+     *  high 32 == this dispatch's reg_task_id) before executing. A payload address
+     *  is never 0, so the gate flag is lossless. */
+    volatile uint64_t src_payload;
 
-    /** Early-dispatch gate. 0 = ready: AICore executes on pickup.
-     *  1 = not-ready: AICore waits until AICPU rings the doorbell
-     *  (DATA_MAIN_BASE high 32 == this dispatch's reg_task_id) before executing. */
-    volatile uint32_t not_ready;
-    uint8_t reserved_payload_abi_pad[4];
+    // === Cache lines 1..7: kernel argument vector ===
+    /** [0..num_args) = GM tensor pointers + scalar values; [SPMD_LOCAL_CONTEXT_INDEX]
+     *  = &local_context, [SPMD_GLOBAL_CONTEXT_INDEX] = &global_context (both prefilled
+     *  once at init). On the gated path the AICPU leaves [0..num_args) unwritten and
+     *  the idle AICore fills them from src_payload during its gate wait. */
+    uint64_t args[PTO2_DISPATCH_MAX_ARGS];
+
+    /** Per-core global context: sub_block_id (AIV lane identity). Cold: written once
+     *  at init, never per dispatch, so it lives in the tail (not the CL0 control
+     *  block). args[SPMD_GLOBAL_CONTEXT_INDEX] points here. */
+    GlobalContext global_context;
+    // No explicit tail padding: alignas(64) rounds sizeof up to 512 (8 cache lines).
 
     static_assert(sizeof(args[0]) == 8);
     static_assert(
@@ -95,3 +124,5 @@ struct alignas(64) PTO2DispatchPayload {
 };
 
 static_assert(sizeof(PTO2DispatchPayload) == 512, "PTO2DispatchPayload hardware ABI size drift");
+static_assert(offsetof(PTO2DispatchPayload, args) == 64, "args[] must start at cache line 1 (control block = CL0)");
+static_assert(offsetof(PTO2DispatchPayload, src_payload) < 64, "src_payload (gate) must live on the CL0 control block");

src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler_cold_path.cpp

@@ -1033,6 +1033,35 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) {
         }
     }
 
+    // Prefill the per-dispatch AsyncCtx constant fields once. Of AsyncCtx's five
+    // fields, four are constant for a given (core, buf_idx): the three pointers
+    // target the fixed deferred_slab_per_core_[core][buf] members, and capacity is
+    // MAX_COMPLETIONS_PER_TASK. Only task_token varies per dispatch, so build_payload
+    // writes just that; these constants survive across dispatches because the
+    // payload buffer is never zeroed between them.
+    // The two context-pointer args are also per-(core, buf_idx) constants — they
+    // target this buffer's own local_context / global_context — so prefill them
+    // here too and drop them from the per-dispatch build_payload writes. This keeps
+    // the per-dispatch write footprint on the CL0 control block only (args[48]/[49]
+    // live on a later line).
+    for (int32_t core_id = 0; core_id < RUNTIME_MAX_WORKER; core_id++) {
+        for (int32_t buf = 0; buf < 2; buf++) {
+            PTO2DispatchPayload &dp = payload_per_core_[core_id][buf];
+            AsyncCtx &ac = dp.local_context.async_ctx;
+            volatile DeferredCompletionSlab *slab = &deferred_slab_per_core_[core_id][buf];
+            ac.completion_count = &slab->count;
+            ac.completion_error_code = &slab->error_code;
+            ac.completion_entries = &slab->entries[0];
+            ac.completion_capacity = MAX_COMPLETIONS_PER_TASK;
+            // Clear the slab once here; thereafter only the completion path re-clears
+            // count (and only when a deferred task dirtied it), never per dispatch.
+            slab->count = 0;
+            slab->error_code = PTO2_ERROR_NONE;
+            dp.args[PAYLOAD_LOCAL_CONTEXT_INDEX] = reinterpret_cast<uint64_t>(&dp.local_context);
+            dp.args[PAYLOAD_GLOBAL_CONTEXT_INDEX] = reinterpret_cast<uint64_t>(&dp.global_context);
+        }
+    }
+
     func_id_to_addr_ = runtime->dev.func_id_to_addr_;
 
     return 0;
@@ -1047,11 +1076,14 @@ void SchedulerContext::deinit() {
     }
 
     // No per-core memset of payload_per_core_ / deferred_slab_per_core_ here
-    // (~300 KB across all cores). Both are fully re-initialized at dispatch
-    // before they can be read: dispatch_task sets deferred_slab->count = 0 /
-    // error_code = NONE and build_payload() overwrites every payload field
-    // (function addr, args[], contexts, not_ready) on the exact [core][buf_idx]
-    // about to run. The consumer side cannot reach a stale slot either: the
+    // (~300 KB across all cores). They are re-initialized before they can be read:
+    // build_payload() overwrites the per-dispatch payload fields (function addr,
+    // args[0..num_args) or src_payload, block_idx/block_num, async_ctx.task_token)
+    // on the exact [core][buf_idx] about to run; the async_ctx slab pointers +
+    // capacity, the two context-pointer args, and the deferred slab (count = 0 /
+    // error_code = NONE) are all cleared once per run in init() — the slab is
+    // thereafter re-cleared only by the completion path after a deferred task.
+    // The consumer side cannot reach a stale slot either: the
     // drain only services a core's running_reg_task_id, and the loop above
     // already reset every core_exec_states_[].running/pending_reg_task_id to
     // AICPU_TASK_INVALID — so no FIN for an undispatched slot is processed, and