Chess (xchesscc) AIE2P cores time out under raw xrt boot while Peano cores complete (NPU2)

[bong-water-water-bong]

Summary

Chess-built (xchesscc) AIE2P cores time out under the raw xrt boot (ERT_CMD_STATE_TIMEOUT) that mlir-aie/IRON's runtime uses, while Peano-built cores from the identical design complete. This holds for every core style tried: the xbridge/me-runtime link, the bare peano-lld link (--xbridge=false), and with the chess_intrinsic_wrapper fences/schedulers removed. The NPU transaction buffer (.bin) is byte-identical between the working Peano and hanging Chess builds, so host-side control is identical; the difference is entirely inside the chess-compiled core ELF.

Environment

• mlir-aie: 1.4.2.dev16+g7e00b57 wheel (also reproduced intent with the v1.4.2 tag built from source)
• llvm-aie: 22.0.0.2026082001+84660bc3
• Vitis AIE tools: 2025.2 (aie2ps target; aie2p name aliased to aie2ps) and 2026.1 — same hang on both
• Device: Ryzen AI NPU2 (npu2_40, AIE2P), Linux, xrt
• IRON: amd/IRON, --compiler chess, silu operator test (single column)

Repro

export AIETOOLS_ROOT=<vitis>/aietools
export PATH=$AIETOOLS_ROOT/bin:<mlir_aie>/bin:<llvm-aie>/bin:$PATH   # llvm-aie bin provides `clang` for the bare link
export XILINXD_LICENSE_FILE=<lic>
cd <IRON>
python -m pytest iron/operators/silu/test.py::test_silu[iter0-input_length_1024-num_aie_columns_1-num_channels_1-tile_size_1024] \
    --compiler chess

Compilation succeeds end-to-end (front-end -> me/bare link -> xclbin). The kernel loads and starts, then the ERT command never completes:
HostRuntimeError: Kernel returned ert_cmd_state.ERT_CMD_STATE_TIMEOUT
The identical test with --compiler peano passes.

Evidence

1. The txn is identical: cmp on the generated *.npu2.bin (the NPU instruction/control buffer) between peano and chess runs = byte-identical (300 B for this design). So the host boots the cores identically.

2. Chess core ELFs (all hang) vs peano core ELF (works) for the same design:

   • peano (bare, clang): 1,956 B; __start (32 B) -> main; objectFifo acquire inlined as acq #51,-1 / acq #48,-1, releases rel #50,1 / rel #49,1.
   • chess xbridge (me-runtime, libme): 12,048 B; _main_init ctor/init prelude before main; acquire via llvm___aie2p___acquire wrapper -> plain acq r0,r1 with the same lock ids/values.
   • chess bare (peano-lld link of chess objects, me-primitive stubs, --xbridge=false): 5,444 B; no me runtime; same wrapper-based acquire with same lock ids/values.
   • chess bare with the wrapper's llvm.chess_memory_fence / chess_separator_scheduler calls removed: still hangs.

3. The core design is an infinite server loop (scf.for to 2^63-1); the ERT completes via the shim-DMA drain after the core consumes one tile. A hang therefore means the core never completes its first acquire/compute/release cycle (or never starts). Peano proves the protocol works with these exact lock values.

4. llvm-objdump decodes peano cores fully but prints <unknown> for many chess VLIW bundles — a chess-specific disassembler or the ISS is needed to inspect the chess main loop. (--get-aiesim, per Tests that still require Chess after #3451, and what each one is waiting on #3479, would be the right tool, but the pip/dev16 aiecc predates it and the current-main aiecc cannot parse the dev16-era IRON design MLIR because aie.objectfifosubview was removed upstream.)

Expected behavior

Chess-compiled cores should run under the same xrt boot as peano cores, or the chess+device path should explicitly refuse/diagnose rather than hanging.

Suspected areas

• Chess (me-product) codegen emits instructions/metadata (.tctmemtab, .rtstab, .eoltab, .chesstypeannotationtab, me_primitive::* references) that assume the aiecompiler/EV boot environment; the raw xrt core-ELF boot (what IRON/peano uses) may not establish whatever the chess code expects.
• Or a chess codegen sequence executes incorrectly on real silicon (needs ISS to confirm vs hypothesis above).

Happy to provide artifacts (both core ELFs, the work dir, full logs) and to test any candidate fix or patch.

[bong-water-water-bong]

Update: narrowed to the chess-compiled core main (not the kernel), via a mixed-core experiment

Additional experiments isolate the hang further:

1. Mixed core test: chess-compiled core main/sync + peano-compiled kernel object linked into one bare core ELF (the peano-lld bare link accepts both, unlike xbridge) -> still ERT_CMD_STATE_TIMEOUT. So the chess-compiled kernel code is NOT the problem; the chess-compiled per-core main (objectFifo acquire/compute/release loop + startup) is.

2. The chess main machine code is structurally equivalent to the working peano main (same lock ids 51/48, values -1/1, plain acq r0,r1 after removing the wrapper fences/schedulers, same tile-buffer addresses 0x78000/0x7c000/0x70400/0x74000, same infinite server-loop shape, core ELF program headers/entry identical). Peano kernels run fine inside peano cores on the same device/flow.

3. An internal project hit the same class of defect previously (chess-compiled cores not executing correctly on this exact NPU2 silicon; "core PC stayed 0" in their cycle-accurate aiesim attempt; all-zeros on hardware; multiple chesscc aie2p codegen bugs documented: leaf-function ret-first codegen, memref base-pointer delivery, pointer-arith miscompiles). Their conclusion matched: a chess-core/kernel runtime-execution defect (control flow / ABI / hardware-compat), needing the chess ISS with a proper ps.so testbench to pin the exact failing instruction/state.

Would it be possible to run this design's chess core through the mlir-aie --get-aiesim/aiesimulator flow (or the chess ISS) on your side? The mixed-core result suggests the failing state is early in the chess core main (before/during the first acquire), which a simulator trace would pinpoint quickly. Happy to provide the work dir + core ELFs.

[bong-water-water-bong]

Update: the "userspace" hypothesis is tested and ruled out — the gap is the core content itself

Follow-up experiments:

1. Ran the design through FLM's exact runtime userspace. mlir-aie's NPU_RUNTIME=hrx backend dispatches the same xclbin+insts through the pinned libhrx release (jtuyls/hrx flm-hrx-amdxdna-v2026.07.30 = the exact asset mlir-aie's utils/hrx-release.env consumes, SHA-verified, with hrx_buffer_map_with_mode).

   • Peano-built silu through HRX: PASSES (integration + xclbin format validated).
   • Chess-built silu through HRX: same ERT state-8 timeout as the XRT insts and XRT full-elf paths.
     So the loader/userspace is not the differentiator: peano cores run under every userspace, chess cores under none.

2. FLM's own chess cores DO run on this NPU (the 35B MoE serves via these exact xclbins). Their mm.xclbin = the same xclbin2 + AIE_PARTITION + IDPP-PDI structure as our aiecc output, and their PDI carries real acquire-based AIE2P code. So the container/boot format is equivalent; the difference is inside the core program (entry/init/crt or codegen) that AMD's internal chess build produces vs mlir-aie's chess output.

Would appreciate: (a) which chesscc flags/crt/PDI flow AMD used for those flm-hrx-amdxdna kernels, or (b) an extractable core image from one of the xclbins to diff against our chess core. Happy to share our full chess core ELFs + build recipes.

[bong-water-water-bong]

Update: FLM's own xclbin cores = peano/FLM-style codegen (immediate acquires, no me-init); mlir-aie chess cores differ

I extracted the core program from AMD's working mm.xclbin (the 35B MoE kernel set) and compared instruction signatures:

Core	acq #imm (immediate)	acq r0,r1 (register)	me-runtime _main_init
FLM mm.xclbin cores	128	0	absent
Peano silu core (RUNS here)	2	0	absent
mlir-aie chess silu core (TIMES OUT here)	0	1	present

So the cores that execute on this NPU2 (FLM's + peano's) share: no me-runtime _main_init prelude + immediate-form lock acquires. The mlir-aie chess output differs on both counts (me-init prelude; acquires routed through llvm___aie2p___acquire wrapper functions that emit register-form acq r0,r1). I attempted to force chess to inline the wrapper (alwaysinline, fences removed) - chess still emits the register-form wrapper call. The chess model's own me_chess.lib only exposes register-arg acquire builtins, so the immediate form is unreachable via the chesscc intrinsic path.

Two questions for you: (a) is register-form acq r0,r1 (or the me-init prelude) known-broken/unimplemented on AIE2P silicon - i.e., would either explain a core that never completes? (b) which chess build/flow produced the flm-hrx-amdxdna xclbin cores (immediate acquires + no me-init)? Replicating that flow is the likely unblock.

[bong-water-water-bong]

Update: guarded-vs-plain acquire delta found; plain-lock shim test still times out

Two more findings:

1. Chess emits the GUARDED acquire variant; peano + the FLM xclbin cores use the PLAIN one. Encodings (both decode as acq r0,r1): chess wrapper = 30 18 12 20 (via chessintr_void_acquire_guarded); peano (llvm-aie __builtin_aie2p_acquire) = 18 18 12 10. The 2025.2 aie2ps model exposes only the guarded + no_fence acquire intrinsics, so the chesscc path cannot emit the plain form directly.

2. Plain-lock shim test: compiled the llvm___aie2p___acquire/release wrappers with PEANO (plain acq/rel), made the chess wrapper .ll declarations-only, and linked the shim into the chess-built core (bare path). The resulting core verifiably contains plain acq r0,r1 (1x) + plain rel r0,r1 (1x), zero guarded forms, with correct call-site args (r0=lock 51/48, r1=-1) -> still ERT_CMD_STATE_TIMEOUT.

Complete isolation matrix (all still time out): me-runtime / bare / fence-free / no-op-acquire / plain-lock-shim core variants; XRT insts / full-elf / HRX userspaces; chess-main + peano-kernel mixed core. Peano passes all of these. The failing component is the chess-compiled core-main CODE itself on real silicon (control flow / memory access / ABI), independent of the loader, the me-init prelude, the guarded-vs-plain lock instruction, and the kernel object.

This looks like a chesscc aie2p codegen vs. AIE2P-silicon execution incompatibility that even a byte-identical design + txn (peano comparison) does not explain. If there is a known-good chesscc build/flags AMD uses for the flm-hrx-amdxdna xclbins, or a silicon errata around chesscc register-form lock ops / me-product codegen, that would directly unblock us.

[bong-water-water-bong]

RESOLVED: root cause = the guarded acquire; the fix uses the 2024-era plain-acquire chess

Chess-compiled cores now RUN and PASS reference tests on the NPU2. The resolution:

1. Root cause: Vitis 2025.2/2026.1 aie2ps chess models only expose the guarded acquire intrinsic (chessintr_void_acquire_guarded), which chesscc lowers to a distinct instruction encoding (30 18 12 20 = guarded acq). On this NPU2 silicon a core stuck on that guarded acquire never completes (ERT_CMD_STATE_TIMEOUT), while the DMA/design side uses the plain protocol.

2. The fix: compile the chess kernels with the Ryzen-AI 1.3.0 vitis_aie_essentials (2024) aie2p chess, whose model still has the unguarded acquire and emits the plain acq encoding (18 18 12 10 — the same form peano and AMD's flm-hrx-amdxdna xclbin cores use). Plus: link cores with the peano-lld bare path (--xbridge=false; the me-runtime prelude also hangs) with me_primitive stubs + libsoftfloat.a for f32 conversions.

3. Verified on strixhalo (NPU2, /dev/accel0): pytest iron/operators/axpy/test.py --compiler chess = 20/20 passed; silu = 20/20 passed; peano regression clean (axpy + silu 20/20 each). Compile+link+load+execute+correctness all work end-to-end.

So the guarded-acquire codegen (2025.2+ models) vs plain-acquire (2024 model + peano) is the silicon-visible difference behind #3690. The 2025.2/2026.1 chesscc guarded acquire either needs a silicon fix or the aiecc chess flow should emit the plain form. Happy to share the full recipe/patches.