Control-packet configuration hangs on device for any design spanning more than one column

[atassis]

What happened?

A design built with --load-pdi-to-ctrl-pkt hangs on device whenever it occupies more than one
column. The same design in a single column completes with correct data. The trigger is the column
span and nothing else.

Variants derived from test/npu-xrt/reconfigure_loadpdi_persistent_memtile/aie.mlir by moving tile
coordinates, with the overlay channels read out of the ctrlpkt_npu_lowered.mlir artifact:

design	overlay channels	result
all tiles in column 0	ctrlpkt_col0_mm2s_chan0	completes, correct data
all tiles in column 1, shim at (1,0)	ctrlpkt_col1_mm2s_chan0	completes, correct data
shim col 0, compute col 1	both	hang
shim col 1, compute col 0	both	hang

One overlay control-packet shim MM2S channel works, two hang, in either direction.
test/npu-xrt/reconfigure_loadpdi passes on this path only because it sits entirely in column 0.

Ruled out, each on device

1. The reconfiguration. It hangs with the second aiex.configure deleted, so with zero
   reconfigurations.
2. Mem tile data seeding. It hangs with initial_value removed.
3. An explicit aie.memtile_dma rather than objectFIFO-generated. This was my first attribution and
   it was wrong: a memtile-free design in one column works, and the memtile design in one column
   works.
4. Unused tile declarations and the unused mem tile lock. Removing both keeps the hang.
5. A column whose only tile is the shim. It hangs with three tiles in column 0.
6. A missing TCT completion-token return path. That was a real omission and is the second commit of
   [AIE] Generate the column control overlay for intermediate columns #3461. Fixing it makes the cross-column routing byte-match the working single-column case in
   input_physical.mlir, and the hang persists.

Both columns' syncs are emitted in the lowered sequence and both channels' DMAs are issued, so
nothing is unissued or unawaited in the IR.

Repro

Take test/npu-xrt/reconfigure_loadpdi_persistent_memtile/aie.mlir and move the compute tiles to
column 1, leaving the shim at (0,0):

aiecc --peano=<llvm-aie> --get-full-elf --load-pdi-to-ctrl-pkt --get-ctrlpkt \
      --no-xchesscc --no-xbridge aie.mlir
g++ -o test test.cpp -std=c++17 -lrt -lstdc++ -luuid -lxrt_coreutil
./test

Single-column variants return in 276-310 us. The cross-column ones never return. The device
recovered after every hang, no wedge.

Hardware / device

Ryzen AI Strix, RyzenAI-npu4, NPU firmware 1.1.2.64, 8 columns, aie.device(npu2).

Version

main at 94537df, built from source.

Relevant logs

xrt::run::aie_error: ERT_CMD_STATE_TIMEOUT
txn_op_idx = 0xFFFFFFFF

[andrej]

#3461 should fix this for many cases, but do not close this issue until we have verified that it is properly fixed. Namely, I haven't verified yet if the "bounding box" solution that that PR implements is correct:"

From that PR:

minCol..maxCol is conservative. The precise set is the columns the router actually traverses, and
the pass runs before routing, so it cannot know them; this matches the conservatism the row logic
already uses (0..maxRow).

Have you verified this? That is, does routing really always happen within the "bounding box" of tiles? I could imagine in a congested scenario, there could be routes that go outside of the bounding box (e.g. "out around" the rest of the design). Routing prefers shorter routes over longer ones, so this would only happen in a congested design.

If that's the case, we would need to solve this more properly: run the control-overlay pass after routing. But it's a bit of a chicken-and-egg problem.