3rd distinct xclbin hw_context in one process silently produces garbage (ERT reports COMPLETED); corrupts Llama prefill

[bong-water-water-bong]

Summary

In a single process, the 3rd distinct xclbin hw_context created by CachedXRTRuntime executes its kernel but the kernel silently produces garbage: the ERT command reports ERT_CMD_STATE_COMPLETED, yet the output BOs contain wrong data. The first two distinct contexts keep working (their handles remain valid), and re-running an old context still gives correct results. Repeating the same xclbin context any number of times is fine; only creating a new context after two others are resident is broken.

This silently corrupts any multi-operator application. In the IRON Llama-3.2-1B NPU app (which uses ~15 distinct xclbin ops in prefill plus a fused decode ELF), the prefill is corrupted and generation becomes nondeterministic: two runs with the same fixed seed produce different text.

Environment

• Ryzen AI MAX+ 395 (Strix Halo), XDNA2 NPU, amdxdna driver, XRT 2.21.75
• CachedXRTRuntime (python/utils/hostruntime/xrtruntime/hostruntime.py), NPU_CONTEXT_CACHE_SIZE["npu2"] = 32 (no eviction in the repro below — only 3 contexts used)

Related: #3594 (hw context cache size / resident context limits).

Reproduction

# 3 GEMM ops with distinct shapes -> 3 distinct xclbins -> 3 distinct hw_contexts
ops = [GEMM(M=1024,K=512,N=1024,...).compile().get_callable(),   # ctx A
       GEMM(M=2048,K=512,N=2048,...).compile().get_callable(),   # ctx B
       GEMM(M=1024,K=1024,N=1024,...).compile().get_callable()]  # ctx C (NEW)
for i, op in enumerate(ops):
    run(op)   # verify vs numpy int32 reference
# ctx A: exact   ctx B: exact   ctx C: 1,047,753 / 1,048,576 elements WRONG

Deterministic: the 3rd (and any later) new context fails; older contexts remain correct (verified by re-running ctx A after ctx C failed).

Impact example: Llama-3.2-1B on NPU

The IRON llama app (iron/applications/llama_3.2_1b/llama_npu.py) runs ~15 distinct xclbin ops in prefill, so prefill ops past the 2nd are corrupted. With torch.manual_seed(1608560892) fixed, two runs produce different continuations:

• run 1: SCENE I. King Leir's Palace. (+ generated text)
• run 2: SCENE I. King Leontes and Pol...

The CPU reference (same seed) produces a third different output. The decode itself (a single fused ELF, one context) executes correctly, so tok/s numbers are unaffected — only correctness is silently lost.

Suggested investigation

• How many hw_contexts can be resident/mapped on the array at once (see Double check hw context cache size #3594; npu_task_max reports 16 hwctx slots, temporal_only=1 allows up to 32 resident)?
• Why does a newly created context after 2 existing ones mis-execute instead of erroring?
• CachedXRTRuntime should detect this condition (and/or evict old contexts when creating a new one beyond the driver's resident limit) rather than letting the kernel silently return garbage.

[bong-water-water-bong]

Related to #3594 (hw context cache size / resident context limits). My repro fails at the 3rd distinct context — well below the 16-slot / 32-resident numbers discussed there — so I'm not sure this is the same limit; worth checking whether it is a per-context BO count or something else. Full repro in the issue body.

[bong-water-water-bong]

Refined trigger: the failure is deterministic when each op's XRTTensors are created between kernel runs (3/3 trials fail at the 3rd distinct context). If all tensors are created upfront before any kernel runs, all 3 contexts pass (1/1). Each XRTTensor constructor opens its own pyxrt.device(0) handle, so the failing pattern interleaves new device handles + BOs with live multi-context execution. Might be worth checking how BOs created on a separate device handle interact with a context on the runtime's handle.