One Tenstorrent vector test fails at VLEN=256 / LMUL=m4 (rv64v_14)

[tommythorn]

Symptom

One test out of the Tenstorrent suite fails, and only this one:

tests/tenstorrent/run.sh machine/paging_bare/rv_v
...
machine/paging_bare/rv_v/xlen_256/vlmul_m4     15 / 16     1 FAIL

The test is xlen_256/vlmul_m4/rv64v_14. Everything else in the suite passes
(8999/8999 scalar, 95/96 vector).

What is known

Self-check mismatch, not an exception: the test reaches test_failed
directly from a bne, having compared an expected 0x1d9c against
simmerv's 0x1d9d.

The trace around it:

vslide1down.vx v24, v0, x0      # funct6=001111 OPMVX
vmv.x.s a0, v24                 # -> 0x18f3, matches expected
vslide1down.vx v0,  v24, x0
vmv.x.s a0, v0                  # -> 0x1d9d, expected 0x1d9c   FAIL

What has been ruled out

The obvious reading -- an off-by-one in vslide1down.vx at LMUL=m4 -- does
not survive checking. tests/vector/gen_vtest.py previously generated the
slides only at LMUL=m1; 3cee2d8 adds vslideup.vx, vslide1up.vx,
vslide1down.vx and vslidedown.vx at mf2/m2/m4 for SEW 8 and 32. The QEMU
differential then passes 2018/2018 identical transcript lines at both
VLEN=128 and VLEN=256. So simmerv agrees with QEMU on these slides at m4.

The two values differ by one in the low bit, which looks more like an
arithmetic result than a misplaced element -- suggesting the wrong value was
produced by something earlier in the test and the slides merely moved it
into view.

Suggested next step

Run the same ELF under qemu-system-riscv64 -M spike (which implements the
HTIF tohost the test ends with) and diff the register trace against
simmerv's -t output to find the first divergence, rather than reasoning
from the failing comparison backwards. The ELF is reproducible with
tests/tenstorrent/run.sh machine/paging_bare/rv_v; the built binary is
left in tests/tenstorrent/out/.

[tommythorn]

Attempted the QEMU-differential plan from the issue body. It did not work, and the reason is worth recording so nobody repeats it.

QEMU's spike machine will not run these tests here. It rejects the ELFs outright with HTIF tohost must be 8 bytes (the suite emits tohost as a size-0 NOTYPE symbol). Patching the symbol table to size 8 / OBJECT gets past that, but QEMU then just spins — and so does a control test that simmerv passes (vlmul_m4/rv64v_13), under -M spike -bios none -kernel, plain -bios <elf>, and with -smp 1. Since the control fails too, this says nothing about simmerv; the invocation or the machine model is the problem. Someone who knows QEMU's spike machine better may get further — the patch script is trivial (rewrite st_size/st_info for tohost/fromhost).

That detour did find a real simmerv bug, now fixed in db3257b. load_image only recorded NOTYPE symbols, so an image declaring tohost as sized data loaded with tohost_addr unset — the riscv-tests handshake never fired and the guest ran forever with no diagnostic. Which is to say: an ELF built to satisfy QEMU's 8-byte requirement was precisely the kind simmerv would hang on. Both directions now work.

So this issue is unchanged in substance: rv64v_14 still fails, still with expected 0x1d9c vs simmerv's 0x1d9d, and the QEMU differential still covers slides at m4 at both VLEN=128 and 256 without complaint (2018/2018 identical).

Next ideas, in rough order of promise:

1. Dump simmerv's vector state around the failure (-t plus the vector CSRs) and hand-evaluate the few instructions that produce that element. The values differ by one in the low bit, so a rounding or carry detail somewhere is likelier than a misplaced element.
2. Build whisper (the suite's own reference, submoduled in the repo) rather than fighting QEMU's spike machine. infra/quals.py --iss whisper is the path the suite is designed around.
3. Bisect the test: it is generated source, so the failing sub-test can be extracted into a minimal .S and run through tests/vector/run.sh's QEMU comparison, which is known to work.

[tommythorn]

Followed up on the debugger idea. It works, and it changes the picture — though not yet the verdict.

QEMU does run these tests; only the exit handshake fails

-d in_asm shows QEMU reaching the end-of-test HTIF loop:

0x800005bc: auipc t0, 265974
0x800005c0: ld    t0, -1388(t0)   # tohost address
0x800005c4: sw    gp, 0(t0)       # verdict
0x800005c8: j     -12             # spin

So the earlier "QEMU hangs" reading was wrong: QEMU finishes, QEMU's HTIF just never acks the write. The verdict is sitting in gp (1 = pass, 3 = fail) and gdb reads it straight out:

qemu-system-riscv64 -M spike ... -S -gdb tcp::PORT &
gdb-multiarch -batch -ex 'target remote :PORT' -ex 'break *0x800005c4' \
              -ex continue -ex 'printf "gp=%d\n", $gp'

That is a working technique for getting a verdict out of QEMU for any test in this suite.

But QEMU is not a valid reference as configured

vlmul_m4/rv64v_13 — which simmerv passes — reports gp=3 (fail) under QEMU, even after matching the suite's own reference ISA from infra/whisper_config_vlen_256.json (rv64imafdcvsuzfh_zba_zbb_zbc_zbs, plus svadu=false for its page_fault_on_first_access). Since the control fails, nothing QEMU says about rv64v_14 is admissible.

Breaking at test_failed shows why QEMU fails it: mcause=1 (instruction access fault) at 0x8000021c, which is the mret at the end of the startup sequence — i.e. before any test body runs.

Likely cause, and it is fixable

tohost in these images sits at 0x80000800 — inside the first page of .text. QEMU's spike machine maps an HTIF MMIO page at tohost & ~0xfff, which shadows 0x80000000–0x80001000, exactly covering the faulting fetch. That would explain both symptoms at once: the instruction access fault in early .text, and why the sw to tohost never acks.

If that is right, QEMU becomes usable by relinking the tests with tohost on a page of its own, away from .text. The linker scripts are per-test and already place a .io_htif section (at 0x70000000 in some variants), so this may be a one-line change to the script plus a rebuild.

Worth trying before falling back to whisper — a working QEMU reference is far more convenient than building whisper, and the gdb verdict technique above already covers the missing HTIF ack.

[tommythorn]

Debugger-on-QEMU works, and it settles what QEMU can and cannot tell us. Correcting my previous comment along the way.

The technique (works, reusable)

QEMU never acks the HTIF write for any of these tests — it always ends up spinning in the end-of-test loop. But the verdict is in gp, so gdb reads it out:

qemu-system-riscv64 -M spike ... -S -gdb tcp::$PORT &
bp=0x$(riscv64-linux-gnu-objdump -d $ELF | grep -m1 'sw	gp,0(t0)' | cut -d: -f1)
gdb-multiarch -q -batch -ex "target remote :$PORT" -ex "break *$bp" \
              -ex continue -ex 'printf "gp=%d\n", $gp'      # 1 = pass, 3 = fail

QEMU is a valid reference for the scalar tests

machine/paging_bare/rv_i/rv64i_5 → QEMU gp=1, simmerv Test Passed. They agree.

It is not, for the rv_v tests

vlmul_m4/rv64v_13 — which simmerv passes — gives gp=3 under QEMU. With the control failing, QEMU's opinion of rv64v_14 is inadmissible, so this issue is still unresolved.

Breaking at test_failed shows QEMU dying early: mcause=1 (instruction access fault) with mepc at the mret that ends the startup sequence, before any test body runs. Matching the suite's own reference ISA from infra/whisper_config_vlen_256.json (rv64imafdcvsuzfh_zba_zbb_zbc_zbs, svadu=false for its page_fault_on_first_access) does not change it.

Correcting my previous comment

I suggested the cause was QEMU's spike machine mapping an HTIF page at tohost & ~0xfff, shadowing .text — the rv_v tests put tohost at 0x80000800, inside .text, while the rv_i tests (which work) put it at 0x70000000. It was a clean theory and it is wrong. Rebuilding rv64v_13/rv64v_14 with tohost moved into a .io_htif section at 0x70000000 (both linker script and source patched; simmerv's verdicts unchanged at pass/fail respectively) leaves QEMU failing with the identical mcause=1 at the identical address.

So the difference between rv_i working and rv_v not is not tohost placement. Still unexplained.

Recommendation

Use spike, as originally intended. The suite documents the exact build:

./configure --enable-tt-stop-if-tohost-nonzero --enable-tt-table-walk-debug \
            --enable-tt-expanded-dram-address-range --enable-dual-endian \
            --with-isa=RV64IMAFDCV_ZBA_ZBB_ZBC_ZBS --with-priv=MSU

Note the spike/ submodule is empty in a --depth 1 clone without --recurse-submodules. Those --enable-tt-* flags are the tell: these tests expect a specifically configured reference, which is consistent with QEMU falling over on the vector ones for reasons that are not simmerv's problem.