fix(truenas): give passthrough drives iothreads and size the VM for ZFS+NFS

[Benehiko]

Problem

The truenas template shipped 4G / 1 vCPU, and every passthrough data drive was serviced on QEMU's main loop. On a storage VM that combination serializes everything: nfsd's thread pool, ZFS checksumming/compression/write aggregation, and vCPU execution all contend for a single core — while disk I/O shares the main loop with the vCPU it is blocking.

This surfaced while investigating a self-hosted CI runner whose boot disk lives on NFS backed by this VM. The guest sat at 80-94% iowait with idle us/sy, and CI jobs crawled.

Measurement

Per-op NFSv4 client stats (/proc/self/mountstats) against a 60T two-drive pool:

op	ops	avg RTT	avg exec
READ	74.3M	38.3ms	63.2ms
WRITE	19.0M	89.1ms	10129.2ms
COMMIT	8.5M	157.3ms	287.0ms

WRITE averaged 89ms of round-trip but 10.1s of total execution — a 113x gap. That gap is client-side queueing, and it is not explained by either end of the wire:

• Not the network — 0.39ms ping RTT, 1 retransmit in 104.9M RPC calls.
• Not the pool — zpool iostat -l showed 1-3ms write latency, FRAG 3%, CAP 20%, both pools ONLINE, no scrub or resilver running.

What is left is the NFS server itself: a single vCPU in front of a pool that retires writes in milliseconds.

Changes

Per-drive iothreads. Each passthrough disk now declares its own iothread object and binds it on the virtio-blk-pci device, moving drive I/O off the main QEMU loop:

-object iothread,id=iothread-disk0
-drive  file=/dev/disk/by-id/ata-...ZXA0S3H6,format=raw,if=none,id=disk0,cache=none,aio=native,discard=unmap
-device virtio-blk-pci,drive=disk0,serial=EXOS22TB-A,iothread=iothread-disk0

IDs derive from the deterministic drive index, so multiple drives cannot collide — a duplicate object id makes QEMU refuse to start outright.

CPUs 1 → 2, exposed as one hyperthreaded core (1 socket, 1 core, 2 threads). Enough to keep nfsd off the critical path without taking a second physical core from the host.

Memory 4G → 6G, sized from measurement rather than convention. On the 4G VM, ARC held 2.1G against a 2.9G cap at a 95% hit rate, with arc_no_grow and memory_throttle_count both zero — ARC was working well and merely short of headroom, not starved. 6G lifts the default cap to ~3G and leaves ~1G free for nfsd.

Passthrough drives also gain discard=unmap, matching the OS disk.

Note the published template docs already specified different defaults than the code shipped — the code had drifted from its own documentation. This brings them back in line.

Expected impact

The vCPU change is what targets the 10.1s writes. Memory is secondary: ARC was already at a 95% hit rate, so more RAM buys little on its own. If 2 vCPUs prove insufficient under load, that is the knob to revisit before RAM.

Testing

go build ./..., make lint (0 issues), and the full go test ./... suite pass.

New tests cover the iothread object/device pairing, declaration ordering (the object must precede the device referencing it), per-drive ID uniqueness across a built machine, that non-passthrough disks are unaffected, and the hyperthreaded -smp topology.

Each assertion was mutation-checked — removing the iothread binding, colliding the IDs, and collapsing threads to 1 all fail the suite.

Generated args verified end-to-end for a two-drive truenas-like config:

-object iothread,id=iothread-disk0
-drive  file=/dev/disk/by-id/ata-ST22000NM000C-3WC103_ZXA0S3H6,...,discard=unmap
-device virtio-blk-pci,drive=disk0,serial=EXOS22TB-A,iothread=iothread-disk0
-object iothread,id=iothread-disk1
-drive  file=/dev/disk/by-id/ata-ST22000NM000C-3WC103_ZXA0WD9J,...,discard=unmap
-device virtio-blk-pci,drive=disk1,serial=EXOS22TB-B,iothread=iothread-disk1

Note for existing VMs

Templates apply at create time, so this does not change an already-created VM. An existing truenas VM needs its persisted config updated and a restart (which takes the NAS down) to pick these up.