Enabling ARC and/or L2ARC causes throughput to drop significantly. #18752

Xaymar · 2026-07-06T07:50:11Z

Xaymar
Jul 6, 2026

Kind of lost on this even after days of searching for solutions. I've got a system with four Samsung 870 QVO 8TBs that when left at default settings underperforms massively:

SEQ_Q4T1
READ:  bw=9.87MiB/s (10.4MB/s), 9.87MiB/s-9.87MiB/s (10.4MB/s-10.4MB/s), io=592MiB (621MB), run=60001-60001msec
WRITE: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=10.0GiB (10.7GB), run=31151-31151msec

RND_Q4T1
READ:  bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=1080MiB (1133MB), run=60001-60001msec
WRITE: bw=47.7MiB/s (50.0MB/s), 47.7MiB/s-47.7MiB/s (50.0MB/s-50.0MB/s), io=2863MiB (3002MB), run=60001-60001msec

The whole thing magically recovers to normal speeds when I set primarycache=none and secondarycache=none on the pool:

SEQ_Q4T1
READ: bw=728MiB/s (764MB/s), 728MiB/s-728MiB/s (764MB/s-764MB/s), io=10.0GiB (10.7GB), run=14058-14058msec
WRITE: bw=431MiB/s (452MB/s), 431MiB/s-431MiB/s (452MB/s-452MB/s), io=10.0GiB (10.7GB), run=23752-23752msec

RND_Q4T1
READ: bw=720MiB/s (755MB/s), 720MiB/s-720MiB/s (755MB/s-755MB/s), io=10.0GiB (10.7GB), run=14223-14223msec
WRITE: bw=26.2MiB/s (27.5MB/s), 26.2MiB/s-26.2MiB/s (27.5MB/s-27.5MB/s), io=1572MiB (1648MB), run=60001-60001msec

What in the world could even be causing this?

At first I thought it was the SATA controller, so I split the whole thing across two SATA controllers, but all that did was slightly increase sequential write speed by about 33%. I was under the impression that ARC (and L2ARC) should improve performance, not horribly ruin it.

This feels like a bug in the software but I have no way to confirm it. I found other users on reddit, proxmox, truenas forum, and various other pages with the exact same issue. Sadly they also ran into the no solution wall that I did.

System Information

lscpu

root@merenka:/home/xaymar# lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             48 bits physical, 48 bits virtual
  Byte Order:                Little Endian
CPU(s):                      32
  On-line CPU(s) list:       0-31
Vendor ID:                   AuthenticAMD
  Model name:                AMD EPYC 7313P 16-Core Processor
    CPU family:              25
    Model:                   1
    Thread(s) per core:      2
    Core(s) per socket:      16
    Socket(s):               1
    Stepping:                1
    Frequency boost:         enabled
    CPU(s) scaling MHz:      35%
    CPU max MHz:             3730.0000
    CPU min MHz:             400.0000
    BogoMIPS:                5999.93
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall
                             nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperf
                             mperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lah
                             f_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext pe
                             rfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall f
                             sgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xge
                             tbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoi
                             nvd amd_ppin brs arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter
                             pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
                             fsrm debug_swap
Virtualization features:
  Virtualization:            AMD-V
Caches (sum of all):
  L1d:                       512 KiB (16 instances)
  L1i:                       512 KiB (16 instances)
  L2:                        8 MiB (16 instances)
  L3:                        128 MiB (4 instances)
NUMA:
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-31
Vulnerabilities:
  Gather data sampling:      Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Mitigation; Safe RET
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BH
                             I Not affected
  Srbds:                     Not affected
  Tsa:                       Vulnerable: Clear CPU buffers attempted, no microcode
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace

lsmem

root@merenka:/home/xaymar# lsmem
RANGE                                  SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x000000407fffffff  258G online       yes 0-128

Memory block size:                  2G
Total online memory:              258G
Total offline memory:               0B

zfs --version

root@merenka:/home/xaymar# zfs --version
zfs-2.3.2-2
zfs-kmod-2.3.2-2

Answered by Xaymar

Jul 13, 2026

For others running into this, it appears that there is a known incompatibility between certain SATA/SAS controllers and Samsung SSDs. Swapping from the built-in controller (seems to be AMD?) to a LSI/Broadcom 9400-16i appears to have completely fixed it. Depending on where you are in the world, they'll either be labelled as Lenovo 430-16i or as HPE

ARC/L2ARC enabled (and configured)

root@merenka:/mnt/main# fio --filename=/mnt/main/test.fio ~/fio-perf.fio
seq_1MQ32T1_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ32T1_MIXED: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=…

View full answer

amotin · 2026-07-06T08:28:32Z

amotin
Jul 6, 2026
Collaborator

I don't believe ARC is ruining anything there. What I guess might happen is some side effects on speculative prefetcher or I/O aggregation, etc. that somehow affect I/O sizes to confuse the drive. The speeds are too low to be reasonable for anything. There must be some disk timeouts, or error or something like that. I'd look towards disk load/latency stats, and if you see there anything insane -- it might be your answer.

2 replies

Xaymar Jul 6, 2026
Author

The disks individually have no issues with performance. Even in a mdadm RAID0,1,5 and 6 the performance is stable as expected with any filesystem - except for ZFS. The speed of the drives can't reasonably be an issue as HDDs are significantly slower than these.

But I'll take a look at iostat. I did notice one of the drives would always get more or less IO utilization during the mdadm tests. No noticable latency spikes or measured timeouts though during the fio test.

Xaymar Jul 6, 2026
Author

I took a few tests but the IO limit doesn't appear to be from the drives themselves. Here's a sequential read with ARC and L2ARC enabled:

ARC/L2ARC enabled and waited 30m for it to be "hot":

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme1n1          0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdb            750.00  12000.00     0.00   0.00    0.22    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.16  14.80
sdc            815.00  13040.00     0.00   0.00    0.22    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.18  18.80
sdd            772.00  12352.00     0.00   0.00    0.22    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.17  16.00
sde              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdf              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdg            764.00  12224.00     0.00   0.00    0.22    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.17  14.80

Horrible.

Immediately after disabling ARC/L2ARC

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00    6.00    720.00    39.00  86.67    1.50   120.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   0.00
nvme1n1          0.00      0.00     0.00   0.00    0.00     0.00    6.00    720.00    39.00  86.67    1.50   120.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   0.00
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdb           1982.00  31712.00     0.00   0.00    0.10    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.19  18.40
sdc           1968.00  31488.00     0.00   0.00    0.10    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.19  18.80
sdd           1952.00  31232.00     0.00   0.00    0.09    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.18  19.20
sde              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdf              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdg           2120.00  33920.00     0.00   0.00    0.09    16.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.19  20.00

Quite bad, so it hasn't recovered from the assault that ARC/L2ARC does yet.

30m after disabling ARC/L2ARC:

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
dm-0             0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1          0.00      0.00     0.00   0.00    0.00     0.00    3.00    200.00    10.00  76.92   13.67    66.67  116.00  10248.00     0.00   0.00    1.50    88.34    0.00    0.00    0.22  20.40
nvme1n1          0.00      0.00     0.00   0.00    0.00     0.00    3.00    200.00    10.00  76.92   14.00    66.67  116.00  10248.00     0.00   0.00    1.49    88.34    0.00    0.00    0.21  18.00
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdb           5736.00 132496.00     0.00   0.00    0.55    23.10    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.18  44.80
sdc           5632.00 131344.00     0.00   0.00    0.53    23.32    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.01  43.20
sdd           5767.00 135888.00     0.00   0.00    0.47    23.56    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.73  38.40
sde              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdf              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdg           5620.00 132112.00     0.00   0.00    0.58    23.51    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.24  46.40

This is the expected read IOPS if the entire drive is considered "full". Oddly enough the drives are 99% empty, so this seems like yet another bug in ZFS - it's not trimming areas that aren't actually used. And we can even measure the recovered performance with fio:

fio ~/fio-perf.fio

seq_1MQ32T1_RW100: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
seq_1MQ32T1_RW50: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
seq_1MQ32T1_RW0: (g=2): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
seq_1MQ4T1_RW100: (g=3): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=4
seq_1MQ4T1_RW50: (g=4): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=4
seq_1MQ4T1_RW0: (g=5): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=4
seq_1MQ1T1_RW100: (g=6): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
seq_1MQ1T1_RW50: (g=7): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
seq_1MQ1T1_RW0: (g=8): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
rnd_4KQ32T1_RW100: (g=9): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
rnd_4KQ32T1_RW50: (g=10): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
rnd_4KQ32T1_RW0: (g=11): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=32
rnd_4KQ4T1_RW100: (g=12): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rnd_4KQ4T1_RW50: (g=13): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rnd_4KQ4T1_RW0: (g=14): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rnd_4KQ1T16_RW100: (g=15): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
rnd_4KQ4T16_RW50: (g=16): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
rnd_4KQ1T16_RW0: (g=17): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
Run status group 0 (all jobs):
   READ: bw=620MiB/s (650MB/s), 620MiB/s-620MiB/s (650MB/s-650MB/s), io=6198MiB (6499MB), run=10001-10001msec

Run status group 1 (all jobs):
   READ: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=2234MiB (2343MB), run=10001-10001msec
  WRITE: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=2355MiB (2469MB), run=10001-10001msec

Run status group 2 (all jobs):
  WRITE: bw=547MiB/s (574MB/s), 547MiB/s-547MiB/s (574MB/s-574MB/s), io=5470MiB (5736MB), run=10001-10001msec

Run status group 3 (all jobs):
   READ: bw=564MiB/s (592MB/s), 564MiB/s-564MiB/s (592MB/s-592MB/s), io=5642MiB (5916MB), run=10001-10001msec

Run status group 4 (all jobs):
   READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=2275MiB (2386MB), run=10003-10003msec
  WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=2254MiB (2363MB), run=10003-10003msec

Run status group 5 (all jobs):
  WRITE: bw=473MiB/s (496MB/s), 473MiB/s-473MiB/s (496MB/s-496MB/s), io=4731MiB (4961MB), run=10001-10001msec

Run status group 6 (all jobs):
   READ: bw=1048MiB/s (1099MB/s), 64.2MiB/s-70.3MiB/s (67.3MB/s-73.7MB/s), io=10.2GiB (11.0GB), run=10001-10010msec

Run status group 7 (all jobs):
   READ: bw=566MiB/s (593MB/s), 31.0MiB/s-41.1MiB/s (32.5MB/s-43.0MB/s), io=5664MiB (5939MB), run=10003-10012msec
  WRITE: bw=555MiB/s (581MB/s), 30.7MiB/s-39.4MiB/s (32.2MB/s-41.3MB/s), io=5552MiB (5822MB), run=10003-10012msec

Run status group 8 (all jobs):
  WRITE: bw=1432MiB/s (1502MB/s), 88.1MiB/s-91.6MiB/s (92.3MB/s-96.0MB/s), io=14.0GiB (15.0GB), run=10001-10005msec

Run status group 9 (all jobs):
   READ: bw=13.0MiB/s (13.6MB/s), 13.0MiB/s-13.0MiB/s (13.6MB/s-13.6MB/s), io=130MiB (136MB), run=10001-10001msec

Run status group 10 (all jobs):
   READ: bw=8867KiB/s (9080kB/s), 8867KiB/s-8867KiB/s (9080kB/s-9080kB/s), io=86.6MiB (90.8MB), run=10001-10001msec
  WRITE: bw=8821KiB/s (9033kB/s), 8821KiB/s-8821KiB/s (9033kB/s-9033kB/s), io=86.2MiB (90.3MB), run=10001-10001msec

Run status group 11 (all jobs):
  WRITE: bw=18.0MiB/s (18.8MB/s), 18.0MiB/s-18.0MiB/s (18.8MB/s-18.8MB/s), io=180MiB (188MB), run=10001-10001msec

Run status group 12 (all jobs):
   READ: bw=16.8MiB/s (17.6MB/s), 16.8MiB/s-16.8MiB/s (17.6MB/s-17.6MB/s), io=168MiB (176MB), run=10001-10001msec

Run status group 13 (all jobs):
   READ: bw=9302KiB/s (9525kB/s), 9302KiB/s-9302KiB/s (9525kB/s-9525kB/s), io=90.8MiB (95.3MB), run=10001-10001msec
  WRITE: bw=9267KiB/s (9489kB/s), 9267KiB/s-9267KiB/s (9489kB/s-9489kB/s), io=90.5MiB (94.9MB), run=10001-10001msec

Run status group 14 (all jobs):
  WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=183MiB (192MB), run=10001-10001msec

Run status group 15 (all jobs):
   READ: bw=158MiB/s (166MB/s), 9.83MiB/s-9.97MiB/s (10.3MB/s-10.5MB/s), io=1584MiB (1661MB), run=10001-10001msec

Run status group 16 (all jobs):
   READ: bw=86.0MiB/s (90.2MB/s), 5459KiB/s-5560KiB/s (5590kB/s-5693kB/s), io=860MiB (902MB), run=10001-10001msec
  WRITE: bw=86.0MiB/s (90.2MB/s), 5453KiB/s-5591KiB/s (5584kB/s-5725kB/s), io=860MiB (902MB), run=10001-10001msec

Run status group 17 (all jobs):
  WRITE: bw=146MiB/s (153MB/s), 9289KiB/s-9381KiB/s (9512kB/s-9606kB/s), io=1461MiB (1532MB), run=10001-10001msec

Edit: More testing later, it seems that having ARC enabled doesn't hurt performance if the read/write is the same as or larger than recordsize, but significantly hurts performance if it is smaller, even by a single byte. Unfortunately considering that the use case is everything from webhosting to network attached storage, there's no way to guarantee any effective recordsize. Keeping it off seems like the best choice so far.

Xaymar · 2026-07-13T14:35:04Z

Xaymar
Jul 13, 2026
Author

For others running into this, it appears that there is a known incompatibility between certain SATA/SAS controllers and Samsung SSDs. Swapping from the built-in controller (seems to be AMD?) to a LSI/Broadcom 9400-16i appears to have completely fixed it. Depending on where you are in the world, they'll either be labelled as Lenovo 430-16i or as HPE

ARC/L2ARC enabled (and configured)

root@merenka:/mnt/main# fio --filename=/mnt/main/test.fio ~/fio-perf.fio
seq_1MQ32T1_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ32T1_MIXED: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ32T1_WRITE: (g=2): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ4T1_READ: (g=3): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T1_MIXED: (g=4): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T1_WRITE: (g=5): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T16_READ: (g=6): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
seq_1MQ4T16_MIXED: (g=7): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
seq_1MQ4T16_WRITE: (g=8): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
rnd_4KQ32T1_READ: (g=9): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ32T1_MIXED: (g=10): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ32T1_WRITE: (g=11): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ4T1_READ: (g=12): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T1_MIXED: (g=13): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T1_WRITE: (g=14): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T16_READ: (g=15): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
rnd_4KQ4T16_MIXED: (g=16): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
rnd_4KQ4T16_WRITE: (g=17): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
fio-3.39
Starting 108 processes
seq_1MQ32T1_READ: Laying out IO file (1 file / 16384MiB)
Jobs: 14 (f=14): [_(92),w(3),_(1),w(1),E(1),w(10)][20.4%][w=149MiB/s][w=38.1k IOPS][eta 07m:45s]
seq_1MQ32T1_READ: (groupid=0, jobs=1): err= 0: pid=3013510: Mon Jul 13 16:27:10 2026
  read: IOPS=2539, BW=2546MiB/s (2669MB/s)(12.5GiB/5025msec)
    slat (nsec): min=720, max=696540, avg=2468.94, stdev=9678.31
    clat (usec): min=41, max=63309, avg=12607.88, stdev=11424.43
     lat (usec): min=178, max=63312, avg=12610.42, stdev=11424.56
    clat percentiles (usec):
     |  1.00th=[  314],  5.00th=[  441], 10.00th=[  490], 20.00th=[  578],
     | 30.00th=[  717], 40.00th=[ 4146], 50.00th=[12125], 60.00th=[19268],
     | 70.00th=[21103], 80.00th=[23200], 90.00th=[26084], 95.00th=[29492],
     | 99.00th=[38536], 99.50th=[42730], 99.90th=[60031], 99.95th=[60556],
     | 99.99th=[61604]
   bw (  MiB/s): min= 1308, max=11486, per=100.00%, avg=2552.63, stdev=3140.29, samples=10
   iops        : min= 1308, max=11486, avg=2552.60, stdev=3140.30, samples=10
  lat (usec)   : 50=0.01%, 100=0.01%, 250=0.31%, 500=10.76%, 750=20.72%
  lat (usec)   : 1000=5.05%
  lat (msec)   : 2=2.04%, 4=1.08%, 10=7.68%, 20=16.36%, 50=35.93%
  lat (msec)   : 100=0.30%
  cpu          : usr=0.26%, sys=1.65%, ctx=9800, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=12761,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ32T1_MIXED: (groupid=1, jobs=1): err= 0: pid=3013805: Mon Jul 13 16:27:10 2026
  read: IOPS=242, BW=243MiB/s (254MB/s)(1233MiB/5083msec)
    slat (nsec): min=810, max=13150, avg=2401.79, stdev=908.55
    clat (usec): min=1274, max=52179, avg=4445.51, stdev=6009.80
     lat (usec): min=1276, max=52183, avg=4447.92, stdev=6009.93
    clat percentiles (usec):
     |  1.00th=[ 1385],  5.00th=[ 1565], 10.00th=[ 1729], 20.00th=[ 2024],
     | 30.00th=[ 2212], 40.00th=[ 2311], 50.00th=[ 2442], 60.00th=[ 2606],
     | 70.00th=[ 3130], 80.00th=[ 4948], 90.00th=[ 8029], 95.00th=[15401],
     | 99.00th=[35390], 99.50th=[41157], 99.90th=[49546], 99.95th=[52167],
     | 99.99th=[52167]
   bw (  KiB/s): min=86188, max=346112, per=100.00%, avg=252381.20, stdev=71174.08, samples=10
   iops        : min=   84, max=  338, avg=246.40, stdev=69.55, samples=10
  write: IOPS=253, BW=259MiB/s (272MB/s)(1318MiB/5083msec); 0 zone resets
    slat (usec): min=7, max=154, avg=25.91, stdev= 8.66
    clat (msec): min=78, max=591, avg=121.11, stdev=70.63
     lat (msec): min=78, max=591, avg=121.13, stdev=70.63
    clat percentiles (msec):
     |  1.00th=[   81],  5.00th=[   85], 10.00th=[   86], 20.00th=[   90],
     | 30.00th=[   92], 40.00th=[   96], 50.00th=[  102], 60.00th=[  109],
     | 70.00th=[  122], 80.00th=[  138], 90.00th=[  153], 95.00th=[  182],
     | 99.00th=[  575], 99.50th=[  584], 99.90th=[  592], 99.95th=[  592],
     | 99.99th=[  592]
   bw (  KiB/s): min=94396, max=362496, per=99.30%, avg=263652.20, stdev=81082.19, samples=10
   iops        : min=   92, max=  354, avg=257.40, stdev=79.21, samples=10
  lat (msec)   : 2=9.21%, 4=28.02%, 10=8.10%, 20=2.02%, 50=1.55%
  lat (msec)   : 100=25.00%, 250=25.75%, 500=0.75%, 750=0.83%
  cpu          : usr=1.06%, sys=0.30%, ctx=6831, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1233,1287,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ32T1_WRITE: (groupid=2, jobs=1): err= 0: pid=3014126: Mon Jul 13 16:27:10 2026
  write: IOPS=453, BW=460MiB/s (482MB/s)(2324MiB/5057msec); 0 zone resets
    slat (usec): min=11, max=3707, avg=32.93, stdev=104.72
    clat (msec): min=56, max=133, avg=70.06, stdev=12.61
     lat (msec): min=56, max=133, avg=70.09, stdev=12.61
    clat percentiles (msec):
     |  1.00th=[   57],  5.00th=[   58], 10.00th=[   59], 20.00th=[   61],
     | 30.00th=[   66], 40.00th=[   68], 50.00th=[   68], 60.00th=[   69],
     | 70.00th=[   71], 80.00th=[   73], 90.00th=[   81], 95.00th=[   89],
     | 99.00th=[  128], 99.50th=[  128], 99.90th=[  134], 99.95th=[  134],
     | 99.99th=[  134]
   bw (  KiB/s): min=387072, max=550912, per=99.78%, avg=469575.60, stdev=53977.61, samples=10
   iops        : min=  378, max=  538, avg=458.40, stdev=52.82, samples=10
  lat (msec)   : 100=96.95%, 250=4.40%
  cpu          : usr=1.56%, sys=0.08%, ctx=2490, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2293,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ4T1_READ: (groupid=3, jobs=1): err= 0: pid=3014374: Mon Jul 13 16:27:10 2026
  read: IOPS=1051, BW=1052MiB/s (1103MB/s)(5263MiB/5004msec)
    slat (nsec): min=740, max=13690, avg=2008.80, stdev=619.31
    clat (usec): min=1277, max=40416, avg=3800.83, stdev=1899.12
     lat (usec): min=1280, max=40417, avg=3802.84, stdev=1899.11
    clat percentiles (usec):
     |  1.00th=[ 1811],  5.00th=[ 2245], 10.00th=[ 2409], 20.00th=[ 2671],
     | 30.00th=[ 2900], 40.00th=[ 3163], 50.00th=[ 3458], 60.00th=[ 3785],
     | 70.00th=[ 4228], 80.00th=[ 4686], 90.00th=[ 5538], 95.00th=[ 6194],
     | 99.00th=[ 7701], 99.50th=[ 8848], 99.90th=[33424], 99.95th=[40109],
     | 99.99th=[40633]
   bw (  MiB/s): min=  842, max= 1462, per=100.00%, avg=1052.27, stdev=209.22, samples=10
   iops        : min=  842, max= 1462, avg=1052.10, stdev=209.11, samples=10
  lat (msec)   : 2=2.07%, 4=63.33%, 10=34.28%, 20=0.15%, 50=0.23%
  cpu          : usr=0.08%, sys=0.70%, ctx=5224, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5260,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T1_MIXED: (groupid=4, jobs=1): err= 0: pid=3014487: Mon Jul 13 16:27:10 2026
  read: IOPS=225, BW=226MiB/s (237MB/s)(1143MiB/5066msec)
    slat (nsec): min=870, max=843130, avg=6278.10, stdev=44775.56
    clat (usec): min=1142, max=75774, avg=4874.48, stdev=8126.59
     lat (usec): min=1332, max=75777, avg=4880.76, stdev=8129.08
    clat percentiles (usec):
     |  1.00th=[ 1401],  5.00th=[ 1500], 10.00th=[ 1565], 20.00th=[ 1713],
     | 30.00th=[ 1844], 40.00th=[ 1991], 50.00th=[ 2147], 60.00th=[ 2376],
     | 70.00th=[ 2868], 80.00th=[ 4752], 90.00th=[ 8979], 95.00th=[20055],
     | 99.00th=[42206], 99.50th=[48497], 99.90th=[70779], 99.95th=[76022],
     | 99.99th=[76022]
   bw (  KiB/s): min=149504, max=328336, per=100.00%, avg=233269.00, stdev=64729.33, samples=10
   iops        : min=  146, max=  320, avg=227.60, stdev=63.10, samples=10
  write: IOPS=227, BW=228MiB/s (239MB/s)(1153MiB/5066msec); 0 zone resets
    slat (usec): min=12, max=191, avg=28.44, stdev= 8.14
    clat (usec): min=2543, max=56470, avg=12722.53, stdev=8040.64
     lat (usec): min=2583, max=56488, avg=12750.99, stdev=8041.23
    clat percentiles (usec):
     |  1.00th=[ 5014],  5.00th=[ 6325], 10.00th=[ 7439], 20.00th=[ 8029],
     | 30.00th=[ 8979], 40.00th=[ 9765], 50.00th=[10159], 60.00th=[10814],
     | 70.00th=[11863], 80.00th=[14091], 90.00th=[22938], 95.00th=[33162],
     | 99.00th=[45351], 99.50th=[47449], 99.90th=[53740], 99.95th=[56361],
     | 99.99th=[56361]
   bw (  KiB/s): min=151552, max=316024, per=100.00%, avg=236105.30, stdev=58586.35, samples=10
   iops        : min=  148, max=  308, avg=230.40, stdev=57.10, samples=10
  lat (msec)   : 2=20.28%, 4=18.62%, 10=29.52%, 20=23.16%, 50=8.16%
  lat (msec)   : 100=0.39%
  cpu          : usr=0.71%, sys=0.43%, ctx=4408, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1143,1150,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T1_WRITE: (groupid=5, jobs=1): err= 0: pid=3014776: Mon Jul 13 16:27:10 2026
  write: IOPS=443, BW=444MiB/s (466MB/s)(2225MiB/5009msec); 0 zone resets
    slat (usec): min=7, max=155, avg=32.81, stdev= 9.76
    clat (usec): min=6010, max=63786, avg=8972.80, stdev=4464.32
     lat (usec): min=6036, max=63816, avg=9005.62, stdev=4464.60
    clat percentiles (usec):
     |  1.00th=[ 6063],  5.00th=[ 6063], 10.00th=[ 6128], 20.00th=[ 7177],
     | 30.00th=[ 8225], 40.00th=[ 8356], 50.00th=[ 8455], 60.00th=[ 8979],
     | 70.00th=[ 9110], 80.00th=[ 9110], 90.00th=[ 9634], 95.00th=[11731],
     | 99.00th=[25035], 99.50th=[46924], 99.90th=[61080], 99.95th=[63177],
     | 99.99th=[63701]
   bw (  KiB/s): min=393216, max=530432, per=100.00%, avg=454840.60, stdev=37601.02, samples=10
   iops        : min=  384, max=  518, avg=444.00, stdev=36.68, samples=10
  lat (msec)   : 10=90.55%, 20=8.01%, 50=1.13%, 100=0.45%
  cpu          : usr=1.44%, sys=0.30%, ctx=2225, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2222,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_READ: (groupid=6, jobs=16): err= 0: pid=3015028: Mon Jul 13 16:27:10 2026
  read: IOPS=1225, BW=1235MiB/s (1295MB/s)(6287MiB/5090msec)
    slat (nsec): min=1020, max=886561, avg=5545.58, stdev=33755.94
    clat (msec): min=2, max=161, avg=51.74, stdev=30.27
     lat (msec): min=2, max=161, avg=51.74, stdev=30.27
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    4], 10.00th=[    6], 20.00th=[   10],
     | 30.00th=[   42], 40.00th=[   51], 50.00th=[   61], 60.00th=[   67],
     | 70.00th=[   72], 80.00th=[   77], 90.00th=[   82], 95.00th=[   95],
     | 99.00th=[  126], 99.50th=[  132], 99.90th=[  153], 99.95th=[  155],
     | 99.99th=[  163]
   bw (  MiB/s): min= 1076, max= 1310, per=100.00%, avg=1242.13, stdev= 4.05, samples=160
   iops        : min= 1074, max= 1310, avg=1241.20, stdev= 4.08, samples=160
  lat (msec)   : 4=7.29%, 10=13.40%, 20=3.67%, 50=14.65%, 100=59.95%
  lat (msec)   : 250=1.81%
  cpu          : usr=0.04%, sys=0.05%, ctx=6342, majf=0, minf=597
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6239,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_MIXED: (groupid=7, jobs=16): err= 0: pid=3015363: Mon Jul 13 16:27:10 2026
  read: IOPS=323, BW=324MiB/s (340MB/s)(1652MiB/5101msec)
    slat (nsec): min=1040, max=20840, avg=3694.48, stdev=1438.44
    clat (msec): min=2, max=416, avg=57.82, stdev=35.90
     lat (msec): min=2, max=416, avg=57.83, stdev=35.90
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   10], 10.00th=[   17], 20.00th=[   26],
     | 30.00th=[   36], 40.00th=[   45], 50.00th=[   53], 60.00th=[   64],
     | 70.00th=[   75], 80.00th=[   88], 90.00th=[  101], 95.00th=[  116],
     | 99.00th=[  157], 99.50th=[  171], 99.90th=[  334], 99.95th=[  418],
     | 99.99th=[  418]
   bw (  KiB/s): min=148284, max=581866, per=99.32%, avg=329362.50, stdev=8634.09, samples=160
   iops        : min=  144, max=  564, avg=320.90, stdev= 8.39, samples=160
  write: IOPS=308, BW=317MiB/s (333MB/s)(1619MiB/5101msec); 0 zone resets
    slat (usec): min=13, max=154, avg=28.24, stdev= 8.45
    clat (msec): min=2, max=506, avg=143.58, stdev=76.88
     lat (msec): min=2, max=506, avg=143.61, stdev=76.88
    clat percentiles (msec):
     |  1.00th=[   16],  5.00th=[   43], 10.00th=[   59], 20.00th=[   82],
     | 30.00th=[  100], 40.00th=[  114], 50.00th=[  131], 60.00th=[  148],
     | 70.00th=[  169], 80.00th=[  197], 90.00th=[  247], 95.00th=[  292],
     | 99.00th=[  393], 99.50th=[  422], 99.90th=[  460], 99.95th=[  506],
     | 99.99th=[  506]
   bw (  KiB/s): min=213096, max=459275, per=99.45%, avg=323226.50, stdev=4680.78, samples=160
   iops        : min=  208, max=  447, avg=314.80, stdev= 4.52, samples=160
  lat (msec)   : 4=0.59%, 10=2.36%, 20=4.62%, 50=20.26%, 100=33.73%
  lat (msec)   : 250=35.18%, 500=4.72%, 750=0.03%
  cpu          : usr=0.02%, sys=0.09%, ctx=6022, majf=0, minf=597
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1648,1575,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_WRITE: (groupid=8, jobs=16): err= 0: pid=3015670: Mon Jul 13 16:27:10 2026
  write: IOPS=583, BW=593MiB/s (621MB/s)(3026MiB/5106msec); 0 zone resets
    slat (usec): min=14, max=2019, avg=31.81, stdev=37.27
    clat (msec): min=18, max=213, avg=108.03, stdev=23.48
     lat (msec): min=18, max=213, avg=108.06, stdev=23.48
    clat percentiles (msec):
     |  1.00th=[   52],  5.00th=[   70], 10.00th=[   81], 20.00th=[   94],
     | 30.00th=[  101], 40.00th=[  104], 50.00th=[  107], 60.00th=[  110],
     | 70.00th=[  115], 80.00th=[  124], 90.00th=[  136], 95.00th=[  146],
     | 99.00th=[  190], 99.50th=[  194], 99.90th=[  199], 99.95th=[  205],
     | 99.99th=[  213]
   bw (  KiB/s): min=529135, max=714752, per=100.00%, avg=607578.00, stdev=3711.29, samples=160
   iops        : min=  515, max=  698, avg=592.30, stdev= 3.67, samples=160
  lat (msec)   : 20=0.03%, 50=0.74%, 100=30.12%, 250=70.72%
  cpu          : usr=0.13%, sys=0.02%, ctx=3050, majf=0, minf=598
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2978,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ32T1_READ: (groupid=9, jobs=1): err= 0: pid=3016004: Mon Jul 13 16:27:10 2026
  read: IOPS=65.7k, BW=257MiB/s (269MB/s)(1284MiB/5001msec)
    slat (nsec): min=720, max=326630, avg=1985.93, stdev=1322.23
    clat (usec): min=3, max=37394, avg=484.29, stdev=613.91
     lat (usec): min=7, max=37396, avg=486.27, stdev=613.92
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   17], 20.00th=[   22],
     | 30.00th=[  269], 40.00th=[  318], 50.00th=[  383], 60.00th=[  478],
     | 70.00th=[  611], 80.00th=[  799], 90.00th=[ 1090], 95.00th=[ 1319],
     | 99.00th=[ 1762], 99.50th=[ 1958], 99.90th=[ 3490], 99.95th=[ 8717],
     | 99.99th=[24249]
   bw (  KiB/s): min=218512, max=272376, per=100.00%, avg=263088.90, stdev=16476.06, samples=10
   iops        : min=54628, max=68094, avg=65772.10, stdev=4118.97, samples=10
  lat (usec)   : 4=0.01%, 10=0.43%, 20=16.73%, 50=9.90%, 100=0.41%
  lat (usec)   : 250=0.98%, 500=33.39%, 750=15.97%, 1000=9.77%
  lat (msec)   : 2=12.00%, 4=0.33%, 10=0.05%, 20=0.03%, 50=0.02%
  cpu          : usr=6.80%, sys=28.16%, ctx=212148, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=328747,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ32T1_MIXED: (groupid=10, jobs=1): err= 0: pid=3016283: Mon Jul 13 16:27:10 2026
  read: IOPS=11.5k, BW=45.1MiB/s (47.3MB/s)(226MiB/5003msec)
    slat (nsec): min=740, max=112361, avg=2194.63, stdev=2042.92
    clat (nsec): min=610, max=25540k, avg=315795.16, stdev=1039094.28
     lat (usec): min=8, max=25542, avg=317.99, stdev=1039.12
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   18], 20.00th=[   23],
     | 30.00th=[  237], 40.00th=[  265], 50.00th=[  273], 60.00th=[  281],
     | 70.00th=[  293], 80.00th=[  310], 90.00th=[  379], 95.00th=[  445],
     | 99.00th=[ 1762], 99.50th=[ 7832], 99.90th=[16909], 99.95th=[20055],
     | 99.99th=[24511]
   bw (  KiB/s): min= 2284, max=64720, per=100.00%, avg=46180.40, stdev=22797.85, samples=10
   iops        : min=  571, max=16180, avg=11545.00, stdev=5699.42, samples=10
  write: IOPS=11.5k, BW=45.1MiB/s (47.3MB/s)(226MiB/5003msec); 0 zone resets
    slat (nsec): min=760, max=110670, avg=2351.97, stdev=1337.41
    clat (usec): min=80, max=113220, avg=2451.52, stdev=5442.45
     lat (usec): min=82, max=113222, avg=2453.87, stdev=5442.50
    clat percentiles (usec):
     |  1.00th=[   676],  5.00th=[   963], 10.00th=[  1139], 20.00th=[  1369],
     | 30.00th=[  1532], 40.00th=[  1680], 50.00th=[  1827], 60.00th=[  1975],
     | 70.00th=[  2147], 80.00th=[  2376], 90.00th=[  2737], 95.00th=[  3130],
     | 99.00th=[ 23725], 99.50th=[ 47449], 99.90th=[ 82314], 99.95th=[ 91751],
     | 99.99th=[102237]
   bw (  KiB/s): min= 2236, max=63816, per=100.00%, avg=46159.10, stdev=22729.02, samples=10
   iops        : min=  559, max=15954, avg=11539.70, stdev=5682.21, samples=10
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.07%, 20=7.71%, 50=5.81%
  lat (usec)   : 100=0.27%, 250=3.48%, 500=31.17%, 750=1.75%, 1000=2.23%
  lat (msec)   : 2=27.63%, 4=18.00%, 10=1.04%, 20=0.29%, 50=0.32%
  lat (msec)   : 100=0.23%, 250=0.01%
  cpu          : usr=3.80%, sys=11.68%, ctx=101756, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=57728,57706,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ32T1_WRITE: (groupid=11, jobs=1): err= 0: pid=3016579: Mon Jul 13 16:27:10 2026
  write: IOPS=10.1k, BW=39.4MiB/s (41.3MB/s)(197MiB/5002msec); 0 zone resets
    slat (nsec): min=760, max=42730, avg=1517.03, stdev=509.81
    clat (usec): min=413, max=118953, avg=3194.18, stdev=8455.50
     lat (usec): min=415, max=118955, avg=3195.69, stdev=8455.60
    clat percentiles (usec):
     |  1.00th=[   660],  5.00th=[   906], 10.00th=[   979], 20.00th=[  1205],
     | 30.00th=[  1418], 40.00th=[  1516], 50.00th=[  1680], 60.00th=[  1811],
     | 70.00th=[  1975], 80.00th=[  2180], 90.00th=[  2474], 95.00th=[  2999],
     | 99.00th=[ 52167], 99.50th=[ 63177], 99.90th=[ 78119], 99.95th=[ 90702],
     | 99.99th=[116917]
   bw (  KiB/s): min= 2888, max=77224, per=100.00%, avg=40332.80, stdev=33986.59, samples=10
   iops        : min=  722, max=19306, avg=10083.10, stdev=8496.65, samples=10
  lat (usec)   : 500=0.65%, 750=2.51%, 1000=7.56%
  lat (msec)   : 2=61.68%, 4=23.53%, 10=0.44%, 20=0.33%, 50=2.30%
  lat (msec)   : 100=1.04%, 250=0.04%
  cpu          : usr=1.16%, sys=4.14%, ctx=50287, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,50406,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ4T1_READ: (groupid=12, jobs=1): err= 0: pid=3016832: Mon Jul 13 16:27:10 2026
  read: IOPS=16.9k, BW=66.0MiB/s (69.2MB/s)(330MiB/5001msec)
    slat (nsec): min=740, max=108720, avg=2011.37, stdev=944.56
    clat (usec): min=4, max=66077, avg=234.23, stdev=560.31
     lat (usec): min=8, max=66079, avg=236.24, stdev=560.33
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   17], 20.00th=[   21],
     | 30.00th=[  235], 40.00th=[  255], 50.00th=[  269], 60.00th=[  273],
     | 70.00th=[  281], 80.00th=[  289], 90.00th=[  334], 95.00th=[  396],
     | 99.00th=[  562], 99.50th=[  627], 99.90th=[ 7177], 99.95th=[12125],
     | 99.99th=[22152]
   bw (  KiB/s): min=34917, max=74896, per=100.00%, avg=67591.40, stdev=12726.59, samples=10
   iops        : min= 8729, max=18724, avg=16897.80, stdev=3181.70, samples=10
  lat (usec)   : 10=0.13%, 20=19.42%, 50=6.76%, 100=0.33%, 250=11.22%
  lat (usec)   : 500=60.53%, 750=1.39%, 1000=0.05%
  lat (msec)   : 2=0.01%, 4=0.02%, 10=0.07%, 20=0.06%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=2.04%, sys=9.22%, ctx=80633, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=84466,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T1_MIXED: (groupid=13, jobs=1): err= 0: pid=3017086: Mon Jul 13 16:27:10 2026
  read: IOPS=9134, BW=35.7MiB/s (37.4MB/s)(178MiB/5001msec)
    slat (nsec): min=740, max=325200, avg=2118.20, stdev=2048.33
    clat (nsec): min=620, max=26075k, avg=269261.04, stdev=766905.01
     lat (usec): min=9, max=26077, avg=271.38, stdev=766.94
    clat percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   17], 20.00th=[   22],
     | 30.00th=[  233], 40.00th=[  258], 50.00th=[  269], 60.00th=[  273],
     | 70.00th=[  281], 80.00th=[  293], 90.00th=[  322], 95.00th=[  388],
     | 99.00th=[  594], 99.50th=[ 4490], 99.90th=[12780], 99.95th=[16909],
     | 99.99th=[21365]
   bw (  KiB/s): min= 1656, max=48208, per=100.00%, avg=36561.70, stdev=14219.65, samples=10
   iops        : min=  414, max=12052, avg=9140.40, stdev=3554.90, samples=10
  write: IOPS=9052, BW=35.4MiB/s (37.1MB/s)(177MiB/5001msec); 0 zone resets
    slat (nsec): min=760, max=1639.2k, avg=2318.18, stdev=7756.64
    clat (nsec): min=870, max=33100k, avg=164336.45, stdev=619749.69
     lat (usec): min=13, max=33101, avg=166.65, stdev=619.82
    clat percentiles (usec):
     |  1.00th=[   19],  5.00th=[   23], 10.00th=[   26], 20.00th=[   30],
     | 30.00th=[   34], 40.00th=[   39], 50.00th=[   46], 60.00th=[   63],
     | 70.00th=[  169], 80.00th=[  281], 90.00th=[  326], 95.00th=[  424],
     | 99.00th=[  668], 99.50th=[ 2999], 99.90th=[ 9110], 99.95th=[12911],
     | 99.99th=[22414]
   bw (  KiB/s): min= 1912, max=46600, per=100.00%, avg=36238.20, stdev=13730.18, samples=10
   iops        : min=  478, max=11650, avg=9059.40, stdev=3432.46, samples=10
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.04%, 20=9.58%, 50=30.47%
  lat (usec)   : 100=6.65%, 250=9.02%, 500=41.69%, 750=1.70%, 1000=0.12%
  lat (msec)   : 2=0.08%, 4=0.19%, 10=0.32%, 20=0.11%, 50=0.02%
  cpu          : usr=2.54%, sys=10.16%, ctx=83872, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=45680,45272,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T1_WRITE: (groupid=14, jobs=1): err= 0: pid=3017322: Mon Jul 13 16:27:10 2026
  write: IOPS=12.1k, BW=47.1MiB/s (49.4MB/s)(235MiB/5001msec); 0 zone resets
    slat (nsec): min=830, max=107700, avg=1834.94, stdev=910.28
    clat (usec): min=25, max=49278, avg=329.88, stdev=1509.16
     lat (usec): min=28, max=49279, avg=331.72, stdev=1509.17
    clat percentiles (usec):
     |  1.00th=[   45],  5.00th=[   50], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   62], 40.00th=[   67], 50.00th=[   73], 60.00th=[  281],
     | 70.00th=[  306], 80.00th=[  318], 90.00th=[  359], 95.00th=[  562],
     | 99.00th=[ 5080], 99.50th=[11207], 99.90th=[22414], 99.95th=[26346],
     | 99.99th=[36439]
   bw (  KiB/s): min= 3158, max=85104, per=100.00%, avg=48223.70, stdev=35539.08, samples=10
   iops        : min=  789, max=21276, avg=12055.80, stdev=8884.95, samples=10
  lat (usec)   : 50=4.87%, 100=51.50%, 250=0.56%, 500=33.69%, 750=7.25%
  lat (usec)   : 1000=0.74%
  lat (msec)   : 2=0.15%, 4=0.16%, 10=0.54%, 20=0.41%, 50=0.15%
  cpu          : usr=1.58%, sys=5.90%, ctx=59866, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,60275,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_READ: (groupid=15, jobs=16): err= 0: pid=3017437: Mon Jul 13 16:27:10 2026
  read: IOPS=63.6k, BW=249MiB/s (261MB/s)(1243MiB/5003msec)
    slat (nsec): min=730, max=231401, avg=2157.16, stdev=1180.02
    clat (nsec): min=220, max=48339k, avg=1002745.09, stdev=1804415.84
     lat (usec): min=9, max=48341, avg=1004.90, stdev=1804.48
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   16], 10.00th=[   18], 20.00th=[   21],
     | 30.00th=[  221], 40.00th=[  326], 50.00th=[  441], 60.00th=[  660],
     | 70.00th=[ 1074], 80.00th=[ 1745], 90.00th=[ 2737], 95.00th=[ 3294],
     | 99.00th=[ 4555], 99.50th=[11469], 99.90th=[25035], 99.95th=[28181],
     | 99.99th=[36963]
   bw (  KiB/s): min=36296, max=299384, per=100.00%, avg=254675.40, stdev=4806.91, samples=160
   iops        : min= 9074, max=74845, avg=63667.70, stdev=1201.70, samples=160
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.03%, 20=18.00%, 50=11.67%
  lat (usec)   : 100=0.13%, 250=0.68%, 500=23.08%, 750=9.15%, 1000=5.75%
  lat (msec)   : 2=14.36%, 4=15.46%, 10=1.15%, 20=0.34%, 50=0.22%
  cpu          : usr=0.94%, sys=2.26%, ctx=315270, majf=0, minf=598
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=318266,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_MIXED: (groupid=16, jobs=16): err= 0: pid=3017763: Mon Jul 13 16:27:10 2026
  read: IOPS=19.3k, BW=75.5MiB/s (79.2MB/s)(379MiB/5016msec)
    slat (nsec): min=750, max=109370, avg=2253.07, stdev=1253.38
    clat (nsec): min=1250, max=59914k, avg=2310442.96, stdev=5261430.66
     lat (usec): min=7, max=59916, avg=2312.70, stdev=5261.45
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   17], 10.00th=[   19], 20.00th=[   22],
     | 30.00th=[   29], 40.00th=[  318], 50.00th=[  469], 60.00th=[  775],
     | 70.00th=[ 1303], 80.00th=[ 2073], 90.00th=[ 6456], 95.00th=[13960],
     | 99.00th=[26608], 99.50th=[30802], 99.90th=[40109], 99.95th=[43254],
     | 99.99th=[57934]
   bw (  KiB/s): min=20255, max=230334, per=100.00%, avg=77558.80, stdev=4665.39, samples=160
   iops        : min= 5060, max=57579, avg=19388.60, stdev=1166.32, samples=160
  write: IOPS=19.2k, BW=75.1MiB/s (78.7MB/s)(377MiB/5016msec); 0 zone resets
    slat (nsec): min=770, max=109771, avg=2428.47, stdev=1196.20
    clat (nsec): min=180, max=82924k, avg=1002481.10, stdev=3693452.88
     lat (usec): min=14, max=82926, avg=1004.91, stdev=3693.49
    clat percentiles (usec):
     |  1.00th=[   21],  5.00th=[   24], 10.00th=[   27], 20.00th=[   31],
     | 30.00th=[   34], 40.00th=[   38], 50.00th=[   42], 60.00th=[   50],
     | 70.00th=[   79], 80.00th=[  553], 90.00th=[ 1745], 95.00th=[ 4293],
     | 99.00th=[21103], 99.50th=[26346], 99.90th=[37487], 99.95th=[42730],
     | 99.99th=[56361]
   bw (  KiB/s): min=19445, max=228146, per=100.00%, avg=77123.70, stdev=4643.35, samples=160
   iops        : min= 4859, max=57031, avg=19279.80, stdev=1160.79, samples=160
  lat (nsec)   : 250=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.02%, 20=8.52%, 50=38.15%
  lat (usec)   : 100=5.73%, 250=1.25%, 500=11.60%, 750=5.57%, 1000=3.93%
  lat (msec)   : 2=10.45%, 4=6.09%, 10=3.45%, 20=3.38%, 50=1.85%
  lat (msec)   : 100=0.02%
  cpu          : usr=0.47%, sys=1.51%, ctx=188825, majf=0, minf=584
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=96925,96385,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_WRITE: (groupid=17, jobs=16): err= 0: pid=3018071: Mon Jul 13 16:27:10 2026
  write: IOPS=30.7k, BW=120MiB/s (126MB/s)(603MiB/5017msec); 0 zone resets
    slat (nsec): min=730, max=1403.7k, avg=2838.77, stdev=8231.76
    clat (usec): min=5, max=53395, avg=2075.69, stdev=5001.81
     lat (usec): min=17, max=53397, avg=2078.53, stdev=5001.76
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   57], 10.00th=[   65], 20.00th=[   76],
     | 30.00th=[   86], 40.00th=[   98], 50.00th=[  113], 60.00th=[  143],
     | 70.00th=[  388], 80.00th=[  734], 90.00th=[ 8586], 95.00th=[13698],
     | 99.00th=[23462], 99.50th=[27132], 99.90th=[35390], 99.95th=[40633],
     | 99.99th=[49021]
   bw (  KiB/s): min=53640, max=232792, per=98.26%, avg=120846.38, stdev=4210.50, samples=157
   iops        : min=13410, max=58198, avg=30211.14, stdev=1052.63, samples=157
  lat (usec)   : 10=0.13%, 20=0.03%, 50=2.24%, 100=39.66%, 250=22.37%
  lat (usec)   : 500=11.20%, 750=4.54%, 1000=1.49%
  lat (msec)   : 2=1.26%, 4=1.64%, 10=6.99%, 20=6.77%, 50=1.71%
  lat (msec)   : 100=0.01%
  cpu          : usr=0.36%, sys=1.10%, ctx=147522, majf=0, minf=582
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,154213,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=2546MiB/s (2669MB/s), 2546MiB/s-2546MiB/s (2669MB/s-2669MB/s), io=12.5GiB (13.4GB), run=5025-5025msec

Run status group 1 (all jobs):
   READ: bw=243MiB/s (254MB/s), 243MiB/s-243MiB/s (254MB/s-254MB/s), io=1233MiB (1293MB), run=5083-5083msec
  WRITE: bw=259MiB/s (272MB/s), 259MiB/s-259MiB/s (272MB/s-272MB/s), io=1318MiB (1382MB), run=5083-5083msec

Run status group 2 (all jobs):
  WRITE: bw=460MiB/s (482MB/s), 460MiB/s-460MiB/s (482MB/s-482MB/s), io=2324MiB (2437MB), run=5057-5057msec

Run status group 3 (all jobs):
   READ: bw=1052MiB/s (1103MB/s), 1052MiB/s-1052MiB/s (1103MB/s-1103MB/s), io=5263MiB (5519MB), run=5004-5004msec

Run status group 4 (all jobs):
   READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=1143MiB (1199MB), run=5066-5066msec
  WRITE: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=1153MiB (1209MB), run=5066-5066msec

Run status group 5 (all jobs):
  WRITE: bw=444MiB/s (466MB/s), 444MiB/s-444MiB/s (466MB/s-466MB/s), io=2225MiB (2333MB), run=5009-5009msec

Run status group 6 (all jobs):
   READ: bw=1235MiB/s (1295MB/s), 1235MiB/s-1235MiB/s (1295MB/s-1295MB/s), io=6287MiB (6592MB), run=5090-5090msec

Run status group 7 (all jobs):
   READ: bw=324MiB/s (340MB/s), 324MiB/s-324MiB/s (340MB/s-340MB/s), io=1652MiB (1732MB), run=5101-5101msec
  WRITE: bw=317MiB/s (333MB/s), 317MiB/s-317MiB/s (333MB/s-333MB/s), io=1619MiB (1698MB), run=5101-5101msec

Run status group 8 (all jobs):
  WRITE: bw=593MiB/s (621MB/s), 593MiB/s-593MiB/s (621MB/s-621MB/s), io=3026MiB (3173MB), run=5106-5106msec

Run status group 9 (all jobs):
   READ: bw=257MiB/s (269MB/s), 257MiB/s-257MiB/s (269MB/s-269MB/s), io=1284MiB (1347MB), run=5001-5001msec

Run status group 10 (all jobs):
   READ: bw=45.1MiB/s (47.3MB/s), 45.1MiB/s-45.1MiB/s (47.3MB/s-47.3MB/s), io=226MiB (236MB), run=5003-5003msec
  WRITE: bw=45.1MiB/s (47.3MB/s), 45.1MiB/s-45.1MiB/s (47.3MB/s-47.3MB/s), io=226MiB (236MB), run=5003-5003msec

Run status group 11 (all jobs):
  WRITE: bw=39.4MiB/s (41.3MB/s), 39.4MiB/s-39.4MiB/s (41.3MB/s-41.3MB/s), io=197MiB (207MB), run=5002-5002msec

Run status group 12 (all jobs):
   READ: bw=66.0MiB/s (69.2MB/s), 66.0MiB/s-66.0MiB/s (69.2MB/s-69.2MB/s), io=330MiB (346MB), run=5001-5001msec

Run status group 13 (all jobs):
   READ: bw=35.7MiB/s (37.4MB/s), 35.7MiB/s-35.7MiB/s (37.4MB/s-37.4MB/s), io=178MiB (187MB), run=5001-5001msec
  WRITE: bw=35.4MiB/s (37.1MB/s), 35.4MiB/s-35.4MiB/s (37.1MB/s-37.1MB/s), io=177MiB (185MB), run=5001-5001msec

Run status group 14 (all jobs):
  WRITE: bw=47.1MiB/s (49.4MB/s), 47.1MiB/s-47.1MiB/s (49.4MB/s-49.4MB/s), io=235MiB (247MB), run=5001-5001msec

Run status group 15 (all jobs):
   READ: bw=249MiB/s (261MB/s), 249MiB/s-249MiB/s (261MB/s-261MB/s), io=1243MiB (1304MB), run=5003-5003msec

Run status group 16 (all jobs):
   READ: bw=75.5MiB/s (79.2MB/s), 75.5MiB/s-75.5MiB/s (79.2MB/s-79.2MB/s), io=379MiB (397MB), run=5016-5016msec
  WRITE: bw=75.1MiB/s (78.7MB/s), 75.1MiB/s-75.1MiB/s (78.7MB/s-78.7MB/s), io=377MiB (395MB), run=5016-5016msec

Run status group 17 (all jobs):
  WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=603MiB (632MB), run=5017-5017msec

ARC/L2ARC disabled (and configured to be as small as possible just in case)

root@merenka:/mnt/main# fio --filename=/mnt/main/test.fio ~/fio-perf.fio
seq_1MQ32T1_READ: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ32T1_MIXED: (g=1): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ32T1_WRITE: (g=2): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=32
seq_1MQ4T1_READ: (g=3): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T1_MIXED: (g=4): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T1_WRITE: (g=5): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
seq_1MQ4T16_READ: (g=6): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
seq_1MQ4T16_MIXED: (g=7): rw=rw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
seq_1MQ4T16_WRITE: (g=8): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=io_uring, iodepth=4
...
rnd_4KQ32T1_READ: (g=9): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ32T1_MIXED: (g=10): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ32T1_WRITE: (g=11): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=32
rnd_4KQ4T1_READ: (g=12): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T1_MIXED: (g=13): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T1_WRITE: (g=14): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
rnd_4KQ4T16_READ: (g=15): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
rnd_4KQ4T16_MIXED: (g=16): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
rnd_4KQ4T16_WRITE: (g=17): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
...
fio-3.39
Starting 108 processes
Jobs: 15 (f=15): [_(92),w(5),E(1),w(10)][20.6%][w=81.5MiB/s][w=20.9k IOPS][eta 07m:40s]
seq_1MQ32T1_READ: (groupid=0, jobs=1): err= 0: pid=3020107: Mon Jul 13 16:29:49 2026
  read: IOPS=1533, BW=1540MiB/s (1615MB/s)(7736MiB/5023msec)
    slat (nsec): min=720, max=15190, avg=2224.25, stdev=834.88
    clat (usec): min=1372, max=64938, avg=20809.60, stdev=5052.36
     lat (usec): min=1374, max=64940, avg=20811.82, stdev=5052.34
    clat percentiles (usec):
     |  1.00th=[ 8029],  5.00th=[15795], 10.00th=[17171], 20.00th=[18482],
     | 30.00th=[19268], 40.00th=[19792], 50.00th=[20579], 60.00th=[21103],
     | 70.00th=[21627], 80.00th=[22414], 90.00th=[23987], 95.00th=[26870],
     | 99.00th=[39584], 99.50th=[57410], 99.90th=[63701], 99.95th=[64226],
     | 99.99th=[64750]
   bw (  MiB/s): min= 1392, max= 1614, per=100.00%, avg=1541.43, stdev=70.36, samples=10
   iops        : min= 1392, max= 1614, avg=1541.40, stdev=70.35, samples=10
  lat (msec)   : 2=0.03%, 4=0.17%, 10=1.35%, 20=40.73%, 50=57.43%
  lat (msec)   : 100=0.70%
  cpu          : usr=0.16%, sys=1.08%, ctx=7215, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=7705,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ32T1_MIXED: (groupid=1, jobs=1): err= 0: pid=3020258: Mon Jul 13 16:29:49 2026
  read: IOPS=247, BW=248MiB/s (260MB/s)(1258MiB/5080msec)
    slat (nsec): min=870, max=349920, avg=3749.01, stdev=16803.13
    clat (usec): min=1424, max=83780, avg=4243.79, stdev=5439.34
     lat (usec): min=1453, max=83783, avg=4247.54, stdev=5439.51
    clat percentiles (usec):
     |  1.00th=[ 1500],  5.00th=[ 1598], 10.00th=[ 1663], 20.00th=[ 1811],
     | 30.00th=[ 1909], 40.00th=[ 2024], 50.00th=[ 2180], 60.00th=[ 2442],
     | 70.00th=[ 3294], 80.00th=[ 5473], 90.00th=[ 8455], 95.00th=[15139],
     | 99.00th=[26608], 99.50th=[30278], 99.90th=[46400], 99.95th=[83362],
     | 99.99th=[83362]
   bw (  KiB/s): min=147751, max=333824, per=100.00%, avg=257311.60, stdev=55482.45, samples=10
   iops        : min=  144, max=  326, avg=251.20, stdev=54.24, samples=10
  write: IOPS=258, BW=264MiB/s (277MB/s)(1341MiB/5080msec); 0 zone resets
    slat (usec): min=11, max=810, avg=29.72, stdev=23.58
    clat (msec): min=82, max=288, avg=119.67, stdev=35.20
     lat (msec): min=82, max=288, avg=119.70, stdev=35.20
    clat percentiles (msec):
     |  1.00th=[   86],  5.00th=[   90], 10.00th=[   92], 20.00th=[   96],
     | 30.00th=[  100], 40.00th=[  103], 50.00th=[  107], 60.00th=[  112],
     | 70.00th=[  124], 80.00th=[  138], 90.00th=[  165], 95.00th=[  205],
     | 99.00th=[  264], 99.50th=[  268], 99.90th=[  279], 99.95th=[  288],
     | 99.99th=[  288]
   bw (  KiB/s): min=145699, max=329728, per=99.36%, avg=268578.10, stdev=58043.28, samples=10
   iops        : min=  142, max=  322, avg=262.20, stdev=56.74, samples=10
  lat (msec)   : 2=18.22%, 4=18.11%, 10=8.49%, 20=2.73%, 50=1.40%
  lat (msec)   : 100=17.17%, 250=34.38%, 500=0.70%
  cpu          : usr=1.34%, sys=0.22%, ctx=6459, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=1257,1311,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ32T1_WRITE: (groupid=2, jobs=1): err= 0: pid=3020657: Mon Jul 13 16:29:49 2026
  write: IOPS=413, BW=419MiB/s (440MB/s)(2122MiB/5061msec); 0 zone resets
    slat (usec): min=11, max=5952, avg=38.02, stdev=152.65
    clat (msec): min=59, max=160, avg=76.85, stdev=17.51
     lat (msec): min=59, max=160, avg=76.89, stdev=17.50
    clat percentiles (msec):
     |  1.00th=[   61],  5.00th=[   62], 10.00th=[   63], 20.00th=[   64],
     | 30.00th=[   67], 40.00th=[   70], 50.00th=[   73], 60.00th=[   77],
     | 70.00th=[   81], 80.00th=[   84], 90.00th=[   90], 95.00th=[  111],
     | 99.00th=[  155], 99.50th=[  159], 99.90th=[  161], 99.95th=[  161],
     | 99.99th=[  161]
   bw (  KiB/s): min=364544, max=503808, per=99.73%, avg=428193.20, stdev=53299.68, samples=10
   iops        : min=  356, max=  492, avg=418.00, stdev=52.13, samples=10
  lat (msec)   : 100=94.98%, 250=6.50%
  cpu          : usr=1.44%, sys=0.06%, ctx=2251, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2091,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
seq_1MQ4T1_READ: (groupid=3, jobs=1): err= 0: pid=3020897: Mon Jul 13 16:29:49 2026
  read: IOPS=963, BW=964MiB/s (1011MB/s)(4824MiB/5004msec)
    slat (nsec): min=750, max=109480, avg=2300.88, stdev=2268.21
    clat (usec): min=1685, max=38567, avg=4147.17, stdev=1628.55
     lat (usec): min=1687, max=38569, avg=4149.47, stdev=1628.50
    clat percentiles (usec):
     |  1.00th=[ 2180],  5.00th=[ 2474], 10.00th=[ 2704], 20.00th=[ 3064],
     | 30.00th=[ 3359], 40.00th=[ 3654], 50.00th=[ 3916], 60.00th=[ 4228],
     | 70.00th=[ 4621], 80.00th=[ 5080], 90.00th=[ 5735], 95.00th=[ 6390],
     | 99.00th=[ 7701], 99.50th=[ 8455], 99.90th=[23200], 99.95th=[38011],
     | 99.99th=[38536]
   bw (  KiB/s): min=790528, max=1310720, per=100.00%, avg=987552.50, stdev=179682.90, samples=10
   iops        : min=  772, max= 1280, avg=964.30, stdev=175.47, samples=10
  lat (msec)   : 2=0.37%, 4=52.44%, 10=46.90%, 20=0.25%, 50=0.10%
  cpu          : usr=0.08%, sys=0.72%, ctx=4781, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=4821,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T1_MIXED: (groupid=4, jobs=1): err= 0: pid=3021161: Mon Jul 13 16:29:49 2026
  read: IOPS=230, BW=231MiB/s (242MB/s)(1158MiB/5010msec)
    slat (nsec): min=770, max=9430, avg=2249.27, stdev=752.67
    clat (usec): min=1365, max=64176, avg=5128.58, stdev=7328.36
     lat (usec): min=1368, max=64178, avg=5130.83, stdev=7328.35
    clat percentiles (usec):
     |  1.00th=[ 1778],  5.00th=[ 2008], 10.00th=[ 2114], 20.00th=[ 2180],
     | 30.00th=[ 2245], 40.00th=[ 2343], 50.00th=[ 2442], 60.00th=[ 2606],
     | 70.00th=[ 2999], 80.00th=[ 4948], 90.00th=[11338], 95.00th=[22676],
     | 99.00th=[38536], 99.50th=[46400], 99.90th=[51119], 99.95th=[64226],
     | 99.99th=[64226]
   bw (  KiB/s): min=133386, max=342701, per=100.00%, avg=237253.50, stdev=65787.11, samples=10
   iops        : min=  130, max=  334, avg=231.60, stdev=64.17, samples=10
  write: IOPS=235, BW=236MiB/s (247MB/s)(1182MiB/5010msec); 0 zone resets
    slat (nsec): min=13180, max=54370, avg=27338.26, stdev=6226.01
    clat (usec): min=2592, max=55707, avg=11915.04, stdev=7668.82
     lat (usec): min=2618, max=55729, avg=11942.37, stdev=7669.06
    clat percentiles (usec):
     |  1.00th=[ 4883],  5.00th=[ 5669], 10.00th=[ 6980], 20.00th=[ 7701],
     | 30.00th=[ 8029], 40.00th=[ 9372], 50.00th=[10159], 60.00th=[10421],
     | 70.00th=[11338], 80.00th=[12911], 90.00th=[18482], 95.00th=[30540],
     | 99.00th=[43254], 99.50th=[50070], 99.90th=[54789], 99.95th=[55837],
     | 99.99th=[55837]
   bw (  KiB/s): min=163840, max=329728, per=99.90%, avg=241348.30, stdev=56959.37, samples=10
   iops        : min=  160, max=  322, avg=235.60, stdev=55.60, samples=10
  lat (msec)   : 2=2.31%, 4=36.54%, 10=29.35%, 20=24.43%, 50=7.15%
  lat (msec)   : 100=0.34%
  cpu          : usr=0.34%, sys=0.80%, ctx=4694, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1157,1180,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T1_WRITE: (groupid=5, jobs=1): err= 0: pid=3021413: Mon Jul 13 16:29:49 2026
  write: IOPS=394, BW=395MiB/s (414MB/s)(1979MiB/5010msec); 0 zone resets
    slat (nsec): min=14500, max=78750, avg=30657.27, stdev=6830.54
    clat (msec): min=7, max=107, avg=10.10, stdev= 5.34
     lat (msec): min=7, max=107, avg=10.13, stdev= 5.34
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[    9], 20.00th=[    9],
     | 30.00th=[    9], 40.00th=[   10], 50.00th=[   10], 60.00th=[   10],
     | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   15],
     | 99.00th=[   22], 99.50th=[   36], 99.90th=[  107], 99.95th=[  108],
     | 99.99th=[  108]
   bw (  KiB/s): min=337920, max=457619, per=100.00%, avg=404518.40, stdev=43353.59, samples=10
   iops        : min=  330, max=  446, avg=394.80, stdev=42.30, samples=10
  lat (msec)   : 10=78.44%, 20=20.50%, 50=0.81%, 100=0.20%, 250=0.20%
  cpu          : usr=1.34%, sys=0.12%, ctx=2032, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1976,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_READ: (groupid=6, jobs=16): err= 0: pid=3021657: Mon Jul 13 16:29:49 2026
  read: IOPS=1218, BW=1228MiB/s (1288MB/s)(6211MiB/5057msec)
    slat (nsec): min=910, max=2687.1k, avg=7144.51, stdev=76378.64
    clat (msec): min=2, max=155, avg=52.09, stdev=17.59
     lat (msec): min=2, max=155, avg=52.09, stdev=17.58
    clat percentiles (msec):
     |  1.00th=[   12],  5.00th=[   25], 10.00th=[   32], 20.00th=[   40],
     | 30.00th=[   45], 40.00th=[   49], 50.00th=[   52], 60.00th=[   55],
     | 70.00th=[   58], 80.00th=[   64], 90.00th=[   74], 95.00th=[   83],
     | 99.00th=[  104], 99.50th=[  122], 99.90th=[  150], 99.95th=[  150],
     | 99.99th=[  157]
   bw (  MiB/s): min= 1016, max= 1396, per=100.00%, avg=1229.60, stdev= 6.71, samples=160
   iops        : min= 1016, max= 1388, avg=1226.70, stdev= 6.61, samples=160
  lat (msec)   : 4=0.05%, 10=0.81%, 20=2.00%, 50=43.18%, 100=53.59%
  lat (msec)   : 250=1.15%
  cpu          : usr=0.02%, sys=0.06%, ctx=6260, majf=0, minf=597
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6163,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_MIXED: (groupid=7, jobs=16): err= 0: pid=3021994: Mon Jul 13 16:29:49 2026
  read: IOPS=306, BW=307MiB/s (322MB/s)(1565MiB/5091msec)
    slat (nsec): min=840, max=77500, avg=3890.30, stdev=4023.86
    clat (usec): min=1855, max=802233, avg=66545.94, stdev=58202.61
     lat (usec): min=1858, max=802238, avg=66549.83, stdev=58203.48
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    8], 10.00th=[   13], 20.00th=[   25],
     | 30.00th=[   35], 40.00th=[   44], 50.00th=[   54], 60.00th=[   65],
     | 70.00th=[   80], 80.00th=[  100], 90.00th=[  127], 95.00th=[  157],
     | 99.00th=[  279], 99.50th=[  347], 99.90th=[  502], 99.95th=[  802],
     | 99.99th=[  802]
   bw (  KiB/s): min=116747, max=630406, per=100.00%, avg=322509.76, stdev=10254.02, samples=156
   iops        : min=  113, max=  613, avg=314.04, stdev= 9.99, samples=156
  write: IOPS=291, BW=293MiB/s (308MB/s)(1493MiB/5091msec); 0 zone resets
    slat (usec): min=13, max=138, avg=28.81, stdev= 8.63
    clat (msec): min=3, max=1244, avg=146.82, stdev=113.07
     lat (msec): min=3, max=1244, avg=146.85, stdev=113.07
    clat percentiles (msec):
     |  1.00th=[   15],  5.00th=[   33], 10.00th=[   47], 20.00th=[   65],
     | 30.00th=[   84], 40.00th=[  101], 50.00th=[  118], 60.00th=[  140],
     | 70.00th=[  167], 80.00th=[  209], 90.00th=[  275], 95.00th=[  363],
     | 99.00th=[  558], 99.50th=[  743], 99.90th=[  936], 99.95th=[ 1250],
     | 99.99th=[ 1250]
   bw (  KiB/s): min=118738, max=446479, per=99.23%, avg=298003.10, stdev=6453.27, samples=160
   iops        : min=  114, max=  435, avg=290.10, stdev= 6.31, samples=160
  lat (msec)   : 2=0.03%, 4=1.51%, 10=2.20%, 20=5.52%, 50=19.89%
  lat (msec)   : 100=32.22%, 250=31.72%, 500=6.67%, 750=0.49%, 1000=0.23%
  lat (msec)   : 2000=0.03%
  cpu          : usr=0.07%, sys=0.05%, ctx=6082, majf=0, minf=596
  IO depths    : 1=0.0%, 2=1.1%, 4=98.9%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1558,1484,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
seq_1MQ4T16_WRITE: (groupid=8, jobs=16): err= 0: pid=3022326: Mon Jul 13 16:29:49 2026
  write: IOPS=586, BW=596MiB/s (625MB/s)(3036MiB/5092msec); 0 zone resets
    slat (usec): min=12, max=4978, avg=36.67, stdev=93.18
    clat (msec): min=15, max=268, avg=107.59, stdev=20.13
     lat (msec): min=15, max=268, avg=107.63, stdev=20.13
    clat percentiles (msec):
     |  1.00th=[   62],  5.00th=[   80], 10.00th=[   91], 20.00th=[   97],
     | 30.00th=[  101], 40.00th=[  104], 50.00th=[  106], 60.00th=[  109],
     | 70.00th=[  112], 80.00th=[  117], 90.00th=[  127], 95.00th=[  138],
     | 99.00th=[  190], 99.50th=[  207], 99.90th=[  234], 99.95th=[  239],
     | 99.99th=[  271]
   bw (  KiB/s): min=499712, max=683779, per=99.61%, avg=608149.20, stdev=3292.17, samples=160
   iops        : min=  488, max=  667, avg=593.00, stdev= 3.20, samples=160
  lat (msec)   : 20=0.03%, 50=0.37%, 100=28.21%, 250=72.96%, 500=0.03%
  cpu          : usr=0.07%, sys=0.09%, ctx=3187, majf=0, minf=598
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2988,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ32T1_READ: (groupid=9, jobs=1): err= 0: pid=3022664: Mon Jul 13 16:29:49 2026
  read: IOPS=39.1k, BW=153MiB/s (160MB/s)(764MiB/5002msec)
    slat (nsec): min=730, max=109750, avg=2019.99, stdev=1013.07
    clat (usec): min=7, max=41268, avg=815.83, stdev=862.20
     lat (usec): min=11, max=41269, avg=817.85, stdev=862.20
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   21], 10.00th=[   29], 20.00th=[  306],
     | 30.00th=[  383], 40.00th=[  490], 50.00th=[  627], 60.00th=[  799],
     | 70.00th=[ 1012], 80.00th=[ 1287], 90.00th=[ 1696], 95.00th=[ 2057],
     | 99.00th=[ 2802], 99.50th=[ 3261], 99.90th=[ 8586], 99.95th=[17171],
     | 99.99th=[25822]
   bw (  KiB/s): min=122172, max=163992, per=100.00%, avg=156508.20, stdev=12270.23, samples=10
   iops        : min=30543, max=40998, avg=39127.00, stdev=3067.53, samples=10
  lat (usec)   : 10=0.01%, 20=4.04%, 50=7.73%, 100=0.30%, 250=0.92%
  lat (usec)   : 500=28.00%, 750=16.47%, 1000=12.06%
  lat (msec)   : 2=24.94%, 4=5.32%, 10=0.15%, 20=0.05%, 50=0.04%
  cpu          : usr=5.66%, sys=18.10%, ctx=158709, majf=0, minf=38
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=195574,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ32T1_MIXED: (groupid=10, jobs=1): err= 0: pid=3022916: Mon Jul 13 16:29:49 2026
  read: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(226MiB/5003msec)
    slat (nsec): min=730, max=28430, avg=1787.19, stdev=732.20
    clat (usec): min=7, max=57855, avg=385.94, stdev=1285.69
     lat (usec): min=9, max=57858, avg=387.73, stdev=1285.74
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   19], 10.00th=[   24], 20.00th=[  243],
     | 30.00th=[  265], 40.00th=[  269], 50.00th=[  277], 60.00th=[  285],
     | 70.00th=[  297], 80.00th=[  334], 90.00th=[  412], 95.00th=[  502],
     | 99.00th=[ 3064], 99.50th=[ 9372], 99.90th=[19792], 99.95th=[24773],
     | 99.99th=[30278]
   bw (  KiB/s): min= 2632, max=69080, per=100.00%, avg=46343.10, stdev=25658.74, samples=10
   iops        : min=  658, max=17270, avg=11585.70, stdev=6414.62, samples=10
  write: IOPS=11.6k, BW=45.2MiB/s (47.4MB/s)(226MiB/5003msec); 0 zone resets
    slat (nsec): min=730, max=104770, avg=2014.27, stdev=870.69
    clat (usec): min=217, max=95328, avg=2373.99, stdev=5495.10
     lat (usec): min=220, max=95331, avg=2376.00, stdev=5495.13
    clat percentiles (usec):
     |  1.00th=[  635],  5.00th=[  914], 10.00th=[ 1074], 20.00th=[ 1287],
     | 30.00th=[ 1467], 40.00th=[ 1614], 50.00th=[ 1762], 60.00th=[ 1909],
     | 70.00th=[ 2089], 80.00th=[ 2278], 90.00th=[ 2606], 95.00th=[ 2933],
     | 99.00th=[32637], 99.50th=[50594], 99.90th=[78119], 99.95th=[83362],
     | 99.99th=[91751]
   bw (  KiB/s): min= 2440, max=68896, per=100.00%, avg=46325.90, stdev=25537.87, samples=10
   iops        : min=  610, max=17224, avg=11581.40, stdev=6384.42, samples=10
  lat (usec)   : 10=0.01%, 20=3.44%, 50=2.66%, 100=0.05%, 250=5.45%
  lat (usec)   : 500=36.15%, 750=2.65%, 1000=2.80%
  lat (msec)   : 2=29.07%, 4=16.42%, 10=0.46%, 20=0.23%, 50=0.38%
  lat (msec)   : 100=0.26%
  cpu          : usr=3.64%, sys=9.92%, ctx=110326, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=57900,57879,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ32T1_WRITE: (groupid=11, jobs=1): err= 0: pid=3023130: Mon Jul 13 16:29:49 2026
  write: IOPS=8454, BW=33.0MiB/s (34.7MB/s)(169MiB/5099msec); 0 zone resets
    slat (nsec): min=820, max=36910, avg=1810.42, stdev=562.10
    clat (usec): min=426, max=185835, avg=3793.97, stdev=10663.55
     lat (usec): min=427, max=185837, avg=3795.78, stdev=10663.62
    clat percentiles (usec):
     |  1.00th=[   955],  5.00th=[  1221], 10.00th=[  1450], 20.00th=[  1680],
     | 30.00th=[  1876], 40.00th=[  1991], 50.00th=[  2180], 60.00th=[  2311],
     | 70.00th=[  2474], 80.00th=[  2671], 90.00th=[  2999], 95.00th=[  3425],
     | 99.00th=[ 65799], 99.50th=[ 85459], 99.90th=[113771], 99.95th=[130548],
     | 99.99th=[170918]
   bw (  KiB/s): min= 1648, max=61731, per=100.00%, avg=34499.80, stdev=28244.52, samples=10
   iops        : min=  412, max=15432, avg=8624.80, stdev=7061.15, samples=10
  lat (usec)   : 500=0.01%, 750=0.23%, 1000=1.56%
  lat (msec)   : 2=39.25%, 4=55.42%, 10=0.69%, 20=0.19%, 50=1.05%
  lat (msec)   : 100=1.43%, 250=0.24%
  cpu          : usr=1.55%, sys=3.45%, ctx=43075, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,43110,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32
rnd_4KQ4T1_READ: (groupid=12, jobs=1): err= 0: pid=3023221: Mon Jul 13 16:29:49 2026
  read: IOPS=14.3k, BW=55.8MiB/s (58.5MB/s)(279MiB/5001msec)
    slat (nsec): min=730, max=105430, avg=1889.26, stdev=809.88
    clat (usec): min=11, max=52309, avg=277.56, stdev=434.88
     lat (usec): min=13, max=52313, avg=279.45, stdev=434.89
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   22], 10.00th=[  212], 20.00th=[  249],
     | 30.00th=[  265], 40.00th=[  269], 50.00th=[  277], 60.00th=[  281],
     | 70.00th=[  289], 80.00th=[  302], 90.00th=[  355], 95.00th=[  408],
     | 99.00th=[  586], 99.50th=[  644], 99.90th=[  947], 99.95th=[ 5932],
     | 99.99th=[18482]
   bw (  KiB/s): min=41146, max=60216, per=100.00%, avg=57159.90, stdev=5698.59, samples=10
   iops        : min=10286, max=15054, avg=14289.90, stdev=1424.80, samples=10
  lat (usec)   : 20=3.23%, 50=5.28%, 100=0.12%, 250=12.11%, 500=77.56%
  lat (usec)   : 750=1.52%, 1000=0.08%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.04%, 20=0.02%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.86%, sys=7.52%, ctx=70329, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=71426,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T1_MIXED: (groupid=13, jobs=1): err= 0: pid=3023529: Mon Jul 13 16:29:49 2026
  read: IOPS=7102, BW=27.7MiB/s (29.1MB/s)(139MiB/5001msec)
    slat (nsec): min=740, max=106280, avg=2199.04, stdev=1019.61
    clat (usec): min=10, max=45202, avg=328.44, stdev=911.86
     lat (usec): min=12, max=45205, avg=330.64, stdev=911.93
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   23], 10.00th=[  212], 20.00th=[  251],
     | 30.00th=[  265], 40.00th=[  273], 50.00th=[  277], 60.00th=[  285],
     | 70.00th=[  289], 80.00th=[  302], 90.00th=[  351], 95.00th=[  412],
     | 99.00th=[  603], 99.50th=[ 4047], 99.90th=[15401], 99.95th=[20055],
     | 99.99th=[30278]
   bw (  KiB/s): min= 1336, max=35472, per=100.00%, avg=28426.10, stdev=11151.26, samples=10
   iops        : min=  334, max= 8868, avg=7106.50, stdev=2787.82, samples=10
  write: IOPS=7060, BW=27.6MiB/s (28.9MB/s)(138MiB/5001msec); 0 zone resets
    slat (nsec): min=760, max=105840, avg=2417.90, stdev=927.41
    clat (usec): min=15, max=49155, avg=230.09, stdev=743.28
     lat (usec): min=16, max=49159, avg=232.51, stdev=743.32
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   28], 10.00th=[   31], 20.00th=[   36],
     | 30.00th=[   43], 40.00th=[   57], 50.00th=[  115], 60.00th=[  260],
     | 70.00th=[  293], 80.00th=[  322], 90.00th=[  461], 95.00th=[  562],
     | 99.00th=[  807], 99.50th=[ 1139], 99.90th=[11338], 99.95th=[15664],
     | 99.99th=[32900]
   bw (  KiB/s): min= 1168, max=35320, per=100.00%, avg=28258.90, stdev=11137.36, samples=10
   iops        : min=  292, max= 8830, avg=7064.60, stdev=2784.32, samples=10
  lat (usec)   : 20=1.64%, 50=21.15%, 100=6.18%, 250=10.08%, 500=55.72%
  lat (usec)   : 750=4.10%, 1000=0.48%
  lat (msec)   : 2=0.11%, 4=0.16%, 10=0.22%, 20=0.13%, 50=0.04%
  cpu          : usr=2.00%, sys=8.62%, ctx=69371, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=35519,35309,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T1_WRITE: (groupid=14, jobs=1): err= 0: pid=3023787: Mon Jul 13 16:29:49 2026
  write: IOPS=8813, BW=34.4MiB/s (36.1MB/s)(172MiB/5001msec); 0 zone resets
    slat (nsec): min=760, max=16710, avg=1427.08, stdev=430.54
    clat (usec): min=25, max=56893, avg=451.77, stdev=1462.63
     lat (usec): min=27, max=56895, avg=453.20, stdev=1462.70
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   59], 10.00th=[   64], 20.00th=[   74],
     | 30.00th=[  277], 40.00th=[  306], 50.00th=[  314], 60.00th=[  326],
     | 70.00th=[  506], 80.00th=[  553], 90.00th=[  586], 95.00th=[  791],
     | 99.00th=[ 1106], 99.50th=[ 9765], 99.90th=[25035], 99.95th=[28181],
     | 99.99th=[35914]
   bw (  KiB/s): min= 1488, max=48657, per=100.00%, avg=35279.80, stdev=19426.04, samples=10
   iops        : min=  372, max=12164, avg=8819.90, stdev=4856.48, samples=10
  lat (usec)   : 50=0.62%, 100=26.47%, 250=0.58%, 500=41.93%, 750=23.45%
  lat (usec)   : 1000=5.57%
  lat (msec)   : 2=0.52%, 4=0.09%, 10=0.30%, 20=0.33%, 50=0.17%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.30%, sys=3.34%, ctx=44034, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,44077,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_READ: (groupid=15, jobs=16): err= 0: pid=3024036: Mon Jul 13 16:29:49 2026
  read: IOPS=46.1k, BW=180MiB/s (189MB/s)(902MiB/5003msec)
    slat (nsec): min=750, max=109000, avg=2321.18, stdev=1119.35
    clat (usec): min=11, max=46589, avg=1386.45, stdev=2047.54
     lat (usec): min=13, max=46591, avg=1388.77, stdev=2047.57
    clat percentiles (usec):
     |  1.00th=[   20],  5.00th=[   29], 10.00th=[  281], 20.00th=[  338],
     | 30.00th=[  424], 40.00th=[  553], 50.00th=[  766], 60.00th=[ 1123],
     | 70.00th=[ 1631], 80.00th=[ 2278], 90.00th=[ 3064], 95.00th=[ 3621],
     | 99.00th=[ 8586], 99.50th=[15008], 99.90th=[26608], 99.95th=[29754],
     | 99.99th=[42206]
   bw (  KiB/s): min=24862, max=217951, per=99.27%, avg=183173.79, stdev=3600.59, samples=147
   iops        : min= 6209, max=54486, avg=45791.90, stdev=900.24, samples=147
  lat (usec)   : 20=1.28%, 50=5.06%, 100=0.05%, 250=0.63%, 500=29.61%
  lat (usec)   : 750=12.81%, 1000=7.63%
  lat (msec)   : 2=19.01%, 4=21.13%, 10=1.94%, 20=0.61%, 50=0.27%
  cpu          : usr=0.87%, sys=1.70%, ctx=231373, majf=0, minf=582
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=230738,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_MIXED: (groupid=16, jobs=16): err= 0: pid=3024375: Mon Jul 13 16:29:49 2026
  read: IOPS=14.8k, BW=57.7MiB/s (60.5MB/s)(290MiB/5027msec)
    slat (nsec): min=740, max=108231, avg=2391.54, stdev=1299.36
    clat (nsec): min=330, max=56047k, avg=2186495.21, stdev=4596941.43
     lat (usec): min=10, max=56048, avg=2188.89, stdev=4596.98
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   24], 10.00th=[   43], 20.00th=[  326],
     | 30.00th=[  420], 40.00th=[  553], 50.00th=[  742], 60.00th=[ 1004],
     | 70.00th=[ 1336], 80.00th=[ 1811], 90.00th=[ 4883], 95.00th=[12780],
     | 99.00th=[23725], 99.50th=[26608], 99.90th=[33817], 99.95th=[41681],
     | 99.99th=[54789]
   bw (  KiB/s): min=10481, max=156704, per=100.00%, avg=59421.50, stdev=3551.38, samples=160
   iops        : min= 2618, max=39176, avg=14854.30, stdev=887.89, samples=160
  write: IOPS=14.7k, BW=57.4MiB/s (60.2MB/s)(289MiB/5027msec); 0 zone resets
    slat (nsec): min=770, max=159450, avg=2568.53, stdev=1423.14
    clat (nsec): min=940, max=73387k, avg=2139941.39, stdev=5453027.74
     lat (usec): min=16, max=73389, avg=2142.51, stdev=5453.10
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   29], 10.00th=[   33], 20.00th=[   40],
     | 30.00th=[   56], 40.00th=[  314], 50.00th=[  545], 60.00th=[  881],
     | 70.00th=[ 1303], 80.00th=[ 1909], 90.00th=[ 3359], 95.00th=[12387],
     | 99.00th=[29230], 99.50th=[35390], 99.90th=[48497], 99.95th=[53216],
     | 99.99th=[59507]
   bw (  KiB/s): min=11070, max=155392, per=100.00%, avg=59071.30, stdev=3525.24, samples=160
   iops        : min= 2766, max=38848, avg=14766.70, stdev=881.36, samples=160
  lat (nsec)   : 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=1.32%, 50=17.85%, 100=3.46%
  lat (usec)   : 250=1.59%, 500=18.12%, 750=11.10%, 1000=8.05%
  lat (msec)   : 2=20.50%, 4=8.20%, 10=3.52%, 20=4.03%, 50=2.23%
  lat (msec)   : 100=0.05%
  cpu          : usr=0.43%, sys=1.25%, ctx=158007, majf=0, minf=582
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=74278,73842,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4
rnd_4KQ4T16_WRITE: (groupid=17, jobs=16): err= 0: pid=3024689: Mon Jul 13 16:29:49 2026
  write: IOPS=22.6k, BW=88.4MiB/s (92.7MB/s)(442MiB/5002msec); 0 zone resets
    slat (nsec): min=740, max=514680, avg=2082.25, stdev=2027.59
    clat (usec): min=5, max=105433, avg=2830.32, stdev=6933.58
     lat (usec): min=22, max=105436, avg=2832.40, stdev=6933.69
    clat percentiles (usec):
     |  1.00th=[   60],  5.00th=[   75], 10.00th=[   91], 20.00th=[  359],
     | 30.00th=[  478], 40.00th=[  627], 50.00th=[  758], 60.00th=[  906],
     | 70.00th=[ 1090], 80.00th=[ 1385], 90.00th=[ 7570], 95.00th=[18482],
     | 99.00th=[34866], 99.50th=[41157], 99.90th=[54789], 99.95th=[61604],
     | 99.99th=[83362]
   bw (  KiB/s): min=12960, max=271648, per=98.10%, avg=88773.16, stdev=5994.78, samples=156
   iops        : min= 3240, max=67912, avg=22192.86, stdev=1498.71, samples=156
  lat (usec)   : 10=0.01%, 20=0.01%, 50=0.18%, 100=11.28%, 250=0.88%
  lat (usec)   : 500=19.06%, 750=18.04%, 1000=15.94%
  lat (msec)   : 2=21.63%, 4=1.24%, 10=2.97%, 20=4.35%, 50=4.32%
  lat (msec)   : 100=0.14%, 250=0.01%
  cpu          : usr=0.39%, sys=0.65%, ctx=112401, majf=0, minf=582
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,113117,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=1540MiB/s (1615MB/s), 1540MiB/s-1540MiB/s (1615MB/s-1615MB/s), io=7736MiB (8112MB), run=5023-5023msec

Run status group 1 (all jobs):
   READ: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=1258MiB (1319MB), run=5080-5080msec
  WRITE: bw=264MiB/s (277MB/s), 264MiB/s-264MiB/s (277MB/s-277MB/s), io=1341MiB (1406MB), run=5080-5080msec

Run status group 2 (all jobs):
  WRITE: bw=419MiB/s (440MB/s), 419MiB/s-419MiB/s (440MB/s-440MB/s), io=2122MiB (2225MB), run=5061-5061msec

Run status group 3 (all jobs):
   READ: bw=964MiB/s (1011MB/s), 964MiB/s-964MiB/s (1011MB/s-1011MB/s), io=4824MiB (5058MB), run=5004-5004msec

Run status group 4 (all jobs):
   READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=1158MiB (1214MB), run=5010-5010msec
  WRITE: bw=236MiB/s (247MB/s), 236MiB/s-236MiB/s (247MB/s-247MB/s), io=1182MiB (1239MB), run=5010-5010msec

Run status group 5 (all jobs):
  WRITE: bw=395MiB/s (414MB/s), 395MiB/s-395MiB/s (414MB/s-414MB/s), io=1979MiB (2075MB), run=5010-5010msec

Run status group 6 (all jobs):
   READ: bw=1228MiB/s (1288MB/s), 1228MiB/s-1228MiB/s (1288MB/s-1288MB/s), io=6211MiB (6513MB), run=5057-5057msec

Run status group 7 (all jobs):
   READ: bw=307MiB/s (322MB/s), 307MiB/s-307MiB/s (322MB/s-322MB/s), io=1565MiB (1641MB), run=5091-5091msec
  WRITE: bw=293MiB/s (308MB/s), 293MiB/s-293MiB/s (308MB/s-308MB/s), io=1493MiB (1566MB), run=5091-5091msec

Run status group 8 (all jobs):
  WRITE: bw=596MiB/s (625MB/s), 596MiB/s-596MiB/s (625MB/s-625MB/s), io=3036MiB (3183MB), run=5092-5092msec

Run status group 9 (all jobs):
   READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=764MiB (801MB), run=5002-5002msec

Run status group 10 (all jobs):
   READ: bw=45.2MiB/s (47.4MB/s), 45.2MiB/s-45.2MiB/s (47.4MB/s-47.4MB/s), io=226MiB (237MB), run=5003-5003msec
  WRITE: bw=45.2MiB/s (47.4MB/s), 45.2MiB/s-45.2MiB/s (47.4MB/s-47.4MB/s), io=226MiB (237MB), run=5003-5003msec

Run status group 11 (all jobs):
  WRITE: bw=33.0MiB/s (34.7MB/s), 33.0MiB/s-33.0MiB/s (34.7MB/s-34.7MB/s), io=169MiB (177MB), run=5099-5099msec

Run status group 12 (all jobs):
   READ: bw=55.8MiB/s (58.5MB/s), 55.8MiB/s-55.8MiB/s (58.5MB/s-58.5MB/s), io=279MiB (293MB), run=5001-5001msec

Run status group 13 (all jobs):
   READ: bw=27.7MiB/s (29.1MB/s), 27.7MiB/s-27.7MiB/s (29.1MB/s-29.1MB/s), io=139MiB (145MB), run=5001-5001msec
  WRITE: bw=27.6MiB/s (28.9MB/s), 27.6MiB/s-27.6MiB/s (28.9MB/s-28.9MB/s), io=138MiB (145MB), run=5001-5001msec

Run status group 14 (all jobs):
  WRITE: bw=34.4MiB/s (36.1MB/s), 34.4MiB/s-34.4MiB/s (36.1MB/s-36.1MB/s), io=172MiB (181MB), run=5001-5001msec

Run status group 15 (all jobs):
   READ: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=902MiB (945MB), run=5003-5003msec

Run status group 16 (all jobs):
   READ: bw=57.7MiB/s (60.5MB/s), 57.7MiB/s-57.7MiB/s (60.5MB/s-60.5MB/s), io=290MiB (304MB), run=5027-5027msec
  WRITE: bw=57.4MiB/s (60.2MB/s), 57.4MiB/s-57.4MiB/s (60.2MB/s-60.2MB/s), io=289MiB (303MB), run=5027-5027msec

Run status group 17 (all jobs):
  WRITE: bw=88.4MiB/s (92.7MB/s), 88.4MiB/s-88.4MiB/s (92.7MB/s-92.7MB/s), io=442MiB (464MB), run=5002-5002msec

With ARC/L2ARC I now have exceeded the expected performance these drives are supposed to deliver, often by as much as 75%. Without it they're performing exactly as you would expect from four SATA3 drives running as a linked device in a raidz1 - (up to) triple drive performance in read, single drive performance in write. Funnily enough, I found the answer on reddit on a completely unrelated post and comment to anything performance related. The numbers now match between ext4 on md raid5 and zfs raidz1 with any io engine (libaio, io_uring, ...)!

0 replies

Navigation Menu

Uh oh!

Enabling ARC and/or L2ARC causes throughput to drop significantly. #18752

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Uh oh!

Enabling ARC and/or L2ARC causes throughput to drop significantly. #18752

Uh oh!

Uh oh!

Xaymar Jul 6, 2026

System Information

Replies: 2 comments · 2 replies

Uh oh!

amotin Jul 6, 2026 Collaborator

Uh oh!

Uh oh!

Xaymar Jul 6, 2026 Author

Uh oh!

Uh oh!

Xaymar Jul 6, 2026 Author

ARC/L2ARC enabled and waited 30m for it to be "hot":

Immediately after disabling ARC/L2ARC

30m after disabling ARC/L2ARC:

Uh oh!

Uh oh!

Xaymar Jul 13, 2026 Author

Xaymar
Jul 6, 2026

Replies: 2 comments 2 replies

amotin
Jul 6, 2026
Collaborator

Xaymar Jul 6, 2026
Author

Xaymar Jul 6, 2026
Author

Xaymar
Jul 13, 2026
Author