Skip to content

Drop a misbehaving paired baseline instead of failing the run - #72

Merged
martian56 merged 1 commit into
mainfrom
bench-resilient
Jul 25, 2026
Merged

Drop a misbehaving paired baseline instead of failing the run#72
martian56 merged 1 commit into
mainfrom
bench-resilient

Conversation

@martian56

Copy link
Copy Markdown
Owner

Follow up to #69. The paired baseline server failed its setup statement on the arm64 profile and failed the whole bench job, which meant #71 could not be measured at all.

The paired baseline is an optimisation on measurement quality, not a requirement. If its server or its loadgen misbehaves the run now discards it and the report falls back to the cross run baseline, which is what happened before #69 anyway.

The baseline server failed its setup statement on one profile and took
the whole bench job with it. If either its server or its loadgen
misbehaves the paired baseline is discarded and the report falls back to
the cross run one.
Repository owner deleted a comment from coderabbitai Bot Jul 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Bench report

✅ Best improvement: linux-x64-full mixed at +19.6%. Nothing regressed beyond noise.

Baseline: main @ same runn. Cells read baseline → this PR. Verdicts need more than ±15% to leave the noise band.

linux-arm64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 30378 → 30577 ⚪ +0.7% 0.23 → 0.23 - -
mixed 13698 → 12641 ⚪ -7.7% 0.51 → 0.43 12.46 → 20.83 30.85 → 31.82
writes 5495 → 5963 ⚪ +8.5% - 9.32 → 7.48 29.17 → 26.19

linux-x64-1cpu-1gb

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 23930 new 0.22 - -
mixed 6221 new 0.34 39.43 278.54
writes 2051 new - 31.46 182.58

linux-x64-2cpu-2gb

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 17424 new 0.38 - -
mixed 11901 new 0.64 3.23 26.00
writes 6190 new - 1.94 22.64

linux-x64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 63521 → 63949 ⚪ +0.7% 0.11 → 0.11 - -
mixed 15698 → 18767 🟢 +19.6% 0.14 → 0.13 24.48 → 19.21 58.63 → 40.18
writes 6212 → 5950 ⚪ -4.2% - 2.46 → 9.70 60.95 → 38.69

windows-x64-full

workload ops/s Δ ops/s read p99 ms write p99 ms write p99.9 ms
reads 22404 → 22344 ⚪ -0.3% 0.35 → 0.35 - -
mixed 6002 → 5890 ⚪ -1.9% 0.34 → 0.35 23.73 → 24.97 38.33 → 37.20
writes 772 → 782 ⚪ +1.3% - 16.21 → 16.42 25.20 → 25.77

Field reference

All engines in identical containers, same driver, measured 2026-07-24 @ ba0ec39. A different harness than the tables above, so read this as standing, not as this PR's delta.

linux-arm64-full

engine reads mixed writes
quantadb 17978 12420 (27ms wp99) 6449 (19ms wp99)
postgres18 15227 14723 (2ms wp99) 9923 (1ms wp99)
mariadb114 4221 4437 (8ms wp99) 6666 (3ms wp99)
mysql84 4258 4394 (9ms wp99) 4274 (4ms wp99)

linux-x64-1cpu-1gb

engine reads mixed writes
quantadb 19419 10854 (29ms wp99) 5170 (39ms wp99)
postgres18 10196 8580 (43ms wp99) 5912 (46ms wp99)
mariadb114 3455 3732 (4ms wp99) 5626 (8ms wp99)
mysql84 3497 3628 (4ms wp99) 3259 (36ms wp99)

linux-x64-2cpu-2gb

engine reads mixed writes
quantadb 22528 16774 (5ms wp99) 8374 (4ms wp99)
postgres18 17060 17006 (2ms wp99) 11404 (1ms wp99)
mariadb114 3735 4006 (3ms wp99) 6741 (3ms wp99)
mysql84 3804 3845 (4ms wp99) 4847 (3ms wp99)

linux-x64-full

engine reads mixed writes
quantadb 18144 15075 (6ms wp99) 8025 (5ms wp99)
postgres18 15402 15154 (2ms wp99) 10872 (1ms wp99)
mariadb114 3437 3699 (4ms wp99) 6138 (3ms wp99)
mysql84 3490 3615 (4ms wp99) 3996 (5ms wp99)

Shared CI runners: treat single-profile swings as suggestive, agreement across profiles as real.

@martian56
martian56 merged commit 5718650 into main Jul 25, 2026
16 checks passed
@martian56
martian56 deleted the bench-resilient branch July 25, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant