One warm-up run does not warm up: the first timed run is the slowest in ~15 of 21 queries

[sandeepkunkunuru]

One warm-up run does not warm the query up. The first timed run is
systematically the slowest, and #749's --print-runs makes it visible on its
first real use.

SF10, 84090b4, --runs 3 --print-runs, series in execution order:

runs IS1    spread  1.77x  0.05ms 0.03ms 0.03ms
runs IS2    spread  1.68x  0.11ms 0.07ms 0.08ms
runs IS6    spread  1.88x  0.06ms 0.03ms 0.03ms
runs IS7    spread  1.42x  0.17ms 0.12ms 0.13ms
runs IC4    spread  1.38x  27.5ms 21.9ms 19.9ms
runs IC7    spread  1.58x  0.41ms 0.28ms 0.26ms
runs IC11   spread  1.12x  54.5ms 50.1ms 48.7ms

Roughly 15 of 21 queries have run 1 as their slowest, and on the short reads
it is 1.4–1.9×. run_benchmark does one warm-up before the timed runs, so this
is a second warm-up effect the first one did not absorb.

Why it matters

Every spread statistic in this repo is inflated by it. #716 argued IC2 is
uniquely unstable partly from spread; the adjacency-tier comparison (#746) read
per-query differences of 1.3–1.9× off 3-run medians and two of them later
reversed. A systematic first-run penalty of up to 1.9× on short reads is a
plausible contributor to both, and nobody could see it because the series was
discarded.

It also biases min (never from run 1) and max (usually run 1), so any
max/min spread — which is what analyse_noise.py and CH-REGRESS reason
about — carries it.

The median of three is the least affected, which is why the published table has
held up. But the published table is not the only thing people read.

What to do

Two options, and I have deliberately not picked one because it shifts published
numbers:

1. Warm up more than once. Two or three warm-ups, discarded. Simple, and it
   makes --runs 3 mean what it says.
2. Discard the first timed run when runs > 3, reporting the rest. Keeps
   the warm-up cheap and is what most benchmark harnesses do.

Either changes every number slightly, so it wants the same treatment as #746:
decide, then re-run the three-engine comparison on that basis rather than let
the default shift under a published table.

What is not in question is that the current output describes a distribution
that includes a warm-up artefact and calls it variance.

[sandeepkunkunuru]

Quantified, with 21 runs and the series printed. The warm-up takes one to
three runs, and --runs 3 reports a median taken from inside it.

SF10, 84090b4, --runs 21 --print-runs, calibration 39 → 39 ms:

runs IS3    1.18x  0.49 0.47 0.44 0.45 0.44 0.43 0.45 0.43 … 0.42 0.42 0.42
runs IC2    1.22x  39.2 35.1 34.4 34.3 34.1 32.0 32.8 34.1 … 34.0 34.4 33.9
runs IC4    1.45x  27.7 22.5 19.9 19.9 19.8 19.7 19.6 19.7 … 19.4 19.3 19.2
runs IC6    1.02x   226  223  222  226  223  223  224  223 …  222  223  223
runs IC11   1.13x  53.9 49.9 48.6 48.2 47.9 47.9 47.7 47.8 … 48.1 48.2 48.4

IC4 needs three runs to settle — 27.7, then 22.5, then 19.9 against a steady
~19.3. IS3 and IC11 need two. IC6 is warm immediately, which is the control that
says this is per-query behaviour rather than a host effect.

What that does to the published numbers

With --runs 3 the reported median is the middle of the three warming runs:

query	3-run median would be	steady state	bias
IC4	~22.5 ms	~19.3 ms	+17%
IS3	~0.47 ms	~0.43 ms	+9%
IC11	~49.9 ms	~48.0 ms	+4%
IC2	~35.1 ms	~34.0 ms	+3%
IC6	~223 ms	~223 ms	0%

So every published Samyama figure is biased against us, by up to 17% on the
worst query — and unevenly, which is worse than uniformly, because it distorts
ratios between queries and between runs.

It also explains a class of confusion directly: a "spread" of 1.45× on IC4 is
not variance at all, it is a warm-up ramp. Tier comparisons read off 3-run
medians (#746) were sampling that ramp, which is a second mechanism — alongside
plain non-reproduction — for why two of them flipped sign between sessions.

Recommendation

Warm up until it settles, or discard the first two timed runs. Given IC4
takes three, one warm-up is not enough and two is marginal; discarding the first
two of --runs 5 would be robust and still cheap.

Either shifts every published number downwards, so it wants deciding
explicitly and then a re-run — the same treatment as #746, and the same reason:
a default that changes underneath a published table is worse than a wrong
default that is documented.

Raw: 40-tier-buffer.log in the 2026-08-25 archive.

[MandarapuMadhulatha]

Audit for work already landed. PARTIAL — the instrument landed, the default did not change.

Done — --warmup N exists and the header discloses it: "Runs per query: N (M warm-up, discarded)" (benches/ldbc_benchmark.rs:992), with the warm-up passes run and discarded at line 638.

Not done — the default is still 1 (benches/ldbc_benchmark.rs:833), deliberately: e33d87d says "The default stays 1... #752 holds the evidence for what the default should become." So every published spread still carries the artifact this issue measured, and this issue is the open decision about the default rather than a missing feature.

Remaining: pick the default from the evidence already in the body, or say in the bench header that the published spreads include a cold first run.

[MandarapuMadhulatha]

Closing: addressed on main by 34fe769 (#1536). The bench now discloses the warm-up count and the cold-first-run bias, and the default of one warm-up run was kept on purpose.

---

Generated by Claude Code