Exp 264: size the initial result buffer from the SQL's high-water row count

[danReynolds]

Hypothesis

Every select() starts by allocating room for 256 rows, whatever the query is
about to return. decodeQuery calls
List<Object?>.filled(colCount * 256, null, growable: true), writes the rows in,
and truncates. For a point read on the repo's canonical six-column product row
that is 1,536 slots allocated and zero-filled to keep 6; on a twenty-column row,
5,376 to keep 21. It is pure waste, paid on every small read.

Someone already tried removing it. Exp 067
shrank the constant in April, regressed four small-query workloads by 40-44%, and
concluded the constant was well-tuned — explaining the regression with a real VM
property, that List.filled(n, null) is cheaper per slot when n is large.

Two things reopen it. The per-slot claim is true and doesn't support the
conclusion: measured per call, the large allocation is about six times cheaper per
slot and twenty times more expensive in total (423 ns for 1,536 slots against
21.5 ns for 12). And exp 067 shrank the constant for every query, so anything
larger than four rows had to double its way up — growth, not allocation, is what
regressed it. Exp 260
has since given the decoder a per-SQL memory of how many rows each statement
returns, so the shrink no longer has to be a guess applied to every query.

The bet: size the initial allocation from that memory, only ever downward, and
point reads get materially cheaper while everything larger runs today's code
unchanged.

Approach

RowSizeMemory gains a second output alongside exp 260's growth hint, and the two
take deliberately opposite statistics, because the two ends of the buffer's life
have opposite failure modes:

• hint (exp 260, untouched) steers growth and takes the smaller of the
  last two row counts. Over-sizing is the expensive mistake there.
• initialRows (new) sizes the initial allocation and takes the largest row
  count ever seen, plus 25% headroom, clamped at the existing 256 rows.
  Under-sizing is the expensive mistake here.

The clamp is what makes this safe rather than tuned, and it is the difference from
what exp 260 explicitly rejected: exp 260 tried applying a hint to the initial
allocation and measured a LIMIT ? statement's 50-row leg going 2.8x slower once
the hint saturated at its large leg. A value that can only ever shrink below the
default cannot do that. Anything that outgrows the initial buffer clamps back to
256 rows and allocates exactly what it allocates on main.

The mark is kept on the main isolate — ReaderPool already stamps exp 260's
growth hint onto each request, so this rides along beside it as
ReadRequest.initialRowHint. A reader worker must not answer this about itself:
it sees only a sample of a statement's executions, and a worker that decodes a
result over sacrificeSlotThreshold is destroyed outright, taking its mark with
it. The writer isolate, which no pool serves, keeps using its own memory — it is
long-lived and executes every read issued inside a transaction, so its local view
is the global one.

Everything else stays on the old path: a statement's first two executions, one
whose entry has been evicted from the pool's 32-entry memory, and every
selectBytes call (which builds no Dart buffer at all).

Full detail, including the sequence in which the two rules were found:
experiments/264-initial-alloc-size-memory.md.

Results

Four alternating-order passes, lane-isolated, 61 samples per lane, three arms:
origin/main, exp 264 without its eviction fix, and exp 264 as it ships. Host at
52-61% CPU idle; controls inside ±1.7%.

Exp 264 (final) against origin/main:

lane	role	pass 1	pass 2	pass 3	pass 4	mean	verdict
point1	primary	-10.3%	-15.2%	-19.9%	-8.5%	-13.5%	reproduced
point1-wide20	primary	-26.3%	-19.5%	-29.5%	-33.1%	-27.1%	reproduced
int20-10k	control	-0.5%	+0.4%	+1.3%	-0.2%	+0.3%	neutral
mixed6-10k	control	+0.1%	+0.8%	-0.7%	-1.7%	-0.4%	neutral
hint-thrash-overflows	guard	-1.3%	+0.9%	-2.6%	+2.2%	-0.2%	neutral

Cost of the eviction fix — exp 264 with it against exp 264 without:

lane	role	pass 1	pass 2	pass 3	pass 4	mean	verdict
point1	primary	-6.8%	-5.6%	-6.9%	+3.3%	-4.0%	sign-flips
point1-wide20	primary	-1.7%	+24.5%	-2.2%	+1.5%	+5.5%	sign-flips
int20-10k	control	+0.3%	+0.4%	-0.8%	+0.3%	+0.0%	neutral
mixed6-10k	control	-0.5%	+0.0%	+1.5%	-0.6%	+0.1%	neutral
hint-thrash-overflows	guard	-30.3%	-28.8%	-30.1%	-28.8%	-29.5%	reproduced

Point reads are 13.5% faster on the canonical six-column row and 27.1% on a
twenty-one-column one, reproduced in all four passes, and the ratio between them
is what the mechanism predicts — what a one-row result wastes is colCount * 255
slots, so the win scales with projection width. In absolute terms that is roughly
0.7 µs and 2.3 µs of worker time per read, against a whole point-read operation of
about 6 µs. Both controls are neutral by construction rather than by luck: a result
larger than 256 rows clamps to the same allocation in both arms.

hint-thrash-overflows is the lane added by this PR (see below) and is now level
with origin/main, so nothing regressed.

The change put exp 260's growth hint at risk, and that is fixed here

Removing exp 260's rowCount <= initialResultRows insert guard is what lets a small
statement be remembered — and it also lets every point read take one of the pool's
32 _rowHints slots. Exp 260 had those slots to itself, so after this change a
report query that runs constantly gets evicted by point-read churn and loses its
hint. Measured at +41.6% on a 5,000-row read behind 40 distinct one-off
statements, systematic enough that the unfixed arm's fastest sample beat the fixed
arm's slowest.

No existing lane could see this — every one of them uses a handful of SQL strings —
so two lanes were added that run never-before-seen statements between timed reads,
bracketing what an eviction policy can and cannot fix.

LRU promotion was the first fix tried and measured no improvement, while costing
the main isolate a map remove-and-reinsert per read. The problem is capacity, not
order. What ships is an eviction preference: drop an entry whose highWater has
never exceeded the initial buffer before dropping one that has. It restores exp 260's
exclusive tenure without a new magic capacity, and the O(32) scan runs only on an
overflowing miss — never per read, which the table confirms (both point lanes
sign-flip, both controls ±0.1%). The property is gated deterministically in
test/reader_pool_test.dart, which fails against insertion-order eviction.

A note on measurement environment

The first measurement pass was collected on a host at 0.0% CPU idle — an
unrelated VM at 190% CPU, under 500 MB free disk, and six run_release.dart
processes wedged 1-4 days from earlier sessions. It surfaced when a confirmation
pass read +50.6% on a lane the candidate cannot reach. Everything above is the
re-measurement after the host freed up. Comparing the two bounds the damage: six
points on the narrow point read (−7.4% → −13.5%), none on the wide one. Recorded as
claim 264.6, with the lesson in JOURNAL.md — a focused harness stamps nothing
about its host, unlike run_release.dart's gitDirty.

Release suite

The suite runs — it completes 14 of 16 scenarios (every read, write, streaming and
app-shaped lane) and then aborts inside the sqlite_async peer at [15/16] Memory,
the pre-existing #282 crash that also stopped exps 260 and 261. Exp 262's
per-scenario persistence made its first real save: the killed run still wrote an
artifact with 169 metrics, correctly self-marked partial: true,
scenariosCompleted: 14, repeatCount: 0.

That artifact is not committed and is not evidence for this experiment —
single-sample with no paired baseline, which is exactly what repeatCount: 0
marks and what the trend charts drop. As a no-regression sanity check it is
reassuring: point query 160,798 qps, 1,000-row select() 0.349 ms, batch insert
of 1,000 rows 0.389 ms, invalidation latency 0.058 ms — all at or better than
README.md's published figures.

Outcome

Accepted. A 7-27% win on point reads with no reachable regression, and the two
properties that make it safe are structural: the clamp means anything larger than
256 rows runs today's code byte-for-byte, and the high-water mark means a
mispredict is one-off (+1.6%) rather than periodic.

Exp 067's rejection stands for what it tested — an unconditional shrink is still
wrong. What it got wrong was the generalisation, and that is now recorded as a
transferable lesson: a rejection whose reasoning is a rate (per slot, per byte,
per row) has to be multiplied back out by the count the caller actually pays before
it closes a direction.

Would reopen if the pool's 32-entry row-size memory turns out to thrash on a real
application's statement mix — an evicted entry now loses a high-water mark as well
as a growth hint, so it would pay the first-jump cost again. The discriminating
measurement is cheap and is filed as an openCandidates entry.

Test plan

• dart run build_runner build --delete-conflicting-outputs — needed in a
  fresh worktree; benchmark/drift/*.g.dart is gitignored and CI generates it
  before analyze and test
• dart analyze --fatal-infos — no issues
• dart test --timeout 60s — 450 tests, all passing
• dart test test/result_buffer_sizing_test.dart — 21 tests covering the
  high-water rule, caller-over-local precedence, a statement that jumps from
  tiny to large, and an empty result that then grows
• focused AOT A/B, four alternating orders, lane-isolated, 61 samples/lane
• 50-process first-jump measurement for the one-time mispredict cost
• finalize_experiment.dart, check_knowledge_links.dart (79 claims, clean),
  check_experiment_dispositions.dart — all green
• CI green. The first CI run aborted (SIGABRT) inside the sqlite_async peer's
  ConnectionLease.notifyUpdates while
  benchmark_keyed_pk_subscriptions_test.dart was running; re-running the
  identical commit passed, and six local repetitions of the same three peer
  workload tests passed. main was green throughout, so this is the Benchmark: survive a peer crash, and pin the baseline the numbers compare against #282
  peer-instability family surfacing in the test job.

---

Correction to an earlier version of this description

An earlier revision of this PR reported that the release suite "no longer
compiles" against drift 2.34.3, and filed it as claim 264.4. That was wrong.
benchmark/drift/*.g.dart is gitignored by design and generated by
build_runner, which CI runs before analyze and test — the fresh experiment
worktree had simply never run it. Reproducing the same failure on origin/main
ruled out this diff but did not establish a repo breakage, because a missing build
step reproduces everywhere. The claim has been restated to what is actually true,
and the general lesson is recorded in JOURNAL.md.

[github-actions]

Belief impact

Learned

• 264.1 · Size the initial result buffer from the SQL's high-water row count
  Sizing decodeQuery's initial result allocation from a main-isolate high-water mark of the rows a SQL has ever returned, clamped into 1..256 rows …
• 264.2 · Size the initial result buffer from the SQL's high-water row count
  A result-buffer size hint whose failure mode is under-prediction must be a high-water mark, not a sliding window. Taking the larger of a SQL's last t…
• 264.3 · Size the initial result buffer from the SQL's high-water row count
  Exp 067's mechanism claim is per-slot true and decision-irrelevant. List<Object?>.filled(n, null, growable: true) followed by a write of k slots …
• 264.4 · Size the initial result buffer from the SQL's high-water row count
  The release suite runs, and exp 262's per-scenario persistence now demonstrably preserves a killed run. The suite completes 14 of 16 scenarios — ever…
• 264.5 · Size the initial result buffer from the SQL's high-water row count
  Exp 264's own change to ReaderPool._record put exp 260's growth hint at risk, and the fix is capacity/priority rather than ordering. Removing exp 2…
• 264.6 · Size the initial result buffer from the SQL's high-water row count
  Every wall-time figure in exp 264 was collected on a host at 0.0% CPU idle. top reported 57% user / 43% sys / 0% idle, with an unrelated Virtualiza…

What this changed

Claim 067.1 said decodeQuery's initial List<Object?>.filled(colCount * 256, null) must not be shrunk. That holds for the change exp 067 actually made — an unconditional cut to colCount * 4, which forces every result larger than four rows to double its way up — but not for the general statement, and the reasoning underneath it was wrong in an instructive way. Exp 067 attributed its regressions to a VM fast path that makes one large null-filled list cheaper per slot than a small one. That is true and it does not support the conclusion: a standalone AOT reproduction of the allocation shape measures 1,536 slots at 0.28 ns/slot against 12 slots at 1.8 ns/slot, so the large allocation is six times cheaper per slot and twenty times more expensive in total. What actually made exp 067 regress was growth, not allocation, and exp 260 has since supplied the thing that lets the shrink avoid growth: a per-SQL memory of how many rows each statement returns. Conditioned on that memory and clamped so it can only ever shrink, the initial allocation is worth a reproduced 7.4% on the canonical 6-column point read and 27.4% on a 21-column one.

A buffer-size hint that can under-predict must be a high-water mark, not a sliding window, and this is the single most transferable thing the run produced. Exp 260's growth hint takes the smaller of a SQL's last two row counts, which is right for growth because over-sizing is the expensive mistake there. Taking the larger of the last two by symmetry, for the initial allocation, was measured at +40% in all four order-flipped passes on a statement returning 3,300 rows behind a burst of 20-row executions: a window of two is defeated by any burst longer than two, because the two observations before the large execution are both small, so the large result is sized for 20 rows and doubles its way up — every time, not once. A high-water mark cannot repeat. What it gives up is a statement that was once large and is now permanently small, which keeps today's allocation forever.

The cost of a mispredict is a one-off +1.6%, and it is bounded by arithmetic rather than by measurement. The doubling chain's landing point depends on where it starts, so an undershoot can be favourable (undershoot-jump, 5,000 rows: from 25 rows the chain lands on 6,400, from 256 it lands on 8,192) or adverse (undershoot-mid, 3,300 rows: 6,400 against 4,096, four extra growths and 56% over-allocation). The adverse case measured 927.5 -> 942.5 us across 50 fresh processes per arm — about 15 us, inside the p10-p90 spread of either arm. Quoting only the favourable lane would have overstated the result; the pair is what brackets it.

Exp 260's argument for where a size memory lives generalises further than exp 260 needed it to, but it was not the cause of the +40%. Exp 260 put its hint on the main isolate because Isolate.exit on a result over sacrificeSlotThreshold ends the worker that produced it (claim 260.3) — an argument about large results, which a mark about small ones appears to escape. It does not: a high-water mark is only worth the observations feeding it, a four-worker pool hands each worker a sample, and a worker that does decode a large result is destroyed and its replacement starts over. Sampling bias and worker sacrifice are separate sufficient reasons to keep the mark on the main isolate. Worth recording that this was the run's first diagnosis of the +40% and was wrong: moving the memory to the pool fixed a real defect and left the lane at +40% in all four passes, because a window of length two is defeated by a burst longer than two wherever the window is kept. A correct fix for a real defect is not evidence that it was the defect being measured.

Peak RSS does not move, and that is the expected reading rather than a null result. Dart releases a truncated growable list's backing store, so the fixed 256-row allocation was never retained by a small result — only allocated, zero-filled and thrown away. Every lane holding one read live is flat within ~1 MB, consistent with claim 261.3. This is a transient-allocation win, in the same category exp 263 put selectBytes in when it separated allocation churn from footprint.

[Copilot]

Pull request overview

This experiment extends resqlite’s result-buffer sizing logic by shrinking decodeQuery’s initial List.filled(...) allocation for statements that have historically returned few rows, while preserving the existing behavior for any statement that might return larger results (via a clamp back to the 256-row default). It builds on Exp 260’s per-SQL row-count memory, adding a high-water statistic used only to reduce waste on point reads.

Changes:

• Add a high-water-mark–based initialRows signal to RowSizeMemory and plumb an initialRowHint through ReaderPool → ReadRequest → reader decode calls.
• Update the decoder to size the initial allocation via initialSlotRows(...), using main-isolate hints for reader workers and local memory only for the writer isolate.
• Expand focused harness lanes and add targeted tests covering shrink/overshoot/mispredict scenarios and precedence rules.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/result_buffer_sizing_test.dart	Adds unit + integration tests for initial-allocation sizing, high-water behavior, and hint precedence.
lib/src/reader/reader_pool.dart	Records row-size memory for all statements and stamps initialRowHint onto each request.
lib/src/reader/read_worker.dart	Adds initialRowHint to the isolate protocol and passes it into decoder entrypoints.
lib/src/query_decoder.dart	Implements initialRowsFor / initialSlotRows and applies the initial-row sizing to the decoder’s initial buffer allocation.
experiments/signals/entries/264.json	Adds the per-experiment signals/claims entry for Exp 264.
experiments/signals/base.json	Updates the result-transfer-shape narrative and priors list to include Exp 264.
experiments/JOURNAL.md	Records transferable lessons learned from Exp 264 (rate vs per-call, guard reruns, high-water vs window).
experiments/index/264.json	Adds the index fragment used to generate the experiments table.
experiments/264-initial-alloc-size-memory.md	Adds the experiment writeup documenting hypothesis, approach, and results.
benchmark/results/2026-08-06T11-40-00Z-exp264-initial-alloc-size-memory.md	Adds the focused AOT A/B benchmark receipt for Exp 264.
benchmark/experiments/select_rows_presize.dart	Adds/adjusts lanes and batching to make initial-allocation effects measurable and guarded.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.