Exp 270: answer the repeated read from memory (moonshot, rejected)

[danReynolds]

Hypothesis

Two experiments ago we learned that for a hot point read, the isolate round trip
is most of the latency — exp 265
priced it and exp 269
reproduced it at 24–90%. Exp 269 tried to collect that headroom by running the
query on the calling isolate and was rejected, because arbitrary SQLite work
cannot be bounded there; it closed by telling future runners not to retry the
same idea with a better predictor.

So this run went the other way: instead of running the query somewhere cheaper,
don't run it at all. resqlite already computes both halves of a read cache and
uses them for something else — the C authorizer records which tables each
prepared statement reads, and the preupdate hook records which tables each write
changed, and the stream engine has consumed both since exp 106. A cache hit runs
no SQLite whatsoever, so it is bounded by construction, which is exactly what
exp 269 could not achieve.

The bet, stated as the assumption being challenged: the invalidation signal
that is good enough for stream() is good enough to answer select() from
memory. Those two consumers fail differently — a stream that misses an
invalidation re-emits late, and a select() that misses one returns the wrong
rows, silently — so this was never mainly a performance question.

Approach

ReadCache sits on the main isolate beside the reader pool, keyed by
(sql, parameters). It refuses to store anything whose invalidation it cannot
prove: statements whose read-table capture is unreliable, statements that read no
table at all (nothing could ever invalidate SELECT 1), statements calling a
function outside a new C-side deterministic allowlist (a new SQLITE_FUNCTION
case in the authorizer, so random(), datetime('now') and every
caller-registered function are excluded), and results past a retention cap.

Invalidation is a version stamp rather than an index walk: each table carries a
write counter, each retained result carries the counters its tables were at when
its read was dispatched, and a lookup that finds a mismatch drops the entry. A
write therefore costs one map write per dirty table and nothing else. Two cases
retire everything at once via an epoch counter — an unknown dirty set, and the
one that is easy to miss: a write reporting an empty dirty set, because DDL and
virtual-table writes fire no preupdate hook and so arrive indistinguishable from
a no-op write.

Transaction reads never reach the pool, so they neither hit nor fill; selectBytes
is excluded because its result is a view over native memory the next query
overwrites. Full detail in
experiments/270-read-result-cache.md.

The runtime is reverted on this branch. It is preserved at archive/exp-270
for inspection, not as a base to build on.

Results

Focused AOT A/B, lane-isolated, eight alternating pairs per lane per collection,
two collections with the order flipped. Both arms come from one worktree and one
native library, differing only in a compile-time constant, so no per-.so
placement offset can exist between them.

lane	collection 1	collection 2	verdict
point1-repeat	−90.0%	−90.1%	REPRODUCED
point1-wide20	−92.1%	−91.9%	REPRODUCED
point1-params (8 cycled ids)	−90.2%	−89.6%	REPRODUCED
page20-repeat	−95.1%	−95.7%	REPRODUCED
read-write-alternate (guard)	+0.7%	+1.8%	neutral
churn-unique (guard)	+3.1%	−4.1%	drift-suspected
uncacheable-fn (guard)	−0.2%	−0.2%	neutral
mixed6-1k (guard)	−2.2%	+0.9%	neutral
concurrent8 (guard)	+2.7%	+0.6%	neutral

A hit costs 0.48 µs against 4.65 µs for the same read dispatched to a worker —
roughly ten times cheaper — and peak RSS is flat or lower. The guards are the
half that matter: nothing regresses on workloads the cache cannot help, and two
of them wrote the design. Describing a statement on its first sighting rather
than its second cost a reproduced +21% on never-repeating SQL, and a
table→queries invalidation index cost +16–19% on the read/write lane before
version stamps moved that cost off the writer entirely.

Then the probe that decides the experiment. select_cache_foreign_writer.dart
opens a second Database on the same file, commits through it, and reads back
through the first:

arm	select() after the foreign commit	a fresh connection sees
parent 1237587	after	after
candidate	before	after

That is a wrong answer, not a late one. resqlite's invalidation is built from its
own preupdate hook, so a connection it does not own commits without it hearing
anything. stream() has always had that boundary and documents it; select()
re-reads the file every time and so has never had it.

PRAGMA data_version would close the hole, and rather than assume it was too
expensive we measured it: 2 µs p50 idle, 1 µs p50 with a foreign connection
committing continuously, 10 µs worst. So a validated hit would cost ~2 µs against
4.65 µs — still ~2× faster, but bought by putting a SQLite connection on the
calling isolate, which is exp 269's rejected architecture.

Finally, incidence, which nothing in the repo had ever measured: across the two
workload simulations, Chat Sim (A5) would serve 17.7% of its 9,006 reads from
cache and Feed Paging (A6) 84.3% of its 140 — a near-5× spread, so the value is
a property of how much a workload writes to what it reads. Worth noting that the
release suite cannot answer this at all: its read scenarios repeat one statement
with nothing writing, so Select → Maps at 100 rows reads 0.040 → 0.005 ms and
that number is the benchmark measuring itself.

The headline sweep ran against a same-host baseline captured from the exact
parent commit and passed explicitly, because the automatic anchor picked exp
269's hand-authored receipt and skipped itself (exp 269's claim 269.5 warned
about precisely this). Against the parent: 7 wins, 0 wall-time regressions,
162 neutral, memory 1 win, 0 regressions. The wins are the measurement
artifact above, not a result.

Flagged lanes and their disposition. One row is red — Streaming (Column Granularity) / Overlapping column writes, −592 re-emits. It is sqlite_async's
row, and it moved 3,466 / 3,894 / 4,347 / 3,663 / 4,185 across the five repeats
inside that one run; resqlite reports 0 disjoint and 10 overlapping in all
five. Peer non-determinism, not attributable here. Separately, a first sweep of
this same commit against this same baseline flagged fourteen lanes at +12% to
+40% (writes, streaming fan-out, subscription rate). No mechanism connects those
to a diff that adds one call to the write path and one map lookup to the read
path, and the cause was the host: a separate multi-Database probe was running
concurrently. The quiet repeat above is the receipt, and both runs are committed.

Outcome

Rejected. Not on performance — the mechanism is a clean 90–96% with no
measurable tax anywhere — but on three things the numbers don't cover:

1. the contract change is silent, and making it opt-in means new public API for a
   trade-off-shaped win;
2. making it honest costs most of the win (~2 µs instead of ~0.5 µs) and buys it
   back with the main-isolate SQLite connection exp 269 rejected;
3. the value is workload-shaped, and the write-mixed workload is the modest one.

Would reopen if the invalidation signal grows cross-connection scope, or if a real
downstream trace shows read repetition much closer to Feed Paging's number than
Chat Sim's. The successor actually worth building is much smaller and is named in
the writeup: serve select() from an active stream's own last result, where the
caller has already accepted stream semantics for that query.

The lasting artifacts are the focused A/B harness, the foreign-writer probe (which
fails on the cached arm and passes on main), the incidence probe, and the
data_version price.

Test plan

• dart analyze --fatal-infos — clean
• dart test — 495 tests with the prototype in place, including 22 new
  test/read_cache_test.dart cases; three load-bearing guards verified to
  fail against a deliberately weakened cache
• focused AOT A/B, nine lanes, two order-flipped collections;
  benchmark/ab_drift_check.dart over all nine
• select_cache_foreign_writer.dart on both arms, and on the parent worktree
• select_cache_data_version.dart (AOT), select_read_cache_incidence.dart
• headline release sweep against an explicit same-host parent baseline
• dart test and dart analyze re-run after the runtime revert
• benchmark/finalize_experiment.dart green

[github-actions]

Belief impact

Learned

• 270.1 · Answer the repeated read from memory
  A select() result served from a main-isolate cache costs 0.48 us against 4.65 us for the same read dispatched to a reader worker. Focused AOT A/B o…
• 270.2 · Answer the repeated read from memory
  resqlite's write invalidation is scoped to writes made through one Database, so a result cache behind select() returns superseded rows whenever a…
• 270.3 · Answer the repeated read from memory
  PRAGMA data_version — SQLite's cross-connection change counter — costs 2 us at p50 and 5 us at max when idle, and 1 us at p50 with a 10 us max whil…
• 270.4 · Answer the repeated read from memory
  An invalidation scheme's cost belongs on the reader, not the writer. A table-to-queries index with column-level intersection measured +16-19% on a la…
• 270.5 · Answer the repeated read from memory
  Read repetition is workload-shaped, not constant. Chat Sim (A5) would serve 17.7% of its 9,006 select() calls from a (sql, parameters) cache and Fe…
• 270.6 · Answer the repeated read from memory
  The release suite's read scenarios cannot evaluate a statement-keyed cache, and the explicit parent comparison shows it directly: 7 wins, 0 wall-time…

What this changed

We believed resqlite's write-invalidation signal was a general-purpose fact about the database, used so far only by streams. It is not: it is a fact about writes made through one Database object. Streams can live with that because a missed invalidation only delays a re-emit, and stream() documents the limitation. select() cannot, because it re-reads the file today and therefore observes every committed write, whoever made it. Anyone reading the stream-engine chapters or the invalidation sections of the architecture doc should read 'invalidation' as scoped to this Database's own writes, not to the file.

The round-trip headroom exp 265 priced and exp 269 confirmed is collectable without running SQLite on the caller. A cache hit measured 0.48 us against a 4.65 us dispatched point read, reproduced at -90% to -96% across four lanes and two order-flipped collections, with every adversarial guard neutral and peak RSS flat or lower. The blocker is not performance and not the bounding problem exp 269 hit; it is that the invalidation signal has the wrong scope.

The cost of a cache is paid on the write path, and that is where an invalidation design should be judged first. A table-to-queries index with column-level elision — the shape the stream engine uses — measured +16-19% on a read/write-alternating lane, because a write pays it whether or not anything is cached. Replacing it with per-table version counters checked at lookup moved the whole cost onto the reader that benefits and took the same lane to neutral.

The release suite cannot evaluate a read cache. Its read scenarios execute one statement thousands of times with nothing writing, so Select -> Maps at 100 rows reports 0.040 ms -> 0.005 ms and the number is the benchmark measuring itself. This is exp 267's observation (every benchmark in the repo uses under ten distinct SQL strings) on a new axis, and it generalises: any candidate keyed on statement identity needs a workload that mixes reads with writes to what they read.

A first incidence number now exists for read repetition. Across the repo's two application-shaped workloads, Chat Sim (A5) would serve 17.7% of 9,006 select() calls from cache and Feed Paging (A6) 84.3% of 140 — a near-5x spread between two workloads in the same repository, so the value of remembering a read is a property of how much a workload writes to what it reads, not a constant. Neither is a production trace.