All notable changes to sqlite-rs. Format follows Keep a Changelog, versioning follows SemVer. Pre-1.0: minor bumps may break the public API.
Versioning policy: one minor version per completed plan phase — the version number tells the plan's story, sub-steps stay inside a phase. V1 (READ CORE) = 0.1.0 through 0.4.0. (History note: internal iterations briefly numbered 0.4.0–0.6.0 were renumbered into the phase scheme on 14 Aug 2026, before any tag or publication of those versions existed.)
hash_agg_findallocated a freshVec<u8>key buffer andVec<Value>key-values buffer per row. It now reusesHashAggState-held scratch buffers via take/give-back, cloning intoGroupSlot/the indexHashMaponly when a row starts a new group.hash_agg_stepsimilarly reuses a scratchVec<Value>for per-row aggregate arguments instead of allocating fresh each row.read_group_by_aggimproves from ~2.5x oracle to ~2.26x oracle onbench_1mb.db(#674).
-
IndexCursor::seek()scanned linearly from the first entry, fully decoding every candidate cell until finding one>= target: O(n) per seek. It now does a real O(log n) tree descent, binary-searching each level's cell array and falling back to the nearest ancestor's qualifying cell when a descended-into subtree has nothing to offer — mirroringTableCursor::seek's binary search, which never got applied to the index-cursor side. Measured ~13.8% faster on an indexed range scan and ~7% faster on an indexed join (#661). -
SorterInsertalways re-decoded its sort-key columns out of the record blob it had just been handed, even though those same values were still sitting in registers moments earlier, beforeMakeRecordencoded them. It now gains an optional source-register run (honored whenp5is nonzero), whichcompile_grouped_scan'sGROUP BYpath opts into; every otherSorterInsertemitter keeps the original decode-from-blob behavior. Measured ~3.5-4% faster ongroup_by_agg(#660). -
INSERT and UPDATE codegen re-
SeekRowid'd back onto the row they had just written, then re-read every index column viaOpcode::Column/Opcode::Rowidto buildIdxInsertkeys — even though those same values were still sitting incol_regs/rowid_regfrom just before the write.emit_index_key_ops_from_regsbuilds the key viaOpcode::Copyfrom those registers instead, dropping the seek and re-read on every indexed INSERT/UPDATE.IdxDeletepaths (removing a different, already-on-disk row's stale entries) are unaffected — they have no such register run to reuse (#663). -
UPDATE ... WHERE col >/>=/</<= lit/BETWEENagainst a leading-indexed column fell back to a fullRewind/Nextscan with a per-rowcompile_condfilter — unlikeSELECT's equivalent fast path, it never used the index at all.try_compile_range_row_seek(aSELECT-agnostic variant of the existing range-seek builders) is now wired intoUPDATEcodegen: a read-onlyIdxNextwalk records matched rowids into an in-memory ephemeral table, then a second pass replays them against the table cursor to do the actual update (the index cursor doing the range walk has no save/restore protection against the same scan's own index-maintenance writes, unlikeTableCursor's snapshotted frames, so the update can't happen inline during the walk). Measured ~28% faster onupdate_filtered_range(#666).DELETE's equivalent fast path was prototyped but not shipped — for a highly selective predicate on a small table, the same two-pass materialization regressed rather than helped, since it needs an unavoidable random-order re-seek per matched row; blocked on cursor save/restore, a separate follow-up.
Tokenizer::newcopied the whole source string on every parse call (src: src.to_string()) even though the tokenizer only ever reads through byte-range slices. Neither the obvious fix (Tokenizer<'a> { src: &'a str }) nor the fallback (Cow<'a, str>) is viable — both tripmake check-mvl-limit, sincesrc/parser/tokenizer.rsis in the qualified subset that bans lifetimes beyond function-scoped elision. InsteadTokenizerno longer stores the source at all: every scan method takessrc: &stras a parameter, keeping every lifetime function-scoped while eliminating the per-parse copy (#644).
-
WalWriter::append_frameissued its ownwrite_atsyscall per dirty page instead of batching a transaction's frames into a single write — an O(n) syscall pattern in the commit hot path. Frames now accumulate in a pending buffer andsync()issues onewrite_atcovering the whole run, still fsyncing exactly once per commit (ADR-0026's per-commit rescan behavior unchanged — only the writes feeding that fsync are batched). Filed as follow-ups rather than chased further here: #639 (VDBE table-scan interpretation overhead) and #640 (ADR-0026's flagged per-commit WAL rescan cost) (#635). -
Implicit-whole-table-group aggregates (
count(*)/sum/etc. with aWHEREclause, noGROUP BY) routed throughcompile_grouped_scan, which unconditionally opened aSortereven though there is noGROUP BYkey to sort by — everyWHERE-matching row paid aMakeRecord/SorterInsert/SorterSort/SorterDataround trip for nothing. Addstry_compile_direct_agg_scan, a new fast-path tier that foldsAggStepinline in a singleRewind/Nextscan instead, reusingflush_group/AggSlot/compile_limit_setupsoHAVING,LIMIT/OFFSET, and #287's zero-row-still-flushes-one-row behavior are unchanged. Declines only when an aggregate call usesDISTINCT(#633). -
compile_scalar_subqueryrouted every aggregate scalar subquery ((SELECT avg(x)/sum(x)/count(*) FROM t ...)) straight intocompile_grouped_scan, bypassing thetry_compile_index_only_count/try_compile_index_only_sumfast-path dispatch top-level aggregate queries already get inentry.rs. Such a subquery now compiles to the same index-onlyIdxRewind/IdxNext/Countbytecode as the identical standalone query, when a qualifying index exists and the subquery has noWHEREclause, instead of a full table scan (#634). -
Fixed a stale spec scenario/dead test link in 009/Req-17 (still described #570's pre-#631 hash-aggregation dispatch, pointed at a renamed test function), caught by
make assurancewhile working on #634. -
group_by_aggran 4.12x slower than the pinned oracle despite sqlite-rs's HashAgg strategy (#570) being algorithmically better than a sort-then-group approach — profiling found the real cost was decode-side overhead (a freshVecallocation per row for the sort key, decoding columns past the key, pseudo cursors re-parsing a row's record header on everyColumnread, andMakeRecord's encode path allocating a fresh serial-type buffer per row), not the opcode family. With those fixed, the Sorter strategy now beats HashAgg outright, so GROUP BY dispatch switches back to it (#631).group_by_agg: 6.38ms (HashAgg) -> 5.04ms (Sorter, fixed) on the 1MB fixture, vs the oracle's 1.52ms (was 4.12x, now ~3.3x). A companion spike (tests/spike/013_raw_pointer_comparator/) confirmed sqlite3's raw-pointer sort-key comparator trick (vdbesort.c) is not the reason for the remaining gap — a safe specialized decode path is 7.5-24.6x faster than the general one, whileunsaferaw pointers buy only ~2% on top of that.
compare()'s REAL ordering reported NaN as "equal" to every other value (partial_cmp().unwrap_or(Ordering::Equal)), breaking the total-order contract's transitivity (spec 008 Req 2) whenever a NaN sat between two ordinary reals. Also fixedcompare_int_real's NaN handling, which disagreed with the REAL-vs-REAL fix's convention (integer-vs-NaN and real-vs-NaN now both treat NaN as the numeric class's maximum). Found bytests/fuzz/fuzz_targets/semantics_compare.rsonce it was actually wired into CI (make fuzz-smoke) — both crashing inputs are now committed seeds undertests/fuzz/seeds/semantics_compare/.tests/fuzz/fuzz_targets/scalar_functions.rsandsemantics_compare.rsdidn't compile — both predated theValue::Text/Value::BlobAPI change fromString/Vec<u8>toRc<str>/Rc<[u8]>, so neither fuzz target had run since. Found while wiringmake fuzz-smokeinto CI.
- Constant propagation and OR-to-IN conversion for WHERE-clause index
eligibility (spec 012-query-constraints Req 1-2, #605):
src/codegen/select/limit_scan.rs::propagate_constantsletsa = b AND b = 5(direct or a multi-hop chain) drive the rowid-seek, covering-index, and skip-scan fast paths using the propagated literal, exactly as if the query had writtena = 5directly;::or_chain_equality_operandsletsx = 1 OR x = 2 OR x = 3(equalities against the same column) probe once per value via the existingSeekRowid/SeekIndexEqopcodes, instead of falling back to a full table scan — no new VDBE opcode needed, since a chain of pure equalities is just a finite list of point seeks (see ADR-0033). #606's LIKE/BETWEEN/IN range-seek work is out of scope here (needs a genuine new range-seek opcode) and is tracked separately as spec 012 Requirement 3. - Fuzzing gap-closing pass:
make fuzz-smokeruns a short crash-only pass of everytests/fuzz/fuzz_targets/*.rstarget and is now a blocking CI job, catching a crash before merge instead of only when someone remembers to runmake fuzz-*by hand. Addedmake fuzz-scalar-functions/fuzz-vdbe-exec/fuzz-semantics-compareMakefile targets for the 3 fuzz binaries that previously had none. Every target now also seeds from a committedtests/fuzz/seeds/<target>/directory (see its README) instead of starting from nothing each run —tests/fuzz/corpus/(the libFuzzer-grown corpus) stays gitignored as before. - Extended MC/DC scan scope (spec 005-assurance) to
src/vdbe/program.rsandsrc/vdbe/control.rs— the opcode-dispatch layer every query executes through, previously the highest-leverage gap since onlyvdbe/exec.rswas covered, not the surrounding dispatch/control machinery.tools/mcdc_report.pynow also prints a finalSUMMARY: PASS/FAILline and returns a non-zero exit code when any multi-leaf obligation is undischarged, somake test-mcdcfails a build instead of silently reporting a gap. All 43/43 multi-leaf obligations in the scanned file set are now discharged, including the two (btree_1081,encode_68) left over from the prior refresh.
- Widened the vendored TCL and sqllogictest corpora (#70) past their
original V2/V3 single-table scope up through V7, per
.openspec/plan.md's own per-block corpus citations: sqllogictest 14 → 17 files (addsselect3-5.test, joins/subqueries/aggregates); TCL 43 → 60 files (adds V4join2/3.test,subquery.test,subquery2.test,select6-8.test,aggnested.test; V6/V7with3-6.testnon-recursive through recursive CTEs; V7savepoint.test,savepoint2.test,pragma.test,pragma2.test,analyze.test).tests/sqllogictest/runner_test.rs'sEXPECTED_FILE_COUNTand doc comments updated;tools/sqllogictest-status.jsonregenerated (34.0% corpus coverage, up from 56.8% of a much smaller denominator — pass rate stays 100% of what's attempted). One coincidental ratchet update:SELECT_INVALID_BASELINE3 → 2 intests/corpus/extracted_sql_test.rs(a known misclassified statement dropped out of the resampled corpus under the per-shape cap, not a parser fix — noted inline so a future re-widening that resurfaces it isn't mistaken for a regression). V8+ files (fkey*,trigger[1,3-9],window*,gencol*,without_rowid*,strict*) deliberately excluded — those features aren't implemented yet. make test-tcl/make test-sqllogictestnow print the corpus's file/ statement/coverage numbers directly (via--nocapture, now the default for these two targets) instead of requiring a separatemake extract-sql-corpus/cat tools/sqllogictest-status.json/make assurancestep to see them.
- DROP TABLE/DROP INDEX (
free_btree_pages_inner) leaked overflow-page chains hanging off individual cells — only the tree's own leaf/interior pages were freed. Table leaf, index leaf, and index interior cells now have their first overflow page located and their whole chain walked and deallocated before the tree structure itself is freed. - Index cells (leaf and interior) were reading their local-payload size
with the table leaf cell's
max_localformula (usable_size - 35) instead of the smaller one SQLite defines for index cells ((usable_size - 12) * 64 / 255 - 23), corrupting reads on any index cell whose payload landed between the two thresholds. Found while closing 006-btree Req 7's documented "no overflowing-index-key fixture" coverage gap — adding one immediately hitPayloadTooShort. Fixing the threshold also surfaced that index entry delete never freed a removed entry's overflow chain at all (table delete already did); index entries essentially never overflowed under the old, too-generous threshold, so the gap was never exercised. tests/fuzz/fuzz_targets/btree_cursor.rs'sFuzzPageSourceimplemented the pre-Rc<[u8]>PageSource::read_pagesignature, breakingmake fuzz-btree. Updated to returnRc<[u8]>.raw_mode_enable_returns_none_without_ttyhard-assertedRawMode::enable()always returnsNone, on the assumption thatcargo testnever runs with a controlling tty attached. True for a piped/CI invocation, false the moment the test binary runs interactively in a real terminal session (e.g.make coveragerun by hand) —enable()correctly returnedSome(_)there, and the test failed on correct behavior. Renamedraw_mode_enable_matches_actual_tty_state; now asserts againsttermios::is_ttyinstead of a hardcoded assumption.
- 006-btree Req 4 (rowid-alias columns) read as unimplemented ("covered
functionally once #34 lands"), but #34's DDL reader
(
rowid_alias_from_sql/TableSchema::rowid_alias) already landed and substitution is already wired throughcodegen/select/projection.rs,codegen/stmt/insert.rs, andcodegen/stmt/update.rs— the note was stale. Added the one missing piece, a btree-layer unit test proving this module itself decodes the alias column asValue::Null, and split the requirement into its two scenarios, both now test-linked. - Audited all 11 specs for drift between prose and implementation and fixed
every confirmed instance; specs 004, 007 and 008 were already clean. No
requirement semantics changed — citations, type definitions and
descriptive prose only.
- 005 claimed crate-wide
#![forbid(unsafe_code)]and that there is nounsafefor miri to check. Both false:src/lib.rsis#![deny(unsafe_code)]andsrc/sys/{fcntl,termios}.rscarry audited#![allow(unsafe_code)]carve-outs (ADR-0031, #592) thatforbidwould make impossible. - 009 carried a stale opcode inventory in four places (61/60/58-of-60 and "65 opcodes"); actual is 68 harvested, all 68 dispatched, so the "two undispatched opcodes" clause described a gap that had closed.
- 001 and 002 documented types that never existed —
Connection,Statement,SelectCompiler,Interpreter,Mem,BTree,MemVfs,WindowsVfs, plus 002's entire Tokenizer and AST code blocks (Token.value,Span.start/end, per-keywordTokenKindvariants, aStmt/SelectStmt/SelectBody/SelectCore/Expr-as-enum tree). Rewritten againstsrc/parser/{tokenizer,ast}.rswith notes on the non-obvious shape decisions. - 002's alternatives table recommended lemon-rs against the accepted
pomelo decision; Requirement 6 retitled to "Generator Swap" and
repointed off
src/parser/parse.y, a file that will not exist. - 003 and 006 still described themselves as read-only after their
write paths landed; 005 still described landed gates (panic-surface
lints,
deny.toml,--lockedCI, SLT runner, crash torture, six fuzz targets) as future work. - Stale citations fixed in 010/011 and elsewhere:
src/codegen/insert.rs→src/codegen/stmt/insert.rs,emit_result_row→projection::emit_row_via_sink,emit_dedup_guard→emit_dedup_check, a nonexistentviews::MAX_DEPTH→ the view-name stack raisingCodegenError::CircularView,src/vdbe/collation.rs→src/record/collation.rs, and threepath:linecitations in 011.
- 005 claimed crate-wide
- Regenerated the MC/DC obligations snapshot (
tests/mcdc/obligations.json) and renamed everymcdc__<id>__vNtagged test (plus doc-comment cross-references) to the obligation id its decision now resolves to, acrosssrc/btree.rs,src/btree/index.rs,src/btree/table/delete.rs,src/parser/grammar.rs,src/parser/tokenizer.rs,src/record/encode.rs,src/vdbe/exec.rs,src/vdbe/functions.rs— the ids had drifted from source line numbers, silently reducing real MC/DC discharge on the scanned file set to near zero. Now correctly reports 40/42 real MC/DC obligations discharged;btree_966andencode_68remain undischarged. - Added a license-header gate (
tools/license_headers.py,make check-license-headers) checking every tracked.rsfile (vendoredthird_party/exempt) for theCopyright 2026 Schuberg Philis/SPDX-License-Identifier: Apache-2.0header pair; backfilled it on the 7 files that were missing it. Dropped the unused"MIT"entry fromdeny.toml's license allow list (cargo denywas warninglicense-not-encountered— nothing in the resolved graph needs it). - Renamed every pass/fail Makefile gate to a consistent
check-*prefix (deny→check-deny,audit→check-audit,grammar-drift→check-grammar-drift,mvl-limit→check-mvl-limit,mod-files→check-mod-files,coverage-gate→check-coverage,assurance-gate→check-assurance), replacing a previous mix of bare names and an inconsistent-gatesuffix. Updated every caller (CI workflow, Makefile dependency chains, doc references). - Removed the Bloom-filter join-probe path (#623, ADR-0033):
choose_bloom_probe/BloomProbe,Opcode::Filter/FilterAdd, andsrc/vdbe/filter.rsimplemented #464's join-level Bloom pre-check, but #545's automatic-index probe shares the identical row threshold and gating conditions and is always tried first, so the Bloom path has compiled into zero real programs since #545 landed. Spec 011 moves from 6 to 5 requirements. - Backfilled
src/vdbe/cursor.rs's andsrc/vdbe/program.rs's highest- leverage coverage gaps (OpenWrite'sIndexWritebranch, severalCursorTypeMismatch/MalformedInstructionarms,IdxRowid's non-integer-trailing-column error,seek_index_eq's prefix-mismatch branch,Program::is_empty):vdbe/cursor.rsregions 88.58% → 89.35%, functions 68.88% → 71.49%, lines 85.30% → 86.51%.
- Raised line coverage to 85%+ across 21 of the 22 files that were below
the repo's threshold (#603): test-only coverage added for error
Display/From-conversion paths, subquery flatten/pushdown expression-rewrite branches, join ordering, integrity-check branches, parser error paths, pager checkpoint, VFS edge cases, and readline dispatch/redraw/terminal logic. TOTAL line coverage 89.22% → 92.88%.src/bin/sqlite-rs/readline/term.rsstays below threshold (66%, up from 16.67%) — its tty-only branches (RawMode::enable's success path,Drop,read_byte) require a real controlling tty thatcargo testnever has, and are flagged rather than faked with a fragile pty harness.
-
Tier 2 query-pipeline performance (#590), no pipeline-structure or output changes — every fix is intra-component and the corpus/parity suites stay bit-exact. The tokenizer walks a byte cursor over its source instead of materializing a
Vec<(usize, char)>(~16× the source size) up front, and identifier/number/parameter/string/quoted- identifier scanners now slice the source directly rather than rebuilding each token char-by-char — a string or quoted identifier only allocates once a''/""escape actually appears in it.lookup_wordbinary-searches the keyword table with an ASCII case-insensitive comparator instead of heap-allocating an uppercased copy of every identifier token, andtokenizepre-sizes its outputVec.TokenKind's rareBlob/Paramvariants are boxed, shrinking the enum from ~40 to 32 bytes (guarded bytest_token_kind_size) and roughly halving token memory traffic. The parser gainedadvance_span(), so the ~16 call sites that only need a token's span no longer deep-clone its payload to immediately discard it. CTE and view expansion returnCow<Select>rather than unconditionally deep-cloning the entireSelectAST: a SELECT with noWITHclause and no view in scope now compiles with zero whole-AST clones, down from two. Statement dispatch andCOLLATEresolution compare witheq_ignore_ascii_caseinstead of allocating uppercased copies.Measured by the new
compile_pathbench (below), criterion baseline against this branch's parent commit, allp = 0.00:Benchmark Before After Change tokenize/short723 ns 344 ns −52.6% tokenize/long3.26 µs 1.62 µs −50.3% tokenize/literals1.19 µs 759 ns −35.8% parse/short1.25 µs 877 ns −30.0% parse/long6.14 µs 3.41 µs −39.1% parse/literals2.80 µs 2.00 µs −31.4% expand_with_clause/no_cte161 ns 1.46 ns −99.1% expand_with_clause/with_cte769 ns 691 ns −6.3% compile_full/short3.89 µs 2.39 µs −39.3% compile_full/no_cte4.82 µs 3.91 µs −18.6% The
no_cteexpansion arm is the copy-on-write change in isolation: with nothing to rewrite it is now a borrow rather than a whole-AST clone. Thewith_ctearm still has to produce an owned, substituted AST, and stays essentially flat — as intended.
tests/performance/compile_path.rs(make bench-compile-path, #590): a fixture-free, oracle-free criterion bench for the Tier 2 compile path — tokenize, parse,WITH-expansion, and full text→Programcompilation.engine.rstimes query execution, where compilation is a rounding error next to B-tree/IO work, so compile-path changes are invisible there. Numbers are a relative signal between revisions, not a parity claim: stock sqlite3 exposes no comparable "compile but don't run" entry point to form an oracle arm against.
- Tier 3 (VDBE/CLI) execution fixes (#591): the #465
row_scratchResultRowbuffer was inert (mem::takehad no return path, so every row already allocated fresh) — removed the dead field. Index scans (idx_rowid/IdxRewind/etc.) decode only the trailing rowid column via the existing header cache instead of fully decoding every index record;SeekIndexEqdecodes only the compared key prefix viadecode_record_uptoplus a header-only column count. Seek/insert collations are borrowed out of the instruction'sP4instead of cloned per call, andAutoIndexNext/AutoIndexRowidno longer clone their encoded keyVecper call. CLI-list/-csv/column/linerendering reuse one line buffer across rows instead of allocating (plus ajoin) per row.#[inline]added to the hot comparison and register-accessor functions.
- Crate-wide
unsafe_codedeny lint added at the Cargo.toml level as a backstop for ADR-0031's single-src/sys/-boundary policy (#592): the underlying drift (unsafe stdin fd construction in the bin crate, unlinted and outside the boundary) was already closed by #587; this makes the deny explicit for lib and bin targets uniformly.
- Tier 0 storage hot paths shed their dominant copies (#588): WAL replay
(
wal::committed_pages) uses an undo log instead of cloning the whole page map at every commit frame; committed WAL overlay pages are shared asRc<[u8]>so WAL-mode reads no longer memcpy a page per read; b-tree insert/delete pre-scan the borrowed page zero-copy (scan_leaf_cells/find_leaf_cell) instead of cloning the page and every cell before the splice fast path; plus a single-lookup page-cache hit, preallocated overflow reassembly, a reused WAL frame buffer, and page-ordered checkpoint backfill.
- The CLI's two raw
BorrowedFd::borrow_raw(0)blocks (src/bin/sqlite-rs/readline/term.rs) moved behind a safe,// SAFETY:-documentedsys::termios::stdin_fd()wrapper, and the binary crate root now carries#![deny(unsafe_code)]— restoring ADR-0031's "unsafe lives only insrc/sys/" invariant and lint-enforcing it for bin targets (#587).
- Added a
// Copyright 2026 Schuberg Philis/// SPDX-License-Identifier: Apache-2.0header to everysrc/**/*.rsandtests/**/*.rsfile (#582), matching the license already declared inLICENSE/Cargo.toml. Excludes the one genuinely vendored file,tests/spike/001_parser/001_lemon-rs/third_party/lemon/lempar.rs.
-
Tier 1 schema performance (#589):
TableSchemanow carries arowid_alias: Option<usize>field resolved once at schema-decode time (rowid_alias_from_sql), so codegen reads a field instead of re-parsing the full CREATE TABLE DDL on every column/expression reference. The ddl_reader hot helpers (column_type,is_table_constraint,column_collation,indexed_column, keyword prefix checks) compare witheq_ignore_ascii_caseinstead of allocating uppercased copies; index→table attachment inread_schemais a hash lookup instead of a linear scan. Newread_schema_and_viewswalkssqlite_masteronce for both catalogs, and the CLI'sexecloop caches the decoded catalog across statements, invalidating only afterCREATE/DROP/ALTER— pure-DML/SELECT scripts no longer re-read the schema per statement. -
Correlated
EXISTS/NOT EXISTSsubqueries (compile_exists,src/codegen/subquery/scalar.rs) now reuse #434'schoose_join_accessseek detection: aWHEREclause that's a single correlated equality against a rowid or unique index compiles to aSeekRowid/SeekIndexEqpoint lookup instead of an unconditionalRewind/Nextscan (#580). The scan's existing jump-to-true-on-first-match behavior is unchanged for non-seekableWHEREclauses. -
encode_record_into(src/record/encode.rs) no longer allocates aVec<u8>per column plus a serial-type-bytes buffer per call: serial types/body lengths are computed up front without allocating, then the header and column bodies are written directly into the caller's reused output buffer (#572). Profiling (spike #449) found this per-row record re-encoding was the dominant cost inGROUP BY's sort-based path, not the sort algorithm itself —group_by_aggimproves ~2.2-2.6x (12.1x/9.5x slower than sqlite3 -> 5.4x/3.8x on the 1MB/50MB fixtures).
- Partial-sort optimization for
ORDER BY <indexed prefix cols>, <suffix cols>(#574): when an index satisfies a strict prefix of the requested order but not all of it, the compiled program now walks that index directly and only sorts the unsatisfied suffix within each prefix-group, instead ofcompile_sorted_scan's single sort over the entire result set (src/codegen/select/index_scan.rs::try_compile_partial_sorted_index_scan). Closes the last open technique in epic #548. .color on|offdot-command for the REPL: toggles ANSI syntax highlighting of the in-progress input line (src/bin/sqlite-rs/readline.rs'sReadline::set_color), alongside the existing.headers/.modetoggles. Query output is unaffected.
- Vendored the
nixcrate'sfcntl/termiosFFI intosrc/sys/(#563): hand-writtenunsafe extern "C"bindings and per-platform (macOS/Linux) ABI structs for POSIX byte-range file locking (src/vfs/lock.rs's cross-process database locking) and raw-mode terminal control (the CLI readline, #558).nixis removed fromCargo.toml— sqlite-rs now has zero external dependencies.src/lib.rs's# is narrowed to#![deny(unsafe_code)]with a single scoped#![allow(unsafe_code)]insrc/sys/; every other module stays unsafe-free. See ADR-0031.
-
Hash-based
GROUP BYaggregation (#570): a secondGROUP BYexecution strategy alongside the existing sort-then-group one. Each row is folded into its group's accumulators as the scan reaches it — O(n) — instead of buffering and sorting every row (O(n log n)) purely to make a group's rows adjacent; only the K groups are ever ordered, which keeps output row order identical to before. NewHashAggOpen/HashAggFind/HashAggStep/HashAggRewind/HashAggData/HashAggNextopcodes plus aP4::GroupKeydescriptor (src/vdbe/hash_agg.rs), shaped after theSorter*family; the per-group flush (HAVING/LIMIT/projection) and the aggregate registry itself are shared verbatim with the sort strategy, so an aggregate cannot mean one thing under each. Selected when no covering index already produces group-ordered rows; aDISTINCTaggregate (or no explicitGROUP BYkey) still falls back to the sorter, which remains the always-correct general path. Spec 009 Requirement 17. -
Automatic index for unindexed equality join columns (#545): when a join level's
ONcondition is a single equality against a column with no usable index, andANALYZEstats judge the table big enough to be worth it, a transient in-memory index is now built over that column once and probed per outer row, instead of a plain nested-loopRewind/Nextscan (sqlite.org/optoverview.html#autoindex). NewAutoIndexInsert/AutoIndexSeek/AutoIndexRowid/AutoIndexNextVDBE opcodes back an exact-key rowid multi-map (opened viaOpenEphemeralwithP5 == 2), distinct from the existing DISTINCT/aggregate dedup guards' single-value-per-keyFound/IdxInsertand from a real index's byte-orderedSeekIndexEq+IdxNextwalk — a join only ever needs "every row sharing this exact key", never a range. Supersedes the existing Bloom-filter join pre-pass (#464) whenever both are eligible for the same level. -
Hash join for equi-joins (#547): #545's automatic-index multi-map now backs onto a
HashMapinstead of aBTreeMap— true O(1) amortized build/probe (nothing ever walked it in key order).EXPLAIN QUERY PLANalso now reports this path (SEARCH t USING AUTOMATIC COVERING INDEX (col=?), matching real sqlite3's own wording) instead of silently falling through to a plainSCANin the human-readable plan while the compiled program used the index at runtime. -
Readline-style line editing and persistent history for the REPL (#551), hand-rolled from scratch rather than depending on
rustyline(#558, superseding #551's originalrustyline-based approach before release):replnow reads input through a zero-dependency-beyond-nixline editor (src/bin/sqlite-rs/readline/) giving up/down arrow history navigation, Ctrl-C-abandons-the-current-line behavior, Ctrl-A/E/K/U, SQL-aware tab completion (keywords, dot-commands, live table/column names from the open database's schema), and tokenizer-backed syntax highlighting (keywords, strings, numbers, comments). Each submitted line is appended to$XDG_STATE_HOME/sqlite-rs/history(falling back to~/.sqlite-rs_historywhen$XDG_STATE_HOMEis unset) and reloaded on the next session; loading/saving is best-effort — a missing$HOME/$XDG_STATE_HOMEor an unwritable history file never blocks the session. Piped/non-tty stdin (used by every existing REPL test) is unaffected, falling back to a plain buffered line read there. -
EXPLAIN QUERY PLANon aUNION/UNION ALLcompound (#539): only the left-most arm's plan was reported before. Now every arm gets its own nested plan under aCOMPOUND QUERYroot (COMPOUND QUERY->LEFT-MOST SUBQUERYplus oneUNION/UNION ALLchild per arm), matching the oracle's own EQP shape — plainUNION's child text calls out its ephemeral-index dedup step (UNION USING TEMP B-TREE),UNION ALLdoesn't need one. -
Bare
EXPLAIN <stmt>(#538): the parser accepted onlyEXPLAIN QUERY PLAN, rejecting plainEXPLAINeven though the opcode/bytecode- listing renderer it should produce (spec 009 Requirement 10) already existed and was only reachable via the CLI's-explainflag.parse_explain_stmtnow accepts both forms identically; thequerybinary renders the bytecode listing whenever the SQL text itself said bareEXPLAIN, regardless of the-explainflag. -
Predicate push-down into
FROM-subqueries and views (#532): a safely-movable outerWHEREconjunct is now moved into a view's or derived table's ownWHEREclause right afterexpand_with_clause/expand_viewsrewrite it into aTableRefKind::Subquery, before it materializes — letting the inner scan use an index on the underlying table instead of always filtering after a full scan. Newsrc/codegen/subquery/pushdown.rspass, gated to provably-safe cases: the target subquery is single-table (noJOINof its own), has noDISTINCT/aggregate/GROUP BY/HAVING/LIMIT/UNION, and every column the conjunct touches is identity-mapped (SELECT *, or a plain, optionally-aliased column list) — anything else stays outer. Recurses through nested views/CTEs.EXPLAIN QUERY PLANnow also recurses into a materializedFROM-subquery's own plan, nesting its rows under the outerSCAN (subquery)row so a pushed-down index search is visible in the report (src/codegen/select/eqp.rs). -
Covering-index scans now treat a table's
INTEGER PRIMARY KEYrowid-alias column as free from any index leaf (#535, found while working on #532):find_covering_index/try_compile_covering_index_scan(src/codegen/select/limit_scan.rs) previously required every projected column to be a declared index column, soSELECT * FROM t WHERE x = 5on at(id INTEGER PRIMARY KEY, x, ...)table with an index onxfell back to a full scan even thoughidneeds no separate table lookup.bare_result_column_namesalso now expands a bareSELECT */table.*against the scan's own schema, instead of bailing the whole covering-index path out for any*projection. -
REPL dot-commands for
sqlite3shell parity (#495):.help,.version,.schema [TABLE],.dump [TABLE],.headers on|off,.mode csv|column|line|list,.databases,.indices [TABLE]— allsqlite3-style prefix-matched, alongside the pre-existing.tables/.quit/.exit(#478)..schema's output byte-matches the pinned oracle. NewOutputModeenum (src/bin/sqlite-rs/mode.rs) plus a.headersflag give the REPL its own result-set renderer (list/csv/column/line), reused for everySELECTthe REPL runs —query/exec's own one-shot output is unaffected..mode column's width-per-column sizing is a documented approximation of stocksqlite3's own heuristic, not a byte-exact match.src/codegen/select/order_by.rs::output_column_namesis nowpub(waspub(super)) so the REPL can derive.headers oncolumn labels for a single-tableSELECTthe same way the compound-SELECTcodegen already does for its ownORDER BYresolution. -
9 read-only introspection
PRAGMAs (#489):table_info,table_list,index_list,index_info,database_list,schema_version,user_version,page_size,page_count. Recognized by a hand-rolled parser (src/bin/sqlite-rs/pragma_query.rs) deliberately outside the main grammar/AST/codegen/VDBE pipeline — these are synthetic in-memory result sets built directly from already-loaded schema/header data (theEXPLAIN QUERY PLANprecedent, not thejournal_modewrite-pragma path), so they never touch aPagertransaction or compile to bytecode. Wired into both thequerysubcommand andrepl; aPRAGMAoutside these 9 names (e.g.journal_mode) falls through to existing behavior unchanged.schema::column_defs/column_type(src/schema/ddl_reader.rs) are nowpub(werepub(crate)) so the CLI-layer pragma module can reuse them instead of re-deriving column-definition splitting. Scope cuts:index_list/index_infoonly report explicitCREATE INDEXentries (originis alwaysc) — auto-indexes for inlinePRIMARY KEY/UNIQUEconstraints are already dropped byschema::read_schemaand not re-derived here;table_listomits the internalsqlite_schema/sqlite_temp_schemarows stocksqlite3lists (no temp-db support). Tests:tests/unit/introspection_pragmas.rs. -
PRAGMA integrity_checkandPRAGMA quick_check(#540, #541), part of epic #421 (V7)'s acceptance gate.Pragma(src/parser/ast.rs) became an enum (JournalMode/IntegrityCheck) to carry a new query-form pragma alongside the existingjournal_modeset-only carve-out; a newIntegrityCheckopcode (src/vdbe/program.rs) compiles from it and emits oneTEXTresult row per problem found (or a single"ok"row). The actual check (src/integrity.rs) walks every table/index b-tree via the existingTableCursor/IndexCursorread APIs — table rowid ordering, index key ordering, and (skipped byquick_check) the index-vs-table cross-check (every index entry's trailing rowid exists in its table; entry counts match) — plus the freelist trunk chain. Auto-vacuum databases (largest_root_btree_page != 0) report a single informational line rather than a false negative: this crate never writes a pointer-map (no auto-vacuum support at all), so pointer-map cross-validation is out of scope, tracked as a follow-up rather than implemented against machinery that doesn't exist yet.
tools/bench_status.pyfails loud instead of silently omitting a scenario/fixture pair frombench-status.jsonwhen its criterion output is missing (#523) — previouslytier1_results()treated a missingestimates.json(e.g. because the bench binary hit the VDBE step-limit guard rail or otherwise crashed partway throughmake bench) the same as "not measured",continue-ing past it with no signal. That's exactly how #303's 785xcorrelated_subqueryregression (fixed by #434) went unnoticed for weeks: it read as "no data" instead of "catastrophically slow".expected_pairs()now states explicitly which (scenario, fixture) combinations a healthy run must produce — accounting fortests/performance/crud.rs's intentionalbench_1mb.db-only scenarios — and any other absence raisesMissingBenchResults, failing the script instead of writing an incomplete status file. (The bench binaries themselves already fail loud viafail()/process::exit(1)on any execution error, includingStepLimitExceeded, since #112 — this closes the one remaining place a missing result could go unreported.)
- CLI
execfailed to bootstrap a brand-new database file (#448):sqlite-rs exec <file> "<sql>"required the target file to already have a valid SQLite header, unlike stocksqlite3 <file> "<sql>"which creates the file lazily on first write. AddedDatabaseHeader::new_empty_page1(src/header.rs) to build a valid empty-database page 1, written byrun_exec(src/bin/sqlite-rs/exec.rs) before opening whenever the target path doesn't exist yet.
- track and apply column-declared
COLLATEacross schema and comparisons (#500) —TableSchema/IndexedColumn(src/schema/ddl_reader.rs) now capture each column's/index-column's declaredCOLLATE(defaultBinary), previously parsed and discarded. A newexpr_collation()(src/codegen/expr/value.rs) falls back to a bare column's declared collation whenever the query has no explicitCOLLATE, wired into WHERE/IN comparisons,ORDER BY,GROUP BY, andmin/maxaggregate comparisons — an explicit query-sideCOLLATEstill wins.SeekIndexEq's probe and the #450/#492 duplicate-key recheck now carry the leading index column's collation via a newP4::SeekKeypayload instead of hardcodingBinary.Collation/compare_textmoved fromsrc/vdbetosrc/recordto keepschema's Tier 0 layer isolation intact.SELECT DISTINCT's ephemeral-index dedup (byte-equality on encoded records, never callingcompare()) is a separate mechanism and was filed as a follow-up (#518) rather than folded in here.
-
SELECT DISTINCTrespects declared/explicitCOLLATE(#518) — the ephemeral-index dedup path (Found/IdxInsertagainst an in-memoryBTreeMap) compared raw encoded record bytes, ignoring collation entirely; aCOLLATE NOCASEcolumn returned case-variant duplicates as distinct rows. Codegen now resolves each result column's collation (mirroring #500'sresolve_order_byfallback) into a newP4::SeekKeyoperand, and the ephemeral-cursor key-building normalizesNoCase/RTrimtext before encoding so byte-equality on the normalized key matchescompare()'s notion of equality. UNION's shared dedup path staysBinary-only, matching its existing conservative ORDER BY handling. -
assurance's
plan_blocks()regex dropped V5/V6 ("V5 Slim"/"V6 Slim" in plan.md's table didn't match the bare-tag-only regex), causing a false "grammar tags not in plan.md value blocks" drift report; also renamedtests/parity/v04.rs-v07.rs's test fns to name the dimensions they actually cover (acceptance/output), fixingmake assurance-gate's parity count from a misleading 3/12 to the accurate 7/12 — no test logic changed, only a naming-convention gap that made real coverage invisible to the dashboard's name-based heuristic. -
JOIN reordering now prefers a rowid/unique-index-seekable inner table over raw ANALYZE row-count ordering (#510) —
join_order::seekable_tablesflags a table whoseONequality is a structural rowid-alias/single-column-UNIQUE-index match (the same shapejoin_access::choose_join_accesslooks for), andscan_costsgives such a tableu64::MAXsoplan_join_order's ascending sort always places it innermost, letting the existing seek codegen fire regardless of the table's own size — a rowid/index seek is O(1)/O(log n) and always cheaper as an inner probe than as the outer scan. Fixes thebench_data JOIN bench_lookup ON bench_data.bucket = bench_lookup.codecase (bench_lookup.codeanINTEGER PRIMARY KEY) where the smaller table's row count previously won it the outer scan slot, forcing a full scan on the larger table's join column instead of aSeekRowidon the smaller one. spend: matched estimate. -
unindexed
GROUP BYaggregate sort-pipeline overhead (#506) —compile_grouped_scan's pass 1 now only serializes columns actually referenced by theGROUP BYkey, aggregate arguments, or plain result/HAVINGcolumns (every other schema column becomes a cheapNullplaceholder rather than a real per-row read);SorterInsertno longer copies the record blob on every insert (reuses the already-Rc'd bytes);OpenPseudois now emitted once before pass 2's loop instead of once per row. Also fixed a pre-existing, ticket-adjacent bug found while adding regression coverage: a plain (non-key, non-aggregate) result/HAVINGcolumn's "arbitrary row" snapshot picked the group's last row instead of the first, mismatching the real oracle's own sort-then-group behavior.group_by_aggbenchmark: 20.8ms -> 11.2ms on the 1MB fixture (~46% faster), still short of the ticket's<3x-oracle target — the residual gap looks architectural (VDBE per-instruction dispatch, the sort pipeline's inherent double-decode), out of this ticket's scope. spend: ~2-3x estimate.
-
trim
run()'s per-instruction dispatch overhead (#509) —checked_add+.ok_oron the step counter and the program-counter increment are replaced withsaturating_add(both are backstops against a pathological program, not values any real program comes close to overflowing, so theOptionconstruction/unwrap on every single instruction was pure overhead), andprogram.get(pc).ok_or(...)is replaced with alet-elseon the sameOptionto skip the extra error-value construction on the hot path. Confirmed via a stashed before/aftercargo bench --bench engineA/B on the 50MB fixture (criterion--save-baseline/--baseline, not just cross-run deltas, since the oracle's own unchanged-code runs showed up to ~2.5% run-to- run noise on this machine):full_scan-2.8%,full_scan_1col-4.3% (both p<0.05, above the noise floor),full_scan_3col-2.1% (within noise). A batched, check-every-4096-steps variant of the step-limit comparison was also tried and measured no further win beyond the above (ours/oracle ratio unchanged within noise), so it was dropped rather than kept as unjustified complexity, per the ticket's own evaluate-and- keep-or-drop mandate. Candidate #1 (streaming rows instead of materializingvm.rows: Vec<Vec<Value>>) was evaluated and deferred: every current caller (CLIquery/repl, the write-path executor) consumes rows fully after the fact, so streaming would mean threading a new execution API through every call site for what both this issue's own data and #506's prior finding indicate is a modest, not gap-closing, win — filed as its own follow-up rather than attempted speculatively here. Candidate #3 (specializingNext/Rewindso a tight scan loop skips re-enteringdispatch()'s generic match) doesn't have a safe, small-scope implementation: Rust already compilesdispatch()'s opcode match to a jump table, so there's no per-arm match-order cost to cut, and a real fast path would mean recognizing and specially executing a whole loop body betweenRewind/Nextand its matching jump — a genuine VM-architecture change (superinstructions / threaded code), not a contained edit; filed as a separate, better- scoped follow-up (#515) with today's benchmark numbers as its starting evidence rather than attempted here. Candidate #4 (Column-opcode- specific flat tax) is answered by the above:full_scan_1col/full_scan_3colstill sit at ~2.0x/~1.6x oracle after this fix, confirming the residual gap is real column-decode cost, not a separate per-call tax — matching #506's own "VDBE per-instruction dispatch... architectural" note.full_scan_1col/full_scan_3col's own remaining ResultRow-side gap is being addressed separately (agent-3, not part of this ticket). spend: matched estimate. -
SorterInsertnow decodes only through theORDER BYkey's highest column index instead of every selected column (#507) —SorterStatecomputesdecode_upto(one past the maxSortKeyColumn.index) once atSorterOpen, and a newdecode_record_upto(src/record/decode.rs, reusingdecode_column's header-walk-then-partial-decode pattern) replaces the prior fulldecode_recordcall on every candidate row. #506 had already fixed the double-copy half of this pattern (Rc<[u8]>reuse instead ofblob.to_vec()); this closes the remaining full-row-decode half for the general-purpose sorter backing plainORDER BY ... LIMIT(as opposed to #506'sGROUP BY-specific codegen path).order_by_limitbenchmark ratio vs the pinned oracle: ~3.2x-6.15x -> ~1.2x (37µs/30µs on the 1MB fixture, 43.5µs/36.4µs on 50MB) — beats the ticket's<2x-oracle target. spend: matched estimate (medium) -
TableCursor::seek(backing theSeekRowidopcode) now binary searches a page's cell-pointer array instead of scanning it linearly, on both leaf pages (rowid comparison) and interior pages (separator-key comparison) (#508). Repeated seeks against the same table — thejointier-1 benchmark's dominant cost, oneSeekRowidper outer row — no longer pay an O(cells-per-page) decode-and-compare loop per call; thejoinbenchmark's ratio against the pinned oracle dropped from ~7.4x to ~2.3x (14.07ms→4.5ms on the 1MB fixture, 1.16s→284ms on 50MB). This was misattributed for a time to ADR-0022's missing-Pager-page-cache gap, which had already been closed by #320/#457/#459 — see ADR-0028, which supersedes ADR-0022's now-stale problem statement. ANALYZE and join-ordering/access-selection were independently confirmed unaffected (EXPLAIN QUERY PLANis identical before and after, on both oracle and ours:SCAN bench_data/SEARCH bench_lookup USING INTEGER PRIMARY KEY) — the gap was purely inseek's own per-page search algorithm, not query planning. spend: matched estimate (medium)
Epic #421's V7.2 phase (Performance & Planner), now complete: the query
planner (ANALYZE + cost model, join ordering, Bloom-filter join
elimination, skip-scan), a run of targeted VDBE/pager performance work
(row-header cache, zero-copy payload, page-cache hashing, correlated-
subquery memoization, CTE materialization sharing), and the /review
follow-up that closed out its warning-level findings. V7.3 (PRAGMAs &
Introspection) is next.
-
share one materialization across repeated
FROM-subquery references (#425, epic #354's V6.1 "10x on repeated subqueries" target) —expand_with_clauserewrote every reference to aWITH-clause CTE into its own independentTableRefKind::Subquery, so a CTE referenced N times re-ran and re-materialized its body N times, same cost as inlining it N times. A new VDBE opcode,OpenDup(p1=new_cursor, p2=source_cursor), opens a second, independently-scanning cursor sharing an already-materialized ephemeral table's row data instead of a freshOpenEphemeral+ populate;RegAlloccaches materializations per statement compile, keyed by structural equality of theSelectbeing materialized (not an AST identity field — that approach grewSelect's inline size enough to trip an unrelated pre-existing depth-guard test sitting at the edge of a real stack overflow in debug builds).cte_reuse_10xbench:cte1.34ms vsinline11.1ms (was statistically identical before this fix) — roughly 8x. -
skip-scan for non-leading composite-index columns (#485) —
WHERE b = ?against a composite index(a, b)(leading columnaunconstrained) now uses the index instead of a full table scan, whenevera'sANALYZE-derivedavg_eqclears the oracle-confirmed skip-scan threshold (empirically measured against sqlite3 3.51.0: `avg_eq
= 18
, matching sqlite.org/optoverview.html's documented "~18 duplicates").is_skip_scan_worthwhile(src/planner.rs) mirrors oracle's own decision;try_compile_skip_scan_index(src/codegen/select/limit_scan.rs) walks the whole index (IdxRewind/IdxNext), checking the constrained column on each narrower index entry andIdxRowid+SeekRowid-ing into the table only for a match;eqp.rsreports the same oracle-verbatim EQP text (SEARCH t USING INDEX idx (ANY(category) AND price=?)). Unlike real SQLite's skip-scan (a genuine per-distinct-leading-value binary seek),IndexCursor::seekin this codebase is a documented Tier 0 linear scan — this walks every index entry rather than truly skipping past a large group, so the measured win (tests/performance/skip_scan.rs,make bench-skip-scan`: ~1.24x, 10.04ms → 8.12ms at 200K rows/3 leading values) comes from narrower index-row decode and selective table lookups, not sub-linear seeking — reported honestly rather than as an oracle-parity ratio. spend: ~1.2M token budget, matched estimate.
-
ORDER BY/LIMITon a compound (UNION/UNION ALL)SELECT(#484) —compile_select_compoundpreviously rejected any top-levelORDER BY/LIMITtrailing a compound statement. Every arm's projected rows now feed a shared sorter (reusing the sorter opcodescompile_sorted_scanalready uses for a single-tableORDER BY) beforeLIMIT/OFFSETand finalResultRowemission; aLIMITwith noORDER BYskips the sorter entirely, reusing the simpler counter-based guardscompile_direct_scanuses. AnORDER BYterm must be an output column name/alias or an ordinal position — matching real SQLite, which rejects any other expression here even when it only references an output column name. Refs: #484. -
REPL mode with
.tablesand.quit/.exitprefix matching (#478) — baresqlite-rs <file>(no subcommand) now enters the REPL directly, matchingsqlite3's shell; adds a.tablesdot-command (reusing thetablessubcommand's listing/columnizing logic) andsqlite3-style prefix matching for.tables/.quit(.t,.ta, ... and.q,.qu, ...);.exitremains an exact-only alias for.quit. -
ANALYZEcommand and cost model for the query planner (#461, spec 011) —ANALYZE/ANALYZE table-namepopulatessqlite_stat1(row counts + per-indexavg_eq);Stats/PlanCost(src/planner.rs) estimate scan/index-probe cost from those stats;choose_join_accessvetoes a structurally-pickedUNIQUE-index seek back to a full scan when the cost model says it isn't actually cheaper, wired live into the CLI'squery/replpath. A database with noANALYZEhistory compiles byte-for-byte as before this change. Filed #470 (join ordering heuristics) as the real follow-up enabled by this ticket.
-
fail closed, not open, when a checkpoint's page-count bound overflows
u32— surfaced by amake silent-swallowrobustness audit (unrelated to epic #421).checkpoint_passive's own comment states the intent plainly: a corrupted-but-checksum-valid WAL frame with apage_numnearu32::MAXmust never drivewrite_atto an arbitrary offset beyond the database's actual extent — but the bound computing the main file's current page count fell back tou32::MAXonu32:: try_fromoverflow, which mademax_pageeffectively unbounded, defeating the exact check the comment describes. Extracted into a testablepage_count_from_size()helper, now falling back to0(max_pagethen falls back to the WAL's own already-validateddb_sizebound) instead. -
support
GROUP BY/aggregate combined withORDER BYand a JOIN (#502, found via the V07 parity suite #72) —compile_joined_grouped_ scanpreviously rejected this combination outright. A third codegen pass now inserts each finalized group row into a second sorter keyed by the resolvedORDER BYtargets instead of sinking it directly;LIMIT/OFFSETmove from per-group-flush time to after that final sort, since which rows aLIMITkeeps isn't known until theORDER BYorder is resolved.ORDER BYterms resolve against a whole aggregate call (matched structurally, not byExprequality, since anORDER BYterm is a separately-parsed AST node from its SELECT-list twin), an ordinal, a result-column alias, or a bare joined column.DISTINCT+ JOIN +GROUP BYstays unsupported (split out of this ticket's scope). -
share one
-shmfd per path per process across all WAL lock guards (#491, follow-up from #412's investigation) —WalWriteLock/WalCheckpointLock/WalReadLock/UnixWalShm(src/vfs/shm.rs) each opened an independentFile, but POSIXfcntlrecord locks are scoped to(process, inode), not to a file descriptor: closing any fd this process holds to a file releases every lock the process holds on that inode, even ones taken through a different, still-live fd. Two such guards held concurrently in one process (e.g.checkpoint::checkpoint_passivecalled directly while a separatePagerholds its own long-livedWalReadLockon the same file) could have one guard'sDropsilently release the other's still- needed lock. A newopen_shm_sharedregistry (Arc/Weak-backed, keyed by path) makes every guard/helper reuse the one fd already open for a path — it only actually closes once nothing in the process needs a lock on that inode anymore. Derived from stock sqlite3's ownos_unix.c, whoseunixClosedefers closing for exactly this reason. -
CTE referenced from more than one arm of a compound
SELECT(#424) —compile_select_compound's per-arm codegen unconditionallyOpenRead'd a resolved-table root page, ignoringTableRefKind::Subqueryentirely, so any arm referencing a CTE (or otherFROM-subquery) hitunsupported: table X has an invalid root page (0)instead of being materialized. Each arm now branches the same way the single-SELECTpath already does, callingmaterialize_from_subqueryper arm on its own cursor. Refs: #424. -
recreate a vanished
-wal/-shmduring flush instead of failing (#422) —Pager::flush_wal_lockedunconditionally calledwal::WalWriter::open_existing, which errored if a concurrent connection (e.g. a realsqlite3client auto-checkpointing on close) had deleted-wal/-shmout from under thisPager, even though its ownjournal_modestill correctly saidWal. Now recreates a fresh-wal/-shmpair on that specificNotFoundcase, mirroringswitch_journal_to_wal's from-scratch creation — matches stocksqlite3's own observed behavior in the same scenario.
-
address V7.2 review warnings from epic #421 (#501) — verified and documented, rather than patched, three warning-level findings that were already structurally safe or a pre-existing crate-wide gap: (1)
take_registerreuse inResultRowis safe becauseRegAllocnever reuses a register number andGROUP BY'sprev_keybookkeeping registers are kept separate from anythingResultRowprojects; (2)SeekIndexEq's duplicate-key recheck hardcodesCollation::Binaryconsistently with the seek it walks past — no column-declaredCOLLATEexists anywhere inTableSchemayet, so patching just the recheck would have been inconsistent (tracked as the real fix in #500); (3) CTE materialization cache reuse is safe today because no volatile expression (random(),CURRENT_TIME, etc.) exists in the parser yet — corrected an overstated "always guaranteed" doc comment and spelled out what the first volatile function must account for. -
add WAL-mode variants to
engine.rs's transaction benchmarks (#436) —insert_single_tx_wal,insert_batch_tx_100_wal,insert_batch_tx_1000_wal,update_batch_tx_walnow run each transaction scenario underjournal_mode=WAL(via aPRAGMA journal_mode=WALswitch excluded from the timed closure, mirroringv6.rs's existingswitch_to_walpattern) against the oracle, alongside the existing DELETE-mode variants — previously onlyv6.rscompared WAL vs DELETE, with no oracle reference point. -
close remaining missing_docs gap for docs.rs (#430) — adds
///doc comments to the ~890 previously-undocumented public items across the nested submodules (parser,vdbe,btree,pager,codegen,vfs,record,schema) that #428 left out of scope, and enables#![warn(missing_docs)]insrc/lib.rsso future regressions are caught bycargo build/clippy. No logic or behavior changes.
-
index-mode memoization cache for correlated scalar subqueries (#494, follow-up from #434/#435) — #314's per-probe-value cache (
src/codegen/subquery/memoize.rs) previously used a table-modeOpenEphemeralcursor with a linearRewind/Eq/Nextscan per lookup, capped atMAX_MEMO_CACHE_ENTRIES = 8distinct probe values to bound worst-case VDBE step counts — any higher-cardinality correlated column fell back to recomputing every row. The cache now uses an index-modeOpenEphemeralcursor (Found/IdxInsert), backed by aBTreeMapfor an O(log n) lookup regardless of cache size, so the cap is removed entirely (bounded only by the ephemeral cursor's existingMAX_EPHEMERAL_ROWSceiling).IdxInsertgains aP5operand (extra payload-only registers beyond theP4key count) andColumngains read support for index-mode ephemeral cursors, so a cache entry's key (the probe value) and its cached result no longer have to be the same registers (src/vdbe/cursor.rs). -
zero-copy
IndexRowpayload for index/WITHOUT ROWID scans (#471) — follow-up to #467, which leftIndexRow::payload(src/btree/index.rs) as an ownedVec<u8>out of scope.IndexFrame.pageis nowRc<[u8]>(matchingPageSource::read_page's return type);IndexRow::payloadreuses thePayloadenum #467 introduced instead ofVec<u8>;decode_leaf_entry/decode_interior_entrypass&frame.pagedirectly intoreassemble_payloadinstead of wrapping it in a throwawayRcand immediately copying the result back to aVec.decode_value_cell(the Pager-only index insert/delete write path) is unchanged. -
non-
UNIQUE-index duplicate-key matches for the covering-index scan and index-onlyCOUNT(*)fast paths (#450, follow-up from #444) — both fast paths previously required aUNIQUEindex becauseSeekIndexEq's one-shot probe couldn't walk forward past duplicate keys.SeekIndexEqnow seeks the index-read cursor's own persisted traversal position (state.cursor) instead of a throwaway one, so a followingIdxNextresumes right after the matched entry; both fast paths add anIdxNext+ leading-column-still-equal recheck loop that walks and emits/counts every duplicate-key sibling, falling out the first time the leading column no longer matches (aUNIQUEindex's single match still falls out on its very firstIdxNext, so this subsumes #444's original single-probe behavior without a separate branch). Refs: #450, 009/Req-16. -
hand-rolled multiplicative hasher for the pager's page-cache
HashMap(#457) — page numbers are plain sequentialu32s, not adversarial input, so the default hasher's SipHash cost onPageCache::entries's hot get/insert path was unneeded. No new dependency (ADR-0022 already ruled that out for this cache);point_lookupbench ~5% faster on both fixtures.
- cache reassembled payload per row position (#469, #475) —
TableCursorState::current_payload()was being called once perColumnopcode instead of once per row (a regression introduced by the lazy-payload change), so an N-columnSELECTpaid for N payload reassembly passes per row instead of 1. Fixed with the same once-per-row cachingheader_cachealready uses, restoringfull_scanto its expected ~1.06x ratio vs the oracle. Also adds regression tests for thePayload::Ownedoverflow-chain case and page-cache-hitRcsharing, plus an overflow-payload reassembly benchmark.
- borrow table row payload from page buffer instead of copying
(#467) —
PageSource::read_pagenow returnsRc<[u8]>instead ofVec<u8>, so aPageCachehit is a refcount bump rather than a copy;TableRow::payloadbecomes aPayloadenum that borrows a range of the shared page for the non-overflow case (zero-copy) and only owns bytes for the overflow-chain case.
- reuse row buffer and move registers in
ResultRow(#465) — eliminates the per-rowVecallocation and per-columnValueclone by reusing aVm-owned scratch buffer (mirroringrecord_scratch, #454) and taking each register's value viatake_registerinstead of cloning it, safe because every scan loop reloads its projected registers before the nextResultRowreads them again. Also corrects a hand-built DISTINCT VDBE test whose instruction order (ResultRowbeforeIdxInsert) didn't match what real codegen produces.
- cache parsed row header for repeated
OP_Columnreads (#458) — table and index-read cursors now parse a row's header (serial types + byte offsets) once, viarecord::parse_header_into, and cache it (RowHeaderCache) on the cursor state, instead ofdecode_columnre-walking the header from byte 0 on everyColumnopcode against the same row. The cache reuses its backingVecallocation across rows (a first, simplerOption<RowHeaderCache>-per-row design measured as a regression onfull_scandue to alloc/free churn) and is invalidated solely throughTableCursorState/IndexReadState::set_current, so a stale cache can never survive a row change.full_scanbench ratio (ours/oracle) improved from 1.39×/2.10× (1MB/50MB fixtures) to 1.25×/1.94×.
- no-stats query optimizations (#444) — two "always wins, no
ANALYZE/cost model needed" optimizations. Covering-index scan: a
single-table
SELECTwhoseWHEREis a top-level equality on aUNIQUEindex's leading column, with every result column already carried by that index, compiles toSeekIndexEq+Columnreads straight off the index cursor, never opening the table cursor. Index-onlyCOUNT(*): counts via the index cursor (IdxRewind/IdxNextor a singleSeekIndexEqprobe) without ever decoding a table row. LIMIT early-out (#128's third example) needed no new codegen — the existingemit_limit_guard/sorter top-K bound already cover it.find_covering_indexis shared between codegen andEXPLAIN QUERY PLANso the two can't drift apart. Non-unique-index duplicate-key matches deferred to #450. See spec 009 Requirement 16.
- correlated scalar subquery equality seeks instead of scanning (#434)
— a correlated scalar subquery's own
WHEREclause always compiled to an unconditionalRewind/Nextscan, even when it was a trivially seekable equality against the subquery table's rowid or aUNIQUEindex. Comparing against the pinned sqlite3 oracle's ownEXPLAINoutput for the reported query showed it uses no caching for this shape at all — it compiles the equality to a singleSeekRowidper row.compile_scalar_subquery(src/codegen/subquery/scalar.rs) now reusesjoin_access::choose_join_access(#243's join-level access-strategy classifier) to take the same fast path; no new VDBE opcode needed.#314's memoization cache (ADR-0021) stays in place for correlated subqueries whoseWHEREisn't a seekable equality.correlated_subquerybenchmark: 785x oracle-relative and unmeasurable againstbench_50mb.db(blew the 50M-step VDBE guard rail) down to ~14-15x on both fixtures. See ADR-0027.
-
decode UTF-8/UTF-16 text straight into
Rc<str>(#441) — text columns were decoded through an intermediateStringbefore converting to theValue::Text(Rc<str>)representation, allocating and copying twice per text value.decode_text(src/record/decode.rs) now builds theRc<str>directly from the decoded bytes, halving text-column decode allocations. Re-scoped from the ticket's originalValue<'a>borrow-from-page-buffer proposal, found premature: the page cache that design needs is deferred (ADR-0022), and even once built is LRU-evicting/mutable in place, which is incompatible with a live borrow under#![forbid(unsafe_code)]. -
cache the WAL
-shmfd across a connection's lifetime (#437) —Vfs::open_wal_shmreturns a persistent handlePagercaches and reuses for every commit's write-lock claim/mxFramepublish, instead of reopening-shmfresh each time. Investigated via a profiling spike (#438,tests/spike/011_wal_performance); ~17.5% faster on a many-commits-per-connection workload (concurrent_read_writebenchmark), though the originalinsert_batch_wal_walbenchmark stays flat since it opens one connection per commit and has nothing to cache across.
- crate-level rustdoc polish (#428) — crate-level
//!docs, Cargo.toml publication metadata (description,repository,documentation,keywords,categories), and doc comments on previously-undocumented top-level public items (DumpError,HEADER_LEN,HeaderError,VfsError,vfs::Result). Closing the remaining ~884missing_docswarnings in nested submodules is tracked in #430.
- lazy per-column record decoding for the VDBE
Columnopcode (#439) —decode_column(payload, idx, encoding)(src/record/decode.rs) walks the record header to find one column's offset and decodes only that column's body, instead ofdecode_record-ing the whole row on every column access. Because WHERE, SET, and SELECT-list column reads all compile to the sameColumnopcode, and a row aWHEREfilter rejects skips its later opcodes via jump-if-false, this gives lazy column decoding to SELECT, UPDATE, and DELETE uniformly with no codegen changes.filter_scanbenchmark (50MB fixture): down from a reported 2.8x gap vs. the oracle to running faster than it (0.65x).
Phase V6.3 of epic #354 (V6 Slim), finalizing V6 and unlocking 1.0: real WAL-mode writes with multi-reader/single-writer concurrency, and the sqlite3-interop demo that was the epic's stated goal. Closes #388, #389, #390, #391.
-
minimal
PRAGMA journal_mode=WAL|DELETEswitching (#388) — a narrow V6 grammar carve-out (.openspec/grammar/sqlite.ebnf, general PRAGMA support stays deferred to V7) parses only this one pragma name/value pair, with everything else falling through to a cleanUnsupported. Codegen/VDBE wiring (Opcode::SetJournalMode) actually runs the switch:Pager::set_journal_modecreates a fresh-wal/-shmand flips the header's version bytes going into WAL, or checkpoints every pending WAL frame, deletes-wal/-shm, and flips the header bytes back going to DELETE. Refuses mid-transaction, matching stock SQLite. -
WAL-mode writes actually go through the WAL (#389) —
Pager::flushnow branches on the tracked journal mode: injournal_mode=WAL, every dirty page is appended as a WAL frame (WalWriter::open_existing, resuming across sessions rather than truncating), the last one marked as the commit frame,mxFramepublished to-shm, and the writer's own subsequent reads served by folding the new pages into its in-memorywal_pagesoverlay — all without ever escalating the main file's SHARED lock to EXCLUSIVE, so readers are never blocked. A newWAL_WRITE_LOCK(src/vfs/shm.rs,Vfs::claim_wal_write_lock) serializes concurrent writers, surfacing contention as the existingVfsError::Lockedpath.rollbackin WAL mode was already correct (frames are only ever appended at commit time). See ADR-0026 for the writer-reopens-and-rescans trade-off. Un-ignoringtests/tiers/tier3.rs'st3_wal_writing_and_live_interopstays for #390 (live interop with a real stocksqlite3process).
-
sqlite-rs + stock sqlite3 concurrent WAL interop (#390) — the "V6 demo" gate for epic #354:
tests/corpus/wal_concurrent_interop_test.rsdrives sqlite-rs through the same SQL-level entry point (execute_transaction_step/compile_statement, the machinerysqlite-rs execalready wraps) a real caller would, proving all four scenarios against a live, pinnedsqlite3process — sqlite-rs writes/ oracle reads, oracle writes/sqlite-rs reads, both alternate commits (round-tripping WAL frames through each other's checksum chains), and a checkpoint by either side is read correctly by the other. Un-ignorestests/tiers/tier3.rs'st3_wal_writing_and_live_interop. Also fixes a real gap this surfaced:dump::open(the CLI'sdump/query/execbootstrap) parsed the database header from the main file's raw bytes only, which fails for a WAL-mode database whose very first schema-creating transaction hasn't been checkpointed yet (that page 1's real content lives only in the-walfile) — it now falls back to a lenientpage_size-only bootstrap and re-derives the header from thePager's WAL-aware read of page 1. -
V6 WAL benchmarks (#391) —
tests/performance/v6.rs(make bench-v6/cargo bench --bench v6), four scenarios adapted from the ticket to what the codebase can measure honestly:insert_batch_wal(1000-row batch INSERT, journal vs WAL mode, driven through the realPRAGMA/compile_statement/execute_transaction_stepSQL path, plus the pinned oracle's own journal-vs-WAL numbers as a sanity check);concurrent_read_write(a documented sequential interleaving — a long-open reader's pinned WAL snapshot alongside 20 writer commits, not a wall-clock-parallel harness; #390's own tests already prove the non-blocking property, this just measures throughput);checkpoint_10mb(checkpoint_passiveagainst a directly-built ~10MB single-commit WAL); andcte_reuse_10x(a CTE referenced 10x via self-join vs. the same subquery repeated 10x inline). First run (--sample-size 10, informal):insert_batch_walours journal ~65ms vs WAL ~59ms (a small WAL win, not the ticket's hoped-for 2.5x — ADR-0026's per-flush-walrescan caps it; oracle's own journal/WAL numbers are ~3.3ms/~3.4ms, indistinguishable at this batch size);concurrent_read_write~338ms/20 cycles;checkpoint_10mb~27-31ms;cte_reuse_10xcte ~40.9ms vs inline ~40.6ms — parity, not a win, becauseexpand_with_clauserewrites every CTE reference into its own independent materialization (confirmed by readingsrc/codegen/subquery/cte.rs), identical cost to inline repetition — there is no shared-materialization optimization yet (filed as #425). Also surfaced, not fixed here (out of scope for a benchmark ticket): a 10-wayUNION ALLofSELECT count(*) FROM ctefails compilation past the first arm ("table cte has an invalid root page (0)") — a compound-arm/CTE codegen gap, filed as #424. spend: roughly matched the issue's 1-day estimate. -
Also filed from this phase's work: #422 (
Pagershould recover, not error, when another connection's auto-checkpoint deletes-wal/-shmout from under it — found while building #390's interop tests). -
spend: V6.3 as a whole ran noticeably over its ~5-day estimate — #388 also had to add a minimal PRAGMA parser (none existed), and #389 had to make
Pager::flushgenuinely WAL-aware (the write path was entirely rollback-journal-only beforehand) — both prerequisites the original per-ticket estimates didn't account for.
parse_insert_stmtpanicked viaexpect()if the firstVALUESrow were ever empty, even though that path was already unreachable (expr_listalways seeds one element before anyOk, with?short-circuiting earlier failures). Replaced with the same safe fallback idiom already used for subsequent rows. The other twoexpect()sites #409 flagged (pager.rs,btree/master.rs) turned out to be test-only code, already correctly lint-allowed — no change needed there. Closes #409.
Phase V6.2 of epic #354 (V6 Slim): the write half of the WAL format, the wal-index checkpoint-coordination pieces, PASSIVE checkpoint, and crash recovery / oracle-parity acceptance tests. Closes #383, #385, #386, #387.
-
WAL frame writer (
WalWriter) andWalHeader::new/serialize, completing the write side of the WAL file format alongside the existingWalHeader::parse/committed_pagesread path. -
wal-index (
-shm) checkpoint coordination —WAL_CKPT_LOCK, a probe for which reader-mark slots are actually held (bounding checkpoint progress), andnBackfillread/publish. -
checkpoint_passive— copies committed WAL frames into the main database file up to the oldest active reader's mark, without blocking on readers (FULL/RESTART deferred to V7).
- WAL crash recovery (torn-frame tolerance, checkpoint-mid-write
consistency) and write-path oracle parity — a
-walfile written by sqlite-rs recovers correctly through a real, pinnedsqlite3.
Phase V6.1 of epic #354 (V6 Slim): non-recursive CTEs, UNION/UNION ALL
compound SELECT, and CREATE VIEW/DROP VIEW.
-
WITHclause (non-recursive CTE) parsing and codegen materialization — a CTE reference inFROM/JOINis rewritten into the sameTableRefKind::Subqueryshape #257's subquery-in-FROM machinery already materializes and scans, including multi-CTE chaining, explicit(col, ...)lists, and self-joins.WITH RECURSIVEparses far enough to report a cleanUnsupportedrather than a syntax error. Closes #375, #376. -
plain
UNIONcompoundSELECT(UNION ALLpre-existed via #240), deduplicated via a shared ephemeral-index cursor reusingSELECT DISTINCT's guard shape. Closes #377, #378. -
CREATE VIEW/DROP VIEWparsing, view storage insqlite_master, and query expansion — a view reference inFROM/JOINis rewritten the same way a CTE is, runs after CTE expansion so CTE-of-view and view-of-CTE both resolve, and detects direct/mutual view-definition cycles with the same "view X is circularly defined" message stock SQLite reports.DROP VIEWparses but is not yet wired into codegen — cleanly rejected rather than panicking. Closes #379, #380.
INSERT ... SELECTwas silently dropping compound-SELECT arms in codegen, and CTE/view materialization was silently scanning only the first arm of a compound-SELECT body — both real correctness gaps found while building this phase, now cleanly rejected instead of producing wrong results. A CTE substituted into an inline derived table's ownFROM, and a view's own body starting withWITH, now also resolve correctly (fixed asymmetries against the already-working paths). Three extracted-SQL-corpus statements ([NOT] MATERIALIZEDCTE hint,WITHfeedingINSERT, single-quoted alias) that the newWITH/UNIONparsing reached further into were reclassified fromInvalidtoUnsupported, loweringSELECT_INVALID_BASELINEfrom 8 to 3.
- oracle-diff parity coverage across
tests/corpus/{cte,union, view}_test.rsfor all of the above, including circular/mutual view references, CTE shadowing a real table, and clean-rejection pins for every not-yet-supported combination (compound CTE/view bodies, compound INSERT source, CTE/view-backed INSERT source,DROP VIEW). Closes #382.
Refs: 009/Req-13 (CTE materialization), 009/Req-14 (compound SELECT), 009/Req-15 (view storage and expansion).
Follow-up fixes from the combined code-review/security-review pass over the eight merged V5 PRs (#353 comment thread). Patch release — no new features, no scope beyond closing gaps the review found in the V5 lock and transaction-control paths.
-
BEGIN IMMEDIATE/BEGIN EXCLUSIVEparsedTransactionModebutcompile_begindiscarded it, so a concurrent writer was only blocked atCOMMITtime (viaPager::flush's EXCLUSIVE escalation), not atBEGINas stock SQLite does.Transaction'sP1now carries the mode;control::transactioncalls the newPager::begin_immediate/begin_exclusiveto escalate to RESERVED/EXCLUSIVE right away.Pager::flush/rollbackrelease that lock back toSharedwhen the transaction ends (commit or rollback), including the no-write case (aBEGIN IMMEDIATEimmediately followed byCOMMIT/ROLLBACK). New subprocess interop test (tests/corpus/begin_immediate_lock_interop_test.rs) proves a compiledBEGIN IMMEDIATE/EXCLUSIVEvisibly blocks a live stocksqlite3writer/reader, mirroringlock_state_interop_test.rs's proof for the raw lock primitive. Closes #395. -
Pager::flushnever used the 5-state lock ladder built for hot-journal recovery — it only ever held the plain SHARED lock everyopen()takes, so two writers (orsqlite-rsracing a live stocksqlite3process) could both pass the SHARED check and interleave journal writes/deletes and page writes with no OS-level mutual exclusion.flush()now escalates to EXCLUSIVE (stepping through RESERVED/PENDING) before touching the journal or main file, and de-escalates back to SHARED afterward, mirroringsqlite3PagerCommitPhaseOne/Two. Closes #398 (Refs #353). -
nested
BEGIN; BEGIN;and a bareCOMMIT/ROLLBACKwith no open transaction silently succeeded instead of erroring like stock SQLite — a divergence the V5 review flagged as untested. Both now return the matching stock-sqlite3error. Closes #396. -
cargo-mvl-limitinstall lacked--force, soSwatinem/rust-cacherestoring a stale cached binary made the mvl-limit gate flaky in CI. Closes #394 (chore, CI-only).
- Spend: matched the review-fix estimate (see #353 review comment for the combined analysis this closed out).
Epic #353. "ACID with a rollback journal": BEGIN/COMMIT/ROLLBACK
(including DEFERRED/IMMEDIATE/EXCLUSIVE), the 5-state file lock
ladder, rollback-journal write path, hot-journal crash recovery, and the
VDBE transaction opcodes that make it all executable. Spend: matched the
epic's 2-3 week estimate.
-
parser support for
BEGIN/COMMIT/ROLLBACK(DEFERRED/IMMEDIATE/EXCLUSIVEtransaction modes) as first-class statements. Closes #356. -
src/vdbe/control.rsgains the transaction opcodes (Transaction/AutoCommit) that make compiledBEGIN/COMMIT/ROLLBACKactually run against the pager instead of just parsing. Closes #360. -
execCLI subcommand runs multi-statement scripts through a single session, so a script'sBEGIN ... COMMITspans multiple statements against one connection instead of one-shot-per-statement. Closes #358. -
minimal
replsubcommand for interactive multi-statementBEGIN/COMMIT/ROLLBACKsessions. Closes #365. -
src/vfs/lock.rsgainsLockLevel/FileLockState, a full 5-state journal-mode lock ladder (UNLOCKED → SHARED → RESERVED → PENDING → EXCLUSIVE) built on byte-identicalfcntlbyte-range locks, matchingos_unix.c'sunixLock/unixUnlocktransition order and PENDING_BYTE probe semantics. Exposed fromsqlite_rs::vfsfor the follow-upPagerwrite-path wiring (#45);Pager/VfsFile::lock_sharedare unchanged. Verified against a live stocksqlite3process in both directions (tests/corpus/lock_state_interop_test.rs). Closes #357. Spend: matched estimate.
Pager::openrecovered a hot rollback journal from its header magic alone, with no check for a live second connection — a race against the oracle's ownhasHotJournal/sqlite3PagerSharedLock(os_unix.c/pager.c) behavior. Now non-blocking-probes RESERVED before recovering (FileLockState::check_reserved,fcntl(F_GETLK)) and fails withVfsError::Lockedif held; on a clear probe, escalates SHARED straight to EXCLUSIVE, deliberately skipping RESERVED, matching stock SQLite. Along the way, fixed a related bug the wiring surfaced: hot-journal recovery opened a second, independent fd to the main db path — exactly the "close()drops allfcntllocks on the inode" trap #45 had flagged and deferred — now consolidated to the one fdPager::openalready holds the lock on.FileLockState(#357's 5-state ladder, previously wired into nothing outside its own unit tests) now backsUnixVfsFile's lock directly. See ADR-0024. Closes #359 (rescoped from a duplicate of #172/ADR-0016). Spend: ~1 session, matched the rescoped 1-day estimate.
-
crash torture test — a kill -9 loop mid-write against the rollback journal, verifying the database always recovers to a consistent state on restart. Discharges the epic's "power-cut torture test" acceptance gate. Closes #361.
-
tests/performance/engine.rsgains four transaction-batching benchmarks (insert_single_tx,insert_batch_tx_100,insert_batch_tx_1000,update_batch_tx), each running aBEGIN/statement(s)/COMMITsession throughexecute_transaction_step(#360) against a fresh scratch copy ofbench_1mb.dbper iteration, with anoraclecounterpart (rusqlite::execute_batch) alongside each — surfacing the per-statement journal/fsync overhead V5's rollback-journal path pays outside a transaction vs. amortizing it across a batch, and lettingmake benchcheck the issue's "within 5× of oracle" batch criterion directly. Current numbers (bench_1mb.db,--quick):insert_single_tx16ms vs oracle 3ms (~5×),insert_batch_tx_100061ms vs oracle 4ms (~17×) — batching cuts our per-row cost far faster than linear, but the ratio to oracle isn't at 5× yet outside the single-row case; left as a follow-up rather than in scope here. Closes #373. Spend: matched the small estimate. -
src/vdbe/cursor.rswas the largest coverage gap in the repo (82.37% lines / 68.39% functions). Adds hand-assembledProgramtests for opcodes no current codegen path emits (Last,NullRow,IdxLE) and for theCursorTypeMismatch/MalformedInstructionerror arms that accounted for most of the file's missed lines — 84.33% line coverage after, repo TOTAL 90.91%. No production code changes. Part of epic #234 (V4). Closes #351.
-
16 codegen call sites (INSERT/UPDATE/DELETE/SELECT/subquery) defaulted an out-of-range
sqlite_master.rootpageto0instead of rejecting it —index_maintenance.rs::open_index_cursorsalready had the correctCodegenError::Unsupportedrejection for an index's root page, but 16 other table/index root-page sites used the naivei32::try_from(...).unwrap_or(0)shortcut instead. A corrupt or adversarialsqlite_masterentry could silently produce a cursor pointed at page 0 (the reserved header page) instead of a compile error — wrong results, no diagnostic. Factored the existing check into two shared helpers (valid_table_root_page,valid_index_root_page) and applied them everywhere. Found viamake silent-swallow's #342 audit (#349). -
honor DISTINCT in aggregates and coerce TEXT/BLOB in sum()/avg() —
count(DISTINCT x)/sum(DISTINCT x)/avg(DISTINCT x)previously silently ignoredDISTINCT, andsum()/avg()skipped TEXT/BLOB inputs instead of coercing them to their numeric-prefix value per SQLite's own text-coercion rule (R-29052-00975). Found via the vendored sqllogictest suite, which now passes in full (#348).
-
hoist uncorrelated WHERE-clause subquery in aggregate scan too (#322, #323) — extends #314's per-outer-value memoization/hoisting to the aggregate-scan codegen path, not just plain SELECTs.
-
UPDATE/DELETE rowid/index-equality seek fast path (#336) — point mutations no longer pay a full table/index scan to find the target row.
-
in-place leaf/index cell splice instead of collect-all/rewrite-all on single-row mutation (#337) — single-row INSERT/UPDATE/DELETE on table and secondary-index leaf pages now splices the cell-pointer array in place (O(1) relative to the page's other cells) instead of decoding and rewriting every cell on the page, when there's enough contiguous free space. Adds real freeblock-chain and fragmented-byte bookkeeping per
fileformat2.html(previously always written zero — see.openspec/adr/0023-leaf-cell-splice.md).deletealways takes the O(1) path;insert/updatefall back to the existing full-rebuild path when the page's contiguous gap is too small, which also defragments the page as a side effect.
- (#338, hash-based aggregation for unindexed GROUP BY, investigated and
closed as not applicable — stock sqlite3 has no hash-aggregation
strategy either; it always sorts via a temporary B-tree for the
unindexed case, so our existing sort-then-group codegen already
matches oracle's algorithm choice. Left open as an
enhancement— optional follow-up if profiling ever justifies closing the constant-factor gap independently of the algorithm.)
- aggregate functions (
count/sum/avg/min/max) combined with a JOIN —SELECT a.name, count(*) FROM a JOIN b ON ... GROUP BY a.namepreviously failed withunsupported: aggregate function count, despite this exact combination being the V4 epic's (#234) stated acceptance gate. Generalizescompile_grouped_scan's sort-then-group codegen shape to a joinedScope, the same way #250 generalizedORDER BY/DISTINCT. Bounded MVP:GROUP BYterms/aggregate arguments must be bare columns, result columns must be*/table.*/a bare column/a whole aggregate call, andHAVINGcombined with a JOIN stays unsupported. Fixestests/tiers/tier3.rs::t3_multi_table_joins_and_aggregates(previously routed around the gap instead of testing it) and activatestests/parity/v04.rs(#333, #335).
-
tests/sqllogictest/runner.rsfailed to compile (&FromClausehas nonamefield — a stale reference left over from #276'sTableRef/FromClauserefactor, invisible tocargo build --all-targets/make lintsincesqllogictestis atest = falsetarget neither reaches). Fixed by reusingcodegen::resolve_from_table_schema(already used elsewhere for the same lookup) againstfrom.first, skipping multi-tableFROM(out-of-slice for V2, per this module's own doc comment) instead of silently mis-resolving it. Also caught two other pre-existing gaps in the same blind spot: an unhandledCodegenErrorvariant (again invisible to normalcargo build) and three clippy violations intests/sqllogictest/{runner,format}.rs(make lintdoesn't reachtest = falsetargets at all — filed as #299 to close that gap properly). Once compiling, the slice's actual coverage jumped from 349→1000 passing queries (15%→43% of the vendored corpus) — the committedtools/sqllogictest-status.jsonbaseline had gone stale while this runner was broken, silently, since CI's ownsqllogicteststep iscontinue-on-error: true(informational, not a gate). -
LIMIT 0returned every matching row instead of none. Every scan shape's LIMIT counter (src/codegen/select/limit_scan.rs'semit_limit_guard, reused by joins, aggregates, and theSeekRowidfast path) checkedDecrJumpZeroafter emitting a row, so aLIMIT 0counter — starting at exactly0— could never stop anything before the first row already leaked through. Restructured as a check-before-act guard (mirroringemit_offset_guard's existing shape):IfNotZerogates whether a row is emitted at all, decrementing only while positive, so a negativeLIMIT(SQLite's "no limit" convention) still falls through unbounded exactly as before. Caught while benchmarking #129, unrelated to that ticket's sorter change. -
EphemeralTable
Insertdecode uses the database's real text encoding (#266).src/vdbe/cursor.rs's subquery-in-FROM materialization path hardcodedTextEncoding::Utf8instead ofdb.header.text_encodinglike every other decode site in the file; a UTF-16 database queried with a subquery in FROM would misdecode text or surface a genericMalformedInstruction. Falls back to UTF-8 only when no db is attached (pure in-memory ephemeral use, e.g. DISTINCT). Regression test added directly against the opcode handler, since building a real UTF-16 fixture through the SQL engine isn't possible yet —MakeRecord(src/vdbe/result.rs) still hardcodes UTF-8 on the encode side, a separate, wider-scope gap left for a follow-up ticket. -
correlated subquery inside a
FROM-subquery's ownSELECTlist (#289, #311).materialize_from_subquery's single-table (non-join) path compiled the subquery'sSELECTlist against aScopecatalog limited to its own resolvedFROMschema(s) instead of the full outer catalog, so any nested subquery referencing another table hit a catalog-visibility rejection — the already-correct joined-FROM path was unaffected. Spend: matched estimate (medium). -
hoist uncorrelated
WHERE-clause subquery out of the outer scan loop (#306, #315). A scalar or single-columnIN (SELECT ...)subquery in a single-tableWHEREclause was re-materialized on every outer row even when uncorrelated — severe enough to hit the 50M-step VDBE guard rail on large tables (#301's bench run). Adds a static, conservative correlation check (subquery_is_correlated/walk_expr_for_correlation— anything uncertain is treated as correlated, which only ever suppresses the optimization) plus a hoist pass: an uncorrelated top-levelWHEREconjunct materializes once, before the scan'sRewind, via a new pointer-identity-keyedScope::hoistedmap, instead of inline per row. Deliberately narrow — only an exactexpr IN (SELECT ...)or scalar-subquery comparison conjunct is recognized;OR/NOT/deeper nesting, multi-columnIN, correlated subqueries, and the joined-queryWHEREpath all fall through unchanged. (A follow-up commit fixed anunreachable!-adjacentmake mvl-limitgate violation the hoist's correlation-walk helper introduced.) Spend: roughly matched the ~200k token estimate.
-
make lintnow covers[[test]] test = falsetargets (corpus/parity/sqllogictest/point_lookup_perf) — the same convention that opts them out of the defaultcargo testrun also opted them out ofcargo clippy --tests/cargo fmt, letting compile errors and lint violations accumulate invisibly (a staleFromClausefield reference insqllogictest/runner.rswas one such compile error, fixed separately). Fixed the 23 accumulated violations this uncovered:&PathBufparameters narrowed to&Path(9 sites acrosstests/corpus/, 1 intests/performance/point_lookup.rs), twoindexing_slicingsites inpoint_lookup.rsreplaced with.get()/destructuring, atype_complexityviolation intests/parity/{driver,v02}.rsfactored into aQueryRunnertype alias, and three violations intests/sqllogictest/{runner,format}.rs(manual_split_once,unnecessary_sort_by,enum_variant_names).make lintnow runscargo clippyagainst these four targets explicitly (named, not a wildcard — matching how--testsitself isn't one either). -
cap ephemeral table/index materialization at 1M rows (#269).
EphemeralTableState.rows/EphemeralState.entries(src/vdbe/cursor.rs) backed a plain in-memoryVec/BTreeMapwith no ceiling — a subquery-in-FROM (#257) or a correlatedIN (SELECT ...)rebuilding its ephemeral index per outer row (compile_in_subquery) could grow memory without limit. AddsExecError::EphemeralRowLimitExceededand aMAX_EPHEMERAL_ROWSconstant checked at both insert sites, following the existing hardcoded-limit pattern (MAX_REGISTERS,MAX_STEPS) rather than a new configurable-limits mechanism. -
assert an aggregate in tier3's joins-and-aggregates stub (#267).
t3_multi_table_joins_and_aggregatesclaimed aggregate coverage by name and ignore-reason but exercised only JOIN + ORDER BY/DISTINCT/INSERT ... SELECT. Added aGROUP BY+count(*)assertion; aggregate functions combined with a JOIN aren't supported by codegen yet, so it runs against a single table rather than the joined query — tracked as a known coverage gap in #268. -
consolidate aggregate codegen onto
AggStep/AggFinal(#263, ADR-0019).src/codegen/select/aggregate.rs'sGROUP BY/plain-aggregate compilation (compile_grouped_scan) now emitsOpcode::AggStep/Opcode::AggFinal— implemented since #241/#242 but never actually emitted by codegen (ADR-0018 tracked this gap) — instead of theAggKind/AggSlothand-rolled register-arithmetic scheme (reset_agg/accumulate_agg), now retired. Surfaced two VM-side gaps along the way:AggStep'smin/maxcomparisons were hardcoded toBINARYcollation (the same class of bug #265 just fixed in the old scheme — fixed here via a newP4::AggFunc{name, arity, collation}descriptor), and there was no way to reset an aggregate-context slot for a newGROUP BYgroup reusing the same slot number (fixed viaAggStep's previously-unusedP5operand as a reset flag). See ADR-0019 for the full design and rejected alternatives (a dedicated reset opcode, threading comparison affinity through as well, folding in plain non-GROUP BYaggregate support). -
aggregate/join/subquery edge-case coverage from the v0.13.0 review (#268). Adds 11 tests across
tests/codegen/select_test.rs,tests/corpus/union_test.rs,tests/corpus/join_test.rs, andtests/corpus/subquery_test.rs: HAVING-filters-all-groups, UNION ALL arm type/affinity mismatch (no coercion, verified), the LEFT/RIGHT/ FULL JOINWHERE ... IS NULLanti-join idiom, two-level-deep correlated subqueries, and multi-columnIN/NOT INsubquery edge cases (zero-row result, NULL tuple component) — all against working functionality. Two sub-items turned out to be missing features, not test gaps, and are documented as cleanUnsupportedrejections rather than fixed here: aggregates with noGROUP BYat all (evencount(*)), andFULL JOINcombined withORDER BY/DISTINCT/LIMIT; also newly discovered, a correlated subquery nested inside a FROM-subquery's own SELECT list. Tracked as follow-on tickets rather than expanding this test-only ticket's scope. -
tighten
src/btreeandsrc/codegenmodule layout (#273, #276). Pure module reorganization, no behavior change:src/btree/groups table b-tree write ops (insert/delete) undertable.rs+table/, and index write ops (index_insert/index_delete) underindex.rs+index/, giving both write paths symmetric naming;ddl.rsrenamed toschema.rs.src/codegen/groups DDL codegen (create_table/drop_table/create_index/drop_index) underddl.rs+ddl/;select.rs(4033 lines) split into a facade plus 9 sub-modules underselect/(entry,joins,join_full,eqp,join_access,order_by,projection,limit_scan,aggregate), each ~1000 lines or fewer. Spec 006Implementation:/Tests:path citations updated for the moved btree files. -
split
src/bin/sqlite-rs.rsinto modules, move dispatch into the library (#292).src/codegen/dispatch.rs'scompile_statement/leading_keywordsnow return a libraryDispatchErrorinstead of anExitCode, so they're usable without depending on the binary crate; the CLI itself splits into per-command modules (src/bin/sqlite-rs/{main,dump,tables,query,exec,common}.rs) behind a thinmain.rsdispatcher. No behavior change. Spend: matched estimate (small). -
dedupe join/subquery codegen (#270, #308). Four independent, behavior-preserving extractions:
compile_join_level_traverse(src/codegen/select/joins.rs) factors the shared nested-loop/ outer-join traversal out ofcompile_join_levelandcompile_join_level_for_sort(#250'sORDER BY+JOIN sorted path), parameterized over row emission via aleafclosure;compile_in_subquerybecomes a thin one-element wrapper aroundcompile_in_subquery_multi; NATURAL/USING join-constraint synthesis is unified into oneresolve_join_constrainthelper shared bycompile_select_joined_scanandcompile_full_join_two_table. Latent-bug fix surfaced along the way:compile_join_level_for_sorthad silently diverged fromcompile_join_leveland never gained #243's seek optimization — a joined query withORDER BYdowngraded an otherwise-seekableSeekRowid/SeekIndexEqpoint lookup to a fullRewind/Nextscan. Both paths now share one traversal, so the optimization applies unconditionally; verified via a newEXPLAIN QUERY PLANregression test. -
design note + benchmark for correlated-subquery rematerialization cost (#303, ADR-0021). ADR-0021 documents deferring a full coroutine rewrite in favor of a scoped follow-up (memoize a correlated subquery's result keyed on its outer-referenced value(s), reusing #306's correlation walk). Adds a
correlated_subquerybench scenario totests/performance/engine.rsdemonstrating the current per-outer-row re-materialization cost, guarded tobench_1mb.dbonly — it blows the VDBE step cap againstbench_50mb.db, itself evidence of the unbounded cost. No fix in this ticket, per its own acceptance criteria. -
add V4 join/aggregate/subquery scenarios to the tier-1 engine bench (#301). Extends
tests/performance/engine.rswithjoin,group_by_agg, andsubqueryscenarios now that V4 phase 1 landed (#235), plus a fixed-sizebench_lookupdimension table andbench_data.bucketcolumn ingen_fixtures.sh --bench.compile_oursnow dispatches single-table vsJOINthe same waysrc/bin/sqlite-rs/query.rsdoes. First measured ratios (bench-status.json) surfaced two follow-ups, filed rather than fixed here per this ticket's scope: an uncorrelated subquery re-executed (and its ephemeral index rebuilt, forIN) on every outer row instead of once (#306), andjoin/GROUP BYratios (14-26x) exceeding #111's 1.5-3x calibration (#310). Spend: roughly matched the 120k token estimate. -
ADR-0022 — profile #317's join ratio, find the missing page cache. #317's own scope was profiling, not fixing. Instrumenting (rather than accepting the provisional "per-row VDBE dispatch overhead" hypothesis from #310/#317) found the real cause:
Pager/VfsPageSourcehave no page cache at all — everyread_pagedoes a fresh syscall + allocation, unconditionally, andjoin's per-rowSeekRowidre-descends the b-tree from the root on every outer row, re-reading the same root/interior pages hundreds of thousands of times on the 830k-row fixture. Ruled out the dispatch-overhead hypothesis via direct instruction-count comparison:full_scan's per-row body has more instructions thanjoin's yet the better ratio (~3.4x vs ~14-16x). No code fix in this ticket (matches #317's own acceptance criteria); filed #320 as the scoped, fully-designed follow-up ("add a bounded page cache toPager's read path"). Spend: roughly matched the profiling-scope estimate.
-
aggregates with no
GROUP BY(#287, #313).SELECT count(*)/sum/ avg/min/max FROM t;(noGROUP BY) now compiles and executes on both populated and empty tables, as a thin extension of the existing sort-then-group machinery (compile_grouped_scan): every row belongs to one synthetic implicit group, and a newimplicit_group: boolparameter ensures a zero-row table still flushes exactly one result row (count(*) = 0, other aggregatesNULL) rather than zero rows.HAVINGwithoutGROUP BYis now accepted too, filtering that single implicit group..openspec/grammar/sqlite.ebnfcorrected:HAVINGis parse.y's own independenthaving_opt, not nested undergroupby_opt.total/group_concatremain unsupported (noAggStateaccumulator yet); the joined-select path is untouched. Spend: matched estimate (medium). -
aggregate function inside a scalar/correlated subquery (#304, #318).
SELECT (SELECT max(x) FROM t) FROM t LIMIT 1(and the correlated form) previously failed with "unsupported: aggregate function max" — a scalar subquery's projected expression compiled throughcompile_value's plain, aggregate-rejecting path instead of #287's aggregate machinery.compile_scalar_subquerynow detects an aggregate call in the subquery's projection and routes throughcompile_grouped_scanas an implicit whole-table group, capturing the result via aCopyinto the destination register;compile_grouped_scangained anouter_scope: Option<&Scope>parameter so a correlated subquery'sWHEREclause still resolves against the enclosing scope. Bug fix found along the way:AggFinalnever cleared itsagg_contextsslot after finalizing — invisible for a top-level query (compiled once), but a correlated aggregate subquery reuses the same slot per outer row, so a zero-row invocation finalized against the previous row's leftover accumulator instead of a fresh NULL/0 (Vm::clear_agg_contextadded). Spend: within estimate (medium). -
.tables [PATTERN]shell parity for thesqlite-rs tablesCLI subcommand (#177). Lists tables and views fromsqlite_master(a newread_table_and_view_namesschema reader,src/schema/ddl_reader.rs, bypassesread_schema's DDL parsing entirely since.tablesneeds neither), excludes internalsqlite_%names, accepts an optional LIKEPATTERNargument (reusingvdbe::like_match), and renders insqlite3's multi-column, space-padded.tableslayout — verified byte-for-byte against the pinned 3.53.4 oracle.temp.-prefixed temp tables remain deferred (needs the V3+ write path's temp-database support). -
ORDER BYandDISTINCTcombined withFULL JOIN(#288, #307). Extendscompile_full_join_two_table's two-pass emitter:DISTINCTthreads adistinct_cursorthrough all three emission sites (matched, left-nulled, right-unmatched), reusing the existing ephemeral-index dedup guard;ORDER BYroutes all three through a newemit_full_join_sort_row, buffering into a sorter cursor (mirroring the ordinary join tree'scompile_joined_sorted_scansplit) with a fourth pass draining the sorter and applyingLIMIT/OFFSETpost-sort.DISTINCT+ORDER BYtogether remains rejected, matching the ordinary join tree's existing restriction. Spend: ~2x estimate — needed sorter-buffering plumbing in the two-pass emitter rather than a config flag.
-
index-ordered scan for
ORDER BY ... LIMIT(#296, #309, ADR-0020).find_ordering_index(src/codegen/select/index_scan.rs) looks for a single index on the FROM table whose column order is a prefix match (forward or exactly-reversed) for the requestedORDER BYterms —BINARYcollation only, and an explicitNULLS FIRST/LASTmust agree with the direction's default. When found,try_compile_index_ordered_scanwalks the index directly (newIdxRewind/IdxLast/IdxNext/IdxPrevopcodes +IdxRowid+SeekRowid) withLIMIT/OFFSETas an early-exit guard — no buffering, no sorter — ahead of the existingcompile_sorted_scanfallback.IndexCursorgainedlast()/prev(), the mirror of its existingfirst()/next(). -
bounded top-K sorter for
ORDER BY ... LIMIT N(#129). The ephemeral sorter (src/vdbe/sorter.rs) previously buffered and sorted every matching row beforeLIMITever applied — a fullO(N log N)sort regardless of how smallLIMITwas.SorterOpennow accepts an optional bound register (P2/P5, wired fromsrc/codegen/select/limit_scan.rs::compile_sorted_scanvia the existing-but-previously-unusedOffsetLimitopcode'sLIMIT + max(OFFSET, 0)/-1-means-unbounded convention), andSorterInsertmaintains a binary max-heap capped at that bound instead of an ever-growing buffer — O(log bound) per insert, provably lossless (a row that loses the eviction comparison can never land within the finalLIMIToutput). Skipped wheneverDISTINCTis present (it dedupes after the sort, so bounding beforehand could evict a row DISTINCT would have kept). ~40% faster on the tier-1 benchmark'sorder_by_limitcase; a linear (non-heap) worst-row scan was tried first and regressed performance wheneverboundexceedslog2(row count)— a genuine dead end worth noting for anyone revisiting this. Index-ordered scanning (skip the sorter entirely when an index matches theORDER BYcolumn) is a separate, larger follow-up — see #296. -
index-ordered scan for
GROUP BY(#310, #316).compile_grouped_scanalways buffered the wholeWHERE-matching table into a sorter before aggregating, even when theGROUP BYcolumns already had a covering index producing rows in the right order (#301's bench found 14-26x tier-1 ratios ongroup_by_agg).try_compile_index_ordered_group_bymirrors #296'sORDER BYMVP: walks a matching index directly (IdxRewind/IdxNextorIdxLast/IdxPrev+IdxRowid+SeekRowid), feeding the same boundary-detection/accumulate/flush logiccompile_grouped_scan's pass 2 already has — no sorter, noMakeRecord, no buffering. Guardrails mirror #296's own MVP: noWHEREclause, an ordinary rowid table, everyGROUP BYterm a bare column. (A follow-up commit precomputedgroup_col_indicesup front to satisfy amake mvl-limitgate the per-row loop'sunreachable!()branch had violated.) Closes thegroup_by_agghalf of #310 only — thejoinper-row dispatch-overhead half is split into #317. Spend: roughly matched the ~150k token estimate.
-
zero-arity scalar functions + FROM-less SELECT (#136, #260, V4 phase 1 epic #235). Registers
sqlite_version()as SQLite's real zero-arity scalar function, exercising codegen's previously-untestedFunctionCallzero-arg branch through a real compiled query.compile_select_no_from(src/codegen/select.rs) adds FROM-lessSELECT <expr>[, ...]support — the normal way built-ins likesqlite_version()are called (SELECT sqlite_version();) — compiling the column list once against an empty schema and emitting exactly one row with no cursor/scan bracketing;*/tbl.*and any clause presuming a table (WHERE/GROUP BY/HAVING/ORDER BY/LIMIT/DISTINCT/ compound) is rejected as unsupported. Wired into thesqlite-rsCLI'squerysubcommand too, which previously hard-refused any FROM-lessSELECTbefore ever reaching codegen. -
UPDATE/DELETEsubquery catalog threading + multi-columnIN(#251, V4 phase 1 epic #235):compile_update/compile_deletegained_with_catalogvariants (mirroringcompile_select_with_catalog's shape) so a subquery in aSETvalue orWHEREclause that references a table other than the statement's own target now resolves instead of failing at codegen time with an empty catalog. Also lands multi-columnIN((a, b) IN (SELECT x, y FROM t)): a newExprKind::InSubqueryMulti, parsed via a token-scan-gated speculative lookahead (so the tuple-vs-grouping-paren ambiguity doesn't regress parser performance on deeply nested plain expressions), and codegen generalizing the existing single-columnIN's ephemeral-index machinery to an N-column key.ANY/ALL/SOMEquantified comparisons, originally also scoped into #251, were dropped entirely — verified against the pinned oracle that SQLite has never implemented that syntax (Postgres/MySQL/standard-SQL only); subqueries inFROMsplit off to a follow-up (#257). -
JOIN: remaining forms (#250, V4 phase 1 epic #235), closing out what #237 deferred. Parser:
NATURALjoins,RIGHT/FULL [OUTER] JOIN,USING (col, ...), and comma-styleFROM a, b(parsed as CROSS-join sugar, which needed no codegen work — it already compiles through #237's CROSS JOIN path). Codegen:NATURAL/USINGsynthesize theON-equivalent equality constraint from schema column names and de-duplicate the shared column inSELECT *output;RIGHT JOINreorders the execution loop nesting so the right-hand table becomes outer (A RIGHT JOIN B == B LEFT JOIN A), generalizing theLEFT JOINmatched/null-extension machinery;FULL JOINadds a second ephemeral-index-tracked pass (same mechanism asDISTINCT) for right-side-unmatched rows.ORDER BY,DISTINCT, andINSERT ... SELECTare now all generalized to work with a JOIN in theFROMclause (previously rejected outright). Deliberate, narrower- than-full-generality scope, each returning a cleanUnsupportederror rather than a silently wrong result: only oneRIGHT JOINperFROMclause;FULL JOINrestricted to a single two-table case; a computedSELECT-list expression combined with a joinedORDER BY; andDISTINCT+ORDER BYcombined with a JOIN.tests/tiers/tier3.rs'st3_multi_table_joins_and_aggregates(the tier-contract acceptance gate for #250) is un-ignored. Spend: ran well past the ticket's "Medium" estimate onceRIGHT/FULL JOIN's loop-reordering and two-pass tracking turned out to need real architectural generalization rather than a local tweak. -
Planner: join-level WHERE/
ONequality index selection (#243, V4 phase 1 epic #235). An inner join table'sONequality against the outer table's rowid, or against aUNIQUEsingle-column index, now compiles to aSeekRowid/newSeekIndexEq+IdxRowid+SeekRowidpoint lookup instead of an unconditionalRewind/Nextfull scan (choose_join_access,src/codegen/select.rs) —LEFT JOIN's null-extension is unaffected. Two new VDBE opcodes (SeekIndexEq,IdxRowid) and a new real secondary-index read cursor (CursorSlot::IndexRead,OpenReadwithP5nonzero) back the index-seek path; non-unique indexes and compound (AND)ONconditions still fall back to a full scan (deliberately narrow, mirroring #137'stry_compile_rowid_seek).EXPLAIN QUERY PLAN(pulled forward from its original V7 grammar slot — see.openspec/grammar/sqlite.ebnf'sexplain-stmt, V4 now) reportsSCAN/SEARCH ... USING ...per table so the planner's choice is observable from the CLI (query "EXPLAIN QUERY PLAN <select>"); bareEXPLAIN(opcode dump) is unchanged, still served by-explain. Spend: ~2x the ticket's original estimate, because scoping surfaced three pieces of missing infrastructure (index-seek opcode, EXPLAIN QUERY PLAN parsing, and the V7→V4 grammar pull-forward) beyond the issue's original "basic WHERE analysis" framing. -
Subqueries in
FROM(#257, V4 phase 1 epic #235, split off from #251). Parser:table-refgains a"(" select-stmt ")" AS identifieralternative (TableRefis nowName/Subquery-shaped in the AST). Codegen: aFROM-subquery materializes into a new VDBE table-mode ephemeral cursor (OpenEphemeralwithP5nonzero —Rewind/Next/Column/Insert/Rowidnow work against an in-memory row list with assigned rowids, alongside the existing index-mode ephemeral cursor DISTINCT already used), bound intoScopevia a syntheticTableSchemaderived from the subquery's own projected columns — then scanned like any real table. Works standalone, as one slot of a joined outerFROM, and when the subquery's ownFROMitself has aJOIN.ANY/ALL/SOMEremain out of scope (never implemented by the pinned oracle). Spend: ran past the issue's own "Medium-Large" estimate — the ephemeral table-mode cursor (Rewind/Next/Insert/Rowid over an in-memory row list) didn't exist yet and had to be added to the VDBE engine, beyond the issue's parser/codegen framing. -
UNION ALLcompoundSELECT(#240, V4 phase 1 epic #235): parser chainsSELECT ... UNION ALL SELECT ...arms intoSelect::compound, withORDER BY/LIMITbinding to the whole compound statement rather than any one arm. Codegen emits each arm's scan/ResultRowblock back to back with per-arm cursor numbers (ScanCursors::for_arm), concatenating with no deduplication and no shared sort/merge step. A column-count mismatch between arms is rejected at compile time. PlainUNION(dedup)/INTERSECT/EXCEPT, joins/subqueries within an arm, andORDER BY/LIMITon the compound statement remain out of scope (deferred to V4 phase 2 or later). -
VDBE
AggStep/AggFinalopcodes (#241/#242, V4 phase 1 epic #235): acount/sum/avg/min/maxaccumulator registry dispatched by a"name(arity)"P4 descriptor, mirroring the existingFunctionopcode's registry-dispatch shape, plus a per-slot aggregate-context table onVmaddressed the same waycursorsis.avgmirrorssum's integer/real promotion and always finalizes REAL (or NULL on zero non-null rows);min/maxcompare viavdbe::compare::compareunder SQLite's type-ordering rules (NULL < INTEGER/REAL < TEXT < BLOB), skipping NULL args likecount(x). Not wired into GROUP BY codegen — #239 (merged first) took a different, opcode-free approach forcount/sum/avg/min/max(reusing existing arithmetic/compare opcodes), soAggStep/AggFinalcurrently have no caller insrc/codegen/; they stand as tested, spec-backed (spec 009 Requirement
- VM primitives for future use.
GROUP BY/HAVING(#239, V4 phase 1 epic #235): parser acceptsGROUP BY(single/multi-column, arbitrary expressions) andHAVING. Codegen groups via the existingSorter*opcode machinery (sort-then-group, mirroring SQLite's ownselect.cshape) and accumulatescount/sum/avg/min/maxper group from existing arithmetic/compare opcodes rather than new dedicatedAggStep/AggFinalopcodes.HAVINGand aggregate result columns compile against a synthetic per-group record via AST substitution of aggregate calls into synthetic column references, reusingcompile_row_values/compile_condunchanged.GROUP BY/HAVINGcombined withORDER BY/DISTINCTin the sameSELECT, and aggregates beyondcount/sum/avg/min/max, are out of scope for this ticket.
- two computed result columns collide (#141).
Copy(r[P2] = r[P1]) harvested from the pinned oracle (SELECT count(*), sum(price) FROM products, alongsideAggStep/AggFinalriding the same harvest — see ADR-0018) closes the gapOpcode::Copywas already hand-added for during #208 but never wired intocompile_row_values's contiguity check.compile_row_values(src/codegen/select.rs) now computes each result column first, and only reserves a fresh contiguous run +Copys into it when the columns didn't land contiguously on their own — no more outright rejection of e.g.SELECT i + 1, i - 1 FROM torSELECT coalesce(i, -1), ifnull(s, 'z') FROM t.emit_branch_into(src/codegen/expr.rs) now accepts arbitrary CASE branch expressions the same way, andFunctionCallargument compilation gained the identical reserve-and-copy fallback for the same underlying contiguity check under a different name.
- Pre-tag
/reviewof the full v0.12.3..v0.13.0 diff (epic #235) found no tag-blocking issues; follow-ups filed as #265–#271 (MIN/MAX collation, subquery-in-FROM text-encoding, tier3 aggregate-stub gap, edge-case test coverage, ephemeral-materialization sizing, join/subquery codegen dedupe, scope-gap tracking confirmation).
-
ORDER BYof a rowid-alias column crashed ("Rowid: cursor slot 2 is a pseudo cursor, not a table cursor") —emit_column_readalways emittedOpcode::Rowidfor the rowid-alias column, valid only against a real table cursor, butORDER BY's second pass re-reads each row from a materializedOpenPseudocursor. Fixed in both call paths that hit it (compile_row_values'sColumnandExprarms), by reading the already-resolved rowid value back viaOpcode::Columninstead when the cursor is the post-sort pseudo cursor. Found via a new full-lifecycle regression test (tests/corpus/cli_write_test.rs) exercising the CLI end to end: schema -> insert -> update -> delete -> select -> export. -
Also fixed a genuine compile break in
tests/sqllogictest/runner.rs(non-exhaustiveCodegenErrormatch missing theRowShapeMismatchvariant #195 added) that meantmake sqllogictestnever actually built — fixing it letselect1.testrun for the first time.
- Assurance tooling:
make mutants(cargo-mutants scoped tosrc/{record,btree,vdbe}/*.rs, reporting totarget/mutants.out) andmake verify(coverage-gate+deny+mvl-limit+mod-fileschained, recording the passing commit totarget/verify.json) are now wired intotools/assurance.py's Evidence/Verification sections — mutation score and a commits-since-last-verify staleness signal, same "read the cache, never run it yourself" discipline as line coverage.
- V03 write-path parity mirror (#72):
tests/parity/v03.rs's stub replaced with real cases — INSERT/UPDATE/DELETE,INSERT ... SELECT,ON CONFLICT IGNORE/REPLACE,CREATE/DROP TABLE/INDEX— driven through thesqlite-rs execCLI and diffed against the pinnedsqlite3oracle across the acceptance/output/schema dimensions.make assurance's Parity line movesV03+ pending->V03 3/4. Along the way, confirmed a known limitation (from #207) applies more broadly than documented: an inline column-levelUNIQUEconstraint inCREATE TABLEcreates no backing index either (not just a composite table-level constraint) —compile_create_tablenever auto-creates one, so UNIQUE enforcement only fires via an explicitCREATE UNIQUE INDEX. Filed as a follow-up, not fixed here.
sqlite-rs --version/-V: reportsCARGO_PKG_VERSIONand exits 0 — the CLI previously had no way to report its own version.
-
UNIQUE constraints on non-rowid columns (#207, split out of #195): new
Opcode::NoConflictreal-index seek+branch primitive (src/vdbe/cursor.rs, built onIndexCursor::seek) fills the gapCursorSlothad no read-capable real-index variant —compile_insertnow probes everyUNIQUEindex before writing a row and dispatchesON CONFLICT(IGNORE/REPLACE/ABORT+FAIL+ROLLBACK) the same way the existing rowid-PK conflict check does. A compositePRIMARY KEY(...)/UNIQUE(...)table constraint with no backing on-disk index still isn't enforced (this codebase doesn't auto-createsqlite_autoindex_*entries yet) — aCREATE TABLE-side gap, not an INSERT-codegen one. -
INSERT ... SELECTcodegen (#208, split out of #195):compile_insertnow drivesselect.rs's scan/filter/project/ORDER BY/DISTINCT/LIMIT machinery (compile_select_scan, factored out ofcompile_select) with a pluggable per-row sink, feeding each projected row into the same per-row constraint-check/write path (compile_row) a literalVALUESrow uses — full parity with plainSELECT, not just scan+WHERE.select.rs's scan cursor numbers are now parameterized (ScanCursors) so the embedded scan never collides with the INSERT's own target-table/index cursors. NewOpcode::Copy(register-to- register, #208) re-materializes a SELECT-scan register into the fresh, contiguous registerMakeRecordneeds once reordered/subset into the target table's schema-column order — mirrorscompile_value's own "always allocate anew" contract. Also fixes a real (pre-existing, found via this ticket's own testing) bug inapply_affinity: TEXT affinity never converted a NUMERIC value to its text rendering, leaving e.g.INSERT INTO t(b) VALUES (1)(b TEXT) storing a raw integer under a TEXT-affinity column —PRAGMA integrity_checkcorrectly flagged this asNUMERIC value in t.b.
- V3 exit gate (#217), closing epic #161: write-path CLI surface
(#215); corpus
PRAGMA integrity_checkcross-validation centralized into a singleassert_integrity_check_okoracle helper, replacing per-file duplicates across b-tree insert/delete, index maintenance, pager flush, and CLI write-path tests (#216). NewexecCLI subcommand wiresINSERT/UPDATE/DELETEthrough existing codegen, plus newCREATE TABLE/DROP TABLE/CREATE INDEX/DROP INDEXcodegen (none existed before #215). Tier 2 (WRITE CORE) stubs flip to real tests:t2_crud_round_trips_on_rowid_tables(CREATE/INSERT/ UPDATE/DELETE round-trip via the CLI) andt2_written_file_passes_integrity_check(stocksqlite3integrity_check-clean on a written file), bringingtests/tiers/tier2.rsto 4/4 active. Scopedcargo-mutantsrun against the V3 write-path modules (b-tree insert/delete/index maintenance, VDBE write-opcode dispatch) as a sanity check ahead of release; a full-crate mutation run remains out of scope for this phase (scoped as a V1 exit-gate deliverable, epic #5).
- Fixes from a phase-level
/reviewof V3 phase 3 (#161), found only by looking at #195/#210/#196 together:INSERT OR REPLACEleft stale secondary-index entries for the row it displaced (#218);UPDATEnever re-validated NOT NULL/CHECK constraints, letting an invalid value propagate into secondary indexes too (#220);INSERTnever wiredAUTOINCREMENTintoNewRowid's opt-in mechanism, so anAUTOINCREMENTtable silently reused rowids after deletion (#221). Adds integration-level regression coverage: an INSERT→UPDATE→DELETE lifecycle test against the same indexed table, and a test pinning today's non-enforcing UNIQUE-index behavior (tracked separately, #207).
- V3 phase 3 complete (#161): write codegen + VDBE. Auto-index
maintenance on write (#196) —
INSERT/DELETE/UPDATEcodegen now opens a write cursor per index and emitsIdxInsert/IdxDeletepairs per row, keeping secondary indexes in sync with table data.DESCindex columns and invalid/untrustedsqlite_masterroot pages are rejected outright rather than silently mis-keyed or misdirected.
Fixes from /review of V3 phase 2 (#188/#190/#191/#192/#193): a parser
bug and an untrusted-input handling gap, plus minor diagnostics/doc
cleanup.
opt_column_constraint()(DDL column-constraint parsing, #192) silently droppedCONSTRAINT <name>when no recognized constraint keyword followed — e.g.CREATE TABLE t (a INTEGER CONSTRAINT foo)was accepted with the constraint text discarded. Now rejected asInvalid, matchingtable_constraint()'s existing behavior.find_master_rootpage()(src/btree/master.rs, #193) cast ani64rootpage read from asqlite_masterrow directly tou32with no validation — a corrupted or malicious.dbfile could store an out-of-range/negative rootpage and get silently mapped to a different page. Now rejected via a newBtreeError::InvalidRootPage.delete_master_rowfabricatedBtreeError::RowidNotFound { rowid: 0 }on a by-name lookup miss, discarding the actual name. Replaced with a dedicatedBtreeError::MasterEntryNotFound { name }.
- Documented
bump_schema_cookie'swrapping_addas deliberate parity with stock SQLite's own cookie wraparound. - Added a tripwire comment on the
PageSource for &Tblanket impl.
V3 phase 2 (epic #161) complete: write-path parser (INSERT/UPDATE/DELETE,
CREATE/DROP TABLE, CREATE/DROP INDEX) plus schema cookie + sqlite_master
maintenance and AUTOINCREMENT tracking.
-
Schema cookie +
sqlite_master/sqlite_sequencewrite maintenance (#193), V3 phase 2 (epic #161). Newsrc/btree/master.rs:bump_schema_cookiepatches the schema cookie (header bytes 40-43) in place, following the offset-patch precedentpager.rsuses for page-count/freelist fields (#167's documented "no header serializer yet" gap);insert_master_row/delete_master_rowwrite/removesqlite_masterrows for CREATE/DROP TABLE/INDEX via the existinginsert_row/delete_rowb-tree primitives;ensure_sqlite_sequence_tableauto-createssqlite_sequenceon first use andupdate_sequencetracks each table's max rowid (monotonic — never decreases). These are write primitives only; wiring them into actual statement execution is VDBE write-opcode scope (#194). Also adds a blanketimpl<T: PageSource> PageSource for &T(src/vfs/page_source.rs) so aTableCursorcan scan a table through a shared&Pagerreference while the samePageris later borrowed mutably for a write. -
Parser: CREATE/DROP TABLE, CREATE/DROP INDEX (#192), V3 phase 2 (epic #161).
parse_create_table/parse_create_index/parse_drop_table/parse_drop_indexacceptCREATE TABLE [IF NOT EXISTS] name (columns, table-constraints) [WITHOUT ROWID | STRICT],CREATE [UNIQUE] INDEX [IF NOT EXISTS] name ON table (indexed-columns) [WHERE ...](partial index), and the twoDROPforms, mirroring the existing three-way accept/unsupported/invalid outcome contract. Column/table constraints cover NOT NULL, PRIMARY KEY [ASC|DESC] [AUTOINCREMENT], UNIQUE, DEFAULT, CHECK, COLLATE, and namedCONSTRAINTs;REFERENCES/FOREIGN KEYare parsed then reportedUnsupported(deferred to V8). NewCreateTable/ColumnDef/ColumnConstraint/TableConstraint/IndexedColumn/CreateIndex/DropTable/DropIndexAST nodes and printer round-trip support; grammar's V3 DDL stub filled in with real detail (indexed-column,COLLATEconstraint,CONSTRAINTprefix). Verified againsttests/corpus/sql/ddl/*.sql: 464/517 CREATE TABLE and 148/149 CREATE INDEX statements accepted. -
Parser: UPDATE statement (#190), V3 phase 2 (epic #161).
parse_updateacceptsUPDATE [OR REPLACE/IGNORE/ABORT/ROLLBACK/FAIL] table SET col=expr, ... [WHERE ...], including the tuple SET form(col1, col2) = (expr1, expr2)(expanded into oneAssignmentper column; mismatched arity is a syntax error, a subquery RHS is unsupported), mirroringparse_insert/parse_delete's three-way accept/unsupported/invalid outcome contract (spec 002-parser). NewUpdate/AssignmentAST nodes reuse the existing expr parser for SET values and WHERE and the existingConflictActionenum;update-stmtgrammar entry in.openspec/grammar/sqlite.ebnfextended to cover the conflict-action clause and tuple-assignment form. -
Parser: DELETE statement (#191), V3 phase 2 (epic #161).
parse_deleteacceptsDELETE FROM table [WHERE ...](no LIMIT/ORDER BY — deferred), mirroringparse_insert's three-way accept/unsupported/invalid outcome contract (spec 002-parser). NewDeleteAST node reuses the existing expr parser for WHERE, plus printer round-trip support. -
Parser: INSERT statement — VALUES + SELECT forms (#188), V3 phase 2 (epic #161).
parse_insertacceptsINSERT [OR REPLACE/IGNORE/ABORT/ ROLLBACK/FAIL] INTO table [(cols)] (VALUES (...), ... | SELECT ... | DEFAULT VALUES), mirroring the existing SELECT recursive-descent parser and its three-way accept/unsupported/invalid outcome contract (spec 002-parser). NewInsert/InsertSource/ConflictActionAST nodes and printer round-trip support;insert-stmtgrammar entry in.openspec/grammar/sqlite.ebnfextended to cover the conflict-action clause and SELECT-source alternative.
tests/tiers/tier0.rs:t0_wal_pending_rows_visibleandt0_any_feature_bearing_file_dumps_all_rowsboth read directly from the shared, committedtests/corpus/fixtures/journalstates/WAL fixtures, unliket0_hot_journal_recovers_committed_state, which already copies its fixture to a scratch dir first. Since tests in one binary run concurrently, and the pinned-oracle shell-out in the "any feature bearing file" test creates a real-shmfile whensqlite3connects to a WAL db, a sibling thread'sdump_databasecould observe that-shmmid-creation and reject it as too short — seen once on main CI after #191 merged (unrelated to that PR's content, not reproducible locally). Both tests now copy their fixture (and-wal/-journalcompanion) into an isolated temp dir via a newIsolatedFixturehelper, matching the hot-journal test's existing isolation convention. Test-only change, nosrc/impact.
Spend: small.
- Index b-tree delete:
extract_max_entry(src/btree/index_delete.rs) permanently orphaned pages whenever a predecessor swap drained a subtree more than one level deep (interior → interior → leaf) — only the outermost page was ever deallocated, leaving deeper already-empty pages unreachable and never returned to the freelist. Fixed by deallocating a subtree's pages bottom-up as each level confirms it's fully drained; added a depth guard (mirroringdescend_index_tree'sMAX_PAGES_VISITEDconvention) since the recursion previously had none. Found during/reviewof #189 (V3 phase 1 exit gate). - Index b-tree insert:
insert_entry(src/btree/index_insert.rs) allocated overflow pages for a large key's payload before checking for a duplicate key, leaking that overflow chain on every rejected duplicate insert. The duplicate check (both the interior-match and leaf-level cases) now runs before any overflow allocation.
Spend: small — both fixes and their regression tests together were well
under a "small" ticket's budget; found and fixed in the same session as
the /review that surfaced them.
-
V3 phase 1 exit gate (epic #161): the b-tree write path is fully shipped — pager write path + freelist (#166, #167), table and index b-tree insert/delete with page split/merge/collapse (#168, #169, #171), overflow chain write/free (#168, #173), and statement-level rollback journaling (#172). Every file this crate writes is opened and
PRAGMA integrity_check-ed by stocksqlite3; round-trip write→read via this crate's own readers is oracle-identical. Next up: V3 phase 2 (0.10.0), the write-path parser + schema layer (INSERT/UPDATE/DELETE, CREATE/DROP TABLE/INDEX,sqlite_mastermaintenance). -
Index b-tree insert/delete — same ops for index b-trees, including WITHOUT ROWID tables (#171), V3 phase 1 (epic #161). New
src/btree/index_insert.rs::insert_entryandsrc/btree/index_delete.rs::delete_entrymirror the table write path (#168/#169) in shape but not mechanism: index interior cells carry a full entry (not just a routing key), so an index leaf split promotes its median entry into the parent (removing it from both halves, unlike a table leaf split's copy-and-keep divider), and a delete target found at interior level requires a predecessor swap (delete_via_predecessor_swap/extract_max_entry) rather than a plain routing-entry removal, to avoid discarding the live value that entry itself carries.descend_index_tree(shared by both write paths) checks for an exact key match at every interior level while descending, not just at the final leaf — needed because a duplicate or delete target may have been promoted to interior level by an earlier split. Verified against stocksqlite3for single-entry insert, bulk insert of 500 entries (forcing splits), delete-all, WITHOUT ROWID insert/delete, and duplicate-key rejection (spec 006-btree Requirements 15-17). -
Fix (found while implementing the above):
delete.rs's (table, #169) andindex_delete.rs's underflow cascade both had a latent bug where an interior page draining to zero routing entries recursed intocollapse_into_ancestorsas if the page itself had "emptied" — silently orphaning its own still-liverightmostsubtree (the grandparent's handling of "child died" repoints/removes its reference to the collapsing page, with nothing carryingrightmostforward). Both nowsplice_childthe survivingrightmostdirectly into the collapsing page's own slot in its parent instead of cascading further. Not confirmed reachable by the table write path's actual test parameters (an exhaustive single-delete probe over 60 rows found no repro there), but the index write path hit it immediately once interior-level values needed preserving — applying the same correction to both, plus a table-side regression test (deleting_one_subtree_never_orphans_a_sibling_rightmost_subtree). -
Table b-tree delete — cell delete + page merge/rebalance (#169), V3 phase 1 (epic #161). New
src/btree/delete.rs::delete_row: locates a cell by rowid and removes it via the shared page-rebuild helpers (promoted frominsert.rstosrc/btree.rsso both write paths reuse them). Underflow policy is a documented simplification of SQLite's proactive half-full-threshold sibling redistribution: a page collapses into its parent only once completely empty, cascading up the ancestor chain and, if it reaches the root, relocating the sole remaining child's content into the fixed root page (the reverse ofinsert.rs::root_split). Emptied pages are returned to the freelist (#167). Verified against stocksqlite3for single-row delete, delete- all (tree collapses to an empty leaf root), bulk delete of 1000 rows, a collapse across a leaf-split boundary, and an insert→delete→insert round trip that reuses freed pages (spec 006-btree Requirements 12-14). -
Table b-tree insert — cell insert + page split (#168), V3 phase 1 (epic #161). New
src/btree/insert.rs: rowid-ordered leaf cell insert (encoding rowid varint + payload + overflow chain, reusingrecord::encode_record/local_payload_size), leaf split with median-key propagation, cascading interior splits, and root split (including the page-1/sqlite_masterroot special case). Verified against stocksqlite3(PRAGMA integrity_check+select) for no-split, single-split, cascading/root-split, 1000-row bulk insert, and overflow+split scenarios (spec 006-btree Requirements 8-11). -
Statement-level journaling — rollback journal for atomicity (#172), V3 phase 1 (epic #161), DELETE mode only (TRUNCATE/PERSIST deferred).
Pager::flushnow journals the pre-transaction content of every page it's about to overwrite before writing to the main file, syncing the journal first;Pager::openreplays a detected hot journal into the main file (truncating back to its pre-transaction page count) instead of refusing to open. Newsrc/pager/journal.rsmirrors stock SQLite'spager.cheader layout andpager_cksumbyte-for-byte, proven both directions: a realsqlite3-written hot journal recovers through ourPager::open(tests/tiers/tier0.rs), and a journal we write recovers through a realsqlite3(tests/corpus/journal_interop_test.rs). Un-ignorestests/tiers/tier2.rs'st2_statement_atomicityandt2_journal_transactions_commit_and_rollback(spec 007-pager Requirement 6, ADR-0016). Spend: ~2x the initial estimate — recovery correctness (real sqlite3 interop, byte-for-byte checksum format, and making sure recovery tests never mutate checked-in fixtures) took more iteration than the write-path plumbing alone. -
Freelist management — allocate/deallocate pages (#167), V3 phase 1 (epic #161).
Pager::allocate_pagepops a page off the freelist (or extends the file when it's empty);Pager::deallocate_pagepushes a page onto the freelist, appending to the current trunk page's leaf array or chaining a new trunk once it's full. Newsrc/pager/freelist.rs::TrunkPageparses/writes freelist trunk pages, never panicking on a truncated/corrupt trunk. An allocate/deallocate round trip still opens andPRAGMA integrity_checks cleanly in stocksqlite3(spec 007-pager Requirement 5). -
Pager write path — dirty page tracking + flush (#166), V3 phase 1 (epic #161).
Pager::get_page_mut/Pager::flushon top of a newVfs::open_write/VfsFile::write_at/syncsurface (implemented for bothUnixVfsandMemoryVfs).Pagernow holds a single read-write file handle instead of opening a second fd, avoiding the documentedclose()-drops-all-fcntl-locks hazard. A page flushed through the new write path still opens andPRAGMA integrity_checks cleanly in stocksqlite3(spec 007-pager Requirement 4).
- Overflow chain pages leaked on b-tree row delete (#173), V3 phase 1
(epic #161).
delete_row(#169) freed emptied leaf/interior pages but never freed the overflow pages a deleted cell's payload had spilled into (#168) — those pages were orphaned instead of returning to the freelist (#167).src/btree/delete.rsnow reads the removed cell's first overflow pointer and walks/deallocates the whole chain, with the same revisited-page cycle guard the read-sidereassemble_payloaduses. #173 was re-scoped to this narrower gap once investigation found the insert-side overflow-chain write was already delivered by #168.
- V2 exit gate (#97): closes epic #56 — V2, single-table queries (Tier 1
QUERY CORE), is fully shipped across all four phases (tokenizer +
SELECT-core parser, value-semantics kernel + scalar function core,
VDBE interpreter,
sqlite-rs queryCLI). tests/tiers/tier1.rs::t1_select_core_accepts_and_rejects— the last Tier 1 stub, flipped live: accept/reject vectors for the SELECT-core parser's three-wayParseOutcomecontract (spec 002-parser Requirement 4). Tier 1 is now 7/7 active, no ignores.
tools/assurance.py's opcode-completeness scan missed dispatch arms combining multiple opcodes (SorterSort | Sort => ...), undercountingSort/SorterSortas unimplemented. Opcode completeness now correctly reads 64/64 against the frozen inventory (#65/#87) — both opcodes were already dispatched..openspec/specs/001-architecture/spec.md: removed a stale(planned)dead link on Requirement 1 (a real test link already covers the scenario) and repointed Requirement 4's dead link to the actual tier0 test (tests/tiers/tier0.rs::t0_feature_bearing_files_are_raw_row_readable).
- Codegen pattern-matches
WHERE rowid = <int literal>/WHERE rowid = ?(or?NNN) — recognized via therowid/_rowid_/oidkeywords or the table's actualINTEGER PRIMARY KEYalias column — and emitsInteger/Variable+SeekRowiddirectly on the table cursor instead of theRewind/Nextfull-table-scan loop: an O(log n) point lookup instead of an O(n) scan (#137). MakingWHERE rowid = ?actually correct (not just compile) required reopening the frozen V2 opcode set to addVariable(re-harvested from the pinned oracle) plus a minimal bind-parameter API —Vm::bind_params,execute_with_params/execute_with_db_and_params— see.openspec/adr/0015-variable-opcode-reopens-frozen-set.md. tests/performance/point_lookup.rs: a quick, dependency-free wall-clock demonstration of the O(n)→O(log n) fix (make test-point-lookup-perf), plus a smalltests/performance/Makefileto run individual test/bench scenarios standalone.
ORDER BYby a genuine computed expression — unary/binary operators, scalar function calls, and an alias whose own result expression is computed rather than a bare column (#155).compile_sorted_scancomputes each such term into its own register, appended after the raw schema-column block already fed toMakeRecord/SorterInsert; theSorterOpensort-key descriptor is patched in once that layout is known (newEmitter::patch_p4), and the record's span is widened to the register allocator's post-compile watermark (newRegAlloc::peek) so expressions with internal temporaries (e.g.CASE) stay record-contiguous. Closes the gap #144 left open.
- Literal fidelity: REAL/BLOB literals compiled to
String8text instead of typed values, integers outsidei32were a hard codegen error, andCASTmisusedMustBeInt/RealAffinity(aborting instead of truncating, leaving TEXT/BLOB/NUMERIC targets as no-ops) (#142). HarvestsReal,Blob,Int64,Castfrom the pinned oracle and addssrc/vdbe/cast.rs, a kernel module implementing SQLite's realCASTconversion rule (longest-numeric-prefix parsing, saturating truncation, the NUMERIC whole-number downgrade that applies only to text/blob sources). Also fixes a%bug this exposed:checked_remalways returnedInteger, but SQLite promotes toREALwhen either operand isREAL. 24 new oracle-harvested CAST vectors added totests/corpus/expr_vectors/walker.jsonl; supersedes the narrower BLOB-only follow-up filed as #151.
ORDER BYresolves 1-based ordinals (ORDER BY 2) and result-column aliases (ORDER BY x), not just bare table-column references (#144). Aliases take precedence over table columns, matching SQLite.ORDER BY ... COLLATE namenow reads the actual collation instead of always comparing under BINARY. Both resolve to the same underlying table-column index a bare column reference already used, so no sorter/SortKeyColumnchange was needed. Genuine expression sort keys (ORDER BY -i,ORDER BY lower(s)) still refuse — extending the sorter's record payload for computed values is tracked separately (#155).
query-list-mode rendering diverged fromsqlite3 -liston NULL, BLOB, and REAL (#143):query's default output reuseddump's.dump-quote()-style renderer (NULLliteral,X'HEX'blobs) instead of the shell's actual-listrules (empty string for NULL, raw blob bytes, truncated at the first embedded NUL byte since the shell prints via a null-terminated C string). A dedicated byte-basedformat_query_valuerenderer now backsquery's-listbranch;dump/exportare unchanged.- REAL columns storing exactly
0.0/1.0(SQLite's integer-serial-type storage optimization) decoded as a bareIntegerand rendered0instead of0.0for any reader, not justquery—emit_column_readnever applied REAL-affinity coercion on column reads.apply_affinitynow convertsInteger -> Realfor REAL affinity (matching SQLite's documented affinity rule), andemit_column_reademitsRealAffinityafterColumnfor REAL-affinity columns.
- Quote-aware DDL column splitting (#135, follow-up from #131 review):
column_defs/split_top_level_commassplit aCREATE TABLEcolumn list on raw top-level commas with no awareness of string literals, quoted identifiers, or comments, androwid_alias_columnscanned that raw text forPRIMARY KEY/INTEGER— since #131 this drivesemit_column_read'sRowid-vs-Columnchoice, so a mis-split was a silently wrong query result, not just wrongdumpoutput. A new length-preservingmask_quotes_and_commentsblanks out'...',"...",`...`,[...],--..., and/*...*/regions before paren-depth/comma-splitting and keyword scanning, while the returned text still slices the original string.rowid_alias_columnalso now recognizes the table-levelPRIMARY KEY(col)constraint form, previously dropped by the table-constraint filter before it was ever checked.
- Comparison affinity was never applied —
WHERE i = '5',WHERE i > 3,WHERE r = 1.5returned no rows instead of matching the oracle (#138).TableSchemanow captures each column's declared type; codegen derives comparison affinity from both operands (columns/CASTs only, per SQLite's owncomparisonAffinityrule) instead of hardcoding the P4 affinity byte, andcompare_jumpapplies it to operand copies before delegating tocompare(). Spec 009 Req 5 gains a scenario for the affinity half of the P4 descriptor. Known remaining gap, filed separately (#151): BLOB literals still compile to text, soWHERE b = x'41'doesn't match yet.
- Performance regime, first results (#112, epic #111): tier-1
(engine-to-engine, criterion) and tier-2 (CLI-to-CLI, hyperfine) bench
harnesses against the pinned 3.53.4 oracle.
tools/gen_fixtures.sh --benchgenerates ~1MB/~50MB fixtures (pure-SQL, deterministic, not committed);tests/performance/engine.rsruns 6 scenarios per fixture with rusqlite linked to the pinned oracle (not itsbundledfeature, so it can't drift);tools/bench_cli.shcomparessqlite-rs dump/queryagainstsqlite3;tools/bench-status.jsonis the committed first-results table.make bench/make bench-cli/make bench-status/make fixtures-bench. Deliberately not wired into CI —make lintscopes clippy to--lib --bins --tests --examplesrather than--all-targetsso benching stays a manual workflow. Findings: full scan/filter/expr/prepare land in the expected 1.5–6× band; point lookup andORDER BY ... LIMITare 500×–41,000× outliers (full scan instead of a rowid seek / no top-K bound — V4 planner-tuning material, filed as #128/#129, not fixed here). - Phase 4B (#96, epic #56): sqllogictest slice runner —
tests/sqllogictest/parses the sqllogictest record format (statement ok/error,query <types> <sort>,----expected blocks with literal values or theN values hashing to <md5>form,onlyif/skipifengine conditionals) and runs the 14 vendored files (#70) through the same read pipelinesqlite-rs queryuses.statement oksetup replays through the pinned oracle, since this engine has no write path yet. - Skip-not-fail policy per spec 004 Req 4: out-of-slice grammar/opcode gaps
skip, only a genuine result divergence fails.
make sqllogictest+ informational (non-gating) CI step, plus a companion step that reports drift between the committed status file and what a run produces. tools/sqllogictest-status.json: committed pass/skip/suspect/fail counts, reported on the assurance dashboard's Model line as a pass-rate AND coverage pair — currently 199/199 passing over 8.6% of the corpus. Thesuspectbucket counts queries declined for reasons that should not occur against oracle-validated input (malformed-SQL verdict, unreadable schema), so an engine regression there surfaces instead of hiding among the skips.tests/unit/codegen.rs: oracle-free program-shape tests pinning each codegen fix below, so a regression failsmake testrather than only the non-gating slice.tests/codegen/expr_test.rs: two end-to-end regression tests (#125/#133) for the scalar-function contiguity fix below —single_arg_function_call_compilesandmulti_arg_function_call_compiles_with_contiguous_registerscompile real SQL through the full parse → codegen → VDBE path and assert on actual output values, complementing the program-shape tests above.- Two opcodes join the frozen V2 set, taking it from 52 to 54 (#134):
Not(r[P2] = !r[P1], NULL in / NULL out) andNull(writes NULL over the register rangeP2..=P3). Both were harvested, not hand-added —tools/harvest_opcodes.pygained the two oracle queries that emit them (SELECT NOT qty FROM products,SELECT CASE WHEN price > 100 THEN 1 END FROM products), somake opcodesreproduces the inventory. Opcode completeness moves 50/52 → 52/54; both are dispatched on arrival. tests/parity/v02.rs: a three-valued-logic parity dimension overserialtypes/values.db(the fixture that actually has NULL rows) — 14 cases coveringNOTover every comparison, connective,IN,BETWEEN, andIS NULLform, plus value-context cases wrapped inIS NULLso the assertion is about semantics rather than about how each engine spells a null.
Codegen defects the runner and its review surfaced, all affecting
sqlite-rs query output (#95's shipped CLI), not just tests:
-
x NOT IN (...)andx NOT BETWEEN a AND breturned rows for NULL operands. Both were compiled as their positive form with true/false jump targets swapped, which turns SQL's "unknown" into "true"; they now lower the way SQLite does (NOT BETWEENasx < lo OR x > hi,NOT INwith an explicit saw-NULL guard). The genericNOT (...)case this left open is fixed by #134, below. -
Every scalar function call with arguments failed to compile (
function argument registers were not contiguous), making V2's scalar functions unreachable through the compiled query path —SELECT abs(id)included.Function's contiguous argument window was reserved before the arguments were compiled, so they always landed past it; the window is now taken from where the arguments actually land. Slice coverage rose from 7.2% to 8.6% as a result. -
Generic
NOT (...)resolved SQL's "unknown" to true, soWHERE NOT (x = 5)returned rows wherex IS NULL, and the two spellingsNOT (x IN (...))andx NOT IN (...)disagreed (#134).compile_condnow carries a third piece of contract alongside its true/false continuations —NullTarget, which names the one the unknown outcome joins, exactly SQLite'sjumpIfNullflag.NOTswaps the two targets and flips it, leaving unknown on the address it already had;AND/OR/BETWEEN/INthread it through unchanged. The same flip fixesx <> 5, which had the identical bug for the identical reason (<>isEqwith the targets exchanged) and also returned NULL rows. -
Conditions used as values (
SELECT x = 5,SELECT a AND b) answered NULL for every row — they fell intocompile_value's catch-all, which allocates an unwritten register. They now materialize all three outcomes, andSELECT NOT xyields NULL for a NULLxinstead of 1.CASE ... ELSE NULLleaked the previous row's result, since its NULL branch emitted no instruction at all to overwrite the shared destination register. -
SELECT *(andSELECT tbl.*) answered NULL for anINTEGER PRIMARY KEYcolumn. #131 routed the WHERE and named-result- column paths throughemit_column_read, which substitutesRowidfor the record's NULL placeholder, but the star-expansion path incompile_row_valuesemitted its own bareColumnand was missed — soSELECT id FROM twas right whileSELECT * FROM twas wrong. No corpus fixture is a plain table with anINTEGER PRIMARY KEY, which is why the oracle suites could not see it; the new parity case borrows an FTS5 shadow table until the corpus gains one. -
tests/codegen/expr_test.rs's walker-vector reader extracted JSON string fields by splitting on the next", without unescaping. That silently changed the SQL under test:'a%b' LIKE 'a\\%b' ESCAPE '\\'reached the compiler with a two-character escape, which SQLite itself rejects. The resulting failure had been filed againstLike's codegen — the engine agreed with the oracle all along. BothESCAPEvectors leaveKNOWN_GAPS, and the pass ratchet moves 44 -> 46. -
Aggregate calls (
count,sum,avg, ...) compiled as ordinary per-row scalar functions, soSELECT count(*) FROM temitted one row per input row instead of one count. Codegen now rejects them as unsupported — V2 has no grouping pass — since a refusal beats silently wrong output. In slice terms this is a boundary re-label rather than a repair: those queries move from the fail column to the skip column, which is why the metric publishes coverage alongside pass rate. -
A rowid-alias column (
INTEGER PRIMARY KEY) read back as NULL, because it is stored as a placeholder in every record and needs the cursor's rowid substituted.SELECT x FROM t WHERE x=2silently matched nothing.rowid_alias_columnmoved fromsrc/dump.rstosrc/schema/so the compiled read path can share the substitutiondumpalready did. Its detection now also excludesINTEGER PRIMARY KEY DESC, which SQLite deliberately does not treat as a rowid alias; two remaining textual misreads are pinned byknown_fragile_*tests. -
&,|,<<,>>,~, and||all parsed (in-grammar since V2) but silently answered NULL (binary ops fell intocompile_value's catch-all, which emits a bareNull) or passed the operand through unchanged (~,UnaryOp::BitNot) (#139). Harvested the six real SQLite opcodesBitAnd/BitOr/ShiftLeft/ShiftRight/BitNot/Concat(54 -> 60 opcodes), added their INTEGER/TEXT coercion and NULL propagation tosrc/vdbe/coerce.rs, and wired dispatch throughsrc/vdbe/arithmetic.rs. Shift handles SQLite's negative-shift-amount reversal and magnitude-≥64 clamp rules, not just Rust's native<</>>. SixKNOWN_GAPSentries close; the walker-vector pass ratchet moves 46 -> 55.
VDBE execution limit, unrelated to codegen, surfaced by #112's bench:
src/vdbe/exec.rs'sMAX_STEPSinfinite-loop backstop was1_000_000, which a real ~830k-row full-table scan already exceeds (a handful of VDBE steps per row). Raised to50_000_000— still a bounded safety net, now sized for real workloads instead of only small test fixtures.ORDER BY ... NULLS FIRST/LAST(#140) was parsed and stored (ast::OrderingTerm::nulls_last) but never read byresolve_order_by, so an explicit modifier was silently ignored — the sorter always placed NULLs first for ASC / last for DESC regardless of the clause.SortKeyColumnnow carriesnulls_first, derived from the parsed term (defaulting to the prior implicit ASC/DESC-driven placement when no clause is given), and the sorter compares NULL-vs-non-NULL independently of thedescendingreversal. Declared-column collation in ORDER BY remains hardcoded toCollation::Binary— schema has no per-column collation storage yet — tracked as a follow-up, not fixed here.
- Phase 3C (#91, epic #56): the codegen convergence ticket —
src/codegen/(select.rs,expr.rs) compiles a parsedSelectAST into a VDBEProgram: full-table scan (Init -> OpenRead -> Rewind -> ... -> Next -> Halt), WHERE/AND/OR/CASE/BETWEEN/IN/LIKE as jump-based control flow (never an intermediate boolean register, per spec 009 Req 11), ORDER BY via the sorter, LIMIT/OFFSET counters, DISTINCT via the ephemeral index. - Wires the previously-missing
Functionopcode dispatch insrc/vdbe/exec.rs(spec 009 Req 7). src/vdbe/explain.rs: theEXPLAINbytecode printer (spec 009 Req 10).- Flips spec 009 Requirements 7/10/11 from
(planned)to active — all 11 requirements now backed, zero dead links. - Un-ignores
tests/tiers/tier1.rs'st1_single_table_where_matches_oracleandt1_explain_prints_bytecode.
Known scope gaps (documented via KNOWN_GAPS in the test files, and now
erroring loudly rather than silently corrupting, per PR #117 review):
no bitwise/concat opcode in the frozen V2 52-opcode set; CAST's
lossy-conversion semantics beyond affinity coercion; REAL literals
represented as text (no OP_Real-equivalent opcode); integer literals
outside i32 range; CASE branch results other than a bare literal or
column reference (no MOVE opcode); full three-valued NULL propagation
through NOT/AND/OR/BETWEEN/IN in value (non-WHERE) context.
0.7.0 completes V2 phase 3 (#87, #88, #89, #90, #91 all closed) — epic #56's engine phase: VDBE interpreter, cursor/sorter/ephemeral opcodes, and now codegen + EXPLAIN, all oracle-parity-tested end to end against the V2 query corpus.
Spend: estimated Large on take (#91 had no prior complexity estimate);
actual spend matched, including a follow-up fix pass for PR #117's
review findings (reachable panics, a silent CASE data leak, and the
project's mvl-limit qualified-subset CI gate).
parse_selectwas reporting several syntactically-valid-but-unimplemented SELECT constructs asParseOutcome::Invalid(genuine syntax error) instead ofUnsupported(#110):IN <table-name>, bareVALUES/ compound,HAVINGwithoutGROUP BY,NOT INDEXED, schema-qualified table names (aux.t5),->/->>operators, andOUTER LEFT NATURAL JOIN. This matters for #96's slice-boundary triage, which otherwise misreads these as our-bug.- Along the way, found and fixed a pre-existing dead-code bug: the
subquery-in-FROM
Unsupportedbranch intable_ref()was unreachable becauseidentifier()was called before the(check. - Spend: on track vs #110's ~120k token estimate.
- Phase 3B (#90, epic #56): the cursor, ephemeral-index (DISTINCT), and
sorter (ORDER BY) VDBE opcode families on top of #89's core —
src/vdbe/cursor.rs(OpenRead/OpenEphemeral/OpenPseudo/Rewind/Last/Next/Column/Rowid/SeekRowid/NullRow/Sequence/Found/IdxInsert/IdxLE/Delete) andsrc/vdbe/sorter.rs(SorterOpen/SorterInsert/SorterSort/Sort/SorterNext/SorterData), keyed by a newP4::SortKey/SortKeyColumndescriptor. TableCursor::last()/prev()(src/btree.rs) — reverse b-tree traversal, mirroringfirst()/next().Vm::with_db/execute_with_db— attaches a shared page source soOpenReadcan open real cursors, alongside the existing register-onlyexecute().- Hand-assembled acceptance programs (
tests/vdbe/cursor_sorter_test.rs) reproduce full-scan, ORDER BY, and DISTINCT against real corpus fixtures. Spec 009's Requirements 4 (cursor) and 9 (sorter) flip from(planned)to active. - Spend: on track vs #90's estimate (large, ~2500-3500 lines).
- Wired
tools/opcodes-v2.json(the oracle-harvested 52-opcode set, #58) intotools/assurance.pyas a VDBE completeness checklist (#65), now that phase 3A (#89) landed a real dispatch table to count against: anOpcode completeness:line in the Model section reports how many opcodessrc/vdbe/exec.rs'sdispatchactually handles versus the harvested total (30/52 as phase 3A lands).Opcode::ALL(src/vdbe/program.rs) plustests/vdbe/opcode_completeness_test.rskeep the enum and the harvested set from drifting apart silently.
- Test coverage raised on every file that
make coverageflagged below 85% line coverage:vdbe/value.rs(80.7% → 100%,sql_ltwas entirely untested),vfs/page_source.rs(76.0% → 97.78%, page-zero and short-read error paths),vdbe/coerce.rs(83.5% → 94.87%,checked_subwas entirely untested plus the Real-operand arithmetic path),vdbe/functions.rs(70.3% → 89.96%,nullif/sign/instr/trim/ltrim/rtrim/replacewere registered but never invoked by any test), andparser/printer.rs(64.47% → 98.48%, itstest_roundtrip_fixpointcorpus expanded from 9 to ~40 SQL strings covering the AST's full print surface — DISTINCT/ALL, table aliases, qualified columns, all unary/remaining binary operators, every literal and param kind, LIKE/GLOB/ESCAPE, COLLATE, CASE variants).parser/grammar.rsalso moved 82.4% → 91.46% as a side effect. Every file in the project is now ≥85%; TOTAL 89.11% → 94.00%. Spend: small, matched estimate.
src/vdbe/functions.rs:like/globscalar functions (spec008-value-semanticsReq 6, #59). Spec 009 Req 7 (#88) dispatcheslike(2)through theFunctionopcode into spec 008's registry, but the registry had nolike/glob— this closes that gap, so no LIKE-specific VDBE logic is needed. ASCII case-insensitive%/_matching withESCAPEforlike; case-sensitive*/?/[...](incl.[^...]negation and-ranges) forglob. Note SQLite's reversed argument order:like(pattern, text[, escape]).tests/corpus/expr_vectors/walker.jsonl: 71 oracle vectors covering CASE/CAST/LIKE/GLOB/BETWEEN/IN-list/short-circuit/arithmetic, harvested by spike 008 (#59) as phase-3 acceptance material.
src/parser/grammar.rs: keyword-named function calls (replace(...),glob(...)) were rejected —REPLACEtokenizes as a keyword, not an identifier, but SQLite accepts most keywords as function names when followed by(. This had silently blocked thefunctions.jsonlcorpus (committed since #78/#79) from ever being executed. Found by spike 008 (#59).src/parser/grammar.rs:-9223372036854775808now parses asLiteral::Integer(i64::MIN)rather than a REAL — the tokenizer folds the positive form to a Float since it has no i64 representation.
tests/tiers/tier1.rs: flipped thet1_expression_kernel_affinity_and_collation_vectorsstub, un-ignored since #78 (value-semantics kernel) shipped in 0.6.0 but was never flipped — a tier-stub-flip process gap caught by a parity review. Mirrors the siblingt1_scalar_functions_match_oraclepattern: a light direct-API smoke test overaffinity_of/apply_affinity/compare/compare_text, with full oracle-vector coverage remaining inexpr_vectors_test.rs. Spend: trivial.
CLAUDE.md: added an "Epic & phase breakdown conventions" section documenting theV{N} phase {M}[{letter}]ticket-naming and one-minor-per-completed-phase versioning pattern already in use on epic #56, so future epics (V3+) follow it consistently instead of re-deriving it each time.
src/vdbe/functions.rs: robustness gaps found in #92's review, #99.zeroblob()now clamps its requested length toMAX_BLOB_LEN(1e9) instead of allocating an unbounded amount — a hugeNpreviously hit Rust's allocator abort path.iif()'s TEXT-condition truthiness now checks bothInteger(0)andReal(0.0)coercion outcomes ('0.0'was incorrectly truthy).round()clampsdigitsto SQLite's[0, 30]range and propagates NULL when the digits argument is NULL instead of silently treating it as 0. Spend: small, matched the review-fix estimate.
src/vdbe/: the value-semantics kernel — spec008-value-semanticsRequirements 1-5, #78.affinity.rs(5-way type affinity derivation + application),compare.rs(cross-type comparison order, NULL < numeric < text < blob, with SQLite's exacti64/f64boundary comparison),collation.rs(BINARY/NOCASE/RTRIM),coerce.rs(longest-valid-numeric-prefix text coercion, checked arithmetic with REAL-overflow promotion),value.rs(NULL propagation, three-valuedAND/OR/NOT,IS/IS NOT). Pure functions onValue, no parser or VDBE-evaluator coupling — runs parallel to the #61 parser work. Spend: matched the medium estimate. Fuzz/proptest coverage deferred to #85.tests/fuzz/fuzz_targets/semantics_compare.rs,tests/semantics_proptest.rs: fuzz + proptest coverage for the value-semantics kernel (#78 follow-up, #85), spec008-value-semanticsRequirements 1, 2, 5 —compareantisymmetry/transitivity/never-panics across arbitraryValuepairs and collations,apply_affinityidempotence,coerce_text_to_numericidempotence on numeric text. Spend: matched the Small (~100k) estimate.src/vdbe/functions.rs: the V2 scalar function set — spec008-value-semanticsRequirement 6, #79.length,upper/lower,substr(a faithful port of SQLite'ssubstrFuncindex arithmetic),abs,coalesce/ifnull/nullif,typeof,hex/unhex,quote, scalarmin/max,round,sign,instr,trim/ltrim/rtrim,replace,zeroblob,iif— purefn(&[Value]) -> Result<Value, FunctionError>, dispatched through a case-insensitive name+arity registry (call_function), ready for phase 3'sFunctionopcode. Known gap:quote()'s REAL rendering doesn't byte-exact-match SQLite's own (observably build-dependent) higher-precision routine — same divergence already scoped out of.dump/-listin #37. Spend: matched the large estimate.
This closes out V2 phase 2 (value semantics + scalar functions) — next
up is V2 phase 3 (single-table SELECT execution, the Function opcode).
tests/fuzz/fuzz_targets/semantics_compare.rs,tests/semantics_proptest.rs: fuzz + proptest coverage for the value-semantics kernel (#78 follow-up, #85), spec008-value-semanticsRequirements 1, 2, 5 —compareantisymmetry/transitivity/never-panics across arbitraryValuepairs and collations,apply_affinityidempotence,coerce_text_to_numericidempotence on numeric text. Spend: matched the Small (~100k) estimate.
src/vfs/shm.rs: bounded-shmfile length against oversized files (#54). #66 had already eliminated theSIGBUSrisk #54 was filed for by switching-shmaccess frommmaptopread/pwrite; this closes the remaining gaps — an upper bound invalidate_shm_lenand a regression test — and records the pread/pwrite decision in.openspec/adr/0001-shm-access-pread-not-mmap.md.
Hand-written recursive-descent parser + typed AST for the SELECT-core V2
slice, spec 002-parser Requirements 2-4, #61. Spend: matched the 1.2M
"Large" estimate.
src/parser/ast.rs: typed AST forSELECT [DISTINCT] ... FROM table [WHERE] [ORDER BY] [LIMIT [OFFSET]]and its full V2 expression grammar (literals, params, column refs, function calls, unary/binary ops at SQLite precedence,IS [NOT] NULL,[NOT] BETWEEN,[NOT] IN,[NOT] LIKE/GLOB [ESCAPE],CASE,CAST,COLLATE, parens). Every node carries aSpan; parenthesization is preserved explicitly viaExprKind::Paren.src/parser/grammar.rs: the recursive-descent parser itself, one method per precedence level mirroringparse.y's%left/%righttable exactly. Recursive-descent entry points (expr/not_expr/unary_expr) are depth-guarded (MAX_EXPR_DEPTH) so pathological nesting fails cleanly instead of overflowing the stack.src/parser/error.rs:parse_select/ParseOutcome— the three-way accept / reject-unsupported / reject-invalid outcome from spike 006 (#57). Unsupported-but-valid constructs (JOIN, GROUP BY, compound SELECT, subqueries, CTEs, window functions) are distinguished from genuine syntax errors, each pointing at the triggering token.src/parser/printer.rs:Displayroundtrip printer, verified as a parse -> print -> parse fixpoint.tests/unit/parser.rs: 32 unit tests covering the full V2 grammar, both diagnostic outcomes, the roundtrip fixpoint, and deeply-nested pathological input.tests/corpus/parser_oracle_test.rs: accept/reject-unsupported/ reject-invalid parity against a livesqlite3oracle across the V2 corpus slice — the ticket's "oracle parity" acceptance bar.tests/fuzz/fuzz_targets/parse_select.rs(make fuzz-parse-select): fuzz target assertingparse_selectnever panics.
.openspec/specs/002-parser/spec.md: Requirements 2-4 flip(planned)→ active, all in-scope V2 scenarios test-linked (CTE and window-function scenarios stay(planned)— V4/V9 per the grammar's future-blocks stubs).
src/vfs/no longer needsunsafe(#66):src/vfs/lock.rs's rawlibc::fcntl(F_SETLK)is nownix::fcntl::fcntl(a safe wrapper);src/vfs/shm.rsno longermmaps the-shmfile —aReadMark/mxFrameaccess isstd::os::unix::fs::FileExt::{read_at, write_at}(pread/pwrite) at the same fixed offsets, and SHM lock slots use the same safefcntlwrapper.src/lib.rsis#![forbid(unsafe_code)]crate-wide again, with no local override anywhere in the crate.libcis no longer a direct dependency;nix(features:fs) replaces it.- Cross-process lock/shm tests now spawn a genuine subprocess (
tests/helpers/lock_probe.rs, a[[bin]]target) viastd::process::Command, instead offork/waitpid/_exit— a fresh address space, closer to a real secondsqlite3process, and needs nounsafe. Makefile'smvl-limitsrc/vfs/*exclusion rationale is nowdynonly (theVfs/VfsFile/SharedLockGuardtrait objects) — theunsaferationale no longer applies.
- The
-shmSIGBUSknown limitation (below, from 0.3.0) is gone: without anmmap, a-shmfile truncated out from under a reader now yields a structuredErrfrom the failingread_at/write_at, not an uncatchable process kill. Coherence between this crate's bufferedpread/pwriteaccess and a concurrentsqlite3process's ownMAP_SHAREDmapping of the same file relies on the OS's unified page cache — true on Linux and macOS, sqlite-rs's supported platforms.
src/parser/tokenizer.rs — a complete SQL tokenizer, spec 002-parser Requirement 1, #60.
src/parser/tokenizer.rs:Token/Span/TokenKind/Keyword/Paramtypes and the scanner. Covers all 146 SQLite reserved keywords (case-insensitive), bare/quoted/bracketed/backticked identifiers, integer/hex/float/string/blob/NULL/TRUE/FALSEliterals, all operators/punctuation (incl.||,->,->>), five parameter forms (?,?NNN,:name,@name,$name), and--//* */comments. Every token carries a line/column/byte-offsetSpan; malformed input always yields aTokenKind::Error, never a panic.tests/tokenizer_proptest.rs: tokenize/print roundtrip and never-panics-on-arbitrary-input property tests.
.openspec/specs/002-parser/spec.md: Requirement 1 flips(planned)→ active, all 4 scenarios test-linked.
sqlite-rs dump/export CLI — V1 step 9, epic #5's acceptance-gate ticket (#37, #49).
sqlite-rs dump <file>/sqlite-rs export <file>(src/bin/sqlite-rs.rs): schema + all rows of every readable table (rowid and WITHOUT ROWID), with rowid-alias substitution forINTEGER PRIMARY KEYcolumns and REAL-affinity 0/1 constant-optimization handling. Virtual tables and any table that fails to decode are skipped with a warning on stderr rather than aborting the whole dump;export's per-table output filenames are sanitized against the source database's (untrusted) table names to prevent path traversal. Both subcommands return a non-SUCCESSexit code when any table was skipped or failed to write, so scripted callers can detect partial output.src/format.rs:-list/-csvvalue rendering verified byte-identical to a real, read-onlysqlite3process — REAL formatting (%.15g-equivalent), blob-as-X'HEX', andsqlite3's actual (non-RFC4180) CSV quoting heuristic.tests/corpus/dump_oracle_test.rs: shells out to a realsqlite3 -readonlyand diffsdump_database's rendering against it across every table of every corpus fixture (list and csv mode);tests/corpus/harness.rs's previously-stubbed fixture reader now does a real open-and-dump.
TableSchemagains asqlfield (the verbatimCREATE TABLEtext), needed to reproduce schema DDL and column type/affinity info.Makefile'smvl-limitqualified-subset gate excludessrc/bin/*— a CLI's stdout/stderr is an I/O boundary, the same waysrc/vfsalready is the designatedunsafe/dynboundary.
- Dump/export oracle parity across all corpus fixture families: done (this release)
- Mutation-testing run, assurance-dashboard check, epic #5 close: tracked separately (#37 item (e)) — completing them finishes V1 without a further version bump
Pager read path, WAL frame reading, and safe-reader locking — epic #5 steps 2 and 6 (#35, #36), the journalstates fixture family (#21), and the safe-reader concurrency scope validated by spike 005 (#8) and implemented via #50/#45. Phase 3 = reading databases mid-life, not just at rest.
Pager(src/pager/mod.rs, #35): aPageSourceimplementation sitting between the VFS and the b-tree cursor. Refuses to open a database with a hot rollback journal (valid magic header) rather than risk serving pre-rollback pages as committed data; otherwise wrapsVfsPageSourceunchanged, soTableCursor<Pager>/IndexCursor<Pager>are byte-identical to theVfsPageSource-based cursors on every at-rest fixture, including auto-vacuum databases.- WAL frame reading (
src/pager/wal.rs, #36): WAL header parsing (both checksum-endianness variants — magic0x377f0682is native byte order, the common case), frame walk with checksum/salt validation, and a committed-page index merged transparently intoPager. Read-only, quiescent-file recovery — no-shmfile required for the recovery path. - Safe-reader locking (#50, #45; byte offsets and sequences validated against a live stock
sqlite3by spike 005, not re-derived):Pager::openacquires the journal-mode SHARED byte-rangefcntllock (PENDING_BYTE+2/SHARED_SIZE) before serving any page, released on drop (src/vfs/lock.rs, opaqueFileLocktype — nodyn/unsafeleaks outsidesrc/vfs/).- Busy detection (
VfsError::Locked): lock contention (EAGAIN/EACCES) surfaces as a distinguishable "database is locked" error, not a generic I/O failure. - WAL
-shmreader-mark protocol (src/vfs/shm.rs): on WAL-mode databases,Pager::openclaims aWAL_READ_LOCKslot and publishes itsaReadMarkat the WAL's currentmxFrame(read only after the exclusive slot claim), so a livesqlite3checkpointer backs off instead of truncating frames the reader depends on. Released on drop. - Cross-process (
fork-based) tests for the locking paths — POSIX record locks never conflict within one process.
journalstatesfixture family (#21): hot-journal and four WAL-pending fixture variants (primary, trailing/spilled, stale/foreign-salt, big-endian checksum), reusing spike #7's fixture-generation tricks.- Spec 007-pager: hot-journal detection, page-view zero-behavior-change, WAL frame reading; new 001-architecture Req-4 scenario "Reader takes a SHARED lock before serving pages".
Vfs::companion_path, closing spec 003 Req-1's previously-unimplemented "Companion file detection" scenario.- Second fuzz target (
fuzz/fuzz_targets/wal_frames.rs,make fuzz-wal) for the "malformed WAL never panics" acceptance criterion.
src/lib.rs:#![forbid(unsafe_code)]→#![deny(unsafe_code)]—forbidcannot be locally overridden andsrc/vfs/lock.rsneeds a scoped#![allow(unsafe_code)]for rawfcntl. Theunsafeboundary stays exactly where the plan designated it:src/vfs/.libcadded as a direct dependency (previously only transitive).
tests/corpus/regen_test.rs's reproducibility check assumed byte-identical regeneration corpus-wide — spec 004 Req-2 already allowed for "byte-identity not required where sqlite3 embeds nondeterminism," but nothing had exercised that carve-out untiljournalstates's WAL salts/journal nonces became the corpus's first nondeterministic fixture family. Now compared by size for that family only.
- Per-inode fd-cache for the POSIX
close()-drops-all-locks trap (#45): deliberately not built — nothing in the crate opens two fds to the same path (main db,-wal,-shmare three distinct paths, each opened once), so there is no bug for it to fix yet. Revisit when a write path or live-refresh read path needs a second fd to an already-locked file. - Linux exercise of the locking interop: owned by #42; CI already runs the full suite (including lock/shm tests) on
ubuntu-latest.
- Mapping a
-shmfile that another process may truncate can raiseSIGBUS(an uncatchable process termination, not a Rust panic) if the mapping outlives the file's backing pages. Inherent to the mmap approach without aSIGBUShandler; the threat model here is a cooperating localsqlite3writer, not an adversarial one — documented insrc/vfs/shm.rsrather than mitigated.
Read-only table and index b-tree cursors, plus the minimal DDL reader — epic #5 steps 4, 5, 7 (#32, #33, #34).
- Table b-tree cursor (
src/btree/, #32):TableCursor(first()/next()/seek(rowid)) over table b-trees (page types 0x05/0x0d), overflow-chain reassembly, page-1 cell-pointer-array trap;src/vfs/page_source.rsgenericPageSourcetrait +VfsPageSourceadapter - Index b-tree cursor (
src/btree/index.rs, #33):IndexCursor(first()/next()/seek(target)) over index b-trees (page types 0x02/0x0a), minimal key comparison (NULL < numeric < text < blob, BINARY collation); makes WITHOUT ROWID tables readable - Minimal DDL reader (
src/schema/ddl_reader.rs, #34):read_schema()decodessqlite_masterintoTableSchema(name, root_page, columns, without_rowid, strict, is_virtual) with zero dependency on a future full parser; unparseable/virtual-table DDL degrades to raw-row access, never an error - Spec 006-btree: page/cell/overflow byte format, transcribed from SQLite's file format and validated against a real oracle
- First fuzz target in the repo (
fuzz/fuzz_targets/btree_cursor.rs,cargo-fuzz,make fuzz-btree)
TableCursor::seekno longer accumulates against thefirst/nexttraversal's page-visited budget, so a long-lived cursor doing many point lookups can't spuriously fail- Overflow-chain reassembly now detects a chain that revisits a page (cycle) instead of relying solely on a flat hop cap, closing a resource-exhaustion path where a small malicious file could force very large reads/allocations
First milestone: the pure-computation core of the Tier 0 READ CORE, plus the assurance machinery. Epic #5 steps 1, 3, 8.
- Record format decoder (
src/record/, #9): varints (1-9 bytes), all serial types (NULL, all integer widths, f64 bit-exact, constants, BLOB, TEXT), all three text encodings (UTF-8/16LE/16BE), structured errors — no panics on malformed input - Database header parser (
src/header.rs, #11): full 100-byte header, page sizes 512-65536 (incl.1= 65536), reserved bytes, WAL-mode detection, text encoding - Read-only VFS (
src/vfs/, #11):Vfs/VfsFiletraits, Unix + in-memory implementations passing a shared contract suite - Fixture corpus + pinned oracle harness (
tests/corpus/, #10): reproducible generation (tools/gen_fixtures.sh), oracle version pinning, diff harness green-with-skips from day one - Assurance tooling:
make assurancedashboard (spec↔code↔test traceability, per-scenario links, symbol validation, dead-link detection),make mvl-limitqualified-subset gate (#23), coverage gate CI (#16, #24) - Specs: 001-architecture (tier model), 002-parser, 003-file-format, 004-corpus; 12-block value plan with drop order and concurrency contract
- Spikes: 001 (parser toolchains), 002 (end-to-end file read — GO, findings in
tests/spike/002_file_reading/findings.md)
#![forbid(unsafe_code)]— whole crate- mvl-limit: all files in the qualified subset
- Traceability: 10/10 requirements implemented (specs 003/004), 22/30 scenarios test-backed, 0 dead links