fix: expose PRAGMA wal_autocheckpoint tuning for bulk-insert-heavy SQLite workloads#251
Merged
Merged
Conversation
…Lite workloads SQLite's default checkpoints WAL back into the main database file every ~4MiB (1,000 pages). Fine for a normal read/write mix, but a large bulk load pays increasingly expensive checkpoints as the file grows over the course of the load — each checkpoint flushes WAL frames into a B-tree that's larger, and less page-cache-resident, than the one before it. Root-caused via the SF10 LDBC SNB benchmark (packages/benchmarks): a ~9-hour SQLite bulk load, ~4.85x slower than Postgres running the exact same bulkInsert code path in-process with no network round trip. A local repro (real bulkCreate() calls, 100K/500K/2M synthetic rows) confirmed per-row insert cost degrades as the table grows and isolated the cause to wal_autocheckpoint's untuned default; raising it cut a 2M-row load's wall-clock time by over 50% at the scale tested, plateauing around 50,000-100,000 pages before regressing again at very large values (an oversized WAL has its own costs). Adds walAutocheckpointPages to LocalSqlitePragmaOptions, following the exact pattern already established by cacheSizeKib/mmapSizeBytes: defaults to undefined (SQLite's own default untouched), validated as a non-negative safe integer (0 disables automatic checkpointing entirely, for callers that would rather run one explicit PRAGMA wal_checkpoint after a bulk load finishes).
pdlug
force-pushed
the
fix/sqlite-bulk-load-checkpoint
branch
from
283e1c2 to
b515b67
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
walAutocheckpointPagestocreateLocalSqliteBackend'spragmasoption (PRAGMA wal_autocheckpoint), following the exact pattern already established bycacheSizeKib/mmapSizeBytes: defaults toundefined(SQLite's own built-in default untouched), so existing callers are unaffected. Values are runtime-validated as integers from0through2,147,483,647, matching SQLite's signed 32-bit C API; oversized values are rejected instead of being silently interpreted by SQLite as0(which disables automatic checkpointing).packages/benchmarks): a real SF10 run showed SQLite taking ~9 hours to bulk-load ~30M rows — ~4.85x slower than Postgres running the exact samebulkInsertcode path in-process with no network round trip, and worse-than-linear relative to SF1 (SQLite's load time grew ~13.4x for a ~10.65x row-count increase; Postgres grew ~9.6x, essentially linear).bulkCreate()calls, 100K/500K/2M synthetic rows, real SNBCommentschema/indexes) confirmed the cause: SQLite's defaultwal_autocheckpoint(1,000 pages, ~4MiB) checkpoints often enough that each checkpoint pays an increasing cost as the database file grows over a bulk load — flushing WAL frames into a B-tree that's larger, and less page-cache-resident, than the one before it.wal_autocheckpointcut the 2M-row case's wall-clock time by over 50% at the best-performing value, plateauing (then regressing — an oversized WAL has its own costs) around 50,000-100,000 pages.