fix: eliminate full-table scans when parsing large Cursor state.vscdb… - #47
Merged
juliantanx merged 1 commit intoAug 18, 2026
Merged
Conversation
…juliantanx#46) The Cursor parser issued one `WHERE key LIKE 'bubbleId:<composerId>:%'` query per composer. SQLite's LIKE is case-insensitive by default and cannot use the `key TEXT UNIQUE` index on cursorDiskKV, so every one of these queries degenerated into a full-table scan (verified via EXPLAIN QUERY PLAN: SCAN cursorDiskKV). On a large state.vscdb (210k rows / 2.75GB, ~850 composers) this meant 850+ sequential full-table scans — `aiusage parse` spun the CPU for 10+ minutes after the progress bar reached 100%, and `aiusage serve` hung on its startup parse the same way. - parse-cursor: replace LIKE prefix matching with index-friendly range bounds (`key >= 'bubbleId:<id>:' AND key < 'bubbleId:<id>;'`, same for composerData), turning each per-composer query into an index seek (SEARCH cursorDiskKV USING INDEX sqlite_autoindex_cursorDiskKV_1). Cursor writes keys with exact lowercase prefixes, so binary range comparison matches the same rows as the ASCII case-insensitive LIKE. - parse-cursor: with the scans now indexed, the existing watermark (CursorCursor lastCreatedAt/lastId) actually short-circuits: an incremental run with no new composers never touches bubble rows. - parse-cursor: skip databases whose schema has no cursorDiskKV table (e.g. residual dbs from old/uninstalled versions) instead of throwing. - parse-cursor/parse: report per-composer progress during the SQLite stage through the existing ProgressReporter instead of a single 1/1 event after the whole parse. No watermark format change and no output change: per-composer token sums were diffed against an independent single-pass aggregation over all 90,260 bubbleId rows of a real 2.75GB state.vscdb — 0 mismatches. Full parse of that db drops from 10+ minutes (never finished) to ~10s; an incremental re-run completes in ~0.7s. Fixes juliantanx#46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #46
Problem
On a large Cursor
state.vscdb,aiusage parseshows the cursor sourceprogress at 100% and then spins the CPU (9–14%) for 10+ minutes without
exiting;
aiusage servehangs the same way during its startup parse,making the tool unusable.
Diagnostic data from the affected machine (2.75GB db,
cursorDiskKV=210,517 rows):
bubbleId:agentKv:checkpointId:composerData:lsofshowed the process holding a read handle on the db;sampleshowed the main thread busy in V8 (CPU-bound, not I/O wait or lock
contention); the db itself is healthy (all sqlite3 CLI queries return
instantly, no corruption, no WAL lock).
Root cause
The parser issues one
WHERE key LIKE 'bubbleId:<composerId>:%'queryper composer. SQLite's LIKE is case-insensitive by default
(
case_sensitive_like=OFFin better-sqlite3), so it cannot use thekey TEXT UNIQUEauto-index —EXPLAIN QUERY PLANconfirmsSCAN cursorDiskKVfor every such query. With ~850 composers that means850+ sequential full-table scans of a 2.75GB table.
Two aggravating factors:
CursorCursor) was applied in JS only afterthe full load, so incremental runs re-scanned the whole table too.
1/1event afterthe entire parse finished.
Fix
(
key >= 'bubbleId:<id>:' AND key < 'bubbleId:<id>;', likewise forcomposerData:). The query plan becomesSEARCH cursorDiskKV USING INDEX sqlite_autoindex_cursorDiskKV_1.Cursor writes keys with exact lowercase prefixes, so binary range
comparison matches the same rows as the ASCII case-insensitive LIKE.
the watermark short-circuits before any bubble query.
cursorDiskKVtable (e.g. residualdbs from old/uninstalled versions) instead of throwing.
existing
ProgressReporter.No watermark format change, no new dependencies, and the db is still
opened read-only.
Testing
degradation, unrelated key-prefix isolation (
agentKv:etc. neverleak into stats), per-composer progress callback, and no progress
events on no-op incremental runs. Full suite: 53 files / 415 tests
passing.
state.vscdb(read-only):aggregation over all 90,260
bubbleId:rows — 0 mismatches.the pre-fix value, no data loss.
Before / after (same machine, same db)
aiusage parse(incremental re-run)aiusage servestartup parse