Skip to content

fix: eliminate full-table scans when parsing large Cursor state.vscdb… - #47

Merged
juliantanx merged 1 commit into
juliantanx:mainfrom
chomoe327:fix/cursor-large-db-full-scan
Aug 18, 2026
Merged

fix: eliminate full-table scans when parsing large Cursor state.vscdb…#47
juliantanx merged 1 commit into
juliantanx:mainfrom
chomoe327:fix/cursor-large-db-full-scan

Conversation

@chomoe327

@chomoe327 chomoe327 commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Fixes #46

Problem

On a large Cursor state.vscdb, aiusage parse shows the cursor source
progress at 100% and then spins the CPU (9–14%) for 10+ minutes without
exiting; aiusage serve hangs the same way during its startup parse,
making the tool unusable.

Diagnostic data from the affected machine (2.75GB db, cursorDiskKV =
210,517 rows):

key prefix rows size
bubbleId: 90,260 1024 MB
agentKv: 103,131 1007 MB
checkpointId: 5,293 141 MB
composerData: 857 78 MB

lsof showed the process holding a read handle on the db; sample
showed the main thread busy in V8 (CPU-bound, not I/O wait or lock
contention); the db itself is healthy (all sqlite3 CLI queries return
instantly, no corruption, no WAL lock).

Root cause

The parser issues one WHERE key LIKE 'bubbleId:<composerId>:%' query
per composer. SQLite's LIKE is case-insensitive by default
(case_sensitive_like=OFF in better-sqlite3), so it cannot use the
key TEXT UNIQUE auto-index — EXPLAIN QUERY PLAN confirms
SCAN cursorDiskKV for every such query. With ~850 composers that means
850+ sequential full-table scans of a 2.75GB table.

Two aggravating factors:

  • The existing watermark (CursorCursor) was applied in JS only after
    the full load, so incremental runs re-scanned the whole table too.
  • The SQLite stage emitted no progress — just a single 1/1 event after
    the entire parse finished.

Fix

  • Replace LIKE prefix matching with index-friendly range bounds
    (key >= 'bubbleId:<id>:' AND key < 'bubbleId:<id>;', likewise for
    composerData:). The query plan becomes
    SEARCH cursorDiskKV USING INDEX sqlite_autoindex_cursorDiskKV_1.
    Cursor writes keys with exact lowercase prefixes, so binary range
    comparison matches the same rows as the ASCII case-insensitive LIKE.
  • Incremental runs with no new composers now never touch bubble rows —
    the watermark short-circuits before any bubble query.
  • Skip databases whose schema has no cursorDiskKV table (e.g. residual
    dbs from old/uninstalled versions) instead of throwing.
  • Report per-composer progress during the SQLite stage through the
    existing ProgressReporter.

No watermark format change, no new dependencies, and the db is still
opened read-only.

Testing

  • New unit tests (real in-memory better-sqlite3 fixtures): missing-table
    degradation, unrelated key-prefix isolation (agentKv: etc. never
    leak into stats), per-composer progress callback, and no progress
    events on no-op incremental runs. Full suite: 53 files / 415 tests
    passing.
  • Verified against the real 2.75GB state.vscdb (read-only):
    • Per-composer token sums diffed against an independent single-pass
      aggregation over all 90,260 bubbleId: rows — 0 mismatches.
    • Cursor token total after parsing: 114,900,780 — identical to
      the pre-fix value, no data loss.

Before / after (same machine, same db)

scenario before after
cursor source full parse (2.75GB db) 10+ min, never finished ~10s
aiusage parse (incremental re-run) full re-scan, hung ~1s
aiusage serve startup parse hung, dashboard never up ~3s

…juliantanx#46)

The Cursor parser issued one `WHERE key LIKE 'bubbleId:<composerId>:%'`
query per composer. SQLite's LIKE is case-insensitive by default and
cannot use the `key TEXT UNIQUE` index on cursorDiskKV, so every one of
these queries degenerated into a full-table scan (verified via
EXPLAIN QUERY PLAN: SCAN cursorDiskKV). On a large state.vscdb
(210k rows / 2.75GB, ~850 composers) this meant 850+ sequential
full-table scans — `aiusage parse` spun the CPU for 10+ minutes after
the progress bar reached 100%, and `aiusage serve` hung on its startup
parse the same way.

- parse-cursor: replace LIKE prefix matching with index-friendly range
  bounds (`key >= 'bubbleId:<id>:' AND key < 'bubbleId:<id>;'`, same
  for composerData), turning each per-composer query into an index
  seek (SEARCH cursorDiskKV USING INDEX sqlite_autoindex_cursorDiskKV_1).
  Cursor writes keys with exact lowercase prefixes, so binary range
  comparison matches the same rows as the ASCII case-insensitive LIKE.
- parse-cursor: with the scans now indexed, the existing watermark
  (CursorCursor lastCreatedAt/lastId) actually short-circuits: an
  incremental run with no new composers never touches bubble rows.
- parse-cursor: skip databases whose schema has no cursorDiskKV table
  (e.g. residual dbs from old/uninstalled versions) instead of
  throwing.
- parse-cursor/parse: report per-composer progress during the SQLite
  stage through the existing ProgressReporter instead of a single
  1/1 event after the whole parse.

No watermark format change and no output change: per-composer token
sums were diffed against an independent single-pass aggregation over
all 90,260 bubbleId rows of a real 2.75GB state.vscdb — 0 mismatches.
Full parse of that db drops from 10+ minutes (never finished) to ~10s;
an incremental re-run completes in ~0.7s.

Fixes juliantanx#46
@juliantanx
juliantanx merged commit d25d23e into juliantanx:main Aug 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cursor 解析在大型 state.vscdb(2.7GB / 21万行 cursorDiskKV)上卡死,增量解析仍会全表重扫

2 participants