fix(db): stop rebuilding the FTS index on every start, and page the rebuild

[ankit-thebigred]

Closes #37.

That issue has been open since 9 July with a correct diagnosis and no fix, so here is the fix. It turned out to need one change beyond what the issue suggested, explained below.

Two compounding bugs

1. The "is the index already populated?" probe can never return true.

messages_fts is created as fts5(text_content, content=''), a contentless table. SQLite stores no column values for contentless FTS5, so reading one back always returns NULL no matter how many rows are indexed. !sample?.text_content was therefore always true, the full rebuild branch ran on every launch, and the fts_version check directly above it never had any effect. The first thing that branch does is DROP TABLE, so a perfectly good index was destroyed on every start.

Reproduction, no app required:

CREATE VIRTUAL TABLE t USING fts5(text_content, content='');
INSERT INTO t(rowid, text_content) VALUES (1, 'hello world');

SELECT text_content FROM t LIMIT 1;  -- NULL, so the index "looks empty"
SELECT COUNT(*) FROM t;              -- 1

2. The rebuild materialises the entire history in one .all().

Every user and assistant message, content JSON included, was loaded into a single array before indexing. On a large history that exhausts the V8 heap. Because the table was already dropped at the top of the branch, each respawn restarted from an empty index and ran out of memory again, which is the crash loop described in #37. The incremental path had the same shape.

Bug 1 is what makes it fire on every launch. Bug 2 is what turns an unnecessary rebuild into a hard crash.

The fix

Probe COUNT(*), and page both the full rebuild and the incremental backfill through one shared helper using a keyset cursor, so peak memory stays flat regardless of history size.

One correction to the fix suggested in #37: it proposes .iterate() with batched inserts, and that does not work. better-sqlite3 refuses to run a write while an iterator is open on the same connection:

This database connection is busy executing a query

I hit this on the bundled better-sqlite3 before settling on paging. Keyset pagination gives the same bounded memory without holding an iterator open, so that is what this PR does. The reasoning is left as a comment on the helper so it does not get "simplified" back into an iterator later.

Verification

Root npm run typecheck is clean.

Behaviour checked against a scratch database using the same schema:

Check	Result
5,000 message rebuild spanning three pages	5,000 indexed
COUNT(*) probe after rebuild	5,000, where the old probe reports NULL
Non user/assistant rows excluded	100 system rows correctly skipped
Second call re-indexes nothing	0
10 newly inserted messages	picked up incrementally
MATCH queries after paged insert	return the expected rowids
Row deep inside the third page	still searchable

Impact

On the database that prompted #37 this was re-reading and re-indexing roughly 600,000 messages on every single launch. Beyond the crash, the repeated drop and rebuild of a ~1 GB index leaves a large amount of free space behind: VACUUM on that database reclaimed 2.6 GiB.

[vercel]

@ankit-thebigred is attempting to deploy a commit to the DevApe Team on Vercel.

A member of the Team first needs to authorize it.