Skip to content

Checkpoint after building the FTS index, or the next boot dies - #432

Merged
jrosskopf merged 1 commit into
mainfrom
fix/fts-index-wal-replay
Aug 31, 2026
Merged

Checkpoint after building the FTS index, or the next boot dies#432
jrosskopf merged 1 commit into
mainfrom
fix/fts-index-wal-replay

Conversation

@jrosskopf

@jrosskopf jrosskopf commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

A gateway that had just rebuilt its index served queries perfectly, and then refused to start again:

escurel-server: fatal: opening DuckDB at …/escurel.duckdb:
Failure while replaying WAL file "…/escurel.duckdb.wal":
Cannot drop entry "fts_main_blocks" because there are entries that depend on it.

PRAGMA create_fts_index(…, overwrite = 1) drops the existing fts_main_blocks schema before recreating it, and DuckDB cannot perform that drop during WAL replay — the dependency ordering replay reconstructs is not the one the live catalog had.

This failure is already documented here, in a different operation

Migrator::ensure_write_attribution names the shape exactly:

…sits in the WAL as a live grenade: the process that ran it keeps working, and the next process to open the file fails to start. That is the worst possible shape for a migration bug — it does not show up until a restart.

The remedy is the one that comment prescribes: CHECKPOINT, folding the operation into the database file and truncating the WAL so there is nothing left to replay.

Observed, not theorised

On the datazoo-loops tenant, whose gateway could not restart at all — every boot needed the index deleted and rebuilt from the lanes, several minutes of re-embedding each time.

I diagnosed it as an unclean shutdown first. That was wrong: it reproduces from a clean stop of a freshly built index.

Verification

By the reproduction that failed twice:

  1. clear the index → boot (rebuilds from the lanes) → serves queries
  2. stop cleanly
  3. boot again — now comes up, where it previously died

Also: no .wal file remains after the stop, and search still returns hits afterwards, so the checkpoint folded in an index that works rather than a broken one.

cargo test --workspace1359 passed, 0 failed.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

… dies

A gateway that had just rebuilt its index served queries perfectly and
then refused to start again:

    escurel-server: fatal: opening DuckDB at …/escurel.duckdb:
    Failure while replaying WAL file "…/escurel.duckdb.wal":
    Cannot drop entry "fts_main_blocks" because there are entries that
    depend on it.

`PRAGMA create_fts_index(…, overwrite = 1)` drops the existing
`fts_main_blocks` schema before recreating it, and DuckDB cannot perform
that drop during WAL replay — the dependency ordering replay reconstructs
is not the one the live catalog had.

**This is the failure `Migrator::ensure_write_attribution` already
documents, in a different operation.** Its comment names the shape
exactly: an unreplayable statement in the WAL is a live grenade — the
process that ran it keeps working, and the NEXT process to open the file
fails to start. It is the worst shape a bug of this kind can take,
because nothing is wrong until a restart, and by then the cause is hours
behind you.

The remedy is the one that comment prescribes: CHECKPOINT, folding the
operation into the database file and truncating the WAL so there is
nothing left to replay.

Observed on the `datazoo-loops` tenant, whose gateway could not restart
at all: every boot needed the index deleted and rebuilt from the lanes,
several minutes of re-embedding each time. Diagnosed as an unclean
shutdown first, which was wrong — it reproduces from a clean stop of a
freshly built index.

Verified by the reproduction that failed twice: clear the index, boot
(rebuilds), stop cleanly, boot again. The second boot now comes up, no
`.wal` file remains after the stop, and search still returns hits — so
the checkpoint folded in an index that works rather than a broken one.
@jrosskopf
jrosskopf merged commit c350c00 into main Aug 31, 2026
4 checks passed
@jrosskopf
jrosskopf deleted the fix/fts-index-wal-replay branch August 31, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant