Checkpoint after building the FTS index, or the next boot dies - #432
Merged
Conversation
… dies
A gateway that had just rebuilt its index served queries perfectly and
then refused to start again:
escurel-server: fatal: opening DuckDB at …/escurel.duckdb:
Failure while replaying WAL file "…/escurel.duckdb.wal":
Cannot drop entry "fts_main_blocks" because there are entries that
depend on it.
`PRAGMA create_fts_index(…, overwrite = 1)` drops the existing
`fts_main_blocks` schema before recreating it, and DuckDB cannot perform
that drop during WAL replay — the dependency ordering replay reconstructs
is not the one the live catalog had.
**This is the failure `Migrator::ensure_write_attribution` already
documents, in a different operation.** Its comment names the shape
exactly: an unreplayable statement in the WAL is a live grenade — the
process that ran it keeps working, and the NEXT process to open the file
fails to start. It is the worst shape a bug of this kind can take,
because nothing is wrong until a restart, and by then the cause is hours
behind you.
The remedy is the one that comment prescribes: CHECKPOINT, folding the
operation into the database file and truncating the WAL so there is
nothing left to replay.
Observed on the `datazoo-loops` tenant, whose gateway could not restart
at all: every boot needed the index deleted and rebuilt from the lanes,
several minutes of re-embedding each time. Diagnosed as an unclean
shutdown first, which was wrong — it reproduces from a clean stop of a
freshly built index.
Verified by the reproduction that failed twice: clear the index, boot
(rebuilds), stop cleanly, boot again. The second boot now comes up, no
`.wal` file remains after the stop, and search still returns hits — so
the checkpoint folded in an index that works rather than a broken one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A gateway that had just rebuilt its index served queries perfectly, and then refused to start again:
PRAGMA create_fts_index(…, overwrite = 1)drops the existingfts_main_blocksschema before recreating it, and DuckDB cannot perform that drop during WAL replay — the dependency ordering replay reconstructs is not the one the live catalog had.This failure is already documented here, in a different operation
Migrator::ensure_write_attributionnames the shape exactly:The remedy is the one that comment prescribes:
CHECKPOINT, folding the operation into the database file and truncating the WAL so there is nothing left to replay.Observed, not theorised
On the
datazoo-loopstenant, whose gateway could not restart at all — every boot needed the index deleted and rebuilt from the lanes, several minutes of re-embedding each time.I diagnosed it as an unclean shutdown first. That was wrong: it reproduces from a clean stop of a freshly built index.
Verification
By the reproduction that failed twice:
Also: no
.walfile remains after the stop, and search still returns hits afterwards, so the checkpoint folded in an index that works rather than a broken one.cargo test --workspace— 1359 passed, 0 failed.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.