fix(db): remove orphaned segment directories during crash recovery - #674
Open
YongqiYin wants to merge 2 commits into
Open
fix(db): remove orphaned segment directories during crash recovery#674YongqiYin wants to merge 2 commits into
YongqiYin wants to merge 2 commits into
Conversation
A crash between Optimize's lock-free directory rename and the
commit-phase manifest flush leaves an on-disk segment directory that
the recovered manifest never references: the directory leaks
permanently, and the first operation that re-allocates that segment id
fails spuriously ("segment path already exists" on writing-segment
switch, ENOTEMPTY on the next Optimize's rename).
Recovery now removes numeric segment directories not referenced by the
recovered manifest (persisted segments plus the writing segment), as
well as leftover *.tmp compact outputs. The exclusive collection file
lock is already held at this point, so no concurrent Optimize can be
producing these directories. Read-only opens skip the cleanup.
Fixes alibaba#673
- collect candidate directory names first and remove them after the
scan: deleting an entry while iterating a directory is
implementation-defined, and the range-for increment could still
throw filesystem_error, breaking the best-effort contract
- log a warning when listing the collection directory fails instead
of failing silently
- guard against a missing writing segment meta
- extend the regression test with non-canonical guard directories
(".tmp", "5.tmpx") and an add-column DDL after recovery, covering
the DDL failure mode listed in the issue
This was referenced Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #673.
Problem
Since #614, Optimize renames its compacted output to the final numeric
directory and opens it in the lock-free phase, before the commit phase
persists
next_segment_id. If the process crashes inside that window,recovery restores the id allocator from the old manifest and leaves the
unreferenced directory on disk:
(
segment path already existson a writing-segment switch fromInsert/DDL, ENOTEMPTY on the next Optimize's rename).
Fix
recovery()now removes on-disk leftovers right after the manifest isloaded, before opening segments:
(persisted set plus the writing segment);
<id>.tmpcompact outputs that were never renamed.Safety:
Optimize can be producing these directories;
no leading zeros, fits uint32) are considered; anything else is left
untouched;
The Optimize three-phase locking model from #614 is unchanged.
Test
New regression test
Feature_Recovery_Orphan_Segment_Dirs_Removed:crash leftovers (colliding dir, far orphan dir,
*.tmpresidue) plusnon-canonical guard dirs (
007,.tmp,5.tmpx) that must survive;the orphans and keeps every referenced segment directory;
Optimize()and an add-column DDL both succeed with alldocs intact.
Fails on main at both the directory assertions and the post-reopen
Optimize(); passes with the fix. Fullcollection_testsuite: 86/86.Note: this makes a read-write
Collection::Opendelete directories underthe collection path that match canonical segment naming but are not
referenced by the manifest — a deliberate, logged self-heal for crash
leftovers.