Validate raw storage segment bounds - #110
Merged
Merged
Conversation
tdenisenko
marked this pull request as ready for review
July 22, 2026 21:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes the opaque
column read out of boundsfailure that appeared during background compaction by validating raw storage segment files before they are accepted by startup integrity or read by the compactor.The investigation found that the warning was caused by real data corruption in the current Mac mini data directory, not only by a reader bug. Segment
11677hastopic2.coltruncated by 6,528 bytes: the header advertises 1,000,000 rows, but the file body only contains 999,796 fixed-width rows. Itstopic2.nullbitmap still marks 153 of the missing rows as present, so padding or truncating would corrupt query results.Changes
data.colvariable-width offsets/blob length.canonical.bitmapwhile removing only raw hot columns and topic null bitmaps.Operational Notes
This PR does not pretend to repair an unrecoverable truncated segment. The current Mac mini data dir still needs a data-repair step before strict validation can run against it: either restore a clean copy of segment
11677or resync/quarantine the affected block range (25484454..25485698).That repair should be a follow-up task because it changes storage recovery behavior and must avoid serving incomplete or unverifiable log rows.
Validation
cargo fmt --checkcargo test -p logex-storagecargo test -p logex-node -p logex-query -p logex-servercargo clippy --workspace --all-targets -- -D warningscargo test --workspace