Skip to content

Validate raw storage segment bounds - #110

Merged
tdenisenko merged 1 commit into
masterfrom
fix/storage-compaction-bounds
Jul 22, 2026
Merged

Validate raw storage segment bounds#110
tdenisenko merged 1 commit into
masterfrom
fix/storage-compaction-bounds

Conversation

@tdenisenko

Copy link
Copy Markdown
Owner

Summary

This PR fixes the opaque column read out of bounds failure that appeared during background compaction by validating raw storage segment files before they are accepted by startup integrity or read by the compactor.

The investigation found that the warning was caused by real data corruption in the current Mac mini data directory, not only by a reader bug. Segment 11677 has topic2.col truncated by 6,528 bytes: the header advertises 1,000,000 rows, but the file body only contains 999,796 fixed-width rows. Its topic2.null bitmap still marks 153 of the missing rows as present, so padding or truncating would corrupt query results.

Changes

  • Added strict raw segment completeness validation for fixed-width raw columns, topic null bitmaps, the canonical bitmap, and data.col variable-width offsets/blob length.
  • Run the validation before raw segment compaction so corrupt segments fail before the low-level reader can emit an ambiguous bounds error.
  • Run the same validation during storage startup integrity so corrupt raw segments are not silently accepted after restart.
  • Added contextual compaction errors with segment id, row count, block range, column name, and length mismatch details.
  • Reused raw column constants for post-compaction raw file cleanup and preserved canonical.bitmap while removing only raw hot columns and topic null bitmaps.
  • Added regression tests for truncated raw columns during startup and background compaction.

Operational Notes

This PR does not pretend to repair an unrecoverable truncated segment. The current Mac mini data dir still needs a data-repair step before strict validation can run against it: either restore a clean copy of segment 11677 or resync/quarantine the affected block range (25484454..25485698).

That repair should be a follow-up task because it changes storage recovery behavior and must avoid serving incomplete or unverifiable log rows.

Validation

  • cargo fmt --check
  • cargo test -p logex-storage
  • cargo test -p logex-node -p logex-query -p logex-server
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace

@tdenisenko
tdenisenko marked this pull request as ready for review July 22, 2026 21:37
@tdenisenko
tdenisenko merged commit 90f3a64 into master Jul 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant