Repository navigation
Values up to 2 MiB, page-slot reuse, and WAL frame cap - #1
Merged
Merged
Conversation
Values were capped at 4,000 bytes, one slotted-page tuple. Larger values (up to MaxValueSize, 2 MiB) are now stored as a CRC-checked chain of chunk tuples headed by a manifest; the B+ tree flags the manifest RID with bit 15 of the slot ID. Small values take the existing path unchanged, existing files need no migration, and the WAL, recovery and replication still log whole values. Batches may carry up to 8 MiB of values. See docs/adr/0001-large-values.md. - Reads verify chunk magic, chain length and CRC (ErrCorruptValue); overwrite and delete free a chain only after it validates. - gRPC: MessageSizeServerOptions / MessageSizeDialOption raise the 4 MiB default; the server, followers and beastctl use them. Write-error status mapping is shared in grpc_status.go. - WAL: frames over 32 MiB are refused on write, and a torn header claiming a huge length now reads as a torn tail instead of allocating up to 4 GiB. - Storage: InsertTuple reuses tombstoned slots. Previously a recycled page's slot directory grew until it could not host a full tuple, and pages too full for one request were dropped from the reuse list; 300 overwrites of a 512 KB value grew the file to 2,120 pages, now 266, and 600 small overwrites added 264 pages, now 3. Tests: boundary sizes, overwrite/delete space reuse, crash and clean restart, CAS and batches on large values, scans, corruption detection, a mixed-size oracle across restarts, gRPC 6 MiB batches, and catch-up plus live replication of 2 MiB values. Full suite passes with -race. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017mWMfRGkKdRgKc5mtSsWEG
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 069adff0a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…g in batch limit - freeValue checked chunk magic and chain termination but not total length or CRC, so a mis-linked chain that still ended cleanly after the right number of chunks could free tuples owned by another key. It now verifies length and CRC first and leaves a failing chain in place. - MaxBatchBytes counted only value bytes. Each WAL batch operation adds 15 bytes of framing, so many tiny puts could commit a WAL frame larger than the replication message limit, and followers could never catch up on it. The limit now covers the encoded frame. Both reported by Codex review on #1; regression tests reproduce each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017mWMfRGkKdRgKc5mtSsWEG
ChromaBeast
added a commit
that referenced
this pull request
Nov 12, 2026
…g in batch limit - freeValue checked chunk magic and chain termination but not total length or CRC, so a mis-linked chain that still ended cleanly after the right number of chunks could free tuples owned by another key. It now verifies length and CRC first and leaves a failing chain in place. - MaxBatchBytes counted only value bytes. Each WAL batch operation adds 15 bytes of framing, so many tiny puts could commit a WAL frame larger than the replication message limit, and followers could never catch up on it. The limit now covers the encoded frame. Both reported by Codex review on #1; regression tests reproduce each.
ChromaBeast
added a commit
that referenced
this pull request
Nov 12, 2026
Values up to 2 MiB, page-slot reuse, and WAL frame cap
ChromaBeast
added a commit
that referenced
this pull request
Nov 13, 2026
Values up to 2 MiB, page-slot reuse, and WAL frame cap
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Values were capped at 4,000 bytes (one slotted-page tuple), so clients storing documents, reports or serialized aggregates had to invent their own chunking. This adds large values as a general engine feature and fixes two allocator defects found while testing it.
MaxRecordSizeare stored as a CRC-checked chain of chunk tuples headed by a 24-byte manifest; the B+ tree flags the manifest RID with bit 15 of the slot ID. Small values take the existing path unchanged and existing files need no migration. The WAL, recovery and replication still log whole values. Design and trade-offs:docs/adr/0001-large-values.md.ErrCorruptValue); overwrite and delete free a chain only after it validates, so a damaged chain cannot tombstone other keys' tuples.api.MessageSizeServerOptions/api.MessageSizeDialOptionraise the 4 MiB default; the server, followers andbeastctluse them. Write-error status mapping moved intogrpc_status.go.InsertTuplenever reused tombstoned slots, so a recycled page's slot directory grew until it couldn't host a full tuple; and pages too full for one request were dropped from the reuse list for good. 300 overwrites of a 512 KB value grew the file to 2,120 pages, now 266; 600 small overwrites added 264 pages, now 3.Compatibility
ErrRecordTooLargenow means "over 2 MiB";ErrBatchTooLargeis new.ErrSlotOutOfBoundsrather than returning manifest bytes.Tests
Boundary sizes, overwrite/delete space reuse, crash and clean restart, CAS and batches on large values, scans, corruption detection, a mixed-size oracle across restarts, page slot reuse, oversized WAL headers, 6 MiB gRPC batches, and catch-up plus live replication of 2 MiB values. Full suite passes with
-race;go vetclean.Downstream: Peerline (ChromaBeast/Peerline) depends on this for releases, reports and indexes larger than 4 KB; its router tests pass against a live server built from this branch.
Generated by Claude Code