An index record is written as a row - #1354
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every record in the index log carried its own field names.
The records are msgpack maps, so each one spelled out
s,q,items,aw, and every row insidespelled
k,rb,ok,mi,oi,pi,a,szagain, once per page a shard has ever indexed.Measured on a 10,000-write store that was 28 bytes of a 126-byte record.
Every record here has the same shape, so the shape does not belong in the record. Records and rows
are now values in field order, read by position. A tag-based design pays one byte for this; a
self-describing one pays a name; a row pays nothing.
Three things had to be true for that to be safe, and each was a bug until it was.
Nothing may be skipped. A row is read by position, so a field that vanishes when it is empty
moves every field behind it.
BlockAddressWirestill skipped absent fields, which shiftedgenerationintoobject_idandband_idintorouting_bucket-- caught by a round-trip test,and it would otherwise have mis-filed every page address on disk.
The container must say which record it holds. Two shapes share this log and one is read as the
other. A map allowed that by matching names and defaulting what was absent; a row cannot, and msgpack
markers cannot tell them apart either, because a whole-index record embeds a
serde_json::Valuewhich is a map whatever the rows around it are. The container now carries a shape byte. A payload
with no container says nothing and goes to the decoder, so a record from an older writer decodes and
a well-framed payload of garbage is REPORTED rather than skipped.
Every cross-shape reader must read by position. There were four. Three announced themselves with
decode errors. The fourth did not: the GC-pressure counter decoded each payload as a whole-index
record and dropped failures with
.ok(), so with rows it counted nothing reclaimable and index GCquietly disagreed with WAL reclaim on the same frontier. The note above that line already described
the failure -- "the record is not counted, and GC reports no reclaimable index-log entries while the
log grows" -- reached by shape rather than by format. All four now take
IndexRecordHead, whichreads the first values by position and ignores what follows.
Measured on 10,000 writes of a 78-byte payload:
Across the sequence that led here the index log has gone 189 -> 85 bytes per record, and the store
writes 6.17x -> 4.62x its payload.
Library suite 1719 passed with
storage_manager_runtime_supports_stop_pause_resume_jitter_backoff_and_phase_flagsand
durable_bytes_never_exceed_the_bytes_the_log_holds, both of which fail on main. Index log 40 of40, block store 72 of 72, recovery suite 8 of 8.