Skip to content

An index record is written as a row - #1354

Merged
bjmeetsfo merged 1 commit into
mainfrom
perf/an-index-record-is-written-as-a-row
Sep 8, 2026
Merged

An index record is written as a row#1354
bjmeetsfo merged 1 commit into
mainfrom
perf/an-index-record-is-written-as-a-row

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Every record in the index log carried its own field names.

The records are msgpack maps, so each one spelled out s, q, items, aw, and every row inside
spelled k, rb, ok, mi, oi, pi, a, sz again, once per page a shard has ever indexed.
Measured on a 10,000-write store that was 28 bytes of a 126-byte record.

Every record here has the same shape, so the shape does not belong in the record. Records and rows
are now values in field order, read by position. A tag-based design pays one byte for this; a
self-describing one pays a name; a row pays nothing.

Three things had to be true for that to be safe, and each was a bug until it was.

Nothing may be skipped. A row is read by position, so a field that vanishes when it is empty
moves every field behind it. BlockAddressWire still skipped absent fields, which shifted
generation into object_id and band_id into routing_bucket -- caught by a round-trip test,
and it would otherwise have mis-filed every page address on disk.

The container must say which record it holds. Two shapes share this log and one is read as the
other. A map allowed that by matching names and defaulting what was absent; a row cannot, and msgpack
markers cannot tell them apart either, because a whole-index record embeds a serde_json::Value
which is a map whatever the rows around it are. The container now carries a shape byte. A payload
with no container says nothing and goes to the decoder, so a record from an older writer decodes and
a well-framed payload of garbage is REPORTED rather than skipped.

Every cross-shape reader must read by position. There were four. Three announced themselves with
decode errors. The fourth did not: the GC-pressure counter decoded each payload as a whole-index
record and dropped failures with .ok(), so with rows it counted nothing reclaimable and index GC
quietly disagreed with WAL reclaim on the same frontier. The note above that line already described
the failure -- "the record is not counted, and GC reports no reclaimable index-log entries while the
log grows" -- reached by shape rather than by format. All four now take IndexRecordHead, which
reads the first values by position and ignores what follows.

Measured on 10,000 writes of a 78-byte payload:

index log   108 -> 85 bytes per record
total       384 -> 361 bytes per record
written     4.92x -> 4.62x the payload

Across the sequence that led here the index log has gone 189 -> 85 bytes per record, and the store
writes 6.17x -> 4.62x its payload.

Library suite 1719 passed with storage_manager_runtime_supports_stop_pause_resume_jitter_backoff_and_phase_flags
and durable_bytes_never_exceed_the_bytes_the_log_holds, both of which fail on main. Index log 40 of
40, block store 72 of 72, recovery suite 8 of 8.

@bjmeetsfo
bjmeetsfo merged commit 19f7aab into main Sep 8, 2026
@bjmeetsfo
bjmeetsfo deleted the perf/an-index-record-is-written-as-a-row branch September 8, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants