Conversation
A varint spends whole bytes on values that need a fraction of one. A column whose values range over four possibilities was costing eight bits each; it now costs two. Two new int column modes take the column minimum as a frame and bit-pack the remainder to exactly the width the span requires — one over the values, one over their differences. The frame is per-column and self-describing rather than drawn from the profile, which means an untrained codec gets it too: the plain Hyperfly row improves alongside the profiled one. Width zero is the degenerate and rather pleasing case — a constant column, or a constant stride under the delta frame, carries no payload bytes at all beyond its base. Per message across the 500-message corpora: route columnar profiled candles 496 -> 384 496 -> 384 devices 896 -> 807 705 -> 576 events 2109 -> 2083 896 -> 795 feed 1908 -> 1882 1536 -> 1511 orders 271 -> 271 184 -> 183 Candles gains most because its timestamps arrive at a constant stride, so every difference is identical and the delta frame packs to width zero. The cross language interop exchange moved from 258 to 215 bytes, 11.4x smaller than the JSON the client first received. Spec section 3.1 pins the bit stream: little-endian, value i at bits [i*w, (i+1)*w), width at most 56, and padding bits in the final byte MUST be zero — without that rule one value would have several encodings. All three implementations agree on the new golden vectors, and Rust passed them on its first run after porting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_hf1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A varint spends whole bytes on values that need a fraction of one.
firmwareMajorranges over four values and was paying eight bits for each; it now pays two.Two new int column modes take the column minimum as a frame and bit-pack the remainder to exactly the width the span requires:
0x02frame of reference —svarint(base), width byte, values packed asv[i] − base0x03delta frame of reference — same over the differencesThe frame is per-column and self-describing, not drawn from the profile. That matters: an untrained codec gets it too, so the plain
Hyperflyrow improves alongside the profiled one. Width zero is the pleasing degenerate case — a constant column, or a constant stride under the delta frame, carries no payload bytes at all beyond its base.Measured, per message over the 500-message corpora
Candles gains most because its timestamps arrive at a constant stride, so every difference is identical and the delta frame packs to width zero. The cross-language interop exchange moved from 258 B to 215 B — 11.4× smaller than the JSON the client first received.
Spec
§3.1 pins the bit stream so three implementations can't drift: little-endian, value
iat bits[i·w, (i+1)·w), width ≤ 56, and padding bits in the final byte MUST be zero — without that rule one value would have several encodings. New golden vectors cover the narrow case, the width-zero case, the shape where plain delta still wins, over-56 widths, and nonzero padding.All three implementations agree on the vectors; Rust passed them on its first run after porting. TS 205, Python 143, Rust 9. Site and bench numbers updated.
🤖 Generated with Claude Code