Skip to content

Commit 98c201b

Browse files
authored
Columnar v5: grammar and derived columns, categorical bit packing, patched frames (#18)
* spec: columnar v5 — categorical bit packing, grammar and derived string modes, PFOR * spec: gate round 1 — pin FOR widths, PFOR exact arithmetic, delta-FOR k>=2, grammar digit semantics, closed profile domain * columnar@5 TS: categorical bit packing, grammar and derived string modes, PFOR, vectors * decode: a single-member enum column has no payload, like a literal * trainer: classify grammar slots across the sample, not per value; bench: profile size accounting for grammar and derived columns * columnar@5 gate round 1: amplification limit replaces the input-affordability bound for columnar, deferred row materialization, span/case/participation vectors, closed-profile vectors, v2 fingerprint case, trainer guards * gate round 2: amplification covers zero-payload row arrays and mirrors at encode, discriminating profile vectors, trainer prices deflate * gate round 3: discriminating vectors for v1 required keys, duplicate leaves, grammar caps, literal and case rules; dictionary ceiling test * site: columnar@5 numbers, profiled rows drop brotli, planner card matches what shipped * readme: columnar@5 numbers and features, amplification wording, spec links * Port columnar plan v5 to Python * columnar: a deflate candidate beyond maxByteLength is no candidate (python port finding) * Port columnar plan v5 to Rust * pr gate: interop bootstraps a v2 profiled artifact, python defers deflate until the aggregate qualifies, spec links to v5
1 parent 5122655 commit 98c201b

41 files changed

Lines changed: 9058 additions & 1982 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -16,16 +16,19 @@ Bytes per message, averaged over 500-message corpora (`bun run bench`):
1616

1717
| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled |
1818
|---|---|---|---|---|---|---|
19-
| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** |
20-
| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** |
21-
| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** |
22-
| single order | 782 | 408 | 388 | 271 | 273 | **188** |
23-
| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 |
24-
25-
Read the spread rather than the best row. Training is worth 57% on audit logs, where
26-
the same user agents recur on every request, and nothing at all on candles, whose
27-
only string sits outside the array. The corpora are synthetic — shaped like real
28-
routes, not captured from one — and no production traffic has been measured yet.
19+
| audit events | 12,687 | 2,512 | 7,190 | 2,013 | 1,974 | **574** |
20+
| device telemetry | 7,994 | 1,422 | 2,007 | 755 | 725 | **542** |
21+
| social feed | 6,863 | 2,294 | 4,396 | 1,871 | 1,868 | **1,466** |
22+
| single order | 782 | 408 | 388 | 271 | 273 | **180** |
23+
| OHLCV candles | 3,225 | 842 | 2,034 | 384 | **362** | 384 |
24+
25+
Read the spread rather than the best row. Training is worth 71% on audit logs —
26+
recurring values become dictionary codes, machine-made ids travel as grammar
27+
lanes, one column derives from another — and nothing at all on candles, whose
28+
only string sits outside the array. Note the Brotli column: on the profiled
29+
stream it now loses bytes on four of five routes. The corpora are synthetic —
30+
shaped like real routes, not captured from one — and no production traffic has
31+
been measured yet.
2932

3033
## Repository
3134

@@ -46,8 +49,9 @@ code.
4649

4750
- [wire v0](spec/wire-v0.md) — envelope, varints, bitmaps, node encodings, canonical
4851
artifacts, decoder limits
49-
- [plan columnar v3](spec/plan-columnar-v3.md) — column layout, delta and XOR and
50-
scaled-decimal numerics, packed text, trained dictionaries
52+
- [plan columnar v5](spec/plan-columnar-v5.md) — column layout, frame-of-reference
53+
bit packing, scaled-decimal and XOR numerics, packed text, trained dictionaries,
54+
grammar lanes, derived columns
5155
- [negotiation v1](spec/negotiation-v1.md) — how peers agree on binary, how a client
5256
bootstraps, how a profile rotates without a cutover
5357

‎apps/bench/src/profiles.ts‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -88,10 +88,13 @@ export function measureCorpus(suite: CorpusSuite): CorpusResult {
8888
const n = suite.corpus.length;
8989
const mean = (total: number) => Math.round(total / n);
9090

91-
const dictBytes = columns.reduce(
92-
(sum, c) => sum + c.dict.reduce((m, e) => m + enc.encode(e).length + 1, 0),
93-
0,
94-
);
91+
const dictBytes = columns.reduce((sum, c) => {
92+
let bytes = 0;
93+
for (const e of c.dict ?? []) bytes += enc.encode(e).length + 1;
94+
for (const e of c.derived?.values ?? []) bytes += enc.encode(e).length + 1;
95+
if (c.grammar) bytes += enc.encode(JSON.stringify(c.grammar)).length;
96+
return sum + bytes;
97+
}, 0);
9598
const savedPerMessage = mean(col) - mean(prof);
9699

97100
return {
@@ -105,7 +108,7 @@ export function measureCorpus(suite: CorpusSuite): CorpusResult {
105108
columnarBrotli: mean(colBr),
106109
profiled: mean(prof),
107110
full: mean(full),
108-
dictEntries: columns.reduce((sum, c) => sum + c.dict.length, 0),
111+
dictEntries: columns.reduce((sum, c) => sum + (c.dict?.length ?? 0) + (c.derived?.values.length ?? 0), 0),
109112
dictBytes,
110113
breakEven: savedPerMessage > 0 ? Math.ceil(dictBytes / savedPerMessage) : null,
111114
};

‎apps/interop/client.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,12 @@ def main() -> int:
3737

3838
_, _, artifact_text = get(f"/.well-known/hyperfly/{offered}")
3939
artifact = json.loads(artifact_text)
40-
codec = compile_ir(artifact["ir"], plan=artifact["plan"]["layout"], profile=artifact.get("profile") and {
41-
"version": 1,
42-
"shared": artifact["profile"],
43-
})
40+
shared = artifact.get("profile")
41+
profile = None
42+
if shared:
43+
v2 = any("grammar" in c or "derived" in c for c in shared["columns"])
44+
profile = {"version": 2 if v2 else 1, "shared": shared}
45+
codec = compile_ir(artifact["ir"], plan=artifact["plan"]["layout"], profile=profile)
4446
assert codec.fingerprint == offered, "a client must verify what it was handed"
4547
registry.add(codec)
4648
print(f"2. fetched artifact, derived fingerprint matches: {codec.fingerprint[:8]}")

‎apps/interop/server.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,11 @@ const sample = (n: number) => ({
3636
});
3737

3838
const profile = train(toIR(EventResponse), Array.from({ length: 20 }, (_, i) => sample(10 + i)));
39+
// the profiled codec first: negotiation offers it, so the client's bootstrap
40+
// walks the full path — fetch a v2 profiled artifact, re-derive its fingerprint
3941
const registry = new CodecRegistry([
40-
compile(EventResponse, { plan: "columnar" }) as never,
4142
compile(EventResponse, { plan: "columnar", profile }) as never,
43+
compile(EventResponse, { plan: "columnar" }) as never,
4244
]);
4345

4446
const port = Number(process.env.PORT ?? 8787);

‎apps/web/app/benchmark.tsx‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,11 @@ const PAYLOADS: Payload[] = [
2525
{ label: "JSON + gzip", bytes: 2503, kind: "generic" },
2626
{ label: "JSON + Brotli — edge q4", bytes: 2512, kind: "generic" },
2727
{ label: "Protobuf", bytes: 7190, kind: "binary" },
28-
{ label: "Hyperfly", bytes: 2083, kind: "hyperfly" },
29-
{ label: "Hyperfly + Brotli", bytes: 2030, kind: "profile" },
30-
{ label: "Hyperfly Profiled", bytes: 795, kind: "full" },
28+
{ label: "Hyperfly", bytes: 2013, kind: "hyperfly" },
29+
{ label: "Hyperfly + Brotli", bytes: 1974, kind: "profile" },
30+
{ label: "Hyperfly Profiled", bytes: 574, kind: "full" },
3131
],
32-
note: "An audit log repeats itself across requests, not within one: the same user agents, the same actor emails, the same resource ids, request after request. A compressor only ever sees one response and has to rediscover them every time — which is why Brotli takes 55 bytes off and the profile takes 1 231. It learned 692 values once, and pays for itself after ten requests.",
32+
note: "An audit log repeats itself across requests, not within one — the same actors, the same user agents, request after request — and its ids are machine-made: evt_, a sequence number, eight hex digits. The profile learns the recurring values as a dictionary, the id shape as a grammar whose lanes ship the sequence as deltas and the hex at four bits a digit, and actorEmail as a function of actorId, which then costs nothing at all. Brotli takes 39 bytes off this route; the profile takes 1 439. Adding Brotli on top of the profiled stream makes it four bytes larger — at this density there is nothing left for it to find.",
3333
},
3434
{
3535
route: "GET /v1/devices",
@@ -39,11 +39,11 @@ const PAYLOADS: Payload[] = [
3939
{ label: "JSON + gzip", bytes: 1473, kind: "generic" },
4040
{ label: "JSON + Brotli — edge q4", bytes: 1422, kind: "generic" },
4141
{ label: "Protobuf", bytes: 2007, kind: "binary" },
42-
{ label: "Hyperfly", bytes: 807, kind: "hyperfly" },
43-
{ label: "Hyperfly + Brotli", bytes: 753, kind: "profile" },
44-
{ label: "Hyperfly Profiled", bytes: 576, kind: "full" },
42+
{ label: "Hyperfly", bytes: 755, kind: "hyperfly" },
43+
{ label: "Hyperfly + Brotli", bytes: 725, kind: "profile" },
44+
{ label: "Hyperfly Profiled", bytes: 542, kind: "full" },
4545
],
46-
note: "An enum with six members is an index, not a string. Bounded integers ship as offsets from their declared minimum and booleans pack into bitmaps — that is the first row, before anything has been compressed or learned. The profile then learns the fleet: the device ids that recur on every page.",
46+
note: "An enum with six members is three bits, not a string and not a byte. Integers ship bit-packed against the narrowest frame the column needs, outliers pay for themselves alone, and booleans pack into bitmaps — that is the first row, before anything has been learned. The profile then learns the fleet: the device ids that recur on every page.",
4747
},
4848
{
4949
route: "GET /v1/orders/:id",
@@ -55,7 +55,7 @@ const PAYLOADS: Payload[] = [
5555
{ label: "Protobuf", bytes: 388, kind: "binary" },
5656
{ label: "Hyperfly", bytes: 271, kind: "hyperfly" },
5757
{ label: "Hyperfly + Brotli", bytes: 273, kind: "profile" },
58-
{ label: "Hyperfly Profiled", bytes: 187, kind: "full" },
58+
{ label: "Hyperfly Profiled", bytes: 180, kind: "full" },
5959
],
6060
note: "The single-entity response, and the case a general compressor handles worst: under a kilobyte there is nothing yet to build a window from. Brotli actually costs two bytes here rather than saving any — at this size its framing outweighs what it finds. What does work is knowing the catalogue in advance.",
6161
},
@@ -67,9 +67,9 @@ const PAYLOADS: Payload[] = [
6767
{ label: "JSON + gzip", bytes: 2307, kind: "generic" },
6868
{ label: "JSON + Brotli — edge q4", bytes: 2294, kind: "generic" },
6969
{ label: "Protobuf", bytes: 4396, kind: "binary" },
70-
{ label: "Hyperfly", bytes: 1882, kind: "hyperfly" },
71-
{ label: "Hyperfly + Brotli", bytes: 1877, kind: "profile" },
72-
{ label: "Hyperfly Profiled", bytes: 1511, kind: "full" },
70+
{ label: "Hyperfly", bytes: 1871, kind: "hyperfly" },
71+
{ label: "Hyperfly + Brotli", bytes: 1868, kind: "profile" },
72+
{ label: "Hyperfly Profiled", bytes: 1466, kind: "full" },
7373
],
7474
note: "Prose is the hard case: the bodies are genuinely new every time and nothing can invent redundancy that is not there. What does recur are the authors, so that is what the profile takes. This is the narrowest margin on the page, and it is the honest one to look at first.",
7575
},
@@ -83,9 +83,9 @@ const PAYLOADS: Payload[] = [
8383
{ label: "Protobuf", bytes: 2034, kind: "binary" },
8484
{ label: "Hyperfly", bytes: 384, kind: "hyperfly" },
8585
{ label: "Hyperfly + Brotli", bytes: 362, kind: "profile" },
86-
{ label: "Hyperfly Profiled", bytes: 362, kind: "full" },
86+
{ label: "Hyperfly Profiled", bytes: 384, kind: "full" },
8787
],
88-
note: "Timestamps arrive at a constant stride, so the differences between them are identical and pack to a width of zero bits — the column carries its first value and nothing else. Exact-decimal prices travel as integer mantissas bit-packed to the span actually present. The last two rows are identical to the byte, and left in to show it: this route's only string sits outside the array, so training buys nothing at all.",
88+
note: "Timestamps arrive at a constant stride, so the differences between them are identical and pack to a width of zero bits — the column carries its first value and nothing else. Exact-decimal prices travel as integer mantissas bit-packed to the span actually present. The profiled row matches the base row exactly, and is left in to show it: this route's only string sits outside the array, so training buys nothing at all — here the honest win still belongs to Brotli's 22 bytes.",
8989
},
9090
];
9191

‎apps/web/app/page.tsx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,8 +32,8 @@ const STAGES = [
3232
{
3333
index: "03",
3434
name: "Planner",
35-
body: "A Rust planner picks a codec per field: entropy coding, dictionary, delta, columnar layout, or raw bytes when nothing beats raw bytes.",
36-
artifact: "plan · 7 columns · 4 codecs",
35+
body: "The planner picks a codec per column: dictionary, delta, frame-of-reference bit packing, grammar lanes for machine-made ids, one column derived from another — or raw, when nothing beats raw.",
36+
artifact: "plan · 12 columns · 5 codecs",
3737
},
3838
{
3939
index: "04",

‎packages/hyperfly/README.md‎

Lines changed: 22 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -44,12 +44,14 @@ same schema:
4444
const codec = compile(EventResponse, { plan: "columnar" });
4545
```
4646

47-
Timestamps become deltas, exact-decimal numbers travel as integer mantissas, enums
48-
become indices, booleans pack into bitmaps, and text columns deflate together.
47+
Timestamps become deltas, integers bit-pack against the narrowest frame the column
48+
needs, exact-decimal numbers travel as integer mantissas, enums become three-bit
49+
indices, booleans pack into bitmaps, and text columns deflate together.
4950

5051
A **profile** adds what only traffic can teach: the values that recur across
51-
*different* responses, which a compressor never sees because it only ever holds
52-
one.
52+
*different* responses, the shape of machine-made ids (`evt_000a1_9fa3` becomes two
53+
small integers), and columns that are functions of other columns — none of which a
54+
compressor can see, because it only ever holds one response.
5355

5456
```ts
5557
import { train } from "hyperfly";
@@ -59,8 +61,8 @@ const codec = compile(EventResponse, { plan: "columnar", profile });
5961
```
6062

6163
The dictionary is an out-of-band artifact — it ships once, not per request. On an
62-
audit-log route in the repo's benchmark it costs 12 KB and pays for itself after
63-
ten requests.
64+
audit-log route in the repo's benchmark it costs 17 KB and pays for itself after
65+
twelve requests.
6466

6567
## Serving it
6668

@@ -101,15 +103,16 @@ Bytes per message, averaged over 500-message corpora, from `apps/bench` in the r
101103

102104
| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled |
103105
|---|---|---|---|---|---|---|
104-
| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** |
105-
| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** |
106-
| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** |
107-
| single order | 782 | 408 | 388 | 271 | 273 | **188** |
108-
| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 |
109-
110-
Read the spread, not the best row. Profiles are worth 57% on audit logs, where the
111-
same user agents recur on every request, and nothing at all on candles, whose only
112-
string sits outside the array. The corpora are synthetic — shaped like real routes,
106+
| audit events | 12,687 | 2,512 | 7,190 | 2,013 | 1,974 | **574** |
107+
| device telemetry | 7,994 | 1,422 | 2,007 | 755 | 725 | **542** |
108+
| social feed | 6,863 | 2,294 | 4,396 | 1,871 | 1,868 | **1,466** |
109+
| single order | 782 | 408 | 388 | 271 | 273 | **180** |
110+
| OHLCV candles | 3,225 | 842 | 2,034 | 384 | **362** | 384 |
111+
112+
Read the spread, not the best row. Profiles are worth 71% on audit logs, where the
113+
same actors recur on every request and the ids share one machine-made shape, and
114+
nothing at all on candles, whose only string sits outside the array. On the
115+
profiled stream, Brotli now loses bytes on four of five routes. The corpora are synthetic — shaped like real routes,
113116
but not captured from one — and no production traffic has been measured yet.
114117

115118
## Guarantees
@@ -118,8 +121,9 @@ but not captured from one — and no production traffic has been measured yet.
118121
fingerprint. A mismatch fails before the body is parsed; it never misreads.
119122
- **Canonical output.** Decode then re-encode returns identical bytes, so a
120123
response is reproducible.
121-
- **Bounded decoding.** Nesting, item counts and byte lengths are limited; a
122-
declared count must be payable by the bytes still on the wire.
124+
- **Bounded decoding.** Nesting, item counts and byte lengths are limited, and an
125+
amplification policy caps how many rows a hostile byte can demand — a bound
126+
input length alone cannot provide once a constant column packs to zero bits.
123127
- **One wire format.** [TypeScript](https://github.com/eliahilse/hyperfly/tree/main/packages/hyperfly),
124128
[Python](https://github.com/eliahilse/hyperfly/tree/main/python) and
125129
[Rust](https://github.com/eliahilse/hyperfly/tree/main/rust) are verified against
@@ -139,7 +143,7 @@ but not captured from one — and no production traffic has been measured yet.
139143

140144
The authorities are the specifications, not this implementation:
141145
[wire v0](https://github.com/eliahilse/hyperfly/blob/main/spec/wire-v0.md),
142-
[columnar v3](https://github.com/eliahilse/hyperfly/blob/main/spec/plan-columnar-v3.md),
146+
[columnar v5](https://github.com/eliahilse/hyperfly/blob/main/spec/plan-columnar-v5.md),
143147
[negotiation v1](https://github.com/eliahilse/hyperfly/blob/main/spec/negotiation-v1.md).
144148
A future implementation ports against the
145149
[golden vectors](https://github.com/eliahilse/hyperfly/tree/main/spec/vectors), not

‎packages/hyperfly/src/canonical.ts‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import type { IRField, IRNode, LiteralValue } from "./ir.js";
2-
import type { SharedProfile } from "./profile.js";
2+
import type { Derivation, GrammarToken, SharedProfile } from "./profile.js";
33
import { sha256 } from "./sha256.js";
44

55
/** Spec §5: not generic JSON canonicalization — key order and escaping are fixed here. */
@@ -60,12 +60,25 @@ export function serializeNode(node: IRNode): string {
6060

6161
export type PlanLayout = "row" | "columnar";
6262

63-
const PLAN_VERSION: Record<PlanLayout, number> = { row: 1, columnar: 4 };
63+
const PLAN_VERSION: Record<PlanLayout, number> = { row: 1, columnar: 5 };
64+
65+
function serializeToken(token: GrammarToken): string {
66+
if ("lit" in token) return `{"lit":${escapeString(token.lit)}}`;
67+
return `{"num":{"base":${token.num.base},"len":${token.num.len},"case":${escapeString(token.num.case)}}}`;
68+
}
69+
70+
function serializeDerived(derived: Derivation): string {
71+
return `{"source":${derived.source},"values":[${derived.values.map(escapeString).join(",")}]}`;
72+
}
6473

6574
export function serializeShared(shared: SharedProfile): string {
66-
const columns = shared.columns.map(
67-
(c) => `{"leaf":${c.leaf},"dict":[${c.dict.map(escapeString).join(",")}]}`,
68-
);
75+
const columns = shared.columns.map((column) => {
76+
let out = `{"leaf":${column.leaf}`;
77+
if (column.dict) out += `,"dict":[${column.dict.map(escapeString).join(",")}]`;
78+
if (column.grammar) out += `,"grammar":[${column.grammar.map(serializeToken).join(",")}]`;
79+
if (column.derived) out += `,"derived":${serializeDerived(column.derived)}`;
80+
return out + "}";
81+
});
6982
return `{"columns":[${columns.join(",")}]}`;
7083
}
7184

‎packages/hyperfly/src/codec.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ export function compileIR<T = unknown>(ir: IRNode, options: CompileOptions = {})
7373
maxDepth: limits.maxDepth,
7474
maxItems: limits.maxItems,
7575
maxByteLength: limits.maxByteLength,
76+
maxAmplification: limits.maxAmplification,
7677
columnar,
7778
deflate: pack.deflate,
7879
canInflate: pack.inflate !== undefined,

0 commit comments

Comments
 (0)