Skip to content

Site: three Hyperfly rows, each adding one thing #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eliahilse merged 1 commit into from
Aug 20, 2026
Merged

Site: three Hyperfly rows, each adding one thing #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions apps/bench/src/profiles.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ export interface CorpusResult {
brotli: number;
protobuf: number;
columnar: number;
columnarBrotli: number;
profiled: number;
full: number;
dictEntries: number;
Expand Down Expand Up @@ -60,6 +61,7 @@ export function measureCorpus(suite: CorpusSuite): CorpusResult {
let br = 0;
let proto = 0;
let col = 0;
let colBr = 0;
let prof = 0;
let full = 0;

Expand All @@ -70,7 +72,9 @@ export function measureCorpus(suite: CorpusSuite): CorpusResult {
br += brotli(j, constants.BROTLI_MODE_TEXT);

proto += suite.proto.encode(message).length;
col += columnar.encode(message as never).length;
const c = columnar.encode(message as never);
col += c.length;
colBr += brotli(c, constants.BROTLI_MODE_GENERIC);
const p = profiled.encode(message as never);
prof += p.length;
full += brotli(p, constants.BROTLI_MODE_GENERIC);
Expand Down Expand Up @@ -98,6 +102,7 @@ export function measureCorpus(suite: CorpusSuite): CorpusResult {
brotli: mean(br),
protobuf: mean(proto),
columnar: mean(col),
columnarBrotli: mean(colBr),
profiled: mean(prof),
full: mean(full),
dictEntries: columns.reduce((sum, c) => sum + c.dict.length, 0),
Expand All @@ -120,12 +125,12 @@ export function runProfileSuite(suites: CorpusSuite[] = defaultSuites()): Corpus
const results = suites.map(measureCorpus);

console.log("\ncorpora — per-message averages, profile trained on the route's own traffic");
console.log(" route msgs json gzip br4 proto col prof full dictionary");
console.log(" route msgs json br4 proto col col+br prof full dictionary");
for (const r of results) {
const cell = (v: number) => String(v).padStart(6);
console.log(
` ${r.route.padEnd(17)} ${String(r.messages).padStart(4)} ${cell(r.json)} ${cell(r.gzip)} ` +
`${cell(r.brotli)} ${cell(r.protobuf)} ${cell(r.columnar)} ${cell(r.profiled)} ${cell(r.full)} ` +
` ${r.route.padEnd(17)} ${String(r.messages).padStart(4)} ${cell(r.json)} ` +
`${cell(r.brotli)} ${cell(r.protobuf)} ${cell(r.columnar)} ${cell(r.columnarBrotli)} ${cell(r.profiled)} ${cell(r.full)} ` +
`${r.dictEntries} entries / ${r.dictBytes}B` +
(r.breakEven ? ` (pays for itself after ${r.breakEven} requests)` : ""),
);
Expand Down
40 changes: 20 additions & 20 deletions apps/web/app/benchmark.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,11 @@ const PAYLOADS: Payload[] = [
{ label: "JSON + gzip", bytes: 2503, kind: "generic" },
{ label: "JSON + Brotli — edge q4", bytes: 2512, kind: "generic" },
{ label: "Protobuf", bytes: 7190, kind: "binary" },
{ label: "Hyperfly · columnar", bytes: 2109, kind: "hyperfly" },
{ label: "Hyperfly · profiled", bytes: 896, kind: "profile" },
{ label: "Hyperfly · full", bytes: 823, kind: "full" },
{ label: "Hyperfly", bytes: 2109, kind: "hyperfly" },
{ label: "Hyperfly + Brotli", bytes: 2054, kind: "profile" },
{ label: "Hyperfly Profiled", bytes: 823, kind: "full" },
],
note: "An audit log repeats itself across requests, not within one: the same user agents, the same actor emails, the same resource ids, request after request. A compressor only ever sees one response and has to rediscover them every time. The profile learned 692 values once, and pays for itself after ten requests.",
note: "An audit log repeats itself across requests, not within one: the same user agents, the same actor emails, the same resource ids, request after request. A compressor only ever sees one response and has to rediscover them every time — which is why Brotli takes 55 bytes off and the profile takes 1 231. It learned 692 values once, and pays for itself after ten requests.",
},
{
route: "GET /v1/devices",
Expand All @@ -39,11 +39,11 @@ const PAYLOADS: Payload[] = [
{ label: "JSON + gzip", bytes: 1473, kind: "generic" },
{ label: "JSON + Brotli — edge q4", bytes: 1422, kind: "generic" },
{ label: "Protobuf", bytes: 2007, kind: "binary" },
{ label: "Hyperfly · columnar", bytes: 896, kind: "hyperfly" },
{ label: "Hyperfly · profiled", bytes: 705, kind: "profile" },
{ label: "Hyperfly · full", bytes: 638, kind: "full" },
{ label: "Hyperfly", bytes: 896, kind: "hyperfly" },
{ label: "Hyperfly + Brotli", bytes: 818, kind: "profile" },
{ label: "Hyperfly Profiled", bytes: 638, kind: "full" },
],
note: "An enum with six members is an index, not a string. Bounded integers ship as offsets from their declared minimum and booleans pack into bitmaps — that is the columnar row. The profile then learns the fleet: the device ids that recur on every page.",
note: "An enum with six members is an index, not a string. Bounded integers ship as offsets from their declared minimum and booleans pack into bitmaps — that is the first row, before anything has been compressed or learned. The profile then learns the fleet: the device ids that recur on every page.",
},
{
route: "GET /v1/orders/:id",
Expand All @@ -53,11 +53,11 @@ const PAYLOADS: Payload[] = [
{ label: "JSON + gzip", bytes: 423, kind: "generic" },
{ label: "JSON + Brotli — edge q4", bytes: 408, kind: "generic" },
{ label: "Protobuf", bytes: 388, kind: "binary" },
{ label: "Hyperfly · columnar", bytes: 271, kind: "hyperfly" },
{ label: "Hyperfly · profiled", bytes: 184, kind: "profile" },
{ label: "Hyperfly · full", bytes: 188, kind: "full" },
{ label: "Hyperfly", bytes: 271, kind: "hyperfly" },
{ label: "Hyperfly + Brotli", bytes: 273, kind: "profile" },
{ label: "Hyperfly Profiled", bytes: 188, kind: "full" },
],
note: "The single-entity response, and the case a general compressor handles worst: under a kilobyte there is nothing yet to build a window from. Note that full is larger than profiled here — at 184 bytes, Brotli's framing costs more than it saves, so the right configuration for this route is to skip it.",
note: "The single-entity response, and the case a general compressor handles worst: under a kilobyte there is nothing yet to build a window from. Brotli actually costs two bytes here rather than saving any — at this size its framing outweighs what it finds. What does work is knowing the catalogue in advance.",
},
{
route: "GET /v1/feed",
Expand All @@ -67,11 +67,11 @@ const PAYLOADS: Payload[] = [
{ label: "JSON + gzip", bytes: 2307, kind: "generic" },
{ label: "JSON + Brotli — edge q4", bytes: 2294, kind: "generic" },
{ label: "Protobuf", bytes: 4396, kind: "binary" },
{ label: "Hyperfly · columnar", bytes: 1908, kind: "hyperfly" },
{ label: "Hyperfly · profiled", bytes: 1536, kind: "profile" },
{ label: "Hyperfly · full", bytes: 1535, kind: "full" },
{ label: "Hyperfly", bytes: 1908, kind: "hyperfly" },
{ label: "Hyperfly + Brotli", bytes: 1902, kind: "profile" },
{ label: "Hyperfly Profiled", bytes: 1535, kind: "full" },
],
note: "Prose is the hard case: the bodies are genuinely new every time and nothing can invent redundancy that is not there. What does recur are the authors, so that is what the profile takes. Full and profiled land within a byte of each other because the dictionary already removed what Brotli was living on.",
note: "Prose is the hard case: the bodies are genuinely new every time and nothing can invent redundancy that is not there. What does recur are the authors, so that is what the profile takes. This is the narrowest margin on the page, and it is the honest one to look at first.",
},
{
route: "GET /v1/candles",
Expand All @@ -81,11 +81,11 @@ const PAYLOADS: Payload[] = [
{ label: "JSON + gzip", bytes: 928, kind: "generic" },
{ label: "JSON + Brotli — edge q4", bytes: 842, kind: "generic" },
{ label: "Protobuf", bytes: 2034, kind: "binary" },
{ label: "Hyperfly · columnar", bytes: 496, kind: "hyperfly" },
{ label: "Hyperfly · profiled", bytes: 496, kind: "profile" },
{ label: "Hyperfly · full", bytes: 372, kind: "full" },
{ label: "Hyperfly", bytes: 496, kind: "hyperfly" },
{ label: "Hyperfly + Brotli", bytes: 372, kind: "profile" },
{ label: "Hyperfly Profiled", bytes: 372, kind: "full" },
],
note: "Timestamps become deltas and exact-decimal prices travel as integer mantissas rather than eight raw bytes. The profile changes nothing at all here, and the row is left in to show it: this route's only string sits outside the array, so there is no column for a dictionary to key on.",
note: "Timestamps become deltas and exact-decimal prices travel as integer mantissas rather than eight raw bytes. The last two rows are identical to the byte, and the row is left in to show it: this route's only string sits outside the array, so there is no column for a dictionary to key on and training buys nothing at all.",
},
];

Expand Down
15 changes: 8 additions & 7 deletions apps/web/app/page.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,13 +164,14 @@ export default function Home() {
<Reveal delay={120}>
<p className="footnote">
Bytes per message, averaged over 500 independent responses per route, measured
with the TypeScript reference implementation — reproduce with `bun run bench`. The
profile is trained on the route&apos;s own traffic, which is what a deployment does;
it is an out-of-band artifact, and the repo reports its size and how many requests
it takes to pay for itself (ten for events, thirty-five for orders). The Brotli row
is q4, the level edges actually run on dynamic responses. Protobuf gets proper
enums and int64. Corpora are synthetic but shaped like real routes: a fixed device
fleet, a fixed product catalogue, a recurring cast of authors.
with the TypeScript reference implementation — reproduce with `bun run bench`. Each
Hyperfly row adds exactly one thing: schema-compiled layout, then generic
compression on top of it, then a dictionary trained on the route&apos;s own traffic.
That dictionary is an out-of-band artifact; the repo reports its size and how many
requests it takes to pay for itself — ten for events, thirty-five for orders. The
Brotli rows are q4, the level edges actually run on dynamic responses. Protobuf
gets proper enums and int64. Corpora are synthetic but shaped like real routes: a
fixed device fleet, a fixed product catalogue, a recurring cast of authors.
</p>
</Reveal>
</section>
Expand Down
Loading