diff --git a/README.md b/README.md index c50f83c..5394b26 100644 --- a/README.md +++ b/README.md @@ -2,37 +2,67 @@ Binary compression for typed APIs at the edge of entropy. -Typed APIs already know what their data can contain. Production traffic reveals what the data -usually contains. Hyperfly intends to use both to generate specialized binary protocols for a -route, instead of shipping generic JSON over a generic compressor. +Typed APIs already know what their data can contain. Production traffic reveals what +the data usually contains. Hyperfly uses both to compile a binary protocol for one +exact route, instead of shipping generic JSON through a generic compressor. -Pre-release. Nothing here is benchmarked yet. +**Pre-release.** The wire format is specified, three implementations agree on it +byte-for-byte, and the benchmarks below are reproducible — but nothing is published +and nothing is stable. + +## What it costs on the wire + +Bytes per message, averaged over 500-message corpora (`bun run bench`): + +| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled | +|---|---|---|---|---|---|---| +| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** | +| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** | +| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** | +| single order | 782 | 408 | 388 | 271 | 273 | **188** | +| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 | + +Read the spread rather than the best row. Training is worth 57% on audit logs, where +the same user agents recur on every request, and nothing at all on candles, whose +only string sits outside the array. The corpora are synthetic — shaped like real +routes, not captured from one — and no production traffic has been measured yet. ## Repository ``` -spec/ wire format spec + golden vectors (the cross-language authority) -packages/hyperfly TypeScript reference implementation — core codec + zod adapter -apps/web hyperfly.dev — landing page (Next.js on OpenNext / Cloudflare Workers) -apps/bench private benchmark harness (JSON, gzip, Brotli baselines) -packages/lb legacy load balancer, previously published as `hyperfly@0.1.x` -packages/tooling shared eslint and typescript configs +spec/ the authority: wire format, plans, negotiation, golden vectors +packages/hyperfly TypeScript reference implementation, zod adapter, HTTP layer +python/ Python implementation and pydantic adapter +rust/ Rust core +apps/interop a TS server and a Python client over real HTTP, run by CI +apps/bench corpora and the benchmark harness +apps/web hyperfly.dev +packages/lb legacy load balancer, previously published as `hyperfly@0.1.x` ``` +The specifications are normative and the implementations are not. A fourth +implementation ports against [the golden vectors](spec/vectors), not against this +code. + +- [wire v0](spec/wire-v0.md) — envelope, varints, bitmaps, node encodings, canonical + artifacts, decoder limits +- [plan columnar v3](spec/plan-columnar-v3.md) — column layout, delta and XOR and + scaled-decimal numerics, packed text, trained dictionaries +- [negotiation v1](spec/negotiation-v1.md) — how peers agree on binary, how a client + bootstraps, how a profile rotates without a cutover + ## Development ```bash bun install -bun run dev # all apps -bun run build # all packages -bun run check-types +bun run test # TypeScript +pytest python/tests -q # Python +cargo test --manifest-path rust/Cargo.toml +cd apps/bench && bun run bench ``` -## Deploy - -```bash -cd apps/web && bun run deploy -``` +CI runs all three suites, a Python version matrix, and the cross-language interop +exchange on every push. ## License diff --git a/apps/web/app/page.tsx b/apps/web/app/page.tsx index 36156f1..5594b98 100644 --- a/apps/web/app/page.tsx +++ b/apps/web/app/page.tsx @@ -105,9 +105,15 @@ export default function Home() { github - - docs — soon - + + docs + +
diff --git a/packages/hyperfly/README.md b/packages/hyperfly/README.md new file mode 100644 index 0000000..b52712a --- /dev/null +++ b/packages/hyperfly/README.md @@ -0,0 +1,150 @@ +# hyperfly + +Binary compression for typed APIs at the edge of entropy. + +Your schema already fixes every field name, type, and bound. Your traffic already +reveals what the values usually look like. Hyperfly compiles both into a binary +protocol for one exact route — and speaks JSON to anything that hasn't been told. + +> **Pre-release.** The wire format is specified and three implementations agree on +> it byte-for-byte, but nothing is stable yet. Expect breaking changes before 1.0. + +```bash +npm install hyperfly zod +``` + +## Two lines at the boundary + +```ts +import { compile } from "hyperfly/zod"; + +const codec = compile(EventResponse); + +const bytes = codec.encode(response); +const value = codec.decode(bytes); +``` + +`compile` walks your Zod schema, derives a canonical description of it, and +fingerprints that description. Anything the schema already settles — field names, +types, enum members, bounds, optionality — never reaches the wire. + +Schemas it cannot encode fail loudly at compile time, with the path: + +```ts +compile(z.object({ meta: z.record(z.string(), z.unknown()) })); +// UnsupportedSchemaError: $.meta: record has no v0 encoding +``` + +## Columns and profiles + +Arrays of records encode far better column-wise, which is a different plan for the +same schema: + +```ts +const codec = compile(EventResponse, { plan: "columnar" }); +``` + +Timestamps become deltas, exact-decimal numbers travel as integer mantissas, enums +become indices, booleans pack into bitmaps, and text columns deflate together. + +A **profile** adds what only traffic can teach: the values that recur across +*different* responses, which a compressor never sees because it only ever holds +one. + +```ts +import { train } from "hyperfly"; + +const profile = train(toIR(EventResponse), lastWeeksResponses); +const codec = compile(EventResponse, { plan: "columnar", profile }); +``` + +The dictionary is an out-of-band artifact — it ships once, not per request. On an +audit-log route in the repo's benchmark it costs 12 KB and pays for itself after +ten requests. + +## Serving it + +A peer decodes only an artifact it holds, so that has to be established before any +bytes are sent. `hyperfly/http` implements the negotiation: + +```ts +import { CodecRegistry } from "hyperfly"; +import { discovery, respond } from "hyperfly/http"; + +const registry = new CodecRegistry([codec, previousCodec]); + +export default { + fetch(request: Request) { + return ( + discovery(request, registry) ?? // .well-known artifact serving + respond(request, payload, registry) // binary if the client can read it, else JSON + ); + }, +}; +``` + +A client that holds nothing gets JSON plus a `Hyperfly-Offer` naming an artifact it +could fetch; once it has it, the same route answers in binary. There is no failure +mode where the response is unreadable. + +Registering the outgoing codec alongside the incoming one is what makes retraining +safe: a new profile is a new fingerprint, so a deployment holding only one codec +turns every rollout into a cutover. + +Works anywhere `Request`/`Response` do — Hono, Cloudflare Workers, Bun.serve, Deno, +Next route handlers. For other stacks, `negotiate()` and `encodeFor()` take headers +and return a decision. + +## What it costs on the wire + +Bytes per message, averaged over 500-message corpora, from `apps/bench` in the repo: + +| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled | +|---|---|---|---|---|---|---| +| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** | +| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** | +| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** | +| single order | 782 | 408 | 388 | 271 | 273 | **188** | +| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 | + +Read the spread, not the best row. Profiles are worth 57% on audit logs, where the +same user agents recur on every request, and nothing at all on candles, whose only +string sits outside the array. The corpora are synthetic — shaped like real routes, +but not captured from one — and no production traffic has been measured yet. + +## Guarantees + +- **Exact-schema compatibility.** Any change to the schema or plan is a new + fingerprint. A mismatch fails before the body is parsed; it never misreads. +- **Canonical output.** Decode then re-encode returns identical bytes, so a + response is reproducible. +- **Bounded decoding.** Nesting, item counts and byte lengths are limited; a + declared count must be payable by the bytes still on the wire. +- **One wire format.** [TypeScript](https://github.com/eliahilse/hyperfly/tree/main/packages/hyperfly), + [Python](https://github.com/eliahilse/hyperfly/tree/main/python) and + [Rust](https://github.com/eliahilse/hyperfly/tree/main/rust) are verified against + the same golden vectors, and CI runs a TypeScript server against a Python client + over real HTTP on every push. + +## Reference + +| | | +|---|---| +| `compile(schema, options?)` | Zod schema → codec. `plan`, `profile`, `limits`, `pack`, `validate`. | +| `compileIR(ir, options?)` | Same, from a canonical IR directly. | +| `train(ir, samples, options?)` | Sampled responses → profile. Non-normative. | +| `CodecRegistry` | Codecs by fingerprint; what makes rotation safe. | +| `negotiate` · `respond` · `discovery` · `readBody` | HTTP integration. | +| `codec.fingerprint` · `codec.artifact` | What identifies and describes a codec. | + +The authorities are the specifications, not this implementation: +[wire v0](https://github.com/eliahilse/hyperfly/blob/main/spec/wire-v0.md), +[columnar v3](https://github.com/eliahilse/hyperfly/blob/main/spec/plan-columnar-v3.md), +[negotiation v1](https://github.com/eliahilse/hyperfly/blob/main/spec/negotiation-v1.md). +A future implementation ports against the +[golden vectors](https://github.com/eliahilse/hyperfly/tree/main/spec/vectors), not +against this code. + +## License + +[Apache License 2.0](./LICENSE) diff --git a/packages/hyperfly/package.json b/packages/hyperfly/package.json index c4d33a1..7f6e033 100644 --- a/packages/hyperfly/package.json +++ b/packages/hyperfly/package.json @@ -32,7 +32,8 @@ "build": "tsc -p tsconfig.build.json", "check-types": "tsc --noEmit", "lint": "eslint --max-warnings 0", - "test": "bun test" + "test": "bun test", + "prepublishOnly": "bun run lint && bun run check-types && bun test && bun run build" }, "peerDependencies": { "zod": "^4.0.0" @@ -49,5 +50,25 @@ "eslint": "^9.39.1", "typescript": "5.9.2", "zod": "^4.0.0" + }, + "publishConfig": { + "access": "public", + "provenance": true + }, + "keywords": [ + "binary", + "compression", + "serialization", + "codec", + "zod", + "schema", + "protobuf", + "msgpack", + "cbor", + "api" + ], + "homepage": "https://hyperfly.dev", + "bugs": { + "url": "https://github.com/eliahilse/hyperfly/issues" } } diff --git a/packages/hyperfly/test/fuzz.test.ts b/packages/hyperfly/test/fuzz.test.ts new file mode 100644 index 0000000..059914c --- /dev/null +++ b/packages/hyperfly/test/fuzz.test.ts @@ -0,0 +1,197 @@ +import { describe, expect, test } from "bun:test"; +import { compileIR, HyperflyError, type IRNode } from "../src/index.js"; + +/** + * The decoder is the one component that eats bytes from the network, so the property + * it must hold is narrow and absolute: for ANY input it either throws a typed + * HyperflyError, or it returns a value. Never a TypeError, a RangeError, an + * out-of-memory, or a hang. + * + * When it does return a value, the guarantee is normalization, not byte identity. + * Plan §4 makes canonicality an ENCODER obligation and lets decoders accept any + * valid mode, so a hand-made non-canonical input re-encodes to different bytes by + * design — the first run of this fuzzer found exactly that and it is correct. + * What must hold is that re-encoding is stable: the normalized bytes decode to the + * same value and encode to themselves. + */ + +function mulberry32(seed: number): () => number { + let a = seed >>> 0; + return () => { + a |= 0; + a = (a + 0x6d2b79f5) | 0; + let t = Math.imul(a ^ (a >>> 15), 1 | a); + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +const SCHEMAS: IRNode[] = [ + { kind: "int" }, + { kind: "int", min: 0, max: 1000 }, + { kind: "float64" }, + { kind: "string" }, + { kind: "bytes" }, + { kind: "enum", members: ["a", "b", "c"] }, + { kind: "nullable", inner: { kind: "string" } }, + { kind: "array", element: { kind: "int" } }, + { kind: "array", element: { kind: "string" }, length: 3 }, + { + kind: "struct", + fields: [ + { name: "a", type: { kind: "int" } }, + { name: "b", type: { kind: "string" }, optional: true }, + { name: "c", type: { kind: "bool" }, nullable: true }, + { name: "d", type: { kind: "float64" }, optional: true, nullable: true }, + ], + }, + { + kind: "array", + element: { + kind: "struct", + fields: [ + { name: "t", type: { kind: "int", min: 0 } }, + { name: "v", type: { kind: "float64" } }, + { name: "s", type: { kind: "string" }, optional: true }, + { name: "e", type: { kind: "enum", members: ["x", "y"] }, nullable: true }, + { name: "b", type: { kind: "bool" } }, + ], + }, + }, + { + kind: "array", + element: { + kind: "struct", + fields: [ + { name: "n", type: { kind: "struct", fields: [{ name: "deep", type: { kind: "string" } }] } }, + { name: "k", type: { kind: "literal", value: "fixed" } }, + ], + }, + }, +]; + +const PROFILE = { + version: 1 as const, + shared: { columns: [{ leaf: 2, dict: ["alpha", "beta", "gamma"] }] }, +}; + +/** + * Decode, re-encode, and require the result to be a fixed point: the normalized bytes + * must decode to the same value and encode to themselves. That is the invariant that + * survives non-canonical input, and a decoder that loses or invents information + * breaks it. + */ +function normalizes( + codec: { decodeBody(b: Uint8Array): unknown; encodeBody(v: never): Uint8Array }, + bytes: Uint8Array, +): void { + const value = codec.decodeBody(bytes); + const normalized = codec.encodeBody(value as never); + expect(codec.decodeBody(normalized)).toEqual(value as never); + expect(codec.encodeBody(codec.decodeBody(normalized) as never)).toEqual(normalized); +} + +/** Any throw that is not a HyperflyError is a decoder bug, so report it usefully. */ +function expectTypedFailure(run: () => unknown, context: string): void { + try { + run(); + } catch (err) { + if (err instanceof HyperflyError) return; + throw new Error(`${context}: expected HyperflyError, got ${(err as Error).name}: ${(err as Error).message}`); + } +} + +describe("decoder fuzzing", () => { + test("random bytes never escape the error type", () => { + const rng = mulberry32(0xf0000d); + for (const plan of ["row", "columnar"] as const) { + for (const ir of SCHEMAS) { + const codec = compileIR(ir, { plan }); + for (let i = 0; i < 400; i++) { + const bytes = new Uint8Array(Math.floor(rng() * 48)); + for (let b = 0; b < bytes.length; b++) bytes[b] = Math.floor(rng() * 256); + expectTypedFailure( + () => normalizes(codec, bytes), + `${plan} ${ir.kind} ${Buffer.from(bytes).toString("hex")}`, + ); + } + } + } + }); + + test("mutating valid output never escapes the error type", () => { + const rng = mulberry32(0xbadf00d); + const value = [ + { t: 1, v: 1.5, s: "alpha", e: "x", b: true }, + { t: 2, v: 2.5, e: null, b: false }, + { t: 3, v: 0, s: "gamma", e: "y", b: true }, + ]; + const ir = SCHEMAS[10]!; + + for (const plan of ["row", "columnar"] as const) { + const codec = compileIR(ir, { plan }); + const valid = codec.encodeBody(value); + for (let i = 0; i < 3000; i++) { + const bytes = new Uint8Array(valid); + const mutations = 1 + Math.floor(rng() * 3); + for (let m = 0; m < mutations; m++) { + bytes[Math.floor(rng() * bytes.length)] = Math.floor(rng() * 256); + } + expectTypedFailure( + () => normalizes(codec, bytes), + `${plan} mutated ${Buffer.from(bytes).toString("hex")}`, + ); + } + } + }); + + test("truncation at every offset is handled", () => { + const value = { a: -5, b: "hello", c: null, d: 1.25 }; + for (const plan of ["row", "columnar"] as const) { + const codec = compileIR(SCHEMAS[9]!, { plan }); + const valid = codec.encodeBody(value); + for (let cut = 0; cut < valid.length; cut++) { + expectTypedFailure(() => normalizes(codec, valid.subarray(0, cut)), `${plan} truncated to ${cut}`); + } + // and trailing garbage must be refused rather than ignored + expectTypedFailure( + () => codec.decodeBody(new Uint8Array([...valid, 0x00])), + `${plan} trailing`, + ); + } + }); + + test("a profiled decoder survives arbitrary dictionary codes", () => { + const rng = mulberry32(0x1c7); + const codec = compileIR(SCHEMAS[10]!, { plan: "columnar", profile: PROFILE, pack: false }); + for (let i = 0; i < 2000; i++) { + const bytes = new Uint8Array(Math.floor(rng() * 32)); + for (let b = 0; b < bytes.length; b++) bytes[b] = Math.floor(rng() * 256); + expectTypedFailure(() => codec.decodeBody(bytes), `profiled ${Buffer.from(bytes).toString("hex")}`); + } + }); + + test("hostile envelopes are refused, not misread", () => { + const rng = mulberry32(0xe14e); + const codec = compileIR(SCHEMAS[9]!); + for (let i = 0; i < 2000; i++) { + const bytes = new Uint8Array(Math.floor(rng() * 40)); + for (let b = 0; b < bytes.length; b++) bytes[b] = Math.floor(rng() * 256); + expectTypedFailure(() => codec.decode(bytes), `envelope ${Buffer.from(bytes).toString("hex")}`); + } + }); + + test("a declared count cannot outrun the input", () => { + const codec = compileIR({ kind: "array", element: { kind: "struct", fields: [{ name: "x", type: { kind: "int" } }] } }); + // every varint length that fits the limit, with no body behind it + for (const declared of [0x7f, 0x8001, 0x808001, 0x80808008]) { + const bytes: number[] = []; + let v = declared; + while (v > 0) { + bytes.push(v & 0xff); + v >>>= 8; + } + expectTypedFailure(() => codec.decodeBody(new Uint8Array(bytes.reverse())), `count ${declared}`); + } + }); +}); diff --git a/python/README.md b/python/README.md index 18dd7e3..2c4645a 100644 --- a/python/README.md +++ b/python/README.md @@ -1,16 +1,87 @@ # hyperfly (Python) -Python implementation of the hyperfly wire format, with a Pydantic adapter. -The authorities are `spec/wire-v0.md`, `spec/plan-columnar-v2.md`, and the -golden vectors — this implementation reproduces them byte-for-byte, and the -test suite proves it against the same files the TypeScript reference uses. +Binary compression for typed APIs at the edge of entropy. + +Your Pydantic models already fix every field name, type, and bound. Your traffic +already reveals what the values usually look like. Hyperfly compiles both into a +binary protocol for one exact route — and speaks JSON to anything that hasn't been +told. + +> **Pre-release.** The wire format is specified and three implementations agree on +> it byte-for-byte, but nothing is stable yet. + +```bash +pip install hyperfly[pydantic] +``` + +## Two lines at the boundary ```python from hyperfly.pydantic import compile -codec = compile(CandleResponse) +codec = compile(EventResponse) + data = codec.encode(response) value = codec.decode(data) ``` -Pre-release; nothing is published to PyPI yet. +Anything the model already settles — field names, types, enum members, bounds, +optionality — never reaches the wire. Models it cannot encode fail loudly at compile +time, with the path: + +```python +compile(Model) # UnsupportedSchemaError: $.meta: dict[str, int] has no v0 encoding +``` + +## Columns and profiles + +```python +from hyperfly import train +from hyperfly.pydantic import compile, to_ir + +profile = train(to_ir(EventResponse), last_weeks_responses) +codec = compile(EventResponse, plan="columnar", profile=profile) +``` + +Column layout turns timestamps into deltas, exact-decimal numbers into integer +mantissas, enums into indices and booleans into bitmaps, and deflates text columns +together. A profile adds what only traffic can teach: the values that recur across +*different* responses, which a compressor never sees because it only ever holds one. + +## Serving it + +```python +from hyperfly import CodecRegistry +from hyperfly.http import negotiate, encode_for, serve_artifact + +registry = CodecRegistry([codec, previous_codec]) + +decision = negotiate(request.headers.get("hyperfly-accept"), registry) +body, headers = encode_for(decision, payload) +``` + +A client that holds nothing gets JSON plus a `Hyperfly-Offer` naming an artifact it +could fetch from `serve_artifact`; once it has it, the same route answers in binary. +There is no failure mode where the response is unreadable. + +Registering the outgoing codec alongside the incoming one is what makes retraining +safe: a new profile is a new fingerprint, so a deployment holding only one codec +turns every rollout into a cutover. + +## Conformance + +The authorities are the specifications, not this implementation. This package runs +the same golden vectors as the TypeScript and Rust implementations, and CI runs a +TypeScript server against this Python client over real HTTP on every push. + +```bash +pip install ./python[test] && pytest python/tests -q +``` + +- [wire v0](https://github.com/eliahilse/hyperfly/blob/main/spec/wire-v0.md) +- [columnar v3](https://github.com/eliahilse/hyperfly/blob/main/spec/plan-columnar-v3.md) +- [negotiation v1](https://github.com/eliahilse/hyperfly/blob/main/spec/negotiation-v1.md) + +## License + +[Apache License 2.0](../LICENSE) diff --git a/rust/README.md b/rust/README.md new file mode 100644 index 0000000..8009c2a --- /dev/null +++ b/rust/README.md @@ -0,0 +1,41 @@ +# hyperfly-core + +Rust implementation of the [hyperfly](https://hyperfly.dev) wire format. + +The authorities are `spec/wire-v0.md`, `spec/plan-columnar-v3.md` and the golden +vectors in `spec/vectors/`. This crate is verified against those vectors, the same +ones the TypeScript and Python implementations run, so all three agree byte-for-byte. + +```rust +use hyperfly_core::{Codec, Field, Limits, Node, Plan, Value}; + +let ir = Node::Struct(vec![Field { + name: "id".into(), + ty: Node::Str, + optional: false, + nullable: false, +}]); +let codec = Codec::compile(ir, Plan::Row, Limits::default(), true)?; + +let value = Value::Object(vec![("id".into(), Value::Str("abc".into()))]); +let bytes = codec.encode(&value)?; +assert_eq!(codec.decode(&bytes)?, value); +``` + +`Codec::compile_with_profile` takes a trained dictionary. There is no schema adapter +here yet — the crate takes a canonical IR directly, so a Rust caller builds `Node` +itself or loads an artifact served over `.well-known` by a peer. + +Decoding is bounded: nesting depth, item counts and byte lengths are all limited, a +declared count must be payable by the bytes still on the wire, and every failure is +a typed `Error` rather than a panic. + +```bash +cargo test --manifest-path rust/Cargo.toml +``` + +Pre-release; not yet published to crates.io. + +## License + +[Apache License 2.0](../LICENSE)