Publish groundwork: docs, package metadata, decoder fuzzing

README.md

@@ -2,37 +2,67 @@
 
 Binary compression for typed APIs at the edge of entropy.
 
-Typed APIs already know what their data can contain. Production traffic reveals what the data
-usually contains. Hyperfly intends to use both to generate specialized binary protocols for a
-route, instead of shipping generic JSON over a generic compressor.
+Typed APIs already know what their data can contain. Production traffic reveals what
+the data usually contains. Hyperfly uses both to compile a binary protocol for one
+exact route, instead of shipping generic JSON through a generic compressor.
 
-Pre-release. Nothing here is benchmarked yet.
+**Pre-release.** The wire format is specified, three implementations agree on it
+byte-for-byte, and the benchmarks below are reproducible — but nothing is published
+and nothing is stable.
+
+## What it costs on the wire
+
+Bytes per message, averaged over 500-message corpora (`bun run bench`):
+
+| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled |
+|---|---|---|---|---|---|---|
+| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** |
+| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** |
+| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** |
+| single order | 782 | 408 | 388 | 271 | 273 | **188** |
+| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 |
+
+Read the spread rather than the best row. Training is worth 57% on audit logs, where
+the same user agents recur on every request, and nothing at all on candles, whose
+only string sits outside the array. The corpora are synthetic — shaped like real
+routes, not captured from one — and no production traffic has been measured yet.
 
 ## Repository
 
 ```
-spec/             wire format spec + golden vectors (the cross-language authority)
-packages/hyperfly TypeScript reference implementation — core codec + zod adapter
-apps/web          hyperfly.dev — landing page (Next.js on OpenNext / Cloudflare Workers)
-apps/bench        private benchmark harness (JSON, gzip, Brotli baselines)
-packages/lb       legacy load balancer, previously published as `hyperfly@0.1.x`
-packages/tooling  shared eslint and typescript configs
+spec/              the authority: wire format, plans, negotiation, golden vectors
+packages/hyperfly  TypeScript reference implementation, zod adapter, HTTP layer
+python/            Python implementation and pydantic adapter
+rust/              Rust core
+apps/interop       a TS server and a Python client over real HTTP, run by CI
+apps/bench         corpora and the benchmark harness
+apps/web           hyperfly.dev
+packages/lb        legacy load balancer, previously published as `hyperfly@0.1.x`
 ```
 
+The specifications are normative and the implementations are not. A fourth
+implementation ports against [the golden vectors](spec/vectors), not against this
+code.
+
+- [wire v0](spec/wire-v0.md) — envelope, varints, bitmaps, node encodings, canonical
+  artifacts, decoder limits
+- [plan columnar v3](spec/plan-columnar-v3.md) — column layout, delta and XOR and
+  scaled-decimal numerics, packed text, trained dictionaries
+- [negotiation v1](spec/negotiation-v1.md) — how peers agree on binary, how a client
+  bootstraps, how a profile rotates without a cutover
+
 ## Development
 
 ```bash
 bun install
-bun run dev           # all apps
-bun run build         # all packages
-bun run check-types
+bun run test            # TypeScript
+pytest python/tests -q  # Python
+cargo test --manifest-path rust/Cargo.toml
+cd apps/bench && bun run bench
 ```
 
-## Deploy
-
-```bash
-cd apps/web && bun run deploy
-```
+CI runs all three suites, a Python version matrix, and the cross-language interop
+exchange on every push.
 
 ## License
 

apps/web/app/page.tsx

@@ -105,9 +105,15 @@ export default function Home() {
                 github
                 <span aria-hidden="true">↗</span>
               </a>
-              <span className="button ghost" aria-disabled="true">
-                docs — soon
-              </span>
+              <a
+                className="button"
+                href={`${GITHUB}#readme`}
+                target="_blank"
+                rel="noopener noreferrer"
+              >
+                docs
+                <span aria-hidden="true">↗</span>
+              </a>
             </div>
           </div>
           <div className="hero-rule" aria-hidden="true" />

packages/hyperfly/README.md

@@ -0,0 +1,150 @@
+# hyperfly
+
+Binary compression for typed APIs at the edge of entropy.
+
+Your schema already fixes every field name, type, and bound. Your traffic already
+reveals what the values usually look like. Hyperfly compiles both into a binary
+protocol for one exact route — and speaks JSON to anything that hasn't been told.
+
+> **Pre-release.** The wire format is specified and three implementations agree on
+> it byte-for-byte, but nothing is stable yet. Expect breaking changes before 1.0.
+
+```bash
+npm install hyperfly zod
+```
+
+## Two lines at the boundary
+
+```ts
+import { compile } from "hyperfly/zod";
+
+const codec = compile(EventResponse);
+
+const bytes = codec.encode(response);
+const value = codec.decode(bytes);
+```
+
+`compile` walks your Zod schema, derives a canonical description of it, and
+fingerprints that description. Anything the schema already settles — field names,
+types, enum members, bounds, optionality — never reaches the wire.
+
+Schemas it cannot encode fail loudly at compile time, with the path:
+
+```ts
+compile(z.object({ meta: z.record(z.string(), z.unknown()) }));
+// UnsupportedSchemaError: $.meta: record has no v0 encoding
+```
+
+## Columns and profiles
+
+Arrays of records encode far better column-wise, which is a different plan for the
+same schema:
+
+```ts
+const codec = compile(EventResponse, { plan: "columnar" });
+```
+
+Timestamps become deltas, exact-decimal numbers travel as integer mantissas, enums
+become indices, booleans pack into bitmaps, and text columns deflate together.
+
+A **profile** adds what only traffic can teach: the values that recur across
+*different* responses, which a compressor never sees because it only ever holds
+one.
+
+```ts
+import { train } from "hyperfly";
+
+const profile = train(toIR(EventResponse), lastWeeksResponses);
+const codec = compile(EventResponse, { plan: "columnar", profile });
+```
+
+The dictionary is an out-of-band artifact — it ships once, not per request. On an
+audit-log route in the repo's benchmark it costs 12 KB and pays for itself after
+ten requests.
+
+## Serving it
+
+A peer decodes only an artifact it holds, so that has to be established before any
+bytes are sent. `hyperfly/http` implements the negotiation:
+
+```ts
+import { CodecRegistry } from "hyperfly";
+import { discovery, respond } from "hyperfly/http";
+
+const registry = new CodecRegistry([codec, previousCodec]);
+
+export default {
+  fetch(request: Request) {
+    return (
+      discovery(request, registry) ??      // .well-known artifact serving
+      respond(request, payload, registry)  // binary if the client can read it, else JSON
+    );
+  },
+};
+```
+
+A client that holds nothing gets JSON plus a `Hyperfly-Offer` naming an artifact it
+could fetch; once it has it, the same route answers in binary. There is no failure
+mode where the response is unreadable.
+
+Registering the outgoing codec alongside the incoming one is what makes retraining
+safe: a new profile is a new fingerprint, so a deployment holding only one codec
+turns every rollout into a cutover.
+
+Works anywhere `Request`/`Response` do — Hono, Cloudflare Workers, Bun.serve, Deno,
+Next route handlers. For other stacks, `negotiate()` and `encodeFor()` take headers
+and return a decision.
+
+## What it costs on the wire
+
+Bytes per message, averaged over 500-message corpora, from `apps/bench` in the repo:
+
+| route | JSON | JSON+Brotli | Protobuf | Hyperfly | + Brotli | Profiled |
+|---|---|---|---|---|---|---|
+| audit events | 12,687 | 2,512 | 7,190 | 2,109 | 2,054 | **823** |
+| device telemetry | 7,994 | 1,422 | 2,007 | 896 | 818 | **638** |
+| social feed | 6,863 | 2,294 | 4,396 | 1,908 | 1,902 | **1,535** |
+| single order | 782 | 408 | 388 | 271 | 273 | **188** |
+| OHLCV candles | 3,225 | 842 | 2,034 | 496 | **372** | 372 |
+
+Read the spread, not the best row. Profiles are worth 57% on audit logs, where the
+same user agents recur on every request, and nothing at all on candles, whose only
+string sits outside the array. The corpora are synthetic — shaped like real routes,
+but not captured from one — and no production traffic has been measured yet.
+
+## Guarantees
+
+- **Exact-schema compatibility.** Any change to the schema or plan is a new
+  fingerprint. A mismatch fails before the body is parsed; it never misreads.
+- **Canonical output.** Decode then re-encode returns identical bytes, so a
+  response is reproducible.
+- **Bounded decoding.** Nesting, item counts and byte lengths are limited; a
+  declared count must be payable by the bytes still on the wire.
+- **One wire format.** [TypeScript](https://github.com/eliahilse/hyperfly/tree/main/packages/hyperfly),
+  [Python](https://github.com/eliahilse/hyperfly/tree/main/python) and
+  [Rust](https://github.com/eliahilse/hyperfly/tree/main/rust) are verified against
+  the same golden vectors, and CI runs a TypeScript server against a Python client
+  over real HTTP on every push.
+
+## Reference
+
+| | |
+|---|---|
+| `compile(schema, options?)` | Zod schema → codec. `plan`, `profile`, `limits`, `pack`, `validate`. |
+| `compileIR(ir, options?)` | Same, from a canonical IR directly. |
+| `train(ir, samples, options?)` | Sampled responses → profile. Non-normative. |
+| `CodecRegistry` | Codecs by fingerprint; what makes rotation safe. |
+| `negotiate` · `respond` · `discovery` · `readBody` | HTTP integration. |
+| `codec.fingerprint` · `codec.artifact` | What identifies and describes a codec. |
+
+The authorities are the specifications, not this implementation:
+[wire v0](https://github.com/eliahilse/hyperfly/blob/main/spec/wire-v0.md),
+[columnar v3](https://github.com/eliahilse/hyperfly/blob/main/spec/plan-columnar-v3.md),
+[negotiation v1](https://github.com/eliahilse/hyperfly/blob/main/spec/negotiation-v1.md).
+A future implementation ports against the
+[golden vectors](https://github.com/eliahilse/hyperfly/tree/main/spec/vectors), not
+against this code.
+
+## License
+
+[Apache License 2.0](./LICENSE)

packages/hyperfly/package.json

@@ -32,7 +32,8 @@
     "build": "tsc -p tsconfig.build.json",
     "check-types": "tsc --noEmit",
     "lint": "eslint --max-warnings 0",
-    "test": "bun test"
+    "test": "bun test",
+    "prepublishOnly": "bun run lint && bun run check-types && bun test && bun run build"
   },
   "peerDependencies": {
     "zod": "^4.0.0"
@@ -49,5 +50,25 @@
     "eslint": "^9.39.1",
     "typescript": "5.9.2",
     "zod": "^4.0.0"
+  },
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  },
+  "keywords": [
+    "binary",
+    "compression",
+    "serialization",
+    "codec",
+    "zod",
+    "schema",
+    "protobuf",
+    "msgpack",
+    "cbor",
+    "api"
+  ],
+  "homepage": "https://hyperfly.dev",
+  "bugs": {
+    "url": "https://github.com/eliahilse/hyperfly/issues"
   }
 }