diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f9ee485..5657254 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,7 +21,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v6 with: - node-version: 20 + node-version: 24 cache: npm - name: Install dependencies diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 95e18af..c5da661 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -16,14 +16,14 @@ jobs: - name: Setup Node uses: actions/setup-node@v6 with: - node-version: 20 + node-version: 24 cache: npm - name: Install dependencies run: npm ci - name: Install Playwright browsers - run: npx playwright install --with-deps chromium + run: npx playwright install --with-deps chromium webkit - name: Run checks and browser tests env: diff --git a/AGENTS.md b/AGENTS.md index 84e8919..1c40b0f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,10 +23,10 @@ Core product traits right now: Treat these as core constraints unless the owner explicitly changes the product direction. - The app is a single exported client-side shell, not a backend product. -- Artifact payloads live in the URL fragment, using the compact `#` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3. Legacy `#agent-render=v1..` links (arx-family carry an extra `.` segment) still decode but are no longer emitted. +- Artifact payloads live in the URL fragment, using the compact `#` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4. Legacy `#agent-render=v1..` links (arx-family carry an extra `.` segment) still decode but are no longer emitted. - The deployed host should not receive artifact contents as part of the initial page request. - Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`. -- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`. +- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`. - The product is zero-retention by host design, not secret-safe in an absolute sense. - Links may still leak through browser history, copied URLs, screenshots, and any future client-side analytics. @@ -84,9 +84,10 @@ The fragment transport is part of the product surface, not an implementation det Current rules: - fragment key: `agent-render` (legacy decode path only; the compact form has no key) -- emitted format: compact `#`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one) +- emitted format: compact `#`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one) - legacy format (still decodable, no longer emitted): `agent-render=v1..` for `plain|lz|deflate`, `agent-render=v1.arx..` for `arx`, `agent-render=v1.arx2..` for `arx2`, and `agent-render=v1.arx3..` for `arx3` -- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3` +- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4` +- `arx4` payloads carry one extra leading char after the tag, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the context mixer ran before the payload; `m`/`c`/`j` additionally need `/arx4-priors.json`, and an encoder that cannot load it emits `s` instead - fragment size budget: `8192` characters - decoded payload budget: `200000` characters - Discord markdown link limit: `2000` characters for the full formatted `[label](url)` string; `createGeneratedArtifactLink*` returns `discordMarkdownLinkWarning` when exceeded @@ -123,6 +124,8 @@ If you change the payload contract, update the code, docs, examples, and the Ope - `src/lib/payload/schema.ts` - type surface, limits, fragment key, supported kinds/codecs - `src/lib/payload/fragment.ts` - encode/decode logic and transport behavior - `src/lib/payload/arx-codec.ts` - arx/arx2/arx3 codecs: dictionary substitution, tuple overlay, brotli, base76/base1k/baseBMP/base64url encoding +- `src/lib/payload/arx4-codec.ts` - arx4 codec: the arx3 stages with brotli replaced by a deterministic integer context mixer, plus the curated priors it primes on +- `public/arx4-priors.json` - curated per-kind arx4 priming corpora (and `.br` pre-compressed variant), regenerated by `scripts/build-arx4-priors.mjs` - `public/arx-dictionary.json` - shared substitution dictionary for the arx codec (served as a static endpoint) - `public/arx-dictionary.json.br` - pre-compressed brotli variant of the dictionary - `public/arx2-dictionary.json` - overlay substitution dictionary for arx2 tuple-envelope transport diff --git a/README.md b/README.md index 908392a..36bdc21 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ Built for the OpenClaw ecosystem, `agent-render` focuses on fragment-based shari ## Status - Markdown, code, diff, CSV, and JSON all render in the static shell -- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`, with automatic shortest-fragment selection across available wire formats -- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx3` reuses those proven bytes and optimizes for compact visible Unicode fragments +- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`, with automatic shortest-fragment selection across available wire formats +- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx3` reuses those proven bytes and optimizes for compact visible Unicode fragments; `arx4` adds the curated context-mixer priors at `/arx4-priors.json` with a pre-compressed `/arx4-priors.json.br` variant - The viewer toolbar copies artifact bodies to the clipboard, downloads them as files, and (for markdown) supports browser print-to-PDF - Deployment target: static hosting, including Cloudflare Pages diff --git a/docs/architecture.md b/docs/architecture.md index f76698e..1986f30 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -99,8 +99,9 @@ The fragment protocol keeps the JSON envelope stable and treats compression stri - `arx` applies domain-dictionary substitution, brotli compression (quality 11), and binary-to-text encoding for best-in-class compression. Four wire shapes are candidates: base76 (ASCII, 77 fragment-safe chars), base64url (RFC 4648 `A-Za-z0-9-_` with a `B.` prefix for detection), base1k (Unicode, 1774 chars from U+00A1–U+07FF), and baseBMP (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). The async encoder tries all four and picks the shortest **transport** length (percent-encoded UTF-8 length for non-ASCII), so base64url can win over Unicode encodings on chat-style surfaces. baseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes, achieving ~70% smaller fragments than deflate on typical payloads (~6.1x compression ratio for 8k markdown). Full pipeline timing is on the order of ~8–14ms for 8k payloads depending on the wire encoding. The substitution dictionary is served as a static file at `/arx-dictionary.json` so agents can fetch it for local compression; a pre-compressed `/arx-dictionary.json.br` variant is also available. The viewer tries the pre-compressed dictionary first on default ARX-family loads, falls back to the JSON file, and only loads external dictionaries when an ARX/ARX2/ARX3 encode or decode path needs them. - `arx2` keeps the arx compression stack but replaces the JSON envelope with a compact tuple envelope and applies `/arx2-dictionary.json` as an overlay before the shared arx dictionary. The viewer tries `/arx2-dictionary.json.br` first for default overlay loads and falls back to JSON. It is emitted with the compact `b` tag (which identifies the codec but does not carry a dictionary version — it implies the current pinned dictionary) and decodes back to the standard envelope before validation/rendering. - `arx3` uses the same tuple envelope, overlay dictionary, shared arx dictionary, and brotli bytes as arx2, then allows the dense baseBMP wire to win by decoded visible character length. This deliberately optimizes copyable visible URL length for trusted Unicode-preserving surfaces; it is not a stronger compressed-byte format than arx2. +- `arx4` keeps the arx3 tuple envelope, overlay dictionary, shared arx dictionary and baseBMP selection rule, and replaces brotli with a deterministic integer context mixer (`arx4-codec.ts`). It is emitted with the compact `e` tag and carries one extra leading payload char, the prior id, naming the priming corpus both sides run before the payload. See `docs/payload-format.md` for the prior ids and the `/arx4-priors.json` asset. - packed wire mode (`p: 1`) shortens transport keys before compression, then unpacks back to the standard envelope during decode -- automatic async codec selection tries `arx3 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx3 use tuple envelopes +- automatic async codec selection tries `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx3/arx4 use tuple envelopes - sync codec selection (used by examples and legacy paths) tries `deflate -> lz -> plain` - decode enforces both visible fragment length and decoded payload size ceilings before UI rendering; arx/arx2/arx3 Brotli decompression uses a streaming output cap before final JSON or tuple parsing - invalid bundle state is normalized or rejected before renderers mount diff --git a/docs/payload-format.md b/docs/payload-format.md index 4faf0df..698db11 100644 --- a/docs/payload-format.md +++ b/docs/payload-format.md @@ -15,6 +15,7 @@ Payload contents are untrusted user content. Viewers, agents, and automations sh #a (arx) #b (arx2) #c (arx3) +#e (arx4) ``` The compact fragment is a single codec tag char followed by the payload. The tag encodes the codec so unsupported formats fail cleanly; the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and rejects a newer one). The legacy `#agent-render=v1..` form (arx-family carry an extra `.` segment) still decodes for back-compatibility but is no longer emitted. Fragment URLs can look long because they carry the artifact payload in the browser-only fragment instead of sending it to the host during the page request. @@ -27,6 +28,7 @@ Supported codecs: - `arx` - domain-dictionary substitution + brotli (quality 11) + binary-to-text encoding. The compact `a` tag identifies the arx codec but does not carry a dictionary version — it implies the build's current pinned dictionary (the build refuses to decode a forward-incompatible newer dictionary). Four wire shapes are tried and the shortest **transport** size wins (see `computeTransportLength` in `fragment.ts` — non-ASCII Unicode may count longer after percent-encoding): **base76** (ASCII-only, 77 fragment-safe chars), **base64url** (standard RFC 4648 alphabet `A-Za-z0-9-_`, no padding, prefixed with `B.` for detection), **base1k** (Unicode, 1774 chars from U+00A1–U+07FF), and **baseBMP** (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). BaseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes. BaseBMP payloads are prefixed with a U+FFF0 marker for detection. The viewer’s `arxDecompress` auto-detects the wire shape (including the rare case where a base76 length prefix is also `B.` — it tries base64url first and falls back to base76 if Brotli fails). The substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the viewer tries the `.br` file first on default loads and falls back to JSON. The arx2 overlay dictionary follows the same `.br`-then-JSON default load pattern. - `arx2` - tuple-envelope transport + arx2 overlay substitution + the shared arx dictionary + brotli (quality 11) + the same four binary-to-text wire shapes. The compact `b` tag identifies arx2 but does not carry a dictionary version — it implies the current pinned shared arx dictionary and arx2 overlay. Existing `arx` links remain valid; async auto-selection keeps arx2 as the conservative transport-measured tuple codec. - `arx3` - the same tuple envelope, overlay substitution, shared arx dictionary, and brotli bytes as arx2, with a different selection rule: baseBMP may win by decoded visible character length instead of conservative percent-encoded transport length. This is the compact visible URL mode for trusted surfaces that preserve Unicode fragments. If a platform rewrites, truncates, or previews links aggressively, prefer arx2/base64url or UUID mode instead. +- `arx4` - the arx3 tuple/overlay/dictionary stages and the arx3 baseBMP selection rule, with brotli replaced by a deterministic integer context mixer (`arx4-codec.ts`). The payload carries one extra leading char, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the coder ran before the payload; an unrecognized prior id is a decode error. Every prior starts from the pinned dictionary slot text, so the `e` tag implies that dictionary twice over; the `m`, `c` and `j` priors append a curated per-kind corpus from `/arx4-priors.json` (pre-compressed `/arx4-priors.json.br` tried first), which the viewer fetches lazily on the first arx4 encode or decode. The `s` id is that shared dictionary text alone, and `n` runs the mixer cold with no prior at all. If the asset is unavailable the encoder falls back to the `s` prior and emits an `s` id, so link creation never blocks; decoding a fragment that names a curated prior fails cleanly instead of coding against a different corpus than the id names. It codes ~10% smaller than arx3 on the sample corpus and is roughly 100x slower, which is why the whole arx family is async-only. The encoder now also supports a packed wire representation (`p: 1`) that shortens key names before compression. Packed mode is transport-only; decoded envelopes normalize back to the standard shape. @@ -106,7 +108,7 @@ Tuple fields: - Larger payloads should fail with a clear error before rendering - Compression is selected automatically across packed/non-packed candidates; arx and arx2 optimize conservative transport length, while arx3 optimizes compact visible length for its dense Unicode wire - Default sync codec priority is `deflate -> lz -> plain` -- Default async codec priority is `arx3 -> arx2 -> arx -> deflate -> lz -> plain` +- Default async codec priority is `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain` - Optional budget-aware encoding can target strict limits and returns the shortest fragment when none fit - `createGeneratedArtifactLink` / `createGeneratedArtifactLinkAsync` return `url`, `markdownLink` (ready to paste verbatim in chat), `markdownLinkLength`, and `discordMarkdownLinkWarning` so agents do not need to reconstruct `[label](url)` themselves diff --git a/docs/url-fragments.md b/docs/url-fragments.md index 1ec93dd..717aad1 100644 --- a/docs/url-fragments.md +++ b/docs/url-fragments.md @@ -22,9 +22,10 @@ The tag char identifies the codec: #a (arx) #b (arx2) #c (arx3) +#e (arx4) ``` -For `arx`, `arx2`, and `arx3`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version. +For `arx`, `arx2`, `arx3`, and `arx4`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version. Older links may use the legacy shape, which the viewer still decodes: diff --git a/package-lock.json b/package-lock.json index d73befc..3179e52 100644 --- a/package-lock.json +++ b/package-lock.json @@ -45,6 +45,7 @@ "@types/papaparse": "^5.3.16", "@types/react": "^19.0.10", "@types/react-dom": "^19.0.4", + "esbuild": "^0.27.3", "eslint": "^9.22.0", "eslint-config-next": "15.5.18", "jsdom": "^28.1.0", @@ -9639,7 +9640,6 @@ "version": "2.3.2", "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz", "integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==", - "dev": true, "hasInstallScript": true, "license": "MIT", "optional": true, diff --git a/package.json b/package.json index 7468651..d19d48d 100644 --- a/package.json +++ b/package.json @@ -32,7 +32,7 @@ "test:e2e": "env -u NO_COLOR playwright test", "test:e2e:update": "env -u NO_COLOR playwright test --update-snapshots", "test:ci": "npm run lint && npm run test && npm run typecheck && npm run test:e2e && npm run check:build-budgets", - "test:browsers": "playwright install chromium", + "test:browsers": "playwright install chromium webkit", "bench:codecs": "node scripts/bench-codecs.mjs", "bench:codecs:update": "node scripts/bench-codecs.mjs --write-baseline", "assets:compress": "node scripts/compress-dictionary.mjs", @@ -82,6 +82,7 @@ "@types/papaparse": "^5.3.16", "@types/react": "^19.0.10", "@types/react-dom": "^19.0.4", + "esbuild": "^0.27.3", "eslint": "^9.22.0", "eslint-config-next": "15.5.18", "jsdom": "^28.1.0", diff --git a/playwright.config.ts b/playwright.config.ts index 3ea64fa..d301e93 100644 --- a/playwright.config.ts +++ b/playwright.config.ts @@ -35,5 +35,12 @@ export default defineConfig({ name: "chromium", use: { ...devices["Desktop Chrome"] }, }, + { + // Second engine for the arx4 wire-format gate: the codec has to code bit-identically here. + // Visual baselines are chromium-only, so the screenshot spec stays on its single engine. + name: "webkit", + testIgnore: ["**/visual.spec.ts"], + use: { ...devices["Desktop Safari"] }, + }, ], }); diff --git a/public/_headers b/public/_headers index 05f6d97..f2ce36d 100644 --- a/public/_headers +++ b/public/_headers @@ -15,6 +15,11 @@ Content-Encoding: br Vary: Accept-Encoding +/arx4-priors.json.br + Content-Type: application/json; charset=utf-8 + Content-Encoding: br + Vary: Accept-Encoding + /vendor/diff-view-pure.css.br Content-Type: text/css; charset=utf-8 Content-Encoding: br diff --git a/public/arx4-priors.json b/public/arx4-priors.json new file mode 100644 index 0000000..dbe9646 --- /dev/null +++ b/public/arx4-priors.json @@ -0,0 +1 @@ +{"version":1,"kinds":{"markdown":"# Northstar rollout status\n\nReporting window: 3-7 February\nOwner: Platform Enablement\nOverall state: amber, trending green\n\n## Executive summary\n\nThe new workspace provisioning path is active for internal teams in two regions. Median setup time fell from eleven minutes to three minutes after the image cache was moved closer to the runners. The remaining risk is not capacity; it is the uneven age of customer-managed DNS records. Eight accounts still point at the retired validation hostname.\n\n| Workstream | Lead | State | Evidence | Next checkpoint |\n| --- | --- | --- | --- | --- |\n| account import | Mina | done | 1,842 records reconciled | archive old export |\n| runner pools | Theo | healthy | p95 queue 18 seconds | add weekend alarm |\n| DNS cutover | Inez | at risk | 8 stale CNAME records | contact account owners |\n| support training | Ravi | in review | handbook draft approved | record walkthrough |\n\nCompleted this week:\n\n- Added an idempotency key to workspace creation so a browser retry cannot produce a duplicate tenant.\n- Backfilled ownership labels for accounts created before the billing system supplied a team identifier.\n- Published a dry-run command that lists planned mutations without applying them.\n- Split the availability graph by region and machine class.\n\nOpen decisions:\n\n1. Keep the compatibility redirect until all eight DNS records move, even though it complicates certificate renewal.\n2. Decide whether suspended trials belong in the nightly reconciliation job.\n3. Choose a single source for customer display names; the sales export and billing ledger disagree on punctuation.\n\n## Risk register\n\n| Risk | Likelihood | Impact | Mitigation |\n| --- | ---: | ---: | --- |\n| stale DNS prevents callback validation | medium | high | preserve redirect and alert per account |\n| large imports exceed job lease | low | medium | checkpoint every 250 rows |\n| retry storm saturates audit writes | low | high | bounded exponential delay and circuit break |\n\nNo data loss was observed. Every imported row can be traced to an immutable batch identifier, and the rollback tool has been exercised against a synthetic tenant.\n\n# Harbor 2.8.0 release notes\n\nThis release focuses on predictable exports, clearer operator feedback, and smaller cold-start costs.\n\n## Added\n\n- A preview subcommand that prints the destination, selected filters, and estimated row count.\n- Support for newline-delimited JSON alongside the existing CSV exporter.\n- Human-readable explanations for policy denials, including the rule name and remediation link.\n- A health endpoint that distinguishes startup, ready, draining, and failed states.\n\n## Changed\n\n- Configuration files are now merged in this order: system defaults, repository settings, user settings, then command-line flags.\n- Empty date ranges return an empty collection instead of a validation error.\n- Temporary files use the same filesystem as the destination so the final rename remains atomic.\n- Progress output moves to stderr, leaving stdout safe for pipes.\n\n## Fixed\n\n- Quoted CSV fields containing carriage returns no longer create phantom records on Windows.\n- A cancelled upload releases its concurrency permit before retry scheduling.\n- The terminal renderer measures visible width after removing color escape sequences.\n- Cache entries now include the normalized locale in their key.\n\n## Upgrade guidance\n\nOperators using a custom wrapper should check whether it assumes progress messages arrive on stdout. The deprecated HARBOR_LEGACY_DATES switch remains accepted for one release but prints a warning. Existing export manifests require no migration.\n\nBefore deployment:\n\n- run a preview against one representative project;\n- compare row counts with the previous stable binary;\n- verify that the service account can create and rename files in the destination;\n- capture a rollback manifest.\n\n## Compatibility\n\nThe command-line interface supports Linux and macOS on current x64 and arm64 machines. The library build targets maintained JavaScript runtimes. Network storage is supported when it provides atomic rename semantics.\n\n# Ledger inventory API\n\nThe inventory service reports stock positions, reservations, and inbound shipments for a warehouse. Quantities are integers in the smallest sellable unit.\n\nBase URL: https://inventory.example.test/v2\n\n## Authentication\n\nSend a short-lived bearer token in the Authorization header. Tokens are scoped to an organization and one or more facilities. Requests without a facility claim receive 403 even when the organization matches.\n\n## List positions\n\nGET /facilities/{facilityId}/positions\n\nQuery parameters:\n\n| Name | Type | Required | Meaning |\n| --- | --- | --- | --- |\n| cursor | string | no | opaque continuation cursor |\n| limit | integer | no | page size from 1 through 250 |\n| sku | string | no | exact normalized SKU |\n| changed_after | timestamp | no | lower bound, exclusive |\n\nExample response:\n\n```json\n{\n \"items\": [\n {\n \"sku\": \"TEA-ASSAM-250\",\n \"on_hand\": 86,\n \"reserved\": 12,\n \"available\": 74,\n \"location\": \"A-03-17\",\n \"revision\": 42\n }\n ],\n \"next_cursor\": \"cD00MiZzPV...\"\n}\n```\n\n## Reserve stock\n\nPOST /facilities/{facilityId}/reservations\n\nThe Idempotency-Key header is required. Repeating an identical request returns the original reservation. Reusing the key with different fields returns 409.\n\nPossible errors:\n\n- 400 malformed JSON or an invalid quantity;\n- 404 unknown facility or SKU;\n- 409 insufficient unreserved stock, stale revision, or conflicting idempotency key;\n- 429 facility write budget exhausted.\n\nClients should treat the error code as stable and the prose message as diagnostic. A retry-after value, when present, is measured in whole seconds.\n\n## Consistency\n\nPosition reads are monotonic within a cursor walk. A reservation becomes visible to subsequent reads after its transaction commits. Webhook delivery is at least once, so consumers must deduplicate by event_id.\n\n# Fieldnote command-line guide\n\nFieldnote turns a directory of interview notes into a searchable local catalog. It does not upload recordings or transcripts.\n\n## Install\n\n```sh\nnpm install --global fieldnote-cli\nfieldnote --version\n```\n\n## Create a catalog\n\n```sh\nfieldnote init ./research\nfieldnote add ./research/notes --author \"Product Research\"\nfieldnote build ./research --format sqlite\n```\n\nThe add command accepts Markdown and plain text. Front matter is optional. When a document has no explicit title, the first non-empty heading is used, followed by the filename.\n\n## Search\n\n```sh\nfieldnote search ./research \"renewal reminder\" --since 2025-01-01\nfieldnote search ./research \"mobile checkout\" --tag usability --json\n```\n\nSearch terms are case-insensitive. Quoted phrases must occur in order. Multiple tag flags are combined with AND semantics.\n\n## Configuration\n\nFieldnote reads fieldnote.toml from the catalog root:\n\n```toml\n[catalog]\nlanguage = \"en\"\nfollow_symlinks = false\nignore = [\"archive/**\", \"**/*.draft.md\"]\n\n[index]\nstemming = true\nmax_file_bytes = 2000000\n```\n\n## Backup and repair\n\nThe source notes remain authoritative. The generated index may be deleted and rebuilt at any time. Use fieldnote doctor to check for unreadable files, duplicate document keys, and index schema drift.\n\nExit codes:\n\n| Code | Meaning |\n| ---: | --- |\n| 0 | command completed |\n| 2 | invalid arguments |\n| 3 | source file could not be read |\n| 4 | catalog needs a rebuild |\n| 8 | unexpected internal failure |\n\nFor automation, prefer --json and inspect the top-level status field instead of parsing terminal prose.\n\n# Launch readiness meeting\n\nDate: 12 March, 14:00 UTC\nFacilitator: Lena\nAttendees: application, infrastructure, support, security, analytics\n\n## Goal\n\nConfirm whether the recurring invoice reminder can move from staff-only traffic to ten percent of eligible accounts.\n\n## Evidence reviewed\n\n- Delivery success was 99.72% across 184,000 test messages.\n- The unsubscribe path completed in under 400 ms at p95.\n- Support reviewed all new message templates in English, French, and German.\n- The analytics event matches the production naming registry.\n- A restored database snapshot reproduced the same eligibility count as the live read replica.\n\n## Discussion\n\nInfrastructure noted that the scheduled burst begins four minutes after the hourly warehouse compaction. Moving the reminder job by ten minutes avoids overlapping CPU peaks. Product accepted the delay because messages are grouped by the recipient's local morning, not by an exact minute.\n\nSecurity asked whether an account administrator can preview the final recipient list. The current preview shows counts by locale but not addresses. The group agreed that exporting addresses would create unnecessary handling risk; support can investigate an individual message by delivery identifier.\n\n## Decisions\n\n1. Proceed with ten percent traffic on Tuesday.\n2. Hold at ten percent for twenty-four hours before considering fifty percent.\n3. Stop automatically if hard bounces exceed 1.5% over any rolling thirty-minute window.\n4. Keep the former reminder workflow disabled but deployable until the experiment ends.\n\n## Actions\n\n| Owner | Action | Due |\n| --- | --- | --- |\n| Omar | shift scheduler window | Monday |\n| Priya | add bounce-rate stop rule | Monday |\n| Jules | publish support lookup steps | Tuesday |\n| Lena | record go or no-go outcome | Tuesday |\n\nParking lot: consolidate template preview tools after launch; it is not required for this rollout.\n\n# Raster pipeline benchmark\n\n## Objective\n\nMeasure the effect of chunk size and worker count on a batch that reads source photographs, applies orientation, produces two thumbnails, and writes checksums.\n\nHost: 8 performance cores, 32 GiB memory, local NVMe\nDataset: 6,400 JPEG files, 18.7 GiB total\nRuns: five measured repetitions after one warm-up\n\n| Workers | Chunk | Median wall time | p95 item time | Peak RSS | Errors |\n| ---: | ---: | ---: | ---: | ---: | ---: |\n| 2 | 8 | 311.4 s | 182 ms | 1.2 GiB | 0 |\n| 4 | 8 | 174.8 s | 201 ms | 2.1 GiB | 0 |\n| 6 | 16 | 132.6 s | 226 ms | 3.5 GiB | 0 |\n| 8 | 16 | 129.9 s | 308 ms | 4.9 GiB | 0 |\n| 8 | 32 | 128.7 s | 421 ms | 7.8 GiB | 0 |\n\nSix workers with sixteen-item chunks is the recommended default. The two faster configurations save less than four seconds while increasing tail latency and memory pressure substantially.\n\n## Notes\n\nThe checksum stage was deliberately included because production verifies every derived file before publishing its manifest. Disabling checksums improves the headline by about seven percent but measures a different workflow.\n\nThe first run after reboot was excluded. Filesystem cache state was then held constant by rotating input order without evicting data. CPU frequency remained within three percent across measured runs.\n\n## Follow-up\n\n- Repeat on network-attached storage with the same manifest.\n- Record energy use on the small office server.\n- Separate decode time from resize time using trace spans.\n- Add one corpus containing unusually tall images.\n\nRaw samples:\n\n```csv\nworkers,chunk,run,seconds,peak_rss_mb\n4,8,1,176.2,2104\n4,8,2,174.8,2121\n6,16,1,132.6,3548\n6,16,2,133.1,3562\n8,32,1,128.7,7974\n```\n\n# Queue delay incident review\n\nIncident window: 09:18-10:07 UTC\nSeverity: SEV-2\nAffected system: outbound document conversion\n\n## Impact\n\nJobs continued to be accepted, but the oldest waiting job reached twenty-six minutes. Approximately 7,300 conversions completed late. No input documents or generated files were lost.\n\n## Timeline\n\n- 09:18: deploy introduced a new metadata lookup before job admission.\n- 09:23: cache miss rate rose from 4% to 61%.\n- 09:29: queue-age warning fired; on-call compared worker saturation and database load.\n- 09:36: feature flag disabled the lookup for newly admitted jobs.\n- 09:44: backlog began shrinking.\n- 10:07: oldest-job age returned below the five-minute objective.\n\n## Root cause\n\nThe lookup cache used the caller's mixed-case organization key, while the database returned a lowercase canonical key. Successful lookups were stored under a value that subsequent requests did not use. The database handled the extra reads, but admission workers spent most of their leases waiting for connections.\n\n## Contributing factors\n\n- The load test reused one lowercase organization key and never exercised case variation.\n- The queue alarm measured depth but not age until a recent dashboard update.\n- The feature was enabled globally instead of by a small account cohort.\n\n## Corrective actions\n\n| Priority | Work | Verification |\n| --- | --- | --- |\n| P0 | normalize keys at the cache boundary | mixed-case unit table |\n| P0 | cap lookup concurrency per worker | saturation test |\n| P1 | canary metadata changes by account | rollout checklist |\n| P1 | alert on connection wait time | forced pool exhaustion |\n| P2 | expose cache-key cardinality | dashboard review |\n\nThe response was effective once the lookup flag was identified. Future drills should practice detecting a latency regression when throughput remains nonzero.\n\n# Archive migration handbook\n\n## Purpose\n\nThis runbook moves completed case files from the primary document store to the lower-cost archive tier while retaining searchable metadata. It is written for the quarterly migration window and assumes the operator has read-only access to production until the approval step.\n\n## Preconditions\n\n- The candidate query has been reviewed by the records owner.\n- Legal holds have been synchronized within the last hour.\n- The archive bucket reports healthy replication in both target regions.\n- A restore drill using the current tool version completed during the last thirty days.\n- The change ticket names an operator and a separate approver.\n\nNever infer eligibility from a folder name. A case is eligible only when its state is closed, its retention start is older than the policy threshold, and no active hold references the case identifier.\n\n## Phase 1: inventory\n\nRun the planner without write credentials:\n\n```sh\narchive-plan --policy policies/quarterly.yaml --as-of 2026-03-31 --output work/candidates.ndjson --summary work/candidates-summary.json\n```\n\nReview the summary by","code":"// delimiter-parser.ts\nexport type Token =\n | { kind: \"word\"; value: string; offset: number }\n | { kind: \"quoted\"; value: string; offset: number }\n | { kind: \"separator\"; value: \",\" | \";\"; offset: number };\n\nexport class ParseFailure extends Error {\n constructor(\n message: string,\n readonly offset: number,\n ) {\n super(message);\n this.name = \"ParseFailure\";\n }\n}\n\nexport function tokenizeList(source: string): Token[] {\n const tokens: Token[] = [];\n let cursor = 0;\n\n while (cursor < source.length) {\n const char = source[cursor];\n if (/\\s/.test(char)) {\n cursor += 1;\n continue;\n }\n if (char === \",\" || char === \";\") {\n tokens.push({ kind: \"separator\", value: char, offset: cursor });\n cursor += 1;\n continue;\n }\n if (char === \"\"\") {\n const start = cursor++;\n let value = \"\";\n let closed = false;\n while (cursor < source.length) {\n const next = source[cursor++];\n if (next === \"\"\") {\n if (source[cursor] === \"\"\") {\n value += \"\"\";\n cursor += 1;\n } else {\n closed = true;\n break;\n }\n } else {\n value += next;\n }\n }\n if (!closed) throw new ParseFailure(\"unterminated quoted value\", start);\n tokens.push({ kind: \"quoted\", value, offset: start });\n continue;\n }\n\n const start = cursor;\n while (cursor < source.length && !/[\\s,;]/.test(source[cursor])) cursor += 1;\n tokens.push({ kind: \"word\", value: source.slice(start, cursor), offset: start });\n }\n return tokens;\n}\n\nexport function parseGroups(source: string): string[][] {\n const groups: string[][] = [[]];\n let expectsValue = true;\n for (const token of tokenizeList(source)) {\n if (token.kind === \"separator\") {\n if (expectsValue) throw new ParseFailure(\"missing value before separator\", token.offset);\n if (token.value === \";\") groups.push([]);\n expectsValue = true;\n continue;\n }\n if (!expectsValue) throw new ParseFailure(\"missing separator\", token.offset);\n groups.at(-1)?.push(token.value);\n expectsValue = false;\n }\n if (expectsValue && source.trim() !== \"\") {\n throw new ParseFailure(\"list cannot end with a separator\", source.length - 1);\n }\n return groups;\n}\n\n// reservation-handler.ts\nimport { createHash } from \"node:crypto\";\nimport type { IncomingMessage, ServerResponse } from \"node:http\";\nimport { inventory } from \"./inventory-store.js\";\n\ntype ReservationBody = {\n facilityId: string;\n sku: string;\n quantity: number;\n expiresAt?: string;\n};\n\nasync function readJson(request: IncomingMessage, limit = 32_768): Promise {\n const chunks: Buffer[] = [];\n let length = 0;\n for await (const chunk of request) {\n const bytes = Buffer.from(chunk);\n length += bytes.length;\n if (length > limit) throw Object.assign(new Error(\"body too large\"), { status: 413 });\n chunks.push(bytes);\n }\n try {\n return JSON.parse(Buffer.concat(chunks).toString(\"utf8\"));\n } catch (cause) {\n throw Object.assign(new Error(\"request body is not valid JSON\", { cause }), { status: 400 });\n }\n}\n\nfunction validateBody(value: unknown): ReservationBody {\n if (!value || typeof value !== \"object\") {\n throw Object.assign(new Error(\"expected an object\"), { status: 400 });\n }\n const body = value as Record;\n if (typeof body.facilityId !== \"string\" || body.facilityId.length < 2) {\n throw Object.assign(new Error(\"facilityId is required\"), { status: 400 });\n }\n if (typeof body.sku !== \"string\" || !/^[A-Z0-9-]+$/.test(body.sku)) {\n throw Object.assign(new Error(\"sku has an invalid format\"), { status: 400 });\n }\n if (!Number.isSafeInteger(body.quantity) || Number(body.quantity) <= 0) {\n throw Object.assign(new Error(\"quantity must be a positive integer\"), { status: 400 });\n }\n return body as ReservationBody;\n}\n\nexport async function reserve(request: IncomingMessage, response: ServerResponse) {\n const requestId = request.headers[\"x-request-id\"]?.toString() ?? crypto.randomUUID();\n try {\n const key = request.headers[\"idempotency-key\"]?.toString();\n if (!key) throw Object.assign(new Error(\"idempotency-key is required\"), { status: 400 });\n const body = validateBody(await readJson(request));\n const fingerprint = createHash(\"sha256\").update(JSON.stringify(body)).digest(\"hex\");\n const result = await inventory.reserve({ key, fingerprint, ...body });\n response.writeHead(result.created ? 201 : 200, {\n \"content-type\": \"application/json\",\n \"x-request-id\": requestId,\n });\n response.end(JSON.stringify(result.reservation));\n } catch (error) {\n const status = Number((error as { status?: number }).status) || 500;\n response.writeHead(status, { \"content-type\": \"application/json\", \"x-request-id\": requestId });\n response.end(JSON.stringify({\n error: status === 500 ? \"internal_error\" : \"invalid_request\",\n message: error instanceof Error ? error.message : \"unexpected failure\",\n }));\n }\n}\n\n#!/usr/bin/env node\n// bin/tally.mjs\nimport { readFile, writeFile } from \"node:fs/promises\";\nimport { resolve } from \"node:path\";\nimport process from \"node:process\";\n\nfunction parseArgs(argv) {\n const options = { input: null, output: null, group: \"category\", pretty: false };\n for (let index = 0; index < argv.length; index += 1) {\n const argument = argv[index];\n if (argument === \"--input\") options.input = argv[++index];\n else if (argument === \"--output\") options.output = argv[++index];\n else if (argument === \"--group\") options.group = argv[++index];\n else if (argument === \"--pretty\") options.pretty = true;\n else if (argument === \"--help\") options.help = true;\n else throw new Error(\"unknown option: \" + argument);\n }\n return options;\n}\n\nfunction summarize(rows, groupName) {\n const totals = new Map();\n for (const row of rows) {\n const group = String(row[groupName] ?? \"unassigned\");\n const amount = Number(row.amount);\n if (!Number.isFinite(amount)) throw new Error(\"invalid amount in row \" + row.id);\n const current = totals.get(group) ?? { count: 0, amount: 0 };\n current.count += 1;\n current.amount += amount;\n totals.set(group, current);\n }\n return [...totals]\n .map(([group, value]) => ({ group, ...value }))\n .sort((left, right) => right.amount - left.amount || left.group.localeCompare(right.group));\n}\n\nasync function main() {\n const options = parseArgs(process.argv.slice(2));\n if (options.help || !options.input) {\n console.error(\"usage: tally --input records.json [--output totals.json] [--group category]\");\n process.exitCode = options.help ? 0 : 2;\n return;\n }\n const inputPath = resolve(options.input);\n const rows = JSON.parse(await readFile(inputPath, \"utf8\"));\n if (!Array.isArray(rows)) throw new Error(\"input must contain a JSON array\");\n const report = {\n source: inputPath,\n groupedBy: options.group,\n rows: rows.length,\n totals: summarize(rows, options.group),\n };\n const serialized = JSON.stringify(report, null, options.pretty ? 2 : 0) + \"\n\";\n if (options.output) await writeFile(resolve(options.output), serialized, { flag: \"wx\" });\n else process.stdout.write(serialized);\n}\n\nmain().catch((error) => {\n console.error(error instanceof Error ? error.message : error);\n process.exitCode = 1;\n});\n\n# reconcile.py\nfrom __future__ import annotations\n\nimport asyncio\nfrom collections.abc import AsyncIterator, Callable\nfrom dataclasses import dataclass\nfrom decimal import Decimal\nfrom typing import Protocol\n\n\n@dataclass(frozen=True)\nclass Entry:\n account_id: str\n reference: str\n amount: Decimal\n currency: str\n\n\nclass Ledger(Protocol):\n async def entries(self, account_id: str) -> AsyncIterator[Entry]: ...\n async def mark_balanced(self, account_id: str, references: list[str]) -> None: ...\n\n\n@dataclass\nclass Reconciliation:\n account_id: str\n debit_total: Decimal\n credit_total: Decimal\n unmatched: list[Entry]\n\n @property\n def balanced(self) -> bool:\n return self.debit_total + self.credit_total == Decimal(\"0\") and not self.unmatched\n\n\nasync def reconcile_account(\n ledger: Ledger,\n account_id: str,\n *,\n report: Callable[[str], None] = lambda message: None,\n) -> Reconciliation:\n by_reference: dict[str, list[Entry]] = {}\n debit_total = Decimal(\"0\")\n credit_total = Decimal(\"0\")\n\n async for entry in ledger.entries(account_id):\n if entry.amount < 0:\n debit_total += entry.amount\n else:\n credit_total += entry.amount\n by_reference.setdefault(entry.reference, []).append(entry)\n\n unmatched: list[Entry] = []\n matched_references: list[str] = []\n for reference, entries in sorted(by_reference.items()):\n currencies = {entry.currency for entry in entries}\n total = sum((entry.amount for entry in entries), start=Decimal(\"0\"))\n if len(currencies) == 1 and total == 0:\n matched_references.append(reference)\n else:\n unmatched.extend(entries)\n report(f\"{account_id}: reference {reference} remains open\")\n\n result = Reconciliation(account_id, debit_total, credit_total, unmatched)\n if result.balanced:\n await ledger.mark_balanced(account_id, matched_references)\n return result\n\n\nasync def reconcile_many(ledger: Ledger, account_ids: list[str], concurrency: int = 8):\n semaphore = asyncio.Semaphore(concurrency)\n\n async def guarded(account_id: str):\n async with semaphore:\n return await reconcile_account(ledger, account_id)\n\n tasks = [asyncio.create_task(guarded(account_id)) for account_id in account_ids]\n return await asyncio.gather(*tasks)\n\n// delimiter-parser.test.ts\nimport { describe, expect, it } from \"vitest\";\nimport { ParseFailure, parseGroups, tokenizeList } from \"./delimiter-parser\";\n\ndescribe(\"tokenizeList\", () => {\n it(\"preserves offsets and quoted separators\", () => {\n expect(tokenizeList('alpha, \"beta,gamma\"; delta')).toEqual([\n { kind: \"word\", value: \"alpha\", offset: 0 },\n { kind: \"separator\", value: \",\", offset: 5 },\n { kind: \"quoted\", value: \"beta,gamma\", offset: 7 },\n { kind: \"separator\", value: \";\", offset: 19 },\n { kind: \"word\", value: \"delta\", offset: 21 },\n ]);\n });\n\n it(\"accepts doubled quotes inside a quoted value\", () => {\n expect(parseGroups('\"a \"\"small\"\" thing\",next')).toEqual([[\"a \"small\" thing\", \"next\"]]);\n });\n\n it.each([\n [\",alpha\", 0, \"missing value before separator\"],\n [\"alpha,\", 5, \"list cannot end with a separator\"],\n [\"alpha beta\", 6, \"missing separator\"],\n ['\"never closed', 0, \"unterminated quoted value\"],\n ])(\"rejects %s\", (source, offset, message) => {\n try {\n parseGroups(source);\n expect.fail(\"expected parse failure\");\n } catch (error) {\n expect(error).toBeInstanceOf(ParseFailure);\n expect(error).toMatchObject({ message, offset });\n }\n });\n});\n\n// reservation-handler.test.ts\ndescribe(\"reserve\", () => {\n it(\"returns the first reservation again for an identical key\", async () => {\n const first = await request(app)\n .post(\"/reservations\")\n .set(\"idempotency-key\", \"order-88\")\n .send({ facilityId: \"east\", sku: \"TEA-1\", quantity: 3 })\n .expect(201);\n const replay = await request(app)\n .post(\"/reservations\")\n .set(\"idempotency-key\", \"order-88\")\n .send({ facilityId: \"east\", sku: \"TEA-1\", quantity: 3 })\n .expect(200);\n expect(replay.body).toEqual(first.body);\n });\n\n it(\"rejects a key reused for a different quantity\", async () => {\n await seedReservation(\"order-91\", { facilityId: \"west\", sku: \"MUG-2\", quantity: 1 });\n const response = await request(app)\n .post(\"/reservations\")\n .set(\"idempotency-key\", \"order-91\")\n .send({ facilityId: \"west\", sku: \"MUG-2\", quantity: 2 })\n .expect(409);\n expect(response.body.error).toBe(\"idempotency_conflict\");\n });\n});\n\ndiff --git a/src/window.ts b/src/window.ts\nindex 120ac91..7f2d8ea 100644\n--- a/src/window.ts\n+++ b/src/window.ts\n@@ -18,12 +18,21 @@ export function collectWindow(samples: Sample[], end: number) {\n- return samples.filter((sample) => sample.timestamp > end - WINDOW_MS);\n+ const start = end - WINDOW_MS;\n+ const selected: Sample[] = [];\n+ for (let index = samples.length - 1; index >= 0; index -= 1) {\n+ const sample = samples[index];\n+ if (sample.timestamp <= start) break;\n+ if (sample.timestamp <= end) selected.push(sample);\n+ }\n+ selected.reverse();\n+ return selected;\n }\n\n export function average(samples: Sample[]) {\n- return samples.reduce((sum, sample) => sum + sample.value, 0) / samples.length;\n+ if (samples.length === 0) return null;\n+ let sum = 0;\n+ for (const sample of samples) sum += sample.value;\n+ return sum / samples.length;\n }\ndiff --git a/tests/test_retry.py b/tests/test_retry.py\nindex 8df21aa..3c2147d 100644\n--- a/tests/test_retry.py\n+++ b/tests/test_retry.py\n@@ -27,9 +27,18 @@ async def test_retries_temporary_failure(client):\n- result = await fetch_with_retry(client, \"/status\")\n+ delays: list[float] = []\n+ result = await fetch_with_retry(\n+ client,\n+ \"/status\",\n+ sleep=delays.append,\n+ jitter=lambda: 0,\n+ )\n assert result.status == 200\n assert client.calls == 3\n+ assert delays == [0.25, 0.5]\n+\n+async def test_does_not_retry_permission_error(client):\n+ client.respond_with(403)\n+ with pytest.raises(PermissionDenied):\n+ await fetch_with_retry(client, \"/private\")\n+ assert client.calls == 1\n\n// lease-pool.ts\nexport interface Clock {\n now(): number;\n sleep(milliseconds: number): Promise;\n}\n\nexport type Lease = {\n value: T;\n release(): void;\n};\n\ntype Waiter = {\n resolve(lease: Lease): void;\n reject(error: Error): void;\n deadline: number;\n signal?: AbortSignal;\n};\n\nexport class LeasePool {\n readonly #available: T[];\n readonly #waiters: Waiter[] = [];\n readonly #leased = new Set();\n #closed = false;\n\n constructor(\n values: Iterable,\n private readonly clock: Clock,\n ) {\n this.#available = [...values];\n if (this.#available.length === 0) throw n","json":"{\n \"service\": {\n \"name\": \"maple-worker\",\n \"environment\": \"staging\",\n \"region\": \"eu-central\",\n \"instance\": {\n \"min\": 2,\n \"max\": 18,\n \"target_cpu_percent\": 65,\n \"shutdown_grace_seconds\": 45\n }\n },\n \"queue\": {\n \"name\": \"document-events\",\n \"visibility_timeout_seconds\": 120,\n \"max_receive_count\": 6,\n \"dead_letter_queue\": \"document-events-failed\",\n \"batch\": {\n \"max_items\": 25,\n \"max_wait_milliseconds\": 400,\n \"parallel_batches\": 4\n }\n },\n \"storage\": {\n \"bucket\": \"maple-staging-derived\",\n \"prefix\": \"normalized/\",\n \"checksum\": \"sha256\",\n \"encryption\": {\n \"mode\": \"managed\",\n \"key_alias\": \"staging-documents\"\n }\n },\n \"logging\": {\n \"level\": \"info\",\n \"format\": \"json\",\n \"redact\": [\n \"request.headers.authorization\",\n \"customer.email\",\n \"document.extracted_text\"\n ],\n \"sample_debug_percent\": 2\n },\n \"features\": {\n \"layout_detection\": true,\n \"language_hint\": false,\n \"parallel_thumbnailing\": true\n }\n}\n\n{\n \"request_id\": \"req_01J8ZP7KFE6V2\",\n \"received_at\": \"2026-02-14T10:22:31.884Z\",\n \"account\": {\n \"id\": \"acct_willow\",\n \"plan\": \"team\",\n \"locale\": \"en-GB\"\n },\n \"order\": {\n \"currency\": \"GBP\",\n \"lines\": [\n {\n \"sku\": \"PAPER-A5-DOT\",\n \"description\": \"A5 dotted notebook\",\n \"quantity\": 3,\n \"unit_price\": \"8.50\",\n \"tax_rate\": \"0.20\"\n },\n {\n \"sku\": \"PEN-GEL-BLACK\",\n \"description\": \"Black gel pen\",\n \"quantity\": 6,\n \"unit_price\": \"1.75\",\n \"tax_rate\": \"0.20\"\n }\n ],\n \"discounts\": [\n {\n \"code\": \"WELCOME5\",\n \"kind\": \"fixed\",\n \"amount\": \"5.00\"\n }\n ],\n \"shipping\": {\n \"method\": \"tracked\",\n \"amount\": \"4.25\",\n \"address\": {\n \"country\": \"GB\",\n \"postal_code\": \"M1 1AE\",\n \"city\": \"Manchester\"\n }\n }\n },\n \"result\": {\n \"subtotal\": \"36.00\",\n \"discount\": \"5.00\",\n \"tax\": \"6.20\",\n \"shipping\": \"4.25\",\n \"total\": \"41.45\",\n \"payment_status\": \"authorized\"\n },\n \"warnings\": []\n}\n\n{\n \"event_id\": \"evt_01J8ZP8A1X\",\n \"type\": \"order.authorized\",\n \"attempt\": 1,\n \"subject\": \"ord_209184\",\n \"data\": {\n \"account_id\": \"acct_willow\",\n \"amount\": \"41.45\",\n \"currency\": \"GBP\",\n \"authorization\": \"auth_77102\"\n },\n \"delivery\": {\n \"destination\": \"https://hooks.example.test/orders\",\n \"timeout_ms\": 5000,\n \"signature_version\": \"v2\"\n }\n}\n\n{\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://schemas.example.test/shipment-notice.json\",\n \"title\": \"Shipment notice\",\n \"type\": \"object\",\n \"additionalProperties\": false,\n \"required\": [\"notice_id\", \"carrier\", \"packages\", \"created_at\"],\n \"properties\": {\n \"notice_id\": {\n \"type\": \"string\",\n \"pattern\": \"^asn_[a-z0-9]{12}$\"\n },\n \"carrier\": {\n \"type\": \"object\",\n \"additionalProperties\": false,\n \"required\": [\"code\", \"service\"],\n \"properties\": {\n \"code\": { \"type\": \"string\", \"minLength\": 2, \"maxLength\": 12 },\n \"service\": { \"enum\": [\"ground\", \"express\", \"freight\"] },\n \"tracking_number\": { \"type\": [\"string\", \"null\"] }\n }\n },\n \"packages\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 100,\n \"items\": {\n \"type\": \"object\",\n \"additionalProperties\": false,\n \"required\": [\"package_id\", \"weight_grams\", \"items\"],\n \"properties\": {\n \"package_id\": { \"type\": \"string\" },\n \"weight_grams\": { \"type\": \"integer\", \"minimum\": 1 },\n \"dimensions_mm\": {\n \"type\": \"array\",\n \"prefixItems\": [\n { \"type\": \"integer\", \"minimum\": 1 },\n { \"type\": \"integer\", \"minimum\": 1 },\n { \"type\": \"integer\", \"minimum\": 1 }\n ],\n \"minItems\": 3,\n \"maxItems\": 3\n },\n \"items\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"required\": [\"sku\", \"quantity\"],\n \"properties\": {\n \"sku\": { \"type\": \"string\", \"minLength\": 1 },\n \"quantity\": { \"type\": \"integer\", \"minimum\": 1 }\n }\n }\n }\n }\n }\n },\n \"created_at\": { \"type\": \"string\", \"format\": \"date-time\" }\n }\n}\n\n{\n \"trace_id\": \"4f817a20be014986a99c7b10d6b253ab\",\n \"root\": {\n \"span_id\": \"a1900f59d144c91a\",\n \"name\": \"POST /imports\",\n \"kind\": \"server\",\n \"start_ns\": 1771048200123000000,\n \"duration_ns\": 184220000,\n \"status\": \"ok\",\n \"attributes\": {\n \"http.method\": \"POST\",\n \"http.route\": \"/imports\",\n \"http.status_code\": 202,\n \"tenant.tier\": \"business\",\n \"request.bytes\": 18342\n }\n },\n \"spans\": [\n {\n \"span_id\": \"4bc9ddc01671c201\",\n \"parent_span_id\": \"a1900f59d144c91a\",\n \"name\": \"validate manifest\",\n \"kind\": \"internal\",\n \"start_offset_ns\": 2200000,\n \"duration_ns\": 18300000,\n \"status\": \"ok\",\n \"events\": [\n {\n \"name\": \"manifest.accepted\",\n \"offset_ns\": 17600000,\n \"attributes\": { \"files\": 12, \"declared_bytes\": 9412000 }\n }\n ]\n },\n {\n \"span_id\": \"41bdf6716d2aac32\",\n \"parent_span_id\": \"a1900f59d144c91a\",\n \"name\": \"insert import\",\n \"kind\": \"client\",\n \"start_offset_ns\": 22600000,\n \"duration_ns\": 105400000,\n \"status\": \"ok\",\n \"attributes\": {\n \"db.system\": \"postgresql\",\n \"db.operation\": \"INSERT\",\n \"db.rows_affected\": 1,\n \"pool.wait_ms\": 3.2\n }\n },\n {\n \"span_id\": \"e919d59cf1079440\",\n \"parent_span_id\": \"a1900f59d144c91a\",\n \"name\": \"publish import.accepted\",\n \"kind\": \"producer\",\n \"start_offset_ns\": 134900000,\n \"duration_ns\": 42100000,\n \"status\": \"ok\",\n \"attributes\": {\n \"messaging.system\": \"kafka\",\n \"messaging.destination\": \"imports.accepted\",\n \"messaging.batch.message_count\": 1\n }\n }\n ],\n \"resource\": {\n \"service.name\": \"intake-api\",\n \"service.version\": \"2026.02.14.3\",\n \"deployment.environment\": \"production\",\n \"cloud.region\": \"us-east\"\n }\n}\n\n{\n \"pipeline\": {\n \"name\": \"nightly-catalog\",\n \"timezone\": \"UTC\",\n \"schedule\": \"15 2 * * *\",\n \"concurrency\": 1,\n \"timeout_minutes\": 90\n },\n \"parameters\": {\n \"source\": \"warehouse_snapshot\",\n \"effective_date\": \"{{ ds }}\",\n \"include_discontinued\": false,\n \"markets\": [\"CA\", \"DE\", \"GB\", \"US\"]\n },\n \"steps\": [\n {\n \"id\": \"extract\",\n \"uses\": \"sql-query\",\n \"with\": {\n \"connection\": \"analytics-readonly\",\n \"query_file\": \"queries/catalog.sql\",\n \"output\": \"work/catalog.ndjson\"\n },\n \"retry\": { \"attempts\": 3, \"delay_seconds\": 20 }\n },\n {\n \"id\": \"validate\",\n \"uses\": \"schema-check\",\n \"needs\": [\"extract\"],\n \"with\": {\n \"input\": \"work/catalog.ndjson\",\n \"schema\": \"schemas/catalog-item.json\",\n \"max_errors\": 25\n }\n },\n {\n \"id\": \"publish\",\n \"uses\": \"object-upload\",\n \"needs\": [\"validate\"],\n \"if\": \"steps.validate.invalid == 0\",\n \"with\": {\n \"input\": \"work/catalog.ndjson\",\n \"destination\": \"catalog/{{ ds }}/items.ndjson\",\n \"content_type\": \"application/x-ndjson\",\n \"cache_control\": \"public, max-age=300\"\n }\n },\n {\n \"id\": \"notify\",\n \"uses\": \"message\",\n \"needs\": [\"publish\"],\n \"with\": {\n \"channel\": \"data-operations\",\n \"template\": \"catalog-published\",\n \"fields\": [\"row_count\", \"duration_seconds\", \"destination\"]\n }\n }\n ],\n \"retention\": {\n \"work_days\": 3,\n \"successful_run_days\": 30,\n \"failed_run_days\": 90\n }\n}\n\ntimestamp,route,status,region,duration_ms,response_bytes,cache\n2026-02-14T10:20:01.120Z,/v1/search,200,us-east,42,1840,hit\n2026-02-14T10:20:01.224Z,/v1/search,200,us-west,77,2912,miss\n2026-02-14T10:20:01.385Z,/v1/items/1842,200,eu-west,31,722,hit\n2026-02-14T10:20:01.412Z,/v1/items/9981,404,eu-west,18,156,bypass\n2026-02-14T10:20:01.609Z,/v1/orders,201,us-east,128,844,bypass\n2026-02-14T10:20:01.944Z,/v1/search,429,us-east,9,188,bypass\n2026-02-14T10:20:02.015Z,/v1/categories,200,ap-south,63,4180,miss\n2026-02-14T10:20:02.276Z,/v1/items/3401,304,us-west,14,0,hit\n\nwarehouse,sku,on_hand,reserved,inbound,reorder_point,last_counted\natlanta,PAPER-A5-DOT,184,26,240,80,2026-02-13\natlanta,PEN-GEL-BLACK,912,144,600,300,2026-02-13\nberlin,PAPER-A5-DOT,72,18,120,60,2026-02-12\nberlin,INK-BLUE-30ML,24,2,48,20,2026-02-11\nmontreal,TAPE-WASHI-GRID,310,54,0,120,2026-02-14\nmontreal,CLIP-BRASS-12,88,9,96,40,2026-02-14\nsydney,NOTE-B6-PLAIN,51,17,144,75,2026-02-10\nsydney,PEN-GEL-BLACK,266,91,300,180,2026-02-10\n\nteam,service,objective,target,window,owner\ncheckout,payment-api,availability,99.95%,30d,kelly\ncheckout,order-worker,latency-p95,800ms,7d,sam\ncatalog,search-api,availability,99.90%,30d,dev\ncatalog,indexer,freshness-p99,15m,7d,morgan\nidentity,session-api,latency-p95,120ms,7d,alex\nplatform,audit-stream,delivery,99.99%,30d,casey\n\n{\n \"dashboard\": {\n \"title\": \"Fulfillment overview\",\n \"description\": \"Order admission, picking, packing, and carrier handoff\",\n \"time_zone\": \"browser\",\n \"refresh\": \"30s\",\n \"variables\": [\n {\n \"name\": \"environment\",\n \"label\": \"Environment\",\n \"type\": \"custom\",\n \"values\": [\"production\", \"staging\"],\n \"default\": \"production\"\n },\n {\n \"name\": \"facility\",\n \"label\": \"Facility\",\n \"type\": \"query\",\n \"query\": \"label_values(fulfillment_orders_total, facility)\",\n \"include_all\": true\n }\n ],\n \"rows\": [\n {\n \"title\": \"Customer impact\",\n \"collapsed\": false,\n \"panels\": [\n {\n \"id\": 1,\n \"title\": \"Orders admitted\",\n \"kind\": \"stat\",\n \"query\": \"sum(rate(fulfillment_orders_total{environment='$environment',facility=~'$facility'}[5m]))\",\n \"unit\": \"orders/s\",\n \"thresholds\": [\n { \"color\": \"green\", \"value\": null },\n { \"color\": \"orange\", \"value\": 40 },\n { \"color\": \"red\", \"value\": 70 }\n ]\n },\n {\n \"id\": 2,\n \"title\": \"Admission failures\",\n \"kind\": \"timeseries\",\n \"queries\": [\n {\n \"ref\": \"A\",\n \"legend\": \"{{reason}}\",\n \"expression\": \"sum by (reason) (rate(fulfillment_admission_failures_total{environment='$environment'}[5m]))\"\n }\n ],\n \"unit\": \"ops/s\",\n \"stack\": \"normal\"\n },\n {\n \"id\": 3,\n \"title\": \"Oldest unshipped order\",\n \"kind\": \"gauge\",\n \"query\": \"max(fulfillment_oldest_unshipped_seconds{environment='$environment',facility=~'$facility'})\",\n \"unit\": \"seconds\",\n \"min\": 0,\n \"max\": 14400\n }\n ]\n },\n {\n \"title\": \"Workflow health\",\n \"collapsed\": false,\n \"panels\": [\n {\n \"id\": 4,\n \"title\": \"Stage duration percentiles\",\n \"kind\": \"timeseries\",\n \"queries\": [\n {\n \"ref\": \"A\",\n \"legend\": \"pick p95\",\n \"expression\": \"histogram_quantile(0.95, sum by (le) (rate(pick_duration_seconds_bucket{environment='$environment'}[10m])))\"\n },\n {\n \"ref\": \"B\",\n \"legend\": \"pack p95\",\n \"expression\": \"histogram_quantile(0.95, sum by (le) (rate(pack_duration_seconds_bucket{environment='$environment'}[10m])))\"\n }\n ],\n \"unit\": \"seconds\"\n },\n {\n \"id\": 5,\n \"title\": \"Worker leases\",\n \"kind\": \"timeseries\",\n \"queries\": [\n {\n \"ref\": \"A\",\n \"legend\": \"active\",\n \"expression\": \"sum(worker_leases{state='active',facility=~'$facility'})\"\n },\n {\n \"ref\": \"B\",\n \"legend\": \"waiting\",\n \"expression\": \"sum(worker_leases{state='waiting',facility=~'$facility'})\"\n }\n ],\n \"unit\": \"short\"\n },\n {\n \"id\": 6,\n \"title\": \"Carrier handoff errors\",\n \"kind\": \"table\",\n \"query\": \"topk(10, sum by (carrier,error_code) (increase(carrier_handoff_failures_total{environment='$environment'}[1h])))\",\n \"columns\": [\n { \"field\": \"carrier\", \"label\": \"Carrier\" },\n { \"field\": \"error_code\", \"label\": \"Code\" },\n { \"field\": \"value\", \"label\": \"Failures\", \"align\": \"right\" }\n ]\n }\n ]\n }\n ],\n \"annotations\": [\n {\n \"name\": \"deployments\",\n \"enabled\": true,\n \"query\": \"changes{service='fulfillment'}\",\n \"color\": \"#6f42c1\"\n },\n {\n \"name\": \"incidents\",\n \"enabled\": true,\n \"query\": \"incident_events{team='commerce'}\",\n \"color\": \"#d73a49\"\n }\n ],\n \"links\": [\n { \"title\": \"Operations guide\", \"url\": \"https://docs.example.test/fulfillment/runbook\" },\n { \"title\": \"Service catalog\", \"url\": \"https://catalog.example.test/services/fulfillment\" }\n ]\n },\n \"alerts\": [\n {\n \"name\": \"OldestOrderTooOld\",\n \"for\": \"10m\",\n \"severity\": \"page\",\n \"expression\": \"max(fulfillment_oldest_unshipped_seconds{environment='production'}) > 3600\",\n \"labels\": { \"team\": \"commerce\", \"service\": \"fulfillment\" },\n \"annotations\": {\n \"summary\": \"An admitted order has remained unshipped for over one hour\",\n \"runbook\": \"https://docs.example.test/fulfillment/runbook#old-orders\"\n }\n },\n {\n \"name\": \"CarrierErrorsElevated\",\n \"for\": \"15m\",\n \"severity\": \"ticket\",\n \"expression\": \"sum(rate(carrier_handoff_failures_total{environment='production'}[10m])) > 0.5\",\n \"labels\": { \"team\": \"commerce\", \"service\": \"carrier-adapter\" },\n \"annotations\": {\n \"summary\": \"Carrier handoff failures exceed the expec"}} \ No newline at end of file diff --git a/public/arx4-priors.json.br b/public/arx4-priors.json.br new file mode 100644 index 0000000..fa2e52d Binary files /dev/null and b/public/arx4-priors.json.br differ diff --git a/scripts/build-arx4-priors.mjs b/scripts/build-arx4-priors.mjs new file mode 100644 index 0000000..d6dc8bb --- /dev/null +++ b/scripts/build-arx4-priors.mjs @@ -0,0 +1,139 @@ +#!/usr/bin/env node + +/** + * Regenerates `public/arx4-priors.json`, the curated per-kind priming corpora the arx4 context mixer + * runs before a payload. + * + * The benched per-kind prior (docs/arx4-cm-bench.md) is `\n` truncated to exactly 16384 bytes. The first 2203 bytes of that are the dictionary-derived + * common prefix, which the codec already rebuilds at runtime, so the asset ships only the 14181-byte + * kind-specific remainder. + * + * The curated text is extracted from the frozen source rather than copied. This script checks what it + * can see locally (the prefix the codec rebuilds still heads the prior, and the prior re-encodes to + * exactly PRIOR_BYTES) and prints each prior's sha256; tests/arx4-priors.test.ts pins those digests, + * so a drift in the frozen script or the shipped dictionaries fails the suite instead of silently + * changing the arx4 wire. + * + * Outputs: + * public/arx4-priors.json : minified asset the viewer fetches + * public/arx4-priors.json.br : brotli-compressed variant, tried first on default loads + * + * Run: node scripts/build-arx4-priors.mjs + */ + +import { createHash } from "node:crypto"; +import { readFileSync, writeFileSync } from "node:fs"; +import { createRequire } from "node:module"; +import { resolve } from "node:path"; + +const require = createRequire(import.meta.url); +const brotli = require("brotli-wasm"); + +const PRIORS_VERSION = 1; +const PRIOR_BYTES = 16 * 1024; +// The frozen benchmark script holding the curated corpus is maintainer-local research +// material and is not tracked in this repo; pass its path to regenerate the asset. +// Integrity does not depend on having it: tests/arx4-priors.test.ts pins the sha256 of +// every prior reassembled from the shipped asset. +const frozenSourcePath = process.argv[2]; +if (!frozenSourcePath) { + console.error( + "usage: node scripts/build-arx4-priors.mjs ", + ); + process.exit(1); +} +const FROZEN_SOURCE = resolve(frozenSourcePath); +const SECTIONS_START = "const curatedMarkdownSections = ["; +const SECTIONS_END = "\nconst curatedSections ="; + +function fail(message) { + console.error(`build-arx4-priors: ${message}`); + process.exit(1); +} + +/** + * Evaluates the three curated section arrays out of the frozen script. The slab between the markers + * holds nothing but those array literals, and both markers are asserted so a reshaped frozen source + * fails loudly rather than yielding a partial corpus. + */ +function extractCuratedSections() { + const source = readFileSync(FROZEN_SOURCE, "utf8"); + const start = source.indexOf(SECTIONS_START); + const end = source.indexOf(SECTIONS_END, start); + if (start < 0 || end < 0) { + fail(`could not locate the curated sections in ${FROZEN_SOURCE}`); + } + + const slab = source.slice(start, end); + const sections = new Function( + `${slab}\nreturn { markdown: curatedMarkdownSections, code: curatedCodeSections, json: curatedJsonSections };`, + )(); + + for (const [kind, texts] of Object.entries(sections)) { + if (!Array.isArray(texts) || texts.length === 0 || texts.some((text) => typeof text !== "string")) { + fail(`extracted ${kind} sections are not a non-empty array of strings`); + } + } + + return sections; +} + +function dictionarySlotText() { + const base = JSON.parse(readFileSync(new URL("../public/arx-dictionary.json", import.meta.url), "utf8")); + const overlay = JSON.parse(readFileSync(new URL("../public/arx2-dictionary.json", import.meta.url), "utf8")); + return [ + ...base.singleByteSlots, + ...base.extendedSlots, + ...overlay.singleByteSlots, + ...overlay.extendedSlots, + ].join("\n"); +} + +/** The frozen `buildExactPrior`: dictionary text, a newline, the corpus, cut to exactly N bytes. */ +function buildExactPrior(commonPrefix, corpusText, kind) { + const available = Buffer.from(`${commonPrefix}${corpusText}`, "utf8"); + if (available.length < PRIOR_BYTES) { + fail(`${kind} corpus has ${available.length} bytes, needs ${PRIOR_BYTES}`); + } + return available.subarray(0, PRIOR_BYTES).toString("utf8"); +} + +function sha256(text) { + return createHash("sha256").update(text, "utf8").digest("hex"); +} + +const curatedSections = extractCuratedSections(); +const commonPrefix = `${dictionarySlotText()}\n`; +const kinds = {}; + +for (const [kind, texts] of Object.entries(curatedSections)) { + const frozenPrior = buildExactPrior(commonPrefix, texts.join("\n\n"), kind); + // The prior is cut to PRIOR_BYTES, so dictionaries that outgrow that budget would ship a prior + // whose head no longer matches the prefix the codec rebuilds from them at runtime. + if (frozenPrior.slice(0, commonPrefix.length) !== commonPrefix) { + fail( + `${kind} prior head does not match the ${commonPrefix.length}-char dictionary-derived prefix; the shipped dictionaries no longer fit the ${PRIOR_BYTES}-byte prior`, + ); + } + + const kindSpecific = frozenPrior.slice(commonPrefix.length); + // A cut through a multibyte character would leave a replacement char that no longer re-encodes to + // the priming bytes the bench measured, so the byte count is checked, not just the char count. + if (Buffer.byteLength(frozenPrior, "utf8") !== PRIOR_BYTES) { + fail(`${kind} prior re-encodes to ${Buffer.byteLength(frozenPrior, "utf8")} bytes, expected ${PRIOR_BYTES}`); + } + + kinds[kind] = kindSpecific; + console.log(`${kind}: prior ${frozenPrior.length} chars, kind block ${kindSpecific.length} chars, sha256 ${sha256(frozenPrior)}`); +} + +const json = JSON.stringify({ version: PRIORS_VERSION, kinds }); +const compressed = brotli.compress(Buffer.from(json, "utf8"), { quality: 11 }); + +writeFileSync(new URL("../public/arx4-priors.json", import.meta.url), json, "utf8"); +writeFileSync(new URL("../public/arx4-priors.json.br", import.meta.url), compressed); + +console.log(`public/arx4-priors.json: ${Buffer.byteLength(json, "utf8")} bytes`); +console.log(`public/arx4-priors.json.br: ${compressed.length} bytes (brotli q11)`); +console.log(`common prefix: ${commonPrefix.length} chars, derived from the shipped dictionaries`); diff --git a/skills/agent-render-linking/SKILL.md b/skills/agent-render-linking/SKILL.md index 47912b9..437fdca 100644 --- a/skills/agent-render-linking/SKILL.md +++ b/skills/agent-render-linking/SKILL.md @@ -32,10 +32,11 @@ character identifying the codec: #a (arx) #b (arx2) #c (arx3) +#e (arx4) ``` -The single tag char identifies the codec; for `arx`/`arx2`/`arx3` it implies the -current dictionary but does not carry a dictionary version. The payload follows +The single tag char identifies the codec; for `arx`/`arx2`/`arx3`/`arx4` it implies +the current dictionary but does not carry a dictionary version. The payload follows immediately after it. The legacy `#agent-render=v1..` form (arx-family carry an extra `.` segment) still decodes, but the viewer no longer emits it — always build the compact form. @@ -47,11 +48,12 @@ Supported codecs: - `arx`: domain-dictionary substitution + brotli (quality 11) + binary-to-text encoding (~70% smaller than deflate with baseBMP). Fetch the shared dictionary from `https://agent-render.com/arx-dictionary.json` to apply substitutions locally before brotli compression. Four wire shapes: baseBMP (~62k safe BMP code points, ~15.92 bits/char, best raw density), base1k (1774 Unicode code points U+00A1–U+07FF), base64url (ASCII `A-Za-z0-9-_`, `B.` prefix — good when Unicode would be percent-encoded), and base76 (77-char ASCII). The product encoder tries all four and picks the shortest **transport** length. - `arx2`: tuple-envelope transport + `https://agent-render.com/arx2-dictionary.json` overlay (or pre-compressed `https://agent-render.com/arx2-dictionary.json.br`) + the shared arx dictionary + brotli (quality 11) + the same four wire shapes. Existing arx links remain valid; prefer arx2 when it is the shortest transport. - `arx3`: same tuple envelope, arx2 overlay, shared arx dictionary, and brotli bytes as arx2, but the dense baseBMP wire may win by decoded visible character length. Use it for trusted surfaces that preserve Unicode fragments and strict visible URL budgets. Prefer arx2/base64url or UUID mode when the target platform rewrites, truncates, or previews long links aggressively. +- `arx4`: the same tuple envelope, arx2 overlay, shared arx dictionary and baseBMP selection rule as arx3, with brotli replaced by a deterministic integer context mixer and one extra leading payload char, the prior id (`m`, `c`, `j`, `s`, or `n`). Recognize and open `#e` links; do not hand-roll them. Reproducing the wire needs the exact frozen mixer plus `https://agent-render.com/arx4-priors.json`, so an agent encoding on its own should stop at `arx3` and let the app or library emit arx4. - packed wire mode (`p: 1`) may be used automatically to shorten transport keys Prefer: 1. shortest valid fragment for the target surface -2. codec priority `arx3 -> arx2 -> arx -> deflate -> lz -> plain` unless explicitly overridden +2. codec priority `arx3 -> arx2 -> arx -> deflate -> lz -> plain` for links you encode yourself; the app itself tries `arx4` first 3. packed wire mode when available ## Envelope shape @@ -198,6 +200,7 @@ https://agent-render.com/#d (deflate) https://agent-render.com/#a (arx) https://agent-render.com/#b (arx2) https://agent-render.com/#c (arx3) +https://agent-render.com/#e (arx4) ``` For `plain`: @@ -241,7 +244,7 @@ To use the dictionary for local `arx` encoding: - Base76 uses 77 ASCII fragment-safe characters. ~6.27 bits/char 5. Prepend the tag `a` to form the fragment (the compact tag does not carry a dictionary version — it implies the current dictionary, so always substitute using the build's current dictionary) -The dictionary includes JSON envelope boilerplate patterns, JSON-escaped Markdown syntax, and programming-language patterns that are already present in the shipped corpus. The viewer tries the pre-compressed dictionary first on default ARX/ARX2/ARX3 encode or decode paths, falls back to the JSON file, and falls back again to its built-in table if external fetches fail. +The dictionary includes JSON envelope boilerplate patterns, JSON-escaped Markdown syntax, and programming-language patterns that are already present in the shipped corpus. The viewer tries the pre-compressed dictionary first on default ARX/ARX2/ARX3/ARX4 encode or decode paths, falls back to the JSON file, and falls back again to its built-in table if external fetches fail. If the dictionary fetch fails, fall back to `deflate` codec. @@ -264,6 +267,8 @@ Then apply substitutions in this order: For `arx3`, use the same tuple, substitution, and brotli bytes as arx2, then try the same four wire shapes, but measure the baseBMP wire by decoded visible character length (rather than conservative transport length) and pick the shortest candidate — so the dense baseBMP wire can win on Unicode-preserving surfaces. Prepend the tag `c` (the compact tag does not carry a dictionary version — it implies the current shared arx dictionary and arx2 overlay). Do not invent a new dictionary entry unless it is backed by corpus evidence and improves the benchmark gate. +For `arx4`, there is no hand-rollable recipe: the payload is arithmetic-coded against a context-mixing model primed on a corpus that must match the encoder bit for bit, so encode arx4 only through the app or `encodeEnvelopeAsync`. Read the tag `e` and the prior id that follows it when parsing a link someone else produced. + ## Practical limits Respect these limits: diff --git a/skills/selfhosted-agent-render/SKILL.md b/skills/selfhosted-agent-render/SKILL.md index 5dda844..316d078 100644 --- a/skills/selfhosted-agent-render/SKILL.md +++ b/skills/selfhosted-agent-render/SKILL.md @@ -49,7 +49,7 @@ Response (`201`): } ``` -The `payload` field is the same payload string used in fragment links — the compact fragment body after `#` (a single codec tag char followed by the payload). Use the same envelope format and codecs (`plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`) described in the `agent-render-linking` skill. The legacy `agent-render=v1..` form (arx-family carry an extra `.` segment) is also accepted for back-compatibility. +The `payload` field is the same payload string used in fragment links — the compact fragment body after `#` (a single codec tag char followed by the payload). Use the same envelope format and codecs (`plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, `arx4`) described in the `agent-render-linking` skill. The legacy `agent-render=v1..` form (arx-family carry an extra `.` segment) is also accepted for back-compatibility. ### Read an artifact @@ -251,8 +251,8 @@ A single `patch` string may contain multiple `diff --git` sections. Encode the envelope using the same codec pipeline as fragment links: 1. Serialize envelope as compact JSON -2. Encode with a codec (`plain` = base64url, `lz` = lz-string, `deflate` = deflate + base64url, or the async arx/arx2/arx3 pipelines) -3. Prepend the single-character codec tag (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3) +2. Encode with a codec (`plain` = base64url, `lz` = lz-string, `deflate` = deflate + base64url, or the async arx/arx2/arx3 pipelines; `arx4` is emitted by the app only, since its context mixer is not hand-rollable) +3. Prepend the single-character codec tag (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4) 4. POST the resulting string as the `payload` field For simple cases, `plain` codec is sufficient: diff --git a/src/lib/payload/arx-codec.ts b/src/lib/payload/arx-codec.ts index 2912560..85e7ae4 100644 --- a/src/lib/payload/arx-codec.ts +++ b/src/lib/payload/arx-codec.ts @@ -252,6 +252,7 @@ function applySubstitutionTrie(text: string, trie: SubstitutionTrieNode): string // a slot so the loaders/getters are written once. Both slots start on their built-in dictionaries. type DictSlot = { table: SubstitutionTable; + dictionary: ArxDictionary; version: number; loaded: boolean; buildPairs: (dict: ArxDictionary) => SubstitutionPair[]; @@ -260,6 +261,7 @@ type DictSlot = { const baseDictSlot: DictSlot = { table: buildSubstitutionTable(buildSubstitutions(BUILTIN_DICTIONARY)), + dictionary: BUILTIN_DICTIONARY, version: BUILTIN_DICTIONARY.version, loaded: false, buildPairs: (dict) => buildSubstitutions(dict), @@ -268,6 +270,7 @@ const baseDictSlot: DictSlot = { const overlayDictSlot: DictSlot = { table: buildSubstitutionTable(buildOverlaySubstitutionPairs(BUILTIN_ARX2_OVERLAY_DICTIONARY)), + dictionary: BUILTIN_ARX2_OVERLAY_DICTIONARY, version: BUILTIN_ARX2_OVERLAY_DICTIONARY.version, loaded: false, buildPairs: buildOverlaySubstitutionPairs, @@ -351,6 +354,7 @@ async function loadDictSlot(slot: DictSlot, source?: string | ArxDictionary): Pr /** Load a dictionary slot from a pre-parsed object (synchronous). */ function loadDictSlotSync(slot: DictSlot, dict: ArxDictionary): number { slot.table = buildSubstitutionTable(slot.buildPairs(dict)); + slot.dictionary = dict; slot.version = dict.version; slot.loaded = true; return dict.version; @@ -409,6 +413,21 @@ export function getActiveArx2OverlayVersion(): number { return overlayDictSlot.version; } +/** + * Newline-joined slot text of the active base + overlay dictionaries, in canonical slot order. + * This is the arx4 context mixer's priming corpus (see arx4-codec.ts): it is the only text both + * encoder and decoder are guaranteed to agree on byte-for-byte, because the compact arx tags + * already pin the dictionary version for the substitution stage. + */ +export function getArxDictionaryPriorText(): string { + return [ + ...baseDictSlot.dictionary.singleByteSlots, + ...baseDictSlot.dictionary.extendedSlots, + ...overlayDictSlot.dictionary.singleByteSlots, + ...overlayDictSlot.dictionary.extendedSlots, + ].join("\n"); +} + function resolveDefaultDictionaryUrl(): string { return withBasePath("/arx-dictionary.json"); } @@ -585,7 +604,7 @@ function decodeArx2ArtifactTuple(value: unknown): ArtifactPayload { } } -function envelopeFromArxTuple(value: unknown, codec: Extract): PayloadEnvelope { +function envelopeFromArxTuple(value: unknown, codec: Extract): PayloadEnvelope { if (!Array.isArray(value)) { throw new Error("Invalid arx2 envelope tuple."); } @@ -731,7 +750,7 @@ export function decodeBase76(str: string): Uint8Array { digitsStart = 2; } - assertWireByteLen(byteLen); + assertArxWireByteLength(byteLen); let num = BIGINT_0; for (let i = digitsStart; i < str.length; i++) { @@ -801,7 +820,7 @@ export function decodeBase1k(str: string): Uint8Array { const lenLow = UNICODE_CHAR_TO_INDEX.get(str[1]) ?? 0; const byteLen = lenHigh * UNICODE_ALPHABET.length + lenLow; - assertWireByteLen(byteLen); + assertArxWireByteLength(byteLen); let num = BIGINT_0; for (let i = 2; i < str.length; i++) { @@ -1042,7 +1061,7 @@ export function decodeBaseBMP(str: string): Uint8Array { const lenLow = BMP_CHAR_TO_INDEX.get(s[1]) ?? 0; const byteLen = lenHigh * BMP_ALPHABET.length + lenLow; - assertWireByteLen(byteLen); + assertArxWireByteLength(byteLen); let num = BIGINT_0; for (let i = 2; i < s.length; i++) { @@ -1171,14 +1190,15 @@ function assertDecodedTextBudget(text: string): void { } /** - * Bounds a wire byte length decoded from an attacker-controlled base-N length prefix BEFORE any - * allocation or per-byte loop. The prefix is tiny but can claim a huge count (baseBMP's 2-char - * prefix reaches ~3.8e9), which would otherwise peg a core for ~a minute on a multi-GB allocation - * — the existing decoded-size budget only runs after decompression, far too late. A real - * compressed payload is always far below MAX_BROTLI_OUTPUT_BYTES, so anything larger is provably - * implausible and rejected as decoded-too-large (caught upstream in decodeFragmentAsync). + * Bounds a byte length decoded from an attacker-controlled length prefix BEFORE any allocation or + * per-byte loop. The prefix is tiny but can claim a huge count (baseBMP's 2-char prefix reaches + * ~3.8e9), which would otherwise peg a core for ~a minute on a multi-GB allocation — the existing + * decoded-size budget only runs after decompression, far too late. A real compressed payload is + * always far below MAX_BROTLI_OUTPUT_BYTES, so anything larger is provably implausible and rejected + * as decoded-too-large (caught upstream in decodeFragmentAsync). Exported for arx4, whose coded + * payload carries its own decoded-length varint. */ -function assertWireByteLen(byteLen: number): void { +export function assertArxWireByteLength(byteLen: number): void { // Reject non-integer lengths too: a malformed prefix can yield NaN (e.g. an out-of-range // charCodeAt), and `NaN > MAX_BROTLI_OUTPUT_BYTES` is false, which would otherwise slip the // guard and decode to a misleading empty array instead of a rejection. @@ -1261,7 +1281,8 @@ async function compressArxJson(json: string): Promise { return compressSubstitutedText(dictEncode(json)); } -function encodeWirePayloads(compressed: Uint8Array): ArxWirePayloads { +/** Every supported binary-to-text wire shape for one compressed payload. */ +export function encodeArxWirePayloads(compressed: Uint8Array): ArxWirePayloads { return { base76: encodeBase76(compressed), base1k: encodeBase1k(compressed), @@ -1270,34 +1291,42 @@ function encodeWirePayloads(compressed: Uint8Array): ArxWirePayloads { }; } -async function decompressWirePayload(encoded: string): Promise { - const brotli = await getBrotli(); - - const decompressFromBytes = (bytes: Uint8Array): string => { - const out = brotliDecompressWithLimit(brotli, bytes); - return new TextDecoder().decode(out); - }; - +/** + * Detects which wire alphabet `encoded` uses and hands the decoded bytes to `decodePayloadBytes`. + * Shared by the Brotli codecs and arx4's context mixer so the alphabet dispatch is written once. + * + * The base76 length prefix can itself be `B.` (e.g. 140-byte payloads), so a base64url read whose + * payload stage rejects the bytes is retried as base76. A decoded-too-large rejection is a real + * answer, not a mis-detection, so it propagates instead of triggering the retry. + */ +export function decodeArxWirePayload(encoded: string, decodePayloadBytes: (bytes: Uint8Array) => T): T { if (isBaseBMPEncoded(encoded)) { - return decompressFromBytes(decodeBaseBMP(encoded)); + return decodePayloadBytes(decodeBaseBMP(encoded)); } if (isBase64urlEncoded(encoded)) { try { - return decompressFromBytes(decodeBase64url(encoded)); + return decodePayloadBytes(decodeBase64url(encoded)); } catch (error) { if (error instanceof ArxDecodedPayloadTooLargeError) { throw error; } - // base76 length prefix can also be `B.` (e.g. 140-byte payloads); retry as base76. } } if (isBase1kEncoded(encoded)) { - return decompressFromBytes(decodeBase1k(encoded)); + return decodePayloadBytes(decodeBase1k(encoded)); } - return decompressFromBytes(decodeBase76(encoded)); + return decodePayloadBytes(decodeBase76(encoded)); +} + +async function decompressWirePayload(encoded: string): Promise { + const brotli = await getBrotli(); + + return decodeArxWirePayload(encoded, (bytes) => ( + new TextDecoder().decode(brotliDecompressWithLimit(brotli, bytes)) + )); } // --------------------------------------------------------------------------- @@ -1317,7 +1346,7 @@ export async function arxCompress(json: string): Promise { */ export async function arxCompressPayloads(json: string): Promise { const compressed = await compressArxJson(json); - return encodeWirePayloads(compressed); + return encodeArxWirePayloads(compressed); } /** @@ -1350,10 +1379,11 @@ export async function arxCompressBase64url(json: string): Promise { } /** - * Compresses a payload envelope with the arx2 tuple-envelope pipeline. - * Returns all supported binary-to-text wire shapes so callers can choose by transport size. + * The tuple-envelope pipeline's pre-entropy stage: envelope → compact tuple JSON → arx2 overlay + * substitution → shared dictionary substitution. Every tuple codec entropy-codes exactly these + * bytes (Brotli for arx2/arx3, the context mixer for arx4), so they all share this one derivation. */ -async function compressTupleEnvelope(envelope: PayloadEnvelope): Promise { +export function substituteArxTupleText(envelope: PayloadEnvelope): string { // The arx2/arx3 overlay repurposes 0x7F (DEL) as a single-byte substitution // code (see ARX2_SINGLE_BYTE_CODES). JSON.stringify escapes every C0 control // byte (< 0x20) — which covers all the other substitution code bytes — but @@ -1364,9 +1394,12 @@ async function compressTupleEnvelope(envelope: PayloadEnvelope): Promise { + const compressed = await compressSubstitutedText(substituteArxTupleText(envelope)); + return encodeArxWirePayloads(compressed); } /** @@ -1394,11 +1427,14 @@ export async function arxDecompress(encoded: string): Promise { } /** - * Decompresses an arx2/arx3 tuple-envelope payload and rebuilds the standard envelope shape. The two - * codecs share the same tuple/overlay wire bytes; only the codec stamped on the rebuilt envelope differs. + * Inverse of {@link substituteArxTupleText}, ending at a rebuilt envelope. Shared by every tuple + * codec; only the codec stamped on the rebuilt envelope differs. */ -async function decompressArxTupleEnvelope(encoded: string, codec: "arx2" | "arx3"): Promise { - const v1Decoded = dictDecode(await decompressWirePayload(encoded)); +export function envelopeFromSubstitutedArxTupleText( + substituted: string, + codec: "arx2" | "arx3" | "arx4", +): PayloadEnvelope { + const v1Decoded = dictDecode(substituted); assertWithinExpansionBudget(v1Decoded); const tupleJson = overlayDecode(v1Decoded); assertWithinExpansionBudget(tupleJson); @@ -1411,6 +1447,10 @@ async function decompressArxTupleEnvelope(encoded: string, codec: "arx2" | "arx3 return envelopeFromArxTuple(tuple, codec); } +async function decompressArxTupleEnvelope(encoded: string, codec: "arx2" | "arx3"): Promise { + return envelopeFromSubstitutedArxTupleText(await decompressWirePayload(encoded), codec); +} + /** Decompresses an arx2 tuple-envelope payload and rebuilds the standard envelope shape. */ export async function arx2DecompressEnvelope(encoded: string): Promise { return decompressArxTupleEnvelope(encoded, "arx2"); diff --git a/src/lib/payload/arx4-codec.ts b/src/lib/payload/arx4-codec.ts new file mode 100644 index 0000000..625cf88 --- /dev/null +++ b/src/lib/payload/arx4-codec.ts @@ -0,0 +1,1011 @@ +/** + * arx4 codec: the context-mixing entropy stage for the arx tuple pipeline. + * + * Pipeline: envelope → compact tuple JSON → overlay + shared dictionary substitution → + * context-mixing arithmetic coder → binary-to-text wire encoding. arx4 replaces only arx3's Brotli + * stage; the tuple and substitution stages are the same functions arx2/arx3 call. + * + * Fragment shape: ``. The prior id names the priming corpus the coder ran + * before the payload, because a decoder has to reproduce the encoder's model state exactly. The + * curated part of those corpora ships as the lazily fetched `/arx4-priors.json` asset. + * + * The model is the frozen ARX4 experiment codec (`CM_SOURCE` in scripts/arx4-cm-determinism.mjs, + * sha256 1f94ebb1fec5207df00e1ecdced7805c0c1d71dbc1c13c4c0baf11bc7b995f01, benchmarked in + * docs/arx4-cm-bench.md) ported to TypeScript, plus one added mixer input: a column-position context + * that predicts a table cell from the cell above it, worth a p50 15.9 percent gain on csv with no + * subset regressed. + * + * Every coding decision is integer-only: no floats, no Date, no Math.random. That is what makes + * encode bit-identical across Node, Chromium and WebKit (docs/arx4-cm-determinism.md), and a single + * ambient or floating-point input in the coding path would silently break every link already shared. + * + * Cost: about 40 ms to code an 8 KB artifact and 770 ms for 60 KB, against ~8 ms for Brotli. That is + * why the whole arx family is async-only. + */ + +import { + assertArxWireByteLength, + decodeArxWirePayload, + encodeArxWirePayloads, + envelopeFromSubstitutedArxTupleText, + getArxDictionaryPriorText, + substituteArxTupleText, + type ArxWirePayloads, +} from "@/lib/payload/arx-codec"; +import type { ArtifactKind, PayloadEnvelope } from "@/lib/payload/schema"; +import { sha256Hex } from "@/lib/sha256"; +import { withBasePath } from "@/lib/site/base-path"; + +// --------------------------------------------------------------------------- +// Model geometry (frozen: these numbers are part of the wire format) +// --------------------------------------------------------------------------- + +const TABLE_BITS = 20; +const TABLE_SIZE = 1 << TABLE_BITS; +const TABLE_SHIFT = 32 - TABLE_BITS; +/** Byte orders per direct-context model; -1 is the current-word model. */ +const MODEL_ORDERS = [0, 1, 2, 3, 4, 6, -1]; +const MODEL_COUNT = MODEL_ORDERS.length; +const MATCH_INPUT_COUNT = 2; +/** Mixer input index of the column-position context, appended after the two match inputs. */ +const COLUMN_INPUT_INDEX = MODEL_COUNT + MATCH_INPUT_COUNT; +const MIXER_INPUT_COUNT = COLUMN_INPUT_INDEX + 1; +const MIXER_CONTEXTS = 256; +const MIXER_WEIGHT_SCALE = 1 << 12; +const MIXER_WEIGHT_LIMIT = 4 * MIXER_WEIGHT_SCALE; +const MIXER_LEARNING_DIVISOR = 25_600; +const STRETCH_SCALE = 1 << 8; +const STRETCH_LIMIT = 8 * STRETCH_SCALE; +const MATCH_HASH_BYTES = 7; +const MATCH_TABLE_BITS = 18; +const MATCH_TABLE_SIZE = 1 << MATCH_TABLE_BITS; +const MATCH_BUFFER_SIZE = 1 << 19; +const MAX_MODEL_COUNT = 31; +const HISTORY_BYTES = 6; + +const COLUMN_TABLE_SIZE = 1 << 20; +const COLUMN_MIN_DELIMITERS = 2; +const COLUMN_MAX_FIELD_INDEX = 31; +const COLUMN_MAX_FIELD_OFFSET = 63; +const COLUMN_MAX_LINE_BYTES = 1024; +/** Stands in for "the row above has no byte here", so it must sit outside the byte range. */ +const COLUMN_ABOVE_NONE = 256; +const COLUMN_DOMAIN_TAG = 0x54; +const NEWLINE_BYTE = 0x0a; +const PIPE_BYTE = 0x7c; +const COMMA_BYTE = 0x2c; + +const FNV_OFFSET_BASIS = 0x811c9dc5; +const FNV_PRIME = 0x01000193; + +// --------------------------------------------------------------------------- +// Integer arithmetic helpers +// --------------------------------------------------------------------------- + +/** Round-half-away-from-zero integer division, symmetric about zero so weights train evenly. */ +function divideRound(numerator: number, denominator: number): number { + if (numerator >= 0) return Math.floor((numerator + Math.floor(denominator / 2)) / denominator); + return -Math.floor((-numerator + Math.floor(denominator / 2)) / denominator); +} + +const BIGINT_1 = BigInt(1); +const BIGINT_31 = BigInt(31); +const BIGINT_2_POW_32 = BigInt(1) << BigInt(32); + +/** log2 of a positive integer in Q16 fixed point, via BigInt squaring so no float ever appears. */ +function log2Q16(value: number): number { + const integerPart = 31 - Math.clz32(value); + let normalized = BigInt(value) << BigInt(31 - integerPart); + let fraction = 0; + for (let bit = 15; bit >= 0; bit--) { + normalized = (normalized * normalized) >> BIGINT_31; + if (normalized >= BIGINT_2_POW_32) { + normalized >>= BIGINT_1; + fraction |= 1 << bit; + } + } + return integerPart * 65_536 + fraction; +} + +/** Q12 probability → signed Q8 log-odds. */ +const stretchTable = new Int16Array(4096); +for (let probability = 1; probability < 4096; probability++) { + const log2RatioQ16 = log2Q16(probability) - log2Q16(4096 - probability); + stretchTable[probability] = divideRound(log2RatioQ16 * 45_426, 1 << 24); +} + +/** Inverse of `stretchTable`, built by search so the two stay consistent by construction. */ +const squashTable = new Uint16Array(STRETCH_LIMIT * 2 + 1); +for (let stretch = -STRETCH_LIMIT; stretch <= STRETCH_LIMIT; stretch++) { + let low = 1; + let high = 4095; + while (low < high) { + const middle = (low + high) >>> 1; + if (stretchTable[middle] < stretch) low = middle + 1; + else high = middle; + } + const previous = Math.max(1, low - 1); + squashTable[stretch + STRETCH_LIMIT] = ( + Math.abs(stretchTable[previous] - stretch) <= Math.abs(stretchTable[low] - stretch) + ? previous + : low + ); +} + +function squashStretch(stretch: number): number { + const clamped = Math.max(-STRETCH_LIMIT, Math.min(STRETCH_LIMIT, stretch)); + return squashTable[clamped + STRETCH_LIMIT]; +} + +/** + * Adaptive slot state: Q12 probability in the low 12 bits, observation count in bits 12-17, and an + * 8-bit check tag in bits 18-25. The tag is what makes a hash collision read as "unseen slot" + * instead of as another context's statistics. + */ +function updatePackedState(packed: number, bit: number): number { + const probability = packed === 0 ? 2048 : packed & 0x0fff; + const count = packed === 0 ? 0 : (packed >>> 12) & 0x3f; + const target = bit === 1 ? 4096 : 0; + const numerator = count === MAX_MODEL_COUNT ? target - probability : 2 * (target - probability); + const denominator = count === MAX_MODEL_COUNT ? 32 : 2 * count + 3; + const nextProbability = Math.max(1, Math.min(4095, probability + divideRound(numerator, denominator))); + const nextCount = Math.min(MAX_MODEL_COUNT, count + 1); + return nextProbability | (nextCount << 12); +} + +/** Probability held in a tagged slot, or the 2048 prior when the slot is empty or collided. */ +function taggedSlotProbability(packed: number, tag: number): number { + return packed === 0 || ((packed >>> 18) & 0xff) !== tag ? 2048 : packed & 0x0fff; +} + +function isWordByte(byte: number): boolean { + return ( + (byte >= 0x30 && byte <= 0x39) + || (byte >= 0x41 && byte <= 0x5a) + || byte === 0x5f + || (byte >= 0x61 && byte <= 0x7a) + ); +} + +type ColumnLineStats = { length: number; pipes: number; commas: number }; + +// --------------------------------------------------------------------------- +// Context mixing model +// --------------------------------------------------------------------------- + +class ContextMixModel { + private readonly tables: Uint32Array[] = Array.from( + { length: MODEL_COUNT }, + () => new Uint32Array(TABLE_SIZE), + ); + private readonly weights = new Int32Array(MIXER_CONTEXTS * MIXER_INPUT_COUNT); + private readonly initializedMixers = new Uint8Array(MIXER_CONTEXTS); + private readonly historyHashes = new Uint32Array(MODEL_COUNT); + private readonly matchTable = new Uint32Array(MATCH_TABLE_SIZE); + private readonly matchBuffer = new Uint8Array(MATCH_BUFFER_SIZE); + private readonly columnTable = new Uint32Array(COLUMN_TABLE_SIZE); + private readonly cachedIndexes = new Uint32Array(MODEL_COUNT); + private readonly cachedTags = new Uint8Array(MODEL_COUNT); + private readonly cachedStretches = new Int16Array(MIXER_INPUT_COUNT); + + private history: number[] = []; + private byteCount = 0; + private matchPosition = -1; + private matchLength = 0; + private wordHash = 0; + private wordLength = 0; + private c0 = 1; + private bitShift = 7; + private cachedMixerOffset = 0; + private cachedRawProbability = 2048; + private cachedMatchBit = -1; + + private columnLineStats: ColumnLineStats[] = []; + private columnPreviousLineBytes: number[] = []; + private columnCurrentLineBytes: number[] = []; + private columnCurrentLength = 0; + private columnCurrentPipes = 0; + private columnCurrentCommas = 0; + private columnRowActive = false; + private columnDelimiter = -1; + private columnFieldIndex = 0; + private columnFieldOffset = 0; + private columnPreviousFields: number[][] | null = null; + private columnSlotIndex = -1; + private columnTag = 0; + private columnHash = 0; + + constructor() { + this.prepareByteContexts(); + } + + private hashHistory(order: number): number { + let hash = (FNV_OFFSET_BASIS ^ order ^ (this.history.length << 24)) >>> 0; + for (let index = 0; index < order; index++) { + const value = index < this.history.length ? this.history[index] + 1 : 0; + hash = Math.imul(hash ^ value ^ (index << 8), FNV_PRIME) >>> 0; + } + return hash; + } + + private prepareByteContexts(): void { + this.historyHashes[0] = 0x243f6a88; + for (let index = 1; index < MODEL_COUNT - 1; index++) { + this.historyHashes[index] = this.hashHistory(MODEL_ORDERS[index]); + } + this.historyHashes[MODEL_COUNT - 1] = ( + this.wordLength === 0 + ? 0x9e3779b9 + : (this.wordHash ^ Math.imul(this.wordLength, 0x85ebca6b)) + ) >>> 0; + } + + private matchByteAt(position: number): number { + return this.matchBuffer[position & (MATCH_BUFFER_SIZE - 1)]; + } + + private isReadableMatchPosition(position: number): boolean { + return ( + position >= 0 + && position < this.byteCount + && this.byteCount - position <= MATCH_BUFFER_SIZE + ); + } + + private matchHash(endPosition: number): number { + let hash = FNV_OFFSET_BASIS; + for (let offset = MATCH_HASH_BYTES - 1; offset >= 0; offset--) { + hash = Math.imul(hash ^ this.matchByteAt(endPosition - offset), FNV_PRIME) >>> 0; + } + return hash >>> (32 - MATCH_TABLE_BITS); + } + + private matchContextsEqual(leftEnd: number, rightEnd: number): boolean { + for (let offset = 0; offset < MATCH_HASH_BYTES; offset++) { + if (this.matchByteAt(leftEnd - offset) !== this.matchByteAt(rightEnd - offset)) return false; + } + return true; + } + + private updateMatch(byte: number): void { + const matchedWholeByte = this.matchPosition >= 0; + if (matchedWholeByte) { + this.matchPosition++; + this.matchLength = Math.min(255, this.matchLength + 1); + } + + const currentPosition = this.byteCount; + this.matchBuffer[currentPosition & (MATCH_BUFFER_SIZE - 1)] = byte; + this.byteCount++; + if (this.byteCount < MATCH_HASH_BYTES) return; + + const hash = this.matchHash(currentPosition); + const previousEnd = this.matchTable[hash] - 1; + this.matchTable[hash] = currentPosition + 1; + if (matchedWholeByte) return; + + const candidateNext = previousEnd + 1; + if ( + previousEnd >= MATCH_HASH_BYTES - 1 + && this.isReadableMatchPosition(candidateNext) + && this.matchContextsEqual(previousEnd, currentPosition) + ) { + this.matchPosition = candidateNext; + this.matchLength = MATCH_HASH_BYTES; + } else { + this.matchPosition = -1; + this.matchLength = 0; + } + } + + /** + * Row detection is causal: only the two already-coded lines before this one decide whether it is + * a table row, so the decoder reaches the same conclusion from the same bytes. + */ + private beginColumnRow(): void { + this.columnRowActive = false; + this.columnDelimiter = -1; + this.columnFieldIndex = 0; + this.columnFieldOffset = 0; + this.columnPreviousFields = null; + + const previous = this.columnLineStats[0]; + const older = this.columnLineStats[1]; + if (previous === undefined || older === undefined) return; + if (previous.length === 0 || older.length === 0) return; + + if (previous.pipes === older.pipes && previous.pipes >= COLUMN_MIN_DELIMITERS) { + this.columnDelimiter = PIPE_BYTE; + } else if (previous.commas === older.commas && previous.commas >= COLUMN_MIN_DELIMITERS) { + this.columnDelimiter = COMMA_BYTE; + } else { + return; + } + + this.columnRowActive = true; + this.columnPreviousFields = this.splitColumnFields(this.columnPreviousLineBytes); + } + + private splitColumnFields(lineBytes: number[]): number[][] { + const fields: number[][] = []; + let field: number[] = []; + + for (let index = 0; index < lineBytes.length; index++) { + const byte = lineBytes[index]; + if (byte === this.columnDelimiter) { + fields.push(field); + if (fields.length > COLUMN_MAX_FIELD_INDEX) return fields; + field = []; + continue; + } + if (field.length <= COLUMN_MAX_FIELD_OFFSET) field.push(byte); + } + + fields.push(field); + return fields; + } + + private updateColumnState(byte: number): void { + if (byte === NEWLINE_BYTE) { + this.columnLineStats.unshift({ + length: this.columnCurrentLength, + pipes: this.columnCurrentPipes, + commas: this.columnCurrentCommas, + }); + if (this.columnLineStats.length > 2) this.columnLineStats.length = 2; + this.columnPreviousLineBytes = this.columnCurrentLineBytes; + this.columnCurrentLineBytes = []; + this.columnCurrentLength = 0; + this.columnCurrentPipes = 0; + this.columnCurrentCommas = 0; + this.beginColumnRow(); + } else { + if (this.columnCurrentLineBytes.length < COLUMN_MAX_LINE_BYTES) { + this.columnCurrentLineBytes.push(byte); + } + this.columnCurrentLength++; + if (byte === PIPE_BYTE) this.columnCurrentPipes++; + else if (byte === COMMA_BYTE) this.columnCurrentCommas++; + if (this.columnRowActive) { + if (byte === this.columnDelimiter) { + this.columnFieldIndex++; + this.columnFieldOffset = 0; + } else { + this.columnFieldOffset++; + } + } + } + + this.columnHash = this.columnContextHash(); + } + + /** Keys on (field index, byte offset in field, byte at the same cell position one row up). */ + private columnContextHash(): number { + if (!this.columnRowActive) return 0; + + const fieldIndex = Math.min(this.columnFieldIndex, COLUMN_MAX_FIELD_INDEX); + const fieldOffset = Math.min(this.columnFieldOffset, COLUMN_MAX_FIELD_OFFSET); + const field = this.columnPreviousFields === null ? undefined : this.columnPreviousFields[fieldIndex]; + const above = field !== undefined && fieldOffset < field.length ? field[fieldOffset] : COLUMN_ABOVE_NONE; + + let hash = FNV_OFFSET_BASIS; + hash = Math.imul(hash ^ COLUMN_DOMAIN_TAG, FNV_PRIME) >>> 0; + hash = Math.imul(hash ^ (fieldIndex + 1), FNV_PRIME) >>> 0; + hash = Math.imul(hash ^ (fieldOffset + 1), FNV_PRIME) >>> 0; + hash = Math.imul(hash ^ (above + 1), FNV_PRIME) >>> 0; + // 0 is reserved for "context inactive", so a hash that lands there is nudged off it. + return hash === 0 ? 1 : hash; + } + + predict(): number { + const mixerContext = this.c0 & 0xff; + const mixerOffset = mixerContext * MIXER_INPUT_COUNT; + if (this.initializedMixers[mixerContext] === 0) { + for (let index = 0; index < MODEL_COUNT; index++) { + this.weights[mixerOffset + index] = divideRound(MIXER_WEIGHT_SCALE, MODEL_COUNT); + } + this.weights[mixerOffset + MODEL_COUNT] = MIXER_WEIGHT_SCALE; + this.weights[mixerOffset + MODEL_COUNT + 1] = MIXER_WEIGHT_SCALE; + this.weights[mixerOffset + COLUMN_INPUT_INDEX] = divideRound(MIXER_WEIGHT_SCALE, MODEL_COUNT); + this.initializedMixers[mixerContext] = 1; + } + + let mixedStretch = 0; + for (let index = 0; index < MODEL_COUNT; index++) { + const slotHash = ( + Math.imul(this.historyHashes[index], 0x9e3779b1) + ^ Math.imul(this.c0, 0x85ebca6b) + ) >>> 0; + const slotIndex = slotHash >>> TABLE_SHIFT; + const tag = slotHash & 0xff; + const stretched = stretchTable[taggedSlotProbability(this.tables[index][slotIndex], tag)]; + this.cachedIndexes[index] = slotIndex; + this.cachedTags[index] = tag; + this.cachedStretches[index] = stretched; + mixedStretch += this.weights[mixerOffset + index] * stretched; + } + + this.cachedMatchBit = -1; + this.cachedStretches[MODEL_COUNT] = 0; + this.cachedStretches[MODEL_COUNT + 1] = 0; + if (this.isReadableMatchPosition(this.matchPosition)) { + const matchBit = (this.matchByteAt(this.matchPosition) >>> this.bitShift) & 1; + const direction = matchBit === 1 ? 1 : -1; + const cappedMatchLength = Math.min(this.matchLength, 32); + this.cachedMatchBit = matchBit; + this.cachedStretches[MODEL_COUNT] = direction * cappedMatchLength * 64; + this.cachedStretches[MODEL_COUNT + 1] = direction * Math.max(0, cappedMatchLength - 11) * 64; + mixedStretch += ( + this.weights[mixerOffset + MODEL_COUNT] * this.cachedStretches[MODEL_COUNT] + + this.weights[mixerOffset + MODEL_COUNT + 1] * this.cachedStretches[MODEL_COUNT + 1] + ); + } + + this.columnSlotIndex = -1; + this.cachedStretches[COLUMN_INPUT_INDEX] = 0; + if (this.columnHash !== 0) { + const slotHash = ( + Math.imul(this.columnHash, 0x9e3779b1) + ^ Math.imul(this.c0, 0x85ebca6b) + ) >>> 0; + const slotIndex = slotHash >>> TABLE_SHIFT; + const tag = slotHash & 0xff; + const stretched = stretchTable[taggedSlotProbability(this.columnTable[slotIndex], tag)]; + this.columnSlotIndex = slotIndex; + this.columnTag = tag; + this.cachedStretches[COLUMN_INPUT_INDEX] = stretched; + mixedStretch += this.weights[mixerOffset + COLUMN_INPUT_INDEX] * stretched; + } + + const rawProbability = squashStretch(divideRound(mixedStretch, MIXER_WEIGHT_SCALE)); + this.cachedMixerOffset = mixerOffset; + this.cachedRawProbability = rawProbability; + return rawProbability; + } + + update(bit: number): void { + const error = bit * 4096 - this.cachedRawProbability; + for (let index = 0; index < MIXER_INPUT_COUNT; index++) { + const weightIndex = this.cachedMixerOffset + index; + const nextWeight = this.weights[weightIndex] + divideRound( + error * this.cachedStretches[index], + MIXER_LEARNING_DIVISOR, + ); + this.weights[weightIndex] = Math.max(-MIXER_WEIGHT_LIMIT, Math.min(MIXER_WEIGHT_LIMIT, nextWeight)); + } + + for (let index = 0; index < MODEL_COUNT; index++) { + const slotIndex = this.cachedIndexes[index]; + const tag = this.cachedTags[index]; + const packed = this.tables[index][slotIndex]; + const matchingState = packed !== 0 && ((packed >>> 18) & 0xff) === tag ? packed : 0; + this.tables[index][slotIndex] = updatePackedState(matchingState, bit) | (tag << 18); + } + + if (this.columnSlotIndex >= 0) { + const packed = this.columnTable[this.columnSlotIndex]; + const matchingState = packed !== 0 && ((packed >>> 18) & 0xff) === this.columnTag ? packed : 0; + this.columnTable[this.columnSlotIndex] = ( + updatePackedState(matchingState, bit) | (this.columnTag << 18) + ); + } + + if (this.cachedMatchBit >= 0 && bit !== this.cachedMatchBit) { + this.matchPosition = -1; + this.matchLength = 0; + } + + this.c0 = (this.c0 << 1) | bit; + this.bitShift--; + } + + private finishByte(byte: number): void { + this.updateMatch(byte); + this.history.unshift(byte); + if (this.history.length > HISTORY_BYTES) this.history.length = HISTORY_BYTES; + + if (isWordByte(byte)) { + if (this.wordLength === 0) this.wordHash = FNV_OFFSET_BASIS; + this.wordHash = Math.imul(this.wordHash ^ byte, FNV_PRIME) >>> 0; + this.wordLength = Math.min(255, this.wordLength + 1); + } else { + this.wordHash = 0; + this.wordLength = 0; + } + + this.c0 = 1; + this.bitShift = 7; + this.prepareByteContexts(); + this.updateColumnState(byte); + } + + processKnownByte(byte: number, consumePrediction: (probability: number, bit: number) => void): void { + for (let shift = 7; shift >= 0; shift--) { + const probability = this.predict(); + const bit = (byte >>> shift) & 1; + consumePrediction(probability, bit); + this.update(bit); + } + this.finishByte(byte); + } + + processDecodedByte(readBit: (probability: number) => number): number { + let byte = 0; + for (let shift = 7; shift >= 0; shift--) { + const probability = this.predict(); + const bit = readBit(probability); + byte |= bit << shift; + this.update(bit); + } + this.finishByte(byte); + return byte; + } +} + +// --------------------------------------------------------------------------- +// Binary arithmetic coder +// --------------------------------------------------------------------------- + +class BinaryArithmeticEncoder { + private x1 = 0; + private x2 = 0xffffffff; + private readonly output: number[] = []; + + writeBit(bit: number, probability: number): void { + const xmid = this.x1 + Math.floor((this.x2 - this.x1) / 4096) * probability; + if (xmid < this.x1 || xmid > this.x2) { + throw new Error(`arithmetic encoder midpoint escaped range: ${this.x1} <= ${xmid} <= ${this.x2}`); + } + + if (bit === 1) { + this.x2 = xmid >>> 0; + } else { + this.x1 = (xmid + 1) >>> 0; + } + + while (((this.x1 ^ this.x2) & 0xff000000) === 0) { + this.output.push(this.x2 >>> 24); + this.x1 = (this.x1 << 8) >>> 0; + this.x2 = ((this.x2 << 8) | 0xff) >>> 0; + } + } + + finish(): Uint8Array { + this.output.push(this.x1 >>> 24, (this.x1 >>> 16) & 0xff, (this.x1 >>> 8) & 0xff, this.x1 & 0xff); + return Uint8Array.from(this.output); + } +} + +class BinaryArithmeticDecoder { + private readonly input: Uint8Array; + private offset = 4; + private x1 = 0; + private x2 = 0xffffffff; + private x: number; + + constructor(input: Uint8Array) { + if (input.length < 4) throw new Error("arithmetic payload is shorter than its four-byte flush"); + this.input = input; + this.x = (input[0] * 0x1000000 + input[1] * 0x10000 + input[2] * 0x100 + input[3]) >>> 0; + } + + private readByte(): number { + if (this.offset >= this.input.length) { + throw new Error("arithmetic decoder exhausted the coded payload"); + } + return this.input[this.offset++]; + } + + readBit(probability: number): number { + const xmid = this.x1 + Math.floor((this.x2 - this.x1) / 4096) * probability; + if (xmid < this.x1 || xmid > this.x2) { + throw new Error(`arithmetic decoder midpoint escaped range: ${this.x1} <= ${xmid} <= ${this.x2}`); + } + + const bit = this.x <= xmid ? 1 : 0; + if (bit === 1) { + this.x2 = xmid >>> 0; + } else { + this.x1 = (xmid + 1) >>> 0; + } + + while (((this.x1 ^ this.x2) & 0xff000000) === 0) { + this.x1 = (this.x1 << 8) >>> 0; + this.x2 = ((this.x2 << 8) | 0xff) >>> 0; + this.x = ((this.x << 8) | this.readByte()) >>> 0; + } + return bit; + } +} + +function encodeVarint(value: number): number[] { + const bytes: number[] = []; + let remaining = value; + do { + let byte = remaining % 128; + remaining = Math.floor(remaining / 128); + if (remaining > 0) byte |= 0x80; + bytes.push(byte); + } while (remaining > 0); + return bytes; +} + +function decodeVarint(input: Uint8Array): { value: number; bytesRead: number } { + let value = 0; + let multiplier = 1; + for (let offset = 0; offset < input.length && offset < 8; offset++) { + const byte = input[offset]; + value += (byte & 0x7f) * multiplier; + if ((byte & 0x80) === 0) return { value, bytesRead: offset + 1 }; + multiplier *= 128; + } + throw new Error("invalid or truncated varint"); +} + +/** + * Runs the priming bytes through the model without coding them, which is what lets a fixed corpus + * shape both sides' statistics for free. Priming also fills the match ring, so the prior supplies + * match candidates as well as byte statistics. + */ +function primeModel(model: ContextMixModel, primeBytes: Uint8Array): void { + for (const byte of primeBytes) { + model.processKnownByte(byte, () => {}); + } +} + +function encodeCm(input: Uint8Array, primeBytes: Uint8Array | null): Uint8Array { + const model = new ContextMixModel(); + if (primeBytes) primeModel(model, primeBytes); + + const coder = new BinaryArithmeticEncoder(); + for (const byte of input) { + model.processKnownByte(byte, (probability, bit) => coder.writeBit(bit, probability)); + } + + const lengthPrefix = encodeVarint(input.length); + const coded = coder.finish(); + const output = new Uint8Array(lengthPrefix.length + coded.length); + output.set(lengthPrefix, 0); + output.set(coded, lengthPrefix.length); + return output; +} + +function decodeCm(input: Uint8Array, primeBytes: Uint8Array | null): Uint8Array { + const { value: byteLength, bytesRead } = decodeVarint(input); + // The varint is attacker-controlled and reaches 2^56, so bound it before allocating the output. + assertArxWireByteLength(byteLength); + + const model = new ContextMixModel(); + if (primeBytes) primeModel(model, primeBytes); + + const coder = new BinaryArithmeticDecoder(input.subarray(bytesRead)); + const output = new Uint8Array(byteLength); + for (let index = 0; index < byteLength; index++) { + output[index] = model.processDecodedByte((probability) => coder.readBit(probability)); + } + return output; +} + +// --------------------------------------------------------------------------- +// Priors +// --------------------------------------------------------------------------- + +/** + * Wire ids for the priming corpus: markdown, code, json, the shared corpus, and none. The id is one + * ASCII char at the head of the arx4 payload, so a fragment always says which prior decodes it. + */ +export const arx4PriorIds = ["m", "c", "j", "s", "n"] as const; + +export type Arx4PriorId = (typeof arx4PriorIds)[number]; + +const arx4PriorIdSet = new Set(arx4PriorIds); + +/** True when `value` is a prior id this build can decode. */ +export function isArx4PriorId(value: string): value is Arx4PriorId { + return arx4PriorIdSet.has(value); +} + +const PRIOR_ID_BY_ARTIFACT_KIND: Record = { + markdown: "m", + code: "c", + json: "j", + csv: "j", + diff: "c", +}; + +/** The prior id an encoder picks for an envelope, from its first artifact's kind. */ +export function arx4PriorIdForEnvelope(envelope: PayloadEnvelope): Arx4PriorId { + const kind = envelope.artifacts[0]?.kind; + return kind === undefined ? "s" : PRIOR_ID_BY_ARTIFACT_KIND[kind]; +} + +/** Curated corpora the priors asset carries, keyed the way the asset keys them. */ +const ARX4_PRIOR_KINDS = ["markdown", "code", "json"] as const; + +type Arx4PriorKind = (typeof ARX4_PRIOR_KINDS)[number]; + +/** + * The `/arx4-priors.json` asset: the kind-specific tail of each curated prior. The 2203-char common + * prefix is left out because {@link getArxDictionaryPriorText} already rebuilds it from the pinned + * dictionaries, which the `e` tag pins anyway. + */ +export type Arx4Priors = { + version: number; + kinds: Record; +}; + +/** Prior ids that need a curated corpus; `s` primes on the dictionaries alone and `n` on nothing. */ +const PRIOR_KIND_BY_ID: Record = { + m: "markdown", + c: "code", + j: "json", + s: null, + n: null, +}; + +/** + * The prior ids that need the curated asset, derived from the kind map so no caller re-spells them. + * Exported because callers route on a fragment's leading char to decide whether the asset is needed + * at all: `s` and `n` fragments must not trigger the fetch. + */ +export const CURATED_PRIOR_IDS: readonly Arx4PriorId[] = arx4PriorIds.filter( + (priorId) => PRIOR_KIND_BY_ID[priorId] !== null, +); + +/** + * The one priors version this build codes curated fragments against. The compact `e` tag carries no + * priors version, so a fragment names only "the curated prior for kind X" and both sides have to + * already agree on which corpus that is. An asset at any other version is unusable, a stale or + * rolled-back copy exactly as much as a forward-deployed one, because priming with corpus bytes the + * fragment was never coded against yields plausible garbage rather than an error. Bumping this is a + * wire change that also needs a new compact tag, like the dictionary pins in fragment-arx.ts. + */ +export const EXPECTED_ARX4_PRIORS_VERSION = 1; + +/** Byte length of every curated prior, the size the ARX4 research benchmarks measured. */ +export const ARX4_PRIOR_BYTES = 16 * 1024; + +/** + * sha256 of each curated prior as {@link reassembleCuratedPrior} rebuilds it: the pinned dictionary + * slot text, a newline, then the asset's kind block. The version field alone cannot say whether an + * asset holds the corpora this build's fragments were coded against, so the install point checks + * identity, not just the label. tests/arx4-priors.test.ts asserts the shipped asset against these with + * node:crypto, and scripts/build-arx4-priors.mjs prints them when it regenerates the asset. + */ +export const PINNED_ARX4_PRIOR_SHA256 = { + markdown: "90da74cfa7a7394099aefd7d8f3ba9ed2acc40237b23d58048f4b8b4dd596c9c", + code: "3596c70d73b7d3f95e5f978a0c3bcb4ae1d4aa8711d563f4a22f39d0123aa6af", + json: "37e1cfa8f8885afda7e560d63616b4e84e891a1c3a63d2ac3a139ebe6558fb18", +} as const satisfies Record; + +/** + * Thrown when a fragment names a curated prior this build cannot rebuild, because the asset is not + * loaded or is not at {@link EXPECTED_ARX4_PRIORS_VERSION}. Decoding it against the shared prior, or + * against a skewed corpus, would return plausible garbage instead, so the failure surfaces and the + * caller can retry once the asset endpoint serves the expected version. + */ +export class Arx4PriorsUnavailableError extends Error { + constructor(priorId: Arx4PriorId) { + super(`The arx4 priors asset is unavailable, so the "${priorId}" prior cannot be rebuilt.`); + this.name = "Arx4PriorsUnavailableError"; + } +} + +// Mirrors the dictionary slots in arx-codec.ts, minus the built-in fallback: there is no compiled-in +// curated corpus, so version 0 means "no asset loaded" and the encoder degrades to the `s` prior +// rather than blocking on a fetch it cannot complete. +const priorsSlot: { priors: Arx4Priors | null; version: number } = { priors: null, version: 0 }; + +/** + * The full curated prior for one kind block: the pinned dictionary slot text, a newline, the block. + * Both the install-time identity check and the coder go through here, so what was validated is exactly + * what primes the mixer. + */ +function reassembleCuratedPrior(kindBlock: string): string { + return `${getArxDictionaryPriorText()}\n${kindBlock}`; +} + +function isArx4Priors(value: unknown): value is Arx4Priors { + if (typeof value !== "object" || value === null) return false; + const asset = value as Record; + // A non-integer or negative version would install while every caller read it as the -1 failure + // sentinel, leaving the priors live but reported unloaded, so the shape check owns it here. + const version = asset.version; + if (typeof version !== "number" || !Number.isInteger(version) || version < 0) return false; + if (typeof asset.kinds !== "object" || asset.kinds === null) return false; + + const kinds = asset.kinds as Record; + // The size bound is part of the shape: a kind block can never exceed the prior it is a tail of, and + // rejecting an oversize one here keeps a corrupt or hostile asset out of the hashing below and out of + // the mixer's byte-by-byte priming walk. + return ARX4_PRIOR_KINDS.every((kind) => { + const block = kinds[kind]; + return typeof block === "string" && block !== "" && block.length <= ARX4_PRIOR_BYTES; + }); +} + +/** + * True when every kind block reassembles to the prior this build codes against, checked by byte length + * and pinned digest. A truncated, padded or swapped block is otherwise indistinguishable from the real + * asset at the version field, and installing it caches it as authoritative: later loads return early on + * the matching version, so the recovered endpoint is never refetched and curated links stay broken. + */ +function priorsMatchPinnedDigests(priors: Arx4Priors): boolean { + return ARX4_PRIOR_KINDS.every((kind) => { + const bytes = new TextEncoder().encode(reassembleCuratedPrior(priors.kinds[kind])); + return bytes.length === ARX4_PRIOR_BYTES && sha256Hex(bytes) === PINNED_ARX4_PRIOR_SHA256[kind]; + }); +} + +function getDefaultArx4PriorsUrls(): string[] { + const url = withBasePath("/arx4-priors.json"); + return [`${url}.br`, url]; +} + +/** + * This fetch sits on the link-creation path, and a connection that never responds would leave the + * awaited promise pending forever instead of degrading to the `s` prior. The dictionary fetches can + * afford no bound because they have a built-in fallback table; the priors slot has none. + */ +const PRIORS_FETCH_TIMEOUT_MS = 10_000; + +async function fetchArx4Priors(url: string): Promise { + try { + const response = await fetch(url, { signal: AbortSignal.timeout(PRIORS_FETCH_TIMEOUT_MS) }); + if (!response.ok) return null; + return (await response.json()) as unknown; + } catch { + return null; + } +} + +/** + * The one install point, so a half-shaped or unpinned asset can never reach the coder. -1 means + * unusable and leaves the slot alone, which is what makes such a load retryable. + */ +function installArx4Priors(value: unknown): number { + if (!isArx4Priors(value) || !priorsMatchPinnedDigests(value)) return -1; + + priorsSlot.priors = value; + priorsSlot.version = value.version; + return value.version; +} + +/** + * Loads the curated priors from a URL or parsed object, trying the pre-compressed asset first on a + * default load. Returns the asset version on success, or -1 on failure (the slot keeps whatever it + * already had), so a transient failure can be retried rather than cached. + * + * A fetched asset is installed only at {@link EXPECTED_ARX4_PRIORS_VERSION}: `.br` and `.json` are two + * files that a mid-deploy CDN can serve at different versions, and installing whatever the first URL + * answers with would both skip the URL that still had the right one and wedge every later retry, which + * hits the same skewed URL again. An off-version asset handed in as an object still installs, because + * that is the injection path tests and offline agents use to put the slot in a known state. + */ +export async function loadArx4Priors(source?: string | Arx4Priors): Promise { + try { + if (source && typeof source === "object") return installArx4Priors(source); + + const urls = typeof source === "string" ? [source] : getDefaultArx4PriorsUrls(); + for (const url of urls) { + const fetched = await fetchArx4Priors(url); + if (isArx4Priors(fetched) && fetched.version === EXPECTED_ARX4_PRIORS_VERSION) { + const installed = installArx4Priors(fetched); + // A digest-mismatched asset from one URL must not short-circuit the others: the + // remaining URL may still serve the intact file (mid-deploy or corrupted cache). + if (installed >= 0) return installed; + } + } + return -1; + } catch { + return -1; + } +} + +/** + * Loads the curated priors from a pre-parsed object (synchronous), for tests and offline agents that + * already hold the asset JSON. Returns the version, or -1 when the object is not a usable asset. + */ +export function loadArx4PriorsSync(priors: Arx4Priors): number { + return installArx4Priors(priors); +} + +/** + * Returns true when the curated priors asset has been loaded, whatever version it is. Deliberately + * NOT a usability check: an asset off {@link EXPECTED_ARX4_PRIORS_VERSION} loads and reports true + * while no curated coding may use it, so the coding paths ask `versionMatchedPriors` instead. + */ +export function isArx4PriorsLoaded(): boolean { + return priorsSlot.priors !== null; +} + +/** Returns the active curated priors version (0 = no asset loaded). */ +export function getActiveArx4PriorsVersion(): number { + return priorsSlot.version; +} + +/** + * The loaded priors, but only at {@link EXPECTED_ARX4_PRIORS_VERSION}: the single gate both coding + * paths ask, so a version-skewed asset can never reach the mixer from either side. + */ +function versionMatchedPriors(): Arx4Priors | null { + return priorsSlot.version === EXPECTED_ARX4_PRIORS_VERSION ? priorsSlot.priors : null; +} + +/** + * Priming bytes for a prior id, or null for "n" (cold model). + * + * Every prior starts from the pinned arx dictionary slot text in its RAW form, not the substituted + * form: the slots are themselves the substitution patterns, so substituting the prior collapses it to + * control bytes and measured ~5% worse than priming on the raw text. + * + * `s` is that text alone. `m`, `c` and `j` append the matching curated corpus from the priors asset, + * which is what the 16 KiB per-kind priors in docs/arx4-cm-bench.md measured, and throw when that + * corpus is missing or version-skewed rather than quietly coding against a different prior than the + * id names. + */ +function priorBytesFor(priorId: Arx4PriorId): Uint8Array | null { + if (priorId === "n") return null; + + const kind = PRIOR_KIND_BY_ID[priorId]; + if (kind === null) return new TextEncoder().encode(getArxDictionaryPriorText()); + + const priors = versionMatchedPriors(); + if (priors === null) throw new Arx4PriorsUnavailableError(priorId); + return new TextEncoder().encode(reassembleCuratedPrior(priors.kinds[kind])); +} + +/** + * The prior id an encode can actually honor. A curated id degrades to `s` when the asset is missing or + * version-skewed, so a failed or stale asset fetch costs compression instead of blocking link + * creation; the emitted id always names the prior the payload was really coded against. + */ +function encodablePriorId(priorId: Arx4PriorId): Arx4PriorId { + if (PRIOR_KIND_BY_ID[priorId] === null) return priorId; + return versionMatchedPriors() === null ? "s" : priorId; +} + +// --------------------------------------------------------------------------- +// Public API +// --------------------------------------------------------------------------- + +/** + * Compresses a payload envelope with the arx4 pipeline. Every returned wire string already carries + * the prior id char, so a candidate is `` + the string returned here. + * + * `priorId` defaults to {@link arx4PriorIdForEnvelope}; pass it explicitly only to exercise a prior + * the kind map does not select. A curated id downgrades to `s` unless the priors asset is loaded at + * {@link EXPECTED_ARX4_PRIORS_VERSION}, so the returned payloads always carry the id they were really + * coded against. + */ +export function arx4CompressEnvelope(envelope: PayloadEnvelope, priorId?: Arx4PriorId): ArxWirePayloads { + const selectedPriorId = encodablePriorId(priorId ?? arx4PriorIdForEnvelope(envelope)); + const substituted = substituteArxTupleText(envelope); + const coded = encodeCm(new TextEncoder().encode(substituted), priorBytesFor(selectedPriorId)); + const payloads = encodeArxWirePayloads(coded); + + return { + base76: `${selectedPriorId}${payloads.base76}`, + base1k: `${selectedPriorId}${payloads.base1k}`, + baseBMP: `${selectedPriorId}${payloads.baseBMP}`, + base64url: `${selectedPriorId}${payloads.base64url}`, + }; +} + +/** + * Decompresses an arx4 payload (prior id char + wire payload) and rebuilds the envelope. + * Throws on an unrecognized prior id rather than guessing, because decoding with the wrong prior + * produces plausible-looking garbage instead of an error. + */ +export function arx4DecompressEnvelope(encoded: string): PayloadEnvelope { + const priorId = encoded.slice(0, 1); + if (!isArx4PriorId(priorId)) { + throw new Error(`Unsupported arx4 prior id "${priorId}".`); + } + + const primeBytes = priorBytesFor(priorId); + const substituted = decodeArxWirePayload(encoded.slice(1), (bytes) => ( + new TextDecoder().decode(decodeCm(bytes, primeBytes)) + )); + return envelopeFromSubstitutedArxTupleText(substituted, "arx4"); +} diff --git a/src/lib/payload/fragment-arx.ts b/src/lib/payload/fragment-arx.ts index 747b735..a2e8b1b 100644 --- a/src/lib/payload/fragment-arx.ts +++ b/src/lib/payload/fragment-arx.ts @@ -14,6 +14,14 @@ import { loadArx2OverlayDictionary, type ArxWirePayloads, } from "@/lib/payload/arx-codec"; +import { + arx4CompressEnvelope, + arx4DecompressEnvelope, + CURATED_PRIOR_IDS, + EXPECTED_ARX4_PRIORS_VERSION, + getActiveArx4PriorsVersion, + loadArx4Priors, +} from "@/lib/payload/arx4-codec"; import { packEnvelope } from "@/lib/payload/wire-format"; import { compactTagForCodec, @@ -22,11 +30,18 @@ import { type PayloadEnvelope, } from "@/lib/payload/schema"; +/** + * One wire encoding, measured under both selection budgets so a single pool can serve both surfaces + * without re-running the codec (arx4's context mixer costs ~770 ms for a 60 KB artifact). + */ export type CandidateFragment = { value: string; codec: PayloadCodec; packed: boolean; + /** Default budget: percent-escaped transport length, except the arx3/arx4 baseBMP wire. */ transportLength: number; + /** Budget for surfaces that URL-serialize the fragment: every wire percent-escaped. */ + urlSerializedLength: number; }; type TransportLengthCalculator = (value: string) => number; @@ -38,8 +53,10 @@ const WIRE_ORDER = ["base76", "base1k", "baseBMP", "base64url"] as const satisfi * Turn an arx codec's four wire payloads into tagged candidates. Shared by all three arx builders, * which previously each re-spelled the tag prefix + transport-length + four-candidate list. * - * `bmpUsesVisibleLength` budgets the dense baseBMP wire by visible URL length instead of percent- - * escaped transport length — see the POLICY note on buildArx3Candidates for why arx3 does this. + * `bmpUsesVisibleLength` gives the dense baseBMP wire its DEFAULT budget in visible URL characters + * instead of percent-escaped transport length — see the POLICY note on buildArx3Candidates for why + * arx3 and arx4 do this. `urlSerializedLength` is unaffected, so the caller can still budget that + * same candidate by transport length for a URL-serializing surface. */ function wirePayloadsToCandidates( codec: ArxCodec, @@ -51,30 +68,67 @@ function wirePayloadsToCandidates( const tag = compactTagForCodec(codec); return WIRE_ORDER.map((wire) => { const value = `${tag}${payloads[wire]}`; + const urlSerializedLength = computeTransportLength(value); return { value, codec, packed, transportLength: - bmpUsesVisibleLength && wire === "baseBMP" ? value.length : computeTransportLength(value), + bmpUsesVisibleLength && wire === "baseBMP" ? value.length : urlSerializedLength, + urlSerializedLength, }; }); } let arxDictionaryLoadPromise: Promise | null = null; let arx2OverlayDictionaryLoadPromise: Promise | null = null; +let arx4PriorsLoadPromise: Promise | null = null; -// Compact ARX fragments (tags `a`/`b`/`c`) do NOT carry a dictionary version — the tag implies the -// CURRENT dictionary, which keeps links short. The safety cost is that a build must not decode with -// a dictionary NEWER than it was built for (a CDN/asset split serving a future dictionary, or a +// Compact ARX fragments (tags `a`/`b`/`c`/`e`) do NOT carry a dictionary version — the tag implies +// the CURRENT dictionary, which keeps links short. The safety cost is that a build must not decode +// with a dictionary NEWER than it was built for (a CDN/asset split serving a future dictionary, or a // version bump), because it would lack the new slots and could produce a structurally-valid-but- // wrong envelope. We pin the newest supported version and reject anything newer so decode hard-fails // instead of mis-decoding. The built-in fallback dictionary (version 0) and the current external // dictionary (version 1) are both <= this and remain usable. Bumping a dictionary version is // therefore a wire change that also requires new compact tags and updating -// tests/arx-dictionary-pin.test.ts. +// tests/arx-dictionary-pin.test.ts. arx4 depends on the same pin twice over, since its context-mixer +// prior is derived from the dictionary slot text as well as its substitution stage. const EXPECTED_ARX_DICTIONARY_VERSION = 1; const EXPECTED_ARX2_OVERLAY_VERSION = 1; +// The arx4 priors asset is pinned the same way and for the same reason, except that it tolerates no +// older version either: it has no built-in fallback table, so every version but the expected one is a +// corpus this build's fragments were never coded against. That pin lives on the codec that codes with +// it (EXPECTED_ARX4_PRIORS_VERSION in arx4-codec.ts); this module only drives the loader toward it. +// +// arx4 holds its DICTIONARIES to that same exact standard, which is where it parts ways with +// arx/arx2/arx3. They tolerate the built-in fallback (version 0) because substitution alone degrades +// predictably; arx4 also primes its context mixer on the dictionary slot text, so a fragment coded +// against any other dictionary is one that healthy viewers cannot decode at all. Both sides therefore +// hold out for the pinned pair: encode leaves the candidate pool, decode refuses. + +/** + * Thrown when an arx4 fragment reaches the decoder while the active dictionaries are not the exact + * pinned pair, which a failed dictionary fetch (built-in fallback) or an asset-version skew both cause. + * Fail closed and retryable: the same fragment decodes once the pinned dictionaries are active, whereas + * coding it against the dictionary text at hand would return plausible garbage. + */ +export class Arx4DictionarySkewError extends Error { + constructor(dictVersion: number, overlayVersion: number) { + super( + `The active arx dictionaries (base ${dictVersion}, overlay ${overlayVersion}) are not the pair arx4 is pinned to (base ${EXPECTED_ARX_DICTIONARY_VERSION}, overlay ${EXPECTED_ARX2_OVERLAY_VERSION}), so arx4 fragments cannot be coded.`, + ); + this.name = "Arx4DictionarySkewError"; + } +} + +/** True when both active dictionaries are exactly the versions arx4 codes against. */ +function arx4DictionariesMatchPins(): boolean { + return ( + getActiveDictVersion() === EXPECTED_ARX_DICTIONARY_VERSION && + getActiveArx2OverlayVersion() === EXPECTED_ARX2_OVERLAY_VERSION + ); +} function assertArxDictionaryNotNewerThanExpected(): void { const version = getActiveDictVersion(); @@ -139,6 +193,31 @@ async function ensureArx2DictionariesLoaded(): Promise { assertArx2OverlayNotNewerThanExpected(); } +/** + * Same retry-on-failure contract as the dictionaries, and never fatal: the codec degrades encoding to + * the `s` prior when the expected-version asset is not there, so only a fragment that names a curated + * prior fails, which is why this resolves instead of throwing. + * + * A version-skewed asset counts as not loaded, for both the early return and the caching: a CDN + * mid-deploy can serve one and then the other, so neither side may pin the wrong corpus for the life + * of the page when a refetch could still install the right one. + */ +async function loadArx4PriorsOnce(): Promise { + if (getActiveArx4PriorsVersion() === EXPECTED_ARX4_PRIORS_VERSION) return; + + arx4PriorsLoadPromise ??= loadArx4Priors() + .then((version) => { + if (version !== EXPECTED_ARX4_PRIORS_VERSION) { + arx4PriorsLoadPromise = null; + } + }) + .catch((error) => { + arx4PriorsLoadPromise = null; + throw error; + }); + await arx4PriorsLoadPromise; +} + function decodeArxEncodedPayload(encoded: string): string { try { return decodeURIComponent(encoded); @@ -167,13 +246,16 @@ async function decodeArxAttempt( codec: ArxCodec, encodedPayload: string, ): Promise { - if (codec === "arx") { - return await arxDecompress(encodedPayload); + switch (codec) { + case "arx": + return await arxDecompress(encodedPayload); + case "arx2": + return await arx2DecompressEnvelope(encodedPayload); + case "arx3": + return await arx3DecompressEnvelope(encodedPayload); + case "arx4": + return arx4DecompressEnvelope(encodedPayload); } - - return codec === "arx2" - ? await arx2DecompressEnvelope(encodedPayload) - : await arx3DecompressEnvelope(encodedPayload); } function normalizeArxDecodeError(error: unknown): Error { @@ -236,6 +318,12 @@ export async function buildArx2Candidates( * CHANGING THIS REQUIRES A MAINTAINER DECISION: switching the arx3 baseBMP budget back to transport * length would make arx2 and arx3 measure the same payload identically and would change which wire * wins auto-selection. Do not flip the metric to "fix" the divergence without owning that trade-off. + * + * PER-SURFACE EXCEPTION: every candidate also carries `urlSerializedLength`, which measures the same + * baseBMP wire by transport length, for surfaces that URL-serialize the fragment (markdown links + * percent-encode baseBMP to ~9x). Selecting on that field is an additional surface-specific + * selection, not a reversal of the default policy above: the primary copy-paste URL keeps the + * visible-length budget. */ export async function buildArx3Candidates( envelope: PayloadEnvelope, @@ -245,10 +333,38 @@ export async function buildArx3Candidates( const payloadEnvelope = { ...envelope, codec: "arx3" as PayloadCodec }; const payloads = await arx3CompressEnvelope(payloadEnvelope); - // The `true` budgets the dense baseBMP wire by visible URL length — see the POLICY note above. + // Visible-length budgeting for the dense baseBMP wire — see the POLICY note above. return wirePayloadsToCandidates("arx3", false, payloads, computeTransportLength, true); } +/** + * Builds deferred `arx4` codec fragment candidates. + * ARX4 reuses the ARX3 tuple/overlay stages and its baseBMP budgeting policy; it swaps Brotli for the + * context mixer in arx4-codec.ts and puts a prior id char in front of the wire payload, so a + * candidate reads ``. + */ +export async function buildArx4Candidates( + envelope: PayloadEnvelope, + computeTransportLength: TransportLengthCalculator, +): Promise { + await ensureArx2DictionariesLoaded(); + // Off the pinned dictionaries, arx4 contributes nothing rather than minting a link no healthy viewer + // can decode. Dropping out of the pool (instead of throwing) keeps the other codecs' candidates, the + // same reason the priors load below resolves; an explicitly requested arx4 then has no candidate, + // which is the honest fail-closed answer. + if (!arx4DictionariesMatchPins()) return []; + + // Resolving rather than throwing on a failed or skewed priors load matters here: these candidates + // share one pool with arx3/arx2/arx/deflate (`buildCandidatesAsync` builds them through the same + // loop), so a throw would take link creation down over a codec the encoder is free to degrade. + // arx4CompressEnvelope owns that degrade, and emits the `s` id it really coded with. + await loadArx4PriorsOnce(); + + const payloadEnvelope = { ...envelope, codec: "arx4" as PayloadCodec }; + const payloads = arx4CompressEnvelope(payloadEnvelope); + return wirePayloadsToCandidates("arx4", false, payloads, computeTransportLength, true); +} + /** * Decodes an ARX fragment remainder with the same versioned-payload fallback behavior as the main decoder. */ @@ -256,14 +372,32 @@ export async function decodeArxFragmentPayload( codec: ArxCodec, remainder: string, ): Promise { - if (codec === "arx3" || codec === "arx2") { - await ensureArx2DictionariesLoaded(); - } else { + if (codec === "arx") { await ensureArxDictionaryLoaded(); + } else { + await ensureArx2DictionariesLoaded(); + } + + if (codec === "arx4" && !arx4DictionariesMatchPins()) { + throw new Arx4DictionarySkewError(getActiveDictVersion(), getActiveArx2OverlayVersion()); } let lastError: Error | null = null; const { parsedDictVersion, versionedPayload } = splitArxFragmentRemainder(remainder); + const decodedPayload = decodeArxEncodedPayload(versionedPayload); + + // Only fragments naming a curated prior (the first payload char) need the priors asset; s and n + // fragments decode without it, so they must not trigger the fetch. A curated fragment that the + // fetch cannot serve at the expected version fails in the codec, which is the retryable outcome: + // decoding it against a skewed corpus would return plausible garbage instead. + // + // The char is read AFTER percent-decoding, so this routes on what the decoder will really see: a + // re-encoding proxy or a handcrafted fragment can deliver `%6d` where the app writes `m`, and routing + // on the raw char would leave that fragment asking for an asset nothing ever fetches. + const priorIdChar = decodedPayload.charAt(0); + if (codec === "arx4" && CURATED_PRIOR_IDS.some((priorId) => priorId === priorIdChar)) { + await loadArx4PriorsOnce(); + } // For a correctly versioned fragment this first attempt (decoding the full remainder, including // the "." prefix) is expected to fail at the decompressor — it exists only for @@ -278,7 +412,7 @@ export async function decodeArxFragmentPayload( } try { - return await decodeArxAttempt(codec, decodeArxEncodedPayload(versionedPayload)); + return await decodeArxAttempt(codec, decodedPayload); } catch (error) { lastError = normalizeArxDecodeError(error); } diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts index a4bc055..d7c4f6e 100644 --- a/src/lib/payload/fragment.ts +++ b/src/lib/payload/fragment.ts @@ -23,14 +23,24 @@ type EncodeOptions = { preferPacked?: boolean; targetMaxFragmentLength?: number; codecPriority?: PayloadCodec[]; + /** + * Budget every candidate (including arx3/arx4 baseBMP) by percent-escaped transport length. + * For surfaces that URL-serialize the fragment, e.g. markdown link destinations. + */ + budgetByTransport?: boolean; }; +/** Which of a candidate's two measurements {@link selectCandidate} compares. */ +type BudgetPolicy = "default" | "urlSerialized"; + const BINARY_STRING_CHUNK_SIZE = 0x8000; const DEFAULT_SYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["deflate", "lz", "plain"]; -const DEFAULT_ASYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["arx3", "arx2", "arx", "deflate", "lz", "plain"]; +const DEFAULT_ASYNC_CODEC_PRIORITY: readonly PayloadCodec[] = ["arx4", "arx3", "arx2", "arx", "deflate", "lz", "plain"]; const PACKED_WIRE_MODES: readonly boolean[] = [true, false]; const UNPACKED_ONLY_WIRE_MODES: readonly boolean[] = [false]; const supportedCodecSet = new Set(codecs); +/** arx4 asset failures (see arx4-codec.ts and fragment-arx.ts), which decode reports as retryable. */ +const ARX4_ASSET_ERROR_NAMES = new Set(["Arx4PriorsUnavailableError", "Arx4DictionarySkewError"]); function isChatSafeAsciiFragmentCodePoint(cp: number): boolean { return ( @@ -73,9 +83,17 @@ function computeTransportLength(value: string): number { return len; } +/** + * Public wrapper over {@link computeTransportLength} so link builders can compare + * how fragment candidates survive URL serialization and chat-surface escaping. + */ +export function getFragmentTransportLength(fragmentBody: string): number { + return computeTransportLength(fragmentBody); +} + /** * Returns the decoded visible length of a fragment body or hash. - * Browsers may expose Unicode fragments as percent-escaped text, while ARX3 budgets by the + * Browsers may expose Unicode fragments as percent-escaped text, while arx3/arx4 budget by the * visible characters a user copies from the URL bar. */ export function getVisibleFragmentLength(fragment: string): number { @@ -127,6 +145,7 @@ function encodePayload(json: string, codec: PayloadCodec): string { case "arx": case "arx2": case "arx3": + case "arx4": throw new Error("arx codec requires async encoding — use encodeEnvelopeAsync instead."); } } @@ -162,6 +181,7 @@ function decodePayload(encoded: string, codec: PayloadCodec): string | null { case "arx": case "arx2": case "arx3": + case "arx4": throw new Error("arx codec requires async decoding — use decodeFragmentAsync instead."); } } @@ -170,8 +190,9 @@ function buildFragment(envelope: PayloadEnvelope, codec: PayloadCodec, packed: b const payloadEnvelope = { ...envelope, codec }; const json = JSON.stringify(packed ? packEnvelope(payloadEnvelope) : payloadEnvelope); const value = `${compactTagForCodec(codec)}${encodePayload(json, codec)}`; - // Non-ARX codecs produce ASCII-only output, so transport length equals string length. - return { value, codec, packed, transportLength: value.length }; + // Non-ARX codecs produce ASCII-only output, so transport length equals string length and neither + // surface's budget can separate them. + return { value, codec, packed, transportLength: value.length, urlSerializedLength: value.length }; } function dedupeCodecs(requested: readonly PayloadCodec[]): PayloadCodec[] { @@ -248,12 +269,22 @@ async function buildArx3Candidates(envelope: PayloadEnvelope): Promise { + const { buildArx4Candidates: buildDeferredArx4Candidates } = await import("@/lib/payload/fragment-arx"); + return buildDeferredArx4Candidates(envelope, computeTransportLength); +} + async function buildCandidatesAsync(envelope: PayloadEnvelope, options: EncodeOptions): Promise { const codecsToTry = getAsyncCandidateCodecs(options); const wireModes = options.preferPacked === false ? UNPACKED_ONLY_WIRE_MODES : PACKED_WIRE_MODES; const candidates: CandidateFragment[] = []; for (const codec of codecsToTry) { + if (codec === "arx4") { + candidates.push(...await buildArx4Candidates(envelope)); + continue; + } + if (codec === "arx3") { candidates.push(...await buildArx3Candidates(envelope)); continue; @@ -276,7 +307,15 @@ async function buildCandidatesAsync(envelope: PayloadEnvelope, options: EncodeOp return candidates; } -function selectCandidate(candidates: CandidateFragment[], budget?: number): CandidateFragment { +function budgetLengthFor(candidate: CandidateFragment, policy: BudgetPolicy): number { + return policy === "urlSerialized" ? candidate.urlSerializedLength : candidate.transportLength; +} + +function selectCandidate( + candidates: CandidateFragment[], + budget?: number, + policy: BudgetPolicy = "default", +): CandidateFragment { if (candidates.length === 0) { throw new Error("No payload codec candidates are available."); } @@ -285,14 +324,16 @@ function selectCandidate(candidates: CandidateFragment[], budget?: number): Cand let shortestInBudget: CandidateFragment | null = null; for (const candidate of candidates) { - if (candidate.transportLength < shortest.transportLength) { + const length = budgetLengthFor(candidate, policy); + + if (length < budgetLengthFor(shortest, policy)) { shortest = candidate; } if ( typeof budget === "number" && - candidate.transportLength <= budget && - (!shortestInBudget || candidate.transportLength < shortestInBudget.transportLength) + length <= budget && + (!shortestInBudget || length < budgetLengthFor(shortestInBudget, policy)) ) { shortestInBudget = candidate; } @@ -334,8 +375,35 @@ export function encodeEnvelope(envelope: PayloadEnvelope, options: EncodeOptions */ export async function encodeEnvelopeAsync(envelope: PayloadEnvelope, options: EncodeOptions = {}): Promise { const candidates = await buildCandidatesAsync(envelope, options); - const selected = selectCandidate(candidates, options.targetMaxFragmentLength); - return selected.value; + const policy: BudgetPolicy = options.budgetByTransport === true ? "urlSerialized" : "default"; + return selectCandidate(candidates, options.targetMaxFragmentLength, policy).value; +} + +/** The two fragments a shareable link needs, both selected from one candidate pool. */ +export type EncodedEnvelopeSurfaces = { + /** Default-policy winner, for the copy-paste URL. */ + fragmentBody: string; + /** URL-serialized-budget winner, for destinations a URL serializer percent-encodes. */ + transportFragmentBody: string; +}; + +/** + * Encodes an envelope once and returns both surface winners. + * + * A link needs two selections over the same candidates: the copy-paste URL keeps the arx3/arx4 + * visible-length budget, while a markdown destination is measured percent-escaped. Running + * {@link encodeEnvelopeAsync} twice would recompress the payload, which for arx4 means running the + * context mixer a second time (~770 ms per 60 KB artifact) for two selections over identical bytes. + */ +export async function encodeEnvelopeSurfacesAsync( + envelope: PayloadEnvelope, + options: Omit = {}, +): Promise { + const candidates = await buildCandidatesAsync(envelope, options); + return { + fragmentBody: selectCandidate(candidates, options.targetMaxFragmentLength, "default").value, + transportFragmentBody: selectCandidate(candidates, options.targetMaxFragmentLength, "urlSerialized").value, + }; } type ParsedFragmentHeader = @@ -503,7 +571,8 @@ function resolveEnvelope(parsed: unknown, rawLength: number): ParsedPayload { * for example when decoding server-injected payloads that are not constrained by URL length. * * Returns structured `ParsedPayload` error responses for malformed fragments or invalid - * envelopes, rather than throwing decode errors. + * envelopes, rather than throwing decode errors. A well-formed fragment whose codec asset is missing or + * version-skewed comes back as `asset-unavailable`, the one code a retry can clear. */ export async function decodeFragmentAsync(hash: string, options?: DecodeOptions): Promise { const header = parseFragmentHeader(hash, options); @@ -521,11 +590,12 @@ export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) const { decodeArxFragmentPayload } = await import("@/lib/payload/fragment-arx"); const decodedFromAttempt = await decodeArxFragmentPayload(codec, remainder); - if (codec === "arx2" || codec === "arx3") { + if (codec === "arx") { + decodedJson = decodedFromAttempt as string; + } else { + // The tuple codecs rebuild the envelope themselves; only `arx` returns envelope JSON. parsed = decodedFromAttempt; decodedJson = null; - } else { - decodedJson = decodedFromAttempt as string; } } else { decodedJson = decodePayload(remainder, codec); @@ -546,6 +616,13 @@ export async function decodeFragmentAsync(hash: string, options?: DecodeOptions) if (error instanceof Error && error.name === "ArxDecodedPayloadTooLargeError") { return { ok: false, code: "decoded-too-large", message: error.message }; } + // A codec asset that is missing or version-skewed is a transport failure, not a malformed fragment: + // say so, because the same link decodes once the asset loads and the dictionary hint below would + // misdirect. Matched by name rather than instanceof so this module stays out of the lazily imported + // arx chunks on a non-arx page load. + if (error instanceof Error && ARX4_ASSET_ERROR_NAMES.has(error.name)) { + return { ok: false, code: "asset-unavailable", message: `${error.message} Reload to try again.` }; + } const arxHint = isArxCodec(codec) ? " It may have been encoded with a different ARX dictionary version." : ""; diff --git a/src/lib/payload/link-creator.ts b/src/lib/payload/link-creator.ts index 5a414be..167953f 100644 --- a/src/lib/payload/link-creator.ts +++ b/src/lib/payload/link-creator.ts @@ -1,5 +1,10 @@ import { normalizeEnvelope } from "@/lib/payload/envelope"; -import { encodeEnvelope, encodeEnvelopeAsync, getVisibleFragmentLength } from "@/lib/payload/fragment"; +import { + encodeEnvelope, + encodeEnvelopeSurfacesAsync, + getFragmentTransportLength, + getVisibleFragmentLength, +} from "@/lib/payload/fragment"; import { buildMarkdownLinkShareInfo } from "@/lib/markdown-link"; import { codecForCompactTag, @@ -30,6 +35,12 @@ export type GeneratedArtifactLink = { hash: string; url: string; fragmentLength: number; + /** + * URL used inside `markdownLink`. Markdown links pass through URL serializers that + * percent-encode non-ASCII fragments (3 UTF-8 bytes become 9 chars), so this may use an + * unpacked ASCII wire encoding of the same payload when that survives transport smaller. + */ + markdownUrl: string; markdownLink: string; markdownLinkLength: number; discordMarkdownLinkWarning: string | null; @@ -160,12 +171,15 @@ export function createGeneratedArtifactLink(draft: LinkCreatorDraft, baseUrl?: s throw new Error(normalized.message); } - return assembleGeneratedLink(normalized.envelope, encodeEnvelope(normalized.envelope), baseUrl); + const fragmentBody = encodeEnvelope(normalized.envelope); + // Sync codecs already emit ASCII wire shapes, so the same fragment serves both surfaces. + return assembleGeneratedLink(normalized.envelope, fragmentBody, fragmentBody, baseUrl); } /** * Async variant of {@link createGeneratedArtifactLink} that can leverage the ARX family of async - * codecs via {@link encodeEnvelopeAsync}. + * codecs via {@link encodeEnvelopeSurfacesAsync}, which encodes once and returns both the + * copy-paste and markdown-destination winners. * * Error and return semantics match the sync variant: throws on invalid draft/normalized payload * or over-budget fragments, and returns `{ hash, url, codec, fragmentLength, envelope, artifact }`. @@ -178,9 +192,11 @@ export async function createGeneratedArtifactLinkAsync(draft: LinkCreatorDraft, } const encodeOptions = draft.codec && draft.codec !== "auto" ? { codec: draft.codec } : {}; + const surfaces = await encodeEnvelopeSurfacesAsync(normalized.envelope, encodeOptions); return assembleGeneratedLink( normalized.envelope, - await encodeEnvelopeAsync(normalized.envelope, encodeOptions), + surfaces.fragmentBody, + surfaces.transportFragmentBody, baseUrl, ); } @@ -190,9 +206,40 @@ export async function createGeneratedArtifactLinkAsync(draft: LinkCreatorDraft, * creators differ only in how `fragmentBody` is produced; everything downstream (budget check, URL, * share info, result shape) lives here once. */ +function toFragmentUrl(fragmentBody: string, baseUrl?: string): string { + if (!baseUrl) { + return `#${fragmentBody}`; + } + + // Concatenate instead of assigning nextUrl.hash: the URL serializer percent-encodes + // non-ASCII fragments, which would hand Copy link a 3x longer string than the visible + // form the fragment budget counts (and that Discord receives on paste). + const nextUrl = new URL(baseUrl); + nextUrl.hash = ""; + return `${nextUrl.toString()}#${fragmentBody}`; +} + +/** + * Picks the fragment used inside markdown links. Markdown destinations get URL-serialized, + * which percent-encodes packed (non-ASCII) fragments to triple size, so the unpacked + * candidate usually survives transport smaller even though it is longer raw. Falls back to + * the primary fragment only when it wins on transport length. + * + * Deliberately NOT gated on the visible fragment budget: an over-budget ASCII candidate is exactly + * where handing back the Unicode fragment hurt most, because URL-serializing it inflates the + * destination roughly ninefold. An over-long markdown link is already reported to the caller through + * `discordMarkdownLinkWarning`, so the surface keeps its smaller-transport contract instead. + */ +function selectMarkdownFragment(fragmentBody: string, transportFragmentBody: string): string { + return getFragmentTransportLength(transportFragmentBody) < getFragmentTransportLength(fragmentBody) + ? transportFragmentBody + : fragmentBody; +} + function assembleGeneratedLink( envelope: PayloadEnvelope, fragmentBody: string, + transportFragmentBody: string, baseUrl?: string, ): GeneratedArtifactLink { const hash = `#${fragmentBody}`; @@ -204,15 +251,9 @@ function assembleGeneratedLink( ); } - let url = hash; - - if (baseUrl) { - const nextUrl = new URL(baseUrl); - nextUrl.hash = fragmentBody; - url = nextUrl.toString(); - } - - const shareInfo = buildGeneratedLinkShareInfo(envelope, url); + const url = toFragmentUrl(fragmentBody, baseUrl); + const markdownUrl = toFragmentUrl(selectMarkdownFragment(fragmentBody, transportFragmentBody), baseUrl); + const shareInfo = buildGeneratedLinkShareInfo(envelope, markdownUrl); return { envelope, @@ -221,6 +262,7 @@ function assembleGeneratedLink( hash, url, fragmentLength, + markdownUrl, markdownLink: shareInfo.markdownLink, markdownLinkLength: shareInfo.length, discordMarkdownLinkWarning: shareInfo.discordWarning, diff --git a/src/lib/payload/schema.ts b/src/lib/payload/schema.ts index cd794c4..b804489 100644 --- a/src/lib/payload/schema.ts +++ b/src/lib/payload/schema.ts @@ -3,17 +3,20 @@ export const MAX_DECODED_PAYLOAD_LENGTH = 200000; export const PAYLOAD_FRAGMENT_KEY = "agent-render"; export const artifactKinds = ["markdown", "code", "diff", "csv", "json"] as const; -export const codecs = ["plain", "lz", "deflate", "arx", "arx2", "arx3"] as const; +export const codecs = ["plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4"] as const; export type ArtifactKind = (typeof artifactKinds)[number]; export type PayloadCodec = (typeof codecs)[number]; -/** The dictionary + Brotli + binary-to-text codecs, as opposed to plain/lz/deflate. */ -export type ArxCodec = "arx" | "arx2" | "arx3"; +/** + * The dictionary + entropy-coder + binary-to-text codecs, as opposed to plain/lz/deflate. + * arx/arx2/arx3 entropy-code with Brotli; arx4 uses the context mixer in arx4-codec.ts. + */ +export type ArxCodec = "arx" | "arx2" | "arx3" | "arx4"; /** True when `codec` is one of the arx-family codecs. */ export function isArxCodec(codec: PayloadCodec): codec is ArxCodec { - return codec === "arx" || codec === "arx2" || codec === "arx3"; + return codec === "arx" || codec === "arx2" || codec === "arx3" || codec === "arx4"; } // Compact fragment header: a single URL-unreserved tag char replaces the legacy @@ -23,6 +26,7 @@ export function isArxCodec(codec: PayloadCodec): codec is ArxCodec { // `B.` prefix, baseBMP U+FFF0 marker, base76/base1k length prefix), so the alphabet is not in the // header. Tags come from the RFC-3986 unreserved set so they never percent-escape, and none can // begin the legacy `agent-render=` literal, which keeps the two header forms unambiguous on decode. +// The arx family runs a, b, c, then e because d is taken by deflate. export const compactCodecTags = { plain: "p", lz: "l", @@ -30,6 +34,7 @@ export const compactCodecTags = { arx: "a", arx2: "b", arx3: "c", + arx4: "e", } as const satisfies Record; const compactTagToCodec = new Map( @@ -126,7 +131,10 @@ export type ParsedPayload = | { ok: true; envelope: PayloadEnvelope; rawLength: number } | { ok: false; - code: "empty" | "missing-key" | "too-large" | "decoded-too-large" | "invalid-format" | "invalid-json" | "invalid-envelope"; + // `asset-unavailable` is the only retryable code: the fragment is well-formed but a codec asset + // this session needs (an arx4 prior corpus or dictionary) is missing or version-skewed, so the + // same link decodes once the asset endpoint serves the expected version. + code: "empty" | "missing-key" | "too-large" | "decoded-too-large" | "invalid-format" | "invalid-json" | "invalid-envelope" | "asset-unavailable"; message: string; }; diff --git a/src/lib/sha256.ts b/src/lib/sha256.ts new file mode 100644 index 0000000..9a8f88d --- /dev/null +++ b/src/lib/sha256.ts @@ -0,0 +1,87 @@ +/** + * Synchronous SHA-256 (FIPS 180-4) over bytes. + * + * WebCrypto's `crypto.subtle.digest` is async and `node:crypto` is not available in the browser, while + * the one place this is used (the arx4 priors identity check in arx4-codec.ts) installs an asset + * synchronously and must reject a mismatch before any coder can reach it. tests/sha256.test.ts pins + * this against `node:crypto` across the padding boundaries. + */ + +const ROUND_CONSTANTS = new Uint32Array([ + 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5, 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5, + 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3, 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174, + 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc, 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da, + 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7, 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967, + 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13, 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85, + 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3, 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070, + 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5, 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3, + 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208, 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2, +]); + +const INITIAL_HASH = new Uint32Array([ + 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, +]); + +const BLOCK_BYTES = 64; + +function rotateRight(value: number, bits: number): number { + return (value >>> bits) | (value << (32 - bits)); +} + +/** SHA-256 digest of `bytes`, as lowercase hex. */ +export function sha256Hex(bytes: Uint8Array): string { + const blockCount = Math.floor((bytes.length + 8) / BLOCK_BYTES) + 1; + const padded = new Uint8Array(blockCount * BLOCK_BYTES); + padded.set(bytes); + padded[bytes.length] = 0x80; + + const view = new DataView(padded.buffer); + // The trailing 64-bit big-endian bit length, split because a bit count over 2^32 does not fit a + // 32-bit write (bytes / 2^29 is the high word of bytes * 8). + view.setUint32(padded.length - 8, Math.floor(bytes.length / 0x20000000)); + view.setUint32(padded.length - 4, (bytes.length * 8) >>> 0); + + const hash = Uint32Array.from(INITIAL_HASH); + const schedule = new Uint32Array(64); + + for (let block = 0; block < blockCount; block += 1) { + for (let index = 0; index < 16; index += 1) { + schedule[index] = view.getUint32(block * BLOCK_BYTES + index * 4); + } + + for (let index = 16; index < 64; index += 1) { + const previous = schedule[index - 15]; + const recent = schedule[index - 2]; + const sigma0 = rotateRight(previous, 7) ^ rotateRight(previous, 18) ^ (previous >>> 3); + const sigma1 = rotateRight(recent, 17) ^ rotateRight(recent, 19) ^ (recent >>> 10); + schedule[index] = (schedule[index - 16] + sigma0 + schedule[index - 7] + sigma1) >>> 0; + } + + let [a, b, c, d, e, f, g, h] = hash; + + for (let index = 0; index < 64; index += 1) { + const sum1 = rotateRight(e, 6) ^ rotateRight(e, 11) ^ rotateRight(e, 25); + const choose = (e & f) ^ (~e & g); + const temp1 = (h + sum1 + choose + ROUND_CONSTANTS[index] + schedule[index]) >>> 0; + const sum0 = rotateRight(a, 2) ^ rotateRight(a, 13) ^ rotateRight(a, 22); + const majority = (a & b) ^ (a & c) ^ (b & c); + const temp2 = (sum0 + majority) >>> 0; + + h = g; + g = f; + f = e; + e = (d + temp1) >>> 0; + d = c; + c = b; + b = a; + a = (temp1 + temp2) >>> 0; + } + + const working = [a, b, c, d, e, f, g, h]; + for (let index = 0; index < 8; index += 1) { + hash[index] = (hash[index] + working[index]) >>> 0; + } + } + + return Array.from(hash, (word) => word.toString(16).padStart(8, "0")).join(""); +} diff --git a/tests/arx-codec.test.ts b/tests/arx-codec.test.ts index efc3096..d3c7a0f 100644 --- a/tests/arx-codec.test.ts +++ b/tests/arx-codec.test.ts @@ -374,7 +374,12 @@ describe("arx fragment round-trip", () => { }; const autoHash = await encodeEnvelopeAsync(bigEnvelope); - const arxTags = [compactTagForCodec("arx"), compactTagForCodec("arx2"), compactTagForCodec("arx3")]; + const arxTags = [ + compactTagForCodec("arx"), + compactTagForCodec("arx2"), + compactTagForCodec("arx3"), + compactTagForCodec("arx4"), + ]; expect(arxTags).toContain(autoHash.charAt(0)); }); diff --git a/tests/arx4-codec.test.ts b/tests/arx4-codec.test.ts new file mode 100644 index 0000000..81e85c4 --- /dev/null +++ b/tests/arx4-codec.test.ts @@ -0,0 +1,238 @@ +import { readFileSync } from "node:fs"; +import { beforeAll, describe, expect, it } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { arx4DeterminismVectors, arx4VectorEnvelope } from "./fixtures/arx4-vectors"; +import { + isBase1kEncoded, + isBase64urlEncoded, + isBaseBMPEncoded, + loadArx2OverlayDictionarySync, + loadArxDictionarySync, +} from "@/lib/payload/arx-codec"; +import { + arx4CompressEnvelope, + arx4DecompressEnvelope, + arx4PriorIdForEnvelope, + loadArx4PriorsSync, + type Arx4PriorId, +} from "@/lib/payload/arx4-codec"; +import { decodeFragment, decodeFragmentAsync, encodeEnvelopeAsync } from "@/lib/payload/fragment"; +import { createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; +import { compactTagForCodec, type ArtifactKind, type PayloadEnvelope } from "@/lib/payload/schema"; + +const ARX4_TAG = compactTagForCodec("arx4"); + +const drafts: Record = { + markdown: { + kind: "markdown", + title: "Release notes", + filename: "notes.md", + content: "# Release notes\n\n- Ship the arx4 codec\n- Keep fragments copyable\n\n| Surface | State |\n| --- | --- |\n| viewer | ready |\n| creator | ready |\n", + language: "", + diffView: "unified", + codec: "arx4", + }, + code: { + kind: "code", + title: "Wire selection", + filename: "wire.ts", + content: "export function selectWire(candidates: Candidate[]): Candidate {\n return candidates.reduce((best, candidate) => (candidate.length < best.length ? candidate : best));\n}\n", + language: "ts", + diffView: "unified", + codec: "arx4", + }, + json: { + kind: "json", + title: "Manifest", + filename: "manifest.json", + content: "{\n \"codec\": \"arx4\",\n \"wire\": [\"base76\", \"base1k\", \"baseBMP\", \"base64url\"],\n \"priors\": 5\n}\n", + language: "", + diffView: "unified", + codec: "arx4", + }, + csv: { + kind: "csv", + title: "Bench rows", + filename: "bench.csv", + content: "codec,bytes,visible\narx2,5410,7416\narx3,5410,2931\narx4,5147,2789\n", + language: "", + diffView: "unified", + codec: "arx4", + }, + diff: { + kind: "diff", + title: "Priority change", + filename: "priority.patch", + content: "diff --git a/src/lib/payload/fragment.ts b/src/lib/payload/fragment.ts\n--- a/src/lib/payload/fragment.ts\n+++ b/src/lib/payload/fragment.ts\n@@ -1 +1 @@\n-const ASYNC = [\"arx3\"];\n+const ASYNC = [\"arx4\", \"arx3\"];\n", + language: "", + diffView: "unified", + codec: "arx4", + }, +}; + +/** Prior ids the encoder never selects on its own, so they only reach a decoder through this path. */ +function envelopeWithPrior(priorId: Arx4PriorId): { envelope: PayloadEnvelope; fragment: string } { + const envelope: PayloadEnvelope = { + v: 1, + codec: "arx4", + title: `Prior ${priorId}`, + activeArtifactId: "doc", + artifacts: [{ + id: "doc", + kind: "markdown", + filename: "doc.md", + content: `# Prior ${priorId}\n\nSame artifact, coded against the ${priorId} prior.\n`, + }], + }; + + return { envelope, fragment: `${ARX4_TAG}${arx4CompressEnvelope(envelope, priorId).baseBMP}` }; +} + +const reportEnvelope: PayloadEnvelope = { + v: 1, + codec: "plain", + title: "Baanish Code Bench", + activeArtifactId: "baanish-code-bench", + artifacts: [{ + id: "baanish-code-bench", + kind: "markdown", + title: "Baanish Code Bench", + filename: "results.md", + content: readFileSync("tests/fixtures/baanish-code-bench-report.md", "utf8"), + }], +}; + +describe("arx4 codec", () => { + beforeAll(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + // Without the curated priors the encoder degrades every kind id to `s` (see + // tests/arx4-priors.test.ts), so the kind-prior expectations below need the asset loaded. + loadArx4PriorsSync(arx4PriorsJson); + }); + + describe("round trip", () => { + it.each([ + ["markdown", "m"], + ["code", "c"], + ["json", "j"], + ["csv", "j"], + ["diff", "c"], + ] as [ArtifactKind, Arx4PriorId][])("round-trips a %s draft on the %s prior", async (kind, priorId) => { + const generatedLink = await createGeneratedArtifactLinkAsync(drafts[kind], "https://agent-render.com/"); + + expect(generatedLink.codec).toBe("arx4"); + expect(generatedLink.hash.startsWith(`#${ARX4_TAG}${priorId}`)).toBe(true); + + const parsed = await decodeFragmentAsync(generatedLink.hash); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + + expect(parsed.envelope).toEqual({ ...generatedLink.envelope, codec: "arx4" }); + }); + + it.each(["s", "n"] as Arx4PriorId[])("round-trips the %s prior a kind never selects", async (priorId) => { + const { envelope, fragment } = envelopeWithPrior(priorId); + expect(fragment.charAt(1)).toBe(priorId); + + const parsed = await decodeFragmentAsync(`#${fragment}`); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + + expect(parsed.envelope).toEqual(envelope); + }); + + it("stays async-only, like the rest of the arx family", () => { + expect(decodeFragment(`#${ARX4_TAG}mZZ`)).toEqual({ + ok: false, + code: "invalid-format", + message: "arx codecs require async decoding — use decodeFragmentAsync instead.", + }); + }); + }); + + describe("wire payloads", () => { + const envelope = envelopeWithPrior("m").envelope; + + it("decodes every wire alphabet back to the same envelope", () => { + const payloads = arx4CompressEnvelope(envelope); + + expect(isBaseBMPEncoded(payloads.baseBMP.slice(1))).toBe(true); + expect(isBase1kEncoded(payloads.base1k.slice(1))).toBe(true); + expect(isBase64urlEncoded(payloads.base64url.slice(1))).toBe(true); + expect(payloads.base76.slice(1)).toMatch(/^[A-Za-z0-9\-._~!$*()',;:@/=]+$/); + + for (const payload of Object.values(payloads)) { + expect(arx4DecompressEnvelope(payload)).toEqual(envelope); + } + }); + + // The kind ids each name their own curated corpus, so the id char is load-bearing: no two ids + // produce the same coded bytes, and none of them matches the shared prior. + it("codes every kind id against its own prior corpus", () => { + const shared = arx4CompressEnvelope(envelope, "s").base64url; + const coded = new Set([shared.slice(1)]); + + for (const priorId of ["m", "c", "j"] as Arx4PriorId[]) { + const payload = arx4CompressEnvelope(envelope, priorId).base64url; + expect(payload.charAt(0)).toBe(priorId); + coded.add(payload.slice(1)); + } + + expect(coded.size).toBe(4); + }); + + it("rejects an unknown prior id instead of guessing a prior", async () => { + const payload = arx4CompressEnvelope(envelope).baseBMP; + const unknown = `z${payload.slice(1)}`; + + expect(() => arx4DecompressEnvelope(unknown)).toThrow(/Unsupported arx4 prior id "z"/); + + const parsed = await decodeFragmentAsync(`#${ARX4_TAG}${unknown}`); + expect(parsed.ok).toBe(false); + if (parsed.ok) return; + expect(parsed.code).toBe("invalid-json"); + }); + }); + + describe("selection", () => { + it("wins auto selection against arx3 on the report fixture", async () => { + const autoFragment = await encodeEnvelopeAsync(reportEnvelope); + const arx3Fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx3" }); + + expect(autoFragment.startsWith(ARX4_TAG)).toBe(true); + expect(autoFragment.length).toBeLessThan(arx3Fragment.length); + + const parsed = await decodeFragmentAsync(`#${autoFragment}`); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope).toEqual({ ...reportEnvelope, codec: "arx4" }); + }); + + it("keeps an ASCII wire when the markdown surface budgets by transport length", async () => { + const fragment = await encodeEnvelopeAsync(reportEnvelope, { codec: "arx4", budgetByTransport: true }); + + expect(fragment.startsWith(ARX4_TAG)).toBe(true); + expect(fragment).toMatch(/^[\x21-\x7e]+$/); + expect(isBaseBMPEncoded(fragment.slice(2))).toBe(false); + }); + }); + + /** + * Characterization vectors. The coder is integer-only and primed from the pinned dictionaries, so + * these strings are fixed for a given build; a diff here means the model, the prior, the tuple + * stage or the dictionary changed, and every arx4 link already shared has stopped decoding. + * base64url is the asserted wire because it is ASCII and diffs readably. + */ + describe("determinism vectors", () => { + it.each(arx4DeterminismVectors)("pins the %s prior payload", (priorId, content, expected) => { + const envelope = arx4VectorEnvelope(content); + + expect(arx4PriorIdForEnvelope(envelope)).toBe("m"); + expect(arx4CompressEnvelope(envelope, priorId).base64url).toBe(expected); + expect(arx4DecompressEnvelope(arx4CompressEnvelope(envelope, priorId).base64url)).toEqual(envelope); + }); + }); +}); diff --git a/tests/arx4-curated-prior-fetch.test.ts b/tests/arx4-curated-prior-fetch.test.ts new file mode 100644 index 0000000..4ce9fad --- /dev/null +++ b/tests/arx4-curated-prior-fetch.test.ts @@ -0,0 +1,71 @@ +import { afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { arx4DeterminismVectors } from "./fixtures/arx4-vectors"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { getActiveArx4PriorsVersion } from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync } from "@/lib/payload/fragment"; +import { compactTagForCodec } from "@/lib/payload/schema"; + +/** + * The lazy priors fetch is routed off the fragment's prior id char, and the decoder reads that char + * after percent-decoding: a re-encoding proxy or a handcrafted fragment can deliver `%6d` where the + * app itself would have written `m`. Routing on the raw char leaves such a fragment permanently + * undecodable, because the asset it needs is never requested however often the viewer retries. + */ +const ARX4_TAG = compactTagForCodec("arx4"); +const [, curatedContent, curatedPayload] = arx4DeterminismVectors[0]; +const [, sharedContent, sharedPayload] = arx4DeterminismVectors[3]; + +function percentEncodeFirstChar(payload: string): string { + return `%${payload.charCodeAt(0).toString(16)}${payload.slice(1)}`; +} + +describe("arx4 curated prior fetch routing", () => { + const fetchSpy = vi.fn(() => + Promise.resolve({ ok: true, json: () => Promise.resolve(arx4PriorsJson) } as Response), + ); + + beforeAll(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + // Deliberately no priors: these cases are about which fragments trigger the lazy fetch. + }); + + afterEach(() => { + vi.unstubAllGlobals(); + fetchSpy.mockClear(); + }); + + // Runs first, while the priors slot is still cold: a fetch here would be an unnecessary one. + it("still skips the fetch for a percent-encoded shared prior id", async () => { + expect(getActiveArx4PriorsVersion()).toBe(0); + vi.stubGlobal("fetch", fetchSpy); + + const parsed = await decodeFragmentAsync( + `#${ARX4_TAG}${percentEncodeFirstChar(sharedPayload)}`, + { skipFragmentBudget: true }, + ); + + expect(fetchSpy).not.toHaveBeenCalled(); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope.artifacts[0]).toMatchObject({ content: sharedContent }); + }); + + it("fetches the curated asset for a percent-encoded curated prior id", async () => { + expect(getActiveArx4PriorsVersion()).toBe(0); + vi.stubGlobal("fetch", fetchSpy); + + const parsed = await decodeFragmentAsync( + `#${ARX4_TAG}${percentEncodeFirstChar(curatedPayload)}`, + { skipFragmentBudget: true }, + ); + + expect(fetchSpy).toHaveBeenCalled(); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope.artifacts[0]).toMatchObject({ content: curatedContent }); + }); +}); diff --git a/tests/arx4-dictionary-pin-guard.test.ts b/tests/arx4-dictionary-pin-guard.test.ts new file mode 100644 index 0000000..079e8ce --- /dev/null +++ b/tests/arx4-dictionary-pin-guard.test.ts @@ -0,0 +1,101 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { arx4DeterminismVectors, arx4VectorEnvelope } from "./fixtures/arx4-vectors"; +import { + getActiveArx2OverlayVersion, + getActiveDictVersion, + loadArx2OverlayDictionarySync, + loadArxDictionarySync, +} from "@/lib/payload/arx-codec"; +import { loadArx4PriorsSync } from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync, encodeEnvelopeAsync } from "@/lib/payload/fragment"; +import { Arx4DictionarySkewError, decodeArxFragmentPayload } from "@/lib/payload/fragment-arx"; +import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; + +// arx4 codes with the dictionary text twice over (substitution stage plus the context-mixer prior) and +// the compact `e` tag carries no dictionary version, so a fragment coded against anything but the +// pinned dictionaries is a link no healthy viewer can decode. Unlike arx/arx2/arx3, which tolerate the +// built-in fallback dictionary, arx4 has to hold out for the exact pinned pair on both sides: encode +// drops out of the candidate pool, decode refuses. +const ARX4_TAG = compactTagForCodec("arx4"); +const envelope: PayloadEnvelope = { + v: 1, + codec: "plain", + activeArtifactId: "doc", + artifacts: [{ id: "doc", kind: "markdown", content: "# Missing assets fallback" }], +}; +const [, curatedContent, curatedPayload] = arx4DeterminismVectors[0]; +const curatedFragment = `#${ARX4_TAG}${curatedPayload}`; + +describe("arx4 dictionary pin guard", () => { + // The dictionary fetch failing is what leaves the built-in version-0 dictionary active, and the + // module registry starts on the built-in pair, so a rejecting fetch reproduces that state exactly. + beforeEach(() => { + vi.stubGlobal("fetch", vi.fn(() => Promise.reject(new Error("dictionary endpoint is down")))); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + }); + + // These first two cases need the built-in fallback dictionary active, which no test can restore once + // another has loaded the shipped one, so they run before the recovery cases below. + it("skips arx4 in the shared candidate pool while the built-in fallback dictionary is active", async () => { + const fragment = await encodeEnvelopeAsync(envelope); + + expect(getActiveDictVersion()).toBe(0); + expect(fragment.startsWith(ARX4_TAG)).toBe(false); + + // The rest of the pool still serves the link, and what it emits decodes. + expect((await decodeFragmentAsync(`#${fragment}`, { skipFragmentBudget: true })).ok).toBe(true); + + // An explicit arx4 request has no pool left to select from, which is the fail-closed outcome: + // no link is minted that healthy viewers would reject. + await expect(encodeEnvelopeAsync(envelope, { codec: "arx4" })).rejects.toThrow(); + }); + + it("refuses a curated fragment as an unavailable asset instead of decoding on the fallback", async () => { + expect(getActiveDictVersion()).toBe(0); + + await expect(decodeArxFragmentPayload("arx4", curatedPayload)).rejects.toThrow(Arx4DictionarySkewError); + + const parsed = await decodeFragmentAsync(curatedFragment, { skipFragmentBudget: true }); + + expect(parsed.ok).toBe(false); + if (parsed.ok) return; + expect(parsed.code).toBe("asset-unavailable"); + expect(parsed.message).toMatch(/reload/i); + }); + + it("codes arx4 again once the pinned dictionaries are active", async () => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); + + const fragment = await encodeEnvelopeAsync(envelope, { codec: "arx4" }); + expect(fragment.startsWith(ARX4_TAG)).toBe(true); + + const parsed = await decodeFragmentAsync(curatedFragment, { skipFragmentBudget: true }); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope).toEqual(arx4VectorEnvelope(curatedContent)); + }); + + // An OLDER overlay passes the family's "not newer than expected" guard, so arx4 has to reject it on + // its own exact pin rather than inherit that tolerance. + it("skips and refuses arx4 when only the overlay dictionary is off the pinned version", async () => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync({ ...arx2DictionaryJson, version: 0 }); + expect(getActiveDictVersion()).toBe(1); + expect(getActiveArx2OverlayVersion()).toBe(0); + + expect((await encodeEnvelopeAsync(envelope)).startsWith(ARX4_TAG)).toBe(false); + + const parsed = await decodeFragmentAsync(curatedFragment, { skipFragmentBudget: true }); + expect(parsed.ok).toBe(false); + if (parsed.ok) return; + expect(parsed.code).toBe("asset-unavailable"); + }); +}); diff --git a/tests/arx4-priors-fetch-skew.test.ts b/tests/arx4-priors-fetch-skew.test.ts new file mode 100644 index 0000000..c557f5e --- /dev/null +++ b/tests/arx4-priors-fetch-skew.test.ts @@ -0,0 +1,69 @@ +import { afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { + EXPECTED_ARX4_PRIORS_VERSION, + getActiveArx4PriorsVersion, + isArx4PriorsLoaded, + loadArx4Priors, + type Arx4Priors, +} from "@/lib/payload/arx4-codec"; + +/** + * A default priors load tries `/arx4-priors.json.br` first and falls back to the plain JSON, so a + * mid-deploy CDN can serve one version from one URL and another from the other. Installing whatever + * the first URL answers with wedges the page: curated coding needs the expected version exactly, and + * a sticky off-version install makes every retry hit the same skewed URL again. + */ +const priors: Arx4Priors = arx4PriorsJson; +const forwardPriors: Arx4Priors = { ...priors, version: priors.version + 1 }; + +function stubPriorsFetch(bodyForUrl: (url: string) => unknown) { + const fetchSpy = vi.fn((input: unknown) => + Promise.resolve({ ok: true, json: () => Promise.resolve(bodyForUrl(String(input))) } as Response), + ); + vi.stubGlobal("fetch", fetchSpy); + return fetchSpy; +} + +describe("arx4 priors fetch skew", () => { + beforeAll(() => { + // The install check reassembles each kind block against the pinned dictionary text, so the + // dictionaries have to be the shipped ones before any priors load can succeed. + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + }); + + afterEach(() => { + vi.unstubAllGlobals(); + }); + + // Runs first: it is the only case that can observe the never-installed slot. + it("leaves the slot uninstalled and refetchable when every URL is off-version", async () => { + const fetchSpy = stubPriorsFetch(() => forwardPriors); + + expect(await loadArx4Priors()).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + expect(getActiveArx4PriorsVersion()).toBe(0); + expect(fetchSpy).toHaveBeenCalledTimes(2); + }); + + it("keeps fetching past an off-version .br asset instead of installing it", async () => { + const fetchSpy = stubPriorsFetch((url) => (url.endsWith(".br") ? forwardPriors : priors)); + + expect(await loadArx4Priors()).toBe(EXPECTED_ARX4_PRIORS_VERSION); + expect(getActiveArx4PriorsVersion()).toBe(EXPECTED_ARX4_PRIORS_VERSION); + expect(fetchSpy).toHaveBeenCalledTimes(2); + }); + + it("keeps fetching past a digest-corrupt right-version .br asset", async () => { + const corruptPriors: Arx4Priors = { ...priors, kinds: { ...priors.kinds, markdown: "x" } }; + const fetchSpy = stubPriorsFetch((url) => (url.endsWith(".br") ? corruptPriors : priors)); + + expect(await loadArx4Priors()).toBe(EXPECTED_ARX4_PRIORS_VERSION); + expect(getActiveArx4PriorsVersion()).toBe(EXPECTED_ARX4_PRIORS_VERSION); + expect(fetchSpy).toHaveBeenCalledTimes(2); + }); +}); diff --git a/tests/arx4-priors-version-guard.test.ts b/tests/arx4-priors-version-guard.test.ts new file mode 100644 index 0000000..da4a6cf --- /dev/null +++ b/tests/arx4-priors-version-guard.test.ts @@ -0,0 +1,104 @@ +import { beforeEach, describe, expect, it } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { arx4DeterminismVectors, arx4VectorEnvelope } from "./fixtures/arx4-vectors"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { + arx4CompressEnvelope, + arx4DecompressEnvelope, + Arx4PriorsUnavailableError, + EXPECTED_ARX4_PRIORS_VERSION, + loadArx4PriorsSync, + type Arx4Priors, +} from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync, encodeEnvelopeAsync } from "@/lib/payload/fragment"; +import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; + +// The compact `e` tag carries no priors version, so a CDN serving any asset other than the one this +// build was coded against is a skew the codec has to notice. The two sides handle it differently on +// purpose: decode refuses, because coding against the wrong corpus yields plausible garbage, while +// encode degrades to the `s` prior, because the arx4 candidates share one pool with +// arx3/arx2/arx/deflate and throwing would take link creation down with them. +const envelope: PayloadEnvelope = { + v: 1, + codec: "arx4", + activeArtifactId: "doc", + artifacts: [{ id: "doc", kind: "markdown", content: "# Title\n\nSome content for the arx4 context mixer." }], +}; + +const priors: Arx4Priors = arx4PriorsJson; +const forwardPriors: Arx4Priors = { ...priors, version: priors.version + 1 }; +const arx4Tag = compactTagForCodec("arx4"); + +describe("arx4 priors version guard", () => { + beforeEach(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(priors); // expected version (1) + }); + + it("degrades encoding to the s prior instead of failing the whole candidate pool", async () => { + expect(await encodeEnvelopeAsync(envelope, { codec: "arx4" })).toMatch(new RegExp(`^${arx4Tag}m`)); + + loadArx4PriorsSync(forwardPriors); + + // arx4 still encodes, against the prior this build does support, and says so in the emitted id. + const degraded = await encodeEnvelopeAsync(envelope, { codec: "arx4" }); + expect(degraded).toMatch(new RegExp(`^${arx4Tag}s`)); + const parsedDegraded = await decodeFragmentAsync(`#${degraded}`, { skipFragmentBudget: true }); + expect(parsedDegraded.ok).toBe(true); + + // And the shared auto pool is never rejected along with it. + const auto = await encodeEnvelopeAsync(envelope); + const parsedAuto = await decodeFragmentAsync(`#${auto}`, { skipFragmentBudget: true }); + expect(parsedAuto.ok).toBe(true); + }); + + it("refuses to decode a curated fragment while a forward-version asset is active", async () => { + const curated = await encodeEnvelopeAsync(envelope, { codec: "arx4" }); + expect((await decodeFragmentAsync(`#${curated}`, { skipFragmentBudget: true })).ok).toBe(true); + + loadArx4PriorsSync(forwardPriors); + const skewed = await decodeFragmentAsync(`#${curated}`, { skipFragmentBudget: true }); + expect(skewed.ok).toBe(false); // hard-fail, not a silent mis-decode + // And it says which kind of failure it is: the fragment is fine, the asset is not, so a caller + // branching on the code must not treat this like a corrupt payload. + if (!skewed.ok) expect(skewed.code).toBe("asset-unavailable"); + + // Retryable: the same link decodes again once the expected-version asset is active. + loadArx4PriorsSync(priors); + const recovered = await decodeFragmentAsync(`#${curated}`, { skipFragmentBudget: true }); + expect(recovered.ok).toBe(true); + }); + + // Regression: an asset OLDER than expected has a valid shape too, so a guard that only rejects + // newer versions installs it and codes curated fragments against a corpus no current-version + // decoder can rebuild. Version 0 is the reported case (a stale or rolled-back asset). + describe("with an older-version asset active", () => { + const stalePriors: Arx4Priors = { ...priors, version: 0 }; + const [, curatedContent, curatedPayload] = arx4DeterminismVectors[0]; + const curatedEnvelope = arx4VectorEnvelope(curatedContent); + + it("degrades encoding to the s prior exactly as a missing asset does", () => { + expect(loadArx4PriorsSync(stalePriors)).toBe(0); + + const payload = arx4CompressEnvelope(curatedEnvelope).base64url; + expect(payload.charAt(0)).toBe("s"); + expect(payload).toBe(arx4CompressEnvelope(curatedEnvelope, "s").base64url); + }); + + it("refuses a curated fragment instead of coding against the stale corpus", () => { + expect(loadArx4PriorsSync(stalePriors)).toBe(0); + expect(() => arx4DecompressEnvelope(curatedPayload)).toThrow(Arx4PriorsUnavailableError); + }); + + it("codes curated fragments again once the shipped asset is active", () => { + expect(loadArx4PriorsSync(stalePriors)).toBe(0); + expect(loadArx4PriorsSync(priors)).toBe(EXPECTED_ARX4_PRIORS_VERSION); + + expect(arx4CompressEnvelope(curatedEnvelope).base64url).toBe(curatedPayload); + expect(arx4DecompressEnvelope(curatedPayload)).toEqual(curatedEnvelope); + }); + }); +}); diff --git a/tests/arx4-priors.test.ts b/tests/arx4-priors.test.ts new file mode 100644 index 0000000..ea08233 --- /dev/null +++ b/tests/arx4-priors.test.ts @@ -0,0 +1,162 @@ +import { createHash } from "node:crypto"; +import { beforeAll, describe, expect, it, vi } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { arx4DeterminismVectors, arx4VectorEnvelope } from "./fixtures/arx4-vectors"; +import { + getArxDictionaryPriorText, + loadArx2OverlayDictionarySync, + loadArxDictionarySync, +} from "@/lib/payload/arx-codec"; +import { + ARX4_PRIOR_BYTES, + arx4CompressEnvelope, + arx4DecompressEnvelope, + Arx4PriorsUnavailableError, + getActiveArx4PriorsVersion, + isArx4PriorsLoaded, + loadArx4Priors, + loadArx4PriorsSync, + PINNED_ARX4_PRIOR_SHA256, + type Arx4Priors, +} from "@/lib/payload/arx4-codec"; + +/** + * Byte-identity gate for public/arx4-priors.json, plus the behavior on either side of loading it. + * + * The asset carries only the kind-specific tail of each curated prior; the codec rebuilds the + * 2203-char common prefix from the pinned dictionaries. The digests are the codec's own install-time + * pins, recomputed here with node:crypto: they are the 16384-char priors the ARX4 research benchmarks + * measured, as the maintainer-local frozen source builds them. A diff here means the asset, the + * dictionaries or the split changed, and every arx4 link on an `m`, `c` or `j` prior has stopped + * decoding. Regenerate with `node scripts/build-arx4-priors.mjs `, which fails + * the same way when it cannot reproduce these from the frozen source. + */ +const COMMON_PREFIX_CHARS = 2203; + +const priors: Arx4Priors = arx4PriorsJson; +const [, curatedVectorContent, curatedVectorPayload] = arx4DeterminismVectors[0]; + +describe("arx4 priors asset", () => { + beforeAll(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + }); + + // Runs first on purpose: the codec holds one module-level priors slot with no unload path, so the + // unloaded behavior can only be observed before any test loads the asset. Each case re-asserts the + // unloaded state so a reordering fails on the precondition instead of on the behavior. + describe("before the asset loads", () => { + it("reports itself unloaded", () => { + expect(isArx4PriorsLoaded()).toBe(false); + expect(getActiveArx4PriorsVersion()).toBe(0); + }); + + it("encodes on the shared prior and emits the s id rather than blocking", () => { + expect(isArx4PriorsLoaded()).toBe(false); + + const envelope = arx4VectorEnvelope(curatedVectorContent); + const payload = arx4CompressEnvelope(envelope).base64url; + + expect(payload.charAt(0)).toBe("s"); + expect(payload).toBe(arx4CompressEnvelope(envelope, "s").base64url); + }); + + it("refuses a curated fragment instead of decoding it against the shared prior", () => { + expect(isArx4PriorsLoaded()).toBe(false); + expect(() => arx4DecompressEnvelope(curatedVectorPayload)).toThrow(Arx4PriorsUnavailableError); + }); + + it("decodes s and n fragments without attempting the priors fetch", async () => { + expect(isArx4PriorsLoaded()).toBe(false); + const fetchSpy = vi.fn(() => { + throw new Error("priors fetch must not happen for s/n fragments"); + }); + vi.stubGlobal("fetch", fetchSpy); + try { + const { decodeArxFragmentPayload } = await import("@/lib/payload/fragment-arx"); + for (const [priorId, content, payload] of arx4DeterminismVectors) { + if (priorId !== "s" && priorId !== "n") continue; + const decoded = await decodeArxFragmentPayload("arx4", payload); + const envelope = typeof decoded === "string" ? JSON.parse(decoded) : decoded; + expect(envelope.artifacts[0].content).toBe(content); + } + expect(fetchSpy).not.toHaveBeenCalled(); + } finally { + vi.unstubAllGlobals(); + } + }); + }); + + describe("loading", () => { + it("reports a fetch failure as -1 so the caller can retry", async () => { + expect(await loadArx4Priors("http://127.0.0.1:1/arx4-priors.json")).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + }); + + it("rejects an asset that is missing a curated kind", async () => { + expect(await loadArx4Priors({ version: 1, kinds: { markdown: "x" } } as unknown as Arx4Priors)).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + }); + + // A version that is not a non-negative integer would install while every caller read it as the + // -1 failure sentinel, so the asset would be live and reported unloaded at the same time. + it("rejects an asset whose version is not a non-negative integer", async () => { + for (const version of [-1, 1.5, Number.NaN]) { + expect(await loadArx4Priors({ ...priors, version })).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + } + }); + + // A shape-valid asset at the expected version is still unusable when its kind blocks are not the + // corpora this build's fragments were coded against: a truncated or swapped block reads as + // "installed and authoritative", so every later load short-circuits and curated links stay broken. + it("rejects an expected-version asset whose kind blocks are not the pinned corpora", async () => { + expect(await loadArx4Priors({ version: 1, kinds: { markdown: "x", code: "x", json: "x" } })).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + + const swapped = { version: 1, kinds: { ...priors.kinds, markdown: priors.kinds.code } }; + expect(await loadArx4Priors(swapped)).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + }); + + // The asset is trusted input, so an oversize kind block would otherwise be primed byte by byte + // through the mixer (a ~5 MB block measured seconds per encode). + it("rejects an oversize kind block before it can reach the mixer", async () => { + const oversize = { version: 1, kinds: { ...priors.kinds, markdown: "X".repeat(5_000_000) } }; + + expect(await loadArx4Priors(oversize)).toBe(-1); + expect(isArx4PriorsLoaded()).toBe(false); + }); + + it("loads the shipped asset at the version this build pins", () => { + expect(loadArx4PriorsSync(priors)).toBe(1); + expect(isArx4PriorsLoaded()).toBe(true); + expect(getActiveArx4PriorsVersion()).toBe(1); + }); + }); + + describe("with the asset loaded", () => { + beforeAll(() => { + loadArx4PriorsSync(priors); + }); + + it("reassembles the benched 16384-char priors byte for byte", () => { + const commonPrefix = `${getArxDictionaryPriorText()}\n`; + expect(commonPrefix).toHaveLength(COMMON_PREFIX_CHARS); + + for (const [kind, expectedSha256] of Object.entries(PINNED_ARX4_PRIOR_SHA256)) { + const prior = `${commonPrefix}${priors.kinds[kind as keyof typeof PINNED_ARX4_PRIOR_SHA256]}`; + + expect(prior).toHaveLength(ARX4_PRIOR_BYTES); + expect(Buffer.byteLength(prior, "utf8")).toBe(ARX4_PRIOR_BYTES); + expect(createHash("sha256").update(prior, "utf8").digest("hex")).toBe(expectedSha256); + } + }); + + it("decodes the curated fragment the unloaded build refused", () => { + expect(arx4DecompressEnvelope(curatedVectorPayload)).toEqual(arx4VectorEnvelope(curatedVectorContent)); + }); + }); +}); diff --git a/tests/compact-header.test.ts b/tests/compact-header.test.ts index 2d09167..31706cb 100644 --- a/tests/compact-header.test.ts +++ b/tests/compact-header.test.ts @@ -34,6 +34,7 @@ describe("compact fragment header", () => { arx: "a", arx2: "b", arx3: "c", + arx4: "e", }); }); @@ -50,7 +51,7 @@ describe("compact fragment header", () => { expect(parsed.ok).toBe(true); if (parsed.ok) expect(markdownContent(parsed)).toBe(envelope.artifacts[0].kind === "markdown" ? envelope.artifacts[0].content : ""); } - for (const codec of ["arx", "arx2", "arx3"] as const) { + for (const codec of ["arx", "arx2", "arx3", "arx4"] as const) { const parsed = await decodeFragmentAsync(`#${await encodeEnvelopeAsync(envelope, { codec })}`); expect(parsed.ok).toBe(true); if (parsed.ok) expect(markdownContent(parsed)).toBe(envelope.artifacts[0].kind === "markdown" ? envelope.artifacts[0].content : ""); diff --git a/tests/components/link-creator.test.tsx b/tests/components/link-creator.test.tsx index d28411d..3c741e4 100644 --- a/tests/components/link-creator.test.tsx +++ b/tests/components/link-creator.test.tsx @@ -46,6 +46,7 @@ function createGeneratedLink(title: string): GeneratedArtifactLink { fragmentLength: 64, hash: `#agent-render=v1.plain.${title}`, url, + markdownUrl: url, markdownLink: shareInfo.markdownLink, markdownLinkLength: shareInfo.length, discordMarkdownLinkWarning: shareInfo.discordWarning, @@ -59,6 +60,14 @@ afterEach(() => { }); describe("LinkCreator", () => { + it("offers every registered codec in the compression selector", () => { + render(); + + for (const option of ["auto", "plain", "lz", "deflate", "arx", "arx2", "arx3", "arx4"]) { + expect(screen.getByRole("button", { name: option })).toBeInTheDocument(); + } + }); + it("keeps the newest generated link when async requests resolve out of order", async () => { const user = userEvent.setup(); diff --git a/tests/e2e/arx4-determinism.spec.ts b/tests/e2e/arx4-determinism.spec.ts new file mode 100644 index 0000000..d838dcf --- /dev/null +++ b/tests/e2e/arx4-determinism.spec.ts @@ -0,0 +1,217 @@ +import path from "node:path"; +import { expect, test, type Page } from "@playwright/test"; +import { build } from "esbuild"; +import { arx4DeterminismVectors, arx4VectorEnvelope } from "../fixtures/arx4-vectors"; +import { goToHash, waitForRendererReady, waitForViewerState } from "./helpers"; +import arx2DictionaryJson from "../../public/arx2-dictionary.json"; +import arx4PriorsJson from "../../public/arx4-priors.json"; +import arxDictionaryJson from "../../public/arx-dictionary.json"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { loadArx4PriorsSync, type Arx4PriorId } from "@/lib/payload/arx4-codec"; +import { createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; +import { compactTagForCodec, type PayloadEnvelope } from "@/lib/payload/schema"; + +/** + * Release gate for the arx4 wire format: a browser that codes even one bit differently from Node + * silently corrupts every arx4 link already shared, so the pinned vectors in + * tests/fixtures/arx4-vectors.ts are asserted character for character in Chromium and WebKit too, + * not only in the Node suite. + * + * Two layers, because no shipped export reaches the codec directly: + * 1. The pinned vectors run against the codec source bundled into the page, which is the only way to + * pick a prior id explicitly (the app always derives it from the artifact kind). + * 2. The link creator and the viewer exercise the same codec inside the real minified app bundle, + * where encode is compared against Node's own output for the same draft and the artifact content + * is read back through the app's copy action. + */ + +const ARX4_TAG = compactTagForCodec("arx4"); + +declare global { + interface Window { + __arx4Determinism?: { + loadPriorAssets: () => Promise<[number, number, number]>; + encodeBase64url: (envelope: PayloadEnvelope, priorId: Arx4PriorId) => string; + }; + } +} + +/** + * Entry point for the in-page harness. It pulls the same modules the app imports, and loads the + * dictionaries and the curated priors from the served assets so the coder is primed from the shipped + * corpus rather than a copy handed in from Node. + * + * The priors load runs after the dictionaries, as it does in the app: installing the asset checks each + * kind block against the pinned prior, which can only be reassembled once the pinned dictionaries are + * the active ones. + */ +const HARNESS_ENTRY = ` +import { loadArx2OverlayDictionary, loadArxDictionary } from "@/lib/payload/arx-codec"; +import { arx4CompressEnvelope, loadArx4Priors } from "@/lib/payload/arx4-codec"; + +window.__arx4Determinism = { + loadPriorAssets: async () => { + const dictionaries = await Promise.all([ + loadArxDictionary(new URL("arx-dictionary.json", window.location.href).toString()), + loadArx2OverlayDictionary(new URL("arx2-dictionary.json", window.location.href).toString()), + ]); + return [...dictionaries, await loadArx4Priors(new URL("arx4-priors.json", window.location.href).toString())]; + }, + encodeBase64url: (envelope, priorId) => arx4CompressEnvelope(envelope, priorId).base64url, +}; +`; + +const repositoryRoot = path.resolve(__dirname, "../.."); +let harnessBundle: Promise | null = null; + +function bundleCodecForBrowser(): Promise { + harnessBundle ??= build({ + stdin: { contents: HARNESS_ENTRY, resolveDir: repositoryRoot, sourcefile: "arx4-harness.ts", loader: "ts" }, + absWorkingDir: repositoryRoot, + tsconfig: "tsconfig.json", + bundle: true, + format: "iife", + platform: "browser", + target: "es2020", + // Only the Brotli codecs reach it, and a browser bundle of the wasm module is not needed here. + external: ["brotli-wasm"], + write: false, + }).then((result) => result.outputFiles[0].text); + + return harnessBundle; +} + +async function installCodecHarness(page: Page) { + await page.addScriptTag({ content: await bundleCodecForBrowser() }); + + // A failed dictionary fetch silently falls back to the built-in dictionary (version 0), and failed + // priors downgrade every kind id to `s`; both code different bytes and would read as a determinism + // failure, so pin the loaded versions. + const versions = await page.evaluate(() => window.__arx4Determinism!.loadPriorAssets()); + expect(versions).toEqual([1, 1, 1]); +} + +async function fillCreatorDraft(page: Page, draft: LinkCreatorDraft) { + await page.getByRole("button", { name: draft.kind, exact: true }).click(); + await page.getByLabel("Title").fill(draft.title); + await page.getByLabel("Filename").fill(draft.filename); + if (draft.kind === "code") { + await page.getByRole("textbox", { name: "Language", exact: true }).fill(draft.language); + } + await page.getByRole("textbox", { name: /^Content\b/ }).fill(draft.content); + await page.getByRole("button", { name: "arx4", exact: true }).click(); +} + +/** Reads the artifact body the viewer decoded, through the app's own copy action. */ +async function copyArtifactBody(page: Page): Promise { + await page.evaluate(() => { + window.localStorage.removeItem("copied-artifact-body"); + Object.defineProperty(navigator, "clipboard", { + configurable: true, + value: { + writeText: (value: string) => { + window.localStorage.setItem("copied-artifact-body", value); + return Promise.resolve(); + }, + }, + }); + }); + + await page.getByRole("button", { name: "Copy", exact: true }).first().click(); + await expect(page.getByRole("button", { name: "Copied" }).first()).toBeVisible(); + return page.evaluate(() => window.localStorage.getItem("copied-artifact-body")); +} + +const creatorDrafts: LinkCreatorDraft[] = [ + { + kind: "markdown", + title: "Release notes", + filename: "notes.md", + content: "# Release notes\n\n- Ship the arx4 codec\n- Keep fragments copyable\n\n| Surface | State |\n| --- | --- |\n| viewer | ready |\n| creator | ready |\n", + language: "", + diffView: "unified", + codec: "arx4", + }, + { + kind: "code", + title: "Wire selection", + filename: "wire.ts", + content: "export function selectWire(candidates: Candidate[]): Candidate {\n return candidates.reduce((best, candidate) => (candidate.length < best.length ? candidate : best));\n}\n", + language: "ts", + diffView: "unified", + codec: "arx4", + }, + { + kind: "json", + title: "Manifest", + filename: "manifest.json", + content: "{\n \"codec\": \"arx4\",\n \"wire\": [\"base76\", \"base1k\", \"baseBMP\", \"base64url\"],\n \"priors\": 5\n}\n", + language: "", + diffView: "unified", + codec: "arx4", + }, +]; + +test.beforeAll(() => { + // The Node side of this spec encodes too, and an unloaded asset would fall back (built-in + // dictionary, `s` prior) and produce different bytes than the browser reads from the served files. + expect(loadArxDictionarySync(arxDictionaryJson)).toBe(1); + expect(loadArx2OverlayDictionarySync(arx2DictionaryJson)).toBe(1); + expect(loadArx4PriorsSync(arx4PriorsJson)).toBe(1); +}); + +test.describe("arx4 pinned vectors in the browser", () => { + for (const [priorId, content, expected] of arx4DeterminismVectors) { + test(`encodes the ${priorId} prior vector to the pinned payload`, async ({ page }) => { + await goToHash(page); + await waitForViewerState(page, "empty"); + await installCodecHarness(page); + + const produced = await page.evaluate( + (vector) => window.__arx4Determinism!.encodeBase64url(vector.envelope, vector.priorId), + { envelope: arx4VectorEnvelope(content), priorId }, + ); + + expect(produced).toBe(expected); + }); + + test(`round-trips the pinned ${priorId} prior fragment through the viewer`, async ({ page }) => { + await goToHash(page, `#${ARX4_TAG}${expected}`); + await waitForViewerState(page, "artifact"); + await waitForRendererReady(page, "markdown"); + + expect(await copyArtifactBody(page)).toBe(content); + }); + } +}); + +test.describe("arx4 links from the shipped app bundle", () => { + for (const draft of creatorDrafts) { + test(`generates and previews the Node-identical arx4 link for a ${draft.kind} draft`, async ({ page }) => { + await goToHash(page); + await waitForViewerState(page, "empty"); + await fillCreatorDraft(page, draft); + await page.getByRole("button", { name: "Generate link" }).click(); + + const generatedLink = page.getByLabel("Generated agent-render link"); + await expect(generatedLink).toBeVisible(); + // The creator encodes against `location.href` with the fragment stripped, so Node has to + // build its comparison URL from the same base. + const baseUrl = await page.evaluate(() => { + const url = new URL(window.location.href); + url.hash = ""; + return url.toString(); + }); + const nodeLink = await createGeneratedArtifactLinkAsync(draft, baseUrl); + + expect(nodeLink.codec).toBe("arx4"); + expect(await generatedLink.inputValue()).toBe(nodeLink.url); + + await page.getByRole("button", { name: "Preview here" }).click(); + await waitForViewerState(page, "artifact"); + await waitForRendererReady(page, draft.kind); + + expect(await copyArtifactBody(page)).toBe(draft.content); + }); + } +}); diff --git a/tests/e2e/visual.spec.ts-snapshots/code-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/code-light-chromium.png index f550864..295a904 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/code-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/code-light-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/csv-compact-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/csv-compact-light-chromium.png index e36914a..dabb9bd 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/csv-compact-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/csv-compact-light-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/diff-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/diff-light-chromium.png index 83edd96..5346db6 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/diff-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/diff-light-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/empty-state-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/empty-state-light-chromium.png index 955c307..cb095fd 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/empty-state-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/empty-state-light-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/json-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/json-light-chromium.png index 49e2790..a66d228 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/json-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/json-light-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/markdown-dark-chromium.png b/tests/e2e/visual.spec.ts-snapshots/markdown-dark-chromium.png index 88f7f11..7c5c276 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/markdown-dark-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/markdown-dark-chromium.png differ diff --git a/tests/e2e/visual.spec.ts-snapshots/markdown-light-chromium.png b/tests/e2e/visual.spec.ts-snapshots/markdown-light-chromium.png index 29d4cb2..cd2830b 100644 Binary files a/tests/e2e/visual.spec.ts-snapshots/markdown-light-chromium.png and b/tests/e2e/visual.spec.ts-snapshots/markdown-light-chromium.png differ diff --git a/tests/fixtures/arx4-vectors.ts b/tests/fixtures/arx4-vectors.ts new file mode 100644 index 0000000..5c130cf --- /dev/null +++ b/tests/fixtures/arx4-vectors.ts @@ -0,0 +1,32 @@ +import type { Arx4PriorId } from "@/lib/payload/arx4-codec"; +import type { PayloadEnvelope } from "@/lib/payload/schema"; + +/** + * Pinned arx4 characterization vectors: `[prior id, artifact content, expected base64url payload]`. + * + * Shared by tests/arx4-codec.test.ts (Node) and tests/e2e/arx4-determinism.spec.ts (Chromium and + * WebKit) so both engines are held to the same strings. base64url is the pinned wire because it is + * ASCII and diffs readably; every other wire is a lossless re-encoding of the same coded bytes. + * + * The `m`, `c` and `j` rows only hold with the curated priors asset loaded (public/arx4-priors.json); + * without it the encoder degrades to the `s` prior and emits an `s` id, so a diff here can also mean + * the asset failed to load rather than that the coder changed. + */ +export const arx4DeterminismVectors: [Arx4PriorId, string, string][] = [ + ["m", "# Vector\n\nOne markdown line.\n", "mB.O_xYpsmnF3Fjw-Z9PGHhL_dTj0p3r3SVWs1mFJtyeMaPfeIlsMWgVsGNqKN86A"], + ["c", "export const vector = 1;\n", "cB.M__uQDGmv78MoPGNjfOs2KgY4fn7UE6oTuQN1uUycnBbOMjiIA"], + ["j", "{\"vector\":true}\n", "jB.Nf-0unuVZkCRtHoDSx9ECYvKuaxxbiRfjODhYu2HBCtZv5bbs75QKeU"], + ["s", "Shared prior vector.\n", "sB.OtTxZjMc80GBVn5NhNnpa9vx1QyfTbf5ubtWtHbfbnGtGrvqMOj6UQ"], + ["n", "Cold model vector.\n", "nB.OOCzvJS_jHbUL4OKSHLkRONp-rl4LRIO11CBl-AaUnR4ajSBpLwkF-smuGWu"], +]; + +/** The envelope every vector codes; only the artifact content varies between rows. */ +export function arx4VectorEnvelope(content: string): PayloadEnvelope { + return { + v: 1, + codec: "arx4", + title: "Vector", + activeArtifactId: "vector", + artifacts: [{ id: "vector", kind: "markdown", filename: "vector.md", content }], + }; +} diff --git a/tests/fragment-arx-selection.test.ts b/tests/fragment-arx-selection.test.ts index 7020142..0706a47 100644 --- a/tests/fragment-arx-selection.test.ts +++ b/tests/fragment-arx-selection.test.ts @@ -91,8 +91,10 @@ describe("arx3-vs-arx2 selection policy", () => { expect(arx2BaseBMP).toBeDefined(); expect(arx2BaseBMP!.transportLength).toBeGreaterThan(chosen.transportLength); - // End-to-end: the public auto encoder commits to the same arx3 baseBMP wire. - const fragment = await encodeEnvelopeAsync(reportEnvelope); + // End-to-end: the public encoder commits to the same arx3 baseBMP wire. The priority is pinned + // to this pair because arx4 shares the policy and outcompresses arx3, so it would otherwise win + // the global auto pool and stop exercising the arx3-vs-arx2 comparison this test is about. + const fragment = await encodeEnvelopeAsync(reportEnvelope, { codecPriority: ["arx3", "arx2"] }); expect(fragment.startsWith(compactTagForCodec("arx3"))).toBe(true); expect(isBaseBMPEncoded(getArx3PayloadBody(fragment))).toBe(true); }); diff --git a/tests/link-creator-encode-once.test.ts b/tests/link-creator-encode-once.test.ts new file mode 100644 index 0000000..3d4c2ab --- /dev/null +++ b/tests/link-creator-encode-once.test.ts @@ -0,0 +1,76 @@ +import { beforeEach, describe, expect, it, vi } from "vitest"; +import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; +import arxDictionaryJson from "../public/arx-dictionary.json"; +import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; +import { loadArx4PriorsSync, type Arx4Priors } from "@/lib/payload/arx4-codec"; +import { decodeFragmentAsync } from "@/lib/payload/fragment"; +import { createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; + +/** + * A generated link needs two selections over the same candidates: the copy-paste URL keeps the + * arx3/arx4 visible-length budget while the markdown destination is measured percent-escaped. Those + * are two reads of one pool, not two encodes: arx4's context mixer costs ~770 ms per 60 KB artifact, + * so a second pass would double every link creation's main-thread stall for identical bytes. + */ +const arx4Compressions = vi.fn(); + +vi.mock("@/lib/payload/arx4-codec", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + arx4CompressEnvelope: (...args: Parameters) => { + arx4Compressions(...args); + return actual.arx4CompressEnvelope(...args); + }, + }; +}); + +const draft: LinkCreatorDraft = { + kind: "markdown", + title: "Launch note", + filename: "brief.md", + content: [ + "# Launch note", + "", + "Share one artifact at a time without uploading it anywhere.", + "", + "- Markdown stays readable", + "- Code keeps its language hint", + "- The link works from a static export", + ].join("\n"), + language: "", + diffView: "unified", + codec: "arx4", +}; + +describe("async link creation", () => { + beforeEach(() => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson as Arx4Priors); + arx4Compressions.mockClear(); + }); + + it("codes the payload once and picks both link surfaces from that pool", async () => { + const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); + + expect(arx4Compressions).toHaveBeenCalledTimes(1); + + // Both surfaces still come out of the pool decodable, and to the same envelope. + const markdownFragment = generatedLink.markdownUrl.slice(generatedLink.markdownUrl.indexOf("#") + 1); + const parsedMarkdown = await decodeFragmentAsync(`#${markdownFragment}`); + const parsedPaste = await decodeFragmentAsync(generatedLink.hash); + expect(parsedMarkdown.ok).toBe(true); + expect(parsedPaste.ok).toBe(true); + if (parsedMarkdown.ok && parsedPaste.ok) { + expect(parsedMarkdown.envelope).toEqual(parsedPaste.envelope); + } + }); + + it("codes the payload once in auto mode too, where arx4 leads the codec priority", async () => { + await createGeneratedArtifactLinkAsync({ ...draft, codec: "auto" }, "https://agent-render.com/"); + + expect(arx4Compressions).toHaveBeenCalledTimes(1); + }); +}); diff --git a/tests/link-creator.test.ts b/tests/link-creator.test.ts index 86c3862..2ff3b7d 100644 --- a/tests/link-creator.test.ts +++ b/tests/link-creator.test.ts @@ -1,12 +1,15 @@ +import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; import { describe, expect, it } from "vitest"; import arx2DictionaryJson from "../public/arx2-dictionary.json"; +import arx4PriorsJson from "../public/arx4-priors.json"; import arxDictionaryJson from "../public/arx-dictionary.json"; import { loadArx2OverlayDictionarySync, loadArxDictionarySync } from "@/lib/payload/arx-codec"; -import { decodeFragment, decodeFragmentAsync } from "@/lib/payload/fragment"; +import { loadArx4PriorsSync } from "@/lib/payload/arx4-codec"; +import { decodeFragment, decodeFragmentAsync, getFragmentTransportLength, getVisibleFragmentLength } from "@/lib/payload/fragment"; import { createDraftEnvelope, createGeneratedArtifactLink, createGeneratedArtifactLinkAsync, type LinkCreatorDraft } from "@/lib/payload/link-creator"; import { DISCORD_MESSAGE_MAX_LENGTH } from "@/lib/markdown-link"; -import { compactTagForCodec } from "@/lib/payload/schema"; +import { compactTagForCodec, MAX_FRAGMENT_LENGTH } from "@/lib/payload/schema"; describe("link creator payloads", () => { it("builds a single-artifact envelope for pasted markdown", () => { @@ -116,7 +119,7 @@ describe("link creator payloads", () => { expect(parsed.ok).toBe(true); }); - it("lets the async auto encoder pick ARX3 when it wins the visible URL budget", async () => { + it("lets the async auto encoder pick ARX4 when it wins the visible URL budget", async () => { loadArxDictionarySync(arxDictionaryJson); loadArx2OverlayDictionarySync(arx2DictionaryJson); @@ -133,13 +136,120 @@ describe("link creator payloads", () => { const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); const parsed = await decodeFragmentAsync(generatedLink.hash); - expect(generatedLink.codec).toBe("arx3"); - expect(generatedLink.hash.startsWith(`#${compactTagForCodec("arx3")}`)).toBe(true); - expect(generatedLink.url).toContain(`#${compactTagForCodec("arx3")}`); + expect(generatedLink.codec).toBe("arx4"); + expect(generatedLink.hash.startsWith(`#${compactTagForCodec("arx4")}`)).toBe(true); + expect(generatedLink.url).toContain(`#${compactTagForCodec("arx4")}`); expect(generatedLink.fragmentLength).toBeLessThan(1900); expect(parsed.ok).toBe(true); }); + it("copies the raw unicode fragment in the paste URL instead of a percent-encoded serialization", async () => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + + const draft: LinkCreatorDraft = { + kind: "markdown", + title: "Launch note", + filename: "brief.md", + content: "# Launch note\n\nA short brief with enough text for a packed wire to win.\n", + language: "", + diffView: "unified", + codec: "arx3", + }; + + const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); + const fragment = generatedLink.url.slice(generatedLink.url.indexOf("#") + 1); + + expect(generatedLink.url).not.toMatch(/%[0-9A-F]{2}/); + expect(fragment).toBe(generatedLink.hash.slice(1)); + expect(fragment.length).toBe(generatedLink.fragmentLength); + }); + + it("builds markdown links from an ASCII wire fragment so URL serialization cannot balloon them", async () => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + + const draft: LinkCreatorDraft = { + kind: "markdown", + title: "Launch note", + filename: "brief.md", + content: [ + "# Launch note", + "", + "Share one artifact at a time without uploading it anywhere.", + "", + "- Markdown stays readable", + "- Code keeps its language hint", + "- The link works from a static export", + ].join("\n"), + language: "", + diffView: "unified", + codec: "arx3", + }; + + const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); + + // The paste URL keeps the packed non-ASCII fragment; the markdown URL must carry a + // percent-escape-free ASCII fragment whose payload decodes identically. + const markdownFragment = generatedLink.markdownUrl.slice(generatedLink.markdownUrl.indexOf("#") + 1); + // eslint-disable-next-line no-control-regex + expect(markdownFragment).toMatch(/^[\x21-\x7e]+$/); + expect(markdownFragment).not.toContain("%"); + // The markdown link must beat the percent-encoded serialization of the packed URL, + // which is what a URL serializer would have produced for that surface. + expect(generatedLink.markdownLinkLength).toBeLessThan(new URL(generatedLink.url).toString().length); + + const parsedMarkdown = await decodeFragmentAsync(`#${markdownFragment}`); + const parsedPacked = await decodeFragmentAsync(generatedLink.hash); + expect(parsedMarkdown.ok).toBe(true); + expect(parsedPacked.ok).toBe(true); + if (parsedMarkdown.ok && parsedPacked.ok) { + expect(parsedMarkdown.envelope).toEqual(parsedPacked.envelope); + } + }); + + // Regression: the markdown surface used to fall back to the primary Unicode fragment when its own + // ASCII candidate went over the visible fragment budget, which is exactly when the fallback hurts + // most: the Unicode fragment serializes ~9x larger inside a markdown destination. An over-budget + // link already tells the agent so through the Discord warning; the surface contract still holds. + it("keeps the markdown surface on the ASCII wire even when that candidate is over the fragment budget", async () => { + loadArxDictionarySync(arxDictionaryJson); + loadArx2OverlayDictionarySync(arx2DictionaryJson); + loadArx4PriorsSync(arx4PriorsJson); + + // Incompressible content sized so the dense paste fragment stays inside the budget while the ASCII + // candidate for the same bytes does not. + let digests = ""; + for (let index = 0; digests.length < 20_000; index += 1) { + digests += createHash("sha256").update(String(index)).digest("hex"); + } + + const draft: LinkCreatorDraft = { + kind: "markdown", + title: "Digest wall", + filename: "digests.md", + content: digests.slice(0, 20_000), + language: "", + diffView: "unified", + codec: "arx4", + }; + + const generatedLink = await createGeneratedArtifactLinkAsync(draft, "https://agent-render.com/"); + const pasteFragment = generatedLink.hash.slice(1); + const markdownFragment = generatedLink.markdownUrl.slice(generatedLink.markdownUrl.indexOf("#") + 1); + + expect(markdownFragment).toMatch(/^[\x21-\x7e]+$/); + // The case only bites while the ASCII candidate is over budget, so assert that it is. + expect(getVisibleFragmentLength(markdownFragment)).toBeGreaterThan(MAX_FRAGMENT_LENGTH); + expect(getFragmentTransportLength(markdownFragment)).toBeLessThan(getFragmentTransportLength(pasteFragment)); + expect(generatedLink.discordMarkdownLinkWarning).not.toBeNull(); + + const parsed = await decodeFragmentAsync(`#${markdownFragment}`, { skipFragmentBudget: true }); + expect(parsed.ok).toBe(true); + if (!parsed.ok) return; + expect(parsed.envelope.artifacts[0]).toMatchObject({ content: draft.content }); + }); + it("rejects empty pasted content", () => { expect(() => createGeneratedArtifactLink({ diff --git a/tests/sha256.test.ts b/tests/sha256.test.ts new file mode 100644 index 0000000..c3a905a --- /dev/null +++ b/tests/sha256.test.ts @@ -0,0 +1,26 @@ +import { createHash } from "node:crypto"; +import { describe, expect, it } from "vitest"; +import { sha256Hex } from "@/lib/sha256"; + +// The browser-side digest that gates the arx4 priors asset has to agree with node:crypto exactly, or +// the shipped asset would be rejected in one runtime and accepted in the other. Lengths cover the +// padding boundaries (the block that has no room for the length field, the exact-block case) and the +// 16 KiB scale the priors check runs at. +const LENGTHS = [0, 1, 3, 55, 56, 63, 64, 65, 119, 120, 127, 128, 16 * 1024]; + +describe("sha256Hex", () => { + it.each(LENGTHS)("matches node:crypto for a %i-byte input", (length) => { + const bytes = new Uint8Array(length); + for (let index = 0; index < length; index += 1) { + bytes[index] = (index * 31 + 7) % 256; + } + + expect(sha256Hex(bytes)).toBe(createHash("sha256").update(bytes).digest("hex")); + }); + + it("matches the published digest of \"abc\"", () => { + expect(sha256Hex(new TextEncoder().encode("abc"))).toBe( + "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad", + ); + }); +});