ARX4: context-mixing codec, 10% smaller fragments, markdown-link fix

.github/workflows/ci.yml

@@ -21,7 +21,7 @@ jobs:
       - name: Setup Node.js
         uses: actions/setup-node@v6
         with:
-          node-version: 20
+          node-version: 24
           cache: npm
 
       - name: Install dependencies

.github/workflows/test.yml

@@ -16,14 +16,14 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v6
         with:
-          node-version: 20
+          node-version: 24
           cache: npm
 
       - name: Install dependencies
         run: npm ci
 
       - name: Install Playwright browsers
-        run: npx playwright install --with-deps chromium
+        run: npx playwright install --with-deps chromium webkit
 
       - name: Run checks and browser tests
         env:

AGENTS.md

@@ -23,10 +23,10 @@ Core product traits right now:
 Treat these as core constraints unless the owner explicitly changes the product direction.
 
 - The app is a single exported client-side shell, not a backend product.
-- Artifact payloads live in the URL fragment, using the compact `#<tag><payload>` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3. Legacy `#agent-render=v1.<codec>.<payload>` links (arx-family carry an extra `<dictVersion>.` segment) still decode but are no longer emitted.
+- Artifact payloads live in the URL fragment, using the compact `#<tag><payload>` form where the single tag char identifies the codec: `p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4. Legacy `#agent-render=v1.<codec>.<payload>` links (arx-family carry an extra `<dictVersion>.` segment) still decode but are no longer emitted.
 - The deployed host should not receive artifact contents as part of the initial page request.
 - Supported artifact kinds are `markdown`, `code`, `diff`, `csv`, and `json`.
-- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`.
+- Supported codecs are `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`.
 - The product is zero-retention by host design, not secret-safe in an absolute sense.
 - Links may still leak through browser history, copied URLs, screenshots, and any future client-side analytics.
 
@@ -84,9 +84,10 @@ The fragment transport is part of the product surface, not an implementation det
 
 Current rules:
 - fragment key: `agent-render` (legacy decode path only; the compact form has no key)
-- emitted format: compact `#<tag><payload>`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one)
+- emitted format: compact `#<tag><payload>`, where the single tag char identifies the codec (`p` plain, `l` lz, `d` deflate, `a` arx, `b` arx2, `c` arx3, `e` arx4); the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and refuses to decode a newer one)
 - legacy format (still decodable, no longer emitted): `agent-render=v1.<codec>.<payload>` for `plain|lz|deflate`, `agent-render=v1.arx.<dictVersion>.<payload>` for `arx`, `agent-render=v1.arx2.<dictVersion>.<payload>` for `arx2`, and `agent-render=v1.arx3.<dictVersion>.<payload>` for `arx3`
-- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`
+- codecs: `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`
+- `arx4` payloads carry one extra leading char after the tag, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the context mixer ran before the payload; `m`/`c`/`j` additionally need `/arx4-priors.json`, and an encoder that cannot load it emits `s` instead
 - fragment size budget: `8192` characters
 - decoded payload budget: `200000` characters
 - Discord markdown link limit: `2000` characters for the full formatted `[label](url)` string; `createGeneratedArtifactLink*` returns `discordMarkdownLinkWarning` when exceeded
@@ -123,6 +124,8 @@ If you change the payload contract, update the code, docs, examples, and the Ope
 - `src/lib/payload/schema.ts` - type surface, limits, fragment key, supported kinds/codecs
 - `src/lib/payload/fragment.ts` - encode/decode logic and transport behavior
 - `src/lib/payload/arx-codec.ts` - arx/arx2/arx3 codecs: dictionary substitution, tuple overlay, brotli, base76/base1k/baseBMP/base64url encoding
+- `src/lib/payload/arx4-codec.ts` - arx4 codec: the arx3 stages with brotli replaced by a deterministic integer context mixer, plus the curated priors it primes on
+- `public/arx4-priors.json` - curated per-kind arx4 priming corpora (and `.br` pre-compressed variant), regenerated by `scripts/build-arx4-priors.mjs`
 - `public/arx-dictionary.json` - shared substitution dictionary for the arx codec (served as a static endpoint)
 - `public/arx-dictionary.json.br` - pre-compressed brotli variant of the dictionary
 - `public/arx2-dictionary.json` - overlay substitution dictionary for arx2 tuple-envelope transport

README.md

@@ -29,8 +29,8 @@ Built for the OpenClaw ecosystem, `agent-render` focuses on fragment-based shari
 ## Status
 
 - Markdown, code, diff, CSV, and JSON all render in the static shell
-- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, and `arx3`, with automatic shortest-fragment selection across available wire formats
-- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx3` reuses those proven bytes and optimizes for compact visible Unicode fragments
+- Fragment transport supports `plain`, `lz`, `deflate`, `arx`, `arx2`, `arx3`, and `arx4`, with automatic shortest-fragment selection across available wire formats
+- The `arx` substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the `arx2` tuple-envelope overlay is served at `/arx2-dictionary.json` with a pre-compressed `/arx2-dictionary.json.br` variant; `arx3` reuses those proven bytes and optimizes for compact visible Unicode fragments; `arx4` adds the curated context-mixer priors at `/arx4-priors.json` with a pre-compressed `/arx4-priors.json.br` variant
 - The viewer toolbar copies artifact bodies to the clipboard, downloads them as files, and (for markdown) supports browser print-to-PDF
 - Deployment target: static hosting, including Cloudflare Pages
 

docs/architecture.md

@@ -99,8 +99,9 @@ The fragment protocol keeps the JSON envelope stable and treats compression stri
 - `arx` applies domain-dictionary substitution, brotli compression (quality 11), and binary-to-text encoding for best-in-class compression. Four wire shapes are candidates: base76 (ASCII, 77 fragment-safe chars), base64url (RFC 4648 `A-Za-z0-9-_` with a `B.` prefix for detection), base1k (Unicode, 1774 chars from U+00A1–U+07FF), and baseBMP (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). The async encoder tries all four and picks the shortest **transport** length (percent-encoded UTF-8 length for non-ASCII), so base64url can win over Unicode encodings on chat-style surfaces. baseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes, achieving ~70% smaller fragments than deflate on typical payloads (~6.1x compression ratio for 8k markdown). Full pipeline timing is on the order of ~8–14ms for 8k payloads depending on the wire encoding. The substitution dictionary is served as a static file at `/arx-dictionary.json` so agents can fetch it for local compression; a pre-compressed `/arx-dictionary.json.br` variant is also available. The viewer tries the pre-compressed dictionary first on default ARX-family loads, falls back to the JSON file, and only loads external dictionaries when an ARX/ARX2/ARX3 encode or decode path needs them.
 - `arx2` keeps the arx compression stack but replaces the JSON envelope with a compact tuple envelope and applies `/arx2-dictionary.json` as an overlay before the shared arx dictionary. The viewer tries `/arx2-dictionary.json.br` first for default overlay loads and falls back to JSON. It is emitted with the compact `b` tag (which identifies the codec but does not carry a dictionary version — it implies the current pinned dictionary) and decodes back to the standard envelope before validation/rendering.
 - `arx3` uses the same tuple envelope, overlay dictionary, shared arx dictionary, and brotli bytes as arx2, then allows the dense baseBMP wire to win by decoded visible character length. This deliberately optimizes copyable visible URL length for trusted Unicode-preserving surfaces; it is not a stronger compressed-byte format than arx2.
+- `arx4` keeps the arx3 tuple envelope, overlay dictionary, shared arx dictionary and baseBMP selection rule, and replaces brotli with a deterministic integer context mixer (`arx4-codec.ts`). It is emitted with the compact `e` tag and carries one extra leading payload char, the prior id, naming the priming corpus both sides run before the payload. See `docs/payload-format.md` for the prior ids and the `/arx4-priors.json` asset.
 - packed wire mode (`p: 1`) shortens transport keys before compression, then unpacks back to the standard envelope during decode
-- automatic async codec selection tries `arx3 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx3 use tuple envelopes
+- automatic async codec selection tries `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain`; arx compares packed + non-packed candidates, while arx2/arx3/arx4 use tuple envelopes
 - sync codec selection (used by examples and legacy paths) tries `deflate -> lz -> plain`
 - decode enforces both visible fragment length and decoded payload size ceilings before UI rendering; arx/arx2/arx3 Brotli decompression uses a streaming output cap before final JSON or tuple parsing
 - invalid bundle state is normalized or rejected before renderers mount

docs/payload-format.md

@@ -15,6 +15,7 @@ Payload contents are untrusted user content. Viewers, agents, and automations sh
 #a<payload>   (arx)
 #b<payload>   (arx2)
 #c<payload>   (arx3)
+#e<payload>   (arx4)
 ```
 
 The compact fragment is a single codec tag char followed by the payload. The tag encodes the codec so unsupported formats fail cleanly; the compact tag does not carry a dictionary version — arx-family tags imply the build's current dictionary (the build pins the newest supported version and rejects a newer one). The legacy `#agent-render=v1.<codec>.<payload>` form (arx-family carry an extra `<dictVersion>.` segment) still decodes for back-compatibility but is no longer emitted. Fragment URLs can look long because they carry the artifact payload in the browser-only fragment instead of sending it to the host during the page request.
@@ -27,6 +28,7 @@ Supported codecs:
 - `arx` - domain-dictionary substitution + brotli (quality 11) + binary-to-text encoding. The compact `a` tag identifies the arx codec but does not carry a dictionary version — it implies the build's current pinned dictionary (the build refuses to decode a forward-incompatible newer dictionary). Four wire shapes are tried and the shortest **transport** size wins (see `computeTransportLength` in `fragment.ts` — non-ASCII Unicode may count longer after percent-encoding): **base76** (ASCII-only, 77 fragment-safe chars), **base64url** (standard RFC 4648 alphabet `A-Za-z0-9-_`, no padding, prefixed with `B.` for detection), **base1k** (Unicode, 1774 chars from U+00A1–U+07FF), and **baseBMP** (high-density Unicode, ~62k safe BMP code points from U+00A1–U+FFEF, ~15.92 bits/char). BaseBMP produces ~32% fewer characters than base1k and ~60% fewer than base76 for the same compressed bytes. BaseBMP payloads are prefixed with a U+FFF0 marker for detection. The viewer’s `arxDecompress` auto-detects the wire shape (including the rare case where a base76 length prefix is also `B.` — it tries base64url first and falls back to base76 if Brotli fails). The substitution dictionary is served at `/arx-dictionary.json` with a pre-compressed `/arx-dictionary.json.br` variant; the viewer tries the `.br` file first on default loads and falls back to JSON. The arx2 overlay dictionary follows the same `.br`-then-JSON default load pattern.
 - `arx2` - tuple-envelope transport + arx2 overlay substitution + the shared arx dictionary + brotli (quality 11) + the same four binary-to-text wire shapes. The compact `b` tag identifies arx2 but does not carry a dictionary version — it implies the current pinned shared arx dictionary and arx2 overlay. Existing `arx` links remain valid; async auto-selection keeps arx2 as the conservative transport-measured tuple codec.
 - `arx3` - the same tuple envelope, overlay substitution, shared arx dictionary, and brotli bytes as arx2, with a different selection rule: baseBMP may win by decoded visible character length instead of conservative percent-encoded transport length. This is the compact visible URL mode for trusted surfaces that preserve Unicode fragments. If a platform rewrites, truncates, or previews links aggressively, prefer arx2/base64url or UUID mode instead.
+- `arx4` - the arx3 tuple/overlay/dictionary stages and the arx3 baseBMP selection rule, with brotli replaced by a deterministic integer context mixer (`arx4-codec.ts`). The payload carries one extra leading char, the prior id (`m`, `c`, `j`, `s`, or `n`), naming the priming corpus the coder ran before the payload; an unrecognized prior id is a decode error. Every prior starts from the pinned dictionary slot text, so the `e` tag implies that dictionary twice over; the `m`, `c` and `j` priors append a curated per-kind corpus from `/arx4-priors.json` (pre-compressed `/arx4-priors.json.br` tried first), which the viewer fetches lazily on the first arx4 encode or decode. The `s` id is that shared dictionary text alone, and `n` runs the mixer cold with no prior at all. If the asset is unavailable the encoder falls back to the `s` prior and emits an `s` id, so link creation never blocks; decoding a fragment that names a curated prior fails cleanly instead of coding against a different corpus than the id names. It codes ~10% smaller than arx3 on the sample corpus and is roughly 100x slower, which is why the whole arx family is async-only.
 
 The encoder now also supports a packed wire representation (`p: 1`) that shortens key names before compression. Packed mode is transport-only; decoded envelopes normalize back to the standard shape.
 
@@ -106,7 +108,7 @@ Tuple fields:
 - Larger payloads should fail with a clear error before rendering
 - Compression is selected automatically across packed/non-packed candidates; arx and arx2 optimize conservative transport length, while arx3 optimizes compact visible length for its dense Unicode wire
 - Default sync codec priority is `deflate -> lz -> plain`
-- Default async codec priority is `arx3 -> arx2 -> arx -> deflate -> lz -> plain`
+- Default async codec priority is `arx4 -> arx3 -> arx2 -> arx -> deflate -> lz -> plain`
 - Optional budget-aware encoding can target strict limits and returns the shortest fragment when none fit
 - `createGeneratedArtifactLink` / `createGeneratedArtifactLinkAsync` return `url`, `markdownLink` (ready to paste verbatim in chat), `markdownLinkLength`, and `discordMarkdownLinkWarning` so agents do not need to reconstruct `[label](url)` themselves
 

docs/url-fragments.md

@@ -22,9 +22,10 @@ The tag char identifies the codec:
 #a<payload>   (arx)
 #b<payload>   (arx2)
 #c<payload>   (arx3)
+#e<payload>   (arx4)
 ```
 
-For `arx`, `arx2`, and `arx3`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version.
+For `arx`, `arx2`, `arx3`, and `arx4`, the compact tag does not carry a dictionary version — it implies the current dictionary (the build pins the newest supported version and rejects a newer one). Only the legacy header below carries an explicit dictionary version.
 
 Older links may use the legacy shape, which the viewer still decodes:
 

package.json

@@ -32,7 +32,7 @@
     "test:e2e": "env -u NO_COLOR playwright test",
     "test:e2e:update": "env -u NO_COLOR playwright test --update-snapshots",
     "test:ci": "npm run lint && npm run test && npm run typecheck && npm run test:e2e && npm run check:build-budgets",
-    "test:browsers": "playwright install chromium",
+    "test:browsers": "playwright install chromium webkit",
     "bench:codecs": "node scripts/bench-codecs.mjs",
     "bench:codecs:update": "node scripts/bench-codecs.mjs --write-baseline",
     "assets:compress": "node scripts/compress-dictionary.mjs",
@@ -82,6 +82,7 @@
     "@types/papaparse": "^5.3.16",
     "@types/react": "^19.0.10",
     "@types/react-dom": "^19.0.4",
+    "esbuild": "^0.27.3",
     "eslint": "^9.22.0",
     "eslint-config-next": "15.5.18",
     "jsdom": "^28.1.0",

playwright.config.ts

@@ -35,5 +35,12 @@ export default defineConfig({
       name: "chromium",
       use: { ...devices["Desktop Chrome"] },
     },
+    {
+      // Second engine for the arx4 wire-format gate: the codec has to code bit-identically here.
+      // Visual baselines are chromium-only, so the screenshot spec stays on its single engine.
+      name: "webkit",
+      testIgnore: ["**/visual.spec.ts"],
+      use: { ...devices["Desktop Safari"] },
+    },
   ],
 });

public/_headers

@@ -15,6 +15,11 @@
   Content-Encoding: br
   Vary: Accept-Encoding
 
+/arx4-priors.json.br
+  Content-Type: application/json; charset=utf-8
+  Content-Encoding: br
+  Vary: Accept-Encoding
+
 /vendor/diff-view-pure.css.br
   Content-Type: text/css; charset=utf-8
   Content-Encoding: br