feat(proxy): native TOON kernel + off-thread token counting + streaming replay fix

docs/pages/platform-adding-llm-providers.md

@@ -3,7 +3,7 @@ title: Adding LLM Providers
 category: Development
 order: 2
 description: Developer guide for implementing new LLM provider support in Archestra Platform
-lastUpdated: 2026-04-29
+lastUpdated: 2026-07-11
 ---
 
 <!-- Renaming/deleting this file? Add a redirect in docs/redirects.json. -->
@@ -171,10 +171,12 @@ The function must:
 
 1. Iterate through provider-specific message array structure
 2. Find tool result messages (e.g., `role: "tool"` in OpenAI, `tool_result` blocks in Anthropic, `functionResponse` parts in Gemini)
-3. Parse JSON content and convert to TOON format using `@toon-format/toon`
-4. Calculate token savings using the appropriate tokenizer
+3. Batch the extracted results through `toonEncodeToolResults()` from `backend/src/routes/proxy/utils/toon-native.ts`. The native addon unwraps, parses, and encodes them — never encode TOON in the adapter
+4. Get token counts to compare. For cl100k providers, pass a before-source to `toonEncodeToolResults()` and read the counts it returns; Anthropic and Bedrock count with their own tokenizer. Keep the TOON version only when it saves tokens
 5. Return compressed messages and compression statistics
 
+`toonEncodeToolResults()` returns `null` when the native addon is unavailable. The adapter must then keep the original results and report the `addon_unavailable` skip reason.
+
 ### Metrics
 
 > **Note:** This is a known abstraction leak that we're planning to address in future versions. Thanks for bearing with us!

docs/pages/platform-costs-and-limits.md

@@ -2,7 +2,7 @@
 title: Costs & Limits
 category: LLM Proxy
 order: 4
-lastUpdated: 2026-06-22
+lastUpdated: 2026-07-10
 ---
 
 <!-- Renaming/deleting this file? Add a redirect in docs/redirects.json. -->
@@ -99,6 +99,7 @@ Compression is skipped when:
 - TOON is disabled
 - a response has no tool results
 - the TOON version would not save tokens
+- the compression engine is unavailable (reported as `addon_unavailable`)
 
 Archestra records before/after token counts and savings when compression is applied, so those savings appear in logs and aggregate cost reporting.
 

platform/.gitignore

@@ -75,6 +75,7 @@ benchmark-config.env
 archestra-rs/sandbox-rs/index.js
 archestra-rs/app-runtime-rs/index.js
 archestra-rs/image-rs/index.js
+archestra-rs/proxy-transform-rs/index.js
 
 # Playwright
 /playwright/

platform/Dockerfile

@@ -154,6 +154,7 @@ COPY archestra-rs/napi-loader/package.json archestra-rs/napi-loader/
 COPY archestra-rs/sandbox-rs/package.json archestra-rs/sandbox-rs/
 COPY archestra-rs/app-runtime-rs/package.json archestra-rs/app-runtime-rs/
 COPY archestra-rs/image-rs/package.json archestra-rs/image-rs/
+COPY archestra-rs/proxy-transform-rs/package.json archestra-rs/proxy-transform-rs/
 
 # Install all dependencies
 RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install --frozen-lockfile
@@ -195,6 +196,7 @@ COPY --from=deps /app/shared/node_modules ./shared/node_modules
 COPY --from=deps /app/archestra-rs/sandbox-rs/node_modules ./archestra-rs/sandbox-rs/node_modules
 COPY --from=deps /app/archestra-rs/app-runtime-rs/node_modules ./archestra-rs/app-runtime-rs/node_modules
 COPY --from=deps /app/archestra-rs/image-rs/node_modules ./archestra-rs/image-rs/node_modules
+COPY --from=deps /app/archestra-rs/proxy-transform-rs/node_modules ./archestra-rs/proxy-transform-rs/node_modules
 
 # Copy source files
 COPY package.json pnpm-lock.yaml pnpm-workspace.yaml turbo.json ./
@@ -209,6 +211,8 @@ COPY archestra-rs/app-runtime-core ./archestra-rs/app-runtime-core
 COPY archestra-rs/app-runtime-rs ./archestra-rs/app-runtime-rs
 COPY archestra-rs/image-core ./archestra-rs/image-core
 COPY archestra-rs/image-rs ./archestra-rs/image-rs
+COPY archestra-rs/proxy-transform-core ./archestra-rs/proxy-transform-core
+COPY archestra-rs/proxy-transform-rs ./archestra-rs/proxy-transform-rs
 
 # Build all workspace
 ENV NEXT_TELEMETRY_DISABLED=1
@@ -224,7 +228,7 @@ FROM builder-base AS rust-napi-musl-smoke
 RUN --mount=type=cache,target=/root/.cargo/registry \
     --mount=type=cache,target=/root/.cargo/git \
     --mount=type=cache,target=/app/archestra-rs/target \
-    pnpm --filter @archestra/sandbox-rs --filter @archestra/app-runtime-rs --filter @archestra/image-rs check:musl
+    pnpm --filter @archestra/sandbox-rs --filter @archestra/app-runtime-rs --filter @archestra/image-rs --filter @archestra/proxy-transform-rs check:musl
 
 # <----- Builder stage ----->
 FROM builder-base AS builder
@@ -259,7 +263,7 @@ RUN --mount=type=cache,target=/root/.cargo/registry \
 # Assemble the production backend with `pnpm deploy` — pnpm's recommended
 # monorepo packaging step (https://pnpm.io/cli/deploy). It produces a
 # self-contained /prod/backend whose node_modules includes every workspace
-# dependency (the three NAPI crates, @archestra/napi-loader, @archestra/shared)
+# dependency (the four NAPI crates, @archestra/napi-loader, @archestra/shared)
 # alongside their transitive production deps, with each crate able to resolve
 # @archestra/napi-loader from a nested store. This replaces the old hand-rolled
 # COPY of each crate's *.node / index.cjs into a workspace-shaped, root-hoisted
@@ -435,7 +439,7 @@ RUN rm -rf /root/.cache/node/corepack /usr/local/bin/pnpm /usr/local/bin/pnpx
 # Copy the fully-assembled backend from the builder's `pnpm deploy` output.
 # /prod/backend contains backend/dist, scripts, drizzle.config.ts and the
 # migrations (scoped by the backend package's `files` field) plus a
-# self-contained node_modules — including the three NAPI crates with their
+# self-contained node_modules — including the four NAPI crates with their
 # compiled *.node addons and a resolvable @archestra/napi-loader. This single
 # tree replaces the previous per-crate .node/index.cjs copy choreography and
 # the runtime `pnpm install`.