feat(proxy): native TOON kernel + off-thread token counting + streaming replay fix - #6502
Draft
arsenyinfo wants to merge 19 commits into
Draft
feat(proxy): native TOON kernel + off-thread token counting + streaming replay fix#6502arsenyinfo wants to merge 19 commits into
arsenyinfo wants to merge 19 commits into
Conversation
Memoize serialized SSE per event at accumulation time in the openai, anthropic, and bedrock stream adapters (2000-fragment replay: 1220ms -> 9.2ms) and document the getRawToolCallEvents contract: full stable history, index-stable, non-destructive. Make the anthropic-openai wrapper conform (append-only instead of splice(0)): previously the handler's index-based dedup dropped every tool-call argument delta on the model-router Anthropic path (streamed AND buffered-approved) - clients got tool names with empty arguments. Exact event-sequence tests pin all four paths x three policy scenarios; native-path bytes are unchanged. The cohere-openai wrapper is documented as known-non-conforming (never translates tool events; pre-existing, tracked separately). Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
proxy-transform-core: batched unwrap -> parse -> TOON-encode kernel
(toon-format 0.5.0, default-features off; serde_json preserve_order).
Positional infallible per-item API; parse failure -> encoded: null.
Ports the unwrap-tool-content semantics exactly (first-text-element
behavior pinned); 120-case golden corpus generated from the crate
(regen via UPDATE_TOON_GOLDENS=1, CI-guarded) plus proptest decode
round-trips with documented upstream-decoder-bug exclusions.
proxy-transform-rs: thin NAPI adapter on the image-rs AsyncTask
pattern (libuv pool, JS-thread input conversion, catch_unwind
firewall, {code,message} error JSON), napi-loader index.cjs, CJS+ESM
smoke tests; check:ci includes clippy -D warnings for both crates.
Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Add routes/proxy/utils/toon-native.ts: one batched call per request
into @archestra/proxy-transform-rs (AsyncTask on the libuv pool),
positional results with a hard length assert, fail-open to
uncompressed on any load/call failure, and an eager startup probe
wired into both web and worker entry modes with an error log and
llm_toon_addon_load_failures_total{context} metric.
Compression skipped due to addon failure is now reported honestly as
toonSkipReason "addon_unavailable" end to end: stats contract,
handler precedence, session-count SQL aggregation, Savings UI branch,
mocks, and regenerated OpenAPI/client types.
The OpenAI adapter keeps its keep/reject and accounting semantics
exactly (tokenizes the unwrapped string, strict fewer-tokens rule,
rejected payloads counted in both totals); TOON bytes now come from
the Rust kernel (toon-format 0.5.0). Tests: 120-case golden gate
(fails in CI without the addon, skips visibly locally), boundary
mocks, exact transformed-request equality incl. interleaved
non-candidate messages, stats matrix, handler-level skip reason;
provider-matrix TOON assertions pass unchanged.
Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
…and Docker turbo build task (cache:false) + @backend#check:ci dependsOn; biome exclusion for the generated index.d.ts; unconditional Tilt prebuild resource; dev-stack.sh addon list; Dockerfile manifest/node_modules/ source copies and musl smoke filter. pnpm deploy output verified to ship index.cjs + index.d.ts + the .node binary with napi-loader resolvable; musl smoke stage itself left to CI. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
anthropic, gemini, bedrock, zhipuai, minimax, cohere now mirror the OpenAI reference: one batched native call per request, positional application via structural locators, fail-open with the addon_unavailable skip reason. Each adapter's accounting semantics are preserved exactly and pinned by per-adapter stats matrices: anthropic per-block counting on shared tool_use_ids, gemini tokenizing the original serialization while parsing unwrapped, bedrock unconditional apply on both branches with content[0] semantics, zhipuai rejected-in-both-totals, minimax unconditional, cohere wins-only. Non-string stringify results are skipped per item so they can never poison the batch. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
BENCH_BACKEND=ts|native selector over the same harness/corpora, a one-off backend output comparator, and a criterion bench in proxy-transform-core. Records the T8 verdict: native FAILS the pre-registered thresholds (1.9x slower at 1KB to 103x at 5MB; peak RSS ~2.4GB vs <705MB cap; only the event-loop-delay guardrail passes). Root cause isolated to the upstream toon-format 0.5.0 encoder: O(k^2) sibling-key scan in write_object_impl plus 36-50MB/s linear paths vs ~80MB/s for the whole TS pipeline; the NAPI boundary itself measured negligible. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
…mentation
The upstream crate's encoder failed the pre-registered gate (O(k^2)
sibling-key scan, 36-50MB/s linear paths). Own encoder + borrowed
JSON DOM (DeserializeSeed over serde_json): byte parity pinned by the
120 goldens and a differential proptest against the crate (now a
dev-dependency, oracle only), 314-344 MiB/s, all six bench corpora at
0.26-0.43x of the TypeScript baseline - the >=50% CPU-reduction
threshold passes everywhere, p99 event-loop delay 13ms vs 95ms
baseline, RSS growth on par with TS.
Hardened for untrusted input after two adversarial review rounds:
per-item output budget max(2x input, 16KiB) + aggregate batch budget
(fail-open to encoded:null - output above 2x bytes can never win the
token comparison), 10MiB per-item input cap bounding parser-DOM
allocation, exact-size pre-write checks bounding capacity overshoot,
O(total fields) reordered-tabular detection, SipHash object index,
runtime fail-closed exponent-format check. Replicates crate quirks
exactly where reachable (saturating as_i64/as_u64 float casts,
[N]{}: arrays, no-exponent floats).
Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Backend no longer depends on the npm TOON implementation: the bench harness is native-only, unwrap-tool-content.ts is deleted (the Rust core owns unwrapping). Docs: provider-authoring guide points at the shared toon-native helper, costs-and-limits documents the addon_unavailable skip reason. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Aligns the sum with the saturating arithmetic already used around it; wrap was unreachable on supported 64-bit targets but inconsistent. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
…ates The Platform Lint job's toolchain does not install the clippy component; clippy already runs workspace-wide in Platform Rust Checks. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Extract the copy-pasted prelude (corpus loading, addon skip/fail gate, deterministic pricing, per-provider token counting) into a shared test harness, and drop the accounting-matrix rows that re-asserted exactly what each suite's full-request equality test already pins. Kept per suite: the full-request and interleaving tests, and the accounting cases a combined request cannot isolate (hadToolResults on uncounted-only input, lone-rejected/lone-larger wasEffective=false, Cohere wins-only, Bedrock branch rules). Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Main's interactions-response changes were generated without the addon_unavailable skip reason added on this branch. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
The TOON keep/reject decision counts tokens with a synchronous WASM tiktoken call on the Node event loop, twice per candidate. Under concurrency that blocks the loop (measured: 0 -> ~260ms p99 event-loop delay at 8-way), spiking every other request's latency, while the encode already runs off-thread. Add cl100k_base counting (tiktoken-rs, byte-identical ranks) to the core, fused into the same off-thread encode pass. The batch-level BeforeSource option selects it: Normalized for most adapters, Raw for Gemini (which tokenizes its pre-unwrap serialization), None to skip (Anthropic/Bedrock keep their own tokenizer). Counts populate beforeTokens/encodedTokens only for encodable items, matching every adapter. Rust-only slice; the TypeScript adapters still pass one arg (optional param) until they cut over. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Cut the five tiktoken-family adapters (openai family, gemini, cohere, zhipuai, minimax) over to the native fused counts: they pass a before-source to toonEncodeToolResults and read beforeTokens/encodedTokens instead of running a synchronous WASM tokenizer per candidate. gemini uses the raw (pre-unwrap) baseline; the rest use normalized. Anthropic and Bedrock keep their own tokenizer (no cl100k Rust equivalent). Switch the JS tiktoken baseline to encode_ordinary so reserved-marker literals (e.g. <|endoftext|>) in tool results count as text instead of throwing, matching the native path; parity is then exact. The wrapper validates the count invariant at the boundary (present iff requested and encodable, else fail open). Measured under 8-way concurrency: event-loop p99 257ms -> 11ms, wall 6.3x faster; +~30MB RSS for the Rust cl100k table. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Review gate: the adversarial strings (unicode, whitespace, reserved-token literals) were passed as bare non-JSON, so they hit encoded=null and only asserted null counts — never comparing native to JS. Wrap them in encodable JSON so the count comparison actually runs on them, and assert it covered at least the adversarial set. The reserved-marker case now doubles as the encode_ordinary regression (counting it would throw under the old encode()). Also disclose the per-item batching in the concurrency bench (mirrors bench-concurrency.ts): wall/RSS are directional, event-loop delay faithful. Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
Step 4 said to always count with the provider tokenizer; cl100k providers now pass a before-source and read counts back from toonEncodeToolResults(). Claude-Session: https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu
arsenyinfo
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the LLM proxy's TOON tool-result compression kernel to Rust and fixes an O(k²) + event-loss defect in the streaming tool-call replay path.
Streaming replay fix
getRawToolCallEventscontract documented (full stable history, index-stable, non-destructive); openai/anthropic/bedrock adapters memoize serialized SSE per event at accumulation time. 2000-fragment replay: 1220ms → 9.2ms.anthropic-openaiwrapper drained its buffer (splice(0)), so the handler's index dedup dropped every tool-call argument event on the model-router Anthropic path (both streamed and buffered-approved). Native-path SSE is byte-identical (pinned by exact event-sequence tests, 4 paths × 3 policy scenarios).Native TOON kernel
proxy-transform-core(pure Rust) +proxy-transform-rs(NAPI AsyncTask, catch_unwind firewall). One batched call per request; all 7 adapters cut over with per-adapter keep/reject accounting preserved exactly (pinned by per-adapter stats-matrix tests).toon-formatcrate's encoder failed the pre-registered perf gate (O(k²) sibling-key scan; 36–50MB/s linear paths). It is now a dev-dependency oracle only; the crate ships an own linear encoder + borrowed JSON DOM parser, byte parity pinned by 120 committed goldens (regression pins) and a differential proptest against the crate encoder (the independence oracle).toonSkipReason: "addon_unavailable"end to end (stats contract → session SQL → Savings UI → OpenAPI/client), never fabricated asnot_effective.Off-thread token counting
The keep/reject decision counted tokens with a synchronous WASM
tiktokencall on the event loop, twice per candidate. Under concurrency that blocked the loop and spiked every other request's latency, while the encode already ran off-thread. The five tiktoken-family adapters (OpenAI family, Gemini, Cohere, ZhipuAI, MiniMax) now pass a before-source totoonEncodeToolResults()and read the cl100k counts back from the same off-thread call (tiktoken-rs, byte-identical ranks); Anthropic and Bedrock keep their own tokenizer. The JStiktokenbaseline switched toencode_ordinaryso reserved-marker literals (e.g.<|endoftext|>) in tool results count as text instead of throwing — byte-identical on all other content, parity pinned by a native↔JS differential test over the 120 goldens + adversarial unicode.Measured under 8-way concurrency (per-item modeling, so wall/RSS are directional; event-loop delay is faithful):
Event-loop p99 257ms → 11ms (~23×), wall 6.3× (counting parallelizes across libuv, no pool saturation). RSS is +~30MB for the resident Rust cl100k table — justified on tail-latency, not memory.
Benchmarks (Apple M4 Pro, 3 runs, same harness/boundary/corpora; baseline at 1e37255)
Pre-registered gate: ≥50% CPU reduction — passes everywhere. p99 event-loop delay 13ms vs 95ms baseline (work moved off-thread). RSS growth under 8-way concurrency: 150–188MB vs TS 159–165MB (absolute peak numbers are confounded by the bench loading the backend module graph; growth is the honest comparison).
Deliberate observable changes
@toon-format/toon2.1.0 to the Rust v3 implementation: hyphen-containing strings quoted, big integers preserved exactly (npm corrupts >2^53 through f64), no exponent notation, document key order. Keep/reject flips measured at 2/57 organic fixtures, all in the conservative direction (original kept).addon_unavailableskip reason (new enum value in stats/API/UI).Notes for reviewers
4223cca0fmixes the encoder introduction with npm-reference deletions (both sides of one swap; builds standalone).platform-deployment.mdaudited, no change needed (addon packaging is generic).openai-responses-from-chatregenerates events losing argument updates on the /responses path; 4 upstream toon-format decoder bugs to report.https://claude.ai/code/session_01QkeLJprfeq75d2NY39aFiu