perf: cross-request prefix-KV cache mechanism [wiring + A/B pending] - #101
Open
leyten wants to merge 1 commit into
Open
perf: cross-request prefix-KV cache mechanism [wiring + A/B pending]#101leyten wants to merge 1 commit into
leyten wants to merge 1 commit into
Conversation
In static-KV mode the stage's reset() is a logical no-op — the KV buffer keeps what was written and reads are :total-bounded — so a follow-up turn on the same ring can reuse the resident KV of the shared prompt prefix instead of re-prefilling it. Add a cached_prefix arg to coordinate_pipe: skip the verify frames for [0:N], prefill only the suffix at start=N, decode as usual. Token-exact by construction — greedy verify commits the same tokens whether [0:N] was just written or reused, given those rows hold the same prefix. Proven on the fake ring: run_coordinator_twoturn drives two jobs on ONE persistent ring (turn 1 warms KV, turn 2 reuses it). The ring's KV-GAP / KV-CONTENT asserts raise if the cached turn reads an unwritten or clobbered slot, so the green multi-turn test proves the cache reused only legitimately-resident, correct KV; both chunked and whole-suffix prefill pass and turn 2's first verify starts at the cache boundary, not 0. Mechanism only. The gateway session->ring pinning + cache-validity bookkeeping (same ring served the prior turn; rows not evicted) is the ring-gated follow-up — NOT wired into serving. Multi-turn TTFT A/B pending a live ring.
leyten
force-pushed
the
perf/prefix-kv-cache
branch
from
July 22, 2026 21:58
eefb7e8 to
52ea40c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE — mechanism only, wiring + A/B pending. This adds the coordinator-side lever and proves it token-exact on the fake ring; it is NOT wired into the gateway and the multi-turn TTFT win is unmeasured.
What: in static-KV mode reset() keeps KV, so a follow-up turn on the same ring reuses the resident KV of the shared prefix.
coordinate_pipe(cached_prefix=N)skips re-prefilling [0:N] and prefills only the suffix at start=N.Why it is correct: greedy verify commits the same tokens whether [0:N] was just written or reused (given the rows hold the same prefix).
run_coordinator_twoturnruns two jobs on one persistent FakeRing; the ring RAISES (KV-GAP / KV-CONTENT) if the cached turn reads an unwritten or clobbered slot — so a green test proves legitimate reuse, not a skipped assertion. Both chunked + whole-suffix pass; turn 2 first verify starts at the cache boundary.Follow-up (ring-gated): gateway session->ring pinning + cache-validity bookkeeping (same ring served the prior turn; rows not evicted by a batched tenant), EAGLE-context reseed over the cached prefix (speed only), and the on-ring multi-turn TTFT A/B. Task #5.