Non-Prefix KV Cache Reuse: Trace Collection and Analysis

A dataset and toolchain for measuring non-prefix KV cache reusability — the kind of token reuse that standard prefix caching misses.

Standard prefix caching only reuses tokens that match from the start of a sequence. In multi-turn RAG, the same supporting documents often reappear across turns but at different positions — retrieval order changes as the query evolves, new documents are added, and the conversation context shifts. These shared tokens are recomputed from scratch by prefix caching. CacheBlend-style systems can recover this waste by matching content-identical chunks regardless of position. This dataset quantifies how much waste there is to recover.

Key Results

110 sessions, 943 requests, 44.3M input tokens across 4 domains from the MT-RAG benchmark.

Domain	Sessions	Requests	Prefix	Non-Prefix	New Compute
clapnq	29	189	36.8%	15.3%	47.9%
cloud	26	251	56.2%	17.1%	26.6%
govt	28	271	54.1%	15.1%	30.8%
fiqa	27	232	80.8%	3.5%	15.7%
Overall	110	943	54.9%	14.2%	31.0%

14.2% of input tokens are reusable but invisible to prefix caching. Recovering these requires content-based chunk matching (CacheBlend). See raw_traces/non_prefix_mtRag/README.md for the full analysis, document-size threshold derivation, and workload comparison.

Repository Layout

NonPrefix_LMCacheDataset/
├── raw_traces/                          # All raw traces (JSONL recordings)
│   ├── README.md
│   ├── non_prefix_mtRag/                #   Main dataset: 110 multi-session MT-RAG traces
│   └── exploratory/                     #   Earlier experiments that motivated the main dataset
│       ├── ClaudeCode/                  #     Claude Code (Anthropic API) traces
│       ├── mtRag_on_OpenClaw/           #     MT-RAG as single long sessions through OpenClaw
│       └── test_OpenClaw/               #     OpenClaw behavior tests (resume, RAG, compaction)
├── html_output/                         # Generated HTML visualizations (gitignored)
│   ├── non_prefix_mtRag/                #   Mirrors raw_traces/non_prefix_mtRag/
│   └── exploratory/                     #   Mirrors raw_traces/exploratory/
│       ├── ClaudeCode/
│       ├── mtRag_on_OpenClaw/
│       └── test_OpenClaw/
├── offline_analysis/                    # Analysis scripts (run against raw_traces, write to html_output)
│   ├── README.md
│   ├── analyze_trace.py                 #   Core analyzer + HTML dashboard generator
│   ├── cacheblend_hashes.py             #   Two-hash engine (rolling prefix + content fingerprint)
│   ├── create_nonprefix_trace.py        #   Rewrite a trace to break prefix-cache reuse (testing)
│   └── trace_viewer.html                #   HTML template for the interactive dashboard
├── mtRag_traces_prompt_building/        # Prompt construction pipeline for the main dataset
│   ├── build_prefix_break_traces.py     #   Build session JSONL from MT-RAG corpus
│   ├── send_to_openclaw.py              #   Send sessions to OpenClaw with proxy management
│   ├── rebuild_capped.py                #   Rebuild with token cap enforcement
│   └── {clapnq,cloud,fiqa,govt}.jsonl   #   Per-domain session prompts
└── proxy/              
                 # Recording proxies (used during collection)
    ├── OpenClaw_proxy.py                #   OpenClaw ↔ vLLM recording proxy
    ├── anthropic_proxy.py               #   Claude Code ↔ Anthropic API proxy
    └── openai_proxy.py                  #   Generic OpenAI-format proxy

Quick Start

Visualize one non_prefix_mtRag trace:

python offline_analysis/analyze_trace.py \
  raw_traces/non_prefix_mtRag/clapnq_clapnq_0208bf26ec357a803445290fa88a2e9e_trace.jsonl \
  --format openclaw \
  --html html_output/non_prefix_mtRag/clapnq_clapnq_0208bf26ec357a803445290fa88a2e9e_trace.html \
  --include-raw-text

Then open html_output/non_prefix_mtRag/clapnq_clapnq_0208bf26ec357a803445290fa88a2e9e_trace.html.

Visualize a Claude Code trace (Anthropic format, default; raw text embedded by default):

python offline_analysis/analyze_trace.py \
  raw_traces/exploratory/ClaudeCode/testing_compact_session_trace.jsonl \
  --html html_output/exploratory/ClaudeCode/testing_compact_session.html

See offline_analysis/README.md for all flags and the analysis methodology.

How the Dataset Was Built

1. Prompt construction (mtRag_traces_prompt_building/). MT-RAG questions are grouped into multi-turn sessions. Each turn includes the question's supporting documents drawn in random order — as they would be in a real retrieval system — naturally producing non-prefix overlap between requests.

2. Trace collection (proxy/). Sessions are sent to OpenClaw (or vLLM directly) through a recording proxy that captures raw /v1/chat/completions traffic as JSONL, one file per session.

3. Offline analysis (offline_analysis/). CacheBlend-style chunk matching (tiktoken o200k_base, 256-token chunks) classifies each token as prefix-reusable, non-prefix-reusable, or new compute. Outputs both a JSON analysis and an interactive HTML dashboard.

Earlier Experiments

Before arriving at the multi-session MT-RAG approach, several other workloads were tried. Most produced near-zero non-prefix reuse because they are inherently prefix-append. See raw_traces/README.md for details.

Experiment	Sessions	Non-Prefix	Why low/high
Claude Code with /compact	1	10.5%	Compaction rewrites history, shifting positions
Claude Code re-read after edit	1	2.9%	Mostly prefix-append
MT-RAG single-session (4 domains)	4	0.5–8.5%	History accumulates in order — docs never move
Multi-session MT-RAG (this dataset)	110	14.2%	Shared docs at different positions across turns