Skip to content

test(benchmark): add chunk reuse accuracy baseline and repair analysis#75

Open
7774596 wants to merge 1 commit into
masterfrom
docs/chunk-cross-attention-optimizations
Open

test(benchmark): add chunk reuse accuracy baseline and repair analysis#75
7774596 wants to merge 1 commit into
masterfrom
docs/chunk-cross-attention-optimizations

Conversation

@7774596

@7774596 7774596 commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add benchmarks/bench_accuracy_baseline.py: measures chunk-mode KV reuse vs full prefill accuracy on LongBench QA (hotpotqa, 2wikimqa, musique) through the production HTTP service (/documents + /infer), with identical greedy decoding in both modes.
  • Scoring methodology: raw QA F1, extracted-answer F1 (bold span / first sentence, separating factual errors from verbosity drift), and gold-answer substring recall; accuracy means are computed only over samples with a TTFT-verified chunk-KV load so silent full-recompute fallbacks cannot dilute the gap.
  • Add docs/optimizations/5_chunk_cross_attention_repair.md: error decomposition (fake attention sinks, missing cross-chunk attention, pad noise, RoPE numerics), hard constraints from the connector interface, ranked repair directions (shared-prefix prefill, composed-prefix partial hits, seam patches, per-chunk correction), and the measured baseline results.
  • Documents two measurement pitfalls with remedies: ring-buffer exhaustion across datasets and generation-cap truncation (the initial 64-token cap truncated 35-69% of reuse answers and conflated cap artifacts with KV damage).

Results (Qwen3-8B, RTX 4090, greedy, max_new_tokens=256)

Dataset N Full F1 Reuse F1 Ext-F1 Gap Rec Gap
HotpotQA 19/20 0.7338 0.1714 +0.4486 +0.2632
2WikimQA 20/20 0.2951 0.0665 +0.2060 +0.0500
MuSiQue 16/16 0.2784 0.0138 +0.2073 +0.1250

Test plan

  • Ran the full three-dataset comparison end to end against the production service stack (vLLM + DaseR, iouring, chunk mode), restarting DaseR with a clean store per dataset
  • ruff check / ruff format --check pass on the benchmark script
  • Verified Loaded column detects and excludes silent full-prefill fallbacks

Measure chunk-mode KV reuse vs full prefill on LongBench QA through the
production HTTP service, scoring raw F1, extracted-answer F1, and gold
recall over samples with a verified chunk-KV load. Document the error
decomposition, hard constraints, ranked repair directions, and corrected
baseline results including the generation-cap truncation pitfall.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant