chore(bench): support block-size and vllm bench load overrides

[GentleCold]

Summary

• Add a benchmark --block-size override and thread it through prepare/load, service startup, vLLM, LMCache, and DaseR runtime config.
• Simplify run_bench.py --backend to canonical rows only: all, baseline, lmcache, daser-prefix, and daser-chunk; comma-separated subsets such as baseline,lmcache,daser-prefix are supported.
• Add --load-generator vllm-bench for synthetic OpenAI-compatible vllm bench serve traffic with controllable input/output length, request rate, concurrency, seed, prefix length, and random range ratio.
• Save detailed vLLM bench outputs for cold/warm phases, compare generated text, and print cold_warm_exact_match_accuracy in the final summary.
• Record backend metric deltas around vLLM bench phases and report token-level backend_server_cache_hit_rate in the same summary field used by the internal loader.
• Reject vllm-bench selections that include daser-chunk because chunk mode uses DaseR-specific endpoints.

Motivation

The IMDB comparison requested with block size 128 could not be represented correctly before this change: vLLM block size, LMCache chunk size, DaseR runtime block tokens, prompt alignment, and capacity sizing were still tied to the default 16-token block in different places. That made block-size-specific benchmark results ambiguous and potentially inconsistent across backends.

We also needed a benchmark path that can drive OpenAI-compatible backends with exact synthetic input lengths. vllm bench serve covers baseline, LMCache, and DaseR prefix through /v1/completions; DaseR chunk remains on the internal loader because it depends on /documents and /infer.

Cache-hit reporting now uses token-level backend counters for the comparison summary: DaseR uses daser_cache_matched_tokens_total / daser_cache_requested_tokens_total; LMCache uses MP server token counters when available and falls back to status prefetch token counters. Request-level and external-prefix counters remain in raw metrics for diagnostics.

Test plan

• Unit tests added / updated
• PYTHONHASHSEED=0 pytest -q -m "not integration" --ignore=tests/integration --ignore=tests/connector/test_daser_connector.py --ignore=tests/connector/test_gds_transfer.py tests/ passes
  • Not run; scoped change was validated with focused benchmark/config tests below.
• Integration tests run, or marked N/A with reason
  • N/A; no connector protocol behavior changed beyond runtime block-size propagation and benchmark orchestration.
• Benchmark / e2e smoke run, or marked N/A with reason
• pre-commit run --all-files passes
  • Not run; commit hook ran ruff check and ruff format, and focused checks below passed.

Validation commands:

• source /data/zwt/vllm/bin/activate && python -m pytest tests/test_config.py tests/unit/test_benchmark_unified_utils.py -q -> 87 passed in 11.10s
• source /data/zwt/vllm/bin/activate && ruff check benchmarks/run_bench.py benchmarks/bench_load.py benchmarks/utils/loadgen.py tests/unit/test_benchmark_unified_utils.py -> All checks passed!
• source /data/zwt/vllm/bin/activate && ruff format --check benchmarks/run_bench.py benchmarks/bench_load.py benchmarks/utils/loadgen.py tests/unit/test_benchmark_unified_utils.py -> 4 files already formatted
• git diff --check
• source /data/zwt/vllm/bin/activate && python benchmarks/run_bench.py --help confirmed backend help now shows --backend {all,baseline,lmcache,daser-prefix,daser-chunk}[,...].
• source /data/zwt/vllm/bin/activate && python - <<'PY' ... confirmed --backend daser-chunk --load-generator vllm-bench raises ValueError before service startup.

Benchmark runs:

• IMDB, 2000 samples, --block-size 128, --gen-max-tokens 1, --max-inflight 16, GPU 2, L1-only no-evict mode.
• LMCache results: cold TTFT mean 340.19 ms, cold total 43.13 s, cold hit rate 0.261; warm TTFT mean 126.19 ms, warm total 16.20 s, warm hit rate 0.739; cold/warm exact match accuracy 1.0.
• DaseR prefix results: cold TTFT mean 253.08 ms, cold total 32.49 s, cold hit rate 0.254; warm TTFT mean 73.07 ms, warm total 9.78 s, warm hit rate 0.958; cold/warm exact match accuracy 1.0.
• Both result configs confirmed block_size=128, num_samples=2000, total_prompt_tokens=1008272, and total_blocks=7549.
• vLLM bench comparison: --backend baseline,lmcache,daser-prefix --load-generator vllm-bench --bench-num-prompts 200 --bench-input-len 8192 --bench-output-len 1 --bench-max-concurrency 16 --bench-request-rate inf --block-size 128, GPU 2.
  • Baseline: 200/200 success, mean TTFT 5170.55 ms, p99 TTFT 5408.25 ms, total 67.17 s.
  • LMCache cold: 200/200 success, mean TTFT 8732.05 ms, p99 TTFT 45208.66 ms, total 111.91 s.
  • LMCache warm: 200/200 success, mean TTFT 484.52 ms, p99 TTFT 613.33 ms, total 6.29 s, cold/warm exact match 196/200 = 0.98.
  • DaseR prefix cold: 200/200 success, mean TTFT 5561.84 ms, p99 TTFT 5804.80 ms, total 71.90 s.
  • DaseR prefix warm: 200/200 success, mean TTFT 539.61 ms, p99 TTFT 856.70 ms, total 6.96 s, cold/warm exact match 199/200 = 0.995.

Checklist

• pre-commit run --all-files passes
• Type hints and docstrings on all new/modified functions
• New features and bug fixes include tests
• No regressions — existing focused tests still pass

[GentleCold]

Update pushed: 5eabf0b perf(bench): reduce prefix benchmark overhead

What changed:

• Batched rolling prefix key generation for prefix lookup/store allocation.
• Deferred cold KV store submission until vLLM reports request completion, so the finished request blocks remain valid while reducing foreground prefill interference.
• Removed foreground store staging synchronization and grouped L1-only store metadata updates.
• Added tests for deferred store lifecycle and grouped L1 store behavior.

Validation:

• python -m pytest tests/connector/test_ipc_client.py tests/server/test_ipc_server.py tests/connector/test_daser_connector.py tests/transfer/test_l1_only_transfer.py tests/retrieval/test_prefix_index.py -q -> 137 passed.
• ruff check ... -> passed.
• ruff format --check ... -> passed.

Current perf finding:

• The no-chunk DaseR prefix path still does not beat LMCache warm in the latest smoke result: LMCache warm TTFT mean ~130.6 ms, DaseR prefix warm TTFT mean ~203.9 ms.
• Debug profiling shows DaseR warm load spends ~21.8 ms per 8192-token request in the server-side pinned-CPU -> GPU staging copy for ~1.208 GB. A direct microbench of the same H2D size on H800 is also ~21.8 ms, so the tens-of-ms cost is the actual transfer cost of the current pinned L1 path, not just scheduler async overhead.
• LMCache server L1 also uses pinned CPU memory, but its retrieve path is native/CUDA-op based and event-driven; DaseR currently copies L1 -> GPU staging and then worker-side staging -> final KV. The next real optimization should reduce that staging path or add a native H2D/scatter path, rather than continuing to tune request scheduling only.