KV-Cache eviction prioritization API

[hyeongyun0916]

Purpose

This PR adds a minimal KV-cache eviction prioritization API to vLLM's chat completion endpoint, to support multi-turn agentic workloads where downstream orchestrators (NVIDIA Dynamo, llm-d) need explicit per-request retention hints.

The directive shape mirrors TensorRT-LLM's KvCacheRetentionConfig.TokenRangeRetentionConfig — same primitives: token range [start, end), priority (0-100), optional duration.
A single orchestrator can target both backends with the same payload semantics; only unit conversion (ms ↔ seconds) and one extra field (our retention_scope) differ.

The Dynamo blog uses example syntax matching this directly:

"system prompt blocks are evicted last (priority: 100); conversation context survives a 30-second tool call (duration: 45s); decode tokens are first to go (priority: 1)"

Implements RFC vllm-project#37003 ([RFC]: Context-Aware KV-Cache Retention API (Prioritized Evictions)).
This supersedes the existing draft PR vllm-project#38514 with the same public API, restructured to keep KVCacheBlock and Request completely untouched.
PR vllm-project#38514 will be closed once this lands; RFC vllm-project#37003 remains open for ongoing design discussion.

Public API

Two optional fields on ChatCompletionRequest:

retention_directives: list[dict] | None
# Each directive: {start: int, end: int|None, priority: int (0-100),
#                  duration: float|None}

retention_scope: str | None
# Opaque ownership identifier (e.g., session ID). Used to enforce
# directive-author ownership rules.

Forwarded to SamplingParams.extra_args and consumed by block_pool at the cache-full transition.
Validator enforces that priorities are non-increasing across rising token positions (prefix-cache constraint).

Behavior

• Zero-overhead fast path: when no request has retention directives, get_new_blocks and get_num_free_blocks are bit-for-bit equivalent to the existing LRU path — a single integer comparison guards the branch.
• Eviction order under pressure: unprioritized blocks drain from the LRU free list first; only after that, the lowest-priority blocks are popped from the priority queue.
• Ownership: any scope can escalate priority; only the owning scope can downgrade or clear.
  Anonymous (scope=None) callers cannot clear another scope's directive.
• Expiry: duration translates into a monotonic expiry timestamp; expired entries are silently treated as unprioritized.

What's changed

The change is purely additive at the API surface and at the data model:

• KVCacheBlock is not modified (0 changes to kv_cache_utils.py)
• Request is not modified (0 changes to request.py)
• All retention state lives in a new self-contained module vllm/v1/core/priority_eviction_queue.py
• block_pool.py integrates the new module via 8 single-line hook calls (+71 lines total)

Two structural-invariant regression tests are included so the "no core changes" property is automatically enforced against future commits.

benchmarks/benchmark_retention_eviction.py            (new)  +233
tests/tool_use/test_chat_completion_request_validations.py   +85
tests/v1/core/test_priority_eviction.py               (new) +558
vllm/entrypoints/openai/chat_completion/protocol.py         +53
vllm/v1/core/block_pool.py                                  +71 / −7
vllm/v1/core/priority_eviction_queue.py               (new) +160

Compared to vllm-project#38514:

Metric	vllm-project#38514	This PR
Production lines added	+380	+284
Core files modified	4	2
block_pool.py inline	+174	+71
KVCacheBlock changes	+17	0
Request changes	+10	0

References

• TensorRT-LLM KvCacheRetentionConfig API: https://nvidia.github.io/TensorRT-LLM/latest/features/kvcache.html
• NVIDIA Dynamo, full-stack agentic inference optimizations: https://developer.nvidia.com/blog/full-stack-optimizations-for-agentic-inference-with-nvidia-dynamo/
• Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live — https://arxiv.org/abs/2511.02230
• Implements RFC [RFC]: Context-Aware KV-Cache Retention API (Prioritized Evictions) vllm-project/vllm#37003. Supersedes draft PR [Feature] Context-Aware KV-Cache Retention API (#37003) vllm-project/vllm#38514 (same public API, minimized surface).

Test Plan

# Retention unit + integration tests
pytest tests/v1/core/test_priority_eviction.py

# Chat completion validator tests (retention fields + monotonic validator)
pytest tests/tool_use/test_chat_completion_request_validations.py

# End-to-end benchmark (no GPU required, runs against real BlockPool)
python benchmarks/benchmark_retention_eviction.py \
  --num-gpu-blocks 64 --num-sessions 4 --blocks-per-session 4

# Pre-commit on touched files
pre-commit run --files \
  vllm/v1/core/priority_eviction_queue.py \
  vllm/v1/core/block_pool.py \
  vllm/entrypoints/openai/chat_completion/protocol.py \
  tests/v1/core/test_priority_eviction.py \
  tests/tool_use/test_chat_completion_request_validations.py \
  benchmarks/benchmark_retention_eviction.py

# Strict mypy
pre-commit run mypy-3.10 --files \
  vllm/v1/core/priority_eviction_queue.py \
  vllm/v1/core/block_pool.py \
  vllm/entrypoints/openai/chat_completion/protocol.py \
  --hook-stage manual

# Broader regression
pytest tests/v1/core/

Test Result

• tests/v1/core/test_priority_eviction.py — 40/40 pass
  • TestPriorityEvictionQueue (10): heap operations, lazy delete, TTL
  • TestApplyDirectives (11): overlap matching, ownership rules
  • TestSidecarLifecycle (6): cleanup invariants
  • TestBlockPoolPriorityEviction (7): integration tests
  • TestStructuralInvariants (2): KVCacheBlock / Request untouched
• tests/tool_use/test_chat_completion_request_validations.py — 10/10 pass (3 pre-existing + 7 new for retention)
• benchmarks/benchmark_retention_eviction.py — end-to-end PASS (prioritized blocks survive; unprioritized blocks evicted)
• pre-commit run --files <all touched> — clean
• pre-commit run mypy-3.10 --hook-stage manual — clean
• Broader pytest tests/v1/core/ — 305 pass; remaining 32 fail / 2 error reproduce identically on the PR base (verified in a separate worktree).
  Unrelated to this PR — test_scheduler.py / test_scheduler_e2e.py EC-connector and multimodal tests that depend on a model artifact not available in the CI environment.

Performance benchmark numbers will be added separately (re-running to ensure reproducibility).

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR — see Purpose section above. Implements RFC [RFC]: Context-Aware KV-Cache Retention API (Prioritized Evictions) vllm-project/vllm#37003, supersedes draft PR [Feature] Context-Aware KV-Cache Retention API (#37003) vllm-project/vllm#38514.
• The test plan — see Test Plan section above.
• The test results — see Test Result section above.
  Performance numbers are being re-measured and will be added in a follow-up.
• (Optional) The necessary documentation update — N/A (no new model or example).