feat(cache): prioritize reclaiming redundant KV replicas

[GentleCold]

Summary

Related RFC: #398

Reduce redundant resident KV copies in P/D deployments with a two-class replacement policy:

• Locally saved blocks enter retained.
• RDMA/P2P-fetched blocks enter reclaimable.
• P-side finished_sending marks the saved blocks for that request as reclaimable.
• MetaServer Insert returns aligned post-insert owner counts; a newly saved local block is asynchronously marked reclaimable when owners >= 3.
• Reclaim evicts reclaimable blocks before retained blocks, with LRU order inside each class.
• Already-existing inserts, local prefix hits, and cross-node serving hits refresh recency without changing class.

The policy changes replacement priority only. It does not change NIXL, add a data-path RPC, delay registration, or introduce backpressure.

PD E2E Validation

Baseline and feature used the same generated multi-turn corpus and seed in a single-host 2P2D deployment:

• Four independent TP2 vLLM instances: 2 prefill and 2 decode.
• P side: Pega read/write + NIXL pull through vLLM MultiConnector.
• D side: NIXL pull + Pega save-only.
• vLLM PD proxy with round-robin P/D selection.
• vLLM internal prefix cache disabled.
• Pega huge pages enabled; pool size 150 GB per instance.
• Model: DeepSeek-V2-Lite-Chat.
• 512 conversations, 16 messages / 8 model requests per conversation, 4096 requests total.
• 8192 input tokens per generated user message and 128 output tokens per request.
• 32 clients, 32 active conversations, closed-loop with no request-rate limit, seed 0.
• MetaServer metrics sampled every 5 seconds with no smoothing.
• The generated input corpus was identical in both runs: SHA256 e3443b744287912e79e2de3a72cfbfade82972a9ec4eed77b94f5b94877ef0d1.

Both runs completed all 4096/4096 requests. The pool was large enough to represent the intended deployment shape while the sustained workload still created eviction pressure.

Results

Metric	Baseline	Feature	Change
Final average redundancy	2.9278	2.1661	-26.02%
Whole-run non-empty mean redundancy	2.7939	2.1605	-22.67%
Tail 20% mean redundancy	2.8886	2.1106	-26.93%
Final blocks with owners >= 4	106,188	3,635	-96.58%
Final live owner records	687,782	657,385	-4.42%
Evicted blocks	6,773,760	8,646,656	+27.65%
RDMA fetch bytes	799.00 GB	787.09 GB	-1.49%
Cache-miss blocks	2,127,355	2,127,355	unchanged
Request throughput	1.226 req/s	1.223 req/s	-0.24%
Mean TTFT	25,512.45 ms	25,521.10 ms	+8.65 ms

The feature substantially suppresses high-replication blocks once reclaim pressure begins. It performs more evictions, but the identical cache-miss count, slightly lower RDMA volume, and effectively unchanged throughput/TTFT indicate that this run did not turn the additional eviction into extra recomputation or transfer cost.

Scope and Tradeoffs

• Owner hints are best-effort replacement signals, not correctness state.
• The owners >= 3 rule applies to every newly inserted local resident, including D-side save-only copies; the policy does not pin all D-side copies as retained.
• More aggressive reclamation increases eviction churn. Metrics for class distribution and evictions by class are included so this can be monitored after rollout.
• Pega Server and MetaServer are upgraded together; a successful Insert response with missing or misaligned owner_counts is treated as a protocol error.

Checks

• Rust formatting and clippy passed with warnings denied.
• pegaflow-core: 139 passed, 1 ignored.
• pegaflow-metaserver: 31 passed.
• Python connector tests: 38 passed.
• Release server, MetaServer, and PyO3 builds passed in the remote validation environment.
• Baseline and feature both completed the 2P2D workload.

[Copilot]

Pull request overview

This PR introduces a cold/warm segmented eviction policy for the in-memory resident KV read cache to reduce redundant P-side copies in P/D deployments. It threads a new “demote to cold” signal from vLLM’s scheduler connector through the Python RPC bindings and gRPC server into pegaflow-core, and adds metrics for resident tiering and tier-based eviction.

Changes:

• Add cold/warm resident classes to the ReadCache, evicting cold blocks before warm blocks and supporting promotions/demotions with metrics.
• Add SetColdBlocks gRPC API + Rust/PyO3/Python stub plumbing, and call it from the vLLM scheduler connector on finished_sending.
• Add unit tests covering “demote saved hashes once” behavior and the new cache tier semantics.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
python/tests/test_combine_hashes.py	Adds tests ensuring finished_sending only triggers demotion once and that SAVE_ONLY does not demote.
python/src/lib.rs	Exposes set_cold_blocks on the Python RPC client and wires it to the new gRPC method.
python/pegaflow/pegaflow.pyi	Updates Python type stubs with the new EngineRpcClient.set_cold_blocks API.
python/pegaflow/connector/scheduler.py	Tracks pending saved hashes per request and demotes them on finished_sending (best-effort).
pegaflow-server/src/service.rs	Implements the SetColdBlocks gRPC endpoint on the server side.
pegaflow-proto/proto/engine.proto	Adds SetColdBlocksRequest/Response messages and SetColdBlocks RPC to the Engine service.
pegaflow-core/src/storage/read_cache.rs	Implements cold/warm class tracking, promotion/demotion, tier-first eviction, and related metrics.
pegaflow-core/src/storage/mod.rs	Adds StorageEngine::set_cold_blocks plumbing into the read cache.
pegaflow-core/src/metrics.rs	Adds resident-class attributes and metrics for resident blocks, promotions/demotions, and evictions-by-class.
pegaflow-core/src/lib.rs	Exposes PegaEngine::set_cold_blocks API.
pegaflow-core/src/internode/p2p_service.rs	Adds a “not served” stub implementation for the new RPC on the P2P transfer service.
pegaflow-core/src/cache.rs	Adjusts TinyLFU cache API to support keyed removals used by tiered eviction.
docs/pd-redundancy-prefix-repetition.svg	Adds the PD redundancy comparison visualization referenced by the PR description.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.