A simulator comparing three KV-cache-aware scheduling policies for LLM serving under memory pressure: greedy, admission control, and memory-first.
For full methodology and design decisions see design.md.
Standard LLM schedulers dispatch requests without visibility into KV cache usage. Accepting a large request can exhaust the KV budget and force preemption of several smaller requests, wasting their partially completed work.
KV-cache-aware scheduling makes this tradeoff explicit at admission time rather than discovering it after the fact.
greedy Accept everything. Preempt active requests when KV overflows. Maximum acceptance rate, but high preemption waste.
admission_control Reject requests that would cause KV overflow before accepting them. Zero preemptions, but explicit rejection of some requests.
memory_first Sort pending requests by KV footprint ascending within each scheduling window. Serve smallest-footprint requests first. Reject if still no room. Best tradeoff between rejection and preemption.
Policy raw_tput clean_tput preemption rejection greedy 8.41 3.46 49.4% 0% admission_control 5.97 5.97 0% 25.7% memory_first 6.06 6.06 0% 25.0%
Greedy appears 39% faster in raw throughput. Under effective throughput (clean completions only), greedy is 43% worse.
Lower rejection than admission_control Lower P99 TTFT than greedy (67.7ms vs 75ms on mixed) Zero preemptions
Greedy's raw throughput advantage disappears under effective throughput. Raw throughput counts preempted work. Effective throughput does not. Under mixed workloads, greedy wastes 59% of its GPU compute on preemptions.
Memory-first is Pareto-dominant. Lower rejection than admission_control. Lower latency tail than greedy. Zero preemptions.
Effective throughput is the correct metric under KV memory pressure. Optimizing for raw throughput leads to greedy, which wastes the most compute.
The tradeoff is controlled rejection vs uncontrolled preemption. Explicit early rejection wastes no compute. Preemption wastes all compute done on the request before eviction.
KV-aware scheduling matters most under heterogeneous workloads. Under uniform workloads, all policies converge. Under heavy-tail and mixed workloads, the gap is large.
kv-cache-aware-scheduler/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── simulator.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs: results/results.csv results/summary.txt plots/ttft_vs_arrival.png plots/short_request_tail.png plots/effective_throughput.png plots/kv_waste.png
Closes a loop across three projects:
llm-inference-scheduler: scheduling by queue and priority paged-attention-sim: KV cache allocation by request kv-cache-aware-scheduler: scheduling that prevents overflow upfront
Also connects to:
kv-cache-eviction-benchmark: what to do when cache is already full sharegpt-workload-bench: real workloads have heavy-tailed prompt lengths
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026