Benchmarking output length prediction quality and calibration for KV-cache-aware LLM serving scheduling. A noisy predictor with 1.5x safety margin captures 99.7% of oracle gains — calibration matters more than accuracy.
python simulation scheduling inference kv-cache admission-control llm-serving serving-infrastructure mlsystems output-length-prediction prediction-calibration
-
Updated
Jul 19, 2026 - Python