[serve][llm] KV cache aware routing & management tracker

[jeffreywang88]

This issue tracks the implementation progress of KV cache-aware routing and KV cache management features.

Milestone 1: KV-cache aware routing for aggregated deployment

• [llm][kv][1/N] Add KV-aware routing interfaces (KVAwareRouter + KVRouterActor) #64084
• [llm][kv][2/N] Enable replica tracking in KVRouterActor #64085
• [llm][kv][3/N] Implement KV aware routing's event plane #64097
• [llm][kv][4+5/N] Pre-routing tokenization & KV cache aware routing #64224
  • Support Anthropic SDK
  • Investigate removing generator return from tokenization endpoint
• [llm][kv][5/N] Implement request routing with KV awareness #64108
• [llm][kv][6/N] Enable token-level request lifecycle tracking #64327
• [llm][kv][7/N] Prefill/decode token load aware request routing #64400
• [llm][kv][8/N] Move tokenization in-process of LLMRouter ingress replica #64642
• [llm][kv][9/N] Determine KV / token routing decision inside the LLMRouter ingress replica #64920
• [llm][kv][10/N] Broadcast KV lifecycle events to all LLMRouter ingress replicas  #64949
  • KVTracker internal state / consistency observability (e.g. token load / KV radix tree similarity): context
• [llm][kv][11/N] Enable atomic selection & reservation broadcast to avoid herding #65010
  • Bandwidth impact for broadcasting to keep LLMRouter replicas state consistent across multi-nodes
  • [Investigation] Ring all-reduce to synchronize states
  • [Investigation] Pure socket implementations
• [llm][kv][12/N] Transmit tokens out-of-band and skip tokenization at the engine #65095

Milestone 2: Advanced deployment patterns support

• LoRA support
• Data-parallel deployments
• PD disaggregated deployments
• Multimodality

Milestone 3: Tiered KV cache management

• KV cache offloading: [serve][llm] Enable KV cache offloading & KV routing awareness of CPU KV caches #65063

Milestone 4: Hybrid KV cache support

• DSV4 hybrid attention

[Arijit6258]

@jeffreywang88 I see quiet a few sub-tasks in the list. I am interested to contribute to this repository. Is it possible to delegate some of the tasks which I can work upon?

[jeffreywang88]

• Substitute asyncio.ensure_future() with asyncio.create_task() under KVRouterActor._schedule (comment): PR.

[Arijit6258]

@jeffrywang88

I would appreciate a comment.

[jeffreywang88]

@Arijit6258 feel free to scope out "Support Anthropic SDK" and/or #64389 (comment). Thank you for your interest!

[Arijit6258]

@jeffreywang88 is there any document of what exactly needs to be done here? I don't see any description.

[jeffreywang88]

@jeffreywang88 is there any document of what exactly needs to be done here? I don't see any description.

For Anthropic SDK support, you can check out #64224 (comment), but the exact details are yet to be scoped out. It'd be great if you can put together a RFC.