DEP: Conditional Disaggregation

[karen-sy]

Summary

Motivation, design, and results so far on conditional disaggregation (PPD).

Implementations are stacked here:

• feat(router): add conditional disagg policy kernel #11718 — feat(router): add conditional disagg policy kernel
• feat(backends): handle conditional disagg bypass annotation #11719 — feat(backends): handle conditional disagg bypass annotation
• feat(router): add decode affinity scoring #11720 — feat(router): add decode affinity scoring
• feat(router): orchestrate conditional disagg bypass #11725 — feat(router): orchestrate conditional disagg bypass
• feat(router): expose conditional disagg configuration #11723 — feat(router): expose conditional disagg configuration

Motivation

Conditional disaggregation (C/D) is a feature that enables a hybrid of aggregated and disaggregated request routing. The router may serve a request prefill worker -> decode worker, or it may send the request directly to a decode worker and the backend runs local prefill plus decode there.

For workloads with a high degree of KV reusage on long-ISL requests, e.g. multi-turn / agentic conversation scenarios, conditional disaggregation can help your deployment maintain predictable SLA. Compared to unconditional disaggregation, it reduces memory pressure / TTFT on prefill workers by optimizing reuse of decode-worker KV cache; compared to unconditionally aggregated deployments, it avoids the heavy ITL penalty incurred by co-scheduling heavy prefill workload onto decode workers.

Proposal

Design

C/D hinges on a "bypass" decision -- whether an input should "bypass" a Prefill worker to perform both ctx+gen on a Decode worker. We propose that this bypass decision is primarily owned by the router, which has a view of per-worker load and kv block ownership state. The decision is parametrized as a ConditionalDisaggPolicy, each implementation of which defines a bypass threshold based on input sequence characteristics and/or worker state.

Concretely, we can start with some static ConditionalDisaggPolicy variants:

• ISL-based: Bypass iff ISL - kv cache hit < const threshold AND (effective ISL / ISL) < const threshold
• Load-based: Bypass iff prefill worker loadedness > const threshold AND decode worker loadedness < const threshold
• ISL-and-Load: Bypass iff ISL-based OR Load-based is true

All of the above three options are implemented across the PR stack listed in the Summary.

We can extend these to dynamic policies based on live signals relative to SLA, for example:

• ISL-based-dynamic: Bypass iff ISL - kv cache hit < f(threshold) AND ISL < g(threshold) where f and g are additionally parameterized by current load signals and user-defined SLA
• Load-based-dynamic
• ISL-and-Load-dynamic

In any case, the system view looks like this:

sequenceDiagram
  participant PrefillRouter
  participant DecodeRouter as KvRouter (decode)
  participant Policy as ConditionalDisaggPolicy
  participant NextStage as next.generate

  PrefillRouter->>DecodeRouter: select best decode worker
  DecodeRouter-->>PrefillRouter: worker, overlap tokens
  PrefillRouter->>Policy: should_bypass_remote_prefill(decision input)
  Policy-->>PrefillRouter: bypass decision
  alt bypass allowed
    PrefillRouter->>PrefillRouter: set phase Decode, add bypass annotation
    PrefillRouter->>NextStage: generate() directly on decode worker
    NextStage-->>PrefillRouter: aggregated response stream
  else no bypass
    PrefillRouter->>PrefillRouter: fall back to normal remote prefill path
  end

Loading

We note that this C/D mode requires some rethinking of the router equation. In particular, consider a common case:

turn 0 does P->D on prefill worker0, decode worker0
turn 1~10 do agg on decode worker0. so this D worker0 has a lot of cache
turn 11 does P->D but doesn't route to D worker0 because default mode is load only. misses cache

Therefore we redefine the decode-side routing equation to be kv-aware:

cost = potential_decode_blocks − radix_overlap_credit

We find that this helps decode-side kv reusage (and therefore TTFT).

Experimental results

Setting: Qwen3-235B-A22B-Instruct-2507 + Claude code agentic code load (up to 256k context, p50 theoretical kv reusage >95%) + GB200.
(14GPU disagg and C/D chosen according to prefill/decode ratematching results):

interactivity (p50) vs throughput:

interactivity (p90) vs throughput:

Future Work

I think there are two pieces which will bring C/D performance to the next level:

• p2p kv transfers, particularly in D->P direction: C/D's main empirical weakness is tail TTFT, due to the prefill-side KV cache misses in modes where most requests run in D (i.e. D accumulates KV cache but P never does). Specifically, once D saturates (in terms of KV and general load), requests that are then routed to P have a big kv miss. A D->P transfer would solve this issue nicely, but I've found so far that vLLM NixlConnector's bidirectional transfer mechanism enforces a block pin on D which penalizes overall throughput too heavily.
• offloading: We can try to delay the point that D saturates (in terms of KV) by using G2 offloading. On vLLM, I haven't been able to find a regime where the G1->G2 write latency is offset by a big enough perf gain; need to follow up with some theoretical analysis of what {HBM/CPU capacity, model, workload, offloading policy} combination would help C/D (and disagg in general).

Requirements

C/D is largely owned by router, but we need the frameworks to support using decode workers as essentially a hybrid decodeonly-or-agg worker. vLLM and TRTLLM do (and therefore Dynamo needs minimal modifications to the respective handlers), but SGLang does not (so we need an upstream workaround like this).

References

Not All Prefills Are Equal: PPD Disaggregation for Multi-turn LLM Serving (https://arxiv.org/abs/2603.13358)
Efficient Multi-round LLM Inference over Disaggregated Serving (https://arxiv.org/abs/2602.14516)