disaggregated-prefill-decode-sim

A discrete-event simulator for studying disaggregated LLM serving, where prefill and decode run on separate nodes and communicate through KV-cache transfer. The project compares a traditional coupled serving architecture against a disaggregated one, measuring latency, throughput, queueing, memory pressure, and transfer overhead across multiple models, workloads, and interconnects.

For the full architectural rationale, modeling assumptions, event flow, and design tradeoffs, see the project design document: design.md.

---

Why this project exists

Serving LLMs efficiently is difficult because the two main serving phases have very different resource profiles:

• Prefill is usually compute-bound
• Decode is usually memory-bandwidth-bound

When both phases share the same GPU, they interfere with each other:

• long prompts delay token generation
• long decodes delay new prompts
• queueing grows quickly under load

Production systems such as Splitwise-style architectures address this by splitting the two phases across nodes. This repository simulates exactly that tradeoff:

• benefit: less interference, higher throughput, better TTFT
• cost: explicit KV-cache transfer between nodes

---

What the simulator models

The simulator includes:

• coupled vs disaggregated execution
• Poisson request arrivals
• variable prompt/output lengths
• model-dependent prefill and decode costs
• GQA-aware KV-cache sizing
• prefill micro-batching
• decode contention with batch-size-dependent slowdown
• transfer latency and bandwidth
• transfer/compute overlap
• GPU memory limits for KV cache
• queueing and drop behavior under saturation

---

Main experiment sweep

The default sweep covers:

• arrival rate: 1, 2, 4, 8, 16 req/s
• prompt length: 128, 256, 512
• output length: 64, 128, 256
• hardware:
  • PCIe 3.0
  • PCIe 4.0
  • NVLink v3
  • NVLink v4
• models:
  • 7B
  • 13B
  • 70B

Total default configurations: 540

---

Final headline results

Using the current final simulator configuration:

• Mean throughput gain: 10.76x

• Median throughput gain: 8.26x

• Max throughput gain: 32.37x

• Mean TTFT speedup: 312.62x

• Median TTFT speedup: 173.73x

• Max TTFT speedup: 1428.11x

• Mean transfer overhead: 0.16% of E2E latency

• Mean transfer latency: 3.96 ms

By model

• 7B: strong TTFT gains under queueing
• 13B: balanced latency and throughput improvements
• 70B: strongest throughput gains due to expensive decode

By hardware

• PCIe 3.0 / 4.0 already make transfer cheap enough in many regimes
• NVLink v3 / v4 further reduce transfer cost, but the largest gains often come from concurrency rather than bandwidth alone

---

Repository structure

disaggregated-prefill-decode-sim/
├── src/
│   ├── __init__.py
│   ├── config.py
│   ├── workload.py
│   ├── nodes.py
│   ├── simulator.py
│   ├── metrics.py
│   └── analysis.py
├── results/
├── plots/
├── LICENSE
├── design.md
├── README.md
├── requirements.txt
└── run.py

---

Quickstart

1. Create environment

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

2. Run the full sweep

python run.py

3. Outputs

The run generates:

CSV outputs

• results/sweep_summary.csv
• results/comparison_table.csv
• results/top_throughput_gain.csv
• results/top_ttft_speedup.csv
• results/analysis_summary.txt

Plots

• throughput gain heatmaps
• TTFT speedup heatmaps
• throughput vs arrival rate
• TTFT vs arrival rate
• transfer overhead plots
• completion-rate plots

---

Key files

run.py

Entry point that runs the full parameter sweep and launches analysis.

src/config.py

Defines:

• model parameters
• hardware profiles
• workload configuration
• KV-cache sizing

src/workload.py

Generates requests with:

• Poisson arrivals
• noisy prompt lengths
• noisy output lengths

src/nodes.py

Contains:

• prefill timing model
• decode timing model
• transfer timing model

src/simulator.py

Implements:

• coupled baseline simulation
• disaggregated two-phase event-driven simulation

src/metrics.py

Computes:

• throughput
• TTFT
• TBT
• E2E latency
• queue waits
• completion/drop rates

src/analysis.py

Produces:

• summary tables
• plots
• result breakdowns

---

Current interpretation

The simulator suggests the following high-level conclusion:

In most practical regimes, disaggregating prefill and decode produces large throughput gains and often dramatic TTFT improvements, while KV transfer overhead remains a small fraction of total latency.

This is especially visible when:

• decode is long
• prompts are frequent
• the coupled baseline enters queueing
• the model is large enough that decode dominates occupancy

---

Limitations

This repository is an analytical simulator, not a full serving engine. It does not model:

• exact CUDA kernel behavior
• exact continuous batching semantics of production systems
• speculative decoding
• prefix caching
• tensor/pipeline parallelism internals
• network contention beyond the transfer abstraction
• cluster-level routing or multi-node decode pools

So results should be read as:

• architecture-level guidance
• system tradeoff exploration
• comparative performance modeling

not exact production measurements.

---

License

This project is licensed under the MIT License.
See LICENSE for details.

---

Author

João Felipe De Souza
2026