This map points to primary papers and official project sources. A layer names the first decision boundary to study, not every layer a system may touch. Some sources span local KV management and cluster movement, so they appear in more than one level. The table deliberately omits paper-reported performance figures. n.d. identifies an undated official documentation or navigational page; its access date identifies the reviewed public version.
| Direction/type | Source | Year | Layer | Central mechanism | Reported objective | Unresolved engineering question | Implementation availability | Accessed |
|---|---|---|---|---|---|---|---|---|
| Navigational | Official Inference Engineering guide, Philip Kiely | n.d. | Guide navigation | Point readers to the companion guide without treating it as evidence for a mechanism. | Provide a public orientation to inference-engineering topics. | Which linked technical claim needs a primary paper or official runtime source before it informs an experiment? | Official public site; navigational only, not source text. | 2026-08-07 |
| Foundational | Roofline, Williams, Waterman, and Patterson | 2008 | L1 and machine model | Relate arithmetic intensity to compute and bandwidth ceilings. | Provide a bound-based model for attainable program performance. | Which operation and byte path dominate the selected serving phase? | Primary technical report. | 2026-08-07 |
| Foundational | Attention Is All You Need, Vaswani et al. | 2017 | Transformer anatomy | Attention-based sequence transformation. | Establish an attention-only sequence architecture. | Which model dimensions determine KV state and attention shape? | Primary paper. | 2026-08-07 |
| Foundational | FlashAttention, Dao et al. | 2022 | L1 | Tile attention to reduce data movement without materializing the full attention matrix. | Improve the memory-access behavior of exact attention execution. | Which shape, layout, and precision constraints define a valid kernel comparison? | Primary paper. | 2026-08-07 |
| Current | ORCA, Yu et al. | 2022 | L2 | Iteration-level scheduling and selective batching. | Serve generative requests with finer scheduling granularity. | Which request state can change safely between iterations? | Primary paper and proceedings artifact. | 2026-08-07 |
| Current | Fast Inference from Transformers via Speculative Decoding | 2023 | Runtime lever | Draft multiple tokens and verify them with the target path. | Accelerate autoregressive decoding while preserving target-model output semantics. | Does verification cost preserve the intended request objective? | Primary paper. | 2026-08-07 |
| Current | Accelerating Large Language Model Decoding with Speculative Sampling | 2023 | L2 and runtime lever | Sample candidate continuations from a proposal model and accept or correct them with the target model. | Reduce autoregressive decoding work while preserving the target sampling distribution. | Which proposal, acceptance, and verification costs remain compatible with the selected latency objective? | Primary paper. | 2026-08-07 |
| Current | GQA, Ainslie et al. | 2023 | Model and KV memory | Use fewer key-value heads than query heads. | Interpolate between multi-query and multi-head attention. | How does the model's KV shape affect admission and layout? | Primary paper. | 2026-08-07 |
| Current | PagedAttention, Kwon et al. | 2023 | L3 | Map logical KV blocks to physical blocks. | Improve memory management for generative-model serving. | Which allocation and eviction rule avoids capacity loss for the workload? | vLLM public project. | 2026-08-07 |
| Official project | vLLM documentation and public source | n.d. | L2 and L3 | Document public scheduler and KV-cache-manager boundaries. | Provide a public runtime and extension surface for serving experiments. | Which documented boundary supports a narrow trace or test without assuming maintainer adoption? | Official documentation and public source. | 2026-08-07 |
| Current | Sarathi-Serve, Agrawal et al. | 2024 | L2 | Chunk prefill work and schedule it with decode work. | Address the serving tradeoff between throughput and tail latency. | Which chunk and decode guarantee satisfy both tail objectives? | Primary paper links a public implementation. | 2026-08-07 |
| Current | vAttention | 2024 | L3 | Use virtual-memory-style KV management without committing to one paged-attention kernel design. | Explore portable KV cache management alternatives for serving systems. | Which mapping abstraction is portable across the chosen backend? | Primary paper. | 2026-08-07 |
| Current | Mooncake, arXiv:2407.00079 | 2024 | L3 and L5 | Treat KV state as a tiered resource while separating prefill and decode responsibilities. | Build KV-cache-centric disaggregated serving that meets latency objectives. | When do cache-tier and handoff costs outweigh reduced phase interference? | Primary paper. | 2026-08-07 |
| Current | NanoFlow, arXiv:2408.12757 | 2024 | L2 | Split work into smaller units to overlap different device resources. | Increase end-to-end serving throughput through intra-device parallelism. | Can the proposed overlap survive the target model, shape mix, and correctness constraints? | Primary paper. | 2026-08-07 |
| Official project | SGLang documentation and public source | n.d. | L2 and L3 | Document public scheduler, attention-backend, and prefix-reuse boundaries. | Provide a public runtime and extension surface for scheduling and cache-state experiments. | Which public interface can expose prefix ownership and request outcomes without reconstructing internal behavior? | Official documentation and public source. | 2026-08-07 |
| Current | FlashInfer, arXiv:2501.01005 | 2025 | L1 | Build serving-oriented attention paths around explicit batch, layout, and paged-KV metadata. | Provide customizable and efficient attention execution for LLM serving. | Which kernel plan is correct for the observed shape distribution, not only one benchmark shape? | Official documentation and public source. | 2026-08-07 |
| Current | Bullet, DOI:10.1145/3779212.3790135 | 2026 | L2 | Coordinate prefill and decode spatially and temporally with a performance model. | Improve GPU utilization while preserving latency objectives. | What guardrail preserves decode service while phase overlap is admitted? | Primary paper preprint and proceedings DOI. | 2026-08-07 |
| Current | LMCache, arXiv:2510.09665 | 2025 | L3 and L5 | Decouple KV reuse, offload, and transfer from one engine through a cache layer and connector boundary. | Enable efficient KV cache offload, transfer, and reuse across queries and engines. | What identity, lifetime, location, and fallback rules preserve correct reuse across engines? | Official documentation and public source. | 2026-08-07 |
| Current | Aegaeon, DOI:10.1145/3731569.3764815 | 2025 | L4 | Pool concurrent models and make fine-grained autoscaling decisions. | Enable effective GPU pooling for concurrent multi-model serving. | Which qualified profiles can share capacity without hiding model-specific objectives? | Primary proceedings paper. | 2026-08-07 |
| Current | Prism, arXiv:2505.04021 | 2025 | L4 | Use elastic memory allocation to co-serve multiple models. | Make multi-model serving more cost efficient under changing demand. | What memory reclamation rule remains safe under changing model demand? | Primary paper. | 2026-08-07 |
| Current | Coral, arXiv:2605.04357 | 2026 | L4 | Jointly choose resource allocation and serving strategy across heterogeneous model replicas. | Use heterogeneous GPU capacity for cost-efficient multi-model serving. | How should a profile qualification boundary represent hardware and availability differences? | Primary paper. | 2026-08-07 |
| Current | DistServe, Zhong et al. | 2024 | L5 | Separate prefill and decode pools. | Optimize goodput under separate TTFT and TPOT objectives. | Is the handoff path faster than the colocated phase interference it removes? | Primary proceedings paper. | 2026-08-07 |
| Current | SmartGen, arXiv:2607.28150 | 2026 | L5 | Transfer selected KV state proactively, on demand, and speculatively across a disaggregated handoff. | Reduce the network burden of disaggregated serving through selective KV transfer. | Which state may be omitted or delayed without changing request correctness or later decode behavior? | Primary paper. | 2026-08-07 |
| Current | Towards Load-Aware Prefill Deflection, arXiv:2607.02043 | 2026 | L2 and L5 | Deflect prefill to a decode node when a bounded chunk schedule protects decode service and avoids a handoff. | Improve tail request delay under bursty disaggregated workloads while protecting decode objectives. | Can queue, transfer, and token-delay estimates support a safe deflection decision? | Primary paper. | 2026-08-07 |
| Official project | llm-d documentation and public source | n.d. | L4 to L6 | Document public routing, disaggregation, and KV-event interfaces. | Provide a public distributed-serving integration surface. | Which eligibility and rollback boundary can a clean-room scenario validate before it becomes a controller? | Official documentation and public source. | 2026-08-07 |
| Survey | System-Aware KV Cache Optimization Survey, arXiv:2607.08057 | 2026 | Future map | Organize KV work by execution and scheduling, placement and migration, and representation and retention. | Systematize system-aware KV optimization and identify future research opportunities. | Which cross-layer experiment is worth a separate, falsifiable study? | Survey, not evidence that a particular mechanism works locally. | 2026-08-07 |
| Speculative | Practical Bayesian Optimization of Machine Learning Algorithms | 2012 | L6 | Use observations to select expensive candidate evaluations. | Make expensive black-box optimization more sample efficient. | Which actions are reversible and safe enough for offline or shadow evaluation? | Primary paper; this guide does not bundle a controller. | 2026-08-07 |
| Companion project | Inference Engineering | 2026 | P0 field guide | Preserve the public location of this guide in the portfolio map. | Register project status and scope; not research evidence. | Which maintenance gate must remain true after the next change? | Public Apache-2.0 repository. | 2026-08-07 |
| Companion project | Inference Bottleneck Lab | 2026 | P0 diagnosis | Apply falsifiable diagnosis rules to synthetic or permitted public observations. | Register project status and scope; not research evidence. | Which bounded public runtime seam can one diagnosis test? | Public Apache-2.0 repository. | 2026-08-07 |
| Companion project | KV Policy Lab | 2026 | P0 cache policy | Replay deterministic cache-policy decisions and counterexamples. | Register project status and scope; not research evidence. | Which policy tradeoff should be tested against a real public runtime boundary next? | Public Apache-2.0 repository. | 2026-08-07 |
- Start with the layer whose metric boundary contains the suspected delay or capacity failure.
- Read the primary source for its workload, assumptions, mechanism, and evaluation boundary.
- Locate a public implementation only after the mechanism matches the measured bottleneck.
- Recreate the mechanism with a synthetic fixture or a controlled measurement before generalizing it.
- Record unresolved engineering questions in an experiment record rather than converting a reported conclusion into a local claim.
The project opportunity map turns these mechanisms into a clean-room learning and contribution order. The research map is not a vendor ranking and does not establish compatibility or performance for any local serving profile.