perf(connector): optimize prefix load staging pipeline#87
Merged
Conversation
- Add a fixed-size staging pool for load buffers. - Cover reuse and oversized-request behavior with focused tests.
- Add FIFO helper for fixed-depth load batch submission. - Cover depth-two submission and buffer reuse order.
- Use two fixed load staging buffers for worker cache-hit loads. - Overlap server load submission with prior batch restore. - Preserve FIFO restore ordering and final CUDA synchronization.
- Add IPC support for registering fixed load staging buffers by index. - Use registered staging buffer indexes on the worker load hot path with a CUDA IPC fallback. - Cover registered staging RPCs and server mapping reuse in tests.
- Rename the reusable staging pool to store-only terminology. - Keep load restore paths on the fixed staging pool. - Update connector tests to use the store staging pool name.
- Reuse one fixed CUDA staging pool implementation for store and load paths. - Block store staging acquisition on completed store futures instead of allocating dynamically. - Consume load pipeline batches in completion order with fixed in-flight depth.
- add fixed load IPC clients so staged load batches use independent server connections - consume io_uring L2 miss reads as they complete and copy each batch to the destination immediately - cover buffer-scoped load clients and completion-driven L2 miss promotion
- track request-level load futures while preserving grouped load pipeline submission - mark each request complete after all batches containing it restore into KV cache - cover independent request load completion and request-id annotated load ranges
- map staging-local split load IDs back to their owning request before updating request-level load futures - cover split staging completion so async loads cannot leave requests waiting forever
- add a fixed-width worker load queue that groups request loads across scheduler windows - drive queued loads through the existing load pipeline with request-level completion futures - cover queued load grouping and pump batching behavior
- replace grouped load queue pumping with request-level max-inflight dispatch - bound active request loads by fixed staging pool depth and configured inflight limit - expand fixed load staging registration and IPC lanes for request-level dispatch - cover immediate dispatch and staging-depth limiting in connector tests
GentleCold
marked this pull request as ready for review
July 9, 2026 06:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Motivation
Warm prefix TTFT in evict mode is significantly slower than LMCache for the 8192/1, block size 128, c8, 200-request workload. This PR keeps the benchmark-positive load-pipeline changes: bounded staging allocation, registered staging mappings, request-level completion, and request inflight dispatch bounded by fixed staging depth.
Follow-up transfer-side experiments that did not improve this workload were dropped from this branch.
Test plan
pre-commit run --all-filespassesValidation run:
pytest tests/connector/test_daser_connector.py tests/connector/test_gds_transfer.py tests/connector/test_ipc_client.py -q-> 134 passedpytest tests/transfer/test_tiered_iouring_transfer.py tests/transfer/test_iouring_skip_l2_transfer.py -q-> 30 passedpre-commit run --all-files-> passedBenchmark evidence, evict mode, block size 128, 8192/1, c8, 200 requests, Qwen3-8B, run id
20260709_140956:Checklist
pre-commit run --all-filespasses