feat(connector): add tiered io_uring KV prefetch#96
Merged
Conversation
- Add an optional TransferLayer prefetch capability for io_uring L2-to-L1 promotion\n- Defer scheduler admission until bounded background prefetch completes\n- Wire the disabled-by-default benchmark control and add focused regression tests
- Queue every vLLM-finished store in request insertion order. - Bound staging snapshot and transfer admission with a FIFO semaphore. - Move authoritative L1 range commit to DaseR and remove duplicate worker commits. - Keep benchmark drain mandatory and remove dummy completion polling. - Add regression coverage, end-to-end validation, and design documentation.
- reserve pinned L1 slices while io_uring promotions are in flight - discard promotions invalidated by newer overlapping stores - release failed promotion reservations and cover both races with regressions
- classify all-L1 external windows atomically and bypass the bounded L2 prefetch window\n- retain transfer leases through CUDA staging and block overlapping stores until release\n- propagate TP-safe lease ranges through scheduler, IPC, and worker load batches\n- add scheduler, transfer, server, IPC, and worker regression coverage
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
l1,mixed, orl2in the existing lookup IPCprefetch_max_requests=2for mixed/L2 requests; all-L1 requests bypass that L2-only boundValidation
365 passed, 101 deselectedgit diff --check: passed200-Request Benchmark
Configuration for all rows: Qwen3-8B, GPU 2, TP1, block size 128, 200 requests, 10,000 input tokens, 1 output token, max-num-seqs/concurrency 16, seed 42, request rate
inf, GPU utilization 0.7, eviction-enabled io_uring, DaseR prefix backend. Feature rows useprefetch_max_requests=2;masterdoes not expose that flag.master(e559308)eaf008d)All rows passed DaseR
/drain. Warm mean deltas:-41.35 ms(-3.45%)-178.99 ms(-13.39%)1158.61 ms):-0.63 msPrefetch counters after warm load:
The overlap fix issued 38 fewer prefetch RPCs while preserving identical L2 bytes, demonstrating that all-L1 requests bypass the L2 window without removing L2 backpressure. It introduced no new cold/warm mismatch; one control mismatch disappeared.
Notes