[GG] fix(sched): tolerate late KV-transfer completions - #196
Conversation
_update_from_kv_xfer_finished asserted that every finished_recving or finished_sending request id is still tracked and in an expected status. A worker-side connector can violate both: an abort can race an async KV load so the completion arrives after the request was freed, and a connector can report the same request twice (seen with the LMCache MP connector under kv_both plus async scheduling; upstream reports in LMCache/LMCache#2356, partial fix for one producer in LMCache/LMCache#4136). Each occurrence killed EngineCore with an AssertionError. Warn and skip unknown request ids, and skip the block free for a live request that is neither WAITING_FOR_REMOTE_KVS nor finished, since freeing the blocks of a live request would corrupt scheduler state.
|
Warning Review limit reached
Next review available in: 57 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
3 participants
Summary
This replaces #192 after its source branch was deleted. It preserves Florian Bernd's original commit and adds the send-side live-request guard requested in review.
Why
LMCache MP with async scheduling can deliver a completion after abort cleanup or report a completion twice. Assertions in the scheduler then terminate EngineCore. A stale send completion must also never free blocks still owned by a running request.
Tests
python3 -m pytest -q tests/v1/kv_connector/unit/test_late_kv_xfer_completions.pyResult: 5 passed.