fix(qwen3): proactive cancellation sweep for queued requests

[sparkzky]

Summary

Qwen3 scheduler only discovered cancellation reactively at token emission — by then the full prefill had already burned GPU. Under a disconnect storm, cancelled work consumed scheduler capacity for tens of seconds.

Closes #642.

Root cause

• admit_deferred_requests never checked token_tx.is_closed() — cancelled deferred requests were admitted and ran full prefill before the first token send failed.
• The single proactive check in the scheduler was ContinuePrefill (effects.rs:316), which only covered mid-chunk cancellation.
• publish_load() ran before the sweep at the top of the loop, so after a sweep cleared everything the scheduler parked idle without republishing (0,0) — load metrics froze stale.

Changes

1. sweep_cancelled_requests() — proactive sweep (scheduler.rs)

One function called at the top of each scheduler iteration (both scheduler_loop and scheduler_loop_with_lora_control), after draining submit_rx and before admission:

• Checks token_tx.is_closed() across all queues: deferred, loading, prefilling, post_control_deferred, active
• Drops cancelled requests via executor.drop_request() (KV block RAII release, prefetch DMA wait, saved_cursor cleanup)
• Retain is driven by collected request IDs, NOT by re-checking is_closed(), to avoid a TOCTOU race (Codex review)
• inflight_prefill_pending is deliberately excluded — its KV is mid-write during decode-overlap prefill; documented in the function doc

2. publish_load() reordered (scheduler.rs)

Moved to after sweep_cancelled_requests() in both loops. Per-iteration order is now: drain → sweep → publish_load → prefetch/admit. This ensures the published metrics reflect post-sweep reality before the scheduler parks idle.

Verification

Test	Result
cargo test -p openinfer-qwen3 --lib	70 passed, 0 failed
cancellation_sweep (GPU, EngineHandle)	6 requests → disconnect → 0/0 in 250ms → follow-up served
http_cancellation (GPU, real HTTP path)	8 streaming clients → disconnect → 0/0 in 59ms → follow-up served
scheduler_robustness (regression)	Passed (pin served: prefill=253 full=506)

New tests

CPU-only scheduler tests (scheduler/tests.rs):

• sweep_drops_cancelled_deferred_request_before_prefill — cancelled deferred + post_control_deferred requests never reach prefill; live sibling survives
• sweep_drops_cancelled_prefilling_request_and_releases_state — cancelled mid-chunk request releases executor state; live sibling keeps its place
• sweep_drops_cancelled_active_request_from_running_set — cancelled decoding request leaves running set and returns KV; live sibling keeps decoding

GPU integration test — EngineHandle (tests/cancellation_sweep.rs):

• cancelled_burst_metrics_collapse_promptly — submits 10 concurrent requests, drops all receivers, asserts num_running=0 && num_waiting=0 within bounded time, then verifies a clean follow-up completion succeeds

GPU integration test — real HTTP disconnect path (tests/http_cancellation.rs):

• http_disconnect_metrics_collapse — starts the real openinfer server, sends 8 concurrent streaming /v1/completions requests, disconnects all clients, polls /metrics for collapse, then sends a clean follow-up

E2E Evidence — real HTTP disconnect path

Exact command

OPENINFER_TEST_MODEL_PATH=/path/to/Qwen3-4B \
  cargo test --release -p openinfer-qwen3 --test http_cancellation -- --nocapture

Full output

=== all 8 streaming clients connected ===
=== in-flight: running=1 waiting=14 ===
=== disconnecting all clients ===
=== polling for metrics collapse ===
auto-aborting request due to dropped stream request_id="cmpl-e20820de-35a4b84e"
auto-aborting request due to dropped stream request_id="cmpl-d1a09093-123bd679"
auto-aborting request due to dropped stream request_id="cmpl-47eca728-ba2f7cbe"
auto-aborting request due to dropped stream request_id="cmpl-d446cfd8-a334a1de"
auto-aborting request due to dropped stream request_id="cmpl-844b1546-66c48b63"
auto-aborting request due to dropped stream request_id="cmpl-0e1971f5-f6ae06b6"
auto-aborting request due to dropped stream request_id="cmpl-eaf21906-53f437a3"
auto-aborting request due to dropped stream request_id="cmpl-74ae281f-5121e561"
=== collapsed to running=0 waiting=0 in 59ms ===
=== follow-up non-streaming request ===
=== follow-up ok: status=200 OK text=" to the new era of" ===
=== PASS: metrics collapsed after disconnect, follow-up served ===

test result: ok. 1 passed; 0 failed; 0 ignored; finished in 31.17s

The auto-aborting request due to dropped stream log lines confirm the full bridge abort path: HTTP client disconnect → bridge detects dropped stream → sets abort_reason → scheduler sweep calls drop_request → /metrics reflects zero.

Acceptance criteria

• A cancelled deferred request is never passed to prefill
• Cancelled loading/prefilling requests release executor-side state and KV reservations
• Cancelled active requests call drop_request at a safe scheduler boundary and leave the running set
• Scheduler load metrics exclude cancelled requests promptly
• CPU-only scheduler tests cover cancellation in deferred, prefilling, and active states with a fake executor
• A one-GPU HTTP integration test sends a concurrent burst, disconnects the clients, and verifies running/waiting metrics return to zero within a bounded number of scheduler steps
• A clean follow-up completion succeeds without waiting for cancelled prompts to prefill

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f766656a2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[CAICAIIs]

Hi, thanks for your PR!
Just a small reminder: we have related existing issues tracking this feature/bug. Next time could you leave a comment on the corresponding issue before submitting PR? This helps us avoid duplicate work and code conflicts with others who are also working on the same task. Much appreciated!

[sparkzky]

Hi, thanks for the heads-up! Noted — I'll make sure to leave a comment on the corresponding issue first next time before opening a PR.

[xiaguan]

Could you please add reproducible E2E evidence to the PR description, including the exact command and full output, ideally covering the real HTTP disconnect path?

[sparkzky]

Hi @xiaguan, thanks for the review!

Added a Rust HTTP integration test (tests/http_cancellation.rs) that covers the real disconnect path, and updated the PR description with the full E2E evidence.

The test starts the actual openinfer server, sends 8 concurrent streaming /v1/completions requests, disconnects all clients, and polls /metrics. Key evidence from the output:

• Bridge abort detection: server logs auto-aborting request due to dropped stream for all 8 requests immediately after disconnect
• Metrics collapse: vllm:num_requests_running and vllm:num_requests_waiting reach 0 within 59ms
• Follow-up succeeds: clean non-streaming request returns 200 OK immediately after

Reproduce with:

OPENINFER_TEST_MODEL_PATH=/path/to/Qwen3-4B \
  cargo test --release -p openinfer-qwen3 --test http_cancellation -- --nocapture

[xiaguan]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33de043b81

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".