fix(qwen3): reclaim completed KV blocks when prefix cache is off (#681)

[Polaris3003]

Qwen3 DFlash disables prefix matching so the target prefill produces fresh hidden states for the draft model, but disabling matching did not disable retention of completed request blocks. Registered blocks entered kvbm's inactive cache even though no later DFlash request can match them; when a later request recomputed the same hash, BlockDuplicationPolicy::Allow registered a duplicate that pinned the old primary via _primary_keepalive, leaving it physically resident but invisible to scheduler accounting. Near capacity, speculative verification then failed to allocate its next page.

Add ImmutableBlock::set_primary_evict_on_reset to mark the canonical primary (followed through a duplicate's keepalive) for reset-on-release, and RequestKv::mark_blocks_reset_on_release to apply it across a request's registered assignments. Qwen3 centralizes the retention decision in retains_completed_kv_blocks / finish_retention_transition: enabling->disabling drains the inactive pool once, and drop_request marks a completed request's blocks for reset when retention is off. Prefix-cache-enabled serving keeps its existing inactive-cache reuse behaviour.

Description

Fixes #681 (issue number)

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update

Checklist

• My code follows the style guidelines of this project (see docs/conventions/coding-style.md).
• I have performed a self-review of my own code.
• I have formatted my commits according to Commitizen conventions.
• I have run the local test suite and all tests pass (see CLAUDE.md).

[xiaguan]

Thanks for the careful lifecycle work here. The direction makes sense: when prefix matching is disabled, completed-request blocks should not remain in the inactive cache, and following a duplicate's keepalive to mark the canonical primary addresses the hidden-capacity case from #681.

The low-level coverage is useful. I ran cargo test -p kvbm-logical -p openinfer-kv-cache --lib locally: all 495 kvbm-logical tests and all 5 openinfer-kv-cache tests passed, including the new duplicate-primary and issue-shaped capacity cases.

Before approval, could you please add contributor-run E2E evidence for the affected Qwen3 path:

1. Reproduce #681 on the baseline and show this branch completing the same sequential 161-block / --gpu-memory-utilization 0.85 workload. Please cover both sampled and greedy DFlash, since the issue reproduces on both paths, and include enough consecutive requests to show capacity remains reusable.
2. Run the affected Qwen3 prefix_cache integration test and HF golden gate, with the exact commands and pass summaries.
3. Let the repository CI complete successfully; the CI run is currently action_required, so only the attribution check has executed.

These results will connect the sound CPU lifecycle model to the actual GPU executor path and verify that disabling retention fixes the reported failure without regressing prefix-cache-enabled serving.

[Polaris3003]

Thanks — here is the contributor-run E2E and integration-test evidence for #681.

Environment

• GPU: NVIDIA GeForce RTX 3090, 24 GiB
• Driver: 580.76.05
• Target model: Qwen3-4B
• Draft model: Qwen3-4B-DFlash-b16
• Prefix cache: disabled for the DFlash reproduction
• Baseline: parent commit 7af750b5
• Baseline binary SHA-256: 30c8831fd03eec6f8698b9d29794cd82f84900e9d2ffdeea524ef353a7002cfc
• PR binary SHA-256: dd08dfddc61708064a05155e86b63242e23fa419240c914cc32a6d05fba45e39

Because this GPU has 24 GiB rather than 16 GiB, literal --gpu-memory-utilization 0.85 would provide substantially more KV capacity. I tuned it to 0.502 to reproduce the same capacity-constrained condition:

memory profile: total=24124 MiB requested=12110 MiB
KV cache (profiled): 161 blocks
KV capacity: total_blocks=160, block_size=16, kv_cache_size_tokens=2560

This is therefore a capacity-equivalent 161-profiled-block reproduction, rather than a hardware-identical 0.85 run.

DFlash E2E command

The server was restarted between every case:

RUST_LOG=info <baseline-or-PR-binary> \
  --model-path /root/openinfer/models/Qwen3-4B \
  --served-model-name qwen3-4b \
  --port 18071 \
  --gpu-memory-utilization 0.502 \
  --no-prefix-cache \
  --dflash-draft-model-path /root/openinfer/models/Qwen3-4B-DFlash-b16

Each mode ran 20 sequential requests:

uv run --no-project \
  --with datasets \
  --with requests \
  --with tqdm \
  --with transformers \
  python scripts/eval_gsm8k_thinking.py \
    --base-url http://127.0.0.1:18071/v1/completions \
    --model qwen3-4b \
    --tokenizer /root/openinfer/models/Qwen3-4B \
    --num-fewshot 8 \
    --limit 20 \
    --max-tokens 512 \
    --seed -1 \
    --temperature <0.8-for-sampled-or-0-for-greedy> \
    --output <case-result.json>

Results:

Code	DFlash mode	Result
Baseline	sampled (temperature=0.8)	Failed on request 3/20 with allocation failed: needed 1 blocks and HTTP 500
Baseline	greedy (temperature=0)	Failed on request 3/20 with allocation failed: needed 1 blocks and HTTP 500
PR	sampled (temperature=0.8)	Completed 20/20 sequential requests; no allocation failure
PR	greedy (temperature=0)	Completed 20/20 sequential requests; no allocation failure

This reproduces #681 on both baseline paths and shows the PR reusing the constrained KV capacity across consecutive requests.

One evaluator detail: the greedy invocation completed all 20 requests and wrote all 20 results, but eval_gsm8k_thinking.py returned status 1 because it returns success only when flexible_extract > 0. The model continued into the next Question: after emitting #### <answer>, so the flexible extractor selected a number from the following question. The strict result was 17/20, and the server log contained no request or allocation failure. I treated request completion plus absence of allocation errors as the lifecycle E2E criterion.

Qwen3 prefix-cache integration test

Exact command:

OPENINFER_TEST_MODEL_PATH=/root/openinfer/models/Qwen3-4B \
cargo test --release \
  -p openinfer-qwen3 \
  --test prefix_cache \
  -- --nocapture

Result:

test prefix_cache_behavior ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Qwen3 HF golden gate

Exact command:

OPENINFER_TEST_MODEL_PATH=/root/openinfer/models/Qwen3-4B \
cargo test --release \
  -p openinfer-qwen3 \
  --test hf_golden_gate \
  -- --nocapture

Result:

test pega_logprobs_match_hf_golden_within_bf16_tolerance ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Repository CI remains a separate gate; it still needs maintainer approval and successful completion.