Skip to content

fix(connector): preserve suffix stores on gpu prefix hits#93

Merged
GentleCold merged 1 commit into
masterfrom
fix/shadowed-prefix-population
Jul 12, 2026
Merged

fix(connector): preserve suffix stores on gpu prefix hits#93
GentleCold merged 1 commit into
masterfrom
fix/shadowed-prefix-population

Conversation

@GentleCold

Copy link
Copy Markdown
Owner

Summary

  • Plan missing rolling-prefix suffix stores before returning zero external tokens.
  • Preserve existing behavior that skips already cached DaseR slots.
  • Extend the prefix-hit regression test to cover a longer GPU-local hit.

Motivation

When vLLM already has a longer GPU-local prefix than DaseR, the DaseR hit contributes no external tokens. The scheduler returned before preparing the missing suffix store, leaving the lower-tier prefix permanently shorter until GPU eviction forced recomputation.

Test plan

  • Unit tests added / updated
  • PYTHONHASHSEED=0 pytest -q -m "not integration" --ignore=tests/integration --ignore=tests/connector/test_daser_connector.py --ignore=tests/connector/test_gds_transfer.py tests/ passes: 353 passed
  • Connector and server tests pass: 240 passed
  • Integration tests: N/A; scheduler lifecycle behavior is covered by the connector regression
  • Benchmark / e2e smoke: validated separately with TP2 long-context replay; no benchmark code is included
  • pre-commit run --all-files passes

Checklist

  • pre-commit run --all-files passes
  • Type hints and docstrings on all new/modified functions
  • New features and bug fixes include tests
  • No regressions - existing tests still pass

- Plan missing prefix suffix stores before returning zero external tokens
- Cover GPU-local hits that fully shadow shorter DaseR hits
@GentleCold
GentleCold merged commit e559308 into master Jul 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant