Skip to content

feat: add pd prefill short request first scheduling. - #2092

Open
immengzi wants to merge 1 commit into
xLLM-AI:mainfrom
immengzi:feat/pd-prefill-short-request-first-v0
Open

feat: add pd prefill short request first scheduling.#2092
immengzi wants to merge 1 commit into
xLLM-AI:mainfrom
immengzi:feat/pd-prefill-short-request-first-v0

Conversation

@immengzi

Copy link
Copy Markdown

Description

Adds opt-in ShortRequestFirst scheduling for PD-prefill requests.

  • Adds enable_short_request_first, short_request_first_threshold, and short_request_first_long_max_wait_ms through CLI flags and JSON configuration.
  • Splits online PD-prefill requests into immediate, short, and long queues. Requests with partial KV state or preemption recovery are scheduled first; fresh short requests are then prioritized over fresh long requests.
  • Supports optional long-request aging to prevent starvation. A value of 0 retains strict short-request-first behavior.
  • Keeps offline requests in the existing queue and preserves online-request scheduling priority.
  • Restricts the feature to FCFS PD-prefill/MIX scheduling with chunked prefill enabled.
  • Adds queue-depth and aged-long-promotion metrics.
  • Adds configuration, queue, and scheduler unit tests, together with PD startup and AISBench benchmark/replay helper scripts.

Related Issues

N/A

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

No known functional issues.

@XuZhang99 XuZhang99 changed the title feat: add pd prefill short request first scheduling feat: add pd prefill short request first scheduling. Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant