feat(frontend): honor continuous_usage_stats in vLLM chat processor#11708
feat(frontend): honor continuous_usage_stats in vLLM chat processor#11708nguyenthai-duong wants to merge 2 commits into
Conversation
The vLLM chat processor (`--dyn-chat-processor vllm`) only attached a `usage` object to a streamed chunk when the engine reported `completion_usage`, which vLLM sends on the final chunk only. As a result, requests with `stream_options.continuous_usage_stats=true` received `usage: null` on every intermediate chunk, unlike vLLM's and SGLang's native OpenAI servers. Read `stream_options.continuous_usage_stats` from the request and, when set, synthesize a per-chunk `usage` object from the running token counters (prompt tokens + cumulative output tokens) the processor already tracks. Engine-provided `completion_usage` still takes precedence on the final chunk so authoritative counts are preserved. Add unit tests covering the flag off (no intermediate usage), flag on (running per-chunk usage), and engine-usage precedence. Signed-off-by: nguyenthai-duong <63569535+nguyenthai-duong@users.noreply.github.com>
nguyenthai-duong
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
nguyenthai-duong
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
|
👋 Hi nguyenthai-duong! Thank you for contributing to ai-dynamo/dynamo. Just a reminder: The 🚀 |
github-actions
Bot
added
feat
external-contribution
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| stream_options = request.get("stream_options") or {} | ||
| continuous_usage_stats = bool( | ||
| stream_options.get("continuous_usage_stats", False) | ||
| ) |
There was a problem hiding this comment.
🔍 Synthesized usage emitted even when include_usage is false
The new code at components/src/dynamo/frontend/vllm_processor.py:678-681 reads only continuous_usage_stats from stream_options, but the OpenAI API spec defines continuous_usage_stats as a sub-option of include_usage — it should only take effect when include_usage is also true. The current implementation will emit per-chunk usage if a client sends {"continuous_usage_stats": true} without "include_usage": true. In practice this is unlikely to cause problems (clients that set the sub-option without the parent are already violating the API contract), but it diverges from vLLM's own handling where include_usage gates all usage emission. If strict OpenAI compatibility matters, an additional and stream_options.get("include_usage", False) check would be warranted.
Was this helpful? React with 👍 or 👎 to provide feedback.
WalkthroughChangesContinuous usage statistics
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@components/src/dynamo/frontend/vllm_processor.py`:
- Around line 678-681: Update the stream-options parsing near
continuous_usage_stats so it is enabled only when include_usage is truthy, while
preserving the existing default behavior when either option is absent. Add
coverage for continuous_usage_stats=true with include_usage missing and with
include_usage=false, verifying that per-chunk usage is not emitted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 389c2bf3-75c1-4166-beae-d943c4718f11
📒 Files selected for processing (2)
components/src/dynamo/frontend/tests/test_vllm_processor_unit.pycomponents/src/dynamo/frontend/vllm_processor.py
| stream_options = request.get("stream_options") or {} | ||
| continuous_usage_stats = bool( | ||
| stream_options.get("continuous_usage_stats", False) | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the file structure and inspect the relevant area with line numbers.
git ls-files 'components/src/dynamo/frontend/vllm_processor.py' 'components/src/dynamo/frontend/*test*' 'components/tests/*' | sed -n '1,120p'
echo '--- vllm_processor.py around the cited lines ---'
sed -n '640,730p' components/src/dynamo/frontend/vllm_processor.py
echo '--- search for include_usage / continuous_usage_stats ---'
rg -n "include_usage|continuous_usage_stats|stream_options" components/src/dynamo/frontend components/tests -SRepository: ai-dynamo/dynamo
Length of output: 7463
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- file outline ---'
ast-grep outline components/src/dynamo/frontend/vllm_processor.py --view expanded || true
echo '--- relevant slices ---'
python3 - <<'PY'
from pathlib import Path
p = Path('components/src/dynamo/frontend/vllm_processor.py')
lines = p.read_text().splitlines()
for start,end in [(650,720),(720,790)]:
print(f'\n### {start}-{end}')
for i in range(start, min(end, len(lines))+1):
print(f'{i:4d}: {lines[i-1]}')
PY
echo '--- tests mentioning include_usage / continuous_usage_stats ---'
rg -n "include_usage|continuous_usage_stats" components/src/dynamo/frontend components/tests -SRepository: ai-dynamo/dynamo
Length of output: 10368
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "include_usage|continuous_usage_stats|usage" components/src/dynamo/frontend/vllm_processor.py -SRepository: ai-dynamo/dynamo
Length of output: 917
🌐 Web query:
vLLM continuous_usage_stats include_usage stream_options docs
💡 Result:
In vLLM, the stream_options parameter allows users to configure how token usage statistics are returned during streaming responses [1][2]. This parameter is part of the OpenAI-compatible API and can be included in chat completion and completion requests when stream=True [2][3]. Key components of stream_options include: include_usage: When set to true, vLLM sends a final usage chunk (containing token usage metrics) after all content chunks have been emitted [1][4]. If this is not enabled, streaming responses typically do not include usage data [1]. continuous_usage_stats: When enabled, this option provides cumulative usage statistics in every streamed token chunk, rather than only in the final usage chunk [5][4]. To function, this generally requires include_usage to also be set to true [5]. These options are managed within the vLLM OpenAI-compatible frontend, specifically within the StreamOptions protocol definition [6][2]. They ensure compatibility with the standard OpenAI API streaming behaviors while providing extended observability for real-time applications [5][7]. Developers can utilize these options by passing them in the stream_options dictionary during the API request [1].
Citations:
- 1: https://docs.vllm.ai/en/latest/features/per_request_metrics/
- 2: https://github.com/vllm-project/vllm/blob/7fe7fa9c/vllm/entrypoints/openai/completion/protocol.py
- 3: [Feature][Frontend]: Add support for
stream_optionsinChatCompletionRequestvllm-project/vllm#5135 - 4: [Rust Frontend] Support streaming
generateendpoint vllm-project/vllm#43779 - 5: [Rust Frontend] Support continuous_usage_stats stream option vllm-project/vllm#43965
- 6: https://github.com/vllm-project/vllm/blob/469f3dcf/vllm/entrypoints/openai/engine/protocol.py
- 7: [Frontend] Continuous usage stats in OpenAI completion API vllm-project/vllm#5742
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '790,845p' components/src/dynamo/frontend/vllm_processor.py
echo '--- include_usage in file ---'
rg -n "include_usage" components/src/dynamo/frontend/vllm_processor.py components/src/dynamo/frontend/tests/test_vllm_processor_unit.py -S
echo '--- relevant tests around continuous usage stats ---'
sed -n '540,670p' components/src/dynamo/frontend/tests/test_vllm_processor_unit.pyRepository: ai-dynamo/dynamo
Length of output: 7505
Gate continuous_usage_stats on include_usage.
continuous_usage_stats is evaluated independently here, so stream_options: {"continuous_usage_stats": true} can emit per-chunk usage without opting into usage reporting. Tie it to include_usage, and add a test for continuous_usage_stats=true with include_usage missing or false.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@components/src/dynamo/frontend/vllm_processor.py` around lines 678 - 681,
Update the stream-options parsing near continuous_usage_stats so it is enabled
only when include_usage is truthy, while preserving the existing default
behavior when either option is absent. Add coverage for
continuous_usage_stats=true with include_usage missing and with
include_usage=false, verifying that per-chunk usage is not emitted.
![dynamo-review-agent[bot]](https://avatars.githubusercontent.com/u/201626793?s=60&v=4){kind=small}
| # asked for per-chunk stats: emit running counts. | ||
| dynamo_out["usage"] = { | ||
| "prompt_tokens": input_tokens, | ||
| "completion_tokens": cumulative_output_tokens, |
There was a problem hiding this comment.
Synthesized continuous usage uses the request-wide cumulative_output_tokens, so n > 1 streams report tokens generated for other choices in each per-choice chunk. Fix: track completion tokens per output_idx and use the current choice's running total when building continuous usage.
🤖 AI Fix
In components/src/dynamo/frontend/vllm_processor.py, inside VllmProcessor._generate_and_stream, add a dict[int, int] counter keyed by output_idx; after deriving and validating output_idx, increment both the existing request-wide cumulative_output_tokens for metrics and the per-choice counter for that output_idx, then in the continuous_usage_stats branch set completion_tokens and total_tokens from the current output_idx counter instead of cumulative_output_tokens.
…hoice Address review feedback on the vLLM chat processor per-chunk usage: - continuous_usage_stats is a sub-option of include_usage per the OpenAI spec, so only emit synthesized per-chunk usage when include_usage is also true (matches vLLM's native gating). (Devin, CodeRabbit) - For n > 1, choices interleave in the stream; the request-wide cumulative counter would report other choices' tokens in each per-choice chunk. Track completion tokens per output index and report the current choice's running total in continuous usage. (dynamo-review-agent) Add a unit test asserting continuous_usage_stats is ignored without include_usage. Signed-off-by: nguyenthai-duong <63569535+nguyenthai-duong@users.noreply.github.com>
nguyenthai-duong
temporarily deployed
to
external_collaborator
GitHub Actions
Inactive
|
Thanks for the reviews. Addressed in b1dbdc1:
Added a unit test asserting Note: the |
1 participant
Overview
stream_options.continuous_usage_stats=truehad no effect when the frontendruns with
--dyn-chat-processor vllm: every intermediate streamed chunk cameback with
usage: null, and usage was reported only once on the final chunk.vLLM's and SGLang's native OpenAI servers emit a
usageobject on every chunkwhen this flag is set, so the vLLM chat processor path diverged from expected
OpenAI-compatible behavior.
Details
Root cause is in
VllmProcessor._generate_and_stream(
components/src/dynamo/frontend/vllm_processor.py). A chunk only received ausagefield when the engine reportedcompletion_usage, which the vLLM enginesends on the final chunk only:
The flag was never read on this path, so it was silently ignored.
The fix reads
stream_options.continuous_usage_statsfrom the request and, whenset, synthesizes a per-chunk
usageobject from the running token counters theprocessor already maintains (
input_tokens+cumulative_output_tokens).Engine-provided
completion_usagestill takes precedence on the final chunk, soauthoritative counts are preserved and the OpenAI-required final usage chunk is
unchanged.
Scope is intentionally minimal (8 lines of logic); it reuses existing counters
and adds no new abstractions.
Where should the reviewer start?
components/src/dynamo/frontend/vllm_processor.py— the two edited blocks in_generate_and_stream.Related Issues
N/A
Summary by CodeRabbit
usage statistics when
stream_options.continuous_usage_statsis enabled,matching vLLM/SGLang native OpenAI-server behavior.
Validation
pre-commit run --files components/src/dynamo/frontend/vllm_processor.py components/src/dynamo/frontend/tests/test_vllm_processor_unit.py— all hooks pass (isort, black, flake8, ruff, codespell, pytest-marker report).TestRoutedEnginePathintest_vllm_processor_unit.py, driving_generate_and_streamwith the existingFakeRoutedEngineharness (CPU-only, no GPU/model):usage{prompt_tokens, completion_tokens, total_tokens}completion_usageon final chunk → engine value used verbatim(
--dyn-chat-processor vllm): before, all content chunks returned"usage":null; after, per-chunk usage is populated while the final chunk isunchanged.
Summary by CodeRabbit
New Features
Bug Fixes