fix(endpoints): count OpenAI Responses streaming output tokens once, not twice

[ajcasagrande]

Summary

The OpenAI Responses endpoint double-counts client-side output tokens on every streaming turn. Output sequence length (OSL) and output-token-throughput are reported at roughly 2× their true value against any compliant streaming server.

The bug

A streaming Responses turn delivers the assistant's text twice on the wire:

• as a chain of incremental response.output_text.delta events (delta = the next text chunk), and
• once more, in full, as a terminal response.output_text.done event (text = the complete finalized text).

ResponsesEndpoint._streaming_event_data maps both to a TextResponseData:

if event_type == "response.output_text.delta":
    delta = json_obj.get("delta")
    return TextResponseData(text=delta) if delta else None
...
if event_type == "response.output_text.done":
    text = json_obj.get("text")
    return TextResponseData(text=text) if text else None

The base extract_response_data calls parse_response for every event in the record, so InferenceResultParser concatenates and tokenizes the delta chain and the done text. The streamed output is counted once from the deltas and again from the done.

Concrete example — a turn that streams "The quick brown fox":

Event	Emitted text
output_text.delta	"The quick "
output_text.delta	"brown fox"
output_text.done	"The quick brown fox" ← full text again

Tokenized output = "The quick brown fox" + "The quick brown fox" = 2× OSL.

The fix

Override extract_response_data so the terminal done is treated as a structural envelope once any delta has carried text, while remaining the sole carrier when no delta did (a server that emits only the terminal event, or the non-streaming convenience field):

def extract_response_data(self, record: RequestRecord) -> list[ParsedResponse]:
    parsed: list[ParsedResponse] = []
    saw_output_text_delta = False
    for response in record.responses:
        json_obj = response.get_json()
        if not json_obj:
            continue
        if isinstance(json_obj, dict):
            event_type = json_obj.get("type")
            if event_type == "response.output_text.delta" and json_obj.get("delta"):
                saw_output_text_delta = True
            elif event_type == "response.output_text.done" and saw_output_text_delta:
                continue  # deltas already carried this text; don't count it twice
        if result := self._route_parsed_json(json_obj, response.perf_ns):
            parsed.append(result)
    return parsed

done is now a fallback: counted only when the deltas didn't already carry the text. The single forward pass is correct because done always trails its deltas in SSE arrival order, which record.responses preserves. parse_response itself is unchanged — the per-event callers (first-token detection, request latency) treat done as a plain data event and never sum tokens, so they see no behavioral change.

Validation

• Real traffic: replayed 2,399 streamed messages captured from live OpenAI Responses streams through the actual parser. concat(deltas) == done.text held on 2,399/2,399; pre-fix output tokens = 2.0000× ground truth, post-fix = 1.0000×.
• Regression test: tests/unit/endpoints/test_openai_responses_streaming_dedup.py pins: deltas + a terminal done tokenize to the delta-only count; a done-only stream (no deltas) still counts once; reasoning and tool-call streams are unaffected.

Test plan

uv run pytest tests/unit/endpoints/test_openai_responses_streaming_dedup.py

Summary by CodeRabbit

• Bug Fixes

  • Prevented duplicated assistant output when streamed deltas are followed by a terminal completion event repeating the full text.
  • Improved response parsing by deserializing once and routing streaming vs non-streaming payloads appropriately, while ignoring invalid/non-JSON bodies.
  • Ensured de-duplication applies per output part, with reasoning and function/tool arguments streamed and tokenized independently.

• Tests

  • Added regression tests covering deduplication scenarios (streaming/non-streaming, done-only, sibling parts, reasoning vs output, and function-call argument streaming).

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@e774a7f3d688334ca0f3bd719a860a4bb86806e6

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@e774a7f3d688334ca0f3bd719a860a4bb86806e6

---

Last updated for commit: e774a7f • Browse code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8dcb61d2-e49c-424f-9b36-13bbcacde1ea

📥 Commits

Reviewing files that changed from the base of the PR and between 89bf792 and e774a7f.

📒 Files selected for processing (2)

• src/aiperf/endpoints/openai_responses.py
• tests/unit/endpoints/test_openai_responses_streaming_dedup.py

🚧 Files skipped from review as they are similar to previous changes (2)

• src/aiperf/endpoints/openai_responses.py
• tests/unit/endpoints/test_openai_responses_streaming_dedup.py

---

Walkthrough

Responses parsing now shares a JSON routing path and suppresses duplicate streamed output text when done repeats text already seen in delta. New regression tests cover streaming, non-streaming, reasoning, and tool-argument cases.

Changes

Streaming Output Text De-duplication

Layer / File(s)	Summary
Parse routing and de-duplication logic src/aiperf/endpoints/openai_responses.py	parse_response now delegates to _route_parsed_json; extract_response_data tracks seen output text per part and suppresses redundant response.output_text.done, with comments describing the sole-carrier fallback path.
Regression tests for dedup behavior tests/unit/endpoints/test_openai_responses_streaming_dedup.py	Helpers and tests cover delta+done de-duplication, full-response extraction, done-only fallback, per-part scoping, reasoning/output separation, and function-call-argument streaming behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

A delta hopped in, then a “done” came by,
This bunny kept count with a careful eye.
One text, one tally, no doubled confetti,
The tests all agree: the hop stays steady 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main fix: preventing double-counting of OpenAI Responses streaming output tokens.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/aiperf/endpoints/openai_responses.py`:
- Around line 306-322: The dedup logic in the response parsing loop is too broad
because a single saw_output_text_delta flag in the parsing method suppresses all
later response.output_text.done events, even for different (output_index,
content_index) pairs. Update the logic in the response handling path around
_route_parsed_json so deltas and terminal done events are tracked per
output/content pair instead of globally, and only skip a done event when the
matching pair has already seen a delta. Add a regression test covering multiple
output/content parts to verify one stream item does not suppress another.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 698ab980-0b04-4e01-ac1b-13a45aee5f8a

📥 Commits

Reviewing files that changed from the base of the PR and between d41bd50 and 26e1a65.

📒 Files selected for processing (2)

• src/aiperf/endpoints/openai_responses.py
• tests/unit/endpoints/test_openai_responses_streaming_dedup.py

[dynamo-review-agent]

Skipping the duplicate response.output_text.done event removes its perf_ns from ParsedResponseRecord.content_responses, so RequestLatencyMetric and derived streaming metrics use the last delta timestamp while the worker still uses the done timestamp. Fix: preserve a non-tokenizing parsed response at the done event timestamp while excluding its text from client-side tokenization.

🤖 AI Fix

In src/aiperf/endpoints/openai_responses.py, update ResponsesEndpoint.extract_response_data so the duplicate response.output_text.done branch appends ParsedResponse(perf_ns=response.perf_ns, data=TextResponseData(text="")) and then continues instead of only continue, and add a regression in tests/unit/endpoints/test_openai_responses_streaming_dedup.py asserting token text is not duplicated while content timing still includes the done event timestamp.

[codecov]

Codecov Report

❌ Patch coverage is 86.36364% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/aiperf/endpoints/openai_responses.py	86.36%	1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!