Skip to content

fix(extensions): reject truly empty length-truncated Chat Completions - #4912

Closed
fscfede-beep wants to merge 1 commit into
openai:mainfrom
fscfede-beep:fix/empty-length-chat-completions-4885
Closed

fix(extensions): reject truly empty length-truncated Chat Completions#4912
fscfede-beep wants to merge 1 commit into
openai:mainfrom
fscfede-beep:fix/empty-length-chat-completions-4885

Conversation

@fscfede-beep

Copy link
Copy Markdown

Summary

Successor to closed #4886 / addresses closed #4885.

The original change correctly rejected truly empty non-streaming Chat Completions responses ending with finish_reason="length", but the first implementation also rejected reasoning-only responses. This PR preserves the intended empty-response protection while retaining provider reasoning that is already surfaced by AnyLLM/LiteLLM.

Behavior

Raise ModelBehaviorError only when all of these hold:

  • finish_reason == "length"
  • no assistant text
  • no refusal
  • no tool calls
  • no provider reasoning (reasoning_content / thinking_blocks, or AnyLLM normalized reasoning)

Preserve partial text, refusals, tool calls, and reasoning-only responses.

Verification

Local regression suite for the affected adapters: 96 passed.

Static verification:

  • Ruff: PASS
  • Pyright: 0 errors, 0 warnings, 0 informations
  • mypy: PASS
  • git diff --check: PASS

Real-provider compatibility test, same provider/model/prompt and max_tokens=8 across pinned code states:

  • provider: Ollama 0.32.14 on isolated loopback server
  • model: rumbo-qwen3-4b-q4km:latest
  • base: 1d471a4775bf2f40179f411824da383deb4c3fca -> one ResponseReasoningItem
  • previous PR fix(extensions): raise on truncated empty Chat Completions responses #4886 head: 7a437cea5bf7a2e10194ee7239a25d87f59f31ad -> ModelBehaviorError
  • this fix: 504dcf93a52fe86732509a83ebb24f986b9f9133 -> one ResponseReasoningItem

Both AnyLLM and LiteLLM were tested end-to-end through the real Ollama OpenAI-compatible endpoint. The raw provider returned finish_reason="length", empty content, and non-empty reasoning.

The original PR was closed after the maintainer requested real-world compatibility evidence; that evidence is now available and the incompatible behavior has been corrected.

Copy link
Copy Markdown
Author

Live-provider compatibility evidence

Following the review concern on closed #4886, I ran the same non-streaming request end-to-end through both affected adapters against the same real local OpenAI-compatible provider.

Provider: Ollama 0.32.14, model rumbo-qwen3-4b-q4km:latest, isolated loopback 127.0.0.1:11435, temperature 0, max_tokens=8.

Raw provider result: finish_reason="length", empty visible content, non-empty reasoning (18 chars). No provider mock was used.

Pinned code-state comparison:

  • base 1d471a4775bf2f40179f411824da383deb4c3fca: AnyLLM -> one ResponseReasoningItem; LiteLLM -> one ResponseReasoningItem.
  • prior PR fix(extensions): raise on truncated empty Chat Completions responses #4886 head 7a437cea5bf7a2e10194ee7239a25d87f59f31ad: AnyLLM -> ModelBehaviorError; LiteLLM -> ModelBehaviorError.
  • this fix 504dcf93a52fe86732509a83ebb24f986b9f9133: AnyLLM -> one ResponseReasoningItem; LiteLLM -> one ResponseReasoningItem.

Additional verification on the fix: 96 affected tests passed; Ruff PASS; Pyright 0 errors/0 warnings/0 informations; mypy PASS; git diff --check PASS.

The experiment demonstrates the compatibility regression and that the reasoning-aware guard restores the prior provider behavior while still rejecting truly empty length-terminated turns.

Receipt: OPENAI_AGENTS_4886_LIVE_REASONING_EVIDENCE_20260907.json, SHA-256 546FDA775E97BCB123F6BFC64756194DDDA903AE802D6ABBDB0A80A15015B822.

Copy link
Copy Markdown
Author

@codex review

Please review the current head 504dcf93a52fe86732509a83ebb24f986b9f9133. Focus on whether the new finish_reason="length" guard is correctly scoped so truly empty responses raise while reasoning-only, refusal, tool-call, and partial-text responses remain valid. Also audit the AnyLLM/LiteLLM adapter normalization and provider-reasoning preservation for P0/P1/P2 correctness issues.

@chatgpt-codex-connector

Copy link
Copy Markdown

Note

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@fscfede-beep fscfede-beep left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review gate check on the current head 504dcf93a52fe86732509a83ebb24f986b9f9133: the proposed condition is appropriately narrow—raise only for truly empty non-streaming length-truncated responses, while preserving reasoning-only, refusal, tool-call, and partial-text responses. The PR documents 96 focused tests and real-provider AnyLLM/LiteLLM + Ollama compatibility evidence. No new review threads are currently present. I am not treating contributor-side validation as upstream CI PASS; the current workflow is action_required with no jobs. Please evaluate the provider compatibility evidence and semantics on the upstream CI/review gate.

@seratch

seratch commented Sep 8, 2026

Copy link
Copy Markdown
Member

Thanks for adding the real-provider compatibility evidence. The Ollama comparison addresses the reasoning-only concern from #4886: both the base and this PR return a reasoning item.

However, that experiment does not exercise the truly empty response targeted by the new exception. It demonstrates compatibility with existing behavior, rather than a user-visible problem that this change fixes. The original reproduction in #4885 supplies a configured response, so the real-world impact needed for this behavior change remains unestablished.

I'm going to close this PR for now. We can reconsider with a real-provider example containing no text, refusal, tool calls, or reasoning, together with the concrete application consequence that requires truncation-specific handling.

@seratch seratch closed this Sep 8, 2026

Copy link
Copy Markdown
Author

Targeted maintainer-review note: this revision is intentionally narrower than #4886. The guard raises only for finish_reason="length" with no text, refusal, tool calls, or provider reasoning; reasoning-only responses remain valid. Live-provider compatibility was exercised against Ollama 0.32.14 through both AnyLLM and LiteLLM (no mock): base and this head preserve ResponseReasoningItem, while #4886 raised ModelBehaviorError. 96 affected tests + Ruff/Pyright/mypy/diff-check pass locally. Please evaluate the semantic boundary and whether the live-provider evidence is sufficient for merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AnyLLM and LiteLLM return successful empty output for truncated responses

2 participants