Skip to content

fix(extensions): raise on truncated empty Chat Completions responses - #4886

Closed
CoralGarden52 wants to merge 1 commit into
openai:mainfrom
CoralGarden52:fix/extensions-truncated-empty-chat
Closed

fix(extensions): raise on truncated empty Chat Completions responses#4886
CoralGarden52 wants to merge 1 commit into
openai:mainfrom
CoralGarden52:fix/extensions-truncated-empty-chat

Conversation

@CoralGarden52

Copy link
Copy Markdown

Summary

This pull request fixes AnyLLM and LiteLLM non-streaming Chat Completions adapters returning successful empty responses when a completion terminates with finish_reason="length" before emitting assistant text, refusal, or tool calls.

Both adapters now raise ModelBehaviorError after usage and tracing data are recorded, matching the built-in OpenAI Chat Completions adapter. Responses containing partial text, refusals, or tool calls remain supported.

Test plan

  • Added regression tests for empty content=None and content="".
  • Added coverage for partial text, refusals, tool calls, and missing LiteLLM finish_reason.
  • Added tracing coverage to verify usage is recorded before ModelBehaviorError is raised.
  • Focused adapter tests: 101 passed.
  • Full parallel test suite: 9360 passed, 29 skipped.
  • Serial test suite: 77 passed, 4 skipped.
  • Formatting, lint, mypy, Pyright, and the final code-change verification script passed.

Live verification:

  • Model: gpt-5.6-luna
  • temperature=0
  • max_tokens=256

The raw Chat Completions response returned finish_reason="length" with content=null. The built-in adapter raised ModelBehaviorError, while AnyLLM and LiteLLM previously returned successful empty outputs.

Issue number

Closes #4885

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. The new condition also rejects length-truncated responses containing reasoning but no assistant text. The existing adapters can return that reasoning as a ResponseReasoningItem, so this may change more than empty-response handling. Please provide a real-provider comparison for a reasoning-only truncated response through the affected adapters, showing the base and head outputs. We need that evidence to assess compatibility before merging.

@fscfede-beep

Copy link
Copy Markdown

I independently reproduced the review concern in a local fork: a finish_reason=length guard that only checks assistant text/refusal/tool calls also rejects reasoning-only completions that the adapters can convert to a ResponseReasoningItem.

I tested a reasoning-aware variant at fscfede-beep/openai-agents-python@504dcf9:

  • AnyLLM also requires _extract_any_llm_reasoning_text(message) to be empty before raising.
  • LiteLLM also requires both reasoning_content and thinking_blocks to be empty before raising.
  • Added reasoning-only finish_reason=length regression coverage for both adapters.
  • Local affected-suite verification: 96 passed; Ruff, Pyright, mypy, and git diff --check pass.

Branch: https://github.com/fscfede-beep/openai-agents-python/tree/fix/empty-length-chat-completions-4885

I have not run the requested live paid-provider base/head comparison, so this is only code-level compatibility evidence, not a substitute for that live verification.

@seratch

seratch commented Sep 7, 2026

Copy link
Copy Markdown
Member

Without confirming real-world impact, we won't move forward with this possibly breaking change to existing use cases.

@seratch seratch closed this Sep 7, 2026

Copy link
Copy Markdown

I now have a real-provider base/head comparison for the reasoning-only truncation case.

Provider/model used:

  • Ollama 0.32.14, isolated local server on loopback
  • rumbo-qwen3-4b-q4km:latest (Qwen3 4B)
  • temperature = 0
  • max output tokens = 8
  • prompt held constant across runs

Raw provider behavior is the case from the review concern: the OpenAI-compatible endpoint returns finish_reason="length", empty assistant content, and non-empty reasoning (reasoning_len=18). I am intentionally not posting the reasoning text itself.

Pinned code states:

AnyLLM, end-to-end against the real Ollama provider:

  • base -> returns one ResponseReasoningItem (output_types=["reasoning"], output_tokens=8)
  • PR fix(extensions): raise on truncated empty Chat Completions responses #4886 -> raises ModelBehaviorError: Chat Completions response terminated with finish_reason='length' but produced no assistant text, tool call, or refusal.
  • reasoning-aware variant -> returns one ResponseReasoningItem (output_types=["reasoning"], output_tokens=8)

LiteLLM, end-to-end via Ollama's real OpenAI-compatible /v1/chat/completions endpoint:

I also independently called litellm.acompletion() against that endpoint; it returned in ~6.65s with finish_reason="length", content_len=0, and reasoning_content_len=18, confirming the LiteLLM provider response itself carries the reasoning before the Agents SDK adapter processes it.

So this is a concrete compatibility break in an existing supported path: before #4886, a length-truncated reasoning-only completion is surfaced as a reasoning item; at the current PR head it becomes an exception. The reasoning-aware guard at 504dcf93... preserves the previous behavior while still rejecting truly empty length-truncated responses.

This is local real-provider evidence, not a hosted/paid-provider claim. The branch remains: https://github.com/fscfede-beep/openai-agents-python/tree/fix/empty-length-chat-completions-4885

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AnyLLM and LiteLLM return successful empty output for truncated responses

3 participants