Skip to content

fix(llm): keep finish_reason and empty content for tool responses without calls - #5633

Merged
qinxuye merged 1 commit into
xorbitsai:mainfrom
MohammadHijjawi97:fix/tool-response-truncated-thinking
Oct 5, 2026
Merged

qinxuye merged 1 commit into
xorbitsai:mainfrom
MohammadHijjawi97:fix/tool-response-truncated-thinking

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

What

ChatModelMixin._post_process_completion (non-streaming chat with tools, used by the vLLM, SGLang, MLX, llama.cpp and transformers engines) has two problems when the model does not call a tool:

  • finish_reason is always "stop", so a response cut off by max_tokens is reported as "stop" instead of "length". The streaming tool path and the request without tools keep "length".
  • When the output only contains reasoning (e.g. max_tokens is reached inside the thinking block), the reasoning parser returns empty content, but the if processed_content: check then keeps the raw output, so the whole reasoning is returned again as content.

Example with Qwen3 (reasoning_content=True), output cut off while thinking:

Before:

with tools   : {"content": "<think>\nThe user wants the weather, I should call", "tool_calls": [], "reasoning_content": "<think>\nThe user wants the weather, I should call"}, "finish_reason": "stop"
without tools: {"content": "", "reasoning_content": "<think>\nThe user wants the weather, I should call"}, "finish_reason": "length"

After:

with tools   : {"content": "", "tool_calls": [], "reasoning_content": "<think>\nThe user wants the weather, I should call"}, "finish_reason": "length"

Fix

Always use the reasoning parser's content for tool parsing, and use the engine's finish_reason when there is no tool call ("stop" if it is missing). Responses with tool calls are unchanged.

Tests

  • Added test_post_process_completion_stopped_while_thinking (fails before the fix, passes after).
  • pytest xinference/model/llm/tests/test_utils.py xinference/model/llm/tool_parsers/tests passes.
  • black / ruff / isort / codespell / mypy (pre-commit) clean on changed files.

…hout calls

When a non-streaming request with tools ends without a tool call,
_post_process_completion always reported finish_reason "stop", and when
the output only contained reasoning (e.g. max_tokens reached inside the
thinking block) the raw output was returned as content as well as
reasoning_content.

Use the engine's finish_reason when there is no tool call and always
use the reasoning parser's content, matching the request without tools
and the streaming tool path.
@XprobeBot XprobeBot added the bug Something isn't working label Oct 4, 2026
@XprobeBot XprobeBot added this to the v3.x milestone Oct 4, 2026

@qinxuye qinxuye left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Verified the reasoning-only empty-content handling and preservation of the engine finish_reason when no tool is called. All 176 focused utility and tool-parser tests pass locally. The sole CI failure is test_fatal_engine_error_closes_model in the image engine, outside the changed LLM post-processing path.

@qinxuye
qinxuye merged commit d201b40 into xorbitsai:main Oct 5, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants