fix: model finishing with no tool calls gets misreported as a decode error - #1348
Open
zambalee wants to merge 1 commit into
Open
Conversation
… error OpenAICompletionsHandler._parse_query_response_FC() (and the same copy-pasted pattern in MistralHandler / FireworksHandler) wraps the tool_calls extraction in a bare `except:` and falls back to the raw text `content` whenever `tool_calls` is falsy. This isn't only an error path -- a model that has already finished the task and replies in plain text (no further tool calls needed) hits the exact same fallback. That raw string then flows into decode_execute() -> convert_to_function_call(), which assumes a list[dict] of function calls. Iterating a string yields its characters, and calling `.items()` on each one raises `'str' object has no attribute 'items'` -- silently miscounted as "Failed to decode the model response" even though the model behaved correctly. Found while running BFCL v4 multi_turn locally: 100% of the `'str' object has no attribute 'items'` errors across three different models/backends traced back to this single fallback path, not to any model or serving-engine issue. Fix: check `tool_calls` explicitly and return `model_responses = []` when there are none, matching what decode_execute/ is_empty_execute_response already expect for "no further tool calls." Co-authored-by: Claude Code Sonnet-5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
OpenAICompletionsHandler._parse_query_response_FC()— and the samecopy-pasted pattern in
MistralHandlerandFireworksHandler—wraps tool_call extraction in a bare
except::When a model has legitimately finished a multi-turn task and replies
with plain text (no further tool calls needed — completely valid
behavior),
tool_callsisNone, the list comprehension raisesTypeError: 'NoneType' object is not iterable, and the bareexceptfalls back to treating the raw text
contentasmodel_responses.That string is then passed to
decode_execute()→convert_to_function_call(), which assumes alist[dict]. Pythonhappily iterates a string character-by-character, and calling
.items()on each single-character string raises:which gets caught by the outer handler and logged as
"Failed to decode the model response"— indistinguishable from anactual model/serving failure, even though the model did nothing
wrong.
Impact
Found while running BFCL v4
multi_turnlocally across threedifferent models on three different backends/hardware:
openmmlab/lmdeploy:latest)vllm/vllm-openai:v0.24.0)The
'str' object has no attribute 'items'error was the onlyerror type observed in the affected result files, at meaningful
volume across all three (different engines, different hardware):
Since this reproduces identically across two engines (vLLM, LMDeploy)
and three hardware platforms, it's clearly the harness's own bug, not
an engine/model/quantization issue. It likely understates Multi-Turn
accuracy for any FC-mode model evaluated through
OpenAICompletionsHandler,MistralHandler, orFireworksHandlerwhenever the model correctly stops calling tools mid-conversation.
Fix
Check
tool_callsexplicitly instead of relying on exceptionfallback. When there are none, return
model_responses = [], whichis_empty_execute_response()already recognizes as "no further toolcalls" and handles as a normal turn-ending case — no exception, no
misreported error.
Testing
No existing unit tests cover this handler. Verified manually with a
mocked
api_response:tool_calls=None+ plain-textcontent→model_responses == [](previously: raw string, crashing downstream)
tool_callspresent → output unchanged from before this fixScope note
CohereHandleralready does this correctly (explicitif len(tool_calls) > 0check) and wasn't touched.local_inference/minicpm_fc.pyhas a structurally different(non-exception-based) fallback to raw
contentthat may warrant aseparate look, but I didn't want to touch a different code path/model
family without dedicated testing — flagging here for maintainers'
awareness rather than bundling an unverified change into this PR.
Co-authored with Claude Code (Sonnet 5).