Fix eval_duration None TypeError in format_utils.py (#162) - #167
Open
AmirF194 wants to merge 1 commit into
Open
Conversation
ollama_chat_to_openai_v1_chat_completion and the streaming final-chunk handler both read eval_duration/prompt_eval_duration/total_duration/ load_duration with dict.get(key, 0), which only supplies the default when the key is absent. A short or aborted generation, or a prompt-cache hit, returns these keys present with value None, and 'None > 0' raises TypeError, killing the worker (NotPunchnox#162). Replace the plain get(key, 0) with get(key) or 0 for all four fields in both code paths so a None value falls back to 0 the same way a missing key already does.
Author
|
Checking in on this one week later, no rush at all. It fixes a real crash: eval_duration (and the other duration fields) can come back None on a short or cached generation, and the current get(key, 0) default only covers a missing key, not a None value, so the request handler throws. Happy to adjust anything if the fix shape needs changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause:
ollama_chat_to_openai_v1_chat_completionand the streaming final-chunkhandler in
format_utils.pyread the four duration fields withollama_response.get("eval_duration", 0)(and the three siblings).dict.get'sdefault only applies when the key is absent, not when it is present with value
None. A short or aborted generation, or a prompt-cache hit, returns these keyspresent and
None, soeval_duration_ns > 0raisesTypeError: '>' not supported between instances of 'NoneType' and 'int'and theworker dies, exactly as reported in #162.
Fix:
get(key) or 0in place ofget(key, 0)for all four duration fields, inboth the non-streaming function and the streaming path's final chunk, so a
Nonevalue falls back to 0 the same way a missing key already does.Note: PR #164 fixed the identical crash in
server_utils.py, but never touchedformat_utils.py, so this specific path stayed broken; the two fixes areindependent.
Verification:
test_format_utils.py(loaded by file path, sincerkllama/api/__init__.pypulls in the RKNN NPU runtime at import time, which is unavailable off Rockchip
hardware): 3 cases with a duration key set to
None, on both the chat and thestreaming path, fail on
mainwith the exact reportedTypeErrorand pass onthis branch; a 4th case confirms real durations still convert correctly.
python:3.12-slimcontainer with only format_utils.py's owndependencies installed (flask, cv2, numpy, pillow, requests, pydantic); no NPU
hardware needed since the bug is in response formatting, not inference.
Nonedurationend-to-end (no NPU hardware in this environment); the repro constructs that
input directly, matching the shapes described in TypeError: '>' not supported between 'NoneType' and 'int' in format_utils.py when eval_duration is None → worker dies #162 and by the merged FIx issue #162 with eval_duration None #164.
Fixes #162