Skip to content

Fix eval_duration None TypeError in format_utils.py (#162) - #167

Open
AmirF194 wants to merge 1 commit into
NotPunchnox:mainfrom
AmirF194:fix/162-eval-duration-none-format-utils
Open

Fix eval_duration None TypeError in format_utils.py (#162)#167
AmirF194 wants to merge 1 commit into
NotPunchnox:mainfrom
AmirF194:fix/162-eval-duration-none-format-utils

Conversation

@AmirF194

Copy link
Copy Markdown

Root cause: ollama_chat_to_openai_v1_chat_completion and the streaming final-chunk
handler in format_utils.py read the four duration fields with
ollama_response.get("eval_duration", 0) (and the three siblings). dict.get's
default only applies when the key is absent, not when it is present with value
None. A short or aborted generation, or a prompt-cache hit, returns these keys
present and None, so eval_duration_ns > 0 raises
TypeError: '>' not supported between instances of 'NoneType' and 'int' and the
worker dies, exactly as reported in #162.

Fix: get(key) or 0 in place of get(key, 0) for all four duration fields, in
both the non-streaming function and the streaming path's final chunk, so a
None value falls back to 0 the same way a missing key already does.

Note: PR #164 fixed the identical crash in server_utils.py, but never touched
format_utils.py, so this specific path stayed broken; the two fixes are
independent.

Verification:

  • Added test_format_utils.py (loaded by file path, since rkllama/api/__init__.py
    pulls in the RKNN NPU runtime at import time, which is unavailable off Rockchip
    hardware): 3 cases with a duration key set to None, on both the chat and the
    streaming path, fail on main with the exact reported TypeError and pass on
    this branch; a 4th case confirms real durations still convert correctly.
  • Ran in a clean python:3.12-slim container with only format_utils.py's own
    dependencies installed (flask, cv2, numpy, pillow, requests, pydantic); no NPU
    hardware needed since the bug is in response formatting, not inference.
  • Not verified: the actual RK3588 inference path that produces a None duration
    end-to-end (no NPU hardware in this environment); the repro constructs that
    input directly, matching the shapes described in TypeError: '>' not supported between 'NoneType' and 'int' in format_utils.py when eval_duration is None → worker dies #162 and by the merged FIx issue #162 with eval_duration None #164.

Fixes #162

ollama_chat_to_openai_v1_chat_completion and the streaming final-chunk
handler both read eval_duration/prompt_eval_duration/total_duration/
load_duration with dict.get(key, 0), which only supplies the default
when the key is absent. A short or aborted generation, or a
prompt-cache hit, returns these keys present with value None, and
'None > 0' raises TypeError, killing the worker (NotPunchnox#162).

Replace the plain get(key, 0) with get(key) or 0 for all four fields
in both code paths so a None value falls back to 0 the same way a
missing key already does.
@AmirF194

AmirF194 commented Aug 7, 2026

Copy link
Copy Markdown
Author

Checking in on this one week later, no rush at all. It fixes a real crash: eval_duration (and the other duration fields) can come back None on a short or cached generation, and the current get(key, 0) default only covers a missing key, not a None value, so the request handler throws. Happy to adjust anything if the fix shape needs changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TypeError: '>' not supported between 'NoneType' and 'int' in format_utils.py when eval_duration is None → worker dies

1 participant