ToolOutputImage results silently dropped from tool messages

[pauamargant]

ToolOutputImage results silently dropped from tool messages (openai-agents ≤0.6.4)

We ran into a subtle bug where tools returning ToolOutputImage appear to succeed but the model never actually sees the image.

We tracked it down to Converter.extract_text_content() in openai_chatcompletions.py, which filters function_call_output content to type == "text" only. Any input_image blocks are silently discarded before the message goes to the API.

We realized that this is an issue of openai-agents version 0.64 and is already fixed upstream in the SDK: openai/openai-agents-python#2214, released in v0.6.5. The fix adds a preserve_tool_output_all_content flag (defaults to True) so image content survives the converter.

The fix on SceneSmith's side would just be bumping the pin:

-"openai-agents>=0.6.4,<0.7",
+"openai-agents>=0.6.5,<0.7",

[nepfaff]

I now investigated this and the issue only was for the Chat Completions API. SceneSmith uses the Response API and everything seems to work fine with the provided code and depencency versions. We know that the Chat Completions API is broken (see #7).

Did you encounter this somehow with the default SceneSmith settings or did you make some changes to discover this?

[nepfaff]

I made some code changes that should now support Chat Completions API (e.g., when forced to use Azure): 67cc408

Let me know if that fixes your issues.