Issues with Azure OpenAI

[Caroline-cyb]

When my prompt is like "A kitchen with a table. There is a fruit bowl on the table with a mix of apples and oranges. The bowl contains at least 6 fruits.", the outcome scene does not have tabletop objects as the prompt describes. In code, early_finish_min_score all set beyond band 7 and the max_critique_rounds is 2. Looking for advise to improve the success rate of tabletop objects . Thanks

[nepfaff]

Your scene doesn't have any small/objects manipulands, which indicates that something is going wrong. Do the logs show that the manipuland stage ran? Was the final output in combined_house dir?

It's hard to debug from this image alone. Would you be able to share a zip of the output dir for this experiment? If that is too heavy, then the main scene.log or room.log file might be sufficient to diagnose the issue.

[nepfaff]

It could also be useful to look at the renders here to see if renders for the manipuland stage exist. For your scene, I'd expect multiple render subdirs with `manipulands_` prefix. (This is how the output dir should look like under `outputs/`)

[Caroline-cyb]

Your scene doesn't have any small/objects manipulands, which indicates that something is going wrong. Do the logs show that the manipuland stage ran? Was the final output in combined_house dir?

It's hard to debug from this image alone. Would you be able to share a zip of the output dir for this experiment? If that is too heavy, then the main scene.log or room.log file might be sufficient to diagnose the issue.

Thank you for your reply, here is my experiment.log. it seems has some errors but never stop and I do have manipulands_ prefix folders.

experiment.log

[nepfaff]

The log reveals what happened: The agent tried to place your manipulands but there was an issue with the Azure OpenAI endpoint that crashed it.

My initial assumption is that the current summarization implementation doesn't play cleanly with the Azure endpoint that is stricter than OpenAI's own endpoint which can lead to crashes. I can't test this directly as I don't have credits for the Azure API. However, if my assumption about the summarization is correct, then I should be able to fix that.

I will give this a go. You will then be able to retry from the manipuland stage, so that you don't have to run from start (you could already do this if the Azure thing is a stochastic thing).

[nepfaff]

It might already work if you use a gpt5 model for summarization. The 4o model doesn't support thinking and hence it crashes:

Summarizing turn 0 with azure_openai/gpt-4o-mini
2026-03-16 09:14:56,476 - httpx - INFO - HTTP Request: POST http://model.mify.ai.srv/v1/responses "HTTP/1.1 400 Bad Request"
2026-03-16 09:14:56,476 - scenesmith.agent_utils.turn_trimming_session - ERROR - Summarization failed: Error code: 400 - {'error': {'code': '400', 'message': 'Request Error', 'param': '{\n  "error": {\n    "message": "Unsupported parameter: \'reasoning.effort\' is not supported with this model.",\n    "type": "invalid_request_error",\n    "param": "reasoning.effort",\n    "code": "unsupported_parameter"\n  }\n}', 'type': ''}, 'request_id': 'ba06eb6d-2e7f-4661-8f69-fa483d35417f'}

[Caroline-cyb]

It might already work if you use a gpt5 model for summarization. The 4o model doesn't support thinking and hence it crashes:

Summarizing turn 0 with azure_openai/gpt-4o-mini
2026-03-16 09:14:56,476 - httpx - INFO - HTTP Request: POST http://model.mify.ai.srv/v1/responses "HTTP/1.1 400 Bad Request"
2026-03-16 09:14:56,476 - scenesmith.agent_utils.turn_trimming_session - ERROR - Summarization failed: Error code: 400 - {'error': {'code': '400', 'message': 'Request Error', 'param': '{\n  "error": {\n    "message": "Unsupported parameter: \'reasoning.effort\' is not supported with this model.",\n    "type": "invalid_request_error",\n    "param": "reasoning.effort",\n    "code": "unsupported_parameter"\n  }\n}', 'type': ''}, 'request_id': 'ba06eb6d-2e7f-4661-8f69-fa483d35417f'}

Make sense. I will try gpt 5 to summarize. i will give you follow up if it works.

[nepfaff]

There is a 2nd issue where tool calls/ tool responses are not treated as atomic units in summarization. Azure might be strict on validation here and crash it. I'm looking into robustifying this

[nepfaff]

There is a 2nd issue where tool calls/ tool responses are not treated as atomic units in summarization. Azure might be strict on validation here and crash it. I'm looking into robustifying this

I was wrong on this. Double checked this and it seems fine.

The fatal error is

openai.BadRequestError: Error code: 400 - {'error': {'code': '400', 'message': 'Request Error', 'param': '[models] Bad Request Error, Error code: 400 - {\'error\': {\'message\': "An assistant message with \'tool_calls\' must be followed by tool messages responding to each \'tool_call_id\'. The following tool_call_ids did not have response messages: call_zAOVmQ6jnssLHIkYAyKVvTlu", \'type\': \'invalid_request_error\', \'param\': \'messages\', \'code\': None}} (model: gpt-5, provider: azure_openai)', 'type': ''}, 'request_id': 'faafacb3-cb29-47e8-867c-4f5e83d91488'}

One possible case might be that Azure OpenAI doesn't play well with non-string return types. I'm struggling to figure this out right now.

With some luck, it's a downstream effect of the summarization error and works after that model switch.

If not, happy to work with you on this. The main issue is that I'm struggling to replicate your setup, as I don't have access to the Azure API. It could be very useful to catch the exact payload that crashed Azure. I'm not sure if Azure has logging tools for that.

[Caroline-cyb]

There is a 2nd issue where tool calls/ tool responses are not treated as atomic units in summarization. Azure might be strict on validation here and crash it. I'm looking into robustifying this

I was wrong on this. Double checked this and it seems fine.

The fatal error is

openai.BadRequestError: Error code: 400 - {'error': {'code': '400', 'message': 'Request Error', 'param': '[models] Bad Request Error, Error code: 400 - {\'error\': {\'message\': "An assistant message with \'tool_calls\' must be followed by tool messages responding to each \'tool_call_id\'. The following tool_call_ids did not have response messages: call_zAOVmQ6jnssLHIkYAyKVvTlu", \'type\': \'invalid_request_error\', \'param\': \'messages\', \'code\': None}} (model: gpt-5, provider: azure_openai)', 'type': ''}, 'request_id': 'faafacb3-cb29-47e8-867c-4f5e83d91488'}

One possible case might be that Azure OpenAI doesn't play well with non-string return types. I'm struggling to figure this out right now.

With some luck, it's a downstream effect of the summarization error and works after that model switch.

If not, happy to work with you on this. The main issue is that I'm struggling to replicate your setup, as I don't have access to the Azure API. It could be very useful to catch the exact payload that crashed Azure. I'm not sure if Azure has logging tools for that.

[Caroline-cyb]

sorry，I accidentally close this issue. and i'm verifying if this caused by summarization error. we'll see what's happen

[Caroline-cyb]

Now，I confirm it's Azure thing. Azure supports '/v1/chat/completions' api rather than /v1/responses. I'm trying to fix this. Just for your information:)

[nepfaff]

Ah, that makes sense then. At some point, I switched everything over to response API as that seemed to be the thing that OpenAI was pushing for.

Hope you manage to get it working with the completions api!