DCI-Agent-Lite with gpt-5.4-nano has the problem of reproducing issues.

[Rinnolo]

According to the settings in DCI-Agent-Lite/scripts/bcplus_eval/run_bcplus_eval_openai.sh , when thinking-level is set to high and using the gpt-5.4-nano model, severe refusal-to-answer behavior occurs. Specifically, the model eventually refuses to answer due to insufficient data, resulting in metrics significantly lower than the results in the repository.

[Zhuofeng-Li]

Thanks for reporting this.

Could you please share the full configuration you used, including the exact command, model name, thinking-level, API parameters, commit hash, and any environment variables that may affect decoding or retry behavior? This would help us reproduce the refusal-to-answer issue more precisely.

For reference, the evaluation logs in our reported runs are available here:

https://huggingface.co/datasets/DCI-Agent/eval-logs/tree/main/dci-agent-lite/browsecomp-plus