[lezama]

What

Runs against the newest Anthropic models (claude-opus-5-5, claude-sonnet-5-5, claude-fable-5-1) fail on every test. The same happens with gpt-5 and o-series models, which LiteLLM rejects with slightly different wording ("don't support temperature="). LiteLLM 1.96 rejects temperature=0 for these models before the request goes out, with an UnsupportedParamsError: "does not support temperature=0. Only temperature=1 is supported."

wp-bench already drops temperature and retries when Anthropic answers "temperature is deprecated". This error has different text, so the fallback never ran and the error was re-raised. The console fills with LiteLLM's "Give Feedback / Get Help" banner, one per failed test.

Fix

• Treat both messages as "this model does not take temperature", then drop it and call again.
• Remember that per model, so later calls skip temperature instead of failing first every time. Records still carry temperature_fallback: true.

Out of scope

• top_p: LiteLLM rejects it on the same models when set, but the default is unset, so default configs are not affected.
• Run-level visibility: the result's model block still shows the configured temperature. The per-record temperature_fallback flag is the signal; the frozen-benchmark work in feat: define one public benchmark release contract #78 already excludes those records.

How to test

• pytest python -q: two new tests in test_models.py. One uses the real LiteLLM exception and checks that only the first call sends temperature. The other checks that an unrelated bad request is still raised.
• Manually: run any Anthropic model above. Before this change, it fails with UnsupportedParamsError. After it, it runs, and the banner appears once at most.

Workaround until this lands: set temperature: 1 on those models in the config.

🤖 Generated with Claude Code

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: lezama <migueluy@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

---

Props Bot commands

You can trigger Props Bot actions by commenting on this PR:

• @propsbot add <username> to give props to a GitHub user. Requires write access.
• @propsbot remove <username> to leave a user out of the props. Requires write access.
• @propsbot refresh to refresh this list. Adding the props-bot label works too.