Same request should return multiple generations at least for the first turn as it is same across all repetitions

[SushantGautam]

n=X means X generated outputs for the same request.

in any_llm

completion(
...
    n=X,
)

Can save tokens

[avalyset]

_call_async calls acompletion(**kwargs) with no n and only reads choices[0], so you'd also need to consume the extra choices. The catch is provider asymmetry (same as #1095/#1102): any-llm's OpenAI path forwards n natively, but Anthropic's Messages API has no n, and the anthropic _convert_params doesn't exclude it, so n leaks into messages.create() and errors. Subjects and the default judge run on Anthropic, so that path has to be handled regardless.

Two ways: pass n through where the provider supports it and fall back to an N-call loop on Anthropic, or just do the N-call loop everywhere. The token saving (the first turn is identical across reps) only shows up with the first option on providers that support n — the second is mostly just tidier code. Which way do you want it? I'll send a PR with a test once you pick.

[SushantGautam]

hmmm. . not that much saving then. . ;)
lets keep in on hold