fix(nim): retry transient HTTP errors and report actual status code

[Aftabbs]

Summary

Fixes #1967 — NIM generator aborts the entire probe on HTTP 408 (request timeout) with a misleading error message instead of retrying.

Two root causes, both fixed here:

1. openai.APIStatusError was not caught by the backoff decorator

OpenAICompatible._call_model uses @backoff.on_exception to retry on transient network failures, but the decorator only listed RateLimitError, InternalServerError, APITimeoutError, and APIConnectionError. NIM raises a bare openai.APIStatusError for HTTP 408, which fell through the decorator and bubbled up unretried.

Fix: Add openai.APIStatusError to the exception tuple, with a giveup=_is_terminal_api_error predicate that retries only transient codes (408, 502, 503, 504) and immediately surfaces terminal errors (400, 401, 403, 404, 409, 422).

2. The catch-all swallowed the HTTP status code

NIM._call_model's bare except Exception block replaced the actual error with "NIM generation failed. Is the model name spelled correctly?" — losing the HTTP status code and URL that would identify the real problem.

Fix: Split the handler: openai.APIStatusError now logs and re-raises with the actual status code and request URL; the fallback branch includes the exception type name.

Changes

File	Change
garak/generators/openai.py	Add _TRANSIENT_HTTP_CODES frozenset and _is_terminal_api_error() giveup predicate; extend backoff decorator to catch APIStatusError
garak/generators/nim.py	Split catch-all into openai.APIStatusError (reports HTTP status) + Exception (reports type name)
tests/generators/test_nim.py	15 new unit tests for retry logic and error message content

Testing

pytest tests/generators/test_nim.py -v -k "not nim_instantiate and not nim_generate and not nim_parallel and not nim_hf and not nim_conservative"
# 16 passed, 5 deselected

The 5 deselected tests require a live NIM_API_KEY and are unchanged.

[jmartin-tech]

DCO signoff is missing from the commit message and the approach here has already been noted an incorrect in a previous PR.

Please see #1973, this PR duplicates the original offering in that PR and needs to account for the same feedback.

[manunicholasjacob]

Nice catch on 408 arriving as APIStatusError rather than InternalServerError.

One thing worth checking: giveup= applies to every exception in the on_exception tuple, not only the newly added APIStatusError. Since RateLimitError and InternalServerError both subclass APIStatusError, _is_terminal_api_error returns True for them whenever their status is not in _TRANSIENT_HTTP_CODES:

RateLimitError       (429) -> giveup=True    # currently retried
InternalServerError  (500) -> giveup=True    # currently retried
InternalServerError  (503) -> giveup=False
APIStatusError       (408) -> giveup=False   # the intended fix

So 429 and 500 would stop being retried, though both are listed in the existing backoff tuple. Adding the 5xx range and 429 to _TRANSIENT_HTTP_CODES, or narrowing the giveup so it only fires for classes not already in the tuple, would keep the current behaviour intact.

[jmartin-tech]

Please carefully read the comments and review code in #1973, us of giveup is not appropriate to address this issue.

Note the final approach before #1973 was retracted by the contributor was moving in the desired direction however needed closer attention from the contributor.