Switch evals/benchmark to LiteLLM gateway, add dynamic model discovery

[saashanair]

Why

Model access should be allowed only through the central i.AI LLM gateway (LiteLLM). This PR makes that true everywhere in this repo — evals, synthetic data generation, and the benchmark CLI — and replaces the hardcoded model list with one that updates automatically as the gateway's model list changes.

What changed

Dynamic model discovery (new)

evals/utils_gateway.py fetches the gateway's /model_group/info and /health/latest and turns them into a list of chat-capable models, each with vendor family and health resolved. Filtering is composable (split_unhealthy, filter_by_family, select_by_name) so callers — not this module — decide what to do with unhealthy or unknown matches.

benchmark.py — no more hardcoded model list

The old MODEL_REGISTRY (Azure/Vertex/locai) is gone, along with the Vertex code path (it never worked — create_llm() unconditionally raised NotImplementedError for it). Instead relevant models are found dynamically by querying the gateway.

Before	After
--provider {azure,vertex,locai,all}	--models <name>... / --family {gpt,claude,gemini,locai}... / --all (exactly one required)
reasoning_effort hardcoded per model name; no way to sweep effort levels	Creates a separate config per effort level by sweeping --reasoning-effort {low,medium,high}... across all models whose supports_reasoning flag, read from the gateway, is true

Model selection (--models vs. --family/--all) is two small functions, _select_named_models/_select_healthy_models, each returning only the fields relevant to its own mode — --models warns on unhealthy matches but still runs them (explicit request by name); --family/--all drops unhealthy matches and reports what was dropped.

Synthetic data generation — off direct Azure

evals/synthetic/* now talks to the gateway (openai.AsyncOpenAI(base_url=LLM_GATEWAY_URL, ...)) instead of constructing an AsyncAzureOpenAI client directly. The two models it uses are now named constants (DRAFTING_MODEL, RESPONSE_GENERATION_MODEL) instead of repeated string literals.

Gateway credentials — one source of truth

LLM_GATEWAY_URL/CONSULT_EVAL_LITELLM_API_KEY were each read directly via os.getenv in 10 separate places across benchmark.py, metrics.py, all four eval_*.py files, synthetic/cli.py, and generate_synthetic.py — only utils_gateway.py's own client validated them before use; everywhere else silently passed None/None into OpenAILLM/AsyncOpenAI on a misconfigured environment. Added utils_gateway.gateway_credentials() as the single source of truth; every call site now gets the same fail-fast validation.

Config cleanup

.env.example and eval.yml no longer list AZURE_OPENAI_*/GOOGLE_CLOUD_*/LOCAI_* — confirmed unused anywhere in the repo post-migration. LLM_GATEWAY_URL/CONSULT_EVAL_LITELLM_API_KEY are documented (previously required but missing from .env.example). AUTO_EVAL_4_1_SWEDEN_DEPLOYMENT (read by the eval judge/scoring path) is now prefilled with a known-working value plus a TODO, instead of being left blank and failing with a confusing 400 partway through a run.

Tests

35 tests (test_utils_gateway.py, test_benchmark.py) covering the discovery and selection logic against mocked gateway responses.

Test plan

Verified live against the real gateway:

• benchmark.py --quick runs end-to-end
• benchmark.py --models gpt-4.1-nano-sweden --evals mapping runs and returns real scores
• benchmark.py --family locai --evals mapping — selection/execution mechanism confirmed working (discovered, filtered to family, excluded the unhealthy match, ran the rest); the eval itself failed for an unrelated, pre-existing reason (locai's structured-output handling), not a regression from this PR
• benchmark.py with no selector errors as expected (exit 2)
• generate_synthetic.py runs end-to-end (1 question, 10 responses) — verified real output on disk
• All 35 unit tests pass
• pre-commit clean (ruff, ruff format, secret/IP/AWS-key scans, nbstripout)

Not yet verified:

• benchmark.py --models <unhealthy-name> through a full run (the warn-but-still-run behavior is unit-tested; no currently-unhealthy model was available on the live gateway to exercise it end-to-end)
• CI (eval.yml, pre-commit.yml) on this actual PR

🤖 Generated with Claude Code

[linear]

PRO-618

[saashanair]

Closing this PR. These changes have been merged into consult's version of themefinder.