Port themefinder#148: switch evals to LiteLLM gateway, add dynamic model discovery

[saashanair]

Context

Model access should be allowed only through the central i.AI LLM gateway (LiteLLM). This PR makes that true everywhere in themefinder/ — evals, synthetic data generation, and the benchmark CLI — and replaces the hardcoded model list with one that updates automatically as the gateway's model list changes.

This ports i-dot-ai/themefinder#148, raised against the old standalone themefinder . That PR was raised without realising that themefinder had been subtree-merged into consult — this replays that PR's diff under the themefinder/ prefix. File paths are the only thing that changed; the logic below is unmodified from the original PR.

Stacked on #1546 (re-adding the missing themes.json ground-truth files) — needed for evals/tests/ and the live eval CI job to actually pass here. This PR will auto-retarget to main once that one merges.

Changes proposed in this pull request

Dynamic model discovery (new)
evals/utils_gateway.py fetches the gateway's /model_group/info and /health/latest and turns them into a list of chat-capable models, each with vendor family and health resolved. Filtering is composable (split_unhealthy, filter_by_family, select_by_name) so callers decide what to do with unhealthy or unknown matches.

benchmark.py — no more hardcoded model list
The old MODEL_REGISTRY (Azure/Vertex/locai) is gone, along with the Vertex code path (it never worked — create_llm() unconditionally raised NotImplementedError for it). Relevant models are now found dynamically by querying the gateway.

Before	After
--provider {azure,vertex,locai,all}	--models <name>... / --family {gpt,claude,gemini,locai}... / --all (exactly one required)
reasoning_effort hardcoded per model name; no way to sweep effort levels	Creates a separate config per effort level by sweeping --reasoning-effort {low,medium,high}... across all models whose supports_reasoning flag, read from the gateway, is true

Model selection (--models vs. --family/--all) is two small functions, _select_named_models/_select_healthy_models, each returning only the fields relevant to its own mode — --models warns on unhealthy matches but still runs them (explicit request by name); --family/--all drops unhealthy matches and reports what was dropped.

Synthetic data generation — off direct Azure
evals/synthetic/* now talks to the gateway (openai.AsyncOpenAI(base_url=LLM_GATEWAY_URL, ...)) instead of constructing an AsyncAzureOpenAI client directly. The two models it uses are now named constants (DRAFTING_MODEL, RESPONSE_GENERATION_MODEL) instead of repeated string literals.

Gateway credentials — one source of truth
LLM_GATEWAY_URL/CONSULT_EVAL_LITELLM_API_KEY were each read directly via os.getenv in 10 separate places across benchmark.py, metrics.py, all four eval_*.py files, synthetic/cli.py, and generate_synthetic.py — only utils_gateway.py's own client validated them before use; everywhere else silently passed None/None into OpenAILLM/AsyncOpenAI on a misconfigured environment. Added utils_gateway.gateway_credentials() as the single source of truth; every call site now gets the same fail-fast validation.

Config cleanup
.env.example and the eval workflow no longer list AZURE_OPENAI_*/GOOGLE_CLOUD_*/LOCAI_* — confirmed unused anywhere in the repo post-migration. LLM_GATEWAY_URL/CONSULT_EVAL_LITELLM_API_KEY are documented (previously required but missing from .env.example). AUTO_EVAL_4_1_SWEDEN_DEPLOYMENT (read by the eval judge/scoring path) is now prefilled with a known-working value plus a TODO, instead of being left blank and failing with a confusing 400 partway through a run.

Tests
35 tests (evals/tests/test_utils_gateway.py, test_benchmark.py) covering the discovery and selection logic against mocked gateway responses.

Guidance to review

Verified in the original PR, against the old standalone repo:

• generate_synthetic.py runs end-to-end (1 question, 10 responses) — verified real output on disk
• All 35 unit tests pass
• benchmark.py --models <unhealthy-name> through a full run — the warn-but-still-run behavior is unit-tested; no currently-unhealthy model was available on the live gateway to exercise it end-to-end

Re-verified directly in consult, after porting:

• All 35 unit tests pass unchanged in this repo (uv run pytest evals/tests/ -v)
• discover_chat_models() runs live against the real gateway, 57 chat-capable models discovered with real health status
• benchmark.py --quick runs end-to-end here too; mapping/condensation/refinement produce real scores. generation currently errors on an unrelated pre-existing bug in src/themefinder's batch-processing retry path (not introduced by this PR — happy to open a separate issue)
• benchmark.py with no selector exits 2 with the expected argparse error, matching test_no_selector_is_invalid
• benchmark.py --models gpt-4.1-nano-sweden --evals mapping runs end-to-end, real scores (f1: 0.479 / 0.503 across the two question parts)
• benchmark.py --family locai --evals mapping — model discovered, selected by family filter, and received live API calls through the gateway. Reproduces the exact same known, pre-existing locai structured-output failure the original PR documented (malformed/non-JSON responses to the mapping prompt) — not a regression from this PR

Things to check

• I have added any new ENV vars in all deployed environments and updated the .env.test files in the repo

n/a — no new env vars. Required a routing/permissions change on the gateway side instead: i-dot-ai/core-llm-gateway#232 which fixed the reading of available models from the gateway, alongside updating the routes that CI key (CONSULT_EVAL_LITELLM_API_KEY) can access.

[linear]

PRO-618

[github-actions]

Claude finished @saashanair's task in 2m 23s —— View job

---

PR Review

Clean port with solid test coverage. Two issues worth addressing:

---

🔴 DO — benchmark.py lines 989–990 and 1004–1006: silent success on config failures

Both early-exit paths use bare return, so the process exits with code 0 even when no models were found or matched. CI will pass silently on a misconfigured environment.

# line 989
if missing:
    console.print(...)
    return          # ← sys.exit(1) needed here

# line 1004
if not selected:
    console.print(...)
    return          # ← and here

Fix this →%20after%20%60console.print(f%22%5Bred%5DNo%20matching%20models...%22)%60%20around%20line%20989%2C%20and%202)%20after%20%60console.print(%22%5Bred%5DNo%20models%20matched...%22)%60%20around%20line%201005.%20Both%20are%20error%20conditions%20that%20should%20exit%20non-zero%20for%20CI%20to%20catch%20them.&repo=i-dot-ai/consult)

---

🟡 TRY — utils_gateway.py line 183: direct key access on gateway response

item["supports_reasoning"] will raise KeyError if the gateway ever omits this field. filter_chat_models already uses .get() for mode — worth being consistent:

# current
supports_reasoning=item["supports_reasoning"],

# safer
supports_reasoning=item.get("supports_reasoning", False),

---

[github-actions]

Claude finished @saashanair's task in 1m 2s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

Clean port with solid test coverage. Two issues from the previous review remain unaddressed:

---

🔴 DO — benchmark.py lines 989 and 1004: silent success on config failures

Both early-exit paths use bare return, so the process exits with code 0 even when no models were found or matched. CI will pass silently on a misconfigured environment.

# line 989
if missing:
    console.print(...)
    return          # ← sys.exit(1) needed here

# line 1004
if not selected:
    console.print(...)
    return          # ← and here

Fix this →%20after%20%60console.print(f%22%5Bred%5DNo%20matching%20models...%22)%60%20around%20line%20989%2C%20and%202)%20after%20%60console.print(%22%5Bred%5DNo%20models%20matched...%22)%60%20around%20line%201005.%20Both%20are%20error%20conditions%20that%20should%20exit%20non-zero%20for%20CI%20to%20catch%20them.&repo=i-dot-ai/consult)

---

🟡 TRY — utils_gateway.py line 184: direct key access on gateway response

item["supports_reasoning"] will raise KeyError if the gateway ever omits this field. filter_chat_models already uses .get() for mode — worth being consistent:

# current
supports_reasoning=item["supports_reasoning"],

# safer
supports_reasoning=item.get("supports_reasoning", False),

---

[github-actions]

Claude finished @saashanair's task in 47s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

Both issues from previous reviews are now fixed — sys.exit(1) at both early-exit paths in benchmark.py, and item.get("supports_reasoning", False) in utils_gateway.py. No further issues found. ✅

[github-actions]

Claude finished @saashanair's task in 1m 9s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

Both previous issues are resolved and no new ones found. Clean port. ✅

[github-actions]

Claude finished @saashanair's task in 2m 22s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

Clean port. Both previous DO/TRY issues (sys.exit(1) at early-exit paths and .get("supports_reasoning", False)) are confirmed fixed. ✅

One small nit:

⚪️ NIT — generate_synthetic.py line 58: carried-over comment references "medium reasoning"

The comment survived from the Azure port but "medium reasoning" isn't an API parameter being set — it describes an assumed default of the gpt-5-nano-sweden model. Worth removing or replacing with the constant name to stay consistent with the pattern introduced by this PR:

# before
# Initialise LLM for response generation (gpt-5-nano with medium reasoning)
# Medium reasoning ≈ o1 performance, 2x faster throughput than mini/low

# after (or just drop the comment — the constant is self-documenting)
# Initialise LLM for response generation

---

[github-actions]

Claude finished @saashanair's task in 1m 4s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

All previous DO/TRY issues (sys.exit(1) at both early-exit paths, .get("supports_reasoning", False)) remain fixed. ✅

One carry-over from the last review:

⚪️ NIT — generate_synthetic.py lines 58–59: stale comment

The two-line comment still describes Azure-specific behaviour that no longer applies:

# Initialise LLM for response generation (gpt-5-nano with medium reasoning)
# Medium reasoning ≈ o1 performance, 2x faster throughput than mini/low

The constant RESPONSE_GENERATION_MODEL is already self-documenting — this comment can just be dropped. Fix this →

---

[github-actions]

Claude finished @saashanair's task in 1m 9s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

All previously flagged issues are resolved: sys.exit(1) at both early-exit paths, .get("supports_reasoning", False), and the stale Azure comment in generate_synthetic.py. TODO for hardcoded model-family matching is now in place. ✅

One open thread from 252afh's review:

⚪️ NIT — utils_gateway.py line 25: _GPT_MARKERS missing "o5-"

"gpt" catches gpt-5* models, but o5- (and o2-) aren't covered by any existing marker. The TODO at lines 20–23 acknowledges this. Fix this →

[github-actions]

Claude finished @saashanair's task in 1m 19s —— View job

---

PR Review

• Gather context and read changed files
• Analyze code quality, patterns, and potential issues
• Post review feedback

---

All previously flagged DO/TRY issues remain fixed (sys.exit(1) at both early-exit paths, .get("supports_reasoning", False), stale Azure comment dropped). ✅

One carry-over NIT still open:

⚪️ NIT — utils_gateway.py line 25: _GPT_MARKERS still missing "o5-" and "o2-"

The TODO noting the limitation is in place, but the markers themselves haven't been extended. "gpt" doesn't catch o5-* or o2-* model names, so they'll get family=None and be unselectable via --family gpt. Fix this →