feat: updating gateway integration

[TEJASNARAYANS]

What Changed?

Briefly describe what this PR changes.

Why?

Explain the problem this solves and why this approach was chosen.

How to Test?

List clear steps for reviewers to verify the change.

Release Label

Select one semantic version bump intent for this PR:

• major - breaking change, next release bumps major version
• minor - backward-compatible feature, next release bumps minor version
• fix - backward-compatible bug fix, next release bumps patch version
• No label (defaults to patch release)

If you do not have permission to apply labels, mention the intended release label here and a maintainer will set it.

Checklist

• I have read the CONTRIBUTING.md guide.
• My code follows the project's style guidelines.
• I have added tests that prove my fix is effective or my feature works.
• I have updated documentation where needed.

[greptile-apps]

Greptile Summary

This PR introduces an optional LLM gateway layer (Bifrost or self-hosted LiteLLM Proxy) that routes batch/eval LiteLLM workloads through a proxy instead of calling providers directly. It adds platform-wide config settings, per-org UI overrides, database migrations, gateway-managed credential support, and wires the gateway into evaluators, GEPA, transcription, and judge-alignment paths.

• New llm_gateway service resolves per-org gateway config, handles Bifrost virtual-key injection, LiteLLM Proxy master-key auth, and native-path routing for Gemini/Google/Vertex models via custom_llm_provider=\"openai\".
• Gateway-managed credentials allow AI provider rows to omit an API key (stored as an encrypted sentinel) when passthrough_provider_keys: false; the gateway_managed flag is surfaced in the AI provider API and UI.
• Frontend modal adds LLM Gateway settings to the Integrations page with mode, type, URL, and key overrides — but clearing the base URL field sends null instead of \"\", so the backend silently preserves the existing value.

Confidence Score: 3/5

The backend gateway plumbing is correct end-to-end, but the frontend/backend mismatch for clearing the base URL override means users who have saved a URL cannot remove it via the UI without a direct API call.

The backend uses an empty-string sentinel to signal clear base URL, but the frontend sends null when the field is emptied. Both omitted and explicitly-null values map to None in Pydantic, so the backend's if base_url is not None guard is never triggered when clearing, and the existing stored URL is silently preserved. Unlike virtual_key and master_key which each have a dedicated clear_* boolean, there is no equivalent escape hatch for base_url.

frontend/src/pages/configurations/Integrations.tsx — the base_url mutation value; app/services/ai/llm_gateway_settings.py — set_org_settings base_url branching logic.

Important Files Changed

Filename	Overview
frontend/src/pages/configurations/Integrations.tsx	Adds LLM Gateway modal; the base_url mutation sends null instead of "" when cleared, so the backend preserves the old value and the user cannot clear the URL from the UI.
app/services/ai/llm_gateway_settings.py	Implements org-level gateway settings persistence; base_url clear semantics rely on empty-string sentinel that the frontend does not send, causing the UI clear action to be silently ignored.
app/services/ai/llm_gateway.py	New module implementing gateway resolution, key handling, and LiteLLM call wrapping for Bifrost and LiteLLM Proxy; well-structured with correct fallback-to-direct logic.
app/services/optimization/gepa_service.py	Routes both batch and reflection LiteLLM calls through the gateway, correctly passing model as routing_model so native-path (Gemini) providers get the OpenAI-compatible routing override.
app/api/v1/routes/aiproviders.py	Extends create endpoint to store a gateway-managed sentinel when passthrough_provider_keys is false and no API key is supplied; response correctly exposes the gateway_managed flag.
app/services/ai/transcription_service.py	Switches to resolve_litellm_api_key for credential resolution and allows null api_key through the GOOGLE STT path so the gateway can handle auth; previously flagged import bug is fixed.

Comments Outside Diff (1)

1. frontend/src/pages/configurations/Integrations.tsx, line 1940-1948 (link)

   Cannot clear org base URL from UI — change silently ignored

   When a user opens the modal with an existing base_url, clears the field, and saves, the frontend sends base_url: null (because "".trim() || null === null). In set_org_settings, the receiving base_url=None hits the elif "base_url" in existing: branch and restores the old value, so the clear is silently discarded.

   virtual_key and master_key both have dedicated clear_* boolean flags to sidestep this ambiguity; base_url has no equivalent. Sending "" instead of null would work — the backend already handles an empty string as a clear operation (trimmed = "" → payload["base_url"] = None).

_{Reviews (5): Last reviewed commit: "fix: updating gepa tests" | Re-trigger Greptile}