feat: add multi-provider LLM gateway and utilization accounting

[mark-e-deyoung]

Objective

Extend the current Ask Sage-to-OpenAI compatibility service into a deployable multi-provider gateway architecture that can route OpenAI-compatible clients to Ask Sage, Groq, Google Gemini, Mistral, OpenRouter, and future providers while centrally accounting for utilization.

The preferred implementation is to evaluate and, if accepted, deploy LiteLLM as the public/client-facing gateway while retaining the existing Ask Sage proxy as an OpenAI-compatible backend adapter. Do not duplicate mature routing, virtual-key, budget, and accounting features inside the Ask Sage translation layer without a documented reason.

Current state

The repository currently translates OpenAI-compatible /v1/chat/completions, /v1/models, and selected audio endpoints to Ask Sage. It can pass through a best-effort upstream usage object when Ask Sage returns one, but it has no durable request ledger, per-client virtual keys, provider routing, budgets, quota awareness, Prometheus metrics, or consolidated cost dashboard.

Scope

1. Document and implement the selected gateway topology:
   • clients and coding CLIs -> multi-provider gateway;
   • gateway -> Groq, Gemini, Mistral, OpenRouter;
   • gateway -> existing Ask Sage proxy as a custom OpenAI-compatible backend;
   • optional future local/Ollama-compatible backends.
2. Preserve the existing Ask Sage protocol translation as a bounded adapter rather than mixing unrelated provider-specific request code into it.
3. Add provider model aliases and explicit routing classes, including:
   • free/preferred models;
   • paid models;
   • fallback chains;
   • models prohibited from automatic paid fallback.
4. Add gateway authentication using per-client or per-workload virtual keys. Never expose provider API keys to downstream clients.
5. Add utilization accounting for all traffic that traverses the gateway:
   • request count, status, latency, provider, model;
   • input, output, reasoning, and cached tokens when reported;
   • provider-reported or calculated cost when available;
   • rate-limit headers and retry information where safe;
   • attribution by virtual key, project, client, and correlation ID.
6. Add budgets and limits:
   • per-key/project RPM and TPM;
   • daily/monthly spend caps;
   • explicit free-only keys that cannot route to paid models;
   • paid keys with bounded maximum spend;
   • fail-closed behavior when cost classification is unknown.
7. Export metrics for the existing monitoring stack and provide a minimal dashboard or documented query set.
8. Support container deployment using Docker or Podman and retain compatibility with RHEL 9/OpenShift deployment patterns already represented in the repository.
9. Keep all credential values in an approved external secret store or deployment secret mechanism. Commit only identifiers, schemas, examples with placeholders, and retrieval procedures.

Utilization boundary

The gateway can account accurately for requests that pass through it. It cannot infer traffic sent directly to provider APIs, and it may not know exact account-wide free-tier balance when a provider does not expose that information programmatically. Distinguish:

• gateway-observed utilization;
• provider-reported token/cost data;
• provider account/project dashboards or quota APIs;
• estimates based on model pricing;
• unknown or unavailable quota state.

Never represent estimated cost or remaining free allowance as authoritative.

Acceptance criteria

• An ADR compares native expansion of this proxy with a LiteLLM front-door deployment and selects one architecture.
• Ask Sage remains usable through the same gateway endpoint as the external providers.
• Groq, Gemini, Mistral, and OpenRouter each pass a non-secret mocked integration test; live smoke tests are separately owner-gated.
• Downstream clients use gateway virtual keys and do not receive provider API keys.
• A free-only policy cannot fall through to a paid model or provider.
• Paid routing requires an explicit bounded budget and returns a clear blocked result when the budget is exhausted.
• Usage is attributable by virtual key/project and includes token/cost fields when the upstream supplies them.
• Unknown token or cost data is represented as unknown, not zero.
• Prometheus-compatible metrics or an equivalent documented export path is available.
• Logs, metrics, traces, error bodies, and receipts are sanitized and contain no provider or gateway secret values.
• Container deployment works with Docker and Podman; OpenShift manifests or overlays are updated when needed.
• Rollback to the current Ask Sage-only proxy is documented and tested.

Security and cost risks

• This work handles credentials and can incur paid inference charges. Provider keys, gateway master keys, database credentials, and virtual-key values must never enter Git history, GitHub comments, CI logs, or artifacts.
• Free-tier availability and model classifications change. Free/paid policy must be configuration-driven and fail closed when stale or ambiguous.
• A public gateway without authentication could spend provider credits. No externally reachable deployment may start without authentication and bounded limits.
• Prompt and response logging must be disabled by default or separately approved because application content may be sensitive.

Coordination

Actor: web-agent/chatgpt/gpt-5.6-thinking
Session: 2026-08-08T16:46:48Z / multi-provider-gateway-web-01
Branch: none
State: planned

Cross-repository consumer: SupraCraft/minecraft-infra Colab orchestration will use a short-lived gateway virtual key and base URL by default; the corresponding issue will be cross-linked after creation.

[mark-e-deyoung]

Cross-repository consumer issue created: SupraCraft/minecraft-infra#116 (feat: add proxy-first external LLM access to Colab workloads).

The integration contract is proxy-first: Colab receives a bounded gateway virtual key and base URL; durable Groq, Gemini, Mistral, OpenRouter, and Ask Sage credentials remain in the gateway's approved secret store. Optional direct-provider injection is owned and gated by minecraft-infra#116.

[mark-e-deyoung]

Correction: this issue was opened from an incorrect assumption that a new generic multi-provider gateway was needed. The maintained gateway is already LiteLLM on TrueNAS and is tracked in SupraCraft/minecraft-infra#116.

The Ask Sage repository should remain a focused OpenAI-compatible Ask Sage adapter. It may later be registered as one backend in the existing LiteLLM gateway, but it should not independently implement Groq, Gemini, Mistral, OpenRouter routing or duplicate LiteLLM virtual-key/accounting functions.

Closing this issue as not planned. No implementation branch or credential mutation occurred.