Route your AI agents across the model providers you already use.
LLMIngress is an open-source, self-hosted AI Gateway for agents. Connect Provider API keys, subscription accounts, and local model servers, expose them through stable Virtual Model names, and control routing, access, limits, fallback, and usage from one Console.
- 🔀 Route Virtual Models with
fixed,cost_first,load_balance,weighted,least_time, or header-driventagpolicies - 🚑 Track the health of each Provider connection and fall back before streaming begins
- 🔐 Create a dedicated API key for each agent or tool, with explicit Virtual Model grants
- 🛡️ Enforce optional budget, RPM, TPM, token, and concurrency limits
- 📊 Track activity, tokens, latency, failures, fallback, connection health, and request cost
- 🕶️ Keep prompts, successful responses, tool arguments, and credentials out of operational logs
git clone https://github.com/IamNotShady/LLMIngress.git
cd LLMIngress
./scripts/deploy.sh./scripts/deploy.sh writes a random ENCRYPTION_KEY into a gitignored .env when missing,
then rebuilds and recreates the Compose containers so stale container-network state is repaired.
The main branch uses the Compose project llmingress; every other branch gets an isolated
project named llmingress-<normalized-branch-name>, including its own containers, network, and
PostgreSQL volume. The selected branch's named volume is preserved across redeploys. Compose still
uses a local-only default PostgreSQL password (llmi-local-db). Published ports bind to
127.0.0.1 by default. Keep a backup of each worktree's .env — the same ENCRYPTION_KEY is
required to decrypt that branch's stored provider credentials.
Branch projects use the same published ports by default, so stop the active branch before starting
another one, or override CONSOLE_PORT, GATEWAY_PORT, and POSTGRES_PORT. Compose derives the
Console's public Gateway URL from GATEWAY_PORT; set GATEWAY_URL only when an explicit external
URL is required.
Every repository startup path resolves each variable in the same order: the current shell,
.env.local, .env, then its code default. ./scripts/deploy.sh passes both files to Compose when
.env.local exists, while ./init.sh and pnpm dev use the shared repository env loader.
DATABASE_URL is the host-process connection string; Docker uses COMPOSE_DATABASE_URL instead
because the app container reaches PostgreSQL through the Compose service name.
Compose runs two containers: the app (Console, Gateway, and Worker in one process group) and PostgreSQL.
| Endpoint | Address | Purpose |
|---|---|---|
| Console | http://localhost:3000 | Configure and observe LLMIngress |
| Gateway | http://localhost:4000 | Serve API key traffic |
| PostgreSQL | localhost:55432 |
Store configuration and operational metadata |
| Worker | Inside the app container | Refresh models, probe connections and quota, and synchronize prices |
Runtime and port overrides are documented in .env.example.
Open http://localhost:3000, create the administrator password, then:
- Add a Provider connection.
- Create a Virtual Model with at least one candidate.
- Create an API key allowed to use that Virtual Model.
- Copy the one-time
llmi_API key.
curl http://localhost:4000/v1/chat/completions \
--header "Authorization: Bearer llmi_your_api_key" \
--header "Content-Type: application/json" \
--data '{
"model": "your-virtual-model",
"messages": [{"role": "user", "content": "Hello"}]
}'LLMIngress supports remote API keys, subscription OAuth, and local model servers. The current built-in templates are:
| Connection type | Built-in templates |
|---|---|
| Subscription | Claude Code, OpenAI Codex, Grok, MiniMax Coding Plan |
| API key | Anthropic, AWS Bedrock, BytePlus ModelArk, Cerebras, ClinePass, Command Code, DeepSeek, Fireworks AI, GLM Coding Plan, Google Gemini, Groq, Kimi Coding Plan, MiniMax, Mistral, Mistral Vibe, Moonshot/Kimi, NousResearch, NVIDIA NIM, Ollama Cloud, OpenAI, OpenCode Go, OpenRouter, Qwen, Qwen Token Plan, xAI, Xiaomi MiMo, Xiaomi MiMo Token Plan, Z.ai |
| Local | Ollama, LM Studio, llama.cpp |
Model refresh can enrich Provider catalogs with capability and price data from models.dev, OpenRouter, LiteLLM, and Vercel. Missing metadata remains unknown, and manual values take precedence.
Health belongs to a Provider connection: each API key or OAuth token is checked independently, while a Local Provider has one logical connection. Confirmed unhealthy connections are filtered from routing until a successful probe recovers them.
Providers that report upstream usage — subscription windows, monthly budgets, or token plans — get a periodic quota probe, and the Console shows the remaining quota per connection.
API keys use the same Virtual Model grants across all supported protocols:
| Protocol | Endpoint |
|---|---|
| OpenAI Chat Completions | POST /v1/chat/completions |
| OpenAI Responses | POST /v1/responses |
| Anthropic Messages | POST /v1/messages |
| Virtual Model discovery | GET /v1/models |
Provider payloads remain protocol-native. LLMIngress replaces the Virtual Model name with the selected Provider model while preserving the Provider request and response contract.
Gateway health endpoints do not require an API key:
| Endpoint | Purpose |
|---|---|
GET /health/live |
Process liveness |
GET /health/ready |
Database and configuration readiness |
GET /health |
Readiness-compatible alias |
- Gateway authenticates API keys, enforces enabled limits, resolves Virtual Models, executes fallback, and records request metadata.
- Console owns configuration and operational views. It does not proxy API key traffic or call Providers.
- Worker performs model discovery, exact Provider-connection probes, upstream quota probes, and price synchronization.
- PostgreSQL stores durable configuration, jobs, usage, cost, fallback, and connection health.
LLMIngress uses Node.js 24, pnpm 11.5.1, and PostgreSQL 18.4.
pnpm install
cp .env.example .env.local
# Set ENCRYPTION_KEY (e.g. openssl rand -base64 32) and confirm DATABASE_URL / TEST_DATABASE_URL.
pnpm run db:migrate
./init.shTo exercise the production-shaped stack from a checkout, run ./scripts/deploy.sh.
Compose builds one multi-role application image, runs it as a single app container alongside
PostgreSQL (published on 127.0.0.1:55432 for development).
./init.sh runs lint, type-checking, unit tests, and the build before starting Console, Gateway,
and Worker. pnpm dev uses the same environment-file precedence without the verification gate.
The standalone verification commands are:
pnpm run verify
pnpm run verify:featuresThe project is pre-release. The current 0001_core_baseline.sql schema is authoritative;
databases created from older development migration histories should be recreated rather than
upgraded in place.
Read AGENTS.md and the coding guide before changing behavior. Work on one feature at a time, write unit and E2E coverage before implementation, and run both verification commands before marking a feature complete.