Skip to content

feat: configurable upstreams for local and self-hosted models#6

Merged
armenr merged 1 commit into
mainfrom
feat/custom-upstreams
Jul 7, 2026
Merged

feat: configurable upstreams for local and self-hosted models#6
armenr merged 1 commit into
mainfrom
feat/custom-upstreams

Conversation

@armenr

@armenr armenr commented Jul 7, 2026

Copy link
Copy Markdown
Owner

Adds a first-class [upstreams] table so an alias can point at any Anthropic-Messages-compatible endpoint — the headline use being a local model, which is what modelmux couldn't do before (its upstreams were hardcoded to the two cloud URLs).

What

  • [upstreams] in routes.toml: name = { base = "...", auth = "..." }. anthropic (passthrough) and openrouter (bearer) stay built in; add your own on top.
  • auth: passthrough | passthrough:ENV | bearer:ENV | none.
  • Upstream names are no longer a fixed enum; loadConfig validates each model's upstream against the merged table and fails loud on an unknown one.
  • No auth leak: Claude Code's inbound auth is never forwarded to a none/bearer upstream, so your Anthropic token stays out of a local process.

Local models

Recent Ollama (v0.14+) speaks the Anthropic Messages API natively, so no shim:

[upstreams]
local = { base = "http://localhost:11434", auth = "none" }
[models]
flagship = "local:qwen3-coder:30b"   # subagents on local Qwen, orchestrator stays Claude

(OpenAI-format runners like LM Studio/vLLM: put LiteLLM in front, point base at it.)

Verified

  • bun run check green: eslint + tsc + 78 tests (+7), incl. an end-to-end integration test (custom local upstream routes correctly, Claude auth not leaked).
  • Hand-driven in the real runtime: a [upstreams] local config parses/lists; a local model without a declared upstream fails loud with a clear message.
  • README gains a Local & self-hosted models section.

- Add an [upstreams] table to routes.toml: point an alias at any
  Anthropic-Messages-compatible endpoint (local Ollama, a self-hosted gateway,
  another provider) with base + auth (passthrough | passthrough:ENV | bearer:ENV
  | none). anthropic and openrouter stay built in.
- Upstream names are no longer a fixed enum; loadConfig validates each model's
  upstream against the merged table and fails loud on an unknown one.
- Never forward Claude Code's inbound auth to a none/bearer upstream, so your
  Anthropic token never reaches a local process.
- README gains a 'Local & self-hosted models' section with an Ollama (v0.14+,
  native Anthropic API) example. Tests cover parsing, validation, auth, and
  end-to-end routing to a local upstream.
@armenr armenr merged commit 47a2aa0 into main Jul 7, 2026
2 checks passed
@armenr armenr deleted the feat/custom-upstreams branch July 7, 2026 07:39
@github-actions github-actions Bot mentioned this pull request Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant