feat: configurable upstreams for local and self-hosted models#6
Merged
Conversation
- Add an [upstreams] table to routes.toml: point an alias at any Anthropic-Messages-compatible endpoint (local Ollama, a self-hosted gateway, another provider) with base + auth (passthrough | passthrough:ENV | bearer:ENV | none). anthropic and openrouter stay built in. - Upstream names are no longer a fixed enum; loadConfig validates each model's upstream against the merged table and fails loud on an unknown one. - Never forward Claude Code's inbound auth to a none/bearer upstream, so your Anthropic token never reaches a local process. - README gains a 'Local & self-hosted models' section with an Ollama (v0.14+, native Anthropic API) example. Tests cover parsing, validation, auth, and end-to-end routing to a local upstream.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a first-class
[upstreams]table so an alias can point at any Anthropic-Messages-compatible endpoint — the headline use being a local model, which is what modelmux couldn't do before (its upstreams were hardcoded to the two cloud URLs).What
[upstreams]inroutes.toml:name = { base = "...", auth = "..." }.anthropic(passthrough) andopenrouter(bearer) stay built in; add your own on top.auth:passthrough|passthrough:ENV|bearer:ENV|none.loadConfigvalidates each model's upstream against the merged table and fails loud on an unknown one.none/bearerupstream, so your Anthropic token stays out of a local process.Local models
Recent Ollama (v0.14+) speaks the Anthropic Messages API natively, so no shim:
(OpenAI-format runners like LM Studio/vLLM: put LiteLLM in front, point
baseat it.)Verified
bun run checkgreen: eslint + tsc + 78 tests (+7), incl. an end-to-end integration test (custom local upstream routes correctly, Claude auth not leaked).[upstreams] localconfig parses/lists; a local model without a declared upstream fails loud with a clear message.