Repository navigation
Support any OpenAI-compatible endpoint (Ollama, vLLM, OpenRouter, Groq, …) - #8
Merged
Merged
Conversation
The library already spoke the OpenAI wire format, but only ever pointed it at
three hardcoded URLs in a private dict. Opening that up reaches Ollama, LM
Studio, vLLM, OpenRouter, Groq, Together and any self-hosted gateway without
writing another client.
Two ways in, matching how often you need it:
agent = Agent(base_url="http://localhost:11434/v1", model="llama3.3")
register_provider("ollama", base_url="http://localhost:11434/v1")
agent = Agent(provider="ollama", model="llama3.3")
No key is required when the provider declares no env_var, which is the shape of
a local server. `dialect="anthropic"` covers a proxy that speaks that format
instead, and `headers=` covers gateways that want attribution.
No third-party model defaults ship with this. Their catalogues move faster than
a release cycle, and guessing one is what produced the gpt-5.2-instant bug. A
provider without a default_model gives a clear error naming itself when the
caller omits model=.
PROVIDERS now holds frozen Provider records instead of plain dicts, so the
registry validates once, in one place: http(s) base URLs, known dialects, no
accidental shadowing of a built-in. Agent(provider=) defaults to None so that
passing it alongside base_url= can be rejected rather than silently ignored.
39 new tests. Docs, llms.txt, API reference, CHANGELOG and a runnable example.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Item 1 of the capability list: reach every endpoint that speaks the OpenAI wire
format, instead of only the three hardcoded URLs in
PROVIDERS.LLMClientalready spoke that dialect. All that was missing was a way to pointit somewhere else, so this is a small change with a large reach: Ollama, LM
Studio, vLLM, OpenRouter, Groq, Together, DeepSeek and any self-hosted gateway,
with no new client code and no new dependency.
Two ways in
base_url=always speaks the OpenAI dialect, which is what "OpenAI-compatible"means. Anything unusual goes through
register_provider, which also takesdialect="anthropic",env_var,default_modelandheaders.No third-party model defaults, on purpose
The obvious next step would be shipping built-in entries for Ollama, Groq and
friends with default models. This deliberately does not.
Their catalogues move faster than a release cycle, and a stale default is
exactly the bug that shipped as
gpt-5.2-instant: a model that did not exist,404-ing for every user of
provider="openai". A provider registered without adefault_modelinstead raises a clear error naming itself when the caller omitsmodel=. The docs carry the base URLs, which are stable facts, and leave modelnames to the caller.
Keyless local servers
Provider.env_varis optional. When it is unset the key is not required and aplaceholder bearer token is sent, because Ollama, LM Studio and vLLM accept any
string. Passing
api_key=still overrides it.Registry safety
PROVIDERSnow maps names to a frozenProviderrecord rather than a plaindict, so validation happens once and in one place:
base_urlmust behttp://orhttps://dialectmust be one of the two known wire formatsoverwrite=True, so a typo cannotsilently shadow
openaiunregister_providerrefuses built-ins outrightHeader support
Gateways like OpenRouter want attribution headers. Both the provider record and
Agent(headers=...)carry them, with the agent's winning on conflict.Type of change
Breaking changes
Both are in
CHANGELOG.md:PROVIDERSvalues areProviderrecords, not dicts. ReadPROVIDERS["openai"].base_url, notPROVIDERS["openai"]["base_url"].PROVIDERSis not in__all__and is undocumented, so the blast radius issmall, but it is a change.
Agent(provider=)defaults toNoneand resolves to"mistral"internally.Behaviour for every existing call is identical; the change exists so that
passing
provider=andbase_url=together can be rejected rather thansilently ignoring one of them.
Checklist
pytest) — 245 total, 39 new, on Python 3.10 and 3.14ruff check .andruff format --check .)reference, local-dev recipe), README,
llms.txt, API reference, CHANGELOG,and
examples/25_custom_endpoints.pyWhat the new tests cover
Registration and lookup, requests landing on the registered URL, optional and
missing default models, keyless providers, env-var-backed providers, the
Anthropic dialect end to end, trailing-slash trimming, duplicate and built-in
name protection, URL and dialect validation, unregistering, ad-hoc
base_urlincluding the mutually-exclusive check with
provider=, header merging andprecedence, error messages listing what is registered, and
_clone()/batch()carrying a custom endpoint.
Related issues
None filed; item 1 of the agreed capability list.