Skip to content

Support any OpenAI-compatible endpoint (Ollama, vLLM, OpenRouter, Groq, …) - #8

Merged
jmbarrancoml merged 1 commit into
mainfrom
feat/01-openai-compatible-endpoints
Aug 18, 2026
Merged

jmbarrancoml merged 1 commit into
mainfrom
feat/01-openai-compatible-endpoints

Conversation

@jmbarrancoml

Copy link
Copy Markdown
Owner

Description

Item 1 of the capability list: reach every endpoint that speaks the OpenAI wire
format, instead of only the three hardcoded URLs in PROVIDERS.

LLMClient already spoke that dialect. All that was missing was a way to point
it somewhere else, so this is a small change with a large reach: Ollama, LM
Studio, vLLM, OpenRouter, Groq, Together, DeepSeek and any self-hosted gateway,
with no new client code and no new dependency.

Two ways in

# Ad-hoc, for a one-off or a local server
agent = Agent(base_url="http://localhost:11434/v1", model="llama3.3")

# Named, when you use it more than once
register_provider("ollama", base_url="http://localhost:11434/v1")
agent = Agent(provider="ollama", model="llama3.3")

base_url= always speaks the OpenAI dialect, which is what "OpenAI-compatible"
means. Anything unusual goes through register_provider, which also takes
dialect="anthropic", env_var, default_model and headers.

No third-party model defaults, on purpose

The obvious next step would be shipping built-in entries for Ollama, Groq and
friends with default models. This deliberately does not.

Their catalogues move faster than a release cycle, and a stale default is
exactly the bug that shipped as gpt-5.2-instant: a model that did not exist,
404-ing for every user of provider="openai". A provider registered without a
default_model instead raises a clear error naming itself when the caller omits
model=. The docs carry the base URLs, which are stable facts, and leave model
names to the caller.

Keyless local servers

Provider.env_var is optional. When it is unset the key is not required and a
placeholder bearer token is sent, because Ollama, LM Studio and vLLM accept any
string. Passing api_key= still overrides it.

Registry safety

PROVIDERS now maps names to a frozen Provider record rather than a plain
dict, so validation happens once and in one place:

  • base_url must be http:// or https://
  • dialect must be one of the two known wire formats
  • registering over an existing name needs overwrite=True, so a typo cannot
    silently shadow openai
  • unregister_provider refuses built-ins outright

Header support

Gateways like OpenRouter want attribution headers. Both the provider record and
Agent(headers=...) carry them, with the agent's winning on conflict.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Refactoring
  • Other (describe):

Breaking changes

Both are in CHANGELOG.md:

  • PROVIDERS values are Provider records, not dicts. Read
    PROVIDERS["openai"].base_url, not PROVIDERS["openai"]["base_url"].
    PROVIDERS is not in __all__ and is undocumented, so the blast radius is
    small, but it is a change.
  • Agent(provider=) defaults to None and resolves to "mistral" internally.
    Behaviour for every existing call is identical; the change exists so that
    passing provider= and base_url= together can be rejected rather than
    silently ignoring one of them.

Checklist

  • Tests pass (pytest) — 245 total, 39 new, on Python 3.10 and 3.14
  • Linting passes (ruff check . and ruff format --check .)
  • Documentation updated — providers guide (endpoint table, registry
    reference, local-dev recipe), README, llms.txt, API reference, CHANGELOG,
    and examples/25_custom_endpoints.py
  • Changes follow the "one parameter = one feature" philosophy

What the new tests cover

Registration and lookup, requests landing on the registered URL, optional and
missing default models, keyless providers, env-var-backed providers, the
Anthropic dialect end to end, trailing-slash trimming, duplicate and built-in
name protection, URL and dialect validation, unregistering, ad-hoc base_url
including the mutually-exclusive check with provider=, header merging and
precedence, error messages listing what is registered, and _clone()/batch()
carrying a custom endpoint.

Related issues

None filed; item 1 of the agreed capability list.

The library already spoke the OpenAI wire format, but only ever pointed it at
three hardcoded URLs in a private dict. Opening that up reaches Ollama, LM
Studio, vLLM, OpenRouter, Groq, Together and any self-hosted gateway without
writing another client.

Two ways in, matching how often you need it:

    agent = Agent(base_url="http://localhost:11434/v1", model="llama3.3")

    register_provider("ollama", base_url="http://localhost:11434/v1")
    agent = Agent(provider="ollama", model="llama3.3")

No key is required when the provider declares no env_var, which is the shape of
a local server. `dialect="anthropic"` covers a proxy that speaks that format
instead, and `headers=` covers gateways that want attribution.

No third-party model defaults ship with this. Their catalogues move faster than
a release cycle, and guessing one is what produced the gpt-5.2-instant bug. A
provider without a default_model gives a clear error naming itself when the
caller omits model=.

PROVIDERS now holds frozen Provider records instead of plain dicts, so the
registry validates once, in one place: http(s) base URLs, known dialects, no
accidental shadowing of a built-in. Agent(provider=) defaults to None so that
passing it alongside base_url= can be rejected rather than silently ignored.

39 new tests. Docs, llms.txt, API reference, CHANGELOG and a runnable example.
@jmbarrancoml
jmbarrancoml merged commit 4b00d59 into main Aug 18, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants