Skip to content

feat: add LiteLLM provider with /v1/models discovery - #220

Open
prodmanpd wants to merge 1 commit into
NanoNets:mainfrom
prodmanpd:feat/add-litellm-provider
Open

feat: add LiteLLM provider with /v1/models discovery#220
prodmanpd wants to merge 1 commit into
NanoNets:mainfrom
prodmanpd:feat/add-litellm-provider

Conversation

@prodmanpd

@prodmanpd prodmanpd commented Aug 24, 2026

Copy link
Copy Markdown

Summary

Adds LiteLLM as a first class provider (provider: "litellm"), so one endpoint reaches 100+ models (OpenAI, Anthropic, Gemini, Bedrock, Vertex, Azure, Groq, and more) through a self hosted LiteLLM proxy.

A LiteLLM proxy speaks the OpenAI compatible wire format, so, per this repo's own design ("adding a wire format is one new adapter"), the adapter reuses OpenAIChatModel's translation and only changes the defaults. The value over provider: "openai" plus a manual baseUrl is a self describing provider:

  • a sensible default proxy address (http://localhost:4000), and
  • /v1/models auto discovery (listLiteLLMModels) so callers can enumerate whatever models the proxy is configured to serve instead of hardcoding a model id.

Changes

  • src/ai/llm/litellm.ts (new): LiteLLMChatModel extends OpenAIChatModel (default proxy base URL and litellm:<model> label) and listLiteLLMModels() which calls GET /v1/models and returns the available model ids.
  • src/ai/llm/factory.ts: ProviderKind gains "litellm"; createChatModel builds a LiteLLMChatModel.
  • src/ai/providers.ts: default model for the litellm provider.
  • test/litellm-adapter.test.ts

Tests

Types: npx tsc -p tsconfig.json --noEmit is clean.

Tests: npm test (repo runner): 840 passed, 0 failed (4 new litellm tests plus no regressions).

test/litellm-adapter.test.ts
  ok litellm: reuses OpenAI-compatible translation and labels as litellm
  ok litellm: /v1/models auto-discovery returns the proxy's model ids
  ok litellm: factory builds a LiteLLMChatModel for provider 'litellm'
  ok litellm: exposes a default proxy base URL

The new tests assert: the adapter forwards the provider prefixed model and temperature and reuses the OpenAI translation; listLiteLLMModels maps /v1/models to ids; and the factory wires provider: "litellm".

Usage

import { createChatModel } from "./ai/llm/factory.js";
import { listLiteLLMModels } from "./ai/llm/litellm.js";

// Chat through a LiteLLM proxy (defaults to http://localhost:4000)
const model = createChatModel({
  provider: "litellm",
  apiKey: process.env.LITELLM_API_KEY ?? "",
  model: "anthropic/claude-3-5-sonnet", // any model the proxy serves
});

// Discover what the proxy offers
const models = await listLiteLLMModels({ apiKey: process.env.LITELLM_API_KEY ?? "" });

Or via env: GRAFT_PROVIDER=litellm, GRAFT_MODEL=anthropic/claude-3-5-sonnet, GRAFT_BASE_URL=http://localhost:4000.

github-actions Bot added a commit that referenced this pull request Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant