feat: add LiteLLM as chat model provider for 100+ LLM backends by RheagalFire · Pull Request #456 · xorbitsai/xagent

RheagalFire · 2026-05-19T15:44:13Z

Summary

Adds LiteLLM as a new chat model provider alongside OpenAI, Claude, DeepSeek, Gemini, and Zhipu, enabling access to 100+ LLM backends through LiteLLM's unified interface.

Changes

File	What
`src/xagent/core/model/chat/basic/litellm.py`	New `LiteLLM(BaseLLM)` with async `chat()` and `stream_chat()`. Uses `litellm.acompletion()` with `drop_params=True`. Supports tool calling, response format, token usage tracking. Maps litellm exceptions to `LLMRetryableError`/`LLMTimeoutError`.
`src/xagent/core/model/chat/basic/__init__.py`	Import and export `LiteLLM`.
`src/xagent/core/model/chat/basic/adapter.py`	Added `"litellm"` provider branch in `create_base_llm()` factory.
`tests/core/model/chat/basic/test_litellm.py`	20 unit tests covering init, chat, tool calling, error handling, factory.

Tests

Unit tests -- 20 passed:

tests/core/model/chat/basic/test_litellm.py::TestLiteLLMInit          7 passed
tests/core/model/chat/basic/test_litellm.py::TestLiteLLMChat          7 passed
tests/core/model/chat/basic/test_litellm.py::TestLiteLLMToolCalling   1 passed
tests/core/model/chat/basic/test_litellm.py::TestLiteLLMErrors        4 passed
tests/core/model/chat/basic/test_litellm.py::TestLiteLLMFactory       1 passed
20 passed in 1.55s

Live e2e via Anthropic:

from xagent.core.model.chat.basic.litellm import LiteLLM
llm = LiteLLM(model_name='anthropic/claude-sonnet-4-6')
result = await llm.chat([{'role': 'user', 'content': 'What is 2+2?'}])
# Result: 4

Lint:

$ ruff check src/xagent/core/model/chat/basic/litellm.py
All checks passed!

Example usage

from xagent.core.model.chat.basic import LiteLLM

llm = LiteLLM(
    model_name="anthropic/claude-sonnet-4-6",
    # api_key omitted -- reads from ANTHROPIC_API_KEY env var
)

response = await llm.chat(
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0.7,
)

Any LiteLLM model string works. See LiteLLM docs for full list.

RheagalFire · 2026-05-19T15:44:32Z

cc @yiboyasss

gemini-code-assist

Code Review

This pull request introduces a new LiteLLM provider to the xagent core model, enabling access to numerous LLM providers through a unified interface. The implementation includes the LiteLLM class, updates to the model adapter factory, and a suite of unit tests. Feedback highlights several critical issues in the new implementation: the chat method fails to include the mandatory raw field in tool call responses, the stream_chat method uses incorrect chunk types and lacks support for tool calls, and there is a risk of an IndexError when accessing response choices without prior validation.

gemini-code-assist · 2026-05-19T15:46:03Z

+                        },
+                    }
+                )
+            return {"type": "tool_call", "tool_calls": tool_calls}


The tool call response dictionary is missing the raw field. According to the BaseLLM.chat interface documentation (see base.py line 163), the returned dictionary for tool calls must include the full response JSON in a raw field to allow for debugging and advanced processing.

return { "type": "tool_call", "tool_calls": tool_calls, "raw": response.model_dump() if hasattr(response, "model_dump") else dict(response), }

gemini-code-assist · 2026-05-19T15:46:04Z

+        async for chunk in response:
+            delta = chunk.choices[0].delta if chunk.choices else None
+            if delta is None:
+                continue
+            content = getattr(delta, "content", None)
+            if content:
+                yield StreamChunk(type=ChunkType.TEXT, content=content)


The stream_chat implementation has several issues that break consistency and functionality:

It uses ChunkType.TEXT instead of the standard ChunkType.TOKEN used in the rest of the codebase (e.g., in BaseLLM.stream_chat).

It omits the delta field in StreamChunk, which is expected by consumers.

It completely ignores tool calls in the stream, which will cause them to be dropped silently during streaming.

Updating the loop to handle these cases ensures compatibility with the expected streaming behavior and supports tool calling in streaming mode.

async for chunk in response: if not chunk.choices: continue delta = chunk.choices[0].delta # Handle text content content = getattr(delta, "content", None) if content: yield StreamChunk(type=ChunkType.TOKEN, content=content, delta=content) # Handle tool calls if hasattr(delta, "tool_calls") and delta.tool_calls: tool_calls = [] for tc in delta.tool_calls: tool_calls.append({ "index": getattr(tc, "index", 0), "id": getattr(tc, "id", None), "type": "function", "function": { "name": getattr(tc.function, "name", None), "arguments": getattr(tc.function, "arguments", ""), }, }) yield StreamChunk( type=ChunkType.TOOL_CALL, tool_calls=tool_calls, raw=chunk.model_dump() if hasattr(chunk, "model_dump") else dict(chunk) )

gemini-code-assist · 2026-05-19T15:46:04Z

+        ) as e:
+            raise LLMRetryableError(str(e)) from e
+
+        choice = response.choices[0]


The code accesses response.choices[0] without verifying that choices is not empty. While LiteLLM typically returns at least one choice on success, certain edge cases (like content filtering or provider-specific errors) could result in an empty list, leading to an IndexError.

Suggested change

choice = response.choices[0]

if not response.choices:

raise LLMRetryableError("LiteLLM returned an empty response (no choices).")

choice = response.choices[0]

qinxuye · 2026-05-20T04:19:16Z

Could you fix the lint, and resolve the gemini comments?

…d empty choices

RheagalFire · 2026-06-03T22:06:03Z

@qinxuye Fixed all three gemini review items:

Added raw field to tool call response (matches zhipu pattern)
Changed ChunkType.TEXT to ChunkType.TOKEN in stream_chat
Added empty choices guard before accessing response.choices[0]

Lint passes clean.

qinxuye · 2026-06-05T10:06:07Z

Lint (pre-commit) failed again... Could you fix it?

RheagalFire · 2026-06-09T20:52:36Z

@qinxuye can u check again

rogercloud

Code Review

Thanks for the work here — adding LiteLLM as a meta-provider is a reasonable direction, the factory wiring follows the existing per-provider pattern, and all three prior Gemini findings (missing raw field, ChunkType.TOKEN/delta/tool calls in the stream, empty-choices guard) are addressed. The design is sound; the main feedback is about how the provider is implemented rather than the direction.

1. (Blocking, design) Don't re-implement OpenAI-shaped parsing — reuse `OpenAILLM`

litellm.acompletion() returns OpenAI-shaped responses (choices[0].delta.content, tool_calls with index/id/function, usage.prompt_tokens, model_dump()). The current LiteLLM.chat()/stream_chat() hand-roll a second, thinner parser for this same shape, which drops capabilities that OpenAILLM already gets right:

Streaming token usage is lost — chat() calls add_token_usage(), but stream_chat() records nothing and never emits a ChunkType.USAGE chunk (OpenAILLM sets stream_options={"include_usage": True} and emits usage).
Streaming tool-call argument fragments are not accumulated across chunks — OpenAILLM._parse_stream_chunk accumulates by index/id; the new code yields each partial fragment standalone, producing incomplete tool calls.
output_config (json_schema structured output) is silently ignored — OpenAILLM converts it into a concrete response_format.
First-token / inter-token timeout control and the empty-content-raises behavior are also absent.

DeepSeekLLM is the precedent: it subclasses OpenAILLM and reuses all of this instead of duplicating it.

Suggested path, sequenced as two PRs:

A prior, standalone refactor PR that extracts a replaceable "completion call" seam in OpenAILLM (e.g. an overridable _acreate()), since today the API call is inlined inside chat/stream_chat.
This PR rebased on top of it, with LiteLLM becoming a thin OpenAILLM subclass that only overrides the call seam (swap AsyncOpenAI client → litellm.acompletion, pass drop_params=True).

This single change subsumes the streaming-usage, tool-call-accumulation, output_config, empty-content, and code-duplication issues at once.

2. (Blocking, packaging) Declare `litellm` as a dependency

litellm.py imports litellm, but the package is absent from pyproject.toml (neither core deps nor an extra). The lazy in-method import defers the failure, but a standard install yields ModuleNotFoundError on first call — an implicit/phantom dependency. Add litellm to the core [project.dependencies], consistent with the other provider SDKs (openai, anthropic, google-genai, zai-sdk), which are all core.

3. (Required, completeness) Register `litellm` in the provider metadata

The adapter has a "litellm" branch, but litellm is not in _SUPPORTED_PROVIDER_METADATA in providers.py, so it never appears in get_supported_provider_metadata() and is not selectable in the UI provider list. Add a metadata entry so it surfaces in the UI.

Tests

The 20 unit tests cover init, non-stream chat, tool calling, error mapping, and the factory well. Gap: there are no stream_chat tests — which is exactly the path with the parity issues above. Once restructured around OpenAILLM, the inherited streaming path should get coverage.

RheagalFire added 4 commits May 19, 2026 20:55

feat: add LiteLLM as chat model provider for 100+ LLM backends

bcbecd7

fix: rename LiteLLMLLM to LiteLLMChat

6c109d9

fix: rename class to LiteLLM

cc9ca25

test: add 20 unit tests for LiteLLM provider

4455ca0

XprobeBot added the feature label May 19, 2026

gemini-code-assist Bot reviewed May 19, 2026

View reviewed changes

fix: address gemini review - add raw field, use ChunkType.TOKEN, guar…

67988a5

…d empty choices

fix: full stream_chat tool call support and raise on empty choices

242e489

fix: isort import ordering and stream_chat tool call formatting

dbc2cb3

rogercloud reviewed Jun 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add LiteLLM as chat model provider for 100+ LLM backends#456

feat: add LiteLLM as chat model provider for 100+ LLM backends#456
RheagalFire wants to merge 7 commits into
xorbitsai:mainfrom
RheagalFire:feat/add-litellm-provider

RheagalFire commented May 19, 2026

Uh oh!

RheagalFire commented May 19, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Uh oh!

gemini-code-assist Bot May 19, 2026

Uh oh!

gemini-code-assist Bot May 19, 2026

Uh oh!

qinxuye commented May 20, 2026

Uh oh!

RheagalFire commented Jun 3, 2026

Uh oh!

qinxuye commented Jun 5, 2026

Uh oh!

RheagalFire commented Jun 9, 2026

Uh oh!

rogercloud left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

RheagalFire commented May 19, 2026

Summary

Changes

Tests

Example usage

Uh oh!

RheagalFire commented May 19, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

qinxuye commented May 20, 2026

Uh oh!

RheagalFire commented Jun 3, 2026

Uh oh!

qinxuye commented Jun 5, 2026

Uh oh!

RheagalFire commented Jun 9, 2026

Uh oh!

rogercloud left a comment

Choose a reason for hiding this comment

Code Review

1. (Blocking, design) Don't re-implement OpenAI-shaped parsing — reuse OpenAILLM

2. (Blocking, packaging) Declare litellm as a dependency

3. (Required, completeness) Register litellm in the provider metadata

Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

1. (Blocking, design) Don't re-implement OpenAI-shaped parsing — reuse `OpenAILLM`

2. (Blocking, packaging) Declare `litellm` as a dependency

3. (Required, completeness) Register `litellm` in the provider metadata