Skip to content

Feature: Introduce LiteLLM client for multi-provider model routing#529

Merged
sng-asyncfunc merged 4 commits into
AsyncFuncAI:mainfrom
marazik:feature/litellm-integration-code
Jun 3, 2026
Merged

Feature: Introduce LiteLLM client for multi-provider model routing#529
sng-asyncfunc merged 4 commits into
AsyncFuncAI:mainfrom
marazik:feature/litellm-integration-code

Conversation

@marazik

@marazik marazik commented May 29, 2026

Copy link
Copy Markdown
Contributor

📄 Summary of PR series:
This PR is part of a 4-PR integration effort to add LiteLLM support to DeepWiki-Open while keeping the system fully backward compatible.

PR breakdown:
PR1 – Add optional LiteLLM Docker Compose setup for local LLM gateway
PR2 (current) – Add LiteLLM as a new provider in the application layer
PR3 – LiteLLM Docker/profile setup (wiring PR1 + PR2)
PR4 – Documentation and usage guide updates

📄 Summary (this PR):
This PR introduces LiteLLM as a new LLM provider in DeepWiki-Open using its OpenAI-compatible API interface. It allows DeepWiki to route inference requests through LiteLLM without modifying existing provider implementations.

🔧 Changes:
Added LiteLLMClient implementing OpenAI-compatible sync and async clients
Integrated LiteLLM into provider system via config.py
Added litellm provider routing in websocket_wiki.py
Updated provider mapping and environment variable support (LITELLM_API_KEY)

🧠 Design Notes:
LiteLLM is treated as a drop-in OpenAI-compatible provider
No changes were made to default providers or existing behavior
Existing OpenAI / OpenRouter / AWS / Google flows remain unchanged
LiteLLM is opt-in via configuration (provider: litellm)

🔄 Compatibility:
This change is fully backward compatible:

Default provider behavior is unchanged
Existing deployments do not require LiteLLM
No breaking changes to existing APIs

🧪 Testing:

Tested with:
Streaming LLM responses via WebSocket
OpenAI-compatible request flow
Multiple model routing via LiteLLM proxy

📝 Notes for reviewers:
This PR focuses strictly on application-level integration
Deployment-specific configuration (Docker profiles, compose overrides) will be handled in follow-up PRs
Embedding support is intentionally kept consistent with existing provider abstraction

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for LiteLLM as an OpenAI-compatible provider by introducing LiteLLMClient, registering the provider in the configuration, and updating the WebSocket chat handler to support LiteLLM streaming and fallback requests. The review feedback identifies design and correctness issues in LiteLLMClient (such as constructor contract violations, ignored arguments, and URL duplication bugs) and suggests overriding __init__ to resolve them. Additionally, the reviewer recommends removing commented-out fallback streaming logic in the WebSocket handler.

Comment thread api/litellm_client.py
Comment thread api/websocket_wiki.py Outdated
@marazik marazik changed the title Introduce LiteLLM client for multi-provider model routing Feature: Introduce LiteLLM client for multi-provider model routing May 30, 2026
@sng-asyncfunc sng-asyncfunc merged commit 16f35a0 into AsyncFuncAI:main Jun 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants