LiteLLM Rust Migration - the fastest and litest AI Gateway (sub 1ms overheads)

[ishaan-berri]

Starting this issue as the parent ticket for our rust migration, please post questions or issues here related to the rust migration

Blog Post: https://docs.litellm.ai/blog/litellm-rust-launch

Sign up for early Beta Tester Group: here

TLDR: Over the past year, we have heard the same thing from our users and our community: they want the fastest, most lightweight AI gateway they can run. We have heard you. We are addressing it by moving LiteLLM to Rust, and committing to sub-1ms overhead with a sub-100MB memory binary you can deploy. By the end of this migration, you will get a pure Rust server that can serve 100% of your AI traffic, with every hot path operation, including auth and rate limiting, running in Rust.

Note: FYI - if users want new features / models / providers on AI Gateway I'm happy to add it for them

Timeline

Aug 15, 2026	litellm.ocr() for Mistral, then all of litellm.ocr(), then the /ocr route
Sep 1, 2026	Same pattern for /messages, then /chat/completions
Sep 15, 2026	The router: load balancing, fallbacks, retries, cooldowns
Dec 1, 2026	The full server: FastAPI thin shell, then pure Rust (axum)

[ishaan-berri]

Implementation PR for the default LiteLLM pip Rust binary packaging path: #31265

[ishaan-berri]

Updated implementation PR for the default LiteLLM pip Rust binary packaging path: #31267 (supersedes #31265; rebased on litellm_internal_staging and uses a canonical litellm_hotfix_* branch).

[ishaan-berri]

FYI - if users want new features / models / providers on AI Gateway I'm happy to add it for them

[eric642]

Do you also provide the Rust SDK?

[impr3ssi0n]

Will the Rust implementation be fully open source as well? Specifically:

• Will the full Rust source code be available in the LiteLLM repository?
• Will the Rust gateway be possible to build from source?
• Or will it be distributed only as prebuilt binaries?

[ishaan-berri]

@eric642 do you want a rust SDK ? we can do it yes

@impr3ssi0n yes to first 2 questions, is that what you want ?

[eric642]

@ishaan-berri Yes, I want the Rust SDK. It would be fantastic if you could develop it.

[wy6688]

The Current Python version is used too much CPU, and Huge, expected Rust version can improve 100x faster, and 10x less memory usages. and won't need use Docker for self-container.

[lazeyliu]

Will i18n be considered?

[ishaan-berri]

@ishaan-berri Yes, I want the Rust SDK. It would be fantastic if you could develop it.

@eric642 what endpoints would you like to start for an initial v0, i'd love feedback on it

[eric642]

@ishaan-berri For my use case, I want /chat/completions to be prioritized. It would be ideal if /response and /message are also available, but it’s fine if they aren’t.

[asthanayak]

@ishaan-berri This is great news!

The overhead was the only blocker for my use case to integrate LiteLLM into a production workflow.
I'd love to see /responses prioritized. OpenResponses is growing rapidly as the open-source interface standard, and with multiple vendors now supporting it, having a low-overhead path through LiteLLM for Responses API would be a huge win.

[wy6688]

It seems Codex already give up /chat/completions, and only support /responses in newer version, so the rust version better support /responses and /messages translation or passthrough, and configurable per model based, ex: Configure the one model for codex /response API, same model with different name for Anthropic /Message API, or automatically adaptive between Client side API and LLM Model side API.