feat: add LiteLLM backend for multi-provider benchmarking#800
feat: add LiteLLM backend for multi-provider benchmarking#800RheagalFire wants to merge 2 commits into
Conversation
|
Hi @RheagalFire, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this. |
RheagalFire
force-pushed
the
feat/add-litellm-provider
branch
2 times, most recently
from
d0d1bda to
1b1135f
Compare
Generated-by: Claude claude-opus-4-6 Signed-off-by: RheagalFire <arishalam121@gmail.com>
RheagalFire
force-pushed
the
feat/add-litellm-provider
branch
from
1b1135f to
8e17f9b
Compare
|
cc @sjmonson |
sjmonson
left a comment
sjmonson
left a comment
There was a problem hiding this comment.
Just a quick issue I noticed. Also I am guessing you didn't run with pre-check locally since at minimum you will need to regen the lock file after changing dependencies. You can do that with tox run -e lock and plain tox run will run all the other CI tasks if you want to have a faster feedback loop.
- Bump minimum litellm version to >=1.83.0 (post supply-chain fix) - Remove <1.87.0 upper bound to allow latest releases - Regenerate uv.lock via tox run -e lock - Fix ruff formatting issues Signed-off-by: Aarish Irani <rheagalfire@gmail.com>
|
@sjmonson Fixed the version bounds. |
sjmonson
left a comment
sjmonson
left a comment
There was a problem hiding this comment.
A few minor nits but otherwise LGTM. Needs a rebase due to lockfile changes. The easiest way I have found to do this is to run rebase and when you get to the lockfile conflict:
git restore --staged --worktree uv.lock
uv lock # or `tox -e lock`
git add uv.lock
git rebase --continue| class _ChunkResult: | ||
| __slots__ = ( | ||
| "first_token", | ||
| "input_tokens", | ||
| "output_tokens", | ||
| "response_id", | ||
| ) | ||
|
|
||
| def __init__( | ||
| self, | ||
| *, | ||
| response_id: str | None, | ||
| input_tokens: int | None, | ||
| output_tokens: int | None, | ||
| first_token: bool, | ||
| ): | ||
| self.response_id = response_id | ||
| self.input_tokens = input_tokens | ||
| self.output_tokens = output_tokens | ||
| self.first_token = first_token |
There was a problem hiding this comment.
Its not critical but this makes a lot more sense as a dataclass or a pydantic model.
There was a problem hiding this comment.
A @dataclass is simple & ideal for a logic-free data-carrier class like this.
| from litellm import acompletion as acompletion | ||
| from litellm import completion as completion |
There was a problem hiding this comment.
So this is my bad but I found out that __all__ is the technically correct way to export from typestubs. I have #893 to fix this on the existing extras but if you could convert this to the following that would be helpful:
| from litellm import acompletion as acompletion | |
| from litellm import completion as completion | |
| from litellm import acompletion, completion | |
| __all__ = ["acompletion", "completion"] |
| class _ChunkResult: | ||
| __slots__ = ( | ||
| "first_token", | ||
| "input_tokens", | ||
| "output_tokens", | ||
| "response_id", | ||
| ) | ||
|
|
||
| def __init__( | ||
| self, | ||
| *, | ||
| response_id: str | None, | ||
| input_tokens: int | None, | ||
| output_tokens: int | None, | ||
| first_token: bool, | ||
| ): | ||
| self.response_id = response_id | ||
| self.input_tokens = input_tokens | ||
| self.output_tokens = output_tokens | ||
| self.first_token = first_token |
There was a problem hiding this comment.
A @dataclass is simple & ideal for a logic-free data-carrier class like this.
3 participants
Summary
Adds a new
litellmbackend that routes generation requests through the LiteLLM SDK, enabling benchmarking across 100+ providers (Anthropic, Gemini, Bedrock, Groq, Cohere, Mistral, etc.) via a unified interface. Timing instrumentation matches the existing OpenAI HTTP backend so benchmark results are directly comparable.Details
LiteLLMBackendandLiteLLMBackendArgsfollowing the existingBackend/BackendArgsregistration patternlitellm.acompletion(stream=True)withdrop_params=Truefor cross-provider compatibilityChatCompletionsRequestHandler.format()to build messages fromGenerationRequest.columnsguidellm.extras.litellmso the optional dep doesn't break imports when not installedlitellm>=1.80.0,<1.87.0added as optional dependency under[project.optional-dependencies].litellmTest Plan
pytest tests/unit/backends/litellm/ -vto verify unit testspytest tests/unit/backends/ -vto verify no regressions in existing backendsanthropic/claude-sonnet-4-6via Azure Foundry:Related Issues
Use of AI
git log
commit 8e17f9b
Author: RheagalFire arishalam121@gmail.com
Date: Tue Jun 16 23:51:47 2026 +0530
commit 93c94ca
Author: Aarish Irani rheagalfire@gmail.com
Date: Wed Jul 1 20:00:26 2026 +0530
Generated-by: Claude claude-opus-4-6
Signed-off-by: RheagalFire arishalam121@gmail.com
Signed-off-by: Aarish Irani rheagalfire@gmail.com