feat(api): add OrcaRouter provider - #1648
Conversation
Add a named OrcaRouter provider to VLMEvalKit as an OpenAI-compatible gateway for frontier open-source and commercial LLMs through a single endpoint. - New vlmeval/api/orcarouter.py: OrcaRouterAPI (BaseAPI subclass), text and image (base64 image_url) support, ORCAROUTER_API_KEY / ORCAROUTER_API_BASE / api_base overrides, default model orcarouter/auto. - Register OrcaRouter_Auto / OrcaRouter_DeepSeek_V4_Pro / OrcaRouter_Qwen3_5_Flash in vlmeval/config.py api_models. - Export OrcaRouterAPI from vlmeval/api/__init__.py. - Add tests/test_orcarouter_api.py covering init, content/message building, generate_inner, and config registration. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: XiaoHuo888-hue <jinhao.song@myflashcloud.com>
|
Thank you for the contribution and for adding the OrcaRouter integration. After reviewing the implementation, we don't think a dedicated OrcaRouterAPI wrapper is necessary at this time. OrcaRouter provides an OpenAI-compatible API, and the existing LMDeployAPI already supports The proposed wrapper largely duplicates existing functionality and would add another provider-specific code path to maintain without introducing new functionality to VLMEvalKit. Therefore, we will close this PR for now. If there is a concrete OrcaRouter-specific behavior that cannot be handled by LMDeployAPI, please feel free to open a follow-up issue or a smaller PR focused on that |
Add a named OrcaRouter provider for VLMEvalKit.
OrcaRouter is an OpenAI-compatible gateway that routes to frontier open-source and commercial LLMs through a single endpoint (
https://api.orcarouter.ai/v1), with model strings of the formorcarouter/<model>(e.g. the specialorcarouter/autoroutes each request to the best available model). It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.Changes
vlmeval/api/orcarouter.py:OrcaRouterAPI(aBaseAPIsubclass), mirroring the existing OpenAI-compatible providers. Supports text and image (base64image_url) inputs,ORCAROUTER_API_KEY/ORCAROUTER_API_BASE/api_baseoverrides, default modelorcarouter/auto.OrcaRouter_Auto,OrcaRouter_DeepSeek_V4_Pro,OrcaRouter_Qwen3_5_Flashinvlmeval/config.pyapi_models.OrcaRouterAPIfromvlmeval/api/__init__.py.tests/test_orcarouter_api.pycovering init, content/message building,generate_inner, and config registration.Verification
pytest tests/test_orcarouter_api.py: 19 passed.pytest tests/test_litellm_api.py: 22 passed (regression).pre-commit run --all-files: all hooks pass.OrcaRouterAPI(model='orcarouter/auto')againsthttps://api.orcarouter.ai/v1/chat/completionsreturnedret_code=0and the expected response.Disclosure: I'm an engineer on the OrcaRouter team.