Skip to content

feat(api): add OrcaRouter provider - #1648

Closed
XiaoHuo888-hue wants to merge 1 commit into
open-compass:mainfrom
XiaoHuo888-hue:add-orcarouter-provider
Closed

feat(api): add OrcaRouter provider#1648
XiaoHuo888-hue wants to merge 1 commit into
open-compass:mainfrom
XiaoHuo888-hue:add-orcarouter-provider

Conversation

@XiaoHuo888-hue

Copy link
Copy Markdown

Add a named OrcaRouter provider for VLMEvalKit.

OrcaRouter is an OpenAI-compatible gateway that routes to frontier open-source and commercial LLMs through a single endpoint (https://api.orcarouter.ai/v1), with model strings of the form orcarouter/<model> (e.g. the special orcarouter/auto routes each request to the best available model). It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis, with no application code changes.

Changes

  • New vlmeval/api/orcarouter.py: OrcaRouterAPI (a BaseAPI subclass), mirroring the existing OpenAI-compatible providers. Supports text and image (base64 image_url) inputs, ORCAROUTER_API_KEY / ORCAROUTER_API_BASE / api_base overrides, default model orcarouter/auto.
  • Register OrcaRouter_Auto, OrcaRouter_DeepSeek_V4_Pro, OrcaRouter_Qwen3_5_Flash in vlmeval/config.py api_models.
  • Export OrcaRouterAPI from vlmeval/api/__init__.py.
  • Add tests/test_orcarouter_api.py covering init, content/message building, generate_inner, and config registration.

Verification

  • pytest tests/test_orcarouter_api.py: 19 passed.
  • pytest tests/test_litellm_api.py: 22 passed (regression).
  • pre-commit run --all-files: all hooks pass.
  • Live test: OrcaRouterAPI(model='orcarouter/auto') against https://api.orcarouter.ai/v1/chat/completions returned ret_code=0 and the expected response.

Disclosure: I'm an engineer on the OrcaRouter team.

Add a named OrcaRouter provider to VLMEvalKit as an OpenAI-compatible
gateway for frontier open-source and commercial LLMs through a single
endpoint.

- New vlmeval/api/orcarouter.py: OrcaRouterAPI (BaseAPI subclass), text
  and image (base64 image_url) support, ORCAROUTER_API_KEY /
  ORCAROUTER_API_BASE / api_base overrides, default model
  orcarouter/auto.
- Register OrcaRouter_Auto / OrcaRouter_DeepSeek_V4_Pro /
  OrcaRouter_Qwen3_5_Flash in vlmeval/config.py api_models.
- Export OrcaRouterAPI from vlmeval/api/__init__.py.
- Add tests/test_orcarouter_api.py covering init, content/message
  building, generate_inner, and config registration.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: XiaoHuo888-hue <jinhao.song@myflashcloud.com>
@ssiq

ssiq commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Thank you for the contribution and for adding the OrcaRouter integration.

After reviewing the implementation, we don't think a dedicated OrcaRouterAPI wrapper is necessary at this time. OrcaRouter provides an OpenAI-compatible API, and the existing LMDeployAPI already supports
OpenAI-compatible endpoints, including message construction, image/video inputs, generation parameters, retries, and streaming. OrcaRouter can be used by configuring the existing wrapper with the appropriate
api_base, API key, and model name.

The proposed wrapper largely duplicates existing functionality and would add another provider-specific code path to maintain without introducing new functionality to VLMEvalKit.

Therefore, we will close this PR for now. If there is a concrete OrcaRouter-specific behavior that cannot be handled by LMDeployAPI, please feel free to open a follow-up issue or a smaller PR focused on that
functionality. Thank you again for your contribution.

@ssiq ssiq closed this Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants