Agent Vendor Verifier

Agent Vendor Verifier is a benchmarking framework for evaluating Agent tool-call effectiveness.

The benchmark aggregates these dimensions into a single, comparable fusion score (IRF), enabling fair cross-vendor comparison for agent-style tool usage.

Agent Vendor Verifier is built upon K2-Vendor-Verifier.

Why Agent Vendor Verifier?

• Multi-dimensional metrics: Go beyond “was a tool called” by measuring:
  • correctness,
  • schema compliance,
  • request success and stability,
  • latency and throughput.
• Comparable fusion score: combine heterogeneous metrics into a single score for ranking and model/vendor selection.

Metrics

For each sample, the benchmark records the finish_reason (e.g. tool_calls, stop, others) and optional tool-call validation results.

Metric	What it Evaluates	Direction
F1 Score	Whether a model triggers tool calls on the right samples, compared against a designated baseline vendor	Higher is better
Success Rate	Whether requests successfully complete without API or runtime errors	Higher is better
Schema Accuracy	Whether generated tool-call arguments conform to the declared JSON Schema	Higher is better
Avg Token	Token usage efficiency per request (prompt + completion)	Lower is better
Avg TTFT	Responsiveness: time from request to first token (ms)	Lower is better
TPS	Generation performance during decoding (e.g. tokens/s)	Higher is better

F1 Score

F1 score measures whether a model triggers tool calls on the correct samples, compared against a designated baseline vendor for the same model.

• A higher F1 indicates closer alignment with the baseline on when to issue tool calls.

Model	Baseline Vendor
Claude	Anthropic
Gemini	Google
Deepseek	Deepseek
Minimax	Minimax
GLM	Bigmodel
Kimi	Moonshot

Treat “ended with tool_calls” as positive, and “ended with stop or others” as negative.

Case	Baseline result	Current vendor result	Meaning
TP	tool_calls	tool_calls	Both agree to trigger a tool call
FP	stop / others	tool_calls	False trigger
FN	tool_calls	stop / others	Missed trigger
TN	stop / others	stop / others	Both agree not to trigger

The baseline is treated as ground truth for whether a tool call should occur.

F1 score computation:

$$ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} $$

$$ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} $$

$$ \text{F1} = \frac{2 \times \text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$

Note: The baseline vendor’s F1 is fixed at 1.0.

Success Rate

Success rate measures the proportion of requests that complete successfully, without API errors, timeouts, or runtime failures.

Success rate computation:

$$ \text{Success Rate} = \frac{\text{successful requests}}{\text{successful requests} + \text{failed requests}} $$

Schema Accuracy

Schema accuracy measures whether the arguments generated in tool calls conform to the declared JSON Schema provided in the request.

• Schema accuracy directly reflects tool-call output quality.

Schema accuracy computation:

$$ \text{Schema Accuracy} = \frac{\text{valid tool calls}}{\text{total tool calls}} $$

Average Tokens

Average tokens measures the average total token usage per request, including both prompt tokens and completion tokens.

• Lower values indicate better token efficiency.

Average TTFT

Average TTFT (Time to First Token) measures the average latency, in milliseconds, from request submission to the arrival of the first generated token.

• Lower values indicate faster initial response.

TPS

TPS measures the average token generation speed during the decoding phase, typically in tokens per second.

• Higher values indicate faster decoding performance.

Together, these six metrics form the basis for the IRF fusion score used for final ranking.

Fusion Score: IRF (Inverse Rank Fusion)

To combine heterogeneous metrics into a single score, Agent Vendor Verifier uses Inverse Rank Fusion (IRF) across six metrics:

• F1 Score
• Success Rate
• Schema Accuracy
• Avg Token
• Avg TTFT
• TPS

They cover correctness, stability, correctness of arguments, cost, and performance, making IRF a balanced indicator of Agent tool-call effectiveness.

Step 1: Rank per metric

For each metric, rank all participating entities:

• Higher is better: F1, Success Rate, Schema Accuracy, TPS → descending order
• Lower is better: Avg Token, Avg TTFT → ascending order

Step 2: Assign ranks

Each entity receives a rank ( r ) starting from 1 (ties handled by standard ranking rules).

Step 3: IRF contribution

For each metric where an entity has a value, compute:

$$ \text{contribution} = \frac{1}{r + k} $$

where k = 5 (In order to map IRF score to $(0, 1]$).

The final IRF score is the sum over all participating metrics:

$$ \text{IRF} = \sum_{\text{metrics}} \frac{1}{r_{\text{metric}} + 5} $$

• Better ranks across more metrics yield higher IRF scores.
• IRF score is comparable within the same model and different vendors.

Performance

We evaluated the tool-call effectiveness of the latest models across multiple vendors and ranked them using the IRF score. The results are shown below:

deepseek-v3.2

Vendor	IRF Score	Success Rate	F1 Score	TPS	Schema Accuracy	TTFT (ms)	Avg Token
deepseek (openrouter)	0.8234	0.9870	1	62.77	0.9854	1268	1859
google-vertex (openrouter)	0.7885	0.9995	0.6216	101.9	0.8480	777.3	1968
siliconflow (openrouter)	0.7706	0.9980	0.7282	151.5	0.8808	2465	2927
atlascloud (openrouter)	0.7567	1	0.7257	81.03	0.8438	1211	2368
siliconflow	0.7345	0.9815	0.7988	15.15	0.8633	5724	1719

gemini-2.5-flash

Vendor	IRF Score	Success Rate	F1 Score	TPS	Schema Accuracy	TTFT (ms)	Avg Token
google-vertex (openrouter)	0.9524	1	1	249.3	0.6875	1651	1731
google-ai-studio (openrouter)	0.9048	0.9933	0.8810	216.1	0.6897	2626	1658

glm-4.7

Vendor	IRF Score	Success Rate	F1 Score	TPS	Schema Accuracy	TTFT (ms)	Avg Token
bigmodel	0.9107	0.9725	1	123.4	0.8409	2243	1789
z.ai (openrouter)	0.8512	0.9980	0.8575	246	0.8291	4986	2303
atlascloud (openrouter)	0.8452	0.9975	0.8624	103.8	0.8327	1914	2311

kimi-k2

Vendor	IRF Score	Success Rate	F1 Score	TPS	Schema Accuracy	TTFT (ms)	Avg Token
siliconflow	0.9158	1	0.8427	58.23	1	1313	1581
siliconflow (openrouter)	0.8690	0.9975	0.8355	92.81	0.9985	1115	1816
moonshot (openrouter)	0.8205	0.9905	1	46.39	1	2662	1857

minimax-m2

Vendor	IRF Score	Success Rate	F1 Score	TPS	Schema Accuracy	TTFT (ms)	Avg Token
google-vertex (openrouter)	0.9217	0.9990	0.7662	347.6	1	491.7	2556
minimax (openrouter)	0.8512	0.9980	1	147.2	0.8205	1483	2278
atlascloud (openrouter)	0.8324	0.9960	0.7622	152.6	1	745.9	2443

Dataset

We made modifications to the open-sourced dataset from K2-Vendor-Verifier to fix errors that occurred when running it on other models and vendors.

Dataset address: Agent Vendor Verifier Dataset

Supported Vendors

Agent Vendor Verifier is designed for vendors that expose an OpenAI-compatible chat completions API with tool-calling support.

If a provider exposes a compatible OpenAI-style endpoint, you can usually benchmark it by adding the appropriate url, model_id, and api_key in config.yaml.

Agent Vendor Verifier can also be extended further in code by inheriting from the Vendor base class, see Vendor-base.

Installation

pip install agent-vendor-verifier

The package requires Python 3.10 or newer and exposes the agent-vendor-verifier command-line entry point.

Quick Start

Prepare:

• A JSONL dataset. See Dataset.
• A config.yaml file describing the model/vendor combinations to benchmark.

Install and run:

pip install agent-vendor-verifier

agent-vendor-verifier \
  --test-file-path "tool-calls/samples.jsonl" \
  --config-file-path "config.yaml" \
  --vendor-concurrency 5 \
  --request-concurrency 30 \
  --retries 10 \
  --timeout 30

CLI Arguments

agent-vendor-verifier supports the following command-line arguments:

• --test-file-path: path to the input dataset in JSONL format. Each line should be one complete request payload.
• --config-file-path: path to the benchmark configuration YAML file.
• --vendor-concurrency: maximum number of vendors evaluated in parallel.
• --request-concurrency: maximum number of concurrent requests sent per vendor.
• --retries: number of retry attempts for failed requests.
• --timeout: per-request timeout in seconds.
• --temperature: optional generation temperature override applied to requests.
• --max-tokens: optional maximum token override applied to requests.
• --incremental: rerun only failed or new requests while keeping previous successful results.
• --use_raw_completions: use the /v1/completions endpoint instead of chat completions.
• --tokenizer-model: tokenizer model name used with --use_raw_completions.
• --extra-body: extra request parameters as a JSON string, merged into the request body.

Benchmark outputs are written to data/ and rank/ under your current working directory.

Configuration

config.yaml defines which model-vendor combinations will be benchmarked. The benchmark accepts any model that can be called through an OpenAI-compatible API.

For a valid config.yaml:

• The file must be valid YAML.
• It must contain at least one top-level model group.
• Each model group must contain a non-empty vendors list.
• Every vendor entry must include name, model_id, api_key, url, validator, and is_baseline.
• If name: openrouter, the vendor entry must also include provider.
• Each model group must have exactly one vendor with is_baseline: true.

Top-level structure

• Each top-level key is a model group you want to compare.
• Under each model group, vendors is a list of vendor entries.
• Exactly one vendor per model group must have is_baseline: true.

The top-level key is just the benchmark group name used in outputs. It can be any label that helps you identify the model, such as gpt-5.2, claude-sonnet-4-5, or my-custom-openai-model.

Vendor fields

Each entry in vendors should include:

• name: vendor label used in result files, for example openrouter, vertex, bedrock, packyapi, foxcode, or yunwu.
• model_id: the exact model string sent to that vendor API. This does not need to match the top-level key.
• api_key: the API key for that vendor endpoint.
• url: the OpenAI-compatible base URL.
• validator: currently use openai.
• is_baseline: set true for exactly one vendor in the group, and false for the rest.
• provider: required only when name: openrouter.

Example

We provide a minimal example in config_example.yaml. A more complete example looks like this:

claude-sonnet-4-5:
  vendors:
    - name: openrouter
      model_id: anthropic/claude-sonnet-4.5
      api_key: sk-or-v1-xxxx
      provider: anthropic
      url: https://openrouter.ai/api/v1
      validator: openai
      is_baseline: true
    - name: packyapi
      model_id: claude-sonnet-4-5-20250929
      api_key: sk-xxxx
      url: https://api.packyapi.com/v1
      validator: openai
      is_baseline: false

How to choose the top-level key

• If you are comparing multiple vendors for the same underlying model, put them under the same top-level key.
• If two vendors require different model_id values for the same model, keep the same top-level key and set vendor-specific model_id values.
• If you want separate rankings, use separate top-level keys.

Baseline rule

• F1 is computed against the vendor marked with is_baseline: true.
• Each model group must have exactly one baseline vendor.
• A common pattern is to use the vendor you trust most as the baseline, such as openrouter with the canonical provider.

API keys

API keys are read from config.yaml directly.

• For local runs, keep a private config.yaml.

Notes

• url should be the vendor's OpenAI-compatible base URL, usually ending in /v1.
• provider is only needed for openrouter.
• validator should currently be openai for every vendor entry.
• The benchmark assumes the configured endpoint supports OpenAI-style chat completions and tool calling.

Run Command

agent-vendor-verifier \
  --test-file-path "tool-calls/samples.jsonl" \
  --config-file-path "config.yaml" \
  --vendor-concurrency 5 \
  --request-concurrency 30 \
  --retries 10 \
  --timeout 30