This document provides complete documentation for all configuration options available in the Kubernetes Inference Performance Benchmark tool.
Controls the API interaction behavior. If SLO headers are present, each request is evaluated for SLO compliance and SLO-related metrics are reported:
api:
type: completion # API type (completion|chat|anthropic_messages)
streaming: true # Enable streaming for TTFT, ITL, and TPOT metrics
headers: # Optional custom HTTP headers
x-inference-model: llama
x-routing-strategy: round-robin
x-slo-tpot-ms: "2"
x-slo-ttft-ms: "1000"
slo_unit: "ms" # Optional SLO unit (e.g., ms, s), default is ms
slo_tpot_header: "x-slo-tpot-ms" # Optional header name for TPOT SLO Header, default is x-slo-tpot-ms
slo_ttft_header: "x-slo-ttft-ms" # Optional header name for TTFT SLO Header, default is x-slo-ttft-msConfigures the test data generation methodology:
data:
type: mock|shareGPT|synthetic|random|shared_prefix|cnn_dailymail|billsum_conversations|infinity_instruct|otel_trace_replay|visionarena # Data generation type
path: ./data/shareGPT/ShareGPT_V3_unfiltered_cleaned_split.json # For shareGPT type, path where dataset to be used is present. Path needs to be set for cnn_dailymail, billsum_conversations and infinity_instruct as well
input_distribution: # For synthetic/random types
min: 10 # Minimum prompt length (tokens)
max: 100 # Maximum prompt length
mean: 50 # Average length
std_dev: 10 # Standard deviation
total_count: 100 # Total prompts to generate
output_distribution: # Same structure as input_distribution
min: 10
max: 100
mean: 50
std_dev: 10
total_count: 100
shared_prefix: # For shared_prefix type
num_groups: 10 # Number of shared prefix groups
num_prompts_per_group: 10 # Unique questions per group
system_prompt_len: 100 # Shared prefix length (tokens)
question_len: 50 # Default question length (tokens), used when question_distribution is absent
output_len: 50 # Default output length (tokens), used when output_distribution is absent
question_distribution: # Optional: distribution for question lengths (overrides question_len)
min: 10
max: 1024
mean: 50
std_dev: 5
output_distribution: # Optional: distribution for output lengths (overrides output_len)
min: 10
max: 1024
mean: 50
std_dev: 5Note: For otel_trace_replay type, see the OpenTelemetry Trace Replay section for complete configuration details.
For VLMs, the synthetic and shared_prefix data types accept an optional multimodal block that produces images, videos, and/or audio alongside the text prompt:
data:
type: synthetic
input_distribution: { ... }
output_distribution: { ... }
multimodal:
image:
count: { type: uniform, min: 1, max: 2, mean: 1.5 } # items per request
insertion_point: 0.0 # 0=prefix, 1=suffix, or a Distribution
resolutions: # AnyResolution or list of weighted resolutions
- resolution: "1080p"
weight: 0.8
- resolution: "4k"
weight: 0.2
video:
count: { type: fixed, min: 1, max: 1, mean: 1 }
insertion_point: 1.0
profiles: # (resolution, frames) tuples
- profile: { resolution: "720p", frames: 32 }
weight: 1.0
audio:
count: { type: fixed, min: 1, max: 1, mean: 1 }
insertion_point: 0.5
durations:
- duration: 5.0
weight: 1.0The reportgen output adds throughput.{images,videos,audios}_per_sec, request_size_bytes, and per-modality distribution blocks (image.{count,pixels,bytes,aspect_ratio}, video.{count,frames,pixels,bytes,aspect_ratio}, audio.{count,seconds,bytes}) to summary_lifecycle_metrics.json.
- Images are PNG by default. Set
image.representation: jpegfor JPEG-encoded payloads (smaller, lossy) — useful when the target VLM expects JPEG or when you want wire-size closer to real client traffic. - Videos carry one
video.representationvalue:mp4(default): onevideo_urlblock carrying an MP4 blob.png_frames: emitframes× PNGimage_urlblocks at one insertion point. No server-side MP4 decode dependency; useful for prefix-cache benchmarks.jpeg_frames: same aspng_framesbut with JPEG-encoded frames — smaller wire payload, matches client pipelines that pre-extract and JPEG-compress frames before sending to the model server.
- Audio is 16-bit mono WAV at 16 kHz (not configurable today).
Multimodal config is passed to the model server as-is with no model-aware validation, so the right starting point is the model's spec sheet.
- Per-request media count — align with vLLM's
--limit-mm-per-prompt(e.g.image=4,video=2,audio=2). Sending more items than the server allows fails at the wire and shows up in the lifecycle report'sfailurescount. - Image resolutions — stay within the vision encoder's pixel cap (Qwen2-VL
min_pixels/max_pixels, LLaVA fixed 336/672, Pixtral tile caps, etc.). Above-cap images get downsized server-side, which means reportedpixels/bytesreflect what was sent, not what the model processed. - Video frames — most VLMs sample to a fixed
num_framesbudget (often 8/16/32). Sending more frames than the server samples wastes wire bytes without changing the workload the model sees. If the server doesn't acceptvideo_urlat all, switch torepresentation: frames. - Audio durations — most audio-capable models cap clips around 30 s (Qwen2-Audio, Whisper-style chunking). Longer clips fail or get truncated.
- Effective context length — images and audio consume context tokens. Large media plus a long text prompt can exceed
--max-model-lenand fail.
When in doubt: start small, watch the failures count in the report, and ramp up resolutions/counts/durations once the success path is solid. There is no portable API across vLLM / SGLang / TGI to query these limits at runtime; a per-model capability registry that validates the multimodal config at load time is tracked as future work.
Defines the benchmarking load pattern:
load:
type: constant|poisson|concurrent|trace_session_replay # Load pattern type
interval: 1.0 # Seconds between request batches
stages: # Load progression stages
- rate: 1 # Requests per second (CONSTANT or POISSON LOADS)
duration: 30 # Seconds to maintain this rate (CONSTANT or POISSON LOADS)
concurrency_level: 3 # Level of concurrency/number of worker threads (CONCURRENT LOADS)
num_requests: 40 # Number of requests to be processed by concurrency_level worker threads (CONCURRENT LOADS)
num_workers: 4 # Concurrent worker threads (default: CPU_cores)
worker_max_concurrency: 10 # Max concurrent requests per worker
worker_max_tcp_connections: 2500 # Max TCP connections per worker
request_timeout: 900 # Optional: per-request timeout in seconds (applies per attempt)
request_retries: 0 # Optional: extra attempts for faults before response headers (default: 0, no retry)
request_retry_backoff_sec: 0.5 # Optional: base backoff before the first retry, doubled + jittered per attempt
base_seed: 12345 # Optional: base random seed for reproducibility (default: current time in ms)
lora_traffic_split: # Optional: MultiLoRA traffic splitting
- name: adapter_1 # LoRA adapter name
split: 0.5 # Traffic weight (must sum to 1.0)
- name: adapter_2
split: 0.5Note: trace_session_replay load type has different stage parameters. See OpenTelemetry Trace Replay for configuration details.
request_retries re-sends a request that fails before response headers were obtained —
a refused connection, a reset, or a connection dropped from the pool. It defaults to 0,
preserving the historical behavior of failing on the first fault.
What is retried:
| Fault | Retried | Why |
|---|---|---|
| Connection refused, reset, dropped from pool | Yes | Nothing was measured on the client side; these recover |
| Anything after a response was established | No | Re-sending could mix or discard token-level measurements and duplicate server work |
| Timeout | No | A real measurement of a slow server, and request_timeout applies per attempt |
| Rejected certificate, fingerprint mismatch | No | Configuration errors — every attempt fails identically |
"No response headers" is not "the server did no work": the request bytes may already have been sent. A retry is therefore bounded and backed off rather than unconditional.
Timing. start_time stays at the logical request's dispatch and is never re-stamped per
attempt, so end-to-end latency counts the failed attempt and its backoff — the workload
really did wait — and schedule_delay, send_duration and achieved_rate stay correct.
The cost is reported beside those numbers instead: a retried request carries
info.retry_wasted_sec plus a matching OTel span attribute, and the retries block totals
it across the run. A request that never succeeded counts its whole life as waste, since no
attempt answered.
Retry fields are omitted rather than zeroed on anything that never retried, so a run with
request_retries: 0 — or with the knob on but no faults — produces the report JSON it
produced before retries existed.
Costs. request_timeout applies per attempt, so worst-case wall time per request
becomes (1 + request_retries) × request_timeout plus backoff; keep request_retries small
when request_timeout is large. Backoff starts at request_retry_backoff_sec, then doubles
and jitters so retries do not resynchronize into a burst.
See Reports for the retries block and the CLI columns.
Defines the preprocessing phase to determine load based on target service saturation.
load:
type: constant|poisson
interval: 15
sweep: # Automatically determine saturation point of the target service and generate stages
type: linear|geometric # Produce a linear distribution [1.0, saturation] of rates for num_stages or geometric distribution clustered around the saturation point
timeout: 60 # Length of time to run load to determine saturation
num_stages: 5 # Number of stages to generate
stage_duration: 180 # Duration of each generated stage
saturation_percentile: 95 # Percentile of sampled rates to select as saturation pointConfigures connection to the model serving backend:
server:
type: vllm # Currently only vLLM supported
model_name: "HuggingFaceTB/SmolLM2-135M-Instruct" # Required model identifier
base_url: "http://0.0.0.0:8000" # Required server endpoint
ignore_eos: true # Whether to ignore End-of-Sequence tokens
api_key: "" # Optional API key for authenticated endpointsSets up performance metrics collection:
metrics:
type: prometheus|default # Metrics backend type
prometheus: # Required when type=prometheus
url: "http://localhost:9090" # Prometheus server URL
scrape_interval: 15 # Metrics scrape interval (seconds)
google_managed: false # Whether using Google Managed Prometheus (see 'Google Managed Prometheus (GMP) Requirements' section)
filters: [] # List of metric names to collectWhen setting google_managed: true, inference-perf queries the GMP API directly. You must configure Application Default Credentials (ADC) in your environment with sufficient permissions.
-
Required Permissions The identity used by ADC must have the Monitoring Viewer role:
roles/monitoring.viewer
-
Environment Configuration
- GKE Cluster: Ensure the Pod is running with Workload Identity enabled and linked to a Google Service Account (GSA) with the required role.
- GCE VM: Ensure the VM's attached Service Account has the required role.
- Local Development: Authenticate using your user credentials:
gcloud auth application-default login
Note: Your personal user account must have the
monitoring.viewerrole on the target GCP project.
Common Error: Failing to configure these permissions will result in API errors similar to:
ERROR - error executing query: 403 Client Error: Forbidden for url: [https://monitoring.googleapis.com/v1/projects/](https://monitoring.googleapis.com/v1/projects/)...
Controls benchmark report generation:
report:
request_lifecycle:
summary: true # Generate high-level summary
per_stage: true # Include breakdown by load stage
per_request: false # Enable detailed per-request logs (verbose)
per_request_fields: # Control fields included in per_request_lifecycle_metrics.json
request: true # Include raw request payloads
response: true # Include raw response payloads
info: true # Include structured request/response metadata
response_chunks: true # Include raw streaming chunks inside info.response_metrics
computed_metrics: false # Include computed per-request latency metrics (TTFT/TPOT/ITL/...)
per_adapter: false # Generate metrics grouped by LoRA adapter
per_adapter_stage: false # Generate metrics grouped by adapter and stage
percentiles: [0.1, 1, 5, 10, 25, 50, 75, 90, 95, 99, 99.9] # List of percentiles to calculate
use_server_output_tokens: false # Treat the server's usage.completion_tokens as the source of truth for output tokens.
max_error_messages: 100 # Max number of unique error messages retained per error label (failures.by_label) and per bad tool call substitution entry
use_server_output_tokens: false # Treat the server's usage.completion_tokens as the source of truth for output tokens.
prometheus:
summary: true # Include Prometheus metrics summary
per_stage: false # Disable Prometheus stage breakdownFor metrics-only per-request reports, disable the raw fields and enable
computed_metrics instead — this removes the multi-GB raw payloads while
keeping full per-request latency analysis:
report:
request_lifecycle:
per_request: true
per_request_fields:
request: false
response: false
info: false
response_chunks: false
computed_metrics: trueSetting info: false removes the entire info block, including
response_chunks; in that case response_chunks has no effect.
computed_metrics (off by default) adds a computed_metrics block to each
successful entry with request_latency, normalized_time_per_output_token,
time_to_first_token, time_per_output_token, inter_token_latency (mean),
inter_token_latencies (per-token deltas), input_tokens, output_tokens,
and the request's ttft_slo_sec/tpot_slo_sec SLOs — the same values (and
the same tokenizer-based correction from raw streaming chunks) used to compute
the summary_lifecycle_metrics.json / stage_*_lifecycle_metrics.json
aggregates. Non-streamed or non-streamable requests report null for
time_to_first_token, time_per_output_token, and inter_token_latency.
Configures storage for benchmark results:
storage:
local_storage:
path: "reports-{timestamp}" # Local directory path
report_file_prefix: null # Optional filename prefix
google_cloud_storage: # Optional GCS configuration
bucket_name: "your-bucket-name" # Required GCS bucket
path: "reports-{timestamp}" # Optional path prefix
report_file_prefix: null # Optional filename prefix
simple_storage_service:
bucket_name: "your-bucket-name" # Required S3 bucket
path: "reports-{timestamp}" # Optional path prefix
report_file_prefix: null # Optional filename prefix
endpoint_url: null # Optional custom endpoint (e.g. for S3-compatible stores)
region_name: null # Optional AWS region name
addressing_style: null # Optional: "auto" (default), "virtual", or "path".Optional tokenizer configuration for specialized tokenization:
tokenizer:
pretrained_model_name_or_path: "model-id" # Required model path
trust_remote_code: true # Whether to trust custom tokenizer code
token: "" # HuggingFace access token for private models
load_timeout: 300.0 # Deadline in seconds for loading the tokenizer,
# including any download from Hugging Face Hub.
# Set to null to disable. Default: 300.load_timeout: null can only be set in a YAML config file; the --tokenizer.load_timeout CLI flag parses a float and rejects null. The deadline applies to each tokenizer construction independently; a run constructs a tokenizer in several stages (data generation, the model server client, report generation), so the worst-case total wait is a small multiple of load_timeout.
data:
type: shareGPT
load:
type: constant
stages:
- rate: 1
duration: 30
api:
type: chat
server:
type: vllm
model_name: HuggingFaceTB/SmolLM2-135M-Instruct
base_url: http://0.0.0.0:8000load:
type: constant
stages:
- rate: 1
duration: 30
api:
type: completion
server:
type: vllm
model_name: HuggingFaceTB/SmolLM2-135M-Instruct
base_url: http://0.0.0.0:8000
ignore_eos: true
tokenizer:
pretrained_model_name_or_path: HuggingFaceTB/SmolLM2-135M-Instruct
data:
type: random
use_chat_template: false # wrap each prompt in the tokenizer's chat template (single user turn) before sending.
# Input lengths then target the fully templated prompt, so the server-side prefill
# token count still matches the configured length. Requires a tokenizer with a chat
# template, and every input length must exceed the template's fixed token overhead.
input_distribution:
min: 10 # min length of the synthetic prompts
max: 100 # max length of the synthetic prompts
mean: 50 # mean length of the synthetic prompts
std_dev: 10 # standard deviation of the length of the synthetic prompts
total_count: 100 # total number of prompts to generate to fit the above mentioned distribution constraints
output_distribution:
min: 10 # min length of the output to be generated
max: 100 # max length of the output to be generated
mean: 50 # mean length of the output to be generated
std_dev: 10 # standard deviation of the length of the output to be generated
total_count: 100 # total number of output lengths to generate to fit the above mentioned distribution constraints
metrics:
type: prometheus
prometheus:
url: http://localhost:9090
scrape_interval: 15
report:
request_lifecycle:
summary: true
per_stage: true
per_request: true
prometheus:
summary: true
per_stage: trueload:
type: constant
stages:
- rate: 1
duration: 30
api:
type: chat
server:
type: vllm
model_name: ./models/SmolLM2-135M-Instruct
base_url: http://0.0.0.0:8000
ignore_eos: true
tokenizer:
pretrained_model_name_or_path: ./models/SmolLM2-135M-Instruct
data:
type: shareGPT
path: ./data/shareGPT/ShareGPT_V3_unfiltered_cleaned_split.json # path to the downloaded shareGPT dataset
metrics:
type: prometheus
prometheus:
url: http://localhost:9090
scrape_interval: 15
report:
request_lifecycle:
summary: true
per_stage: true
per_request: false
prometheus:
summary: true
per_stage: trueReplay real-world LLM workloads captured as OpenTelemetry traces. This feature enables benchmarking with production traffic patterns, including complex dependency graphs, multi-turn conversations, and agent workflows.
OTel trace replay reconstructs the original call graph from trace files, preserving:
- Sequential dependencies — requests that must wait for predecessors
- Parallel fan-outs — concurrent requests with no dependencies
- Shared-prefix patterns — requests sharing common message history (KV-cache opportunities)
- Output-aware replay — substitutes recorded assistant messages with actual generated text for realistic growing-context behavior
-
Trace → Replay Graph: Each trace file is converted to a directed acyclic graph (DAG) where:
- LLM spans become nodes
- Dependencies are inferred from message content (assistant messages matching predecessor outputs)
- Timing gaps between calls are preserved as
wait_msdelays
-
Session-Based Execution: Each trace file represents one session. The load generator controls:
- How many sessions run concurrently (
concurrent_sessions) - How many sessions to process per stage (
num_sessions) - Optional rate limiting for session starts (
session_rate)
- How many sessions run concurrently (
-
Output Substitution: When a request depends on a predecessor's output, the recorded assistant message is replaced at runtime with the actual generated text, ensuring realistic KV-cache behavior for multi-turn conversations and agent chains.
data:
type: otel_trace_replay
otel_trace_replay:
# Source — specify one:
trace_files: # List of specific trace files
- "path/to/trace1.json"
- "path/to/trace2.json"
trace_directory: "path/to/traces/" # OR: all .json files in directory
# Model configuration
use_static_model: true # Override recorded model names
static_model_name: "my-model" # Model to use for all requests
model_mapping: # OR: remap per recorded name
"gpt-4": "my-model"
"gpt-3.5-turbo": "my-other-model"
# Generation parameters
default_max_tokens: 1000 # Fallback if output tokens are set to 0 in the otel file
# Error handling
include_errors: false # Skip spans with error status, that is, status != 0 (default)
skip_invalid_files: true # Skip unparseable trace files during replay
# Tool-call mitigation (client-side, default disabled)
bad_tool_call_handling: none # none|use_recorded — see docs/otel_trace_replay.md#bad-tool-call-handling
# Output replay fidelity (default disabled)
disable_output_substitution: false # true = send recorded assistant outputs as-is (no live substitution); conflicts with inject_random_session_id / duplicate_sessions_target
load:
type: trace_session_replay # Required for otel_trace_replay
stages:
- concurrent_sessions: 4 # Max sessions active simultaneously
num_sessions: 20 # Run 20 sessions in this stage
session_rate: 2.0 # Optional: start max 2 sessions/sec
timeout: 300 # Optional: stage timeout in seconds
num_workers: 4 # Worker processes
worker_max_concurrency: 10 # Max concurrent requests per workerconcurrent_sessions (required): Controls session-level concurrency
0= unlimited (all sessions active at once, stress test mode)N > 0= at most N sessions active; when one completes, the next starts
num_sessions (optional): Number of sessions to run in this stage
- If omitted, runs all remaining sessions in the corpus
- Stages advance through the corpus sequentially (like standard load stages)
session_rate (optional): Rate limit for starting new sessions
- Omit for no rate limiting
- Useful for controlled ramp-up scenarios
timeout (optional): Wall-clock safety limit
- If exceeded, in-flight sessions are cancelled and stage exits as FAILED
Traces must be JSON files with a spans array. Each LLM span requires:
{
"span_id": "unique-id",
"trace_id": "trace-id",
"start_time": "2024-01-01T00:00:00Z",
"end_time": "2024-01-01T00:00:01Z",
"name": "chat gpt-4",
"attributes": {
"gen_ai.request.model": "gpt-4",
"gen_ai.input.messages": "[{\"role\":\"user\",\"content\":\"hello\"}]",
"gen_ai.output.text": "hi there",
"gen_ai.usage.prompt_tokens": 10,
"gen_ai.usage.completion_tokens": 5
}
}Token counts are read from gen_ai.usage.prompt_tokens / gen_ai.usage.completion_tokens (also accepts input_tokens / output_tokens). If absent, a 4 chars/token estimate is used.
Simple Sequential Replay:
data:
type: otel_trace_replay
otel_trace_replay:
trace_files:
- "examples/otel/test_traces/simple/simple_chain.json"
use_static_model: true
static_model_name: "HuggingFaceTB/SmolLM2-135M-Instruct"
default_max_tokens: 100
load:
type: trace_session_replay
stages:
- concurrent_sessions: 1 # One session at a time
num_workers: 4
worker_max_concurrency: 10
server:
type: vllm
base_url: "http://localhost:8000"
model_name: "HuggingFaceTB/SmolLM2-135M-Instruct"Multi-Stage with Rate Limiting:
data:
type: otel_trace_replay
otel_trace_replay:
trace_directory: "examples/otel/test_traces/advanced"
use_static_model: true
static_model_name: "my-model"
load:
type: trace_session_replay
stages:
- concurrent_sessions: 2
num_sessions: 10
session_rate: 1.0 # Warm-up: 2 concurrent, 1/sec start rate
- concurrent_sessions: 5
num_sessions: 20
session_rate: 2.0 # Ramp-up: 5 concurrent, 2/sec start rate
- concurrent_sessions: 10
# Final stage: 10 concurrent, all remaining sessions
num_workers: 8
worker_max_concurrency: 20- Production Traffic Replay: Benchmark with real user interaction patterns
- Agent Workflow Testing: Replay complex multi-step agent traces with tool calls
- Multi-Turn Conversation Analysis: Test KV-cache efficiency with realistic conversation flows
- Dependency Graph Validation: Verify server behavior under complex request dependencies
- Comparative Analysis: Replay the same traces against different model configurations
Unlike standard data generators that produce independent requests, OTel trace replay operates at the session granularity. Each session is a complete trace file with an internal dependency graph. The load generator:
- Activates sessions according to
concurrent_sessionslimit - Dispatches all events for a session immediately (workers handle internal parallelism)
- Each event blocks until its predecessors complete and outputs are available
- Tracks session completion and starts new sessions as slots become available
This design preserves the causal structure of the original workload while allowing the load generator to control session-level concurrency and throughput.