performance.md

Performance

Last modified: 2026-08-16

What SBproxy delivers on real hardware, with the methodology you'd need to reproduce it.

TL;DR

On an 8 vCPU GCE instance, single binary, zero tuning beyond the defaults:

• 77,758 rps through a passthrough proxy at 0.6 ms p99.
• 138,770 rps on a cache hit at 0.3 ms p99.
• 50,713 rps running the full chain (auth, rate limit, transforms, cache) at 0.6 ms p99.
• 77,784 rps for non-streaming AI gateway requests against a mocked LLM upstream.
• 0.3 ms p50 at the median proxy path. Most p99s land under 1 ms.

These are publishable medians from 60-second runs across three replicates. Run details below; raw artifacts and the full reproducibility recipe live in sbproxy-bench.

None of these numbers can be reproduced from this repository. The scenario files, the Terraform that provisions the GCE instances, the load-generator configs, and the raw per-replicate oha output all live in that separate repo. Cloning this one and running something will not regenerate the table. What this tree does ship is a smaller local harness, described under Reproduce locally, which is the right tool for measuring your own config anyway.

This page also says nothing about memory. For resident set, pod sizing, and OOM triage, see capacity-planning.md, which is honest about the fact that memory under load has never been measured here.

Headline numbers

Matrix-v7 publishable run, c3-standard-8 GCE instances, LTO-enabled release build (lto = "fat", codegen-units = 1), 60 s × 3 replicates per scenario, medians shown.

Scenario	rps	p50	p99	What it tests
Passthrough	77,758	0.233 ms	0.618 ms	Bare proxy. No policies, no transforms.
WAF blocking	185,049	0.103 ms	0.166 ms	Requests rejected by WAF before upstream.
Rate limit (sliding window)	67,312	0.287 ms	0.443 ms	Per-IP rate limit at admit threshold.
CEL policy	55,810	0.356 ms	0.530 ms	Custom CEL expression on every request.
Cache hit	138,770	0.132 ms	0.302 ms	Response served from in-process cache.
Cache (stale-while-revalidate)	142,108	0.131 ms	0.284 ms	SWR path returns cache, refreshes async.
Full chain	50,713	0.382 ms	0.618 ms	Auth + rate limit + cache + transforms + proxy.
Idle connections	126,270	3.8 ms	8.4 ms	500 mostly-idle keep-alives plus traffic.
AI proxy (non-streaming)	77,784	0.242 ms	0.515 ms	OpenAI-compatible request, mocked LLM upstream.
AI proxy (streaming)	196	101.8 ms	102.4 ms	SSE streaming. Throughput is upstream-bound.
AI failover	11,460	1.721 ms	2.161 ms	Provider primary errors, fallback served.
AI streaming guardrails	22,228	0.897 ms	1.139 ms	Output guardrails scanning each SSE chunk.

How to read this

Latency, not just throughput. SBproxy's design priority is tight tail latency. The p99 column is the one that matters in production. Most proxy-path scenarios land p99 under 1 ms; the cache and WAF scenarios land under 0.5 ms.

The full-chain number is the realistic one. "Passthrough" is a useful ceiling, but real configs do work: parse a JWT, check a rate limiter, run a transform, look at the cache, then call upstream. Full-chain at 50k rps with 0.6 ms p99 is what you should expect when you stack features.

The AI streaming row looks slow on purpose. SSE streaming throughput is gated by the upstream model's token generation rate. The interesting numbers there are the per-chunk overhead and time-to-first-byte, not rps.

WAF "blocking" is fast because it short-circuits. That 185k rps is requests SBproxy rejects before they ever touch upstream. It's a different number from "throughput when traffic is clean," but it's the right number when you're sizing for an attack.

Where these numbers are weak

Be honest with yourself about coverage:

• Two scenarios are upstream-bound, not proxy-bound. AI streaming (196 rps) and AI failover (11,460 rps) reflect upstream behavior, not Pingora's ceiling.
• Localhost numbers in older docs are lower. Single-laptop runs hit ephemeral-port exhaustion around 150 concurrent connections and conflate proxy work with the load generator's CPU. Use the c3 numbers above as the trustworthy floor; expect higher on bigger hardware.
• Hardware matters. c3-standard-8 is a Sapphire Rapids instance with dedicated cores. Burstable VMs (e2, t-series) or AMD Milan (n2d) will land lower; recent EPYC and bare metal will land higher.
• Configuration matters. Logging at debug, full-body logging, or expensive Lua transforms can each cut throughput in half.
• There is no memory column, and that is a real gap. Every scenario above reports rps and latency and nothing about resident set, so this table cannot tell you how big an instance to buy. The one memory figure published anywhere is an idle process on a developer laptop. capacity-planning.md has the empty table and the commands that fill it.
• The AI rows predate the current AI-gateway feature set. The matrix-v7 run measured the AI proxy path before the usage ledger, the CEL policy engine, the guardrail mesh, outcome-aware routing, and predictive budgets landed, and before the model host could serve weights on a local GPU. Each of those adds work per request, so treat the AI rows as a ceiling for a fully configured gateway and re-run the recipe with your config. For AI router strategy comparisons (round-robin vs peak-EWMA vs prefix-affinity and friends), see ai-lb-benchmark.md. Requests answered by the model host are bound by the engine's token generation rate, like the streaming row, not by the proxy.

If you need numbers for your scenario, run the benchmark recipe yourself. Don't take the table above on faith.

Clustered: a 3-node mesh, measured

Everything above is single-node. This section is a 3-node gossip-mesh run (July 2026), because a proxy that markets clustering owes you multi-node numbers. Same zone and instance classes as the single-node matrix: c3-standard-8 per proxy node, a c3-standard-8 origin running the echo + mock-ai container, and a c3-standard-22 load generator, all in us-central1-a. The scenario is the AI proxy non-streaming recipe (OpenAI-compatible POST against the mocked upstream, 20 connections per stream, 30 s warmup, three 60 s replicates, median reported). Nodes were meshed with proxy.cluster in shared_key mode and per-minute governed-key limits in the approximate tier. Run these numbers against each other, not against the matrix-v7 table: the build and image tuning differ enough that cross-table comparisons mislead.

Measurement	Result
Single node, standalone (no cluster block)	43,958 rps, 0.436 ms p50, 0.840 ms p99
Same node, mesh enabled, 3-node cluster formed	43,129 rps, 0.445 ms p50, 0.851 ms p99
All 3 nodes loaded concurrently (aggregate)	119,178 rps, ~0.45 ms p50, ~1.33 ms p99, zero errors
Governed-key spend visible on a peer node	15 to 20 s
Node killed mid-run: surviving node's error rate	zero (throughput held)
Killed node restarted: back in the 3-node directory	about 10 s

What the rows mean:

• The mesh stays off the request path. Joining a 3-node cluster cost the measured node 1.9 percent of throughput and about 10 microseconds of p99, which is within run-to-run noise for this recipe. Gossip, membership, and typed-state exchange are out-of-band of request serving.
• Aggregate scales like independent nodes, because on this path they are. Three loaded nodes delivered 2.71x one node. The missing 8 percent per node is consistent with the shared origin and single load generator approaching their own limits, so read 119k as this fleet's floor rather than its ceiling.
• Approximate governance converges at the dissemination cadence. Spend recorded on one node appeared in a peer's admission view in 15 to 20 s (the counter dissemination loop ticks every 15 s). Spend landing in the final seconds of a fixed window can roll over before peers merge it, which is the bounded-overshoot tradeoff key-management.md documents for the approximate tier; use the strict tier when a limit must be exact under concurrent traffic.
• A dying peer is a local event. Killing one node mid-run left the surviving nodes' success rate at 100 percent with throughput intact. Clients of the dead node saw immediate connection-refused (fail-fast, no hangs), which is what a fronting load balancer's health check needs to eject it. After restart the node rejoined the mesh and served traffic again in about 10 s.

Reproduce it from the sbproxy-bench clustered topology, or by hand: three nodes with a proxy.cluster block (shared_key security, symmetric seed lists), the A01 scenario config pointed at a shared mock origin, and oha streams per node. Watch mesh_peer_count, mesh_owner_route_total, and GET /admin/cluster/status to confirm the mesh formed before trusting a number.

Hardware and methodology

Setting	Value
Instance type (proxy + origin)	c3-standard-8 (8 vCPU Sapphire Rapids, dedicated)
Instance type (loadgen)	c3-standard-22
Region / zone	us-central1-a
Build profile	release with lto = "fat", codegen-units = 1, strip = "symbols"
Allocator	mimalloc
Run duration	60 seconds, 3 replicates per scenario, median reported
Logging	Compile-stripped debug/trace via tracing release_max_level_info
Origin	Echo server returning a small JSON body

The full set of scenarios, the harness code, the loadgen config, and the raw per-replicate output live in the sbproxy-bench repo.

Reproduce locally

You don't need GCE to get a useful read. The microbenchmarks and the local recipe below run on a laptop.

There is no microbenchmark suite

Earlier versions of this page documented a criterion suite and a target/criterion/ report directory. Neither exists. No crate in this workspace declares a [[bench]] target, there is no benches/ directory, and criterion does not appear in Cargo.lock, so cargo bench --workspace compiles the workspace and benches nothing. If you came here looking for in-process numbers on the config compiler or pipeline dispatch, there are none to find; the two scripts below are the whole local story.

The scripted harness

Two scripts in scripts/ drive load against a real proxy process and report both latency and resident set:

# Boots a release binary, samples idle and peak RSS, runs oha, writes JSON.
scripts/perf-regression-run.sh /tmp/bench.json my-label

# Three scenario shapes (static, full middleware, echo) at concurrency 64.
SBPROXY_BIN=./target/release/sbproxy scripts/perf-compare.sh

The first is what the perf-regression CI lane runs on a PR labeled run-perf; it compares the PR against a freshly built main baseline and fails the check when p99, idle RSS, or max RSS regresses more than 5%. Both scripts need oha on PATH, pinned to 1.4.5 in CI. capacity-planning.md covers the tuning variables and how to read what comes back.

End-to-end local run

make build-release
./target/release/sbproxy --config examples/basic-proxy/sb.yml &

# In another terminal, drive load against the local proxy.
# oha is a simple choice; wrk and hey work too.
oha -n 10000 -c 100 http://127.0.0.1:8080/get

Localhost runs hit ephemeral-port exhaustion around 150 concurrent connections. They're useful for relative comparisons (before vs after a code change) and unreliable for absolute production numbers.

Cloud benchmark

The full c3 benchmark used for the headline numbers is in the sbproxy-bench repo, including the Terraform that provisions the GCE instances and the harness that runs each scenario through three replicates.

Profiling a hot path

When you need to know why a scenario is slower than expected:

# Linux: perf + flamegraph
cargo flamegraph --bin sbproxy --release -- --config sb.yml

# macOS: samply (no sudo)
samply record ./target/release/sbproxy --config sb.yml

# Heap profiling
heaptrack ./target/release/sbproxy --config sb.yml

For per-request CPU breakdown, enable OpenTelemetry tracing in the config (telemetry block) and view spans in your collector of choice. The phase pipeline emits a span per phase, so you can pinpoint which middleware is dominating.

Why the numbers look like this

A few design choices do most of the work:

• Pingora foundation. The same proxy framework Cloudflare runs at scale. Tokio runtime, careful epoll integration, no garbage collector to pause it.
• mimalloc allocator. Roughly 5 to 10% faster than glibc malloc on server workloads.
• Compile-stripped logging. tracing is configured with release_max_level_info, so debug and trace calls evaporate at compile time. No runtime filter cost on the hot path.
• LTO + codegen-units = 1. Across-crate inlining and smaller binaries. Costs build time, gives a 5 to 15% rps lift at the tail.
• ArcSwap for hot reload. New configs swap in atomically. Old requests finish on their snapshot, new ones pick up the new config. No locks on the request path.
• bumpalo per-request arenas, compact_str for short strings, smallvec for small collections. Fewer heap allocations per request.
• Bloom filter + radix tree host routing. O(1) negative lookup before any per-origin work.

See architecture.md for the full pipeline and comparison.md for how the numbers stack against other proxies.

What to watch in production

For your own dashboards, the metrics that move first:

• sbproxy_request_duration_seconds (p50, p95, p99). The single most useful gauge.
• sbproxy_phase_duration_seconds{phase="upstream_ttfb"}. Time from the request's first byte to the first upstream response byte. Compare against the total request duration to see what the proxy itself adds.
• sbproxy_active_connections. Sustained climb means your upstream is slower than incoming.
• sbproxy_cache_results_total. Compute the hit ratio as hit / (hit + miss) from the result label. It is the number that moves p99 the most when caching is configured.
• sbproxy_config_reload_total. A spike means your reload tooling is flapping.

Memory is not on that list because the proxy does not export its own resident set. Read it from the layer below (container_memory_working_set_bytes under Kubernetes, ps -o rss= on a plain host); capacity-planning.md covers what to alert on.

See metrics-stability.md for the full catalog and stability tier of every metric.