perf: black-box load/verification client for deployed aggregator/gateway#166
perf: black-box load/verification client for deployed aggregator/gateway#166MastaP wants to merge 1 commit into
Conversation
A purely client-side load test for a deployed aggregator or subscription
gateway. Unlike cmd/performance-test it scrapes no /metrics and reads no
server logs — everything reported is observed from the client.
It speaks the same wire protocol as the state-transition SDK:
- certification_request params = hex(CBOR(CertificationRequest))
- get_inclusion_proof.v2 params = {"stateId":"<32-byte hex>"}
- X-API-Key carries the shared API key
- X-State-ID carries the state id hex (the gateway routes on it)
Commitments are uniform (fresh key + random state per request, no shard
rejection sampling) so the gateway spreads them evenly across shards.
After a configurable delay it fetches each inclusion proof back and
verifies it locally against the SMT root, confirming every accepted
commitment actually landed in the tree. Reports submit/accept rates,
submit and submit->included latencies, per-shard-prefix distribution,
and an error breakdown.
Flags cover rate, duration/count, proof delay/retry/timeout, crypto
verification, connection/concurrency pools, TLS skip, and a -proofs
toggle to measure accept rate only (for rate ramps).
testnet2 findingsRuns against the 8-shard subscription gateway ( Correctness ✅
Throughput ceiling
|
| Offered | Submitted | Accepted | Accept % | Shed (503) |
|---|---|---|---|---|
| 300/s | 3000 | 3000 | 100% | 0 |
| 350/s | 4200 | 4200 | 100% | 0 |
| 400/s | 4800 | 4480 | 93.3% | 320 |
| 1000/s | 10000 | 3533 | 35.3% | 6467 |
- Knee is between 350 and 400 req/s; sustained accepted throughput ≈ 350–375/s.
- Everything accepted is aggregated correctly (100% SMT inclusion at every rate) — the shards are not the bottleneck.
- This is ~27× below the shared key's 10,000 req/s limit, so the API-key rate limiter is not the cause.
Interpretation
Each ramp step above is a fresh ~12s burst. A horizontal autoscaler reacts to sustained load over minutes, so a 12s burst likely hits a single gateway instance and never triggers scale-out. ~375/s is therefore plausibly single-instance capacity (a per-instance concurrency / in-flight cap emitting 503 Service Overloaded), with autoscaling never engaging.
Next step
A sustained run (e.g. -rate 1000 -duration 180s -proofs=false) while watching the gateway instance count + HAProxy logs will distinguish:
- autoscaling lag — accepted/s climbs from ~350 toward ~1400 as instances spin up, vs.
- a fixed per-instance or global cap — accepted/s stays pinned at ~350/s.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces perf-blackbox, a pure black-box load and verification client for a deployed aggregator that generates commitments, submits them via JSON-RPC, and verifies inclusion proofs. The review feedback highlights several critical issues: a potential early context timeout when using the -count flag due to unadjusted duration, connection reuse and error reporting issues in HTTP response handling, a blocking semaphore acquisition that ignores context cancellation, and an artificial 32-core cap in numCPU that limits performance on high-core servers.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if c.count <= 0 { | ||
| c.count = int(float64(c.rate) * c.duration.Seconds()) | ||
| } | ||
| if c.count <= 0 { | ||
| c.count = c.rate | ||
| } |
There was a problem hiding this comment.
If the -count flag is explicitly provided, cfg.duration is ignored for determining the total count, but it is still used to calculate the runCtx timeout in main(). If the combination of count and rate requires more time than the default 10s duration (e.g., -count 10000 -rate 100 requires 100 seconds), the context will timeout early, cutting the load test short.
Updating c.duration dynamically in parseConfig when c.count is provided fixes this issue.
if c.count <= 0 {
c.count = int(float64(c.rate) * c.duration.Seconds())
} else if c.rate > 0 {
c.duration = time.Duration(float64(c.count)/float64(c.rate)) * time.Second
}
if c.count <= 0 {
c.count = c.rate
}| resp, err := c.client().Do(req) | ||
| if err != nil { | ||
| return nil, nil, err | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| ct := strings.ToLower(resp.Header.Get("Content-Type")) | ||
| if !strings.Contains(ct, "application/json") { | ||
| snippet := readSnippet(resp.Body, 256) | ||
| return nil, nil, fmt.Errorf("non-JSON response: HTTP %d ct=%q body=%s", resp.StatusCode, ct, snippet) | ||
| } | ||
|
|
||
| var rpcResp rpcResponse | ||
| if err := json.NewDecoder(resp.Body).Decode(&rpcResp); err != nil { | ||
| return nil, nil, fmt.Errorf("decode response (HTTP %d): %w", resp.StatusCode, err) | ||
| } | ||
| return rpcResp.Result, rpcResp.Error, nil | ||
| } |
There was a problem hiding this comment.
There are two major issues with the current HTTP response handling:
- Connection Reuse: Decoding directly from
resp.Bodyusingjson.NewDecoderwithout fully draining the body to EOF (especially when errors occur or whenreadSnippetonly reads up to 256 bytes) prevents HTTP connection reuse. This leads to connection churn, high latency, and ephemeral port exhaustion under high load. - Error Reporting: HTTP status codes other than
200 OK(e.g., 401, 403, 429, 502) are not explicitly checked. If the gateway returns a non-200 status code with a non-JSON-RPC body, the tool fails to report the HTTP error clearly, resulting in confusing "submit status: " or JSON decoding errors.
Reading the entire response body using io.ReadAll first guarantees the connection is fully drained and can be reused, while also allowing robust handling of non-200 status codes.
Note: Remember to add "io" to the imports list.
resp, err := c.client().Do(req)
if err != nil {
return nil, nil, err
}
defer resp.Body.Close()
bodyBytes, err := io.ReadAll(resp.Body)
if err != nil {
return nil, nil, fmt.Errorf("read response body: %w", err)
}
if resp.StatusCode != http.StatusOK {
var rpcResp rpcResponse
if json.Unmarshal(bodyBytes, &rpcResp) == nil && rpcResp.Error != nil {
return nil, rpcResp.Error, nil
}
snippet := strings.TrimSpace(string(bodyBytes))
if len(snippet) > 256 {
snippet = snippet[:256] + "..."
}
return nil, nil, fmt.Errorf("HTTP %d: %s", resp.StatusCode, snippet)
}
ct := strings.ToLower(resp.Header.Get("Content-Type"))
if !strings.Contains(ct, "application/json") {
snippet := strings.TrimSpace(string(bodyBytes))
if len(snippet) > 256 {
snippet = snippet[:256] + "..."
}
return nil, nil, fmt.Errorf("non-JSON response: HTTP %d ct=%q body=%s", resp.StatusCode, ct, snippet)
}
var rpcResp rpcResponse
if err := json.Unmarshal(bodyBytes, &rpcResp); err != nil {
return nil, nil, fmt.Errorf("decode response (HTTP %d): %w", resp.StatusCode, err)
}
return rpcResp.Result, rpcResp.Error, nil| return | ||
| } | ||
|
|
||
| proofSem <- struct{}{} |
There was a problem hiding this comment.
When acquiring a slot in proofSem, the code blocks directly on proofSem <- struct{}{}. If the context is cancelled, the goroutine will remain blocked until it can acquire the semaphore, delaying shutdown and wasting resources.
Using a select statement to acquire the semaphore or abort if ctx.Done() is closed ensures a clean and responsive shutdown.
| proofSem <- struct{}{} | |
| select { | |
| case proofSem <- struct{}{}: | |
| case <-ctx.Done(): | |
| return | |
| } |
| func numCPU() int { | ||
| n := runtime.NumCPU() | ||
| if n < 1 { | ||
| return 1 | ||
| } | ||
| if n > 32 { | ||
| return 32 | ||
| } | ||
| return n | ||
| } |
There was a problem hiding this comment.
The numCPU function artificially caps the number of CPU cores at 32. On modern high-core servers (e.g., 64 or 128 cores), this limits the number of commitment generation workers, creating an artificial bottleneck for high-rate load tests.
Removing the arbitrary cap of 32 allows the tool to scale fully on larger machines.
func numCPU() int {
n := runtime.NumCPU()
if n < 1 {
return 1
}
return n
}1 participant
What
cmd/perf-blackbox— a purely client-side load and inclusion-verification tool for a deployed aggregator or subscription gateway. Unlikecmd/performance-test, it scrapes no/metricsand reads no server logs; every number it reports is observed from the client.Wire protocol
Speaks exactly what the state-transition SDK speaks:
certification_request— params =hex(CBOR(CertificationRequest))(tag 39030, v1)get_inclusion_proof.v2— params ={"stateId":"<32-byte hex>"}X-API-Keyheader carries the shared API keyX-State-IDheader carries the state-id hex (the gateway routes on it)Behaviour
internal/proofverify.VerifyInclusionProofLocal) — confirming every accepted commitment actually landed in the tree.Flags
-url -api-key -rate -duration -count -proof-delay -proof-retry -proof-timeout -verify -proofs -submit-concurrency -proof-concurrency -conns -http-timeout -insecure -gen-workers-proofs=falsemeasures accept rate only (used for rate ramps).Build / run
Repo needs Go 1.25+.
The API key is only ever read from
-api-key/UNICITY_API_KEY; nothing is hardcoded.Findings from testnet2 runs are in a comment below.