feat(proof-gen): request admission limits, per-chain fill budget, single-flight block fills

[DylanVerstraete]

Fifth and last PR of the prover liveness audit, finding 6. Stacked on #1353 → #1352 → #1351 → #1350; retarget to usc-dev as those merge.

Problem

max_batch_size bounded the work inside one batch, not how many requests ran at once, and nothing ever timed a request out. The audit fixture sent 32 identical cold proof-by-tx requests and observed 32 concurrent tip reads and 32 duplicate block fetch+builds.

Changes

• Admission middleware around /api/v1/proof*:
  • process-wide cap on requests in flight and a per-chain cap so one chain's storm cannot starve the others; refusals are immediate 503 + Retry-After: 1 with code: Overloaded | ChainOverloaded, retriable: true
  • end-to-end deadline per request → 504, code: RequestTimeout
  • /livez, /readyz, /api/v1/health, /metrics are never limited, so probes and recovery keep working under load
  • config: admission.{max_in_flight_requests (64), max_in_flight_per_chain (32), request_timeout_secs (120)} in YAML; MAX_IN_FLIGHT_REQUESTS, MAX_IN_FLIGHT_PER_CHAIN, REQUEST_TIMEOUT_SECS env overrides (both config paths)
• Per-chain block-fill budget: cache.max_concurrent_block_fills (default 16) bounds simultaneous source-block fetches for the merkle cache across all requests and the backfill worker together.
• Single-flight fills: concurrent misses for one height share one fetch+build. Leader/followers on a Notify; a failed leader wakes the followers, which retry as leader on their own, so one bad fetch is not broadcast as everyone's error. Cached blocks and processed-empty blocks are served without a fetch (previously every cold proof-by-tx on an empty block refetched it).
• Metrics: proof_gen_requests_in_flight gauge, proof_gen_requests_rejected{reason} counter.
• config.example.yaml documents both knobs.

Tests

proof-gen-api-server/tests/liveness_admission.rs:

• 32 simultaneous identical cold requests, budget 8 → exactly 8 admitted (200), 24 refused (Overloaded), peak concurrent tip reads ≤ 8, one block fetch; 24 sequential retries all 200 with no new fetch
• 2 s upstream under a 200 ms deadline → 504 RequestTimeout in < 1 s
• with the single admission slot held by a slow proof request: /api/v1/health, /livez, /readyz, /metrics all answer immediately

cargo test -p proof-gen-api-server green (81 unit, 27 route, 3+3+4+2 liveness), clippy -D warnings, fmt.

[cursor]

PR Summary

Medium Risk
Changes proof-serving behavior under load (503/504 responses and stricter RPC concurrency) and merkle fill coordination; mis-tuned limits could increase client retries or slow cold proofs, but health paths are exempt and defaults are conservative.

Overview
Adds HTTP admission control and merkle cache fill throttling so proof traffic cannot fan out into unbounded concurrent RPC and long-running requests.

Admission wraps /api/v1/proof* with process-wide and per-chain in-flight limits (defaults 64 / 32). Over-limit callers get 503 with Retry-After: 1 and retriable JSON (Overloaded / ChainOverloaded). Admitted work is bounded by an end-to-end timeout (default 120s) → 504 RequestTimeout. Health, readiness, and /metrics stay outside admission. Limits are configurable via YAML admission: and env MAX_IN_FLIGHT_REQUESTS, MAX_IN_FLIGHT_PER_CHAIN, REQUEST_TIMEOUT_SECS (CLI path applies env after building config).

Per-chain cache fills: new cache.max_concurrent_block_fills (default 16) caps simultaneous source-block fetches (requests + backfill). Single-flight per block height dedupes concurrent cold misses via leader/follower Notify; FillGuard cleans up on cancel so admission timeouts do not strand waiters. On-demand merkle paths use shared fill instead of duplicate fetches.

Observability: proof_gen_requests_in_flight and proof_gen_requests_rejected{reason}. Integration tests in liveness_admission.rs cover burst caps, single fetch under concurrency, 504, probe exemption, and leader cancellation recovery.

^{Reviewed by Cursor Bugbot for commit 91725d0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bb72d48. Configure here.}