Skip to content

fix: throttle upstream RPC-provider error reporting to stop log amplification#1453

Open
tony8713 wants to merge 1 commit into
masterfrom
fix/rpc-error-log-amplification
Open

fix: throttle upstream RPC-provider error reporting to stop log amplification#1453
tony8713 wants to merge 1 commit into
masterfrom
fix/rpc-error-log-amplification

Conversation

@tony8713

Copy link
Copy Markdown
Contributor

Problem

score.snapshot.org was flapping with Better Stack "Timeout (no headers received)" every few minutes (incident 2026-06-22, ~15:37 UTC onward).

Root cause chain:

  • The shared RPC proxy rpc.snapshot.org (served by brovider on DigitalOcean App Platform) intermittently returns HTTP 403 for strategy RPC calls — overwhelmingly /1 (mainnet) and /137 (Polygon). The 403 carries x-do-orig-status: 403, so the DO origin (brovider passing through an upstream provider's 403), not Cloudflare, emits it.
  • ethers then throws SERVER_ERROR / CALL_EXCEPTION for every failing strategy call.
  • score-api called capture() (Sentry) and console.log once per failed request. Under an outage that is thousands of failures/minute, each serializing the full ethers error object (request body + upstream HTML). Observed log bursts of 20k–54k lines/min. That log/Sentry amplification stalls the event loop, so score-api stops answering within the HTTP header timeout → Better Stack opens an incident; it recovers between bursts → flapping.

The 403 itself is an infra issue (tracked separately). This PR removes the self-inflicted amplification that turns a transient upstream blip into a score-api outage — the isolated, safe resilience fix.

Fix

New src/helpers/providerErrors.ts:

  • isProviderError(e) — recognises upstream RPC-provider failures (ethers SERVER_ERROR/TIMEOUT/NETWORK_ERROR/CALL_EXCEPTION, HTTP 403/429/5xx, or an rpc.snapshot.org URL signature) vs. genuine input/validation errors.
  • summarizeError(e) — bounded (≤256 char) single-line summary; never serializes the full ethers object.
  • shouldReport(key) — time-windowed (30s) throttle keyed by a coarse fingerprint, so a 403 storm reports ~once/window per (method/network + error code) instead of thousands of times.

src/rpc.ts — both error paths (handlePostError and the /api/scores catch) now: if the error is an upstream-provider error, only capture() + log a sampled trickle; otherwise behave exactly as before. The HTTP response (500) is unchanged in all cases.

How to test

  • yarn test:unit — new src/helpers/providerErrors.test.ts (10 cases) plus existing rpc.test.ts / methods.test.ts pass unchanged.
  • Manual: simulate a provider 403 storm (point BROVIDER_URL at an endpoint returning 403 and fire repeated /api/scores); confirm Sentry captures + [rpc] ... (upstream provider, throttled) log lines are emitted at most ~once per 30s per key instead of per request, while clients still receive a 500.

Notes / non-goals

  • No version bump. Behaviour for non-provider errors is byte-for-byte unchanged.
  • Per-strategy provider timeout tightening (currently the snapshot.js 25s default via getProvider(network) in src/strategies/utils.ts) was deliberately not included — it risks altering scoring for slow-but-valid strategies and warrants its own change. Recommended follow-up.

Infra (team action, outside this PR)

The 403 root cause lives in brovider / its upstream providers, not here. See #uptime for the exact steps.

🤖 Generated with Claude Code

…fication

When rpc.snapshot.org (the shared RPC proxy) returns HTTP 403/5xx, ethers
throws a SERVER_ERROR/CALL_EXCEPTION for every failing strategy call.
score-api was calling capture() (Sentry) and console.log once per failed
request. During a provider outage that is thousands of failures per minute,
each serializing the full ethers error object, which stalls the event loop
and makes score-api miss the HTTP header timeout (the score.snapshot.org
flapping).

Add helpers/providerErrors with isProviderError(), summarizeError(), and a
time-windowed shouldReport() throttle, and apply it in both error paths in
rpc.ts so a provider outage produces a sampled trickle of reports instead of
a self-inflicted flood. Behaviour for non-provider errors is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@tony8713

Copy link
Copy Markdown
Contributor Author

CI note: the lint job passes. The test job's single failure is test/strategies/integration/utils.test.ts › getVp() › should calculate VP for EVM address, which makes a real RPC call to https://rpc.snapshot.org/1 and received the exact 403 this PR is about:

bad response (status=403, ... server:"cloudflare", x-do-orig-status:"403", x-do-app-origin:"dd20df60-...", url="https://rpc.snapshot.org/1", code=SERVER_ERROR)

That is the live production incident reproducing inside CI (origin 403, not Cloudflare), not a regression from this change. All unit tests — including the new providerErrors suite — and every non-network test pass. The integration test will go green once the upstream provider key behind brovider for chains 1/137 is rotated (infra step tracked in #uptime). Happy to re-run CI after that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant