fix: throttle upstream RPC-provider error reporting to stop log amplification#1453
Open
tony8713 wants to merge 1 commit into
Open
fix: throttle upstream RPC-provider error reporting to stop log amplification#1453tony8713 wants to merge 1 commit into
tony8713 wants to merge 1 commit into
Conversation
…fication When rpc.snapshot.org (the shared RPC proxy) returns HTTP 403/5xx, ethers throws a SERVER_ERROR/CALL_EXCEPTION for every failing strategy call. score-api was calling capture() (Sentry) and console.log once per failed request. During a provider outage that is thousands of failures per minute, each serializing the full ethers error object, which stalls the event loop and makes score-api miss the HTTP header timeout (the score.snapshot.org flapping). Add helpers/providerErrors with isProviderError(), summarizeError(), and a time-windowed shouldReport() throttle, and apply it in both error paths in rpc.ts so a provider outage produces a sampled trickle of reports instead of a self-inflicted flood. Behaviour for non-provider errors is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
|
CI note: the lint job passes. The test job's single failure is That is the live production incident reproducing inside CI (origin 403, not Cloudflare), not a regression from this change. All unit tests — including the new |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
score.snapshot.orgwas flapping with Better Stack "Timeout (no headers received)" every few minutes (incident 2026-06-22, ~15:37 UTC onward).Root cause chain:
rpc.snapshot.org(served by brovider on DigitalOcean App Platform) intermittently returns HTTP 403 for strategy RPC calls — overwhelmingly/1(mainnet) and/137(Polygon). The 403 carriesx-do-orig-status: 403, so the DO origin (brovider passing through an upstream provider's 403), not Cloudflare, emits it.SERVER_ERROR/CALL_EXCEPTIONfor every failing strategy call.capture()(Sentry) andconsole.logonce per failed request. Under an outage that is thousands of failures/minute, each serializing the full ethers error object (request body + upstream HTML). Observed log bursts of 20k–54k lines/min. That log/Sentry amplification stalls the event loop, so score-api stops answering within the HTTP header timeout → Better Stack opens an incident; it recovers between bursts → flapping.The 403 itself is an infra issue (tracked separately). This PR removes the self-inflicted amplification that turns a transient upstream blip into a score-api outage — the isolated, safe resilience fix.
Fix
New
src/helpers/providerErrors.ts:isProviderError(e)— recognises upstream RPC-provider failures (ethersSERVER_ERROR/TIMEOUT/NETWORK_ERROR/CALL_EXCEPTION, HTTP 403/429/5xx, or anrpc.snapshot.orgURL signature) vs. genuine input/validation errors.summarizeError(e)— bounded (≤256 char) single-line summary; never serializes the full ethers object.shouldReport(key)— time-windowed (30s) throttle keyed by a coarse fingerprint, so a 403 storm reports ~once/window per (method/network + error code) instead of thousands of times.src/rpc.ts— both error paths (handlePostErrorand the/api/scorescatch) now: if the error is an upstream-provider error, onlycapture()+ log a sampled trickle; otherwise behave exactly as before. The HTTP response (500) is unchanged in all cases.How to test
yarn test:unit— newsrc/helpers/providerErrors.test.ts(10 cases) plus existingrpc.test.ts/methods.test.tspass unchanged.BROVIDER_URLat an endpoint returning 403 and fire repeated/api/scores); confirm Sentry captures +[rpc] ... (upstream provider, throttled)log lines are emitted at most ~once per 30s per key instead of per request, while clients still receive a 500.Notes / non-goals
getProvider(network)insrc/strategies/utils.ts) was deliberately not included — it risks altering scoring for slow-but-valid strategies and warrants its own change. Recommended follow-up.Infra (team action, outside this PR)
The 403 root cause lives in brovider / its upstream providers, not here. See #uptime for the exact steps.
🤖 Generated with Claude Code