During a Helius mainnet incident (2026-09-02, ~09:37Z), getBalance/getAccountInfo returned HTTP 500 while getHealth passed. Kora's balance collector (kora_lib::metrics::balance) logged Failed to get balance for signer …: Account FCy1… not found and the signer_balance_lamports gauge dropped to 0 — while the account held 1.85 SOL on-chain.
Two problems:
- An RPC transport/server error is reported as "Account not found" — misleading during triage (the account existed; the provider was down).
- The gauge going to 0 on fetch failure is indistinguishable from a genuinely empty wallet. Downstream, our balance-low and burn-rate alerts both paged as a critical "fee-payer drain" with top-up guidance during what was a vendor outage.
Proposed:
- On fetch failure, hold the last-known-good gauge value (or mark the series stale) instead of writing 0.
- Export a companion counter, e.g.
signer_balance_fetch_errors_total{signer_name}, so operators can alert on "balance unknown" as its own condition.
- Log the actual error class (HTTP status / transport error) rather than mapping everything to "Account not found".
Observed on the image deployed as kora-mainnet-int (SDP prod), config current as of 2026-08-22.
During a Helius mainnet incident (2026-09-02, ~09:37Z),
getBalance/getAccountInforeturned HTTP 500 whilegetHealthpassed. Kora's balance collector (kora_lib::metrics::balance) loggedFailed to get balance for signer …: Account FCy1… not foundand thesigner_balance_lamportsgauge dropped to 0 — while the account held 1.85 SOL on-chain.Two problems:
Proposed:
signer_balance_fetch_errors_total{signer_name}, so operators can alert on "balance unknown" as its own condition.Observed on the image deployed as kora-mainnet-int (SDP prod), config current as of 2026-08-22.