Skip to content

Balance metric conflates RPC fetch failure with zero balance #660

Description

@niks3089

During a Helius mainnet incident (2026-09-02, ~09:37Z), getBalance/getAccountInfo returned HTTP 500 while getHealth passed. Kora's balance collector (kora_lib::metrics::balance) logged Failed to get balance for signer …: Account FCy1… not found and the signer_balance_lamports gauge dropped to 0 — while the account held 1.85 SOL on-chain.

Two problems:

  1. An RPC transport/server error is reported as "Account not found" — misleading during triage (the account existed; the provider was down).
  2. The gauge going to 0 on fetch failure is indistinguishable from a genuinely empty wallet. Downstream, our balance-low and burn-rate alerts both paged as a critical "fee-payer drain" with top-up guidance during what was a vendor outage.

Proposed:

  • On fetch failure, hold the last-known-good gauge value (or mark the series stale) instead of writing 0.
  • Export a companion counter, e.g. signer_balance_fetch_errors_total{signer_name}, so operators can alert on "balance unknown" as its own condition.
  • Log the actual error class (HTTP status / transport error) rather than mapping everything to "Account not found".

Observed on the image deployed as kora-mainnet-int (SDP prod), config current as of 2026-08-22.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions