Skip to content

feat(prom): watch every signing wallet's balance on the chain it spends on - #54

Merged
DylanVerstraete merged 3 commits into
mainfrom
feat/signer-balance-metric
Sep 1, 2026
Merged

feat(prom): watch every signing wallet's balance on the chain it spends on#54
DylanVerstraete merged 3 commits into
mainfrom
feat/signer-balance-metric

Conversation

@DylanVerstraete

Copy link
Copy Markdown
Collaborator

Third of the observability trio (with #53): the most standard relayer metric there is, and we had no view of it.

Problem

Every tx-sending role runs off a plain EOA that only needs gas — delivery on the destination chain, ack and claim submission on the Creditcoin chain. When a wallet runs dry, sends fail with whatever wording that node gives an underfunded sender, which at a glance reads like an RPC problem. Nothing pointed at the wallet. Hyperlane (hyperlane_wallet_balance) and OpenZeppelin Defender both treat this as a headline metric.

Change

New balance worker publishing relayer_signer_balance_ether{chain_key, role, address}, polled every 60s.

  • Watch list derived from config, each role on the chain it actually spends gas on: delivery → the route's destination RPC; ack/claim → the Creditcoin RPC. Watching the right address on the wrong chain would report a healthy balance for a wallet that is empty where it matters — a test pins this mapping.
  • address is a label, so one wallet reused across roles/chains stays correlatable in queries.
  • Providers are built lazily and dropped on a failed read (re-dial next tick, never retry a dead socket forever).
  • An unparseable signer_key is skipped with a warning, not an error — the worker that uses the key raises the real failure with context, and the relayer must not refuse to start over its least important worker.
  • Health is heartbeated per completed sweep even if every read failed: an unreachable RPC already starves the indexing workers' success-only heartbeats and trips the liveness restart; this auxiliary worker should not be a second finger on that trigger.

Alert this enables

relayer_signer_balance_ether < 0.1

warning while there is still time to top up, instead of when deliveries are already failing. Units are ether (Hyperlane convention) so thresholds are human; the encode test pins metric name, labels and units so a rename or wei/ether mixup fails in CI rather than in a silently never-firing alert.

Verification

  • cargo test --workspace — 169 pass, 0 fail (3 new: role→chain mapping, bad-key skip, encode shape).
  • clippy -D warnings clean, fmt clean.

…ds on

Every tx-sending role runs off a plain EOA that only needs gas — delivery on
the destination chain, ack and claim submission on the Creditcoin chain. When
one runs dry, sends fail with whatever wording that node gives an underfunded
sender, which reads like an RPC problem; nothing anywhere pointed at the
wallet. Industry relayers (Hyperlane's hyperlane_wallet_balance, OpenZeppelin
Defender's balance monitors) treat signer balance as a headline metric for
exactly this reason.

Add a balance watcher worker publishing

    relayer_signer_balance_ether{chain_key, role, address}

polled every 60s. The watch list is derived from config, each role on the
chain it actually spends gas on: delivery on the route's destination RPC, ack
and claim on the Creditcoin RPC — watching the right address on the wrong
chain would report a healthy balance for a wallet that is empty where it
matters (a test pins this). Address is a label so one wallet reused across
roles stays correlatable.

Resilience choices, both deliberate:
 - a signer_key that fails to parse is skipped with a warning, not an error —
   the worker that uses the key reports the real failure with context, and the
   relayer must not refuse to start over its least important worker;
 - health is heartbeated once per completed sweep even if every read failed.
   An unreachable RPC already starves the indexing workers' success-only
   heartbeats and trips the liveness restart; this auxiliary worker adding a
   second finger to that trigger would only worsen the restart storm.

Units are ether (Hyperlane convention) so alert thresholds are human
(< 0.1); the encode test pins name, labels and units so a rename or wei/ether
mixup fails in CI rather than in a silently never-firing alert.
@cursor

cursor Bot commented Sep 1, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Observability-only background worker with defensive health and startup behavior; no changes to delivery, ack, or claim transaction paths.

Overview
Adds a signer balance watcher so operators can alert before gas wallets run dry (instead of misreading underfunded-send failures as RPC issues).

A new balance worker runs alongside existing relayer tasks: it builds a watch list from route config (delivery on each route’s destination RPC; ack / claim on the Creditcoin RPC), polls balances every 60s, and publishes relayer_signer_balance_ether with labels chain_key, role, and address (ether units for simple thresholds like < 0.1). Unparseable keys are skipped with a warning so startup isn’t blocked; with no signers configured the worker stays idle and does not register with /health (avoids liveness restart loops). RPC providers are recreated after failed reads; health heartbeats once per sweep even when reads fail, so this auxiliary path doesn’t amplify restart storms from unreachable RPCs.

Prometheus wiring extends MetricsTrait / RelayerMetrics with set_signer_balance; tests pin role→chain mapping, idle health behavior, bad-key handling, and the encoded metric shape.

Reviewed by Cursor Bugbot for commit f59c6f1. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread message-relayer/src/balance.rs
…atch

The worker heartbeated before the empty-targets early return, then parked on
cancel without ever beating again. Health kept tracking the registered-but-
silent component, so past PROGRESS_DEADLINE the liveness probe would restart-
loop a relayer that simply has no signing keys configured. Register only when
there is actual work; a regression test runs the idle path against a zero
deadline, where anything registered would already read as stale.

Found by Bugbot on #54.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit f7fa2f5. Configure here.

Comment thread message-relayer/src/balance.rs
…e regression

Health::status compares age with a strict greater-than, so under the test's
zero deadline a registration made in the same millisecond still read as alive.
With the pre-cancelled token the idle path completes well inside a millisecond,
meaning a reintroduced register-then-park bug would usually slip past the
assertion. Sleep 20ms before asserting: after any positive age a registered
component is stale, so only a never-registered watcher keeps /health green.
Verified by temporarily reintroducing the bug — the test now fails against it.

Found by Bugbot on #54.
@DylanVerstraete
DylanVerstraete merged commit 1549d49 into main Sep 1, 2026
10 checks passed
@DylanVerstraete
DylanVerstraete deleted the feat/signer-balance-metric branch September 1, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants