feat(prom): watch every signing wallet's balance on the chain it spends on - #54
Conversation
…ds on
Every tx-sending role runs off a plain EOA that only needs gas — delivery on
the destination chain, ack and claim submission on the Creditcoin chain. When
one runs dry, sends fail with whatever wording that node gives an underfunded
sender, which reads like an RPC problem; nothing anywhere pointed at the
wallet. Industry relayers (Hyperlane's hyperlane_wallet_balance, OpenZeppelin
Defender's balance monitors) treat signer balance as a headline metric for
exactly this reason.
Add a balance watcher worker publishing
relayer_signer_balance_ether{chain_key, role, address}
polled every 60s. The watch list is derived from config, each role on the
chain it actually spends gas on: delivery on the route's destination RPC, ack
and claim on the Creditcoin RPC — watching the right address on the wrong
chain would report a healthy balance for a wallet that is empty where it
matters (a test pins this). Address is a label so one wallet reused across
roles stays correlatable.
Resilience choices, both deliberate:
- a signer_key that fails to parse is skipped with a warning, not an error —
the worker that uses the key reports the real failure with context, and the
relayer must not refuse to start over its least important worker;
- health is heartbeated once per completed sweep even if every read failed.
An unreachable RPC already starves the indexing workers' success-only
heartbeats and trips the liveness restart; this auxiliary worker adding a
second finger to that trigger would only worsen the restart storm.
Units are ether (Hyperlane convention) so alert thresholds are human
(< 0.1); the encode test pins name, labels and units so a rename or wei/ether
mixup fails in CI rather than in a silently never-firing alert.
PR SummaryLow Risk Overview A new Prometheus wiring extends Reviewed by Cursor Bugbot for commit f59c6f1. Bugbot is set up for automated code reviews on this repo. Configure here. |
…atch The worker heartbeated before the empty-targets early return, then parked on cancel without ever beating again. Health kept tracking the registered-but- silent component, so past PROGRESS_DEADLINE the liveness probe would restart- loop a relayer that simply has no signing keys configured. Register only when there is actual work; a regression test runs the idle path against a zero deadline, where anything registered would already read as stale. Found by Bugbot on #54.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f7fa2f5. Configure here.
…e regression Health::status compares age with a strict greater-than, so under the test's zero deadline a registration made in the same millisecond still read as alive. With the pre-cancelled token the idle path completes well inside a millisecond, meaning a reintroduced register-then-park bug would usually slip past the assertion. Sleep 20ms before asserting: after any positive age a registered component is stale, so only a never-registered watcher keeps /health green. Verified by temporarily reintroducing the bug — the test now fails against it. Found by Bugbot on #54.

Third of the observability trio (with #53): the most standard relayer metric there is, and we had no view of it.
Problem
Every tx-sending role runs off a plain EOA that only needs gas — delivery on the destination chain, ack and claim submission on the Creditcoin chain. When a wallet runs dry, sends fail with whatever wording that node gives an underfunded sender, which at a glance reads like an RPC problem. Nothing pointed at the wallet. Hyperlane (
hyperlane_wallet_balance) and OpenZeppelin Defender both treat this as a headline metric.Change
New
balanceworker publishingrelayer_signer_balance_ether{chain_key, role, address}, polled every 60s.addressis a label, so one wallet reused across roles/chains stays correlatable in queries.signer_keyis skipped with a warning, not an error — the worker that uses the key raises the real failure with context, and the relayer must not refuse to start over its least important worker.Alert this enables
warning while there is still time to top up, instead of when deliveries are already failing. Units are ether (Hyperlane convention) so thresholds are human; the encode test pins metric name, labels and units so a rename or wei/ether mixup fails in CI rather than in a silently never-firing alert.
Verification
cargo test --workspace— 169 pass, 0 fail (3 new: role→chain mapping, bad-key skip, encode shape).-D warningsclean, fmt clean.