feat(relayer): make the settlement path observable, and alert on it

[DylanVerstraete]

Why the 20 August incident is still unalertable

Two days of deliveries succeeding while relay fees silently never settled. Adding the relayer_ack_submissions / relayer_claim_submissions counters in #42 was necessary but not sufficient, for a structural reason:

acknowledge_tx returns early when there is nothing to settle and increments nothing:

if !needs_ack && claim_ids.is_empty() {
    return Ok(AckOutcome::Terminal("nothing left to settle (...)".into()));
}

An unincremented Prometheus family emits zero bytes. So these three states are byte-identical from outside the process:

• settlement is idle (nothing required an ack, no unsettled fee)
• settlement is not configured for this route
• the ack worker is dead

No expression over those counters can separate them. And the outage shape does not appear in them anyway: a failed proof fetch defers the tx rather than resolving it, so during those two days the tx sat in the queue while the counters barely moved.

What this measures on usc-devnet right now

messages indexed, 24h	226
messages delivered, 24h	226
relayer_ack_submissions_total, all time	1
relayer_claim_submissions_total, all time	4

226 deliveries, zero settlement attempts recorded. I cannot tell from metrics whether that is correct (soak traffic requires no ack and has no claimable fee) or a repeat of 20 August. That ambiguity is the bug this PR fixes.

Two additions

relayer_settlement_queue_depth (gauge, per chain_key) — delivered txs discovered but not yet settled, published on every ack tick whether or not there is work. Two properties matter:

• It is a gauge, so it encodes at zero. absent() therefore genuinely means "the worker has never ticked", not "nothing happened yet".
• A deferred tx stays counted, so a queue that never returns to empty is exactly the 20 August signature — the one signal that stays pinned for the whole duration of such an outage.

SettlementOutcome::NothingToSettle — the no-op early return is counted instead of silent, so an idle route is distinguishable from a stopped one in the counters too. Claim is only incremented in AckAndClaim mode, since an Ack-only route has no claim to make.

PendingTxs gains len/is_empty to feed the gauge.

Alert rules: not in this repo

Three rules were written and verified live against usc-devnet, then removed from this PR per review — aks-grafana-alloy-iac is the home for deployed alerts, and two sources of truth for what is live is worse than one that arrives late. They are parked outside git until someone with access to that repo picks them up. I could not apply them directly: /v1/provisioning/alert-rules returns 403 with my token.

What stays here is the metric side, and the reasoning with it: the rustdoc on set_settlement_queue_depth states the alert shape and why a gauge rather than a counter is what makes absent() meaningful.

Two ordering constraints travel with the rules wherever they land:

• the queue-not-draining rule stays paused until a relayer carrying relayer_settlement_queue_depth is deployed to usc-devnet;
• the stuck-pool rule needs fix(relayer): stop retrying an under-funded delivery once a top-up is impossible #49 merged first, or its first act is to page about the message stuck since 24 Aug that cannot be rescued — the fastest way to get a new alert muted.

While writing them I did hit one thing worth recording, because it also applies to any future rule in this shape: absent() emits a series carrying the selector's equality matchers while vector(0) is unlabelled, so absent(...) or vector(0) returns two series and a following + silently discards the absent signal and evaluates to a hard 0 — reading healthy while the thing is entirely gone. Any aggregation without by (max(), sum()) drops labels and fixes it. Verified both ways against the live stack.

Tests

prom::tests::an_idle_settlement_path_is_still_visible asserts the two properties the alerting depends on: that a zero queue depth still encodes, and that the no-op outcome carries its own label value. metrics_encode_round_trips extended. Workspace green, clippy clean, fmt applied.

[cursor]

PR Summary

Low Risk
Metrics-only changes with no settlement or delivery behavior changes beyond counting no-op paths.

Overview
Adds observability so an idle settlement worker can be told apart from a stuck or dead one—addressing gaps where per-outcome counters stay empty during proof-fetch deferrals or when there is legitimately nothing to settle.

Each ack tick now publishes relayer_settlement_queue_depth (from PendingTxs::len()), including zero, so scrapes show liveness and a non-draining queue is visible while txs are re-deferred. When acknowledge_tx exits early with nothing to ack or claim, it increments SettlementOutcome::NothingToSettle on ack (and claim in AckAndClaim mode) instead of emitting no metrics. PendingTxs exposes len/is_empty; Prometheus wiring and a test assert zero depth and the new outcome label encode on scrape.

^{Reviewed by Cursor Bugbot for commit 4d9f9b3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 75fe0a6. Configure here.}