alerting: node-health detectors, warnings mirroring, and gauges

[bkeroack]

PR 2 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §3). Stacked on #480 — merge that first.

PR 1 added the StatusEvent wire schema with no emitters. This adds the detectors that fill it.

What

satd now watches six conditions about itself and reports each through three surfaces simultaneously, driven from one state machine so they cannot disagree:

• a status streaming event (category bit 16),
• an entry in getwarnings — which fires the Core-compatible alertnotify hook,
• a satd_alert_active{kind="..."} gauge on /metrics.

kind	severity	raises	clears
ibd_complete	info	IBD finishes	one-shot
tip_stall	critical	no block for alerttipstallseconds, outside IBD	next block connects
disk_low	critical	free space under alertdiskfreemb	1.5× the floor
mempool_congested	warning	alertmempoolfullpct of the byte cap	below 75 % of the raise line
peer_floor	warning	under alertpeerfloor for 60 s	at/above for 60 s
deep_reorg	critical	reorg ≥ alertreorgdepth blocks	one-shot

Decisions worth reviewing

• Level-triggered with a real hysteresis gap. Clearing at the same value that raises turns a metric hovering at the threshold into a pager storm, so every standing condition has a ratio, a hold time, or both between its two lines. Fixed constants — six raise thresholds is already plenty of operator surface.
• Durability by re-evaluation. Status events aren't replayable, so what makes health alerting at-least-once across a restart is that the task re-evaluates from scratch and re-raises anything still standing. A condition that raised and cleared while nobody was listening is stale by definition. Corollary, documented in the manual: a subscriber that attaches after a raise won't see it until the condition changes — getwarnings is the current-state query. (Snapshot-on-subscribe is deferred per design §8.)
• API runtime, never the consensus core (design D7). A stall detector whose poll can be delayed by block connection is the one thing it must not be.
• deep_reorg reports the true depth. ChainEvent::Reorg carries old/new tip heights but no fork point, so the detector counts the reorg's disconnects — exactly old_height − fork_height, the same number ReorgRecord stores — rather than reporting the threshold it crossed. A truncation reorg (invalidateblock) has no replacement chain to connect, so a count that goes quiet is finalized by the poll loop instead of waiting for a connect that may never come.
• Thresholds are hot-reloadable, because an operator retuning an alert is usually doing it while it's firing. Disabling a threshold also clears a standing condition rather than stranding a warning nothing will ever retract.
• ibd_complete deliberately creates no warning — it's good news, and a warning for it would sit in getwarnings forever with nothing to clear it.
• tip_stall's clock is seeded at startup, not from the tip timestamp. A node that was down for hours has a stale tip but connects its backlog within seconds; seeding from the tip would page for a stall that is really a restart.
• satd_disk_free_bytes is omitted, not zeroed, when the filesystem can't be interrogated — a zero there is exactly the alarm an operator must not be shown falsely. Same reason render_health_metrics emits nothing at all when no detector is running.

Also generalizes free_disk_bytes out of the three index runners that had each grown a private copy, and adds satd_tip_last_connect_age_seconds / satd_disk_free_bytes. Disk headroom previously had no metric or alerting — which is how a silent disk-fill wedged a dogfood node in May.

Config

alerttipstallseconds=3600, alertdiskfreemb=10240, alertmempoolfullpct=90, alertpeerfloor=3, alertreorgdepth=3. All hot-reloadable; 0 disables a detector. Registered in KNOWN_CONFIG_KEYS and the reload table, so the anti-drift coverage tests cover them.

Acceptance (design §10 PR 2)

• E2E raise/clear over a real socket: tip_stall raise→clear, disk_low raised by a live SIGHUP threshold change (which also covers the reload path), deep_reorg's derived depth/fork height, and the explicit-only category guarantee (a categories=0 subscriber sees chain events but never a status event while one is firing).
• getwarnings gains/loses alert.* in lockstep with the events.
• Detector-off (=0) emits nothing and clears anything standing.
• Threshold defaults come from one source (node::health::defaults), asserted from the config layer.
• clippy --workspace --all-targets --all-features clean.

peer_floor is intentionally not covered E2E — it holds 60 s in either direction by design, which is longer than an E2E test should sit; its logic is unit-tested.

Local test run: green except the 10 pre-existing test_address_index_backfill_* regtest cases, which fail on this machine's disk-space guard (needs ~80 GB, 68 GB free) and are unrelated.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

[bkeroack]

Round-1 review fix (Medium — ops reliability): tip_stall now also clears from the poll path when the tip age falls back under the threshold, not only on BlockConnected.

The event-driven clear stays as the fast path — the alert should lift the instant the chain moves. The problem was the other direction: a node that genuinely is not receiving blocks has no event to clear on, and raising alerttipstallseconds via SIGHUP is the documented way to quiet a firing alert. Without a poll-side clear the condition, its getwarnings entry, the satd_alert_active{kind="tip_stall"} gauge and the stream state all stayed raised until some future block. Every other standing detector already cleared from the poll.

New e2e test raises the alert via SIGHUP, then raises the threshold via SIGHUP and asserts the clear arrives with no block mined. Verified it fails without the change. Manual and config-reference updated.

[bkeroack]

Implemented the -connect fix discussed in review (commit 7f8c0e23).

peer_floor_for now takes the -connect= count and caps the default at it:

pub fn peer_floor_for(network: bitcoin::Network, connect_peers: usize) -> u64 {
    match network {
        bitcoin::Network::Regtest => 0,
        _ if connect_peers > 0 => PEER_FLOOR.min(connect_peers as u64),
        _ => PEER_FLOOR,
    }
}

Why. -connect= pins the node to exactly the addresses named and suppresses both DNS seeding and the fixed seeds (satd/src/main.rs:2918, :2946), so a node given one upstream can never reach a floor of 3 — no code path is left that would add a peer. The raise would stand forever in getwarnings, -alertnotify, the TUI modal, and getblockchaininfo.warnings, which wallet software renders to end users. satd_alert_active{kind="peer_floor"} pinned high also stops distinguishing peer-starved from configured-that-way.

This is the regtest trap in different clothes, and peer_floor_for's own doc comment already argued the case — it just keyed off the network when the condition is a property of the configuration.

Capped, not disabled, so the alert still does the one useful thing it can here: report a -connect= node that had all its configured peers and then lost one. An explicit alertpeerfloor overrides in either direction, including to 0.

The default had to move out of the Config struct literal — that literal has already moved connect by the time it reaches the alert_peer_floor field.

Tests (both run against the previous behavior and confirmed failing there):

• node: peer_floor_default_is_disabled_only_on_regtest extended — 1 and 2 -connect peers give floors of 1 and 2, 8 caps at 3, regtest stays 0.
• satd: new the_peer_floor_default_follows_the_connect_list — CLI and bitcoin.conf connect= paths, the cap, explicit-override in both directions, and regtest.

Docs updated in the same commit: CHANGELOG.md, docs/release-notes/0.5.0-pre.md, and the alertpeerfloor row in docs/manual/src/config-reference.md.

Stack rebased on top: #482 → #484 → #485 → #486, all force-pushed with --force-with-lease, bases unchanged.

[bkeroack]

Option A implemented for alerttipstallseconds (commit 5a978b9c) — finding 2(a).

pub fn tip_stall_for(network: bitcoin::Network) -> u64 {
    match network {
        bitcoin::Network::Regtest => 0,
        _ => TIP_STALL_SECS,
    }
}

This was the last threshold in the set without a network-conditional default; peer_floor_for and reorg_depth_for already turned regtest off for the same reason. last_connect is seeded at detector start and advanced only by BlockConnected, so a regtest node idling an hour between test runs raised a critical alert — pinning getwarnings, holding has_errors() true, and putting up the TUI's blocking modal on a chain where having no blocks is the resting state.

Test networks keep the hour rather than being relaxed the way reorg_depth_for relaxes them: unlike a shallow reorg, going an hour without a block is not an ordinary property of thin hashrate — signet and testnet4 are still expected to make blocks.

Finding 2(b) — the mainnet rate — is deliberately not changed. At a 600 s mean interval, P(gap > 3600 s) = e⁻⁶ ≈ 0.25 %, so a healthy node raises roughly every 2.8 days. The reviewer is right about the arithmetic and right that an error-severity page twice a week is the alert-fatigue mechanism this codebase warns about elsewhere. But 3600 is documented on TIP_STALL_SECS as an intentional trade, and raising it to 7200 (≈ once every 3 years) costs an hour of detection latency — that is a product call, not a review fix, so it stays the operator's to tune via alerttipstallseconds. Happy to revisit.

Tests: tip_stall_default_is_disabled_only_on_regtest in node, and the regtest/mainnet/signet assertions in health_alert_threshold_defaults_and_parsing. That test previously asserted 3600 on a regtest config and passed, so it is a direct negative check on the new value.

Docs: CHANGELOG.md, 0.5.0-pre.md (the "two defaults are network-dependent" paragraph is now three), and the alerttipstallseconds row in config-reference.md.

Stack rebased: #482 → #484 → #485 → #486, force-pushed with --force-with-lease, bases unchanged. One conflict in config-reference.md where #482 adds the alertfile row adjacent to this one — resolved keeping both.