alerting: StatusEvent wire schema + status category (bit 16)

[bkeroack]

PR 1 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §10). Wire schema only — no emitters, no config, no behavior change.

Why

Every body on the streaming API describes the chain or the mempool. None describes the node, so an operator who wanted to know their daemon had stalled, filled its disk, or lost its peers had to poll getblockchaininfo on a timer and diff the results.

Per design decision D2, health events are first-class on the firehose rather than internal to the webhook dispatcher. Every carrier and both SDKs then get them with zero extra plumbing, and the dispatcher landing in PR 3 is just another bus consumer.

What

• proto: StatusEvent + StatusKind/StatusState/StatusSeverity enums; NodeEvent.status = 31 (28/29/30 taken by the SP work). No SubscribeRequest field — the category bit is the opt-in.
• node: node/src/events/status.rs (the Rust types), NodeEventBody::Status, CATEGORY_STATUS = 16 joining EXPLICIT_ONLY_CATEGORIES, EventPublisher::publish_status.
• carriers: gRPC status_event_to_proto mapping arm; WS/SSE serde passthrough; ZMQ nodeevent-only (no Core-compat topic — Core's equivalent is -alertnotify, which satd drives from the same detectors in PR 2).
• docs: docs/api/streaming.md §7.8, CHANGELOG + 0.5.0-pre.md.

Decisions worth reviewing

• Bit 16 is explicit-request only. Follows the tweaks precedent: a categories=0 subscriber written against an older node never starts receiving a body it can't parse after an upgrade.
• WS/SSE serves status, unlike tweaks. The WS mask strips CATEGORY_TWEAKS from every mask because tweaks are firehose-scale and gRPC-only; status is low-volume JSON with no index prerequisite, so an explicit bit-16 request survives the mask. Asserted in a test.
• No cursor, not in the replay ring. Health events are not replayable at all — durability comes from detectors re-evaluating and re-raising standing conditions at startup (PR 2). Keeping them out of the ring also stops them evicting the mempool transitions the ring exists for.
• details is a BTreeMap, not a HashMap. The webhook body is HMAC-signed; golden signature vectors would be unreproducible under map-iteration order.
• Schema version stays 1 — adding a body variant is a no-bump change under the documented evolution policy (node/src/events/schema.rs).

Acceptance (design §10 PR 1)

• Schema rules hold, no version bump.
• categories=0 subscriber receives nothing new — asserted in status_category_is_explicit_only and the WS mask test.
• cargo check --workspace --tests (the recurring proto-literal gotcha — no SubscribeRequest field was added this time, so only one exhaustive-match test site needed a new arm).
• clippy --workspace --all-targets --all-features clean; full test suite green.

Local test run: everything passes except the 10 pre-existing test_address_index_backfill_* regtest cases, which fail on this machine's disk-space guard (need ~80 GB, have 68 GB free) and are unrelated to this change.

Stack

PR 1 (this) → PR 2 detectors + warnings + metrics → PR 3 satd-alert crate + webhook dispatcher → PR 4 watch-set hooks → PR 5 bench + docs → PR 6 SDK.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

[bkeroack]

Deep review — findings (#480 + #481)

Independent review; these are reviewer-reported and I have not personally re-verified them — worth confirming before acting.

Wire-schema compatibility is clean. Bit 16 is genuinely explicit-only end to end: ALL_CATEGORIES_DEFAULT = !(TWEAKS|STATUS) is applied at every mask origin (gRPC Subscribe, Watch init, SetWatchSet, WS/SSE mask_from and SetCategories; ZMQ excluded wholesale). Proto tag 31 and the new enums collide with nothing, reserved 15 is respected, no existing message changed. Detector state machines also check out — hysteresis, clear_if_threshold_relaxed, the ReorgSeen identity set (sized 512 > the log ring's 256, so an evicted key can't be re-presented), history(u64::MAX) overflow, and MAX_DETAIL_LEN char-boundary truncation are all right.

1. peer_floor default of 3 permanently warns on a standard -connect= node — MEDIUM/HIGH

node/src/health.rs disables the floor only on regtest, and detectors are spawned unconditionally. A non-empty connect disables DNS seeding and fixed-seed fallback, and the reconnect loop only re-dials connect_addrs — so the node holds exactly 1 peer forever. ~60–150 s after start peer_floor raises, and getwarnings / getblockchaininfo.warnings become permanently non-empty, the TUI warning modal opens over every view, and -alertnotify fires — on a correctly configured node whose operator never opted into alerting. Same for onlynet=onion and firewalled nodes. getblockchaininfo.warnings is Core-compat surface that downstream clients render to end users.

2. alerttipstallseconds is not network-conditional, unlike the other two defaults — MEDIUM

TIP_STALL_SECS = 3600 is used verbatim everywhere, while peer_floor_for and reorg_depth_for are per-network.
(a) Regtest: idling an hour without mining raises tip_stall at Severity::Error → has_errors() → blocking TUI modal + -alertnotify. A regtest node only advances on generatetoaddress, so idling is its normal state — exactly the "poor greeting for every developer's first run" the PR cites when disabling the other two on regtest.
(b) Mainnet: P(interblock gap > 3600 s) = e⁻⁶ ≈ 0.25 %, ≈ 0.36/day. A healthy mainnet node raises an error-severity standing warning about every three days.

3. check_disk's spawn_blocking does not give the isolation its comment claims — MEDIUM

The comment says calling statvfs inline would "freeze all of them". spawn_blocking(...).await with no timeout has the identical effect on the detectors — it only spares the API worker. With -blocksdir on a hard NFS mount whose server disappears, the detector task parks on the JoinHandle forever: tip_stall, deep_reorg, check_mempool, check_peers never run again, chain_rx is never polled, and every gauge freezes at its last value — so an external Prometheus rule on tip-age can't fire either. A wedged storage mount silently disables the whole alerting subsystem. A tokio::time::timeout around the join (elapsed ⇒ None, already handled) fixes it.

4. The status category is gated only by stream:subscribe — MEDIUM (information disclosure)

satd-auth deliberately separates rpc:read from stream:subscribe, and neither carrier applies an extra check for bit 16. A token issued ["stream:subscribe"] — specifically to withhold getpeerinfo/getblockchaininfo/getwarnings — can set categories=16 and receive free bytes on the node's volume, peer topology (in/out counts), mempool cap + occupancy + mempoolminfee, tip heights, IBD state, and reorg depth/fork heights. The datadir path is correctly withheld; the rest is host telemetry that was previously behind a different capability.

5. deep_reorg fires -alertnotify per occurrence with no dedup, on an unbounded channel — LOW

Unlike record, notify_event has no per-id dedup and no rate limit; alert_tx is an UnboundedSender and each message spawns a shell command. A burst of at-threshold reorgs (thin-hashrate test network, or a scripted invalidateblock/reconsiderblock loop) queues one exec each.

6. Docs and the proto contradict the implementation on IBD suppression — LOW

check_tip_stall_values explicitly ignores IBD (let _ = in_ibd;), but 0.5.0-pre.md says both "tip_stall is suppressed during initial block download" and "is not suppressed during initial block download" in the same paragraph; the proto comment on STATUS_KIND_TIP_STALL says "while not in IBD"; config.rs and the CLI help say "outside IBD". Only config-reference.md and the StatusKind::TipStall rustdoc are right.

[bkeroack]

Tier 1 + Tier 2 review findings fixed

Working bottom-up across the stack; every branch was rebased and force-pushed with --force-with-lease, bases unchanged.

On this PR (#480)

Finding 4 — status gated only by stream:subscribe (the highest-severity open item). Fixed in c757408e. The category now additionally requires rpc:read.

StatusEvent reports free disk, peer counts split in/out, mempool cap + occupancy + mempoolminfee, tip height, IBD state, reorg depth — the content of getblockchaininfo, getpeerinfo, getmempoolinfo, getwarnings. satd-auth separates the two capabilities precisely so a streaming token can be issued without node read RPC, so serving status on stream:subscribe alone routed around the split rather than extending it.

• Refused at the handshake — PERMISSION_DENIED (gRPC Subscribe) / 403 (WS/SSE). Not stripped: bit 16 is never in the categories=0 default, so asking for it is deliberate, and a silently-absent category reads to a dashboard as "nothing is wrong".
• Stripped on the mid-stream control paths — SetCategories and SetWatchSet on both carriers, which have no per-message error channel. Closes the second door.
• No new capability. The vocabulary is a Tier 1 surface and the data already has an owner in it; status is new this release so no deployed token loses anything.
• Defaults untouched. With no token store may_receive_status returns true (loopback trust), and the operator principal holds CapabilitySet::ALL. Both asserted.

New events/src/catauth.rs holds the policy with 4 unit tests; 2 more cover the gRPC/WS wiring. All were run against the ungated code and fail there.

Finding 6 (part) — the STATUS_KIND_TIP_STALL proto comment claimed IBD suppression; check_tip_stall_values deliberately ignores IBD. Corrected here; the release-notes paragraph that asserted both readings in consecutive sentences, and config.rs's field doc + CLI help, are fixed on #481.

Finding 2(a) — alerttipstallseconds is now 0 on regtest (tip_stall_for), on #481. 2(b), the ~2.8-day mainnet raise rate, is deliberately unchanged — see the earlier comment.

Finding 3 — check_disk's unbounded spawn_blocking fixed on #481.

Deferred (Tier 3), with reasons

Finding 5 — deep_reorg fires -alertnotify per occurrence, no dedup, unbounded channel. Not fixed. An edge event should fire per occurrence — that is what makes it an edge rather than a standing condition — so the correct fix is a rate limit, and choosing the interval is an operator policy call rather than a review fix. alertreorgdepth already defaults to 10 on test networks and 0 on regtest, which removes the realistic burst sources. Worth revisiting if anyone reports exec pressure.

Full deferral list with rationale is on #486.