alerting: satd-alert crate + webhook dispatcher, absorbing reorgwebhook

CHANGELOG.md

@@ -153,6 +153,24 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md).
   free-space probe is bounded and carried across polls, so a `blocksdir` on an
   unresponsive network mount stalls only the disk alert — never the other
   detectors — and strands at most one blocking thread rather than one per poll.
+- Alerting: outbound webhooks. `alertfile=<path>` configures any number of
+  signed HTTP hooks, each filtered by category/kind/severity. Bodies are
+  byte-identical to the streaming API's JSON for the same event; delivery
+  metadata rides in headers (`X-Satd-Signature`, `X-Satd-Delivery`,
+  `X-Satd-Hook`, `X-Satd-Attempt`). Delivery is serial and in-order per hook,
+  retried with exponential backoff on transient failures and skipped on a
+  permanent 4xx. The per-hook queue is bounded and overflow **drops silently**
+  — there is no in-band gap notice; the record is
+  `satd_alertwebhook_dropped_total` and a log line. Redirects are never followed — a 3xx is a permanent drop, so a
+  signed body cannot be steered to a host the alertfile never named. Delivery is
+  **best-effort**: nothing is persisted, a hook that was down resumes at the
+  live head, and drops are counted in `satd_alertwebhook_dropped_total`. The
+  Streaming Consumption API remains the surface for guaranteed, resumable
+  consumption. Hooks reload on SIGHUP
+  (keep-last-good on error); per-hook counters are exported as
+  `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with
+  their original payload, now delivered by the same dispatcher — which also
+  moves that outbound HTTP off the consensus runtime.
 
 ## Releases
 

Cargo.toml

@@ -1,5 +1,5 @@
 [workspace]
-members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-policy"]
+members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-alert", "satd-policy"]
 # `fuzz/` is a standalone cargo-fuzz workspace (needs a nightly + sanitizer
 # toolchain); keep it out of the normal `cargo build` / `cargo test --all`.
 exclude = ["fuzz"]

docs/manual/src/config-reference.md

@@ -408,6 +408,7 @@ the taxonomy and the details each event carries.
 
 | Key | Default | Reload | Compat | Description |
 |---|---|---|---|---|
+| `alertfile` | none | path restart, contents hot | satd | TOML file describing outbound alert webhooks. Must be mode `0600` — it holds signing secrets. The *path* is read once at startup; the file's *contents* are re-read on every SIGHUP, so a hook can be added, edited, or removed live. A parse error keeps the last-good hook set. See [Observability → Alert webhooks](observability.md#alert-webhooks). |
 | `alerttipstallseconds` | `3600` (`0` on regtest) | hot | satd | Raise `tip_stall` after this many seconds with no connected block. Defaults to disabled on regtest only, where blocks exist just when a test mines them and an idle chain is normal; every other network — test networks included — keeps the hour, since going an hour without a block is not an ordinary property of thin hashrate the way a shallow reorg is. *Not* suppressed during initial block download — `is_initial_block_download()` compares the tip header's timestamp against the wall clock rather than tracking sync progress, so a node that was caught up and then wedged re-enters it precisely when you need paging. A node that is genuinely syncing connects blocks continuously and so never crosses the threshold. Cleared the moment a block connects — or, if you raise this value past the current tip age, on the next detector poll. |
 | `alertdiskfreemb` | `10240` | hot | satd | Raise `disk_low` below this many MiB free on the blocks directory (or the data directory when `blocksdir` is not split out). Clears at 1.5× the floor, or as soon as you lower the floor below the current reading. |
 | `alertmempoolfullpct` | `90` | hot | satd | Raise `mempool_congested` at this percentage of `maxmempool`. Clears below 75 % of the raise line, or as soon as you raise the threshold above the current occupancy. Values above 100 are clamped. |

docs/manual/src/observability.md

@@ -145,6 +145,174 @@ and `satd_disk_free_bytes` (free space on the watched directory). The latter is
 omitted rather than reported as zero when the filesystem cannot be
 interrogated.
 
+## Alert webhooks
+
+The three surfaces above all require something to be *watching* the node. A
+webhook pushes instead: point `alertfile=<path>` at a TOML file and satd POSTs
+each matching event to your endpoint.
+
+```toml
+version = 1
+
+[[webhook]]
+id = "pager"                        # unique; appears in the X-Satd-Hook header and metric labels
+url = "https://alerts.example/satd"
+secret = "a-long-random-string"     # required — it signs every delivery
+categories = ["status"]             # status | chain | mempool | heartbeat
+kinds = ["tip_stall", "disk_low"]   # optional, status only
+min_severity = "warning"            # optional, status only
+
+[[webhook]]
+id = "deadman"
+url = "https://hc-ping.example/abc123"
+secret = "another-long-random-string"
+categories = ["heartbeat"]
+heartbeat_interval_secs = 300       # one ping per 5 min, not the bus's 1 Hz
+```
+
+The file must be mode `0600` — it holds signing secrets, and satd refuses to
+read a group- or world-accessible one. Its **contents** are re-read on every
+`SIGHUP`, so hooks can be added, edited, or removed without a restart; the
+**path** is fixed at startup. A parse error on reload keeps the last-good hook
+set and logs why, because alerting that silently stopped after a typo is the
+worse failure.
+
+### What arrives
+
+```http
+POST /your/endpoint HTTP/1.1
+Content-Type: application/json
+X-Satd-Signature: sha256=<hex HMAC-SHA256(secret, canonical string)>
+X-Satd-Timestamp: 1753400000
+X-Satd-Delivery: <node_id>-<instance_id>-<seq>
+X-Satd-Hook: pager
+X-Satd-Attempt: 1
+X-Satd-Webhook-Version: 2
+
+{"schema_version":1,"stamp":{...},"body":{"category":"status", ...}}
+```
+
+The body is **byte-identical** to the JSON a WebSocket subscriber receives for
+the same event, so a receiver parses webhook bodies and streaming frames with
+one code path. Delivery metadata rides in headers and never in the body — which
+is what makes the signature stable across retries.
+
+### Verifying a delivery
+
+The signature covers a **canonical string**, not the bare body — five fields
+joined by newlines, with the raw body last:
+
+```
+"2" LF <X-Satd-Timestamp> LF <X-Satd-Delivery> LF <X-Satd-Hook> LF <raw body>
+```
+
+Signing only the body would leave `X-Satd-Delivery` unauthenticated *and*
+predictable, and that header is the one this page tells you to deduplicate on.
+One captured `(body, signature)` pair could then be replayed under forged future
+delivery ids, poisoning your dedup cache so the real alerts were discarded on
+arrival while satd counted them delivered. Binding the id, the hook, and a
+timestamp into the signed material closes that.
+
+So a receiver must:
+
+1. Read `X-Satd-Timestamp` and reject anything older than **600 seconds**. This
+   is not optional — it is what stops a captured delivery being a permanent
+   replay token. A delivery still being retried after the window ages out by
+   design; a 20-minute-old "disk is filling" alert is not worth acting on.
+2. Rebuild the canonical string above from the raw body, **before parsing it**.
+3. Compare the HMAC in constant time.
+4. Deduplicate on `X-Satd-Delivery` — stable across retries of one event, and
+   unique across restarts, so a retry and a genuine repeat are distinguishable.
+5. Reply `2xx` to acknowledge.
+
+> **Upgrading from the pre-release v1 scheme.** Earlier drafts signed the raw
+> body alone and sent `X-Satd-Webhook-Version: 1`. The legacy `reorgwebhook=`
+> keys still use exactly that, unchanged, and still report version `1` — branch
+> on the version header rather than assuming one scheme.
+
+The `X-Satd-Delivery` value is opaque; do not parse it. Deduplicate on the whole
+header: it is unique per event and stable across the retries of one delivery, so
+the only duplicate you can receive is a retry of something you already saw.
+
+### Delivery behavior
+
+- **Serial and in-order per hook** — one request in flight at a time, so events
+  arrive in the order the node produced them.
+- **Retried with backoff** on 5xx, 408, 429, timeouts, and connection failures:
+  1 s doubling to a 5-minute ceiling — but **not forever**. A delivery is
+  abandoned once it ages past the 600 s freshness window it was signed with,
+  which lands somewhere around the tenth or eleventh attempt. That is
+  deliberate: the signed timestamp is the only staleness signal a receiver
+  checks, so a delivery that could no longer pass that check is not worth
+  sending. The practical consequence is that a receiver down for longer than
+  ten minutes — a relay redeploy, a restart that takes a while — loses the
+  events raised during the outage. Alert on
+  `satd_alertwebhook_dropped_total` if that matters to you; the detectors
+  re-raise standing health conditions, so those recover on their own, but
+  chain and mempool events do not.
+  Any *other* 4xx is treated as permanent, counted, and skipped — a receiver
+  answering 404 forever must not pin the queue and turn every later event into
+  a drop. The skip is counted in `satd_alertwebhook_dropped_total` and logged.
+- **Redirects are not followed.** A 3xx is a permanent drop. The URL in the
+  alertfile is where the signed body goes; following a redirect would move it —
+  signature, hook identity and all — to a host you never named, and the useful
+  targets for that are exactly the ones you cannot see: a cloud metadata
+  endpoint, an RFC1918 admin port, the node's own RPC. If your receiver moves,
+  update the alertfile.
+- **Bounded queue.** A hook that falls far enough behind drops events. They are
+  counted in `satd_alertwebhook_dropped_total` and logged; nothing is held for
+  later and nothing is inserted into the stream to tell the receiver.
+- **Nothing reaches consensus.** Deliveries run on the isolated API runtime;
+  a stalled endpoint cannot affect block connection.
+- **Best-effort, and that is the whole contract.** Nothing is persisted. A
+  webhook fires when something happens and is retried while your endpoint is
+  briefly unreachable; that is all it promises. A node that was down did not
+  deliver those events and will not go back for them — when it comes up, its
+  hooks resume at the live head. Health alerts are the exception, and they get
+  it for free: the detectors re-evaluate at startup and re-raise anything still
+  true (see **Durability** above), so a standing problem still reaches you.
+
+  If you need guaranteed delivery, resumability across downtime, or history,
+  use the [Streaming Consumption API](streaming.md). That is the recommended
+  way to integrate with satd and it does all three properly — real cursors,
+  backpressure, and a bounded `RescanBlocks`. Webhooks are for automation you
+  are happy to miss occasionally: page me, poke a script, ping a dead-man's
+  switch.
+- **`chain` alerts are suppressed during initial block download.** A node
+  syncing from scratch does not POST its entire block history. `status`,
+  `heartbeat` **and `mempool`** keep flowing: health alerts stay live because
+  "this node is unhealthy" is exactly as true mid-sync, and the heartbeat keeps
+  flowing so an external dead-man's switch does not declare a syncing node dead.
+  Note the consequence for `mempool` hooks — a mainnet mempool subscription is
+  thousands of events a minute, and a multi-day sync does not quiet it. What was
+  suppressed is counted in `satd_alertwebhook_dropped_total`. The suppression is
+  latched on leaving IBD once, so a node whose tip later goes stale keeps
+  alerting — which is the whole point of a stalled-tip alert.
+
+Plaintext `http://` is accepted for loopback and private-network targets. For a
+public host, use `https://` or set `allow_insecure_http = true` on the hook.
+
+Per-hook counters are exported: `satd_alertwebhook_delivered_total`,
+`_failed_attempts_total`, `_dropped_total` (events lost, not held —
+there is no dead-letter queue),
+`_queue_depth`, and `_last_success_age_seconds`, all labelled `hook="<id>"`.
+Nothing is exported when no hook is configured.
+
+> **Note.** The older `reorgwebhook=` / `reorgwebhooksecret=` keys still work
+> and are now served by this dispatcher, with their original `ReorgRecord`
+> payload and v1 body-only signature unchanged.
+>
+> **One behavior did change:** redirects are no longer followed. A receiver
+> that answers 301/302 — an `http`→`https` proxy hop, a trailing-slash
+> redirect, a load balancer that relocates — used to be chased and now
+> classifies as a permanent drop. If your reorg endpoint relies on a redirect,
+> point `reorgwebhook=` at the final URL; otherwise every reorg record is
+> silently discarded. Everything else about the payload and signature is
+> byte-identical, so a receiver on a stable URL needs no edits.
+>
+> New deployments should prefer an `alertfile` hook with
+> `categories = ["chain"]`, which delivers the standard event envelope.
+
 ## Structured JSON Logging
 
 `satd` logs to stdout. Use `--log-format=json` to switch from the text format

docs/release-notes/0.5.0-pre.md

@@ -383,6 +383,80 @@ alert does not delete the series out from under a Prometheus rule of your own.
 Disk headroom previously had no alerting or metric at all, which is how a silent
 disk-fill wedged a dogfood node in May.
 
+### Alert webhooks
+
+Health events on the streaming API still require something to be *watching*.
+`alertfile=<path>` pushes instead — a TOML file describing any number of signed
+HTTP hooks, each filtered by category, kind, and severity floor:
+
+```toml
+version = 1
+
+[[webhook]]
+id = "pager"
+url = "https://alerts.example/satd"
+secret = "a-long-random-string"
+categories = ["status"]
+kinds = ["tip_stall", "disk_low"]
+```
+
+- **One wire schema, not two.** The POST body is byte-identical to the JSON a
+  WebSocket subscriber receives for the same event, so a receiver parses both
+  with one code path. Delivery metadata (`X-Satd-Signature`, `X-Satd-Delivery`,
+  `X-Satd-Hook`, `X-Satd-Attempt`) rides in headers and never in the body —
+  which is what keeps the signature stable across retries and makes
+  `X-Satd-Delivery` usable as an idempotency key.
+- **Failure behavior.** Delivery is serial and in-order per hook, with
+  exponential backoff to a five-minute ceiling on transient failures. A
+  permanent 4xx is counted and skipped rather than retried forever — a receiver
+  answering 404 must not pin the queue and convert every later event into a
+  drop — and the skip advances the resume position, so a hard-rejecting
+  endpoint does not rebuild the same refused span on every restart. Redirects
+  are never followed: a 3xx is a permanent drop, because chasing one would
+  carry the signed body to a host the alertfile never named. A hook that falls
+  far enough behind **drops events silently** — there is no in-band gap notice
+  and no resume cursor, because there is no durable log to resume from. Watch
+  `satd_alertwebhook_dropped_total` if you need to know. Do not read the
+  absence of a notice as the absence of a gap; if you need that guarantee, the
+  Streaming Consumption API is the surface that offers it.
+- **Best-effort by design.** Webhooks are the basic way to automate off chain
+  and mempool events — page me, poke a script, ping a dead-man's switch. They
+  are not an integration API and deliberately do not try to be one: nothing is
+  persisted, a hook that was down resumes at the live head, and events dropped
+  by a full queue or a lagging bus are counted in
+  `satd_alertwebhook_dropped_total` and logged. Retries cover the one failure
+  the surface promises to survive, a receiver that is briefly unreachable.
+  Health alerts are the exception and get it for free: the detectors re-evaluate
+  at startup and re-raise anything still standing, so a live problem reaches you
+  across a restart without any delivery machinery. Anything needing guaranteed
+  delivery, resumability, or history uses the **Streaming Consumption API**,
+  which is the canonical way to integrate with satd and already does all of it
+  properly — real cursors, backpressure, and a bounded `RescanBlocks`.
+- **Reloadable.** Hook definitions are re-read on every SIGHUP even when no
+  `bitcoin.conf` key changed, so a hook can be added, edited, or removed live.
+  A parse error keeps the last-good set — alerting that silently stopped after
+  a typo is the worse failure. The file must be mode 0600; it holds signing
+  secrets. A reload is a handover, not a restart: a hook whose stanza you did
+  not touch keeps its delivery task, so its queue and retry backoff survive.
+  Editing one hook does not discard a page that another hook was still
+  retrying — which matters because a status event has no replay behind it and
+  the detector that raised it will not raise it twice.
+- Per-hook counters are exported (`satd_alertwebhook_delivered_total`,
+  `_failed_attempts_total`, `_dropped_total`, `_queue_depth`,
+  `_last_success_age_seconds`), and a node with no alertfile exports none of
+  them.
+
+**`reorgwebhook=` is absorbed, not replaced.** The existing keys keep working
+and their payload is unchanged byte for byte — `ChainEvent::Reorg` does not
+carry the `depth`, `fork_height`, or disconnected/reconnected lists that a
+`ReorgRecord` does, so re-shaping it would have silently broken deployed
+receivers. It is now served by the new dispatcher, which also moves that
+outbound HTTP off the consensus runtime, where it should never have been. New
+deployments should prefer an `alertfile` hook with `categories = ["chain"]`.
+
+The signature scheme is specified with cross-language test vectors so a
+receiver can be verified without a running node.
+
 ## Storage / Core compatibility
 
 ### Core v28+ XOR-obfuscated block files (`blocksxor`)

mcp/src/context.rs

@@ -28,4 +28,6 @@ pub struct McpContext {
     /// Health-detector readings, so `get_metrics_snapshot` renders the same
     /// health gauges as the HTTP scrape. `None` in tests that bypass main.rs.
     pub health: Option<Arc<node::health::HealthState>>,
+    /// Webhook delivery counters, for the same reason. `None` in tests.
+    pub webhooks: Option<Arc<node::metrics::WebhookMetrics>>,
 }

mcp/src/tools/ergonomics.rs

@@ -47,6 +47,7 @@ pub fn get_metrics_snapshot(ctx: &McpContext) -> String {
         addr_subs: ctx.addr_subs.clone(),
         addr_enabled: ctx.addr_enabled,
         health: ctx.health.clone(),
+        webhooks: ctx.webhooks.clone(),
     };
     let body = metrics_ctx.render_prometheus();
     let result = json!({
@@ -82,6 +83,7 @@ pub fn get_readiness(ctx: &McpContext) -> String {
         // Readiness reads only chain heights; the health gauges are not
         // consulted, so there is nothing to thread through here.
         health: None,
+        webhooks: None,
     };
     let (ready, reason) = match metrics_ctx.is_ready() {
         Ok(()) => (true, None),

mcp/tests/tools.rs

@@ -59,6 +59,7 @@ fn make_test_ctx() -> (McpContext, tempfile::TempDir) {
         addr_enabled: false,
         addr_subs: None,
         health: None,
+        webhooks: None,
     };
     (ctx, dir)
 }
@@ -616,6 +617,7 @@ mod mining {
             addr_enabled: false,
             addr_subs: None,
             health: None,
+            webhooks: None,
         };
 
         let result = mine::generate_blocks(&ctx, 1, REGTEST_ADDR);