diff --git a/CHANGELOG.md b/CHANGELOG.md index f9c842913..740b7eaec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -153,6 +153,24 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). free-space probe is bounded and carried across polls, so a `blocksdir` on an unresponsive network mount stalls only the disk alert — never the other detectors — and strands at most one blocking thread rather than one per poll. +- Alerting: outbound webhooks. `alertfile=` configures any number of + signed HTTP hooks, each filtered by category/kind/severity. Bodies are + byte-identical to the streaming API's JSON for the same event; delivery + metadata rides in headers (`X-Satd-Signature`, `X-Satd-Delivery`, + `X-Satd-Hook`, `X-Satd-Attempt`). Delivery is serial and in-order per hook, + retried with exponential backoff on transient failures and skipped on a + permanent 4xx. The per-hook queue is bounded and overflow **drops silently** + — there is no in-band gap notice; the record is + `satd_alertwebhook_dropped_total` and a log line. Redirects are never followed — a 3xx is a permanent drop, so a + signed body cannot be steered to a host the alertfile never named. Delivery is + **best-effort**: nothing is persisted, a hook that was down resumes at the + live head, and drops are counted in `satd_alertwebhook_dropped_total`. The + Streaming Consumption API remains the surface for guaranteed, resumable + consumption. Hooks reload on SIGHUP + (keep-last-good on error); per-hook counters are exported as + `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with + their original payload, now delivered by the same dispatcher — which also + moves that outbound HTTP off the consensus runtime. ## Releases diff --git a/Cargo.lock b/Cargo.lock index bdfdbb7c7..1a92c50a0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4081,6 +4081,7 @@ dependencies = [ "rand 0.8.6", "rcgen", "reqwest", + "satd-alert", "satd-auth", "satd-events", "satd-events-client", @@ -4088,6 +4089,7 @@ dependencies = [ "sd-notify", "serde", "serde_json", + "subtle", "tempfile", "tikv-jemallocator", "tls-config", @@ -4100,6 +4102,22 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "satd-alert" +version = "0.4.0" +dependencies = [ + "hex", + "hmac", + "node", + "serde", + "sha2", + "tempfile", + "thiserror 2.0.18", + "toml_edit", + "tracing", + "url", +] + [[package]] name = "satd-auth" version = "0.4.0" diff --git a/Cargo.toml b/Cargo.toml index d25b3e632..5bc115363 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,5 @@ [workspace] -members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-policy"] +members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-alert", "satd-policy"] # `fuzz/` is a standalone cargo-fuzz workspace (needs a nightly + sanitizer # toolchain); keep it out of the normal `cargo build` / `cargo test --all`. exclude = ["fuzz"] diff --git a/docs/manual/src/config-reference.md b/docs/manual/src/config-reference.md index edb8b57b9..5568c335e 100644 --- a/docs/manual/src/config-reference.md +++ b/docs/manual/src/config-reference.md @@ -408,6 +408,7 @@ the taxonomy and the details each event carries. | Key | Default | Reload | Compat | Description | |---|---|---|---|---| +| `alertfile` | none | path restart, contents hot | satd | TOML file describing outbound alert webhooks. Must be mode `0600` — it holds signing secrets. The *path* is read once at startup; the file's *contents* are re-read on every SIGHUP, so a hook can be added, edited, or removed live. A parse error keeps the last-good hook set. See [Observability → Alert webhooks](observability.md#alert-webhooks). | | `alerttipstallseconds` | `3600` (`0` on regtest) | hot | satd | Raise `tip_stall` after this many seconds with no connected block. Defaults to disabled on regtest only, where blocks exist just when a test mines them and an idle chain is normal; every other network — test networks included — keeps the hour, since going an hour without a block is not an ordinary property of thin hashrate the way a shallow reorg is. *Not* suppressed during initial block download — `is_initial_block_download()` compares the tip header's timestamp against the wall clock rather than tracking sync progress, so a node that was caught up and then wedged re-enters it precisely when you need paging. A node that is genuinely syncing connects blocks continuously and so never crosses the threshold. Cleared the moment a block connects — or, if you raise this value past the current tip age, on the next detector poll. | | `alertdiskfreemb` | `10240` | hot | satd | Raise `disk_low` below this many MiB free on the blocks directory (or the data directory when `blocksdir` is not split out). Clears at 1.5× the floor, or as soon as you lower the floor below the current reading. | | `alertmempoolfullpct` | `90` | hot | satd | Raise `mempool_congested` at this percentage of `maxmempool`. Clears below 75 % of the raise line, or as soon as you raise the threshold above the current occupancy. Values above 100 are clamped. | diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index c2b0cfe09..1ebf88287 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -145,6 +145,174 @@ and `satd_disk_free_bytes` (free space on the watched directory). The latter is omitted rather than reported as zero when the filesystem cannot be interrogated. +## Alert webhooks + +The three surfaces above all require something to be *watching* the node. A +webhook pushes instead: point `alertfile=` at a TOML file and satd POSTs +each matching event to your endpoint. + +```toml +version = 1 + +[[webhook]] +id = "pager" # unique; appears in the X-Satd-Hook header and metric labels +url = "https://alerts.example/satd" +secret = "a-long-random-string" # required — it signs every delivery +categories = ["status"] # status | chain | mempool | heartbeat +kinds = ["tip_stall", "disk_low"] # optional, status only +min_severity = "warning" # optional, status only + +[[webhook]] +id = "deadman" +url = "https://hc-ping.example/abc123" +secret = "another-long-random-string" +categories = ["heartbeat"] +heartbeat_interval_secs = 300 # one ping per 5 min, not the bus's 1 Hz +``` + +The file must be mode `0600` — it holds signing secrets, and satd refuses to +read a group- or world-accessible one. Its **contents** are re-read on every +`SIGHUP`, so hooks can be added, edited, or removed without a restart; the +**path** is fixed at startup. A parse error on reload keeps the last-good hook +set and logs why, because alerting that silently stopped after a typo is the +worse failure. + +### What arrives + +```http +POST /your/endpoint HTTP/1.1 +Content-Type: application/json +X-Satd-Signature: sha256= +X-Satd-Timestamp: 1753400000 +X-Satd-Delivery: -- +X-Satd-Hook: pager +X-Satd-Attempt: 1 +X-Satd-Webhook-Version: 2 + +{"schema_version":1,"stamp":{...},"body":{"category":"status", ...}} +``` + +The body is **byte-identical** to the JSON a WebSocket subscriber receives for +the same event, so a receiver parses webhook bodies and streaming frames with +one code path. Delivery metadata rides in headers and never in the body — which +is what makes the signature stable across retries. + +### Verifying a delivery + +The signature covers a **canonical string**, not the bare body — five fields +joined by newlines, with the raw body last: + +``` +"2" LF LF LF LF +``` + +Signing only the body would leave `X-Satd-Delivery` unauthenticated *and* +predictable, and that header is the one this page tells you to deduplicate on. +One captured `(body, signature)` pair could then be replayed under forged future +delivery ids, poisoning your dedup cache so the real alerts were discarded on +arrival while satd counted them delivered. Binding the id, the hook, and a +timestamp into the signed material closes that. + +So a receiver must: + +1. Read `X-Satd-Timestamp` and reject anything older than **600 seconds**. This + is not optional — it is what stops a captured delivery being a permanent + replay token. A delivery still being retried after the window ages out by + design; a 20-minute-old "disk is filling" alert is not worth acting on. +2. Rebuild the canonical string above from the raw body, **before parsing it**. +3. Compare the HMAC in constant time. +4. Deduplicate on `X-Satd-Delivery` — stable across retries of one event, and + unique across restarts, so a retry and a genuine repeat are distinguishable. +5. Reply `2xx` to acknowledge. + +> **Upgrading from the pre-release v1 scheme.** Earlier drafts signed the raw +> body alone and sent `X-Satd-Webhook-Version: 1`. The legacy `reorgwebhook=` +> keys still use exactly that, unchanged, and still report version `1` — branch +> on the version header rather than assuming one scheme. + +The `X-Satd-Delivery` value is opaque; do not parse it. Deduplicate on the whole +header: it is unique per event and stable across the retries of one delivery, so +the only duplicate you can receive is a retry of something you already saw. + +### Delivery behavior + +- **Serial and in-order per hook** — one request in flight at a time, so events + arrive in the order the node produced them. +- **Retried with backoff** on 5xx, 408, 429, timeouts, and connection failures: + 1 s doubling to a 5-minute ceiling — but **not forever**. A delivery is + abandoned once it ages past the 600 s freshness window it was signed with, + which lands somewhere around the tenth or eleventh attempt. That is + deliberate: the signed timestamp is the only staleness signal a receiver + checks, so a delivery that could no longer pass that check is not worth + sending. The practical consequence is that a receiver down for longer than + ten minutes — a relay redeploy, a restart that takes a while — loses the + events raised during the outage. Alert on + `satd_alertwebhook_dropped_total` if that matters to you; the detectors + re-raise standing health conditions, so those recover on their own, but + chain and mempool events do not. + Any *other* 4xx is treated as permanent, counted, and skipped — a receiver + answering 404 forever must not pin the queue and turn every later event into + a drop. The skip is counted in `satd_alertwebhook_dropped_total` and logged. +- **Redirects are not followed.** A 3xx is a permanent drop. The URL in the + alertfile is where the signed body goes; following a redirect would move it — + signature, hook identity and all — to a host you never named, and the useful + targets for that are exactly the ones you cannot see: a cloud metadata + endpoint, an RFC1918 admin port, the node's own RPC. If your receiver moves, + update the alertfile. +- **Bounded queue.** A hook that falls far enough behind drops events. They are + counted in `satd_alertwebhook_dropped_total` and logged; nothing is held for + later and nothing is inserted into the stream to tell the receiver. +- **Nothing reaches consensus.** Deliveries run on the isolated API runtime; + a stalled endpoint cannot affect block connection. +- **Best-effort, and that is the whole contract.** Nothing is persisted. A + webhook fires when something happens and is retried while your endpoint is + briefly unreachable; that is all it promises. A node that was down did not + deliver those events and will not go back for them — when it comes up, its + hooks resume at the live head. Health alerts are the exception, and they get + it for free: the detectors re-evaluate at startup and re-raise anything still + true (see **Durability** above), so a standing problem still reaches you. + + If you need guaranteed delivery, resumability across downtime, or history, + use the [Streaming Consumption API](streaming.md). That is the recommended + way to integrate with satd and it does all three properly — real cursors, + backpressure, and a bounded `RescanBlocks`. Webhooks are for automation you + are happy to miss occasionally: page me, poke a script, ping a dead-man's + switch. +- **`chain` alerts are suppressed during initial block download.** A node + syncing from scratch does not POST its entire block history. `status`, + `heartbeat` **and `mempool`** keep flowing: health alerts stay live because + "this node is unhealthy" is exactly as true mid-sync, and the heartbeat keeps + flowing so an external dead-man's switch does not declare a syncing node dead. + Note the consequence for `mempool` hooks — a mainnet mempool subscription is + thousands of events a minute, and a multi-day sync does not quiet it. What was + suppressed is counted in `satd_alertwebhook_dropped_total`. The suppression is + latched on leaving IBD once, so a node whose tip later goes stale keeps + alerting — which is the whole point of a stalled-tip alert. + +Plaintext `http://` is accepted for loopback and private-network targets. For a +public host, use `https://` or set `allow_insecure_http = true` on the hook. + +Per-hook counters are exported: `satd_alertwebhook_delivered_total`, +`_failed_attempts_total`, `_dropped_total` (events lost, not held — +there is no dead-letter queue), +`_queue_depth`, and `_last_success_age_seconds`, all labelled `hook=""`. +Nothing is exported when no hook is configured. + +> **Note.** The older `reorgwebhook=` / `reorgwebhooksecret=` keys still work +> and are now served by this dispatcher, with their original `ReorgRecord` +> payload and v1 body-only signature unchanged. +> +> **One behavior did change:** redirects are no longer followed. A receiver +> that answers 301/302 — an `http`→`https` proxy hop, a trailing-slash +> redirect, a load balancer that relocates — used to be chased and now +> classifies as a permanent drop. If your reorg endpoint relies on a redirect, +> point `reorgwebhook=` at the final URL; otherwise every reorg record is +> silently discarded. Everything else about the payload and signature is +> byte-identical, so a receiver on a stable URL needs no edits. +> +> New deployments should prefer an `alertfile` hook with +> `categories = ["chain"]`, which delivers the standard event envelope. + ## Structured JSON Logging `satd` logs to stdout. Use `--log-format=json` to switch from the text format diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index a6ac85419..57929d552 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -383,6 +383,80 @@ alert does not delete the series out from under a Prometheus rule of your own. Disk headroom previously had no alerting or metric at all, which is how a silent disk-fill wedged a dogfood node in May. +### Alert webhooks + +Health events on the streaming API still require something to be *watching*. +`alertfile=` pushes instead — a TOML file describing any number of signed +HTTP hooks, each filtered by category, kind, and severity floor: + +```toml +version = 1 + +[[webhook]] +id = "pager" +url = "https://alerts.example/satd" +secret = "a-long-random-string" +categories = ["status"] +kinds = ["tip_stall", "disk_low"] +``` + +- **One wire schema, not two.** The POST body is byte-identical to the JSON a + WebSocket subscriber receives for the same event, so a receiver parses both + with one code path. Delivery metadata (`X-Satd-Signature`, `X-Satd-Delivery`, + `X-Satd-Hook`, `X-Satd-Attempt`) rides in headers and never in the body — + which is what keeps the signature stable across retries and makes + `X-Satd-Delivery` usable as an idempotency key. +- **Failure behavior.** Delivery is serial and in-order per hook, with + exponential backoff to a five-minute ceiling on transient failures. A + permanent 4xx is counted and skipped rather than retried forever — a receiver + answering 404 must not pin the queue and convert every later event into a + drop — and the skip advances the resume position, so a hard-rejecting + endpoint does not rebuild the same refused span on every restart. Redirects + are never followed: a 3xx is a permanent drop, because chasing one would + carry the signed body to a host the alertfile never named. A hook that falls + far enough behind **drops events silently** — there is no in-band gap notice + and no resume cursor, because there is no durable log to resume from. Watch + `satd_alertwebhook_dropped_total` if you need to know. Do not read the + absence of a notice as the absence of a gap; if you need that guarantee, the + Streaming Consumption API is the surface that offers it. +- **Best-effort by design.** Webhooks are the basic way to automate off chain + and mempool events — page me, poke a script, ping a dead-man's switch. They + are not an integration API and deliberately do not try to be one: nothing is + persisted, a hook that was down resumes at the live head, and events dropped + by a full queue or a lagging bus are counted in + `satd_alertwebhook_dropped_total` and logged. Retries cover the one failure + the surface promises to survive, a receiver that is briefly unreachable. + Health alerts are the exception and get it for free: the detectors re-evaluate + at startup and re-raise anything still standing, so a live problem reaches you + across a restart without any delivery machinery. Anything needing guaranteed + delivery, resumability, or history uses the **Streaming Consumption API**, + which is the canonical way to integrate with satd and already does all of it + properly — real cursors, backpressure, and a bounded `RescanBlocks`. +- **Reloadable.** Hook definitions are re-read on every SIGHUP even when no + `bitcoin.conf` key changed, so a hook can be added, edited, or removed live. + A parse error keeps the last-good set — alerting that silently stopped after + a typo is the worse failure. The file must be mode 0600; it holds signing + secrets. A reload is a handover, not a restart: a hook whose stanza you did + not touch keeps its delivery task, so its queue and retry backoff survive. + Editing one hook does not discard a page that another hook was still + retrying — which matters because a status event has no replay behind it and + the detector that raised it will not raise it twice. +- Per-hook counters are exported (`satd_alertwebhook_delivered_total`, + `_failed_attempts_total`, `_dropped_total`, `_queue_depth`, + `_last_success_age_seconds`), and a node with no alertfile exports none of + them. + +**`reorgwebhook=` is absorbed, not replaced.** The existing keys keep working +and their payload is unchanged byte for byte — `ChainEvent::Reorg` does not +carry the `depth`, `fork_height`, or disconnected/reconnected lists that a +`ReorgRecord` does, so re-shaping it would have silently broken deployed +receivers. It is now served by the new dispatcher, which also moves that +outbound HTTP off the consensus runtime, where it should never have been. New +deployments should prefer an `alertfile` hook with `categories = ["chain"]`. + +The signature scheme is specified with cross-language test vectors so a +receiver can be verified without a running node. + ## Storage / Core compatibility ### Core v28+ XOR-obfuscated block files (`blocksxor`) diff --git a/mcp/src/context.rs b/mcp/src/context.rs index 3e72989ce..b710d52b5 100644 --- a/mcp/src/context.rs +++ b/mcp/src/context.rs @@ -28,4 +28,6 @@ pub struct McpContext { /// Health-detector readings, so `get_metrics_snapshot` renders the same /// health gauges as the HTTP scrape. `None` in tests that bypass main.rs. pub health: Option>, + /// Webhook delivery counters, for the same reason. `None` in tests. + pub webhooks: Option>, } diff --git a/mcp/src/tools/ergonomics.rs b/mcp/src/tools/ergonomics.rs index 9b0d1b1ac..9be0c11f8 100644 --- a/mcp/src/tools/ergonomics.rs +++ b/mcp/src/tools/ergonomics.rs @@ -47,6 +47,7 @@ pub fn get_metrics_snapshot(ctx: &McpContext) -> String { addr_subs: ctx.addr_subs.clone(), addr_enabled: ctx.addr_enabled, health: ctx.health.clone(), + webhooks: ctx.webhooks.clone(), }; let body = metrics_ctx.render_prometheus(); let result = json!({ @@ -82,6 +83,7 @@ pub fn get_readiness(ctx: &McpContext) -> String { // Readiness reads only chain heights; the health gauges are not // consulted, so there is nothing to thread through here. health: None, + webhooks: None, }; let (ready, reason) = match metrics_ctx.is_ready() { Ok(()) => (true, None), diff --git a/mcp/tests/tools.rs b/mcp/tests/tools.rs index 1ef2292fa..a96df16f4 100644 --- a/mcp/tests/tools.rs +++ b/mcp/tests/tools.rs @@ -59,6 +59,7 @@ fn make_test_ctx() -> (McpContext, tempfile::TempDir) { addr_enabled: false, addr_subs: None, health: None, + webhooks: None, }; (ctx, dir) } @@ -616,6 +617,7 @@ mod mining { addr_enabled: false, addr_subs: None, health: None, + webhooks: None, }; let result = mine::generate_blocks(&ctx, 1, REGTEST_ADDR); diff --git a/node/src/events/publisher.rs b/node/src/events/publisher.rs index 7e9a3cb6f..39bc4b9cc 100644 --- a/node/src/events/publisher.rs +++ b/node/src/events/publisher.rs @@ -204,6 +204,16 @@ impl EventPublisher { self.out.subscribe() } + /// How many receivers are currently subscribed to the envelope stream. + /// + /// A published event reaches exactly these; anything published while this + /// is zero is not buffered anywhere and cannot be recovered. That makes it + /// the observable a handover has to keep above zero — see + /// `satd::alert::AlertReloader::apply`. + pub fn subscriber_count(&self) -> usize { + self.out.receiver_count() + } + /// Edge identity stamped on every envelope. pub fn edge(&self) -> &EdgeIdentity { &self.edge diff --git a/node/src/events/replay.rs b/node/src/events/replay.rs index 961a9b309..6a33b1ea0 100644 --- a/node/src/events/replay.rs +++ b/node/src/events/replay.rs @@ -42,6 +42,17 @@ pub trait BlockCursorSource: Send + Sync { /// Current active-chain tip height. fn current_tip_height(&self) -> u32; + /// Whether the node is still in initial block download. + /// + /// Not used by replay itself — it is here because the webhook dispatcher + /// needs to suppress its firehose during IBD and already holds this handle; + /// threading a second chain reference through the dispatcher's five + /// constructor layers to answer one boolean is worse. Defaults to `false` + /// so test doubles and any other implementor are unaffected. + fn in_initial_block_download(&self) -> bool { + false + } + /// Active-chain block hashes for the heights in `[from, to]` (inclusive), /// height-ascending. Resolved by walking the active chain back from the tip /// **once** (O(tip − from), not O(span²)) so it is reorg-safe: it follows @@ -60,6 +71,10 @@ impl BlockCursorSource for crate::chain::state::ChainState { self.tip_height() } + fn in_initial_block_download(&self) -> bool { + self.is_initial_block_download() + } + fn active_chain_range(&self, from: u32, to: u32) -> Vec<(u32, BlockHash)> { let (tip_hash, tip_height) = self.tip_snapshot(); let hi = to.min(tip_height); diff --git a/node/src/metrics.rs b/node/src/metrics.rs index 83e6b546b..89610ed19 100644 --- a/node/src/metrics.rs +++ b/node/src/metrics.rs @@ -54,6 +54,9 @@ pub struct MetricsContext { /// `satd_disk_free_bytes` is exactly the alarm an operator must not be /// shown falsely). pub health: Option>, + /// Per-hook webhook delivery counters. `None` (or empty) when no alertfile + /// is configured, in which case the block renders nothing at all. + pub webhooks: Option>, } impl MetricsContext { @@ -337,6 +340,9 @@ impl MetricsContext { // Node-health gauges, rendered only when the detector task is running. render_health_metrics(&mut out, self.health.as_deref()); + // Webhook delivery counters, rendered only when a hook is configured. + render_webhook_metrics(&mut out, self.webhooks.as_deref()); + // Transaction-filtering policy metrics (design §10, PR 7c). Extracted to // a free function so the I8-invisibility invariant (a node with no // non-empty ruleset renders a byte-identical page) is unit-testable @@ -404,6 +410,142 @@ fn metric_sample(out: &mut String, name: &str, labels: &[(&str, &str)], value: u } } +/// Per-hook webhook delivery counters, written by the alert dispatcher in the +/// `satd` binary and rendered here. +/// +/// Lives in `node` rather than beside the dispatcher because `satd-alert` +/// depends on `node` (for the health taxonomy), so the reverse dependency +/// needed to render them from here would be a cycle. +#[derive(Debug, Default)] +pub struct HookCounters { + pub delivered: std::sync::atomic::AtomicU64, + /// Failed *attempts*, not failed events: a single event retried five times + /// before succeeding contributes five. This is the signal that an endpoint + /// is flaky even while it is ultimately keeping up. + pub failed_attempts: std::sync::atomic::AtomicU64, + /// Events never delivered: queue overflow, broadcast lag, or a permanent + /// 4xx. The dead-letter count. + pub dropped: std::sync::atomic::AtomicU64, + pub queue_depth: std::sync::atomic::AtomicU64, + /// Unix seconds of the last 2xx, or 0 if there has never been one. + pub last_success_unix: std::sync::atomic::AtomicU64, +} + +/// Registry of per-hook counters, keyed by hook id. +/// +/// A `BTreeMap` so `/metrics` renders hooks in a stable order (Prometheus does +/// not care, but a diffable scrape is worth the ordering), behind an `RwLock` +/// because a SIGHUP reload can add or remove hooks while the endpoint is being +/// scraped. +#[derive(Debug, Default)] +pub struct WebhookMetrics { + hooks: parking_lot::RwLock>>, +} + +impl WebhookMetrics { + pub fn new() -> Self { + Self::default() + } + + /// Get or create a hook's counters. Called on hook registration and on + /// every delivery, so it must not allocate on the hot path — the common + /// case is a read-lock hit. + pub fn hook(&self, id: &str) -> Arc { + if let Some(c) = self.hooks.read().get(id) { + return c.clone(); + } + let mut w = self.hooks.write(); + w.entry(id.to_string()).or_default().clone() + } + + /// Forget hooks that are no longer configured, so a removed hook's series + /// stops being exported instead of freezing at its last value forever. + pub fn retain(&self, ids: &[String]) { + self.hooks.write().retain(|k, _| ids.iter().any(|i| i == k)); + } + + pub fn is_empty(&self) -> bool { + self.hooks.read().is_empty() + } + + fn snapshot(&self) -> Vec<(String, Arc)> { + self.hooks + .read() + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .collect() + } +} + +/// Append the webhook dispatcher's per-hook counters — but only when at least +/// one hook is configured, so a node with no alertfile renders a page +/// byte-identical to a build without the dispatcher (the same invisibility +/// property the policy metrics hold). +fn render_webhook_metrics(out: &mut String, metrics: Option<&WebhookMetrics>) { + use std::sync::atomic::Ordering; + let Some(metrics) = metrics.filter(|m| !m.is_empty()) else { + return; + }; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + // Family headers once, then one labelled sample per hook. Emitting a + // header per hook (as a per-sample `metric` call in this loop does) is + // well-formed with a single webhook configured and invalid with two — the + // kind of bug that passes every test written against one hook and breaks + // an operator's whole scrape page the day they add a second. + let snapshot = metrics.snapshot(); + for (name, help, kind) in [ + ( + "satd_alertwebhook_delivered_total", + "Webhook events acknowledged (2xx) by the receiver.", + "counter", + ), + ( + "satd_alertwebhook_failed_attempts_total", + "Webhook delivery attempts that failed (retried or dropped).", + "counter", + ), + ( + "satd_alertwebhook_dropped_total", + "Webhook events never delivered (queue overflow, broadcast lag, or a permanent 4xx).", + "counter", + ), + ( + "satd_alertwebhook_queue_depth", + "Events currently queued for this webhook.", + "gauge", + ), + // Age rather than a timestamp: an alerting rule wants "no successful + // delivery in N minutes", which is awkward to express against an + // absolute epoch value. 0 before the first success, which reads as + // "fresh" — so pair it with `delivered_total > 0` in a rule. + ( + "satd_alertwebhook_last_success_age_seconds", + "Seconds since this webhook's last acknowledged delivery (0 if none yet).", + "gauge", + ), + ] { + metric_header(out, name, help, kind); + for (id, c) in &snapshot { + let labels = [("hook", id.as_str())]; + let last = c.last_success_unix.load(Ordering::Relaxed); + let value = match name { + "satd_alertwebhook_delivered_total" => c.delivered.load(Ordering::Relaxed), + "satd_alertwebhook_failed_attempts_total" => { + c.failed_attempts.load(Ordering::Relaxed) + } + "satd_alertwebhook_dropped_total" => c.dropped.load(Ordering::Relaxed), + "satd_alertwebhook_queue_depth" => c.queue_depth.load(Ordering::Relaxed), + _ if last == 0 => 0, + _ => now.saturating_sub(last), + }; + metric_sample(out, name, &labels, value); + } + } +} + /// Append the node-health gauges (§A3): tip age, free disk, and one 0/1 series /// per standing alert condition. /// @@ -796,6 +938,99 @@ mod tests { assert_eq!(scope_label(false, false), "none"); } + #[test] + fn webhook_metrics_are_valid_exposition_format_with_several_hooks() { + // Two hooks, because one hook cannot expose this: a header-per-sample + // renderer is well-formed with a single series and invalid with two. + // A strict parser rejects the entire page on the duplicate `# HELP`, + // so an operator adding a second webhook would lose every metric satd + // exports, not just these. + let m = WebhookMetrics::default(); + let _ = m.hook("pager"); + let _ = m.hook("deadman"); + let _ = m.hook("relay"); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert_one_header_per_family(&out); + // And every hook is still represented. + for id in ["pager", "deadman", "relay"] { + assert!( + out.contains(&format!("satd_alertwebhook_delivered_total{{hook=\"{id}\"}}")), + "missing series for {id}:\n{out}" + ); + } + } + + #[test] + fn webhook_metrics_invisible_until_a_hook_is_configured() { + use std::sync::atomic::Ordering; + // No registry, and an empty registry, both render nothing: a node with + // no alertfile must produce a page byte-identical to one built without + // the dispatcher. + let mut out = String::new(); + render_webhook_metrics(&mut out, None); + assert!(out.is_empty(), "{out}"); + + let m = WebhookMetrics::new(); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.is_empty(), "an empty registry must leak nothing:\n{out}"); + + // A registered hook exports its counters, labelled by id. + let c = m.hook("ops"); + c.delivered.store(7, Ordering::Relaxed); + c.dropped.store(2, Ordering::Relaxed); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.contains(r#"satd_alertwebhook_delivered_total{hook="ops"} 7"#), "{out}"); + assert!(out.contains(r#"satd_alertwebhook_dropped_total{hook="ops"} 2"#), "{out}"); + assert!(out.contains(r#"satd_alertwebhook_queue_depth{hook="ops"} 0"#), "{out}"); + } + + #[test] + fn removed_hooks_stop_being_exported() { + // Otherwise a hook deleted on SIGHUP would keep exporting its last + // values forever, and an alerting rule on it could never recover. + let m = WebhookMetrics::new(); + m.hook("old"); + m.hook("new"); + m.retain(&["new".to_string()]); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.contains(r#"hook="new""#), "{out}"); + assert!(!out.contains(r#"hook="old""#), "{out}"); + } + + #[test] + fn last_success_age_is_zero_before_the_first_delivery() { + use std::sync::atomic::Ordering; + // An age computed from an unset (0) timestamp would render as ~57 years + // and read as a catastrophic outage on a node that simply has not + // delivered anything yet. + let m = WebhookMetrics::new(); + m.hook("ops"); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!( + out.contains(r#"satd_alertwebhook_last_success_age_seconds{hook="ops"} 0"#), + "{out}" + ); + + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + m.hook("ops") + .last_success_unix + .store(now.saturating_sub(30), Ordering::Relaxed); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!( + out.contains(r#"satd_alertwebhook_last_success_age_seconds{hook="ops"} 3"#), + "expected an age around 30s:\n{out}" + ); + } + #[test] fn health_metrics_absent_without_a_detector_task() { // Rendering zeros here would show a `satd_disk_free_bytes 0` — the diff --git a/satd-alert/Cargo.toml b/satd-alert/Cargo.toml new file mode 100644 index 000000000..34efcde93 --- /dev/null +++ b/satd-alert/Cargo.toml @@ -0,0 +1,36 @@ +[package] +name = "satd-alert" +version.workspace = true +edition.workspace = true +license.workspace = true +publish.workspace = true +repository.workspace = true + +# Operator-alerting core: alertfile parsing/validation, the webhook delivery +# contract (headers + HMAC signature), and the retry policy. Deliberately +# depends on NO async runtime or HTTP client — `reqwest` and the dispatcher task +# live in the `satd` binary, so everything here is a pure function or a plain +# data type and is unit-testable without a socket. Mirrors `satd-auth`. +# +# The one heavyweight dependency is `node`, and it is deliberate: the health- +# event taxonomy (kinds, severities, category bits) has exactly one definition, +# and an alertfile's `kinds`/`categories`/`min_severity` are validated against +# it rather than against a copy that could drift. +[dependencies] +node = { path = "../node" } +sha2 = { workspace = true } +hmac = { workspace = true } +hex = { workspace = true } +serde = { workspace = true } +thiserror = { workspace = true } +tracing = { workspace = true } +# Parse-only, like satd-auth: the `display` default feature pulls an +# unpublished toml_writer and nothing here round-trips the file. +toml_edit = { version = "0.25", default-features = false, features = ["parse"] } +# The same WHATWG parser `reqwest` resolves a hook URL with. Validating against +# it rather than against a prefix check is what makes "the alertfile loaded" and +# "this hook can actually deliver" the same statement. +url = "2" + +[dev-dependencies] +tempfile = "3" diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs new file mode 100644 index 000000000..c8581461d --- /dev/null +++ b/satd-alert/src/config.rs @@ -0,0 +1,1015 @@ +//! The `alertfile=` format: parsing, validation, and permission checks. +//! +//! ```toml +//! version = 1 +//! +//! [[webhook]] +//! id = "ops" # unique; appears in headers and metric labels +//! url = "https://alerts.example/satd" +//! secret = "a-long-random-string" # required — it signs the body +//! categories = ["status"] # status | chain | mempool | heartbeat +//! kinds = ["tip_stall", "disk_low"] # optional, status only +//! min_severity = "warning" # optional, status only +//! heartbeat_interval_secs = 60 # optional dead-man ping; default off +//! ``` +//! +//! Why a file rather than flat `bitcoin.conf` keys: a hook has a secret and a +//! filter. Core's config format is +//! flat and first-wins, so several hooks cannot be expressed in it without +//! inventing an index syntax. The precedent is `authfile=`, which is the same +//! shape for the same reason — down to the 0600 permission check, since both +//! files hold secrets. +//! +//! Validation is **recognize-and-reject**, matching satd's config posture: an +//! unknown key, an unknown category, a duplicate id, or a missing secret is a +//! hard error. An accepted-but-ignored alerting rule is worse than a refused +//! one — the operator believes they are covered when they are not. + +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; + +use node::events::{StatusKind, StatusSeverity}; + +/// Per-hook outbound queue depth. Delivery is serial per hook, so this is how +/// far a hook may fall behind before events are dropped. Nothing is announced +/// to the receiver when that happens — webhooks are best-effort, and the loss +/// is visible only on `satd_alertwebhook_dropped_total`. 1024 is the same order +/// as the publisher's own replay ring: enough to ride out a receiver restart, +/// small enough that a permanently dead endpoint cannot grow memory without +/// bound. +pub const HOOK_QUEUE_CAPACITY: usize = 1024; + +/// The hook id the built-in `reorgwebhook=` dispatcher registers under. +/// +/// Reserved rather than merely documented as reserved: hook ids key the metrics +/// registry, so an alertfile hook sharing this id would be handed the *same* +/// `HookCounters` as the legacy dispatcher and silently sum two unrelated +/// series. Both would also send `X-Satd-Hook: reorg-legacy` under different +/// secrets and different contract versions, so a receiver keying its secret +/// lookup off that header would verify with the wrong key. The crate already +/// hard-errors on duplicate ids for this exact reason; this is the one +/// collision the duplicate check cannot see. +/// +/// Kept in step with `satd::alert::LEGACY_REORG_HOOK_ID` by a test there — +/// `satd-alert` cannot depend on the binary crate. +pub const RESERVED_LEGACY_REORG_ID: &str = "reorg-legacy"; + +/// Cap on a hook id's length. It becomes an HTTP header value and a Prometheus +/// label on every single delivery, so an unbounded id is paid for forever. +pub const MAX_HOOK_ID_LEN: usize = 64; + +/// The only alertfile schema version this build understands. +pub const ALERTFILE_VERSION: u64 = 1; + +#[derive(Debug, thiserror::Error)] +pub enum AlertFileError { + #[error("alertfile {path}: {source}")] + Io { + path: PathBuf, + #[source] + source: std::io::Error, + }, + #[error( + "alertfile {path} is group/world accessible (mode {mode:04o}); it holds webhook signing \ + secrets — run: chmod 600 {path}" + )] + Permissions { path: PathBuf, mode: u32 }, + /// A TOML syntax error, rendered as **message and line only**. + /// + /// `toml_edit::TomlError`'s own `Display` quotes the offending source line + /// back verbatim. That is a good diagnostic for a config file and a bad one + /// for this file: the most likely line to be mid-edit when the syntax + /// breaks is `secret = "..."` — an operator rotating a key drops the + /// closing quote — and both callers log this with `error = %e` + /// (`satd/src/main.rs`, `satd/src/reload.rs`), which would put the + /// plaintext HMAC signing key in `debug.log` and anywhere that log is + /// shipped. The whole point of `Hook`'s hand-written `Debug` below is to + /// keep that key out of logs; rendering the snippet here would defeat it + /// from a different direction. + #[error("alertfile {path}: not valid TOML at line {line}: {message}")] + Toml { + path: PathBuf, + line: usize, + message: String, + }, + #[error("alertfile {path}: {message}")] + Invalid { path: PathBuf, message: String }, +} + +impl AlertFileError { + fn invalid(path: &Path, message: impl Into) -> Self { + Self::Invalid { + path: path.to_path_buf(), + message: message.into(), + } + } +} + +/// Which event classes a hook receives, as a mask of the streaming API's +/// category bits. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct CategoryMask(pub u32); + +impl CategoryMask { + pub fn contains(self, bit: u32) -> bool { + self.0 & bit != 0 + } +} + +/// A hook's event filter: categories, then (for status events) kinds and a +/// severity floor. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct HookFilter { + pub categories: CategoryMask, + /// `None` = every kind in the subscribed categories. Only meaningful for + /// status events; a hook that filters kinds still receives whatever other + /// categories it asked for. + pub kinds: Option>, + /// `None` = no floor. Status events only. + pub min_severity: Option, +} + +impl HookFilter { + /// Whether a status event passes this filter. `category_bit` is checked by + /// the caller for non-status bodies; this is the status-specific half. + pub fn accepts_status(&self, kind: StatusKind, severity: StatusSeverity) -> bool { + if !self.categories.contains(node::events::CATEGORY_STATUS) { + return false; + } + if let Some(kinds) = &self.kinds + && !kinds.contains(&kind) + { + return false; + } + if let Some(min) = self.min_severity + && severity < min + { + return false; + } + true + } +} + +/// One configured webhook. +/// +/// `Debug` is hand-written: a derived one renders `secret` in full, and a +/// single `tracing::debug!(?hook)` added later would put a signing key in the +/// log. The crate already takes this posture for watch-set scan keys. +#[derive(Clone, PartialEq, Eq)] +pub struct Hook { + pub id: String, + pub url: String, + /// Required. An unsigned webhook invites anyone who learns the URL to + /// forge node events into the operator's alerting pipeline, and a secret + /// costs nothing. + pub secret: String, + pub filter: HookFilter, + /// Forward at most one heartbeat per interval, as a dead-man's switch + /// ("tell me when my node goes quiet"). `None` = off. The bus heartbeat is + /// 1 Hz, which no HTTP receiver wants unthrottled. + pub heartbeat_interval_secs: Option, + /// Permit a plaintext `http://` URL to a non-loopback, non-private + /// address. Off by default: webhook bodies carry chain data rather than + /// secrets, but signed-then-cleartext is still a footgun. + pub allow_insecure_http: bool, +} + +impl std::fmt::Debug for Hook { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Hook") + .field("id", &self.id) + .field("url", &self.url) + .field("secret", &"") + .field("filter", &self.filter) + .field("heartbeat_interval_secs", &self.heartbeat_interval_secs) + .field("allow_insecure_http", &self.allow_insecure_http) + .finish() + } +} + + + +/// A parsed, validated alertfile. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct AlertFile { + pub hooks: Vec, +} + +impl AlertFile { + /// Read, permission-check, parse, and validate an alertfile. + pub fn load(path: &Path) -> Result { + let file = std::fs::File::open(path).map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + check_perms(path, &file)?; + let text = std::io::read_to_string(&file).map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + Self::parse(path, &text) + } + + /// Parse and validate alertfile contents. Split from [`load`](Self::load) + /// so the rules are testable without a file on disk. + pub fn parse(path: &Path, text: &str) -> Result { + let doc: toml_edit::DocumentMut = + text.parse() + .map_err(|source: toml_edit::TomlError| AlertFileError::Toml { + path: path.to_path_buf(), + // `span()` is a byte range into `text`; turn it into a line + // number without carrying any of the text along with it. + line: source.span().map_or(0, |s| { + text[..s.start.min(text.len())].bytes().filter(|b| *b == b'\n').count() + 1 + }), + message: source.message().to_string(), + })?; + + match doc.get("version").and_then(|v| v.as_integer()) { + Some(v) if v as u64 == ALERTFILE_VERSION => {} + Some(v) => { + return Err(AlertFileError::invalid( + path, + format!("unsupported version {v} (this build understands {ALERTFILE_VERSION})"), + )); + } + None => { + return Err(AlertFileError::invalid( + path, + "missing required `version` key", + )); + } + } + + for key in doc.as_table().iter().map(|(k, _)| k) { + if !matches!(key, "version" | "webhook") { + return Err(AlertFileError::invalid( + path, + format!("unknown top-level key `{key}`"), + )); + } + } + + let mut hooks = Vec::new(); + if let Some(array) = doc.get("webhook") { + let Some(entries) = array.as_array_of_tables() else { + return Err(AlertFileError::invalid( + path, + "`webhook` must be an array of tables (`[[webhook]]`)", + )); + }; + for entry in entries { + hooks.push(parse_hook(path, entry)?); + } + } + + // An id is a hook's identity in delivery headers and in metric labels, + // so two hooks sharing one would conflate two unrelated series and make + // `X-Satd-Hook` ambiguous at the receiver. + let mut seen = BTreeSet::new(); + for h in &hooks { + if !seen.insert(h.id.clone()) { + return Err(AlertFileError::invalid( + path, + format!("duplicate webhook id `{}`", h.id), + )); + } + } + Ok(Self { hooks }) + } +} + +fn parse_hook(path: &Path, t: &toml_edit::Table) -> Result { + const KNOWN: &[&str] = &[ + "id", + "url", + "secret", + "categories", + "kinds", + "min_severity", + "heartbeat_interval_secs", + "allow_insecure_http", + ]; + for key in t.iter().map(|(k, _)| k) { + if !KNOWN.contains(&key) { + return Err(AlertFileError::invalid( + path, + format!("unknown key `{key}` in [[webhook]] (known: {})", KNOWN.join(", ")), + )); + } + } + + let id = req_str(path, t, "id")?; + if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') { + return Err(AlertFileError::invalid( + path, + format!("webhook id `{id}` must be non-empty [A-Za-z0-9_-] (it is used in headers and metric labels)"), + )); + } + if id == RESERVED_LEGACY_REORG_ID { + return Err(AlertFileError::invalid( + path, + format!( + "webhook id `{id}` is reserved for the built-in `reorgwebhook=` dispatcher; \ + pick another id" + ), + )); + } + if id.len() > MAX_HOOK_ID_LEN { + return Err(AlertFileError::invalid( + path, + format!( + "webhook id is {} characters; the limit is {MAX_HOOK_ID_LEN} (it becomes an HTTP \ + header value and a Prometheus label on every delivery)", + id.len() + ), + )); + } + let url = req_str(path, t, "url")?; + let secret = req_str(path, t, "secret")?; + if secret.is_empty() { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `secret` must not be empty — it signs every delivery"), + )); + } + + // Typed strictly, like every sibling field. `allow_insecure_http = "true"` + // is a plausible thing to write, and silently reading it as `false` would + // reject the very hook the operator just opted in for — with an error + // message about plaintext that makes no sense next to the line they added. + let allow_insecure_http = match t.get("allow_insecure_http") { + None => false, + Some(v) => v.as_bool().ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `allow_insecure_http` must be a bool (true or false)"), + ) + })?, + }; + validate_url(path, &id, &url, allow_insecure_http)?; + + let categories = parse_categories(path, &id, t)?; + let kinds = parse_kinds(path, &id, t)?; + let min_severity = match t.get("min_severity") { + None => None, + Some(v) => { + let s = v.as_str().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `min_severity` must be a string")) + })?; + Some(StatusSeverity::from_str_exact(s).ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: unknown min_severity `{s}` (info, warning, critical)"), + ) + })?) + } + }; + let heartbeat_interval_secs = match t.get("heartbeat_interval_secs") { + None => None, + Some(v) => { + let n = v.as_integer().filter(|n| *n > 0).ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `heartbeat_interval_secs` must be a positive integer"), + ) + })?; + Some(n as u64) + } + }; + if heartbeat_interval_secs.is_some() + && !categories.contains(node::events::CATEGORY_HEARTBEAT) + { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: `heartbeat_interval_secs` needs the `heartbeat` category — \ + without it there is nothing to downsample" + ), + )); + } + + Ok(Hook { + id, + url, + secret, + filter: HookFilter { + categories, + kinds, + min_severity, + }, + heartbeat_interval_secs, + allow_insecure_http, + }) +} + +fn parse_categories( + path: &Path, + id: &str, + t: &toml_edit::Table, +) -> Result { + use node::events::{CATEGORY_CHAIN, CATEGORY_HEARTBEAT, CATEGORY_MEMPOOL, CATEGORY_STATUS}; + let Some(v) = t.get("categories") else { + // A hook with no categories would receive nothing, which is never what + // anyone meant to configure. + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` is required (status, chain, mempool, heartbeat)"), + )); + }; + let arr = v.as_array().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `categories` must be an array")) + })?; + let mut mask = 0u32; + for item in arr { + let name = item.as_str().ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` entries must be strings"), + ) + })?; + mask |= match name { + "status" => CATEGORY_STATUS, + "chain" => CATEGORY_CHAIN, + "mempool" => CATEGORY_MEMPOOL, + "heartbeat" => CATEGORY_HEARTBEAT, + // Rejected rather than silently ignored: the tweak firehose is + // per-block bulk data and an HTTP receiver is the wrong consumer + // for it. Tweaks stay on the streaming API. + "tweaks" => { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: the `tweaks` category is not deliverable over webhooks \ + (it is a per-block firehose); consume it on the streaming API" + ), + )); + } + other => { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: unknown category `{other}` \ + (status, chain, mempool, heartbeat)" + ), + )); + } + }; + } + if mask == 0 { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` must not be empty"), + )); + } + Ok(CategoryMask(mask)) +} + +fn parse_kinds( + path: &Path, + id: &str, + t: &toml_edit::Table, +) -> Result>, AlertFileError> { + let Some(v) = t.get("kinds") else { + return Ok(None); + }; + let arr = v.as_array().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `kinds` must be an array")) + })?; + let mut out = BTreeSet::new(); + for item in arr { + let name = item.as_str().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `kinds` entries must be strings")) + })?; + // Validated against the live taxonomy, not a copy: a typo'd kind would + // otherwise match nothing and silently disable the hook. + let kind = StatusKind::from_str_exact(name).ok_or_else(|| { + AlertFileError::invalid( + path, + format!( + "webhook `{id}`: unknown status kind `{name}` (known: {})", + StatusKind::ALL + .iter() + .map(|k| k.as_str()) + .collect::>() + .join(", ") + ), + ) + })?; + out.insert(kind); + } + if out.is_empty() { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `kinds` must not be empty (omit it to accept every kind)"), + )); + } + Ok(Some(out)) +} + +/// Validate a hook URL against the parser that will actually resolve it. +/// +/// This parses rather than prefix-matching, because the two disagree on strings +/// an operator really types. `https://alerts.example:99999/satd`, +/// `https://[::1`, and `https://` all pass a `starts_with("https://")` test and +/// all fail `Url::parse`. Accepting one is not a cosmetic slip: the dispatcher +/// only discovers it at send time, where `reqwest` reports a builder error with +/// no status, `classify_response(None)` reads that as *transient*, and the hook +/// retries a permanently-invalid URL every five minutes forever — filling its +/// queue and dropping every real alert behind it, on a node that logged +/// `alertfile loaded` at startup. An accepted-but-undeliverable hook is the +/// "accepted-but-ignored rule" this module's posture exists to prevent. +fn validate_url( + path: &Path, + id: &str, + url: &str, + allow_insecure_http: bool, +) -> Result<(), AlertFileError> { + let parsed = url::Url::parse(url).map_err(|e| { + AlertFileError::invalid(path, format!("webhook `{id}`: url is not a valid URL: {e}")) + })?; + match parsed.scheme() { + "https" => Ok(()), + "http" => { + if allow_insecure_http || is_local_host(parsed.host()) { + Ok(()) + } else { + Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: plaintext http:// to a non-local address; use https://, \ + or set allow_insecure_http = true to accept it" + ), + )) + } + } + other => Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: url scheme must be https:// or http:// (got `{other}://`)"), + )), + } +} + +/// Whether a parsed host is loopback or RFC1918 — the cases where plaintext is +/// normal (a relay on the same host, a receiver inside a private network) and +/// demanding TLS would just push operators to set the override. +/// +/// Taking the host from `Url` rather than slicing the authority out by hand is +/// what closes the two smuggling tricks this check used to unpick manually. +/// `http://127.0.0.1:8332@evil.example/hook` has host `evil.example`, not +/// `127.0.0.1` — userinfo runs to the *last* `@`. And `\` terminates the +/// authority for a special scheme, so `http://evil.example\@127.0.0.1/hook` is +/// also `evil.example`. Both previously read as loopback, waiving the +/// `allow_insecure_http` gate and posting the signed body in cleartext to the +/// attacker's host. The parser gets both right by construction. +fn is_local_host(host: Option>) -> bool { + match host { + Some(url::Host::Domain(d)) => d == "localhost", + Some(url::Host::Ipv4(v4)) => v4.is_loopback() || v4.is_private() || v4.is_link_local(), + Some(url::Host::Ipv6(v6)) => v6.is_loopback() || v6.is_unique_local(), + None => false, + } +} + +fn req_str(path: &Path, t: &toml_edit::Table, key: &str) -> Result { + t.get(key) + .and_then(|v| v.as_str()) + .map(str::to_string) + .ok_or_else(|| { + AlertFileError::invalid(path, format!("[[webhook]] is missing required string `{key}`")) + }) +} + +/// Refuse a group/world-accessible alertfile. Checked from the **open handle**, +/// not the path, so the permissions that are validated are the ones of the file +/// actually read (no TOCTOU window against a swap). +#[cfg(unix)] +fn check_perms(path: &Path, file: &std::fs::File) -> Result<(), AlertFileError> { + use std::os::unix::fs::MetadataExt; + let meta = file.metadata().map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + let mode = meta.mode() & 0o777; + if mode & 0o077 != 0 { + return Err(AlertFileError::Permissions { + path: path.to_path_buf(), + mode, + }); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_perms(_path: &Path, _file: &std::fs::File) -> Result<(), AlertFileError> { + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn parse(text: &str) -> Result { + AlertFile::parse(Path::new("/test/alertfile"), text) + } + + const MINIMAL: &str = r#" +version = 1 +[[webhook]] +id = "ops" +url = "https://alerts.example/satd" +secret = "s3cret" +categories = ["status"] +"#; + + #[test] + fn parses_a_minimal_hook() { + let f = parse(MINIMAL).unwrap(); + assert_eq!(f.hooks.len(), 1); + let h = &f.hooks[0]; + assert_eq!(h.id, "ops"); + assert!(h.filter.categories.contains(node::events::CATEGORY_STATUS)); + assert!(h.filter.kinds.is_none()); + assert!(h.filter.min_severity.is_none()); + assert_eq!(h.heartbeat_interval_secs, None); + } + + #[test] + fn parses_filters() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "pager" +url = "https://alerts.example/satd" +secret = "s" +categories = ["status", "chain"] +kinds = ["tip_stall", "disk_low"] +min_severity = "warning" +"#, + ) + .unwrap(); + let h = &f.hooks[0]; + assert!(h.filter.categories.contains(node::events::CATEGORY_STATUS)); + assert!(h.filter.categories.contains(node::events::CATEGORY_CHAIN)); + let kinds = h.filter.kinds.as_ref().unwrap(); + assert!(kinds.contains(&StatusKind::TipStall)); + assert!(!kinds.contains(&StatusKind::PeerFloor)); + assert_eq!(h.filter.min_severity, Some(StatusSeverity::Warning)); + } + + #[test] + fn status_filter_applies_kind_and_severity_floors() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "pager" +url = "https://x.example/h" +secret = "s" +categories = ["status"] +kinds = ["tip_stall", "peer_floor"] +min_severity = "critical" +"#, + ) + .unwrap(); + let filter = &f.hooks[0].filter; + assert!(filter.accepts_status(StatusKind::TipStall, StatusSeverity::Critical)); + // Right kind, too quiet. + assert!(!filter.accepts_status(StatusKind::PeerFloor, StatusSeverity::Warning)); + // Loud enough, wrong kind. + assert!(!filter.accepts_status(StatusKind::DiskLow, StatusSeverity::Critical)); + } + + #[test] + fn a_hook_without_the_status_category_never_takes_status_events() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "chain-only" +url = "https://x.example/h" +secret = "s" +categories = ["chain"] +"#, + ) + .unwrap(); + assert!(!f.hooks[0] + .filter + .accepts_status(StatusKind::TipStall, StatusSeverity::Critical)); + } + + #[test] + fn missing_or_wrong_version_is_rejected() { + assert!(matches!( + parse("[[webhook]]\nid=\"a\"\nurl=\"https://x/h\"\nsecret=\"s\"\ncategories=[\"status\"]\n"), + Err(AlertFileError::Invalid { .. }) + )); + assert!(parse(&MINIMAL.replace("version = 1", "version = 2")).is_err()); + } + + #[test] + fn unknown_keys_are_rejected_not_ignored() { + // An accepted-but-ignored alerting rule is worse than a refused one: + // the operator believes they are covered when they are not. + // + // Both levels, placed deliberately. Appending to MINIMAL does *not* + // test the top-level check: TOML puts a trailing key inside the last + // open table, so `{MINIMAL}\nnot_a_key = 1` lands in `[[webhook]]` and + // exercises `parse_hook`'s KNOWN list twice over. The top-level loop + // needs a key written before the first `[[webhook]]` header. + let top_level = MINIMAL.replace("[[webhook]]", "not_a_key = 1\n[[webhook]]"); + let err = parse(&top_level).expect_err("a stray top-level key must be refused"); + assert!(err.to_string().contains("not_a_key"), "{err}"); + assert!(parse(&MINIMAL.replace("categories = [\"status\"]", "categories = [\"status\"]\nkindz = [\"tip_stall\"]")).is_err()); + } + + /// A URL that satisfies the old `starts_with("https://")` check but that + /// `reqwest` cannot resolve. Accepting one is not cosmetic: the dispatcher + /// sees a builder error with no status, classifies it transient, and + /// retries forever without ever delivering — on a node whose startup log + /// says the alertfile loaded cleanly. + #[test] + fn a_well_prefixed_but_unparseable_url_is_refused_at_load() { + for bad in [ + "https://", // no host at all + "https://:8080/hook", // empty host with a port + "https://alerts.example:99999/hook", // port out of range + "https://[::1/hook", // unterminated IPv6 literal + "https:///hook", // template placeholder, left unedited + ] { + let f = MINIMAL.replace("https://alerts.example/satd", bad); + assert!( + parse(&f).is_err(), + "`{bad}` parses as an alertfile but cannot ever deliver", + ); + } + } + + #[test] + fn the_legacy_reorg_hook_id_is_reserved() { + // Sharing this id hands the alertfile hook the same metrics counters as + // the built-in `reorgwebhook=` dispatcher, and makes `X-Satd-Hook` + // ambiguous between two hooks with different secrets and different + // contract versions. + let f = MINIMAL.replace(r#"id = "ops""#, r#"id = "reorg-legacy""#); + let err = parse(&f).expect_err("the reserved id must be refused"); + assert!(err.to_string().contains("reserved"), "{err}"); + } + + #[test] + fn a_wrong_typed_allow_insecure_http_is_refused_not_ignored() { + // Reading `"true"` as `false` would reject the very hook the operator + // just opted in for, with an error about plaintext that makes no sense + // beside the line they added. + let f = MINIMAL.replace( + "https://alerts.example/satd", + "http://alerts.example/satd", + ) + "allow_insecure_http = \"true\"\n"; + let err = parse(&f).expect_err("a non-bool must be refused"); + assert!(err.to_string().contains("must be a bool"), "{err}"); + } + + #[test] + fn an_over_long_hook_id_is_refused() { + let f = MINIMAL.replace(r#"id = "ops""#, &format!(r#"id = "{}""#, "x".repeat(65))); + assert!(parse(&f).is_err(), "an id becomes a header value on every delivery"); + } + + /// A syntax error must not quote the offending line back, because the line + /// most likely to be mid-edit when the syntax breaks is the secret. + #[test] + fn a_toml_syntax_error_never_echoes_the_secret() { + let broken = MINIMAL.replace(r#"secret = "s3cret""#, r#"secret = "s3cret"#); + let err = parse(&broken).expect_err("unterminated string must fail"); + let rendered = err.to_string(); + assert!( + !rendered.contains("s3cret"), + "the signing secret reached the error text, and both callers log it: {rendered}", + ); + assert!(rendered.contains("line"), "should still locate the error: {rendered}"); + } + + #[test] + fn unknown_categories_and_kinds_are_rejected() { + assert!(parse(&MINIMAL.replace("\"status\"", "\"blocks\"")).is_err()); + let with_kind = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nkinds = [\"tip_stal\"]", + ); + assert!(parse(&with_kind).is_err(), "a typo'd kind must not silently match nothing"); + } + + #[test] + fn tweaks_category_is_rejected_with_a_pointer_to_the_streaming_api() { + let err = parse(&MINIMAL.replace("\"status\"", "\"tweaks\"")).unwrap_err(); + assert!( + err.to_string().contains("streaming API"), + "the error should say where tweaks are consumed: {err}" + ); + } + + #[test] + fn secret_is_required_and_must_be_non_empty() { + assert!(parse(&MINIMAL.replace("secret = \"s3cret\"", "")).is_err()); + assert!(parse(&MINIMAL.replace("s3cret", "")).is_err()); + } + + #[test] + fn duplicate_ids_are_rejected() { + // Two hooks sharing an id would be indistinguishable to a receiver. + let two = format!("{MINIMAL}\n[[webhook]]\nid = \"ops\"\nurl = \"https://b.example/h\"\nsecret = \"t\"\ncategories = [\"chain\"]\n"); + let err = parse(&two).unwrap_err(); + assert!(err.to_string().contains("duplicate"), "{err}"); + } + + #[test] + fn ids_are_restricted_to_header_and_label_safe_characters() { + for bad in ["has space", "quote\"", "new\nline", ""] { + let text = MINIMAL.replace("id = \"ops\"", &format!("id = {bad:?}")); + assert!(parse(&text).is_err(), "id {bad:?} should be rejected"); + } + } + + #[test] + fn plaintext_http_is_gated_unless_local_or_opted_in() { + let remote = MINIMAL.replace("https://alerts.example/satd", "http://alerts.example/satd"); + assert!(parse(&remote).is_err(), "plaintext to a public host needs the opt-in"); + + // Loopback and RFC1918 are fine unannotated — a relay on the same host + // or inside a private network is the normal deployment. + for local in [ + "http://127.0.0.1:9000/hook", + "http://localhost:9000/hook", + "http://10.1.2.3/hook", + "http://192.168.0.9:8443/h", + "http://[::1]:9000/hook", + ] { + let text = MINIMAL.replace("https://alerts.example/satd", local); + assert!(parse(&text).is_ok(), "{local} should be accepted"); + } + + // And the escape hatch works. + let opted = remote.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nallow_insecure_http = true", + ); + assert!(parse(&opted).is_ok()); + } + + #[test] + fn userinfo_cannot_impersonate_a_local_host() { + // The authority's host is what comes after the last `@`. Reading the + // userinfo as the host would let a config pasted from a third-party + // setup guide send signed cleartext to a public endpoint while + // *looking* like a loopback target, silently skipping the acknowledgement + // the plaintext gate exists to force. + for sneaky in [ + "http://127.0.0.1:8332@evil.example/hook", + "http://localhost@evil.example/hook", + "http://10.0.0.1@evil.example:8080/hook", + "http://user:127.0.0.1@evil.example/hook", + ] { + let text = MINIMAL.replace("https://alerts.example/satd", sneaky); + assert!( + parse(&text).is_err(), + "{sneaky} resolves to a public host and must need the opt-in" + ); + } + + // A genuine loopback target with userinfo is still local. + let text = MINIMAL.replace("https://alerts.example/satd", "http://user:pw@127.0.0.1:9000/h"); + assert!(parse(&text).is_ok(), "userinfo on a real loopback host is fine"); + } + + #[test] + fn a_backslash_cannot_smuggle_a_public_host_past_the_local_check() { + // The WHATWG URL parser (what `reqwest` resolves these with) treats `\` + // as a path separator for special schemes, so the host here is + // `evil.example` — but splitting the authority on `/` alone leaves + // `evil.example\@127.0.0.1`, whose last `@` yields a loopback address. + // Same bypass as the userinfo case, through a different separator. + // Written as TOML *literal* strings (single-quoted). A backslash in a + // basic string is an escape, so `"...\@..."` fails at the TOML layer + // and would make this test pass without ever reaching the URL check. + for sneaky in [ + r"http://evil.example\@127.0.0.1/hook", + r"http://evil.example\@10.0.0.1/hook", + r"http://evil.example\@localhost:8080/hook", + ] { + let text = format!( + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = '{sneaky}'\n\ + secret = \"s3cret\"\n\ + categories = [\"status\"]\n" + ); + assert!( + parse(&text).is_err(), + "{sneaky} resolves to a public host and must need the opt-in" + ); + } + + // A backslash in the *path* of a genuinely local target is unaffected. + let text = format!( + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = '{}'\n\ + secret = \"s3cret\"\n\ + categories = [\"status\"]\n", + r"http://127.0.0.1:9000/a\b" + ); + assert!(parse(&text).is_ok()); + } + + #[test] + fn non_http_schemes_are_rejected() { + for url in ["ftp://x/h", "file:///etc/passwd", "x.example/h", ""] { + let text = MINIMAL.replace("https://alerts.example/satd", url); + assert!(parse(&text).is_err(), "{url} should be rejected"); + } + } + + #[test] + fn heartbeat_interval_requires_the_heartbeat_category() { + // Otherwise the knob silently does nothing. + let text = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nheartbeat_interval_secs = 60", + ); + assert!(parse(&text).is_err()); + let ok = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\", \"heartbeat\"]\nheartbeat_interval_secs = 60", + ); + assert_eq!( + parse(&ok).unwrap().hooks[0].heartbeat_interval_secs, + Some(60) + ); + // Zero would mean "every heartbeat", i.e. 1 Hz of HTTP. + let zero = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\", \"heartbeat\"]\nheartbeat_interval_secs = 0", + ); + assert!(parse(&zero).is_err()); + } + + #[test] + fn debug_never_renders_the_signing_secret() { + // A derived `Debug` prints `secret` in full, so one + // `tracing::debug!(?hook)` added later would write a signing key to the + // log — and to anywhere the log is shipped. Assert on the secret's + // *value*, not on the presence of "redacted", so the test cannot pass + // by rendering both. + let f = parse( + "version = 1\n\ + [[webhook]]\n\ + id = \"pager\"\n\ + url = \"https://example.invalid/h\"\n\ + secret = \"correct-horse-battery-staple\"\n\ + categories = [\"status\"]\n", + ) + .unwrap(); + let hook_dbg = format!("{:?}", f.hooks[0]); + assert!( + !hook_dbg.contains("correct-horse-battery-staple"), + "Hook Debug leaked the secret: {hook_dbg}" + ); + // The whole file renders through the same impl. + let file_dbg = format!("{f:?}"); + assert!( + !file_dbg.contains("correct-horse-battery-staple"), + "AlertFile Debug leaked the secret: {file_dbg}" + ); + } + + #[test] + fn empty_file_with_only_a_version_is_valid() { + // "configured but no hooks" is a legitimate intermediate state during + // an edit; it must not fail the daemon. + assert!(parse("version = 1\n").unwrap().hooks.is_empty()); + } + + #[cfg(unix)] + #[test] + fn group_or_world_readable_file_is_refused() { + use std::io::Write as _; + use std::os::unix::fs::PermissionsExt as _; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("alertfile"); + let mut f = std::fs::File::create(&path).unwrap(); + f.write_all(MINIMAL.as_bytes()).unwrap(); + drop(f); + + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + assert!( + matches!(AlertFile::load(&path), Err(AlertFileError::Permissions { .. })), + "a world-readable file holding signing secrets must be refused", + ); + + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + assert!(AlertFile::load(&path).is_ok()); + } +} diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs new file mode 100644 index 000000000..3cae74628 --- /dev/null +++ b/satd-alert/src/contract.rs @@ -0,0 +1,254 @@ +//! The webhook delivery contract: header names, the idempotency key, and the +//! body signature. +//! +//! Normatively specified in `docs/api/webhooks.md`. The rules that matter: +//! +//! - The **body is the event**, byte-for-byte identical to the JSON a +//! WebSocket subscriber would receive for the same event. One documented +//! wire schema, not two — a receiver written against the streaming spec +//! parses webhook bodies with the same code. +//! - Delivery metadata rides in **headers**, never in the body. Putting the +//! attempt counter or hook id inside the JSON would change the bytes between +//! retries of the same event, which breaks both signature caching and +//! receiver-side deduplication. +//! - The signature is computed over the **raw request body**, so a receiver +//! verifies before parsing — the standard order, and the only one that is +//! safe against a parser that normalizes. + +use hmac::{Hmac, Mac}; +use sha2::Sha256; + +/// `sha256=` over the raw body, keyed by the hook's secret. Retained +/// unchanged from the shipped `reorgwebhook` so existing receivers keep working +/// when that hook is served by this dispatcher. +pub const SIGNATURE_HEADER: &str = "X-Satd-Signature"; +/// Idempotency key: `--`. Stable across retries of +/// one event, unique across restarts (the instance id is a per-process nonce). +pub const DELIVERY_HEADER: &str = "X-Satd-Delivery"; +/// Which configured hook this delivery belongs to. +pub const HOOK_HEADER: &str = "X-Satd-Hook"; +/// 1-based attempt counter. Present so a receiver can tell a retry from a +/// genuine duplicate event without keeping state. +pub const ATTEMPT_HEADER: &str = "X-Satd-Attempt"; +/// Contract version. Bumped only for a breaking change to the header set or +/// signature scheme — the *body* schema is versioned independently by the +/// event envelope's `schema_version`. +pub const WEBHOOK_VERSION_HEADER: &str = "X-Satd-Webhook-Version"; +/// Contract version for alertfile hooks: signature covers the delivery +/// metadata, not just the body, and `X-Satd-Timestamp` is present. +pub const WEBHOOK_VERSION: &str = "2"; +/// Contract version the legacy `-reorgwebhook` alias keeps emitting. +/// +/// That surface shipped before this crate existed: body-only signature, no +/// delivery id, `ReorgRecord` rather than the event envelope. Deployed +/// receivers verify it as-is, so it is frozen — the version header is how a +/// receiver tells the two apart. +pub const LEGACY_WEBHOOK_VERSION: &str = "1"; + +/// Unix seconds at which a delivery was signed. Covered by the v2 signature; +/// a receiver rejects a delivery outside [`MAX_TIMESTAMP_SKEW_SECS`]. +pub const TIMESTAMP_HEADER: &str = "X-Satd-Timestamp"; + +/// Sign a raw body with a hook secret, rendering the `X-Satd-Signature` value. +/// +/// HMAC-SHA256 over the exact bytes that go on the wire. The `sha256=` prefix +/// is part of the value (GitHub-webhook convention) so the algorithm can be +/// migrated later without a header rename. +pub fn sign_body(secret: &str, body: &[u8]) -> String { + let mut mac = as Mac>::new_from_slice(secret.as_bytes()) + .expect("HMAC accepts a key of any length"); + mac.update(body); + format!("sha256={}", hex::encode(mac.finalize().into_bytes())) +} + +/// Build the idempotency key for an event. +/// +/// All three components already exist on every envelope (`EdgeStamp.node_id`, +/// `EdgeStamp.seq`, and the publisher's per-process `instance_id`), so this +/// adds no state: it is stable across retries of one event because the stamp is +/// fixed at publish time, and unique across restarts because `instance_id` is. +pub fn delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { + format!("{node_id_hex}-{instance_id}-{seq}") +} + +/// Maximum age a receiver should accept for a v2 delivery, in seconds. +/// +/// Normative for receivers, and now enforced on the sending side too: a +/// delivery older than this is refused by a conforming receiver, so continuing +/// to retry it is guaranteed-futile work that pins the head of a serial queue. +/// The dispatcher abandons an event once it crosses this age. +/// +/// Note the bound that matters is the *total* retry span, not the 300 s cap on +/// any one backoff interval — a doubling curve reaches 600 s of cumulative age +/// around the tenth attempt. +/// +/// Wide enough to tolerate ordinary clock skew, tight enough that a captured +/// delivery is not a permanent replay token. +pub const MAX_TIMESTAMP_SKEW_SECS: u64 = 600; + +/// Build the canonical string a v2 signature covers. +/// +/// ```text +/// "2" LF LF LF LF +/// ``` +/// +/// Unambiguous without escaping because every field before the body is +/// constrained to a character set that excludes LF: the version is a literal, +/// the timestamp is decimal digits, the delivery id is hex plus `-` and an +/// optional `w`/`r` tag, and the hook id is restricted to `[A-Za-z0-9_-]` at +/// parse time. The body is last, so its content cannot be confused with a +/// preceding field however it is shaped. +pub fn v2_signing_string(timestamp: u64, delivery_id: &str, hook_id: &str, body: &[u8]) -> Vec { + let mut buf = + Vec::with_capacity(body.len() + delivery_id.len() + hook_id.len() + 32); + buf.extend_from_slice(b"2\n"); + buf.extend_from_slice(timestamp.to_string().as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(delivery_id.as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(hook_id.as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(body); + buf +} + +/// Sign a v2 delivery: HMAC-SHA256 over [`v2_signing_string`]. +/// +/// v1 (see [`sign_body`]) covers the body and nothing else, which leaves the +/// delivery id — the value the contract instructs receivers to deduplicate +/// on — unauthenticated *and* predictable, since `seq` is dense and every +/// component of the id is visible in any single captured delivery. Anyone +/// holding one valid `(body, signature)` pair could therefore replay it under +/// forged future ids, filling a receiver's dedup cache so that the genuine +/// alerts bearing those ids are discarded on arrival — while satd counts them +/// delivered and advances its cursor. Signing the id closes that, and signing a +/// timestamp bounds replay of the capture itself. +pub fn sign_v2( + secret: &str, + timestamp: u64, + delivery_id: &str, + hook_id: &str, + body: &[u8], +) -> String { + let mut mac = as Mac>::new_from_slice(secret.as_bytes()) + .expect("HMAC accepts a key of any length"); + mac.update(&v2_signing_string(timestamp, delivery_id, hook_id, body)); + format!("sha256={}", hex::encode(mac.finalize().into_bytes())) +} + +#[cfg(test)] +mod tests { + use super::*; + + // Golden vectors. These are the contract: they are reproduced verbatim in + // `docs/api/webhooks.md` and in the reference push relay's tests, so an + // independent receiver implementation can verify its HMAC without running + // a node. Changing any expected value here is a breaking protocol change + // and must come with a `WEBHOOK_VERSION` bump. + /// v2 golden vectors. Computed independently (Python `hmac`) against the + /// canonical string, not captured from this implementation's own output — + /// a vector derived from the code under test only proves the code agrees + /// with itself. Reproduced verbatim in `docs/api/webhooks.md` and in the + /// reference relay's tests. + #[test] + fn v2_signature_golden_vectors() { + let node = "ab".repeat(16); + let did = delivery_id(&node, 7, 42); + assert_eq!(did, "abababababababababababababababab-7-42"); + assert_eq!( + sign_v2("hunter2", 1_753_400_000, &did, "pager", br#"{"hello":"world"}"#), + "sha256=0dcd7bcc563327beab8a0ec4464a261288b43825415ae4c1ebdc91e79c83e031", + ); + assert_eq!( + sign_v2("hunter2", 1_753_400_000, &did, "pager", b""), + "sha256=abb61065799427bc98456c493fe12ac9adec92fd10bebbc1bd00720becb69b6c", + ); + } + + #[test] + fn v2_signature_covers_every_metadata_field() { + let node = "ab".repeat(16); + let did = delivery_id(&node, 7, 42); + let base = sign_v2("s", 1_000, &did, "pager", b"body"); + // Each of these is a field v1 left unauthenticated. Changing any one + // must change the signature, or the field is not really covered. + assert_ne!(base, sign_v2("s", 1_001, &did, "pager", b"body"), "timestamp"); + assert_ne!( + base, + sign_v2("s", 1_000, &delivery_id(&node, 7, 43), "pager", b"body"), + "delivery id — the dedup key an attacker would forge" + ); + assert_ne!(base, sign_v2("s", 1_000, &did, "other", b"body"), "hook id"); + assert_ne!(base, sign_v2("s", 1_000, &did, "pager", b"body2"), "body"); + assert_ne!(base, sign_v2("t", 1_000, &did, "pager", b"body"), "secret"); + } + + #[test] + fn v2_canonical_string_is_unambiguous() { + // The delimiter is LF and no field before the body may contain one, so + // no reshuffling of field contents can produce the same signing string. + // Concatenation without a delimiter would let ("ab","c") collide with + // ("a","bc"); this asserts it does not. + assert_ne!( + v2_signing_string(1, "ab", "c", b"x"), + v2_signing_string(1, "a", "bc", b"x"), + ); + } + + #[test] + fn signature_golden_vectors() { + for (secret, body, expected) in [ + ( + "hunter2", + r#"{"hello":"world"}"#, + "sha256=12f1ef94c239895aafefa2a6804ec6136d8c23fff17c08064cfc75b33e3fbaf5", + ), + ( + "", + "", + "sha256=b613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad", + ), + ] { + assert_eq!(sign_body(secret, body.as_bytes()), expected, "secret={secret:?}"); + } + } + + #[test] + fn signature_covers_the_exact_bytes() { + // A single-byte change must change the signature — the point of signing + // the raw body rather than a parsed/normalized form. + let a = sign_body("k", br#"{"a":1}"#); + let b = sign_body("k", br#"{"a":2}"#); + assert_ne!(a, b); + // And whitespace is not normalized away. + assert_ne!(sign_body("k", br#"{"a":1}"#), sign_body("k", br#"{"a": 1}"#)); + } + + #[test] + fn signature_is_keyed() { + assert_ne!(sign_body("k1", b"body"), sign_body("k2", b"body")); + } + + #[test] + fn delivery_ids_are_stable_and_unique() { + let node = "ab".repeat(16); + // Same event, any attempt ⇒ same id (the caller does not vary it). + assert_eq!(delivery_id(&node, 7, 42), delivery_id(&node, 7, 42)); + // Different event ⇒ different id. + assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 7, 43)); + // Same seq after a restart ⇒ different id, because the instance nonce + // changed. Without this a receiver would dedupe away real events. + assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 8, 42)); + } + + /// Ids are unique per event within a process, across every field that + /// varies: node, instance, and counter. + #[test] + fn a_delivery_id_varies_with_every_component() { + let node = "ab".repeat(16); + let other = "cd".repeat(16); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&other, 7, 1)); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&node, 8, 1)); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&node, 7, 2)); + } +} diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs new file mode 100644 index 000000000..3ce6d1cad --- /dev/null +++ b/satd-alert/src/lib.rs @@ -0,0 +1,38 @@ +//! Operator alerting: the `alertfile` webhook configuration, the delivery +//! contract, and the retry policy. +//! +//! This crate is the pure half of the alerting subsystem. It parses and +//! validates an alertfile, decides whether a given event matches a hook, +//! renders the delivery headers, signs a body, and classifies a response as +//! delivered / retryable / dead. It does **not** open sockets: the dispatcher +//! task and its `reqwest` client live in the `satd` binary, so every rule here +//! is testable without a network, and the signature contract can be pinned by +//! golden vectors that a third-party receiver implementation can check itself +//! against. +//! +//! The split mirrors `satd-auth`, and for the same reason: policy that decides +//! *whether* and *how* something is delivered belongs somewhere a test can +//! reach without standing up a daemon. +//! +//! Module layout: +//! - [`config`] — the alertfile format, its validation rules, and permission +//! checks. +//! - [`contract`] — the wire contract: header names, delivery ids, HMAC +//! signing, and the version constant. +//! - [`retry`] — response classification and the backoff curve. + +pub mod config; +pub mod contract; +pub mod retry; + +pub use config::{ + AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY, MAX_HOOK_ID_LEN, + RESERVED_LEGACY_REORG_ID, +}; +pub use contract::{ + delivery_id, sign_body, sign_v2, v2_signing_string, + ATTEMPT_HEADER, + DELIVERY_HEADER, HOOK_HEADER, LEGACY_WEBHOOK_VERSION, MAX_TIMESTAMP_SKEW_SECS, + SIGNATURE_HEADER, TIMESTAMP_HEADER, WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, +}; +pub use retry::{classify_response, retry_delay, Disposition, MAX_BACKOFF}; diff --git a/satd-alert/src/retry.rs b/satd-alert/src/retry.rs new file mode 100644 index 000000000..a726539d9 --- /dev/null +++ b/satd-alert/src/retry.rs @@ -0,0 +1,154 @@ +//! Response classification and the retry curve. +//! +//! The governing constraint is that a webhook endpoint is *outside* the +//! operator's node and can behave arbitrarily badly. Nothing here may be able +//! to block, unbound, or spin: delivery is serial per hook, a hook's queue is +//! bounded, and a failing endpoint degrades that hook's delivery and nothing +//! else. + +use std::time::Duration; + +/// What to do with an attempt's outcome. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Disposition { + /// 2xx — the receiver acked. + Delivered, + /// Transient: retry this same event after a backoff. + Retry, + /// Permanent for this event: count it and move on. + /// + /// A misconfigured receiver (wrong path → 404, bad auth → 401, a body it + /// rejects → 400) would otherwise pin the head of the queue forever and + /// convert every subsequent event into a queue-overflow drop. Skipping the + /// event loses one delivery; retrying it forever loses all of them. + Drop, +} + +/// Longest backoff between attempts. Matches the peer-reconnect curve's cap: +/// an endpoint that has been down for five minutes is being restarted or +/// redeployed, and probing it faster than that helps nobody. +pub const MAX_BACKOFF: Duration = Duration::from_secs(300); + +/// Base delay before attempt `n` (1-based): 1s · 2^(n-1), capped at +/// [`MAX_BACKOFF`]. Jitter is applied separately by [`jitter`] so this stays a +/// pure function of the attempt number and can be asserted exactly. +pub fn retry_delay(attempt: u32) -> Duration { + // `attempt` is 1-based, so the first retry waits 1s. Saturating shift keeps + // a long-dead endpoint from overflowing into a zero delay. + let secs = 1u64.checked_shl(attempt.saturating_sub(1)).unwrap_or(u64::MAX); + Duration::from_secs(secs).min(MAX_BACKOFF) +} + +/// Spread a delay by up to ±25 %, given a caller-supplied random value. +/// +/// Randomness is a parameter rather than a call to a global RNG so the curve is +/// deterministic under test. The purpose is the usual one: a node with several +/// hooks pointed at the same receiver must not retry them all in lockstep. +pub fn jitter(base: Duration, rand: u64) -> Duration { + let base_ms = base.as_millis() as u64; + if base_ms == 0 { + return base; + } + let span = base_ms / 2; // ±25 % of base + let offset = rand % span.max(1); + Duration::from_millis(base_ms.saturating_sub(span / 2).saturating_add(offset)) +} + +/// Classify a delivery attempt. +/// +/// `status` is the HTTP status if one was received; `None` means the request +/// never produced a response (connect error, TLS failure, timeout) — always +/// retryable, since it says nothing about whether the receiver would accept the +/// event. +pub fn classify_response(status: Option) -> Disposition { + match status { + None => Disposition::Retry, + Some(s) if (200..300).contains(&s) => Disposition::Delivered, + // 408 Request Timeout and 429 Too Many Requests are explicitly + // "try again" despite being 4xx; 5xx is the server admitting fault. + Some(408) | Some(429) => Disposition::Retry, + Some(s) if (500..600).contains(&s) => Disposition::Retry, + // Every other 4xx is the receiver saying this request is wrong, and it + // will be just as wrong next time. 3xx lands here too: the delivery + // client does not follow redirects (a followed redirect would carry the + // signed body to a host the alertfile never named), so a redirecting + // endpoint is a misconfiguration to fix in the alertfile, not something + // to retry into. + Some(_) => Disposition::Drop, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn success_is_delivered() { + for s in [200, 201, 202, 204, 299] { + assert_eq!(classify_response(Some(s)), Disposition::Delivered, "{s}"); + } + } + + #[test] + fn server_errors_and_transport_failures_retry() { + for s in [500, 502, 503, 504, 599] { + assert_eq!(classify_response(Some(s)), Disposition::Retry, "{s}"); + } + assert_eq!(classify_response(None), Disposition::Retry, "no response"); + } + + #[test] + fn the_two_retryable_4xx_are_retried() { + assert_eq!(classify_response(Some(408)), Disposition::Retry); + assert_eq!(classify_response(Some(429)), Disposition::Retry); + } + + #[test] + fn client_errors_are_dropped_not_retried_forever() { + // A misconfigured receiver must not pin the head of the queue and + // convert every later event into an overflow drop. + for s in [400, 401, 403, 404, 410, 422] { + assert_eq!(classify_response(Some(s)), Disposition::Drop, "{s}"); + } + } + + #[test] + fn redirects_are_dropped_because_they_are_never_followed() { + // The delivery client sets `Policy::none()`, so a 3xx arrives here + // rather than being chased to a host the alertfile never named. + for s in [301, 302, 303, 307, 308] { + assert_eq!(classify_response(Some(s)), Disposition::Drop, "{s}"); + } + } + + #[test] + fn backoff_doubles_then_caps() { + assert_eq!(retry_delay(1), Duration::from_secs(1)); + assert_eq!(retry_delay(2), Duration::from_secs(2)); + assert_eq!(retry_delay(3), Duration::from_secs(4)); + assert_eq!(retry_delay(9), Duration::from_secs(256)); + assert_eq!(retry_delay(10), MAX_BACKOFF, "capped"); + // A hook that has been failing for days must stay at the cap, not + // overflow back to something small. + assert_eq!(retry_delay(64), MAX_BACKOFF); + assert_eq!(retry_delay(u32::MAX), MAX_BACKOFF); + } + + #[test] + fn jitter_stays_within_a_quarter_of_the_base() { + let base = Duration::from_secs(8); + for r in [0u64, 1, 999, u64::MAX / 2, u64::MAX] { + let j = jitter(base, r); + assert!( + j >= Duration::from_secs(6) && j <= Duration::from_secs(10), + "jitter({base:?}, {r}) = {j:?} escaped ±25%", + ); + } + } + + #[test] + fn jitter_actually_varies() { + let base = Duration::from_secs(8); + assert_ne!(jitter(base, 0), jitter(base, 1_234_567)); + } +} diff --git a/satd/Cargo.toml b/satd/Cargo.toml index b85586a65..811831bbe 100644 --- a/satd/Cargo.toml +++ b/satd/Cargo.toml @@ -52,6 +52,7 @@ node-index = { path = "../node-index" } node-filter-index = { path = "../node-filter-index" } esplora-handlers = { path = "../esplora-handlers" } satd-auth = { path = "../satd-auth" } +satd-alert = { path = "../satd-alert" } electrum-proto = { path = "../electrum-proto" } tls-config = { path = "../tls-config" } satd-events = { path = "../events", features = ["grpc", "zmq", "ws"] } @@ -81,6 +82,10 @@ hex = { workspace = true } base64 = { workspace = true } tempfile = "3" rcgen = { workspace = true } +# Constant-time signature comparison in the webhook e2e helper: that helper is +# the most likely thing a third party copies when writing a receiver, and the +# spec requires a constant-time compare. +subtle = { workspace = true } # Streaming E2E harness (tests/e2e/streaming.rs, folded into the `e2e` test # target): a real gRPC client (tonic, # pinned to the workspace version so it matches the `satd_events::proto` diff --git a/satd/src/alert.rs b/satd/src/alert.rs new file mode 100644 index 000000000..3cae8d1ce --- /dev/null +++ b/satd/src/alert.rs @@ -0,0 +1,1323 @@ +//! The alert webhook dispatcher. +//! +//! One fan-in task reads the event bus; one delivery task per configured hook +//! owns an outbound HTTP client and a bounded queue. `satd-alert` holds the +//! rules (what matches, how it is signed, when to retry); everything here is +//! the plumbing that runs them against a real socket. +//! +//! # Invariants +//! +//! **Nothing blocks consensus.** The fan-in `try_send`s into bounded queues and +//! drops on overflow; the delivery tasks run on the isolated API runtime. There +//! is no path from a slow endpoint back to block connection — the isolation is +//! structural, not a timeout budget. (The dispatcher this replaces ran its +//! outbound HTTP on the consensus runtime.) +//! +//! **Best-effort, and deliberately so.** Nothing is persisted and nothing is +//! replayed. A full hook queue or a lagged broadcast receiver drops events, +//! moves `satd_alertwebhook_dropped_total`, and logs; a restart resumes at the +//! live head. Anything that needs to know what it missed wants the Streaming +//! Consumption API, which is the canonical integration surface and has real +//! cursors, backpressure and a bounded `RescanBlocks`. Adding durability here +//! would be a worse reimplementation of that one surface over. +//! +//! **Delivery is serial and ordered per hook.** One request in flight at a +//! time, so a receiver observes events in the order the node produced them and +//! a retry cannot be overtaken by the event behind it. +//! +//! **A dead endpoint degrades only itself.** Retries back off to a five-minute +//! ceiling and never give up on a transient failure, but a permanent 4xx skips +//! the event rather than pinning the head of the queue — a receiver returning +//! 404 forever must not convert every subsequent event into an overflow drop. + +use std::sync::Arc; +use std::sync::atomic::Ordering; + +use node::events::{EventPublisher, NodeEvent, NodeEventBody}; +use node::metrics::{HookCounters, WebhookMetrics}; +use satd_alert::{AlertFile, Hook, HOOK_QUEUE_CAPACITY}; +use tokio::sync::{mpsc, watch}; + +/// Hook id the legacy `reorgwebhook=` alias reports under, in metric labels and +/// the `X-Satd-Hook` header. Namespaced so it can never collide with an +/// operator-chosen alertfile id (`-` is allowed in ids, but this exact string +/// is documented as reserved). +pub const LEGACY_REORG_HOOK_ID: &str = "reorg-legacy"; + +/// Per-attempt HTTP timeout. Matches the shipped reorg webhook. +const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + +/// Composite stop signal: the process-wide shutdown, plus a narrower channel +/// the SIGHUP reload flips to retire one task. +/// +/// Two channels rather than one because a reload must stop *some* of the +/// dispatcher's tasks without touching the global signal every other subsystem +/// watches. The scope of `local` differs by task on purpose: a fan-in is +/// retired per *generation*, because every reload rebuilds the hook list it +/// iterates; a delivery task is retired per *hook*, because a reload that +/// leaves a hook's stanza untouched must leave its queue alone (see +/// `AlertReloader::apply`). +#[derive(Clone)] +pub struct Stop { + global: watch::Receiver, + local: watch::Receiver, +} + +impl Stop { + fn stopped(&self) -> bool { + *self.global.borrow() || *self.local.borrow() + } + + /// Resolve once either channel signals. Cancel-safe: `changed()` only + /// marks a value seen when it completes, and the loop re-checks both + /// borrows, so being dropped inside a `select!` loses nothing. + /// + /// A dropped sender counts as stopped. `changed()` returns `Err` forever + /// once the sender is gone, so ignoring it would spin this loop at 100% + /// CPU — and the reading is right anyway: nothing that could retire this + /// task still exists. + async fn wait(&mut self) { + loop { + if self.stopped() { + return; + } + tokio::select! { + r = self.global.changed() => if r.is_err() { return }, + r = self.local.changed() => if r.is_err() { return }, + } + } + } +} + +/// What a hook's delivery task receives. +enum Delivery { + /// An event to deliver, pre-rendered to the exact bytes that will be + /// signed and sent. Rendering happens once in the fan-in rather than per + /// hook so a body is never serialized differently for two receivers. + Event { + body: Arc>, + delivery_id: String, + /// When the dispatcher took this event off the bus — NOT when the + /// delivery task got round to it. + /// + /// This is what `X-Satd-Timestamp` carries, and the spec makes it the + /// only staleness signal a receiver checks. Stamping it at dequeue + /// instead meant the freshness window bounded the *retry span* rather + /// than the event's actual age: delivery is serial per hook and a + /// hard-down endpoint burns the full retry budget on each event in + /// turn, so a `tip_stall` raised at T+0 could be popped at T+2h, + /// signed with `now`, sail through the receiver's 600 s check, and + /// page on-call for a condition that raised and cleared two hours + /// earlier. Stamped here, that delivery is correctly rejected as + /// stale. + queued_at: u64, + }, +} + +struct HookChannel { + id: String, + hook: Hook, + tx: mpsc::Sender, + counters: Arc, + /// Whether the "delivery task is gone" warning has already been logged for + /// this generation. Per-generation on purpose: a reload should re-report a + /// genuinely dead hook once, not stay quiet because a retired generation + /// already said so. + reported_closed: std::sync::atomic::AtomicBool, +} + +/// Process-lived dispatcher state that must outlive any single generation. +/// +/// A reload retires the generation that accumulated a drop count, and +/// `HookCounters` is process-lived so a reload does not reset an operator's +/// counters. +/// +/// Held in a process `static` (`DISPATCHER_STATE`). Note the consequence for +/// testing: `left_ibd` is a latch, so the first test in a process to set it +/// makes every later test of the IBD gate vacuous, non-deterministically. The +/// gate is covered end-to-end rather than by unit tests for that reason; a unit +/// test of it would need this state threaded through `AlertReloader::new`. +#[derive(Default)] +struct DispatcherState { + /// Whether this node has ever been observed out of initial block download. + /// + /// `is_initial_block_download()` is the tip header's age, not a sync flag, + /// so a node whose tip stops advancing crosses back into "IBD" a day later. + /// This keeps a flapping predicate from re-suppressing a caught-up node. + /// + /// It is deliberately not load-bearing. The latch lives in this process, so + /// a node restarted while already wedged never observes a non-IBD tip and + /// never arms it — and an earlier version of this gate went permanently + /// silent in exactly that case. What makes that harmless now is the *scope* + /// of the gate: only chain events are suppressed, and chain events are by + /// definition not arriving on a node whose tip has stopped. + left_ibd: std::sync::atomic::AtomicBool, +} + +/// The process-wide instance. Tests construct their own. +static DISPATCHER_STATE: std::sync::LazyLock> = + std::sync::LazyLock::new(|| Arc::new(DispatcherState::default())); + +/// A delivery task the reloader keeps alive across reloads. +/// +/// Held by `AlertReloader` rather than by the fan-in, so a reload can decide +/// per hook whether to keep the task — and its queue of pending deliveries — +/// or retire it. +struct RunningHook { + /// The stanza this task was started for. A reload reuses the task when the + /// new stanza compares equal, and retires it when it does not. + config: Hook, + tx: mpsc::Sender, + counters: Arc, + /// Retires this hook's delivery task alone. + stop: watch::Sender, +} + +/// Start one hook's delivery task. +/// +/// Must be called from within the API runtime — the task does outbound HTTP, +/// which is exactly what must never share the consensus runtime. +fn start_hook( + hook: &Hook, + metrics: &WebhookMetrics, + global_stop: watch::Receiver, +) -> RunningHook { + let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); + let (stop_tx, stop_rx) = watch::channel(false); + let counters = metrics.hook(&hook.id); + tokio::spawn(deliver_loop( + hook.clone(), + rx, + counters.clone(), + Stop { + global: global_stop, + local: stop_rx, + }, + )); + RunningHook { + config: hook.clone(), + tx, + counters, + stop: stop_tx, + } +} + +/// Fan-in: one broadcast receiver, filtered and enqueued per hook. +/// +/// The bus receiver is created by the caller, not here. When it is created +/// relative to retiring the previous generation is the whole handover protocol; +/// see `AlertReloader::apply`. +async fn fan_in( + hooks: Vec, + mut rx: tokio::sync::broadcast::Receiver, + publisher: Arc, + block_source: Option>, + state: Arc, + mut stop: Stop, +) { + // Per-hook heartbeat downsampling state (D11): last forwarded instant. + let mut last_heartbeat: Vec> = vec![None; hooks.len()]; + + tracing::info!( + target: "alert", + hooks = hooks.len(), + "alert webhook dispatcher started", + ); + + loop { + tokio::select! { + _ = stop.wait() => break, + recv = rx.recv() => match recv { + Ok(env) => { + // Suppress the block firehose during initial block + // download. + // + // The dispatcher starts long before P2P, so without this a + // brand-new node with an alertfile POSTs its entire sync — + // one `block_connected` per historical block, for as long as + // IBD takes. The failure mode is a multi-day firehose at the + // receiver rather than anything that looks like a bug + // locally. + // + // Scoped to chain events, and that scope is what makes the + // gate safe. `is_initial_block_download()` is the tip + // header's age, not a sync flag, so a node that is fully + // caught up and then *stops* reads as "syncing" a day later + // — and a node restarted while already wedged reads that way + // from its first event, with no chance to latch. An earlier + // version gated everything on that predicate and so went + // totally dark on a stalled node: no status, no mempool, and + // watch matches destroyed outright since they have no replay + // to recover them. The one thing a stalled node is *not* + // producing is chain events, so suppressing only those costs + // nothing in that state and still stops the firehose in the + // state it was written for. + // + // Status and heartbeat therefore always pass, which they + // must: "this node is unhealthy" is exactly as true during + // IBD, and a heartbeat is a dead-man's switch — suppressing + // it during a multi-day sync makes an external watchdog + // declare a healthy node dead. + let suppressible = matches!(env.body, NodeEventBody::Chain(_)); + let syncing = suppressible + && !state.left_ibd.load(Ordering::Relaxed) + && { + let in_ibd = block_source + .as_deref() + .is_some_and(|s| s.in_initial_block_download()); + if !in_ibd { + state.left_ibd.store(true, Ordering::Relaxed); + } + in_ibd + }; + // Decide who wants this *before* rendering it. Serializing + // first meant a `BlockTweaks` envelope — hundreds of KB of + // per-block silent-payment rows — was rendered in full and + // then discarded, since tweaks are refused at parse and can + // never be in any hook's mask. + let wanted: Vec = hooks + .iter() + .enumerate() + .filter(|(i, hook)| accepts(&hook.hook, &env, &mut last_heartbeat[*i])) + .map(|(i, _)| i) + .collect(); + if wanted.is_empty() { + continue; + } + if syncing { + // Counted rather than skipped silently, so an operator + // can see on `/metrics` that the sync suppressed + // deliveries rather than the hook being broken. + for i in wanted { + hooks[i].counters.dropped.fetch_add(1, Ordering::Relaxed); + } + continue; + } + // Render once for every hook: the body a receiver verifies + // must be the bytes a WS subscriber would have seen, and + // rendering per hook risks two receivers disagreeing. + let Ok(bytes) = serde_json::to_vec(&env) else { + tracing::warn!(target: "alert", "skipping event: serialization failed"); + continue; + }; + let body = Arc::new(bytes); + // One clock read for every hook this event fans out to, so + // the same event is never stale for one receiver and fresh + // for another. + let queued_at = unix_secs(); + let delivery_id = satd_alert::delivery_id( + &hex::encode(env.stamp.node_id), + publisher.instance_id(), + env.stamp.seq, + ); + for i in wanted { + let hook = &hooks[i]; + enqueue(hook, Delivery::Event { + body: body.clone(), + delivery_id: delivery_id.clone(), + queued_at, + }); + } + } + Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => { + // The dispatcher itself fell behind the bus. Best-effort + // means those events are simply gone: the counter and this + // line are the whole record. + tracing::warn!(target: "alert", dropped = n, "alert dispatcher lagged the event bus"); + for hook in &hooks { + hook.counters.dropped.fetch_add(n, Ordering::Relaxed); + } + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => break, + }, + } + } + tracing::info!(target: "alert", "alert webhook dispatcher stopped"); +} + +/// Queue an event for one hook. +/// +/// Returns whether the event was queued. An overflow drops the event and moves +/// `satd_alertwebhook_dropped_total`; nothing is held for later and nothing is +/// synthesized into the stream. Webhooks are best-effort by design — a consumer +/// that needs to know what it missed wants the streaming API, which has real +/// cursors and backpressure. +fn enqueue(hook: &HookChannel, item: Delivery) -> bool { + match hook.tx.try_send(item) { + Ok(()) => { + hook.counters + .queue_depth + .store(queue_depth(&hook.tx) as u64, Ordering::Relaxed); + true + } + Err(mpsc::error::TrySendError::Full(_)) => { + // A receiver that cannot keep up loses events rather than + // back-pressuring the bus. Consensus is never the thing that waits. + hook.counters.dropped.fetch_add(1, Ordering::Relaxed); + false + } + Err(mpsc::error::TrySendError::Closed(_)) => { + // The delivery task is gone: it failed to build its HTTP client, or + // it panicked. Silently ignoring this makes the hook look perfectly + // healthy on `/metrics` — `dropped_total` flat, `queue_depth` flat + // — while it delivers nothing at all, forever. Count it like any + // other loss so the existing "no successful delivery in N minutes" + // rule fires. + hook.counters.dropped.fetch_add(1, Ordering::Relaxed); + // Logged once per hook, not once per event. This arm is reached on + // every event once a delivery task is gone — including for a + // generation being retired by a reload, where it is expected — and + // an unrated line here is thousands per second on a busy mempool, + // filling the very disk `disk_low` is watching. + if !hook.reported_closed.swap(true, Ordering::Relaxed) { + tracing::warn!( + target: "alert", + hook = %hook.id, + "hook delivery task is gone; events for this hook are being dropped", + ); + } + false + } + } +} + +fn queue_depth(tx: &mpsc::Sender) -> usize { + HOOK_QUEUE_CAPACITY.saturating_sub(tx.capacity()) +} + +/// Whether a hook wants this event, applying category, kind, severity, and +/// heartbeat-downsampling filters in that order. +fn accepts(hook: &Hook, env: &NodeEvent, last_heartbeat: &mut Option) -> bool { + match &env.body { + NodeEventBody::Status(s) => hook.filter.accepts_status(s.kind, s.severity), + NodeEventBody::Heartbeat { .. } => { + let Some(interval) = hook.heartbeat_interval_secs else { + return false; + }; + if !hook.filter.categories.contains(node::events::CATEGORY_HEARTBEAT) { + return false; + } + // The bus beats at 1 Hz; a dead-man's-switch receiver wants one + // ping per interval, not sixty. + let now = std::time::Instant::now(); + let due = last_heartbeat + .map(|t| now.duration_since(t).as_secs() >= interval) + .unwrap_or(true); + if due { + *last_heartbeat = Some(now); + } + due + } + // A `Lagged` body is built per-connection by the streaming carriers and + // is never published to the shared bus, so the dispatcher cannot see + // one. Treated as any other unsubscribed body rather than special-cased + // into a path that no event takes. + other => { + // Tweaks are refused at parse (never in a hook's mask), and the + // remaining bodies map to their category bit. + let bit = match other { + NodeEventBody::Mempool(_) => node::events::CATEGORY_MEMPOOL, + NodeEventBody::Chain(_) => node::events::CATEGORY_CHAIN, + _ => return false, + }; + hook.filter.categories.contains(bit) + } + } +} + +/// The HTTP client every webhook delivery goes through. +/// +/// Redirects are **not** followed. The alertfile URL is validated once, at load +/// (scheme, and a warning for a non-loopback plaintext target), and a followed +/// redirect would silently move the request — body, `X-Satd-Signature`, and the +/// hook's identity — to a host that never passed that check. The interesting +/// destinations are exactly the ones an operator cannot see: a cloud metadata +/// endpoint, an RFC1918 admin port, the node's own RPC. HTTPS does not help; +/// a 302 to `http://169.254.169.254/` is a perfectly valid response. +/// +/// A receiver that wants to move must publish a stable final URL and have the +/// operator update the alertfile. A 3xx classifies as a permanent drop, so the +/// misconfiguration shows up in the logs rather than as silent non-delivery. +fn webhook_client() -> reqwest::Result { + reqwest::Client::builder() + .timeout(REQUEST_TIMEOUT) + .redirect(reqwest::redirect::Policy::none()) + .build() +} + +/// One hook's delivery loop: serial, in-order, retrying with backoff. +async fn deliver_loop( + hook: Hook, + mut rx: mpsc::Receiver, + counters: Arc, + mut stop: Stop, +) { + let client = match webhook_client() { + Ok(c) => c, + Err(e) => { + tracing::error!(target: "alert", hook = %hook.id, error = %e, "failed to build webhook HTTP client; this hook will not deliver"); + return; + } + }; + + loop { + let item = tokio::select! { + _ = stop.wait() => return, + item = rx.recv() => match item { + Some(i) => i, + None => return, + }, + }; + let Delivery::Event { body, delivery_id, queued_at } = item; + + counters + .queue_depth + .store(rx.len() as u64, Ordering::Relaxed); + + // Signed once, outside the retry loop, and reused for every attempt: + // the signature must be stable across retries of one event (the + // attempt counter rides in a header, deliberately not in the body or + // the signed material). + // + // The timestamp is the moment the event left the bus, not the moment + // this task reached it — see `Delivery::Event::queued_at`. So the + // freshness window bounds the age of the *alert*, which is what a + // receiver actually wants to reason about, rather than the length of + // one retry chain. A 20-minute-old "disk is filling" alert is not + // worth acting on however long it spent queued versus retrying. + let signed_at = queued_at; + let signature = + satd_alert::sign_v2(&hook.secret, signed_at, &delivery_id, &hook.id, &body); + let mut attempt: u32 = 0; + loop { + attempt = attempt.saturating_add(1); + let result = client + .post(&hook.url) + .header(reqwest::header::CONTENT_TYPE, "application/json") + .header(satd_alert::SIGNATURE_HEADER, &signature) + .header(satd_alert::TIMESTAMP_HEADER, signed_at.to_string()) + .header(satd_alert::DELIVERY_HEADER, &delivery_id) + .header(satd_alert::HOOK_HEADER, &hook.id) + .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) + .header(satd_alert::WEBHOOK_VERSION_HEADER, satd_alert::WEBHOOK_VERSION) + .body(body.as_ref().clone()) + .send() + .await; + let status = match &result { + Ok(r) => Some(r.status().as_u16()), + Err(_) => None, + }; + match satd_alert::classify_response(status) { + satd_alert::Disposition::Delivered => { + counters.delivered.fetch_add(1, Ordering::Relaxed); + counters + .last_success_unix + .store(unix_secs(), Ordering::Relaxed); + break; + } + satd_alert::Disposition::Drop => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + status = ?status, + delivery = %delivery_id, + "webhook receiver rejected the delivery permanently; skipping this event", + ); + break; + } + satd_alert::Disposition::Retry => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + // A builder error is not transient. reqwest defers URL + // parsing into the builder, so a URL that satisfies the + // alertfile's checks but that the WHATWG parser rejects + // fails here with no status — which `classify_response` + // reads as a transient network problem. `validate_url` + // parses at load precisely so this cannot happen, but if + // one ever slips through, retrying it every five minutes + // forever would pin the head of a serial queue and drop + // every real alert behind it. + if result.as_ref().err().is_some_and(reqwest::Error::is_builder) { + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::error!( + target: "alert", + hook = %hook.id, + "webhook url cannot be resolved; dropping this event (fix the \ + alertfile — retrying cannot help)", + ); + break; + } + // Stop once the delivery is older than the freshness window + // the contract publishes. `signed_at` is fixed for the life + // of the event, so past this age a conforming receiver is + // required to refuse every remaining attempt. Continuing + // would be guaranteed-futile work — and if the rejection + // arrives as a 503 from a gateway rather than a 4xx, the + // event would never reach `Drop` and would pin the queue + // permanently. + if unix_secs().saturating_sub(signed_at) > satd_alert::MAX_TIMESTAMP_SKEW_SECS { + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + delivery = %delivery_id, + attempt, + "webhook delivery aged past the freshness window; giving up on it", + ); + break; + } + match result { + Ok(r) => tracing::warn!(target: "alert", hook = %hook.id, status = %r.status(), attempt, "webhook delivery failed; retrying"), + // `without_url` because a webhook URL is frequently the + // credential itself (Slack, Discord, PagerDuty) and may + // carry userinfo; reqwest's `Display` appends it + // verbatim to transport and timeout errors, so the + // plain form writes that credential to the log on every + // endpoint blip. + Err(e) => { + let e = e.without_url(); + tracing::warn!(target: "alert", hook = %hook.id, error = %e, attempt, "webhook request failed; retrying"); + } + } + let delay = satd_alert::retry::jitter( + satd_alert::retry_delay(attempt), + rand::random::(), + ); + tokio::select! { + _ = tokio::time::sleep(delay) => {} + _ = stop.wait() => return, + } + } + } + } + } +} + +fn unix_secs() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0) +} + +/// Owns the running dispatcher generation and re-spawns it on SIGHUP. +/// +/// `alertfile=` follows the `authfile=` model: the *path* is restart-only, the +/// *contents* are re-read on every SIGHUP even when no `bitcoin.conf` key +/// changed — editing a hook in place is the whole point. A parse or permission +/// error keeps the last-good dispatcher running, because the failure mode of +/// "your alerting silently stopped after a typo" is worse than the failure mode +/// of "your edit did not take effect and the log says why". +pub struct AlertReloader { + path: std::path::PathBuf, + /// The API runtime. The reload runs on the consensus runtime's signal loop, + /// so the new generation's tasks must be spawned onto the API runtime + /// explicitly rather than inherited from the caller. + api_handle: tokio::runtime::Handle, + publisher: Arc, + block_source: Option>, + metrics: Arc, + global_stop: watch::Receiver, + /// Everything a handover has to hand over. + running: parking_lot::Mutex, + /// The last successfully-applied alertfile, so a SIGHUP that did not change + /// it can be a no-op instead of destroying in-flight deliveries. + last_applied: parking_lot::Mutex>, +} + +/// The live dispatcher, as much of it as a reload has to reason about. +/// +/// Under one lock because the handover in `apply` is a sequence over these +/// fields whose *order* is the correctness argument; splitting them would turn +/// that order into an unstated convention between independent critical +/// sections. +#[derive(Default)] +struct Running { + /// Retires the fan-in of the generation currently running. + fan_in_stop: Option>, + /// Live delivery tasks by hook id, carried across reloads. + hooks: std::collections::HashMap, +} + +impl AlertReloader { + #[allow(clippy::too_many_arguments)] + pub fn new( + path: std::path::PathBuf, + api_handle: tokio::runtime::Handle, + publisher: Arc, + block_source: Option>, + metrics: Arc, + global_stop: watch::Receiver, + ) -> Self { + Self { + path, + api_handle, + publisher, + block_source, + metrics, + global_stop, + running: parking_lot::Mutex::new(Running::default()), + last_applied: parking_lot::Mutex::new(None), + } + } + + pub fn path(&self) -> &std::path::Path { + &self.path + } + + /// Load the alertfile and start a dispatcher generation, replacing any + /// generation already running. + /// + /// Returns the hook count on success. On failure the previous generation is + /// left untouched and the error is returned for the caller to log with the + /// right severity (fatal at startup, warn-and-continue on reload). + pub fn apply(&self) -> Result { + let file = AlertFile::load(&self.path)?; + let ids: Vec = file.hooks.iter().map(|h| h.id.clone()).collect(); + + // A SIGHUP that did not change the alertfile is a no-op. + // `reload_from_sighup` calls this on *every* SIGHUP, whatever key the + // operator actually edited, so without this an unrelated + // `maxconnections` edit would churn the whole dispatcher. + // + // Belt and braces rather than the only defence: the handover below + // carries each hook's delivery task across a reload when that hook's + // stanza is unchanged, so even a real edit leaves the untouched hooks' + // queues and retry backoff intact. + // + // Comparing the parsed file rather than the file bytes means + // reformatting or a comment edit is also a no-op. + // + // The one exception is a hook whose delivery task is no longer running. + // "Unchanged" is the common case for a SIGHUP — the operator is + // reloading precisely *because* something is wrong — so short-circuiting + // on it unconditionally would make the restart below unreachable on the + // only path an operator would take to trigger it, leaving "edit the + // stanza to force a mismatch" as the sole recovery. + { + let last = self.last_applied.lock(); + let dead = { + let running = self.running.lock(); + running.hooks.values().any(|h| h.tx.is_closed()) + }; + if last.as_ref() == Some(&file) && !dead { + tracing::debug!( + target: "alert", + "alertfile unchanged; keeping the running dispatcher generation", + ); + return Ok(file.hooks.len()); + } + if dead { + tracing::warn!( + target: "alert", + "at least one webhook delivery task is no longer running; \ + rebuilding the dispatcher generation to restart it", + ); + } + } + + let hook_count = file.hooks.len(); + + // ---- Handover ---------------------------------------------------- + // + // Subscribe the incoming generation to the bus here, synchronously, + // before anything is retired — and hand the receiver to the task rather + // than letting it subscribe for itself. + // + // A `broadcast::Receiver` only sees what is published after it is + // created. Subscribing inside the spawned fan-in means the subscription + // does not exist until the executor first polls that task, so every + // event published between retiring the outgoing generation and that + // first poll reaches nobody. Not delayed: gone. Status events have no + // replay by design and the detectors that raise them are edge-triggered + // against a `HealthState` that outlives the reload, so a `disk_low` + // that lands in the window is never re-raised and the page never + // arrives. The window is short but it is scheduler latency, which is + // longest exactly when the node is loaded enough to be raising alerts. + // + // Holding both subscriptions open for a moment is the safe direction: a + // bus delivery id is `node-instance-`, so the two + // generations mint the *same* id for the same event and a receiver + // deduplicating on `X-Satd-Delivery`, as the contract instructs, + // collapses them. + let bus_rx = self.publisher.subscribe(); + + let mut running = self.running.lock(); + + // Reconcile the delivery tasks. A hook whose stanza is unchanged keeps + // the task it already has, queue and retry backoff included — a reload + // is an operator editing one stanza, and it must not destroy pending + // deliveries for every *other* hook in the file. A status event has no + // replay behind it, so one sitting in backoff for an untouched hook is + // lost outright. Only hooks actually edited or removed are retired. + let mut next_hooks: std::collections::HashMap = + std::collections::HashMap::with_capacity(file.hooks.len()); + let mut channels: Vec = Vec::with_capacity(file.hooks.len()); + { + let _guard = self.api_handle.enter(); + for hook in &file.hooks { + let kept = match running.hooks.remove(&hook.id) { + // An unchanged stanza keeps its task AND its queued + // backlog — but only if that task is still alive. A + // delivery loop that returned early (its `webhook_client()` + // failed to build at startup, say) leaves a closed channel + // behind, and re-adopting it on every SIGHUP made the hook + // permanently dark: the config compares equal forever, so + // nothing ever restarted it, and the operator's only + // recoveries were a full restart or editing the stanza to + // force a mismatch. A closed sender means the task is gone, + // so fall through and start a fresh one. + Some(r) if r.config == *hook && !r.tx.is_closed() => Some(r), + Some(dead) if dead.config == *hook => { + tracing::warn!( + target: "alert", + hook = %hook.id, + "webhook delivery task is no longer running; restarting it", + ); + let _ = dead.stop.send(true); + None + } + Some(edited) => { + // This one's queue is dropped on purpose: the operator + // changed where or how it delivers, and flushing the + // backlog to the superseded endpoint is not what they + // asked for. Nothing is announced — webhooks are + // best-effort, and the loss shows up only on + // `satd_alertwebhook_dropped_total`. + let _ = edited.stop.send(true); + None + } + None => None, + }; + let running_hook = kept.unwrap_or_else(|| { + start_hook(hook, &self.metrics, self.global_stop.clone()) + }); + channels.push(HookChannel { + id: hook.id.clone(), + hook: hook.clone(), + tx: running_hook.tx.clone(), + counters: running_hook.counters.clone(), + reported_closed: std::sync::atomic::AtomicBool::new(false), + }); + next_hooks.insert(hook.id.clone(), running_hook); + } + // Whatever is left was removed from the file. + for (_, gone) in running.hooks.drain() { + let _ = gone.stop.send(true); + } + } + running.hooks = next_hooks; + + // Retire the outgoing fan-in. Its queued deliveries are not lost with + // it: the queues belong to the delivery tasks, which this handover has + // already decided the fate of, one hook at a time. + if let Some(old) = running.fan_in_stop.take() { + let _ = old.send(true); + } + + if channels.is_empty() { + // An alertfile with no hooks runs no tasks at all. + drop(bus_rx); + } else { + let (gen_tx, gen_rx) = watch::channel(false); + { + let _guard = self.api_handle.enter(); + tokio::spawn(fan_in( + channels, + bus_rx, + self.publisher.clone(), + self.block_source.clone(), + Arc::clone(&DISPATCHER_STATE), + Stop { + global: self.global_stop.clone(), + local: gen_rx, + }, + )); + } + running.fan_in_stop = Some(gen_tx); + } + drop(running); + // Stop exporting counters for hooks that are no longer configured, + // rather than freezing their series at the last value forever. + // + // The legacy reorg alias is preserved explicitly: it registers its + // counters outside the alertfile, so retaining only alertfile ids would + // evict them the first time `apply` runs — and it runs at startup — + // leaving the still-running legacy dispatcher incrementing a snapshot + // nothing renders. Reorg-webhook delivery would become permanently + // unobservable, and only on nodes that configure both. + let mut keep = ids; + keep.push(LEGACY_REORG_HOOK_ID.to_string()); + self.metrics.retain(&keep); + + *self.last_applied.lock() = Some(file); + Ok(hook_count) + } +} + +/// Deliver legacy `reorgwebhook=` records. +/// +/// Absorbed into this module so it shares the API runtime and the HTTP client +/// shape, but **the body stays the shipped `ReorgRecord` JSON, byte for byte**. +/// A `ChainEvent::Reorg` envelope does not carry `depth`, `fork_height`, or the +/// disconnected/reconnected hash lists, so switching this hook to the envelope +/// schema would silently break every deployed receiver. Operators who want the +/// envelope shape configure a new-style hook with `categories = ["chain"]`. +pub async fn legacy_reorg_dispatcher( + webhook: crate::reload::SharedWebhook, + mut rx: mpsc::Receiver, + counters: Arc, +) { + let client = match webhook_client() { + Ok(c) => c, + Err(e) => { + tracing::warn!(target: "alert", error = %e, "failed to build reorg webhook HTTP client"); + return; + } + }; + tracing::info!(target: "alert", "reorg webhook dispatcher started (legacy alias)"); + while let Some(record) = rx.recv().await { + // Re-read the live target per record so a SIGHUP that changes or + // removes the URL takes effect without a restart. The guard is dropped + // before any await. + let Some(target) = webhook.read().clone() else { + continue; + }; + let Ok(body) = serde_json::to_vec(&record) else { + tracing::warn!(target: "alert", "failed to serialize reorg record for webhook"); + continue; + }; + let mut attempt: u32 = 0; + loop { + attempt += 1; + let mut req = client + .post(&target.url) + .header(reqwest::header::CONTENT_TYPE, "application/json") + .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) + .header(satd_alert::HOOK_HEADER, LEGACY_REORG_HOOK_ID) + // v1, frozen: this surface shipped with a body-only signature, + // no delivery id, and a `ReorgRecord` body. Deployed receivers + // verify exactly that, so it does not move to the v2 contract; + // the version header is how a receiver tells them apart. + .header( + satd_alert::WEBHOOK_VERSION_HEADER, + satd_alert::LEGACY_WEBHOOK_VERSION, + ); + if let Some(secret) = &target.secret { + req = req.header(satd_alert::SIGNATURE_HEADER, satd_alert::sign_body(secret, &body)); + } + let result = req.body(body.clone()).send().await; + let status = match &result { + Ok(r) => Some(r.status().as_u16()), + Err(_) => None, + }; + match satd_alert::classify_response(status) { + satd_alert::Disposition::Delivered => { + counters.delivered.fetch_add(1, Ordering::Relaxed); + counters.last_success_unix.store(unix_secs(), Ordering::Relaxed); + break; + } + d => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + // Consumes `result` (nothing below reads it) so the error + // can be stripped of its URL before it is formatted. + match result { + Ok(r) => tracing::warn!(target: "alert", status = %r.status(), attempt, "reorg webhook returned non-2xx"), + // `without_url` for the same reason the alertfile + // dispatcher applies it: a webhook URL is frequently + // the credential itself (Slack, Discord, PagerDuty) + // and may carry userinfo, and reqwest's `Display` + // appends it verbatim to transport and timeout errors. + Err(e) => { + let e = e.without_url(); + tracing::warn!(target: "alert", error = %e, attempt, "reorg webhook request failed") + } + } + // Bounded retries, matching the shipped behavior: the legacy + // hook has no queue to fall behind on, so a failing endpoint + // is given three tries and the record is dropped. + // + // Deliberately NOT keyed on `Disposition` the way alertfile + // hooks are. The shipped dispatcher retried *any* non-2xx + // three times, so classifying 4xx as a one-shot drop here + // would quietly change behavior on a flag operators already + // depend on. (The redirect change is not reverted: the + // shipped client followed 30x, and that is an SSRF vector + // for a signed request, so 3xx now fails — documented as a + // breaking change in the release notes.) + let _ = d; + if attempt >= 3 { + counters.dropped.fetch_add(1, Ordering::Relaxed); + break; + } + tokio::time::sleep(std::time::Duration::from_millis( + 200u64 * (1 << (attempt - 1)), + )) + .await; + } + } + } + } + tracing::info!(target: "alert", "reorg webhook dispatcher stopped"); +} + +#[cfg(test)] +mod tests { + use super::*; + use node::events::{StatusEvent, StatusKind, StatusSeverity}; + + fn hook_from(toml: &str) -> Hook { + AlertFile::parse(std::path::Path::new("/test"), toml) + .expect("valid alertfile") + .hooks + .remove(0) + } + + /// The alertfile parser reserves this id so an operator hook cannot + /// collide with the built-in `reorgwebhook=` dispatcher. The two constants + /// live in different crates — `satd-alert` cannot depend on this binary — + /// so nothing but this assertion keeps them in step. If they drift, the + /// reservation silently protects the wrong string and both hooks share one + /// set of metrics counters again. + #[test] + fn the_reserved_hook_id_matches_the_one_the_parser_refuses() { + assert_eq!(LEGACY_REORG_HOOK_ID, satd_alert::RESERVED_LEGACY_REORG_ID); + } + + fn status_env(kind: StatusKind, severity: StatusSeverity) -> NodeEvent { + let mut ev = StatusEvent::raised(StatusKind::TipStall, "x"); + ev.kind = kind; + ev.severity = severity; + NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Status(ev), + ) + } + + const STATUS_HOOK: &str = r#" +version = 1 +[[webhook]] +id = "ops" +url = "https://x.example/h" +secret = "s" +categories = ["status"] +min_severity = "warning" +"#; + + #[test] + fn status_filter_is_applied_per_hook() { + let hook = hook_from(STATUS_HOOK); + let mut hb = None; + assert!(accepts( + &hook, + &status_env(StatusKind::DiskLow, StatusSeverity::Critical), + &mut hb + )); + // Below the severity floor. + assert!(!accepts( + &hook, + &status_env(StatusKind::IbdComplete, StatusSeverity::Info), + &mut hb + )); + } + + #[test] + fn a_status_only_hook_does_not_receive_chain_events() { + let hook = hook_from(STATUS_HOOK); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Chain(node::chain::events::ChainEvent::BlockConnected { + hash: bitcoin::BlockHash::from_raw_hash( + bitcoin::hashes::Hash::from_byte_array([3u8; 32]), + ), + height: 5, + }), + ); + let mut hb = None; + assert!(!accepts(&hook, &env, &mut hb)); + } + + #[test] + fn heartbeats_are_downsampled_to_the_configured_interval() { + let hook = hook_from( + r#" +version = 1 +[[webhook]] +id = "deadman" +url = "https://x.example/h" +secret = "s" +categories = ["heartbeat"] +heartbeat_interval_secs = 3600 +"#, + ); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Heartbeat { uptime_ns: 1 }, + ); + let mut hb = None; + // The bus beats at 1 Hz; the first one goes through and the rest of the + // hour's worth do not. + assert!(accepts(&hook, &env, &mut hb)); + for _ in 0..100 { + assert!(!accepts(&hook, &env, &mut hb)); + } + } + + #[test] + fn a_hook_without_the_heartbeat_interval_gets_no_heartbeats() { + let hook = hook_from(STATUS_HOOK); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Heartbeat { uptime_ns: 1 }, + ); + let mut hb = None; + assert!(!accepts(&hook, &env, &mut hb)); + } + + // === Handover ========================================================== + + /// A reloader over a one-line alertfile in `dir`, plus the publisher it + /// dispatches from. + /// + /// The URL is a discard port on loopback: these tests are about what the + /// reloader does to its own tasks, not about delivery, and nothing here + /// waits on a response. + fn handover_fixture(dir: &std::path::Path) -> (AlertReloader, Arc) { + let publisher = EventPublisher::new( + node::events::EdgeIdentity::new([9; 16], None).expect("edge identity"), + 64, + ); + let (_tx, global_stop) = watch::channel(false); + // `_tx` must outlive the reloader or every `Stop` reads "sender gone" + // and the tasks retire themselves immediately. Leak it: the fixture + // owns nothing else that could hold it for the test's duration. + std::mem::forget(_tx); + let reloader = AlertReloader::new( + dir.join("alertfile.toml"), + tokio::runtime::Handle::current(), + publisher.clone(), + None, + Arc::new(WebhookMetrics::new()), + global_stop, + ); + (reloader, publisher) + } + + fn write_hooks(dir: &std::path::Path, stanzas: &str) { + use std::os::unix::fs::PermissionsExt as _; + let path = dir.join("alertfile.toml"); + std::fs::write(&path, format!("version = 1\n{stanzas}")).expect("write alertfile"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + } + + fn stanza(id: &str, categories: &str) -> String { + format!( + "\n[[webhook]]\nid = \"{id}\"\nurl = \"http://127.0.0.1:9/{id}\"\n\ + secret = \"{}\"\ncategories = [{categories}]\n", + "s".repeat(32) + ) + } + + /// `apply` must take the incoming generation's bus subscription itself, + /// not leave it to the task it spawns. + /// + /// A `broadcast::Receiver` only sees what is published after it is created. + /// If the fan-in subscribes for itself, the subscription does not exist + /// until the executor first polls it — and every event published between + /// retiring the outgoing generation and that first poll reaches nobody. + /// A status event lost there is lost for good: there is no replay, and the + /// detectors are edge-triggered against a `HealthState` that outlives the + /// reload, so the condition is never re-raised. + /// + /// This is a single-threaded runtime and nothing is awaited across the + /// second `apply`, so no spawned task has run: the subscriber count is + /// exactly what `apply` did synchronously. Deferring the subscription to + /// the task leaves it at 1 and fails here. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_subscribes_the_incoming_generation_before_retiring_the_old_one() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + // Let generation one's fan-in reach its select loop. + for _ in 0..8 { + tokio::task::yield_now().await; + } + assert_eq!( + publisher.subscriber_count(), + 1, + "one generation running should mean exactly one bus subscriber" + ); + + write_hooks( + dir.path(), + &format!( + "{}{}", + stanza("pager", "\"status\""), + stanza("ops", "\"chain\"") + ), + ); + reloader.apply().expect("second apply"); + + assert_eq!( + publisher.subscriber_count(), + 2, + "apply() must subscribe the incoming generation itself; deferring it \ + to the spawned task leaves the bus unsubscribed for as long as the \ + executor takes to poll, and events published in that window are gone" + ); + } + + /// Editing one hook must not destroy another hook's pending deliveries. + /// + /// A reload used to retire the whole generation, taking every hook's queue + /// and retry backoff with it. For chain events that is survivable — the + /// durable cursor did not advance. A status event has no replay by design, + /// so one sitting in backoff for a hook the operator never touched is + /// simply lost, and the edge-triggered detector will not raise it again. + /// + /// Identity of the `mpsc::Sender` is the observable: a carried-over hook + /// keeps the same channel, and therefore the same queue. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_carries_over_the_delivery_task_of_an_untouched_hook() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is running"); + + // Add an unrelated second hook. `pager`'s stanza is byte-identical. + write_hooks( + dir.path(), + &format!( + "{}{}", + stanza("pager", "\"status\""), + stanza("ops", "\"chain\"") + ), + ); + reloader.apply().expect("second apply"); + + let after = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is still running"); + assert!( + before.same_channel(&after), + "an untouched hook must keep its delivery task across a reload; a \ + fresh channel means its queued deliveries were destroyed" + ); + assert!( + !before.is_closed(), + "the carried-over hook's delivery task must still be running" + ); + } + + /// A hook whose delivery task has died must be restarted by the next + /// reload, not re-adopted forever. + /// + /// The keep-the-task rule compared only the stanza, so a task that exited + /// early — `webhook_client()` failing to build at startup is the realistic + /// way — left a closed channel that every subsequent SIGHUP happily + /// re-adopted, because an unchanged stanza compares equal every time. The + /// hook was dark for the life of the process and the only recoveries were a + /// restart or editing the stanza to force a mismatch. Reloading is exactly + /// what an operator does to fix things, so it must actually fix this. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_restarts_a_hook_whose_delivery_task_has_died() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader.running.lock().hooks["pager"].tx.clone(); + + // Kill this hook's delivery task the way an early return would, and let + // it observe the stop so the receiver drops and the channel closes. + reloader.running.lock().hooks["pager"] + .stop + .send(true) + .expect("stop the delivery task"); + for _ in 0..100 { + if before.is_closed() { + break; + } + tokio::task::yield_now().await; + } + assert!(before.is_closed(), "fixture must actually kill the task"); + + // Same file, same stanza — the exact case that used to re-adopt it. + reloader.apply().expect("second apply"); + let after = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is still registered"); + assert!( + !after.is_closed(), + "a reload must restart a dead delivery task, not re-adopt it", + ); + assert!( + !before.same_channel(&after), + "and that means a fresh channel, not the corpse", + ); + } + + /// The mirror: a hook the operator *did* edit is retired, so the new + /// endpoint does not inherit a backlog addressed to the old one. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_retires_the_delivery_task_of_an_edited_hook() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader.running.lock().hooks["pager"].tx.clone(); + + write_hooks(dir.path(), &stanza("pager", "\"status\", \"chain\"")); + reloader.apply().expect("second apply"); + + let after = reloader.running.lock().hooks["pager"].tx.clone(); + assert!( + !before.same_channel(&after), + "an edited hook must get a fresh delivery task" + ); + } +} diff --git a/satd/src/config.rs b/satd/src/config.rs index 050ca497c..9c80f4914 100644 --- a/satd/src/config.rs +++ b/satd/src/config.rs @@ -1008,6 +1008,10 @@ pub struct Config { /// node warning, with `%s` replaced by the warning text. `None` = no /// dispatcher. Read once at startup (restart to change), like Core. pub alert_notify: Option, + /// Path to the TOML alertfile describing outbound webhooks. `None` = no + /// dispatcher. The path is restart-only; the file's *contents* are re-read + /// on every SIGHUP (the `authfile` model). + pub alertfile: Option, /// Health detector: raise `tip_stall` after this many seconds with no /// connected block. Deliberately not gated on IBD — see /// `node::health::check_tip_stall_values`. `0` disables. SIGHUP-live. @@ -3273,6 +3277,10 @@ impl Config { .or_else(|| file_get("reorgwebhooksecret")), block_notify: cli.blocknotify.clone().or_else(|| file_get("blocknotify")), alert_notify: cli.alertnotify.clone().or_else(|| file_get("alertnotify")), + alertfile: cli + .alertfile + .clone() + .or_else(|| file_get("alertfile").map(PathBuf::from)), alert_tip_stall_seconds: cli .alert_tip_stall_seconds .or_else(|| file_get("alerttipstallseconds").and_then(|v| v.parse().ok())) @@ -5155,6 +5163,13 @@ pub struct CliArgs { )] pub reorg_webhook_secret: Option, + #[arg( + long = "alertfile", + value_name = "PATH", + help = "TOML file describing outbound alert webhooks (see the Operator Manual). Must be mode 0600 — it holds signing secrets." + )] + pub alertfile: Option, + #[arg( long = "alerttipstallseconds", value_name = "SECS", @@ -6213,7 +6228,8 @@ pub const KNOWN_CONFIG_KEYS: &[&str] = &[ "shutdownnotify", "reorgwebhook", "reorgwebhooksecret", - // Health-alert detector thresholds (A3) + // Health alerts + webhook dispatcher (A3) + "alertfile", "alerttipstallseconds", "alertdiskfreemb", "alertmempoolfullpct", @@ -7505,6 +7521,7 @@ rpcport=8332 shutdownnotify: None, reorg_webhook: None, reorg_webhook_secret: None, + alertfile: None, alert_tip_stall_seconds: None, alert_disk_free_mb: None, alert_mempool_full_pct: None, @@ -7789,6 +7806,7 @@ rpcport=8332 shutdownnotify: None, reorg_webhook: None, reorg_webhook_secret: None, + alertfile: None, alert_tip_stall_seconds: None, alert_disk_free_mb: None, alert_mempool_full_pct: None, diff --git a/satd/src/main.rs b/satd/src/main.rs index a7a972e94..fe3b62f29 100644 --- a/satd/src/main.rs +++ b/satd/src/main.rs @@ -1,3 +1,4 @@ +mod alert; mod config; mod fast_start; mod notify; @@ -685,6 +686,12 @@ async fn main() { // of claiming success. Handed to `ReloadHandles` further down. let mut reorg_webhook_handle: Option = None; + // Shared per-hook delivery counters, rendered on `/metrics`. Created here + // because both the legacy reorg hook (below) and the alertfile dispatcher + // (further down) register into it; a node with neither configured leaves it + // empty, and an empty registry renders nothing. + let webhook_metrics = Arc::new(node::metrics::WebhookMetrics::new()); + // Open reorg log + webhook dispatcher. Failure is non-fatal — the node // still runs, just without persistent reorg history (and, since reorgs are // then not recorded, without webhook delivery either). @@ -704,7 +711,18 @@ async fn main() { Arc::new(parking_lot::RwLock::new(reload::webhook_target_from(&config))); let (tx, rx) = tokio::sync::mpsc::channel::(64); reorg_log.set_webhook_sender(tx); - tokio::spawn(reorg_webhook_dispatcher(shared.clone(), rx)); + // On the API runtime, not the consensus core: this task does + // outbound HTTP to an endpoint satd does not control, and the + // previous placement (a plain `tokio::spawn` here) put that on the + // same runtime as block connection. + { + let _api_guard = api_handle.enter(); + tokio::spawn(alert::legacy_reorg_dispatcher( + shared.clone(), + rx, + webhook_metrics.hook(alert::LEGACY_REORG_HOOK_ID), + )); + } chain_state.attach_reorg_log(reorg_log); reorg_webhook_handle = Some(shared); } @@ -2164,6 +2182,42 @@ async fn main() { } }; + // Outbound alert webhooks (`alertfile=`). The dispatcher is a plain event-bus + // consumer: it starts only when a file is configured, so a node without one + // spawns no tasks and its `/metrics` page is unchanged. + // + // A bad alertfile is fatal AT STARTUP (the operator explicitly configured + // alerting; coming up "successfully" with none is the failure they would + // notice last), but only a warning on SIGHUP, where the last-good set keeps + // running. + let alert_reloader = config.alertfile.clone().map(|path| { + std::sync::Arc::new(alert::AlertReloader::new( + path, + api_handle.clone(), + event_publisher.clone(), + // Read-only active-chain access, used only to answer "is this node + // still in initial block download" for the firehose gate. + Some(chain_state.clone() as std::sync::Arc), + webhook_metrics.clone(), + shutdown_rx.clone(), + )) + }); + if let Some(reloader) = &alert_reloader { + match reloader.apply() { + Ok(hooks) => tracing::info!( + target: "alert", + hooks, + path = %reloader.path().display(), + "alertfile loaded", + ), + Err(e) => { + tracing::error!(target: "alert", error = %e, "alertfile is unusable"); + auth.cleanup(); + std::process::exit(1); + } + } + } + // Start MCP server if enabled if config.mcp { let mcp_ctx = std::sync::Arc::new(satd_mcp::McpContext { @@ -2178,6 +2232,7 @@ async fn main() { addr_enabled: config.addressindex, addr_subs: Some(address_index_concrete.subscription_registry()), health: health_state.clone(), + webhooks: Some(webhook_metrics.clone()), }); // `--mcp` only enables the feature; `--mcpport` provides the transport. @@ -2288,6 +2343,7 @@ async fn main() { addr_subs: Some(address_index_concrete.subscription_registry()), addr_enabled: config.addressindex, health: health_state.clone(), + webhooks: Some(webhook_metrics.clone()), }; let rx = shutdown_rx.clone(); api_handle.spawn(async move { @@ -3088,6 +3144,7 @@ async fn main() { rpc_auth: auth.clone(), token_store: token_store.clone(), alert_thresholds: alert_thresholds.clone(), + alert_reloader: alert_reloader.clone(), }; loop { tokio::select! { @@ -3427,92 +3484,6 @@ fn reload_tls_certificates() { tracing::info!(reloaded, failed, "TLS certificate reload complete"); } -/// Forwards reorg records to the configured HTTP webhook. Best effort — -/// failures are logged and dropped. Never blocks the consensus path: -/// the only backpressure is the channel itself, which `ReorgLog::record` -/// `try_send`s into (full queue = silent drop, counted). -/// -/// The target (URL + optional signing secret) is read from `webhook` per -/// record, so a SIGHUP reload that changes or removes it takes effect on the -/// next reorg without restarting this task. When the target is `None` the -/// record is drained and dropped (no webhook configured). -async fn reorg_webhook_dispatcher( - webhook: reload::SharedWebhook, - mut rx: tokio::sync::mpsc::Receiver, -) { - use reqwest::header::{HeaderMap, HeaderValue}; - let client = match reqwest::Client::builder() - .timeout(std::time::Duration::from_secs(10)) - .build() - { - Ok(c) => c, - Err(e) => { - tracing::warn!(error = %e, "Failed to build reorg webhook HTTP client"); - return; - } - }; - tracing::info!("Reorg webhook dispatcher started"); - while let Some(record) = rx.recv().await { - // Snapshot the live target. `None` => no webhook configured right now; - // drop the record. The guard is released before any `.await`. - let target = match webhook.read().clone() { - Some(t) => t, - None => continue, - }; - let url = target.url; - let secret = target.secret; - - let body = match serde_json::to_vec(&record) { - Ok(b) => b, - Err(e) => { - tracing::warn!(error = %e, "Failed to serialize reorg record for webhook"); - continue; - } - }; - - let mut headers = HeaderMap::new(); - headers.insert( - reqwest::header::CONTENT_TYPE, - HeaderValue::from_static("application/json"), - ); - if let Some(ref key) = secret { - let sig = hmac_sha256_hex(key.as_bytes(), &body); - if let Ok(h) = HeaderValue::from_str(&format!("sha256={}", sig)) { - headers.insert("X-Satd-Signature", h); - } - } - - // Simple retry loop: 3 attempts with jittered backoff. A failing - // webhook must not back up consensus, so we stop after 3 and move - // on to the next record. - let mut attempt = 0u32; - loop { - attempt += 1; - match client - .post(&url) - .headers(headers.clone()) - .body(body.clone()) - .send() - .await - { - Ok(r) if r.status().is_success() => break, - Ok(r) => { - tracing::warn!(status = %r.status(), attempt, "Reorg webhook returned non-2xx"); - } - Err(e) => { - tracing::warn!(error = %e, attempt, "Reorg webhook request failed"); - } - } - if attempt >= 3 { - break; - } - let backoff_ms = 200u64 * (1 << (attempt - 1)); - tokio::time::sleep(std::time::Duration::from_millis(backoff_ms)).await; - } - } - tracing::info!("Reorg webhook dispatcher stopped"); -} - /// Address-index backfill supervisor. Owns serialization (one runner /// at a time) and crash recovery (auto-respawns on startup if the /// persisted cursor state is Running and the index is enabled). @@ -3912,10 +3883,3 @@ async fn persist_sp_failed_with_cleanup( } } -fn hmac_sha256_hex(key: &[u8], msg: &[u8]) -> String { - use bitcoin::hashes::{Hash, HashEngine, Hmac, HmacEngine, sha256}; - let mut hmac: HmacEngine = HmacEngine::new(key); - hmac.input(msg); - let out = Hmac::::from_engine(hmac); - hex::encode(out.to_byte_array()) -} diff --git a/satd/src/reload.rs b/satd/src/reload.rs index 6532afe42..bcdefc9d9 100644 --- a/satd/src/reload.rs +++ b/satd/src/reload.rs @@ -181,6 +181,10 @@ pub struct ReloadHandles { /// are rotatable live (the cookie is preserved). Shared with every RPC /// listener surface, so a reload covers all of them at once. pub rpc_auth: Arc, + /// Alertfile dispatcher, present only when `alertfile=` is set. Re-read on + /// every SIGHUP independently of the rest of the config, so editing a hook + /// in place takes effect — the `TokenStore` precedent. + pub alert_reloader: Option>, /// Health-detector raise thresholds. Always present (the detector task is /// always configured, even when every threshold is 0), so a threshold /// change always has somewhere to land. @@ -416,8 +420,13 @@ const LOAD_ONLY_KEYS: &[&str] = &[ /// /// `policyfile`: the transaction-filtering ruleset — re-read, recompiled, swapped, /// and re-placed live (path change, in-place content edit, and removal all live). +/// +/// `alertfile`: the outbound webhook set — re-read, re-validated, and the +/// dispatcher generation replaced. A parse/permission error keeps the last-good +/// dispatcher running (silently losing alerting after a typo is the worse +/// failure); the *path* itself is restart-only. #[allow(dead_code)] -const HANDLER_RELOAD_KEYS: &[&str] = &["policyfile"]; +const HANDLER_RELOAD_KEYS: &[&str] = &["policyfile", "alertfile"]; /// Build the field-disposition table. Constructed at call time (once per /// reload) so the non-capturing `diff`/`apply` closures coerce to `fn` @@ -990,6 +999,25 @@ pub fn reload_from_sighup(handles: &ReloadHandles, running: &Config) -> Config { } } + // Outbound alert webhooks, same precedent: re-read the file's *contents* + // on every SIGHUP, even when the path is unchanged. A failure keeps the + // last-good dispatcher — an operator whose edit did not apply reads it in + // the log, whereas an operator whose alerting silently stopped finds out + // the next time something breaks. + if let Some(reloader) = &handles.alert_reloader { + match reloader.apply() { + Ok(hooks) => tracing::info!( + hooks, + path = %reloader.path().display(), + "alertfile reloaded" + ), + Err(e) => tracing::error!( + error = %e, + "alertfile reload failed — keeping the last-good webhook set" + ), + } + } + // Transaction-filtering policy file (§8), the `TokenStore` precedent: // re-read the external file's *contents* on every SIGHUP — even when the // path is unchanged — recompile, swap, and run the lossless re-placement diff --git a/satd/tests/e2e.rs b/satd/tests/e2e.rs index e7f95716e..15803a478 100644 --- a/satd/tests/e2e.rs +++ b/satd/tests/e2e.rs @@ -23,6 +23,10 @@ mod streaming; // against a live `satd`. #[path = "e2e/sdk.rs"] mod sdk; +// Alert webhook dispatcher E2E: a real satd delivering signed POSTs to a +// scriptable loopback receiver. +#[path = "e2e/webhooks.rs"] +mod webhooks; use common::{ DeterministicWallet, TestNode, build_signed_p2wpkh_spend_from_block1_coinbase, diff --git a/satd/tests/e2e/streaming.rs b/satd/tests/e2e/streaming.rs index c3273e044..6e6fb9c17 100644 --- a/satd/tests/e2e/streaming.rs +++ b/satd/tests/e2e/streaming.rs @@ -44,12 +44,23 @@ fn mine_addr(seed: u8) -> String { /// readiness with `reqwest::blocking`, whose internal runtime panics if dropped /// on a tokio worker thread — so the (blocking) startup runs on a blocking /// task. -async fn start_streaming_async(args: Vec<&'static str>) -> StreamingNode { +pub(crate) async fn start_streaming_async(args: Vec<&'static str>) -> StreamingNode { tokio::task::spawn_blocking(move || StreamingNode::start(&args)) .await .unwrap() } +/// Like [`start_streaming_async`] but for arguments computed at runtime (a +/// temp path, an allocated port), which cannot be `&'static str` literals. +pub(crate) async fn start_streaming_owned(args: Vec) -> StreamingNode { + tokio::task::spawn_blocking(move || { + let refs: Vec<&str> = args.iter().map(String::as_str).collect(); + StreamingNode::start(&refs) + }) + .await + .unwrap() +} + /// `getserverstatus` reports the runtime-bound gRPC + streamws listener /// addresses, and the harness discovers them. This exercises the /// `ServerListenerStatus` wiring (no client needed). @@ -288,7 +299,7 @@ async fn coinbase1(sn: &StreamingNode) -> String { .unwrap() } -async fn mine_n(sn: &StreamingNode, n: u32) { +pub(crate) async fn mine_n(sn: &StreamingNode, n: u32) { let rpc = sn.node.rpc_handle(); let addr = mine_addr(0x99); tokio::task::spawn_blocking(move || rpc.mine(n, &addr)) @@ -1792,8 +1803,16 @@ async fn grpc_watch_txid_unconfirmed_on_invalidateblock() { /// authoritative), so the value is written there — which means it must not have /// been passed on the command line as anything other than its startup default. async fn set_alert_threshold(sn: &StreamingNode, key: &str, value: &str) { + sighup_with_conf(sn, &format!("{key}={value}\n")).await; +} + +/// Replace the node's `bitcoin.conf` `[regtest]` section and SIGHUP, waiting +/// for the reload to land. Only the config *file* is re-read on SIGHUP (startup +/// CLI args stay authoritative), so a key set on the command line cannot be +/// changed this way. +pub(crate) async fn sighup_with_conf(sn: &StreamingNode, body: &str) { let conf = sn.node.datadir.join("bitcoin.conf"); - std::fs::write(&conf, format!("[regtest]\n{key}={value}\n")).expect("write bitcoin.conf"); + std::fs::write(&conf, format!("[regtest]\n{body}")).expect("write bitcoin.conf"); let pid = sn.node.process.id().to_string(); let status = tokio::task::spawn_blocking(move || { std::process::Command::new("kill") diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs new file mode 100644 index 000000000..709a696bf --- /dev/null +++ b/satd/tests/e2e/webhooks.rs @@ -0,0 +1,736 @@ +//! Alert webhook dispatcher E2E: a real `satd` POSTing to a real socket. +//! +//! The dispatcher's rules (matching, signing, retry classification) are +//! unit-tested in `satd-alert`. What only an end-to-end test can establish is +//! that a condition detected inside the daemon reaches an external HTTP +//! receiver with the right bytes and headers, that a failing receiver is +//! retried rather than dropped, and that a permanently-broken one cannot wedge +//! the queue behind it. +//! +//! The receiver is a raw TCP listener rather than an HTTP framework: the tests +//! need to script exact status codes per request and hold a connection open +//! without responding, which is easier to do with 20 lines of socket code than +//! to coax out of a server library. + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; + +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpListener; +use tokio::sync::Mutex; + +use crate::common::StreamingNode; + +/// One captured POST. +#[derive(Debug, Clone)] +pub struct Received { + pub headers: HashMap, + pub body: String, +} + +impl Received { + fn json(&self) -> serde_json::Value { + serde_json::from_str(&self.body).expect("webhook body is JSON") + } + + fn header(&self, name: &str) -> &str { + self.headers + .get(&name.to_ascii_lowercase()) + .map(String::as_str) + .unwrap_or_default() + } + + /// Verify `X-Satd-Signature` exactly as a third-party receiver would. + /// + /// v2: the HMAC covers the timestamp, the delivery id, the hook id, and the + /// body — not the body alone. Signing only the body would leave the + /// delivery id unauthenticated, and since the contract tells receivers to + /// deduplicate on that header, anyone holding one captured delivery could + /// replay it under forged ids and pre-empt the genuine alerts. + /// + /// The comparison is constant-time. It is not a real secret-dependent + /// branch here — the node is the signer — but this helper is the most + /// likely thing someone copies when writing a receiver against satd, and + /// the spec requires constant-time comparison. + fn signature_valid(&self, secret: &str) -> bool { + use subtle::ConstantTimeEq as _; + let Ok(ts) = self.header("x-satd-timestamp").parse::() else { + return false; + }; + let expected = satd_alert::sign_v2( + secret, + ts, + self.header("x-satd-delivery"), + self.header("x-satd-hook"), + self.body.as_bytes(), + ); + let got = self.header("x-satd-signature"); + got.len() == expected.len() + && got.as_bytes().ct_eq(expected.as_bytes()).into() + } +} + +/// How the mock receiver answers the next request. +#[derive(Clone, PartialEq, Eq)] +enum Behavior { + Ok, + /// Respond with this status for the first `n` requests, then 200. + FailFirst(u16, usize), + /// Answer every request with `302 Location: `. + Redirect(String), + /// Answer with `code` for any `block_connected` at one of these heights, + /// 200 for everything else. Keyed on content rather than request ordinal so + /// a script does not depend on how many events a block produces; a 5xx + /// leaves the delivery in retry, which is what an endpoint outage looks + /// like from the node's side. + FailHeights(Vec, u16), +} + +struct Inner { + behavior: Behavior, + seen: usize, + received: Vec, + /// Every request, accepted or refused — what `received` deliberately is not. + attempted: Vec, +} + +/// A scriptable HTTP receiver on loopback. +pub struct MockReceiver { + port: u16, + inner: Arc>, +} + +impl MockReceiver { + async fn start(behavior: Behavior) -> Self { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let port = listener.local_addr().unwrap().port(); + let inner = Arc::new(Mutex::new(Inner { + behavior, + seen: 0, + received: Vec::new(), + attempted: Vec::new(), + })); + let task_inner = inner.clone(); + tokio::spawn(async move { + loop { + let Ok((mut sock, _)) = listener.accept().await else { + return; + }; + let inner = task_inner.clone(); + tokio::spawn(async move { + let Some(req) = read_request(&mut sock).await else { + return; + }; + let (status, location) = { + let mut g = inner.lock().await; + g.seen += 1; + let (status, location) = match &g.behavior { + Behavior::Ok => (200, None), + Behavior::FailFirst(code, n) if g.seen <= *n => (*code, None), + Behavior::FailFirst(..) => (200, None), + Behavior::Redirect(to) => (302, Some(to.clone())), + Behavior::FailHeights(hs, code) => { + let h = req.json()["body"]["height"].as_u64(); + match h { + Some(h) if hs.contains(&h) => (*code, None), + _ => (200, None), + } + } + }; + // Only a request the receiver actually accepted counts + // as received; a 503'd attempt is a retry, not a + // delivery. Every attempt is kept separately, so a test + // can assert on what changes (and what must not) across + // the retries of one event. + g.attempted.push(req.clone()); + if status == 200 { + g.received.push(req); + } + (status, location) + }; + let reason = if status == 200 { "OK" } else { "Error" }; + let loc = location + .map(|l| format!("Location: {l}\r\n")) + .unwrap_or_default(); + let _ = sock + .write_all( + format!("HTTP/1.1 {status} {reason}\r\n{loc}Content-Length: 0\r\nConnection: close\r\n\r\n") + .as_bytes(), + ) + .await; + let _ = sock.flush().await; + }); + } + }); + Self { port, inner } + } + + pub async fn ok() -> Self { + Self::start(Behavior::Ok).await + } + + /// Reject the first `n` requests with `code`, then accept. + pub async fn failing_first(code: u16, n: usize) -> Self { + Self::start(Behavior::FailFirst(code, n)).await + } + + /// Answer `code` for `block_connected` at these heights, 200 otherwise. + pub async fn failing_heights(heights: Vec, code: u16) -> Self { + Self::start(Behavior::FailHeights(heights, code)).await + } + + /// Answer every request with a 302 pointing at `to`. + pub async fn redirecting_to(to: String) -> Self { + Self::start(Behavior::Redirect(to)).await + } + + pub fn url(&self) -> String { + format!("http://127.0.0.1:{}/hook", self.port) + } + + /// Wait for a delivery matching `pred`, up to `secs`. + async fn wait_for( + &self, + secs: u64, + pred: impl Fn(&Received) -> bool, + ) -> Received { + let deadline = std::time::Instant::now() + Duration::from_secs(secs); + loop { + if let Some(r) = self + .inner + .lock() + .await + .received + .iter() + .find(|r| pred(r)) + .cloned() + { + return r; + } + assert!( + std::time::Instant::now() < deadline, + "timed out waiting for a matching webhook delivery; got: {:?}", + self.inner.blocking_lock_bodies() + ); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + + async fn attempts(&self) -> usize { + self.inner.lock().await.seen + } + + /// Every request the receiver saw, accepted or not, in arrival order. + async fn all_attempts(&self) -> Vec { + self.inner.lock().await.attempted.clone() + } + + /// Every accepted delivery, in arrival order. + /// + /// Unused on this branch — the dispatcher tests assert on *attempts*, since + /// a refused delivery is still evidence of what was sent. The watch-hook + /// tests stacked above do use it, and each branch is linted on its own. + #[allow(dead_code)] + async fn all(&self) -> Vec { + self.inner.lock().await.received.clone() + } +} + +trait BodiesForPanic { + fn blocking_lock_bodies(&self) -> Vec; +} + +impl BodiesForPanic for Mutex { + fn blocking_lock_bodies(&self) -> Vec { + self.try_lock() + .map(|g| g.received.iter().map(|r| r.body.clone()).collect()) + .unwrap_or_default() + } +} + +/// Read one HTTP request (headers + Content-Length body) off a socket. +async fn read_request(sock: &mut tokio::net::TcpStream) -> Option { + let mut buf = Vec::new(); + let mut tmp = [0u8; 4096]; + // Headers first. + let head_end = loop { + let n = sock.read(&mut tmp).await.ok()?; + if n == 0 { + return None; + } + buf.extend_from_slice(&tmp[..n]); + if let Some(pos) = find_double_crlf(&buf) { + break pos; + } + }; + let head = String::from_utf8_lossy(&buf[..head_end]).to_string(); + let mut headers = HashMap::new(); + for line in head.lines().skip(1) { + if let Some((k, v)) = line.split_once(':') { + headers.insert(k.trim().to_ascii_lowercase(), v.trim().to_string()); + } + } + let len: usize = headers + .get("content-length") + .and_then(|v| v.parse().ok()) + .unwrap_or(0); + let mut body = buf[head_end + 4..].to_vec(); + while body.len() < len { + let n = sock.read(&mut tmp).await.ok()?; + if n == 0 { + break; + } + body.extend_from_slice(&tmp[..n]); + } + Some(Received { + headers, + body: String::from_utf8_lossy(&body).to_string(), + }) +} + +fn find_double_crlf(buf: &[u8]) -> Option { + buf.windows(4).position(|w| w == b"\r\n\r\n") +} + +const SECRET: &str = "an-operator-chosen-signing-secret"; + +/// Write (or rewrite) the alertfile for a test node. +/// +/// `nudge` exists so a test can produce a *materially different* file: the +/// dispatcher deliberately treats a SIGHUP that did not change the alertfile as +/// a no-op, so a test that needs a fresh generation — and therefore a fresh +/// catch-up from the persisted cursor — has to actually change something. It +/// sets `allow_insecure_http`, which parses under every category and is a +/// no-op for the loopback URLs these tests use, so it changes the parsed hook +/// without changing behavior. +fn write_alertfile( + dir: &tempfile::TempDir, + receiver: &MockReceiver, + id: &str, + categories: &str, + nudge: bool, +) { + use std::os::unix::fs::PermissionsExt as _; + let path = dir.path().join("alertfile.toml"); + let hb = if nudge { "allow_insecure_http = true\n" } else { "" }; + std::fs::write( + &path, + format!( + r#"version = 1 +[[webhook]] +id = "{id}" +url = "{}" +secret = "{SECRET}" +categories = [{categories}] +{hb}"#, + receiver.url() + ), + ) + .expect("write alertfile"); + // The dispatcher refuses a group/world-readable file; it holds the signing + // secret. + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); +} + +/// Write a two-hook alertfile. `cats_b` is the only thing callers vary between +/// two writes, so hook `a`'s stanza stays byte-identical across a reload. +fn write_two_hook_alertfile( + dir: &tempfile::TempDir, + a: (&MockReceiver, &str, &str), + b: (&MockReceiver, &str, &str), +) { + use std::os::unix::fs::PermissionsExt as _; + let path = dir.path().join("alertfile.toml"); + let stanza = |(r, id, cats): (&MockReceiver, &str, &str)| { + format!( + "\n[[webhook]]\nid = \"{id}\"\nurl = \"{}\"\nsecret = \"{SECRET}\"\n\ + categories = [{cats}]\n", + r.url() + ) + }; + std::fs::write( + &path, + format!("version = 1\n{}{}", stanza(a), stanza(b)), + ) + .expect("write alertfile"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); +} + +/// Start a node whose `alertfile=` points at a one-hook file delivering +/// `categories` to `receiver`. +/// +/// The alertfile *path* is restart-only (the `authfile` model — the dispatcher +/// binds to a path at startup), so it must be on the command line rather than +/// SIGHUP'd in. The file lives in its own temp dir, which the returned guard +/// keeps alive for the test's duration. +async fn node_with_hook( + receiver: &MockReceiver, + id: &str, + categories: &str, + extra: Vec, +) -> (StreamingNode, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("tempdir"); + write_alertfile(&dir, receiver, id, categories, false); + let path = dir.path().join("alertfile.toml"); + + let mut args = vec![format!("--alertfile={}", path.display())]; + args.extend(extra); + let sn = crate::streaming::start_streaming_owned(args).await; + (sn, dir) +} + +// =========================================================================== + +/// A detected condition reaches the receiver as a correctly-signed POST whose +/// body is the same JSON a streaming subscriber would have received. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_delivers_a_signed_status_event() { + let receiver = MockReceiver::ok().await; + let (sn, _dir) = node_with_hook(&receiver, "ops", "\"status\"", vec![]).await; + + // Drive `disk_low` to a known-cleared state first, so the raise below is + // guaranteed to be an edge. + // + // At the 10 GiB default this test otherwise depends on the host's free + // space: on a machine under the floor the condition is already raised + // before the dispatcher attaches, the raise SIGHUP hits `raise_if_new`'s + // no-op, and the whole v2-signature guard dies as a bare 60 s timeout with + // nothing to say why. A 1 MiB floor is below any filesystem that can hold a + // datadir, so this clears on every host. It has to go through the conf file + // rather than the command line — a CLI value wins over the conf, so a CLI + // threshold would make every later SIGHUP retune a no-op. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + + // Raise `disk_low` *after* the dispatcher is attached, by moving the + // threshold live rather than starting with it tripped: a status event is + // not replayable, so a condition raised during startup would never reach a + // hook that had not finished subscribing. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + + assert!( + got.signature_valid(SECRET), + "X-Satd-Signature must verify over the v2 signing string", + ); + assert_eq!(got.header("x-satd-hook"), "ops"); + assert_eq!(got.header("x-satd-webhook-version"), "2"); + assert_eq!(got.header("x-satd-attempt"), "1"); + assert!( + !got.header("x-satd-delivery").is_empty(), + "an idempotency key is always present", + ); + // The timestamp is part of the signed material, so a receiver can bound + // replay of a captured delivery. + let ts: u64 = got + .header("x-satd-timestamp") + .parse() + .expect("X-Satd-Timestamp is present and numeric"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + assert!(now.abs_diff(ts) < 300, "timestamp should be recent, got {ts} vs {now}"); + // Tampering with any signed field must invalidate the signature. The + // delivery id is the one that matters most: the contract tells receivers to + // deduplicate on it, so if it were unsigned an attacker holding one valid + // delivery could pre-poison a receiver's dedup cache against future alerts. + let mut forged = got.clone(); + forged + .headers + .insert("x-satd-delivery".into(), format!("{}-forged", got.header("x-satd-delivery"))); + assert!( + !forged.signature_valid(SECRET), + "the delivery id must be covered by the signature", + ); + assert_eq!(got.header("content-type"), "application/json"); + + let body = got.json(); + assert_eq!(body["body"]["kind"], "disk_low", "body: {body}"); + assert_eq!(body["body"]["state"], "raised"); + // The same envelope a WS subscriber sees — schema version and stamp too. + assert_eq!(body["schema_version"], 1); + assert!(body["stamp"]["seq"].is_number()); +} + +/// A receiver that is down comes back to find the event still waiting: the +/// dispatcher retries with backoff instead of dropping. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_retries_until_the_receiver_recovers() { + // 503 the first two attempts, then accept. Backoff is 1s then 2s, so the + // delivery lands a few seconds in. + let receiver = MockReceiver::failing_first(503, 2).await; + let (sn, _dir) = node_with_hook(&receiver, "flaky", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 1).await; + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "chain") + .await; + assert!(got.signature_valid(SECRET)); + // The attempt counter proves this was a retry rather than a fresh event — + // and it is what lets a receiver tell the two apart without keeping state. + let attempt: u32 = got.header("x-satd-attempt").parse().unwrap_or(0); + assert!(attempt >= 3, "expected the 3rd attempt to succeed, got {attempt}"); + assert!(receiver.attempts().await >= 3); + + // One event is signed exactly once, and every attempt carries that same + // signature, timestamp, and delivery id — only `X-Satd-Attempt` varies. + // + // This is the invariant, not an implementation detail. Re-signing per + // attempt would refresh `X-Satd-Timestamp`, which is what makes a delivery + // age out of the receiver's freshness window while it is still being + // retried — deliberate, since a 20-minute-old alert is not worth acting on. + // Asserting only on the accepted attempt (as this test used to) passes just + // as happily with the signing moved inside the retry loop. + let attempts = receiver.all_attempts().await; + let chain: Vec<_> = attempts + .iter() + .filter(|r| r.json()["body"]["category"] == "chain") + .collect(); + assert!(chain.len() >= 3, "expected 3 attempts at one event, got {}", chain.len()); + let first = chain[0]; + for (i, r) in chain.iter().enumerate() { + assert_eq!( + r.header("x-satd-timestamp"), + first.header("x-satd-timestamp"), + "attempt {} re-stamped the timestamp; the event must be signed once", + i + 1 + ); + assert_eq!(r.header("x-satd-signature"), first.header("x-satd-signature")); + assert_eq!(r.header("x-satd-delivery"), first.header("x-satd-delivery")); + assert_eq!(r.body, first.body); + } + // ...and the attempt counter is the one thing that does move. + assert_ne!(chain[0].header("x-satd-attempt"), chain[2].header("x-satd-attempt")); +} + +/// A SIGHUP that did not touch the alertfile must not destroy a queued status +/// event. +/// +/// `reload_from_sighup` calls `AlertReloader::apply()` on *every* SIGHUP, +/// whatever key the operator actually edited, and retiring a generation drops +/// everything queued in it. For chain events that is survivable — the cursor +/// did not advance, so the next generation's catch-up re-queues them. A status +/// event has no replay by design, and the detectors are edge-triggered against +/// a `HealthState` that outlives the reload, so a `disk_low` sitting in retry +/// backoff when the operator SIGHUPs to change `maxconnections` would be +/// dropped, never replayed, and never re-raised. The page simply never arrives. +/// +/// The receiver 503s the first two attempts, so the event is provably still +/// in the dispatcher — mid-backoff — when the unrelated SIGHUP lands. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn an_unrelated_sighup_does_not_destroy_a_queued_status_event() { + // Refuse the first four attempts. Backoff is 1s, 2s, 4s, 8s, so the event + // stays in the dispatcher for ~15s — a wide window for the unrelated reload + // below to land inside. Too few refusals and the delivery could succeed + // before the SIGHUP, leaving the test green without ever exercising the + // reload path. + let receiver = MockReceiver::failing_first(503, 4).await; + let (sn, _dir) = node_with_hook(&receiver, "pager", "\"status\"", vec![]).await; + + // Known-cleared first, so the raise is an edge on any host (see + // `webhook_delivers_a_signed_status_event` for why this goes through the + // conf file rather than the command line). + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + + // Raise `disk_low`. The detectors re-evaluate on a 15s poll, so the raise + // does not follow the SIGHUP immediately — wait for the delivery to have + // been attempted rather than assuming a fixed delay. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + let deadline = std::time::Instant::now() + Duration::from_secs(45); + while receiver.attempts().await == 0 { + assert!( + std::time::Instant::now() < deadline, + "the status event was never attempted; the raise did not reach the hook" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + } + + // Now an unrelated reload, while that delivery is still being retried. The + // alertfile is untouched; only a daemon key changes. The threshold is + // carried over so the condition stays raised — a cleared-and-re-raised + // alert would produce a *new* event and mask the loss of the old one. + crate::streaming::sighup_with_conf( + &sn, + "alertdiskfreemb=17592186044416\nmaxconnections=42\n", + ) + .await; + + // The original event must still land. + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + assert_eq!(got.json()["body"]["kind"], "disk_low"); + assert!(got.signature_valid(SECRET)); +} + +/// Editing one hook must not destroy a *different* hook's queued event. +/// +/// The companion to `an_unrelated_sighup_does_not_destroy_a_queued_status_event`, +/// and the harder half. That one is protected by the unchanged-file early +/// return in `apply`; here the alertfile genuinely changes, so the reload runs +/// the whole handover. A reload that rebuilt every delivery task would take +/// `pager`'s in-flight `disk_low` down with it even though the operator only +/// touched `ops` — and a status event has no replay, so it is lost outright and +/// the edge-triggered detector never raises it again. +/// +/// `pager` 503s the first four attempts, so the event is provably still in its +/// queue, mid-backoff, when the reload lands. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn editing_one_hook_does_not_destroy_another_hooks_queued_event() { + let pager = MockReceiver::failing_first(503, 4).await; + let ops = MockReceiver::ok().await; + + let dir = tempfile::tempdir().expect("tempdir"); + write_two_hook_alertfile( + &dir, + (&pager, "pager", "\"status\""), + (&ops, "ops", "\"status\""), + ); + let path = dir.path().join("alertfile.toml"); + let sn = + crate::streaming::start_streaming_owned(vec![format!("--alertfile={}", path.display())]) + .await; + + // Known-cleared, then raised, so the detector fires an edge on any host. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let deadline = std::time::Instant::now() + Duration::from_secs(45); + while pager.attempts().await == 0 { + assert!( + std::time::Instant::now() < deadline, + "the status event was never attempted; the raise did not reach the hook" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + } + + // Edit `ops` only. `pager`'s stanza is byte-identical, so its delivery task + // — and the `disk_low` still in its queue — must survive. + write_two_hook_alertfile( + &dir, + (&pager, "pager", "\"status\""), + (&ops, "ops", "\"status\", \"chain\""), + ); + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let got = pager + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + assert_eq!(got.json()["body"]["kind"], "disk_low"); + assert!(got.signature_valid(SECRET)); +} + +/// A redirect is never followed: the signed body does not go to a host the +/// alertfile never named. +/// +/// The configured URL is validated once, at load. If a 302 moved the request, +/// that check would be advisory only — and the interesting destinations are the +/// ones an operator cannot see from the outside: a cloud metadata endpoint, an +/// RFC1918 admin port, the node's own RPC. Here the redirect target is a second +/// mock receiver, which must never be touched. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_does_not_follow_redirects() { + let elsewhere = MockReceiver::ok().await; + let receiver = MockReceiver::redirecting_to(elsewhere.url()).await; + let (sn, _dir) = node_with_hook(&receiver, "redirector", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 2).await; + // Give the dispatcher room to deliver, retry, and give up. + tokio::time::sleep(Duration::from_secs(5)).await; + + assert!( + receiver.attempts().await >= 1, + "the configured endpoint should have been contacted", + ); + assert_eq!( + elsewhere.attempts().await, + 0, + "the redirect target must never receive the signed body", + ); +} + +/// A receiver that permanently rejects one event must not pin the queue: the +/// event is skipped and later events still arrive. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_permanent_rejection_does_not_wedge_the_queue() { + // 404 the first request only. If a permanent 4xx were retried forever the + // second event would never be delivered. + let receiver = MockReceiver::failing_first(404, 1).await; + let (sn, _dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 2).await; + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "chain") + .await; + assert_eq!(got.json()["body"]["kind"], "block_connected"); + assert!(got.signature_valid(SECRET)); + // Exactly one rejection, and it did not stall what followed. + assert!(receiver.attempts().await >= 2); +} + +/// A hook that was not running does not go back for what it missed. +/// +/// This is the whole durability contract, asserted from the outside. Webhooks +/// are best-effort (design D6): there is no cursor, no replay and no gap +/// notice. A hook that comes back resumes at the live head, and the events that +/// happened while it was down are simply not delivered. +/// +/// The assertion is deliberately that the node does *not* do something. Earlier +/// revisions delivered the missed span, then announced it — each of which cost +/// a durable keyspace, a `Store` trait extension across four backends, a GC +/// pass and a synthesized delivery-id space, to serve a use case the streaming +/// API already serves properly with real cursors and backpressure. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn a_hook_that_was_down_resumes_at_the_live_head() { + // 503 blocks 2..=5 so they stay queued and un-acked when the hook is + // retired. Everything else is accepted. + let receiver = MockReceiver::failing_heights(vec![2, 3, 4, 5], 503).await; + let (sn, dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 1).await; + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 1) + .await; + + // The outage: four blocks queued and retrying, none acked. + crate::streaming::mine_n(&sn, 4).await; + tokio::time::sleep(Duration::from_secs(3)).await; + + let missed = |r: &Received| (2..=5).contains(&r.json()["body"]["height"].as_u64().unwrap_or(0)); + let before = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); + assert!(before > 0, "the outage span should have been attempted at least once"); + + // Retire the generation with a same-URL alertfile edit — the same code path + // a restart takes. The queued blocks go with it. + write_alertfile(&dir, &receiver, "chain", "\"chain\"", true); + crate::streaming::sighup_with_conf(&sn, "").await; + tokio::time::sleep(Duration::from_secs(3)).await; + + // Nothing from the missed span comes back... + let after = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); + assert_eq!( + after, before, + "the missed span was re-sent; a best-effort hook resumes at the head and \ + does not carry history across a reload" + ); + + // ...and no in-band notice is synthesized about it either. + assert!( + !receiver + .all_attempts() + .await + .iter() + .any(|r| r.json()["body"]["category"] == "lagged"), + "a lagged body was delivered; gap accounting was removed with the cursor" + ); + + // The hook is live: the next block still arrives. + crate::streaming::mine_n(&sn, 1).await; + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 6) + .await; +}