From 11a07718b90f02f200db7bfe20c4a39c2a64658f Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Fri, 24 Jul 2026 13:07:27 -0600 Subject: [PATCH 01/11] alerting: satd-alert crate + webhook dispatcher, absorbing reorgwebhook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PR 3 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §4). PR 2 made the node detect its own problems; this delivers them somewhere. `alertfile=` configures any number of signed HTTP hooks, each filtered by category, status kind, and severity floor. The pure half — parsing, validation, permissions, the delivery contract, retry classification — lives in a new `satd-alert` crate so every rule is testable without a socket and the signature scheme can be pinned by golden vectors a third-party receiver checks itself against. `reqwest` and the dispatcher task stay in the binary. The split, and the reason for it, mirror `satd-auth`. Design points: - One wire schema, not two. The POST body is byte-identical to the JSON a WebSocket subscriber receives for the same event, so a receiver parses both with one code path. Delivery metadata rides in headers — putting the attempt counter or hook id in the body would change the bytes between retries of one event, breaking both signing and receiver-side dedup. - Honest failure behavior. Serial and in-order per hook; exponential backoff to a 5-minute ceiling on 5xx/408/429/timeouts; any other 4xx counted and skipped, because a receiver answering 404 forever must not pin the head of the queue and convert every later event into an overflow drop. A bounded queue that overflows emits a `lagged` body carrying the count and a resume cursor ahead of the next delivery: a gap is never silent. - At-least-once for confirmed chain events across a restart, via a per-hook cursor in the chainstate metadata CF and a startup replay through the same machinery the streaming carriers use (same 10k clamp, same clamp signal). Health events come back by re-evaluation; mempool stays best-effort — the contract the bus itself offers. - Config follows the authfile model: path restart-only, contents re-read on every SIGHUP, 0600 enforced from the open handle. A parse error keeps the last-good hook set — alerting that silently stopped after a typo is the worse failure. Validation is recognize-and-reject throughout: an unknown category, a typo'd kind, a duplicate id, or a missing secret is an error, never a rule that quietly matches nothing. - `reorgwebhook=` is absorbed, not replaced. Its payload stays the shipped ReorgRecord JSON byte for byte — ChainEvent::Reorg carries neither depth, fork_height, nor the disconnected/reconnected lists, so re-shaping it would have silently broken deployed receivers. Routing it through this module also moves that outbound HTTP off the consensus runtime, where it should never have been (D7). Per-hook counters (`satd_alertwebhook_*`) render only when a hook is configured, so an unconfigured node's /metrics page is unchanged. E2E covers a signed status delivery verified as a third party would, retry-until-recovery via the attempt counter, and a permanent 4xx not wedging the queue behind it. --- CHANGELOG.md | 13 + Cargo.lock | 16 + Cargo.toml | 2 +- docs/manual/src/config-reference.md | 1 + docs/manual/src/observability.md | 90 +++ docs/release-notes/0.5.0-pre.md | 56 ++ mcp/src/context.rs | 2 + mcp/src/tools/ergonomics.rs | 2 + mcp/tests/tools.rs | 2 + node/src/metrics.rs | 206 +++++++ node/src/storage/coin_cache.rs | 8 + node/src/storage/mod.rs | 20 + node/src/storage/rocksdb_store.rs | 17 + node/src/storage/split_store.rs | 11 + satd-alert/Cargo.toml | 32 + satd-alert/src/config.rs | 755 +++++++++++++++++++++++ satd-alert/src/contract.rs | 115 ++++ satd-alert/src/lib.rs | 33 ++ satd-alert/src/retry.rs | 141 +++++ satd/Cargo.toml | 1 + satd/src/alert.rs | 890 ++++++++++++++++++++++++++++ satd/src/config.rs | 20 +- satd/src/main.rs | 153 ++--- satd/src/reload.rs | 30 +- satd/tests/e2e.rs | 4 + satd/tests/e2e/streaming.rs | 25 +- satd/tests/e2e/webhooks.rs | 343 +++++++++++ 27 files changed, 2888 insertions(+), 100 deletions(-) create mode 100644 satd-alert/Cargo.toml create mode 100644 satd-alert/src/config.rs create mode 100644 satd-alert/src/contract.rs create mode 100644 satd-alert/src/lib.rs create mode 100644 satd-alert/src/retry.rs create mode 100644 satd/src/alert.rs create mode 100644 satd/tests/e2e/webhooks.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index f9c842913..40c7f8f5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -153,6 +153,19 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). free-space probe is bounded and carried across polls, so a `blocksdir` on an unresponsive network mount stalls only the disk alert — never the other detectors — and strands at most one blocking thread rather than one per poll. +- Alerting: outbound webhooks. `alertfile=` configures any number of + signed HTTP hooks, each filtered by category/kind/severity. Bodies are + byte-identical to the streaming API's JSON for the same event; delivery + metadata rides in headers (`X-Satd-Signature`, `X-Satd-Delivery`, + `X-Satd-Hook`, `X-Satd-Attempt`). Delivery is serial and in-order per hook, + retried with exponential backoff on transient failures and skipped on a + permanent 4xx, with a bounded queue whose overflow is reported in-band as a + `lagged` event rather than silently. Confirmed chain events are at-least-once + across a restart via a durable per-hook cursor. Hooks reload on SIGHUP + (keep-last-good on error); per-hook counters are exported as + `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with + their original payload, now delivered by the same dispatcher — which also + moves that outbound HTTP off the consensus runtime. ## Releases diff --git a/Cargo.lock b/Cargo.lock index bdfdbb7c7..48ac6d827 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4081,6 +4081,7 @@ dependencies = [ "rand 0.8.6", "rcgen", "reqwest", + "satd-alert", "satd-auth", "satd-events", "satd-events-client", @@ -4100,6 +4101,21 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "satd-alert" +version = "0.4.0" +dependencies = [ + "hex", + "hmac", + "node", + "serde", + "sha2", + "tempfile", + "thiserror 2.0.18", + "toml_edit", + "tracing", +] + [[package]] name = "satd-auth" version = "0.4.0" diff --git a/Cargo.toml b/Cargo.toml index d25b3e632..5bc115363 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,5 +1,5 @@ [workspace] -members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-policy"] +members = ["satd", "sat-cli", "node", "node-index", "node-filter-index", "node-sp-index", "esplora-handlers", "electrum-proto", "sat-tui", "mcp", "consensus", "events", "satd-events-proto", "satd-events-client", "tls-config", "satd-auth", "satd-alert", "satd-policy"] # `fuzz/` is a standalone cargo-fuzz workspace (needs a nightly + sanitizer # toolchain); keep it out of the normal `cargo build` / `cargo test --all`. exclude = ["fuzz"] diff --git a/docs/manual/src/config-reference.md b/docs/manual/src/config-reference.md index edb8b57b9..5568c335e 100644 --- a/docs/manual/src/config-reference.md +++ b/docs/manual/src/config-reference.md @@ -408,6 +408,7 @@ the taxonomy and the details each event carries. | Key | Default | Reload | Compat | Description | |---|---|---|---|---| +| `alertfile` | none | path restart, contents hot | satd | TOML file describing outbound alert webhooks. Must be mode `0600` — it holds signing secrets. The *path* is read once at startup; the file's *contents* are re-read on every SIGHUP, so a hook can be added, edited, or removed live. A parse error keeps the last-good hook set. See [Observability → Alert webhooks](observability.md#alert-webhooks). | | `alerttipstallseconds` | `3600` (`0` on regtest) | hot | satd | Raise `tip_stall` after this many seconds with no connected block. Defaults to disabled on regtest only, where blocks exist just when a test mines them and an idle chain is normal; every other network — test networks included — keeps the hour, since going an hour without a block is not an ordinary property of thin hashrate the way a shallow reorg is. *Not* suppressed during initial block download — `is_initial_block_download()` compares the tip header's timestamp against the wall clock rather than tracking sync progress, so a node that was caught up and then wedged re-enters it precisely when you need paging. A node that is genuinely syncing connects blocks continuously and so never crosses the threshold. Cleared the moment a block connects — or, if you raise this value past the current tip age, on the next detector poll. | | `alertdiskfreemb` | `10240` | hot | satd | Raise `disk_low` below this many MiB free on the blocks directory (or the data directory when `blocksdir` is not split out). Clears at 1.5× the floor, or as soon as you lower the floor below the current reading. | | `alertmempoolfullpct` | `90` | hot | satd | Raise `mempool_congested` at this percentage of `maxmempool`. Clears below 75 % of the raise line, or as soon as you raise the threshold above the current occupancy. Values above 100 are clamped. | diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index c2b0cfe09..e5ccb133b 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -145,6 +145,96 @@ and `satd_disk_free_bytes` (free space on the watched directory). The latter is omitted rather than reported as zero when the filesystem cannot be interrogated. +## Alert webhooks + +The three surfaces above all require something to be *watching* the node. A +webhook pushes instead: point `alertfile=` at a TOML file and satd POSTs +each matching event to your endpoint. + +```toml +version = 1 + +[[webhook]] +id = "pager" # unique; appears in headers, metrics, and the cursor key +url = "https://alerts.example/satd" +secret = "a-long-random-string" # required — it signs every delivery +categories = ["status"] # status | chain | mempool | heartbeat +kinds = ["tip_stall", "disk_low"] # optional, status only +min_severity = "warning" # optional, status only + +[[webhook]] +id = "deadman" +url = "https://hc-ping.example/abc123" +secret = "another-long-random-string" +categories = ["heartbeat"] +heartbeat_interval_secs = 300 # one ping per 5 min, not the bus's 1 Hz +``` + +The file must be mode `0600` — it holds signing secrets, and satd refuses to +read a group- or world-accessible one. Its **contents** are re-read on every +`SIGHUP`, so hooks can be added, edited, or removed without a restart; the +**path** is fixed at startup. A parse error on reload keeps the last-good hook +set and logs why, because alerting that silently stopped after a typo is the +worse failure. + +### What arrives + +```http +POST /your/endpoint HTTP/1.1 +Content-Type: application/json +X-Satd-Signature: sha256= +X-Satd-Delivery: -- +X-Satd-Hook: pager +X-Satd-Attempt: 1 +X-Satd-Webhook-Version: 1 + +{"schema_version":1,"stamp":{...},"body":{"category":"status", ...}} +``` + +The body is **byte-identical** to the JSON a WebSocket subscriber receives for +the same event, so a receiver parses webhook bodies and streaming frames with +one code path. Delivery metadata rides in headers and never in the body — which +is what makes the signature stable across retries. + +Verify the signature over the **raw body, before parsing**. Deduplicate on +`X-Satd-Delivery`: it is stable across retries of one event and unique across +restarts, so a retry and a genuine repeat are distinguishable. Reply `2xx` to +acknowledge. + +### Delivery behavior + +- **Serial and in-order per hook** — one request in flight at a time, so events + arrive in the order the node produced them. +- **Retried with backoff** on 5xx, 408, 429, timeouts, and connection failures: + 1 s doubling to a 5-minute ceiling, indefinitely. Any *other* 4xx is treated + as permanent, counted, and skipped — a receiver answering 404 forever must + not pin the queue and turn every later event into a drop. +- **Bounded queue.** A hook that falls far enough behind drops events, and the + next delivery is preceded by a `lagged` body carrying how many were lost and + the cursor to resume from. A gap is never silent. +- **Nothing reaches consensus.** Deliveries run on the isolated API runtime; + a stalled endpoint cannot affect block connection. +- **At-least-once for confirmed chain events across a restart.** Each hook's + resume position is persisted, and on startup the hook replays what it missed + (bounded by the same 10 000-block window the streaming API uses; beyond that + you get a `lagged` notice instead). Health events are re-raised by + re-evaluation rather than replayed, and mempool events are best-effort — the + same contract the event bus itself offers. + +Plaintext `http://` is accepted for loopback and private-network targets. For a +public host, use `https://` or set `allow_insecure_http = true` on the hook. + +Per-hook counters are exported: `satd_alertwebhook_delivered_total`, +`_failed_attempts_total`, `_dropped_total` (the dead-letter count), +`_queue_depth`, and `_last_success_age_seconds`, all labelled `hook=""`. +Nothing is exported when no hook is configured. + +> **Note.** The older `reorgwebhook=` / `reorgwebhooksecret=` keys still work +> and are now served by this dispatcher, with their original `ReorgRecord` +> payload unchanged — existing receivers need no edits. New deployments should +> prefer an `alertfile` hook with `categories = ["chain"]`, which delivers the +> standard event envelope. + ## Structured JSON Logging `satd` logs to stdout. Use `--log-format=json` to switch from the text format diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index a6ac85419..0851940b4 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -383,6 +383,62 @@ alert does not delete the series out from under a Prometheus rule of your own. Disk headroom previously had no alerting or metric at all, which is how a silent disk-fill wedged a dogfood node in May. +### Alert webhooks + +Health events on the streaming API still require something to be *watching*. +`alertfile=` pushes instead — a TOML file describing any number of signed +HTTP hooks, each filtered by category, kind, and severity floor: + +```toml +version = 1 + +[[webhook]] +id = "pager" +url = "https://alerts.example/satd" +secret = "a-long-random-string" +categories = ["status"] +kinds = ["tip_stall", "disk_low"] +``` + +- **One wire schema, not two.** The POST body is byte-identical to the JSON a + WebSocket subscriber receives for the same event, so a receiver parses both + with one code path. Delivery metadata (`X-Satd-Signature`, `X-Satd-Delivery`, + `X-Satd-Hook`, `X-Satd-Attempt`) rides in headers and never in the body — + which is what keeps the signature stable across retries and makes + `X-Satd-Delivery` usable as an idempotency key. +- **Honest failure behavior.** Delivery is serial and in-order per hook, with + exponential backoff to a five-minute ceiling on transient failures. A + permanent 4xx is counted and skipped rather than retried forever — a receiver + answering 404 must not pin the queue and convert every later event into a + drop. A hook that falls far enough behind drops events and the next delivery + is preceded by a `lagged` body carrying the count and a resume cursor: a gap + is never silent. +- **At-least-once for confirmed chain events across a restart.** Each hook's + resume position is persisted with the chainstate, and on startup the hook + replays what it missed, bounded by the same 10 000-block window the streaming + API uses. Health events come back by re-evaluation instead; mempool events + stay best-effort, which is the contract the bus itself offers. +- **Reloadable.** Hook definitions are re-read on every SIGHUP even when no + `bitcoin.conf` key changed, so a hook can be added, edited, or removed live. + A parse error keeps the last-good set — alerting that silently stopped after + a typo is the worse failure. The file must be mode 0600; it holds signing + secrets. +- Per-hook counters are exported (`satd_alertwebhook_delivered_total`, + `_failed_attempts_total`, `_dropped_total`, `_queue_depth`, + `_last_success_age_seconds`), and a node with no alertfile exports none of + them. + +**`reorgwebhook=` is absorbed, not replaced.** The existing keys keep working +and their payload is unchanged byte for byte — `ChainEvent::Reorg` does not +carry the `depth`, `fork_height`, or disconnected/reconnected lists that a +`ReorgRecord` does, so re-shaping it would have silently broken deployed +receivers. It is now served by the new dispatcher, which also moves that +outbound HTTP off the consensus runtime, where it should never have been. New +deployments should prefer an `alertfile` hook with `categories = ["chain"]`. + +The signature scheme is specified with cross-language test vectors so a +receiver can be verified without a running node. + ## Storage / Core compatibility ### Core v28+ XOR-obfuscated block files (`blocksxor`) diff --git a/mcp/src/context.rs b/mcp/src/context.rs index 3e72989ce..b710d52b5 100644 --- a/mcp/src/context.rs +++ b/mcp/src/context.rs @@ -28,4 +28,6 @@ pub struct McpContext { /// Health-detector readings, so `get_metrics_snapshot` renders the same /// health gauges as the HTTP scrape. `None` in tests that bypass main.rs. pub health: Option>, + /// Webhook delivery counters, for the same reason. `None` in tests. + pub webhooks: Option>, } diff --git a/mcp/src/tools/ergonomics.rs b/mcp/src/tools/ergonomics.rs index 9b0d1b1ac..9be0c11f8 100644 --- a/mcp/src/tools/ergonomics.rs +++ b/mcp/src/tools/ergonomics.rs @@ -47,6 +47,7 @@ pub fn get_metrics_snapshot(ctx: &McpContext) -> String { addr_subs: ctx.addr_subs.clone(), addr_enabled: ctx.addr_enabled, health: ctx.health.clone(), + webhooks: ctx.webhooks.clone(), }; let body = metrics_ctx.render_prometheus(); let result = json!({ @@ -82,6 +83,7 @@ pub fn get_readiness(ctx: &McpContext) -> String { // Readiness reads only chain heights; the health gauges are not // consulted, so there is nothing to thread through here. health: None, + webhooks: None, }; let (ready, reason) = match metrics_ctx.is_ready() { Ok(()) => (true, None), diff --git a/mcp/tests/tools.rs b/mcp/tests/tools.rs index 1ef2292fa..a96df16f4 100644 --- a/mcp/tests/tools.rs +++ b/mcp/tests/tools.rs @@ -59,6 +59,7 @@ fn make_test_ctx() -> (McpContext, tempfile::TempDir) { addr_enabled: false, addr_subs: None, health: None, + webhooks: None, }; (ctx, dir) } @@ -616,6 +617,7 @@ mod mining { addr_enabled: false, addr_subs: None, health: None, + webhooks: None, }; let result = mine::generate_blocks(&ctx, 1, REGTEST_ADDR); diff --git a/node/src/metrics.rs b/node/src/metrics.rs index 83e6b546b..cc0fd5f6c 100644 --- a/node/src/metrics.rs +++ b/node/src/metrics.rs @@ -54,6 +54,9 @@ pub struct MetricsContext { /// `satd_disk_free_bytes` is exactly the alarm an operator must not be /// shown falsely). pub health: Option>, + /// Per-hook webhook delivery counters. `None` (or empty) when no alertfile + /// is configured, in which case the block renders nothing at all. + pub webhooks: Option>, } impl MetricsContext { @@ -337,6 +340,9 @@ impl MetricsContext { // Node-health gauges, rendered only when the detector task is running. render_health_metrics(&mut out, self.health.as_deref()); + // Webhook delivery counters, rendered only when a hook is configured. + render_webhook_metrics(&mut out, self.webhooks.as_deref()); + // Transaction-filtering policy metrics (design §10, PR 7c). Extracted to // a free function so the I8-invisibility invariant (a node with no // non-empty ruleset renders a byte-identical page) is unit-testable @@ -404,6 +410,136 @@ fn metric_sample(out: &mut String, name: &str, labels: &[(&str, &str)], value: u } } +/// Per-hook webhook delivery counters, written by the alert dispatcher in the +/// `satd` binary and rendered here. +/// +/// Lives in `node` rather than beside the dispatcher because `satd-alert` +/// depends on `node` (for the health taxonomy), so the reverse dependency +/// needed to render them from here would be a cycle. +#[derive(Debug, Default)] +pub struct HookCounters { + pub delivered: std::sync::atomic::AtomicU64, + /// Failed *attempts*, not failed events: a single event retried five times + /// before succeeding contributes five. This is the signal that an endpoint + /// is flaky even while it is ultimately keeping up. + pub failed_attempts: std::sync::atomic::AtomicU64, + /// Events never delivered: queue overflow, broadcast lag, or a permanent + /// 4xx. The dead-letter count. + pub dropped: std::sync::atomic::AtomicU64, + pub queue_depth: std::sync::atomic::AtomicU64, + /// Unix seconds of the last 2xx, or 0 if there has never been one. + pub last_success_unix: std::sync::atomic::AtomicU64, +} + +/// Registry of per-hook counters, keyed by hook id. +/// +/// A `BTreeMap` so `/metrics` renders hooks in a stable order (Prometheus does +/// not care, but a diffable scrape is worth the ordering), behind an `RwLock` +/// because a SIGHUP reload can add or remove hooks while the endpoint is being +/// scraped. +#[derive(Debug, Default)] +pub struct WebhookMetrics { + hooks: parking_lot::RwLock>>, +} + +impl WebhookMetrics { + pub fn new() -> Self { + Self::default() + } + + /// Get or create a hook's counters. Called on hook registration and on + /// every delivery, so it must not allocate on the hot path — the common + /// case is a read-lock hit. + pub fn hook(&self, id: &str) -> Arc { + if let Some(c) = self.hooks.read().get(id) { + return c.clone(); + } + let mut w = self.hooks.write(); + w.entry(id.to_string()).or_default().clone() + } + + /// Forget hooks that are no longer configured, so a removed hook's series + /// stops being exported instead of freezing at its last value forever. + pub fn retain(&self, ids: &[String]) { + self.hooks.write().retain(|k, _| ids.iter().any(|i| i == k)); + } + + pub fn is_empty(&self) -> bool { + self.hooks.read().is_empty() + } + + fn snapshot(&self) -> Vec<(String, Arc)> { + self.hooks + .read() + .iter() + .map(|(k, v)| (k.clone(), v.clone())) + .collect() + } +} + +/// Append the webhook dispatcher's per-hook counters — but only when at least +/// one hook is configured, so a node with no alertfile renders a page +/// byte-identical to a build without the dispatcher (the same invisibility +/// property the policy metrics hold). +fn render_webhook_metrics(out: &mut String, metrics: Option<&WebhookMetrics>) { + use std::sync::atomic::Ordering; + let Some(metrics) = metrics.filter(|m| !m.is_empty()) else { + return; + }; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0); + for (id, c) in metrics.snapshot() { + let labels = [("hook", id.as_str())]; + metric( + out, + "satd_alertwebhook_delivered_total", + "Webhook events acknowledged (2xx) by the receiver.", + "counter", + &labels, + c.delivered.load(Ordering::Relaxed), + ); + metric( + out, + "satd_alertwebhook_failed_attempts_total", + "Webhook delivery attempts that failed (retried or dropped).", + "counter", + &labels, + c.failed_attempts.load(Ordering::Relaxed), + ); + metric( + out, + "satd_alertwebhook_dropped_total", + "Webhook events never delivered (queue overflow, broadcast lag, or a permanent 4xx).", + "counter", + &labels, + c.dropped.load(Ordering::Relaxed), + ); + metric( + out, + "satd_alertwebhook_queue_depth", + "Events currently queued for this webhook.", + "gauge", + &labels, + c.queue_depth.load(Ordering::Relaxed), + ); + // Age rather than a timestamp: an alerting rule wants "no successful + // delivery in N minutes", which is awkward to express against an + // absolute epoch value. 0 before the first success, which reads as + // "fresh" — so pair it with `delivered_total > 0` in a rule. + let last = c.last_success_unix.load(Ordering::Relaxed); + metric( + out, + "satd_alertwebhook_last_success_age_seconds", + "Seconds since this webhook's last acknowledged delivery (0 if none yet).", + "gauge", + &labels, + if last == 0 { 0 } else { now.saturating_sub(last) }, + ); + } +} + /// Append the node-health gauges (§A3): tip age, free disk, and one 0/1 series /// per standing alert condition. /// @@ -796,6 +932,76 @@ mod tests { assert_eq!(scope_label(false, false), "none"); } + #[test] + fn webhook_metrics_invisible_until_a_hook_is_configured() { + use std::sync::atomic::Ordering; + // No registry, and an empty registry, both render nothing: a node with + // no alertfile must produce a page byte-identical to one built without + // the dispatcher. + let mut out = String::new(); + render_webhook_metrics(&mut out, None); + assert!(out.is_empty(), "{out}"); + + let m = WebhookMetrics::new(); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.is_empty(), "an empty registry must leak nothing:\n{out}"); + + // A registered hook exports its counters, labelled by id. + let c = m.hook("ops"); + c.delivered.store(7, Ordering::Relaxed); + c.dropped.store(2, Ordering::Relaxed); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.contains(r#"satd_alertwebhook_delivered_total{hook="ops"} 7"#), "{out}"); + assert!(out.contains(r#"satd_alertwebhook_dropped_total{hook="ops"} 2"#), "{out}"); + assert!(out.contains(r#"satd_alertwebhook_queue_depth{hook="ops"} 0"#), "{out}"); + } + + #[test] + fn removed_hooks_stop_being_exported() { + // Otherwise a hook deleted on SIGHUP would keep exporting its last + // values forever, and an alerting rule on it could never recover. + let m = WebhookMetrics::new(); + m.hook("old"); + m.hook("new"); + m.retain(&["new".to_string()]); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!(out.contains(r#"hook="new""#), "{out}"); + assert!(!out.contains(r#"hook="old""#), "{out}"); + } + + #[test] + fn last_success_age_is_zero_before_the_first_delivery() { + use std::sync::atomic::Ordering; + // An age computed from an unset (0) timestamp would render as ~57 years + // and read as a catastrophic outage on a node that simply has not + // delivered anything yet. + let m = WebhookMetrics::new(); + m.hook("ops"); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!( + out.contains(r#"satd_alertwebhook_last_success_age_seconds{hook="ops"} 0"#), + "{out}" + ); + + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + m.hook("ops") + .last_success_unix + .store(now.saturating_sub(30), Ordering::Relaxed); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert!( + out.contains(r#"satd_alertwebhook_last_success_age_seconds{hook="ops"} 3"#), + "expected an age around 30s:\n{out}" + ); + } + #[test] fn health_metrics_absent_without_a_detector_task() { // Rendering zeros here would show a `satd_disk_free_bytes 0` — the diff --git a/node/src/storage/coin_cache.rs b/node/src/storage/coin_cache.rs index 90f352b57..39bf17d8f 100644 --- a/node/src/storage/coin_cache.rs +++ b/node/src/storage/coin_cache.rs @@ -1284,6 +1284,14 @@ impl Store for CoinCache { self.inner.write_sp_backfill_last_error(msg) } + fn read_alert_cursor(&self, key: &[u8]) -> Option> { + self.inner.read_alert_cursor(key) + } + + fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { + self.inner.write_alert_cursor(key, value) + } + fn lookup_spend( &self, outpoint: &OutPoint, diff --git a/node/src/storage/mod.rs b/node/src/storage/mod.rs index d467b73ba..60b5807a0 100644 --- a/node/src/storage/mod.rs +++ b/node/src/storage/mod.rs @@ -845,4 +845,24 @@ pub trait Store: Send + Sync { "write_sp_backfill_last_error not supported on this backend".into(), )) } + + /// Read a webhook hook's durable resume cursor from metadata, by its + /// `alertwebhook.cursor.` key. `None` when the hook has never acked a + /// cursor-bearing delivery (a fresh hook starts at the live head). + /// + /// Deliberately narrow rather than a generic metadata get/put: the alert + /// dispatcher is the only writer, and a general "write any key into the + /// chainstate metadata CF" accessor is an invitation to collide with the + /// index markers that share it. + fn read_alert_cursor(&self, _key: &[u8]) -> Option> { + None + } + + /// Persist a webhook hook's resume cursor. Default: error, so a non-Rocks + /// backend fails loud rather than silently losing at-least-once delivery. + fn write_alert_cursor(&self, _key: &[u8], _value: &[u8]) -> Result<(), StoreError> { + Err(StoreError::Database( + "write_alert_cursor not supported on this backend".into(), + )) + } } diff --git a/node/src/storage/rocksdb_store.rs b/node/src/storage/rocksdb_store.rs index cb435721a..9746ef2e8 100644 --- a/node/src/storage/rocksdb_store.rs +++ b/node/src/storage/rocksdb_store.rs @@ -1249,6 +1249,23 @@ impl RocksDbStore { } impl Store for RocksDbStore { + /// Read a hook's `alertwebhook.cursor.` blob. The key namespace is + /// disjoint from the index markers that share this CF. + fn read_alert_cursor(&self, key: &[u8]) -> Option> { + let cf = self.db.cf_handle(CF_METADATA)?; + self.db.get_cf(&cf, key).ok().flatten() + } + + fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { + let cf = self + .db + .cf_handle(CF_METADATA) + .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; + self.db + .put_cf(&cf, key, value) + .map_err(|e| StoreError::Database(e.to_string())) + } + fn get_block_index(&self, hash: &BlockHash) -> Option { let cf = self.cf(CF_BLOCK_INDEX); let value = self.db.get_cf(&cf, hash_bytes(hash)).ok()??; diff --git a/node/src/storage/split_store.rs b/node/src/storage/split_store.rs index 41ca0970f..264121bbc 100644 --- a/node/src/storage/split_store.rs +++ b/node/src/storage/split_store.rs @@ -89,6 +89,17 @@ impl Store for SplitStore { self.block_store.get_cumulative_tx_count(hash) } + // Webhook resume cursors live with the block index in the shared store, for + // the same reason cumulative tx counts do: they must survive the + // snapshot→background handoff, which replaces the coins store. + fn read_alert_cursor(&self, key: &[u8]) -> Option> { + self.block_store.read_alert_cursor(key) + } + + fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { + self.block_store.write_alert_cursor(key, value) + } + fn chain_tx_backfill_complete(&self) -> bool { self.block_store.chain_tx_backfill_complete() } diff --git a/satd-alert/Cargo.toml b/satd-alert/Cargo.toml new file mode 100644 index 000000000..f03cfda17 --- /dev/null +++ b/satd-alert/Cargo.toml @@ -0,0 +1,32 @@ +[package] +name = "satd-alert" +version.workspace = true +edition.workspace = true +license.workspace = true +publish.workspace = true +repository.workspace = true + +# Operator-alerting core: alertfile parsing/validation, the webhook delivery +# contract (headers + HMAC signature), and the retry policy. Deliberately +# depends on NO async runtime or HTTP client — `reqwest` and the dispatcher task +# live in the `satd` binary, so everything here is a pure function or a plain +# data type and is unit-testable without a socket. Mirrors `satd-auth`. +# +# The one heavyweight dependency is `node`, and it is deliberate: the health- +# event taxonomy (kinds, severities, category bits) has exactly one definition, +# and an alertfile's `kinds`/`categories`/`min_severity` are validated against +# it rather than against a copy that could drift. +[dependencies] +node = { path = "../node" } +sha2 = { workspace = true } +hmac = { workspace = true } +hex = { workspace = true } +serde = { workspace = true } +thiserror = { workspace = true } +tracing = { workspace = true } +# Parse-only, like satd-auth: the `display` default feature pulls an +# unpublished toml_writer and nothing here round-trips the file. +toml_edit = { version = "0.25", default-features = false, features = ["parse"] } + +[dev-dependencies] +tempfile = "3" diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs new file mode 100644 index 000000000..efbe3594a --- /dev/null +++ b/satd-alert/src/config.rs @@ -0,0 +1,755 @@ +//! The `alertfile=` format: parsing, validation, and permission checks. +//! +//! ```toml +//! version = 1 +//! +//! [[webhook]] +//! id = "ops" # unique; appears in headers and the cursor key +//! url = "https://alerts.example/satd" +//! secret = "a-long-random-string" # required — it signs the body +//! categories = ["status"] # status | chain | mempool | heartbeat +//! kinds = ["tip_stall", "disk_low"] # optional, status only +//! min_severity = "warning" # optional, status only +//! heartbeat_interval_secs = 60 # optional dead-man ping; default off +//! ``` +//! +//! Why a file rather than flat `bitcoin.conf` keys: a hook has a secret, a +//! filter, and (from the watch-set work) a watch-set. Core's config format is +//! flat and first-wins, so several hooks cannot be expressed in it without +//! inventing an index syntax. The precedent is `authfile=`, which is the same +//! shape for the same reason — down to the 0600 permission check, since both +//! files hold secrets. +//! +//! Validation is **recognize-and-reject**, matching satd's config posture: an +//! unknown key, an unknown category, a duplicate id, or a missing secret is a +//! hard error. An accepted-but-ignored alerting rule is worse than a refused +//! one — the operator believes they are covered when they are not. + +use std::collections::BTreeSet; +use std::path::{Path, PathBuf}; + +use node::events::{StatusKind, StatusSeverity}; + +/// Per-hook outbound queue depth. Delivery is serial per hook, so this is how +/// far a hook may fall behind before events are dropped and the gap is reported +/// to the receiver as a `Lagged` notice. 1024 is the same order as the +/// publisher's own replay ring: enough to ride out a receiver restart, small +/// enough that a permanently dead endpoint cannot grow memory without bound. +pub const HOOK_QUEUE_CAPACITY: usize = 1024; + +/// The only alertfile schema version this build understands. +pub const ALERTFILE_VERSION: u64 = 1; + +#[derive(Debug, thiserror::Error)] +pub enum AlertFileError { + #[error("alertfile {path}: {source}")] + Io { + path: PathBuf, + #[source] + source: std::io::Error, + }, + #[error( + "alertfile {path} is group/world accessible (mode {mode:04o}); it holds webhook signing \ + secrets — run: chmod 600 {path}" + )] + Permissions { path: PathBuf, mode: u32 }, + #[error("alertfile {path}: not valid TOML: {source}")] + Toml { + path: PathBuf, + #[source] + source: toml_edit::TomlError, + }, + #[error("alertfile {path}: {message}")] + Invalid { path: PathBuf, message: String }, +} + +impl AlertFileError { + fn invalid(path: &Path, message: impl Into) -> Self { + Self::Invalid { + path: path.to_path_buf(), + message: message.into(), + } + } +} + +/// Which event classes a hook receives, as a mask of the streaming API's +/// category bits. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub struct CategoryMask(pub u32); + +impl CategoryMask { + pub fn contains(self, bit: u32) -> bool { + self.0 & bit != 0 + } +} + +/// A hook's event filter: categories, then (for status events) kinds and a +/// severity floor. +#[derive(Debug, Clone, Default)] +pub struct HookFilter { + pub categories: CategoryMask, + /// `None` = every kind in the subscribed categories. Only meaningful for + /// status events; a hook that filters kinds still receives whatever other + /// categories it asked for. + pub kinds: Option>, + /// `None` = no floor. Status events only. + pub min_severity: Option, +} + +impl HookFilter { + /// Whether a status event passes this filter. `category_bit` is checked by + /// the caller for non-status bodies; this is the status-specific half. + pub fn accepts_status(&self, kind: StatusKind, severity: StatusSeverity) -> bool { + if !self.categories.contains(node::events::CATEGORY_STATUS) { + return false; + } + if let Some(kinds) = &self.kinds + && !kinds.contains(&kind) + { + return false; + } + if let Some(min) = self.min_severity + && severity < min + { + return false; + } + true + } +} + +/// One configured webhook. +#[derive(Debug, Clone)] +pub struct Hook { + pub id: String, + pub url: String, + /// Required. An unsigned webhook invites anyone who learns the URL to + /// forge node events into the operator's alerting pipeline, and a secret + /// costs nothing. + pub secret: String, + pub filter: HookFilter, + /// Forward at most one heartbeat per interval, as a dead-man's switch + /// ("tell me when my node goes quiet"). `None` = off. The bus heartbeat is + /// 1 Hz, which no HTTP receiver wants unthrottled. + pub heartbeat_interval_secs: Option, + /// Permit a plaintext `http://` URL to a non-loopback, non-private + /// address. Off by default: webhook bodies carry chain data rather than + /// secrets, but signed-then-cleartext is still a footgun. + pub allow_insecure_http: bool, +} + +impl Hook { + /// The `CF_METADATA` key holding this hook's durable resume cursor. + pub fn cursor_key(&self) -> Vec { + format!("alertwebhook.cursor.{}", self.id).into_bytes() + } +} + +/// A parsed, validated alertfile. +#[derive(Debug, Clone, Default)] +pub struct AlertFile { + pub hooks: Vec, +} + +impl AlertFile { + /// Read, permission-check, parse, and validate an alertfile. + pub fn load(path: &Path) -> Result { + let file = std::fs::File::open(path).map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + check_perms(path, &file)?; + let text = std::io::read_to_string(&file).map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + Self::parse(path, &text) + } + + /// Parse and validate alertfile contents. Split from [`load`](Self::load) + /// so the rules are testable without a file on disk. + pub fn parse(path: &Path, text: &str) -> Result { + let doc: toml_edit::DocumentMut = + text.parse().map_err(|source| AlertFileError::Toml { + path: path.to_path_buf(), + source, + })?; + + match doc.get("version").and_then(|v| v.as_integer()) { + Some(v) if v as u64 == ALERTFILE_VERSION => {} + Some(v) => { + return Err(AlertFileError::invalid( + path, + format!("unsupported version {v} (this build understands {ALERTFILE_VERSION})"), + )); + } + None => { + return Err(AlertFileError::invalid( + path, + "missing required `version` key", + )); + } + } + + for key in doc.as_table().iter().map(|(k, _)| k) { + if !matches!(key, "version" | "webhook") { + return Err(AlertFileError::invalid( + path, + format!("unknown top-level key `{key}`"), + )); + } + } + + let mut hooks = Vec::new(); + if let Some(array) = doc.get("webhook") { + let Some(entries) = array.as_array_of_tables() else { + return Err(AlertFileError::invalid( + path, + "`webhook` must be an array of tables (`[[webhook]]`)", + )); + }; + for entry in entries { + hooks.push(parse_hook(path, entry)?); + } + } + + // Ids name a hook in delivery headers, in metrics labels, and in the + // durable cursor key, so a duplicate would silently make two hooks + // share one resume position. + let mut seen = BTreeSet::new(); + for h in &hooks { + if !seen.insert(h.id.clone()) { + return Err(AlertFileError::invalid( + path, + format!("duplicate webhook id `{}`", h.id), + )); + } + } + Ok(Self { hooks }) + } +} + +fn parse_hook(path: &Path, t: &toml_edit::Table) -> Result { + const KNOWN: &[&str] = &[ + "id", + "url", + "secret", + "categories", + "kinds", + "min_severity", + "heartbeat_interval_secs", + "allow_insecure_http", + ]; + for key in t.iter().map(|(k, _)| k) { + if !KNOWN.contains(&key) { + return Err(AlertFileError::invalid( + path, + format!("unknown key `{key}` in [[webhook]] (known: {})", KNOWN.join(", ")), + )); + } + } + + let id = req_str(path, t, "id")?; + if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') { + return Err(AlertFileError::invalid( + path, + format!("webhook id `{id}` must be non-empty [A-Za-z0-9_-] (it is used in headers, metric labels, and the cursor key)"), + )); + } + let url = req_str(path, t, "url")?; + let secret = req_str(path, t, "secret")?; + if secret.is_empty() { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `secret` must not be empty — it signs every delivery"), + )); + } + + let allow_insecure_http = t + .get("allow_insecure_http") + .and_then(|v| v.as_bool()) + .unwrap_or(false); + validate_url(path, &id, &url, allow_insecure_http)?; + + let categories = parse_categories(path, &id, t)?; + let kinds = parse_kinds(path, &id, t)?; + let min_severity = match t.get("min_severity") { + None => None, + Some(v) => { + let s = v.as_str().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `min_severity` must be a string")) + })?; + Some(StatusSeverity::from_str_exact(s).ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: unknown min_severity `{s}` (info, warning, critical)"), + ) + })?) + } + }; + let heartbeat_interval_secs = match t.get("heartbeat_interval_secs") { + None => None, + Some(v) => { + let n = v.as_integer().filter(|n| *n > 0).ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `heartbeat_interval_secs` must be a positive integer"), + ) + })?; + Some(n as u64) + } + }; + if heartbeat_interval_secs.is_some() + && !categories.contains(node::events::CATEGORY_HEARTBEAT) + { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: `heartbeat_interval_secs` needs the `heartbeat` category — \ + without it there is nothing to downsample" + ), + )); + } + + Ok(Hook { + id, + url, + secret, + filter: HookFilter { + categories, + kinds, + min_severity, + }, + heartbeat_interval_secs, + allow_insecure_http, + }) +} + +fn parse_categories( + path: &Path, + id: &str, + t: &toml_edit::Table, +) -> Result { + use node::events::{CATEGORY_CHAIN, CATEGORY_HEARTBEAT, CATEGORY_MEMPOOL, CATEGORY_STATUS}; + let Some(v) = t.get("categories") else { + // A hook with no categories would receive nothing, which is never what + // anyone meant to configure. + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` is required (status, chain, mempool, heartbeat)"), + )); + }; + let arr = v.as_array().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `categories` must be an array")) + })?; + let mut mask = 0u32; + for item in arr { + let name = item.as_str().ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` entries must be strings"), + ) + })?; + mask |= match name { + "status" => CATEGORY_STATUS, + "chain" => CATEGORY_CHAIN, + "mempool" => CATEGORY_MEMPOOL, + "heartbeat" => CATEGORY_HEARTBEAT, + // Rejected rather than silently ignored: the tweak firehose is + // per-block bulk data and an HTTP receiver is the wrong consumer + // for it. Tweaks stay on the streaming API. + "tweaks" => { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: the `tweaks` category is not deliverable over webhooks \ + (it is a per-block firehose); consume it on the streaming API" + ), + )); + } + other => { + return Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: unknown category `{other}` \ + (status, chain, mempool, heartbeat)" + ), + )); + } + }; + } + if mask == 0 { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `categories` must not be empty"), + )); + } + Ok(CategoryMask(mask)) +} + +fn parse_kinds( + path: &Path, + id: &str, + t: &toml_edit::Table, +) -> Result>, AlertFileError> { + let Some(v) = t.get("kinds") else { + return Ok(None); + }; + let arr = v.as_array().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `kinds` must be an array")) + })?; + let mut out = BTreeSet::new(); + for item in arr { + let name = item.as_str().ok_or_else(|| { + AlertFileError::invalid(path, format!("webhook `{id}`: `kinds` entries must be strings")) + })?; + // Validated against the live taxonomy, not a copy: a typo'd kind would + // otherwise match nothing and silently disable the hook. + let kind = StatusKind::from_str_exact(name).ok_or_else(|| { + AlertFileError::invalid( + path, + format!( + "webhook `{id}`: unknown status kind `{name}` (known: {})", + StatusKind::ALL + .iter() + .map(|k| k.as_str()) + .collect::>() + .join(", ") + ), + ) + })?; + out.insert(kind); + } + if out.is_empty() { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: `kinds` must not be empty (omit it to accept every kind)"), + )); + } + Ok(Some(out)) +} + +fn validate_url( + path: &Path, + id: &str, + url: &str, + allow_insecure_http: bool, +) -> Result<(), AlertFileError> { + if url.starts_with("https://") { + return Ok(()); + } + let Some(rest) = url.strip_prefix("http://") else { + return Err(AlertFileError::invalid( + path, + format!("webhook `{id}`: url must start with https:// or http://"), + )); + }; + if allow_insecure_http || is_local_target(rest) { + return Ok(()); + } + Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: plaintext http:// to a non-local address; use https://, or set \ + allow_insecure_http = true to accept it" + ), + )) +} + +/// Whether an `http://` authority is loopback or RFC1918 — the cases where +/// plaintext is normal (a relay on the same host, a receiver inside a private +/// network) and demanding TLS would just push operators to set the override. +fn is_local_target(rest: &str) -> bool { + let host = rest + .split('/') + .next() + .unwrap_or("") + .rsplit_once(':') + .map(|(h, _)| h) + .unwrap_or_else(|| rest.split('/').next().unwrap_or("")) + .trim_start_matches('[') + .trim_end_matches(']'); + if host == "localhost" || host == "::1" { + return true; + } + let Ok(ip) = host.parse::() else { + return false; + }; + match ip { + std::net::IpAddr::V4(v4) => v4.is_loopback() || v4.is_private() || v4.is_link_local(), + std::net::IpAddr::V6(v6) => v6.is_loopback() || v6.is_unique_local(), + } +} + +fn req_str(path: &Path, t: &toml_edit::Table, key: &str) -> Result { + t.get(key) + .and_then(|v| v.as_str()) + .map(str::to_string) + .ok_or_else(|| { + AlertFileError::invalid(path, format!("[[webhook]] is missing required string `{key}`")) + }) +} + +/// Refuse a group/world-accessible alertfile. Checked from the **open handle**, +/// not the path, so the permissions that are validated are the ones of the file +/// actually read (no TOCTOU window against a swap). +#[cfg(unix)] +fn check_perms(path: &Path, file: &std::fs::File) -> Result<(), AlertFileError> { + use std::os::unix::fs::MetadataExt; + let meta = file.metadata().map_err(|source| AlertFileError::Io { + path: path.to_path_buf(), + source, + })?; + let mode = meta.mode() & 0o777; + if mode & 0o077 != 0 { + return Err(AlertFileError::Permissions { + path: path.to_path_buf(), + mode, + }); + } + Ok(()) +} + +#[cfg(not(unix))] +fn check_perms(_path: &Path, _file: &std::fs::File) -> Result<(), AlertFileError> { + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn parse(text: &str) -> Result { + AlertFile::parse(Path::new("/test/alertfile"), text) + } + + const MINIMAL: &str = r#" +version = 1 +[[webhook]] +id = "ops" +url = "https://alerts.example/satd" +secret = "s3cret" +categories = ["status"] +"#; + + #[test] + fn parses_a_minimal_hook() { + let f = parse(MINIMAL).unwrap(); + assert_eq!(f.hooks.len(), 1); + let h = &f.hooks[0]; + assert_eq!(h.id, "ops"); + assert!(h.filter.categories.contains(node::events::CATEGORY_STATUS)); + assert!(h.filter.kinds.is_none()); + assert!(h.filter.min_severity.is_none()); + assert_eq!(h.heartbeat_interval_secs, None); + assert_eq!(h.cursor_key(), b"alertwebhook.cursor.ops".to_vec()); + } + + #[test] + fn parses_filters() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "pager" +url = "https://alerts.example/satd" +secret = "s" +categories = ["status", "chain"] +kinds = ["tip_stall", "disk_low"] +min_severity = "warning" +"#, + ) + .unwrap(); + let h = &f.hooks[0]; + assert!(h.filter.categories.contains(node::events::CATEGORY_STATUS)); + assert!(h.filter.categories.contains(node::events::CATEGORY_CHAIN)); + let kinds = h.filter.kinds.as_ref().unwrap(); + assert!(kinds.contains(&StatusKind::TipStall)); + assert!(!kinds.contains(&StatusKind::PeerFloor)); + assert_eq!(h.filter.min_severity, Some(StatusSeverity::Warning)); + } + + #[test] + fn status_filter_applies_kind_and_severity_floors() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "pager" +url = "https://x.example/h" +secret = "s" +categories = ["status"] +kinds = ["tip_stall", "peer_floor"] +min_severity = "critical" +"#, + ) + .unwrap(); + let filter = &f.hooks[0].filter; + assert!(filter.accepts_status(StatusKind::TipStall, StatusSeverity::Critical)); + // Right kind, too quiet. + assert!(!filter.accepts_status(StatusKind::PeerFloor, StatusSeverity::Warning)); + // Loud enough, wrong kind. + assert!(!filter.accepts_status(StatusKind::DiskLow, StatusSeverity::Critical)); + } + + #[test] + fn a_hook_without_the_status_category_never_takes_status_events() { + let f = parse( + r#" +version = 1 +[[webhook]] +id = "chain-only" +url = "https://x.example/h" +secret = "s" +categories = ["chain"] +"#, + ) + .unwrap(); + assert!(!f.hooks[0] + .filter + .accepts_status(StatusKind::TipStall, StatusSeverity::Critical)); + } + + #[test] + fn missing_or_wrong_version_is_rejected() { + assert!(matches!( + parse("[[webhook]]\nid=\"a\"\nurl=\"https://x/h\"\nsecret=\"s\"\ncategories=[\"status\"]\n"), + Err(AlertFileError::Invalid { .. }) + )); + assert!(parse(&MINIMAL.replace("version = 1", "version = 2")).is_err()); + } + + #[test] + fn unknown_keys_are_rejected_not_ignored() { + // An accepted-but-ignored alerting rule is worse than a refused one: + // the operator believes they are covered when they are not. + assert!(parse(&format!("{MINIMAL}\nnot_a_key = 1\n")).is_err()); + assert!(parse(&MINIMAL.replace("categories = [\"status\"]", "categories = [\"status\"]\nkindz = [\"tip_stall\"]")).is_err()); + } + + #[test] + fn unknown_categories_and_kinds_are_rejected() { + assert!(parse(&MINIMAL.replace("\"status\"", "\"blocks\"")).is_err()); + let with_kind = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nkinds = [\"tip_stal\"]", + ); + assert!(parse(&with_kind).is_err(), "a typo'd kind must not silently match nothing"); + } + + #[test] + fn tweaks_category_is_rejected_with_a_pointer_to_the_streaming_api() { + let err = parse(&MINIMAL.replace("\"status\"", "\"tweaks\"")).unwrap_err(); + assert!( + err.to_string().contains("streaming API"), + "the error should say where tweaks are consumed: {err}" + ); + } + + #[test] + fn secret_is_required_and_must_be_non_empty() { + assert!(parse(&MINIMAL.replace("secret = \"s3cret\"", "")).is_err()); + assert!(parse(&MINIMAL.replace("s3cret", "")).is_err()); + } + + #[test] + fn duplicate_ids_are_rejected() { + // Two hooks sharing an id would share a durable cursor key. + let two = format!("{MINIMAL}\n[[webhook]]\nid = \"ops\"\nurl = \"https://b.example/h\"\nsecret = \"t\"\ncategories = [\"chain\"]\n"); + let err = parse(&two).unwrap_err(); + assert!(err.to_string().contains("duplicate"), "{err}"); + } + + #[test] + fn ids_are_restricted_to_header_and_label_safe_characters() { + for bad in ["has space", "quote\"", "new\nline", ""] { + let text = MINIMAL.replace("id = \"ops\"", &format!("id = {bad:?}")); + assert!(parse(&text).is_err(), "id {bad:?} should be rejected"); + } + } + + #[test] + fn plaintext_http_is_gated_unless_local_or_opted_in() { + let remote = MINIMAL.replace("https://alerts.example/satd", "http://alerts.example/satd"); + assert!(parse(&remote).is_err(), "plaintext to a public host needs the opt-in"); + + // Loopback and RFC1918 are fine unannotated — a relay on the same host + // or inside a private network is the normal deployment. + for local in [ + "http://127.0.0.1:9000/hook", + "http://localhost:9000/hook", + "http://10.1.2.3/hook", + "http://192.168.0.9:8443/h", + "http://[::1]:9000/hook", + ] { + let text = MINIMAL.replace("https://alerts.example/satd", local); + assert!(parse(&text).is_ok(), "{local} should be accepted"); + } + + // And the escape hatch works. + let opted = remote.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nallow_insecure_http = true", + ); + assert!(parse(&opted).is_ok()); + } + + #[test] + fn non_http_schemes_are_rejected() { + for url in ["ftp://x/h", "file:///etc/passwd", "x.example/h", ""] { + let text = MINIMAL.replace("https://alerts.example/satd", url); + assert!(parse(&text).is_err(), "{url} should be rejected"); + } + } + + #[test] + fn heartbeat_interval_requires_the_heartbeat_category() { + // Otherwise the knob silently does nothing. + let text = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\"]\nheartbeat_interval_secs = 60", + ); + assert!(parse(&text).is_err()); + let ok = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\", \"heartbeat\"]\nheartbeat_interval_secs = 60", + ); + assert_eq!( + parse(&ok).unwrap().hooks[0].heartbeat_interval_secs, + Some(60) + ); + // Zero would mean "every heartbeat", i.e. 1 Hz of HTTP. + let zero = MINIMAL.replace( + "categories = [\"status\"]", + "categories = [\"status\", \"heartbeat\"]\nheartbeat_interval_secs = 0", + ); + assert!(parse(&zero).is_err()); + } + + #[test] + fn empty_file_with_only_a_version_is_valid() { + // "configured but no hooks" is a legitimate intermediate state during + // an edit; it must not fail the daemon. + assert!(parse("version = 1\n").unwrap().hooks.is_empty()); + } + + #[cfg(unix)] + #[test] + fn group_or_world_readable_file_is_refused() { + use std::io::Write as _; + use std::os::unix::fs::PermissionsExt as _; + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("alertfile"); + let mut f = std::fs::File::create(&path).unwrap(); + f.write_all(MINIMAL.as_bytes()).unwrap(); + drop(f); + + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + assert!( + matches!(AlertFile::load(&path), Err(AlertFileError::Permissions { .. })), + "a world-readable file holding signing secrets must be refused", + ); + + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + assert!(AlertFile::load(&path).is_ok()); + } +} diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs new file mode 100644 index 000000000..f6f4cf7cb --- /dev/null +++ b/satd-alert/src/contract.rs @@ -0,0 +1,115 @@ +//! The webhook delivery contract: header names, the idempotency key, and the +//! body signature. +//! +//! Normatively specified in `docs/api/webhooks.md`. The rules that matter: +//! +//! - The **body is the event**, byte-for-byte identical to the JSON a +//! WebSocket subscriber would receive for the same event. One documented +//! wire schema, not two — a receiver written against the streaming spec +//! parses webhook bodies with the same code. +//! - Delivery metadata rides in **headers**, never in the body. Putting the +//! attempt counter or hook id inside the JSON would change the bytes between +//! retries of the same event, which breaks both signature caching and +//! receiver-side deduplication. +//! - The signature is computed over the **raw request body**, so a receiver +//! verifies before parsing — the standard order, and the only one that is +//! safe against a parser that normalizes. + +use hmac::{Hmac, Mac}; +use sha2::Sha256; + +/// `sha256=` over the raw body, keyed by the hook's secret. Retained +/// unchanged from the shipped `reorgwebhook` so existing receivers keep working +/// when that hook is served by this dispatcher. +pub const SIGNATURE_HEADER: &str = "X-Satd-Signature"; +/// Idempotency key: `--`. Stable across retries of +/// one event, unique across restarts (the instance id is a per-process nonce). +pub const DELIVERY_HEADER: &str = "X-Satd-Delivery"; +/// Which configured hook this delivery belongs to. +pub const HOOK_HEADER: &str = "X-Satd-Hook"; +/// 1-based attempt counter. Present so a receiver can tell a retry from a +/// genuine duplicate event without keeping state. +pub const ATTEMPT_HEADER: &str = "X-Satd-Attempt"; +/// Contract version. Bumped only for a breaking change to the header set or +/// signature scheme — the *body* schema is versioned independently by the +/// event envelope's `schema_version`. +pub const WEBHOOK_VERSION_HEADER: &str = "X-Satd-Webhook-Version"; +pub const WEBHOOK_VERSION: &str = "1"; + +/// Sign a raw body with a hook secret, rendering the `X-Satd-Signature` value. +/// +/// HMAC-SHA256 over the exact bytes that go on the wire. The `sha256=` prefix +/// is part of the value (GitHub-webhook convention) so the algorithm can be +/// migrated later without a header rename. +pub fn sign_body(secret: &str, body: &[u8]) -> String { + let mut mac = as Mac>::new_from_slice(secret.as_bytes()) + .expect("HMAC accepts a key of any length"); + mac.update(body); + format!("sha256={}", hex::encode(mac.finalize().into_bytes())) +} + +/// Build the idempotency key for an event. +/// +/// All three components already exist on every envelope (`EdgeStamp.node_id`, +/// `EdgeStamp.seq`, and the publisher's per-process `instance_id`), so this +/// adds no state: it is stable across retries of one event because the stamp is +/// fixed at publish time, and unique across restarts because `instance_id` is. +pub fn delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { + format!("{node_id_hex}-{instance_id}-{seq}") +} + +#[cfg(test)] +mod tests { + use super::*; + + // Golden vectors. These are the contract: they are reproduced verbatim in + // `docs/api/webhooks.md` and in the reference push relay's tests, so an + // independent receiver implementation can verify its HMAC without running + // a node. Changing any expected value here is a breaking protocol change + // and must come with a `WEBHOOK_VERSION` bump. + #[test] + fn signature_golden_vectors() { + for (secret, body, expected) in [ + ( + "hunter2", + r#"{"hello":"world"}"#, + "sha256=12f1ef94c239895aafefa2a6804ec6136d8c23fff17c08064cfc75b33e3fbaf5", + ), + ( + "", + "", + "sha256=b613679a0814d9ec772f95d778c35fc5ff1697c493715653c6c712144292c5ad", + ), + ] { + assert_eq!(sign_body(secret, body.as_bytes()), expected, "secret={secret:?}"); + } + } + + #[test] + fn signature_covers_the_exact_bytes() { + // A single-byte change must change the signature — the point of signing + // the raw body rather than a parsed/normalized form. + let a = sign_body("k", br#"{"a":1}"#); + let b = sign_body("k", br#"{"a":2}"#); + assert_ne!(a, b); + // And whitespace is not normalized away. + assert_ne!(sign_body("k", br#"{"a":1}"#), sign_body("k", br#"{"a": 1}"#)); + } + + #[test] + fn signature_is_keyed() { + assert_ne!(sign_body("k1", b"body"), sign_body("k2", b"body")); + } + + #[test] + fn delivery_ids_are_stable_and_unique() { + let node = "ab".repeat(16); + // Same event, any attempt ⇒ same id (the caller does not vary it). + assert_eq!(delivery_id(&node, 7, 42), delivery_id(&node, 7, 42)); + // Different event ⇒ different id. + assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 7, 43)); + // Same seq after a restart ⇒ different id, because the instance nonce + // changed. Without this a receiver would dedupe away real events. + assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 8, 42)); + } +} diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs new file mode 100644 index 000000000..a5f9b1925 --- /dev/null +++ b/satd-alert/src/lib.rs @@ -0,0 +1,33 @@ +//! Operator alerting: the `alertfile` webhook configuration, the delivery +//! contract, and the retry policy. +//! +//! This crate is the pure half of the alerting subsystem. It parses and +//! validates an alertfile, decides whether a given event matches a hook, +//! renders the delivery headers, signs a body, and classifies a response as +//! delivered / retryable / dead. It does **not** open sockets: the dispatcher +//! task and its `reqwest` client live in the `satd` binary, so every rule here +//! is testable without a network, and the signature contract can be pinned by +//! golden vectors that a third-party receiver implementation can check itself +//! against. +//! +//! The split mirrors `satd-auth`, and for the same reason: policy that decides +//! *whether* and *how* something is delivered belongs somewhere a test can +//! reach without standing up a daemon. +//! +//! Module layout: +//! - [`config`] — the alertfile format, its validation rules, and permission +//! checks. +//! - [`contract`] — the wire contract: header names, delivery ids, HMAC +//! signing, and the version constant. +//! - [`retry`] — response classification and the backoff curve. + +pub mod config; +pub mod contract; +pub mod retry; + +pub use config::{AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY}; +pub use contract::{ + delivery_id, sign_body, ATTEMPT_HEADER, DELIVERY_HEADER, HOOK_HEADER, SIGNATURE_HEADER, + WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, +}; +pub use retry::{classify_response, retry_delay, Disposition, MAX_BACKOFF}; diff --git a/satd-alert/src/retry.rs b/satd-alert/src/retry.rs new file mode 100644 index 000000000..b1a64f3f4 --- /dev/null +++ b/satd-alert/src/retry.rs @@ -0,0 +1,141 @@ +//! Response classification and the retry curve. +//! +//! The governing constraint is that a webhook endpoint is *outside* the +//! operator's node and can behave arbitrarily badly. Nothing here may be able +//! to block, unbound, or spin: delivery is serial per hook, a hook's queue is +//! bounded, and a failing endpoint degrades that hook's delivery and nothing +//! else. + +use std::time::Duration; + +/// What to do with an attempt's outcome. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Disposition { + /// 2xx — the receiver acked. Advance the cursor. + Delivered, + /// Transient: retry this same event after a backoff. + Retry, + /// Permanent for this event: count it and move on. + /// + /// A misconfigured receiver (wrong path → 404, bad auth → 401, a body it + /// rejects → 400) would otherwise pin the head of the queue forever and + /// convert every subsequent event into a queue-overflow drop. Skipping the + /// event loses one delivery; retrying it forever loses all of them. + Drop, +} + +/// Longest backoff between attempts. Matches the peer-reconnect curve's cap: +/// an endpoint that has been down for five minutes is being restarted or +/// redeployed, and probing it faster than that helps nobody. +pub const MAX_BACKOFF: Duration = Duration::from_secs(300); + +/// Base delay before attempt `n` (1-based): 1s · 2^(n-1), capped at +/// [`MAX_BACKOFF`]. Jitter is applied separately by [`jitter`] so this stays a +/// pure function of the attempt number and can be asserted exactly. +pub fn retry_delay(attempt: u32) -> Duration { + // `attempt` is 1-based, so the first retry waits 1s. Saturating shift keeps + // a long-dead endpoint from overflowing into a zero delay. + let secs = 1u64.checked_shl(attempt.saturating_sub(1)).unwrap_or(u64::MAX); + Duration::from_secs(secs).min(MAX_BACKOFF) +} + +/// Spread a delay by up to ±25 %, given a caller-supplied random value. +/// +/// Randomness is a parameter rather than a call to a global RNG so the curve is +/// deterministic under test. The purpose is the usual one: a node with several +/// hooks pointed at the same receiver must not retry them all in lockstep. +pub fn jitter(base: Duration, rand: u64) -> Duration { + let base_ms = base.as_millis() as u64; + if base_ms == 0 { + return base; + } + let span = base_ms / 2; // ±25 % of base + let offset = rand % span.max(1); + Duration::from_millis(base_ms.saturating_sub(span / 2).saturating_add(offset)) +} + +/// Classify a delivery attempt. +/// +/// `status` is the HTTP status if one was received; `None` means the request +/// never produced a response (connect error, TLS failure, timeout) — always +/// retryable, since it says nothing about whether the receiver would accept the +/// event. +pub fn classify_response(status: Option) -> Disposition { + match status { + None => Disposition::Retry, + Some(s) if (200..300).contains(&s) => Disposition::Delivered, + // 408 Request Timeout and 429 Too Many Requests are explicitly + // "try again" despite being 4xx; 5xx is the server admitting fault. + Some(408) | Some(429) => Disposition::Retry, + Some(s) if (500..600).contains(&s) => Disposition::Retry, + // Every other 4xx is the receiver saying this request is wrong, and it + // will be just as wrong next time. + Some(_) => Disposition::Drop, + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn success_is_delivered() { + for s in [200, 201, 202, 204, 299] { + assert_eq!(classify_response(Some(s)), Disposition::Delivered, "{s}"); + } + } + + #[test] + fn server_errors_and_transport_failures_retry() { + for s in [500, 502, 503, 504, 599] { + assert_eq!(classify_response(Some(s)), Disposition::Retry, "{s}"); + } + assert_eq!(classify_response(None), Disposition::Retry, "no response"); + } + + #[test] + fn the_two_retryable_4xx_are_retried() { + assert_eq!(classify_response(Some(408)), Disposition::Retry); + assert_eq!(classify_response(Some(429)), Disposition::Retry); + } + + #[test] + fn client_errors_are_dropped_not_retried_forever() { + // A misconfigured receiver must not pin the head of the queue and + // convert every later event into an overflow drop. + for s in [400, 401, 403, 404, 410, 422] { + assert_eq!(classify_response(Some(s)), Disposition::Drop, "{s}"); + } + } + + #[test] + fn backoff_doubles_then_caps() { + assert_eq!(retry_delay(1), Duration::from_secs(1)); + assert_eq!(retry_delay(2), Duration::from_secs(2)); + assert_eq!(retry_delay(3), Duration::from_secs(4)); + assert_eq!(retry_delay(9), Duration::from_secs(256)); + assert_eq!(retry_delay(10), MAX_BACKOFF, "capped"); + // A hook that has been failing for days must stay at the cap, not + // overflow back to something small. + assert_eq!(retry_delay(64), MAX_BACKOFF); + assert_eq!(retry_delay(u32::MAX), MAX_BACKOFF); + } + + #[test] + fn jitter_stays_within_a_quarter_of_the_base() { + let base = Duration::from_secs(8); + for r in [0u64, 1, 999, u64::MAX / 2, u64::MAX] { + let j = jitter(base, r); + assert!( + j >= Duration::from_secs(6) && j <= Duration::from_secs(10), + "jitter({base:?}, {r}) = {j:?} escaped ±25%", + ); + } + } + + #[test] + fn jitter_actually_varies() { + let base = Duration::from_secs(8); + assert_ne!(jitter(base, 0), jitter(base, 1_234_567)); + } +} diff --git a/satd/Cargo.toml b/satd/Cargo.toml index b85586a65..91e7c032d 100644 --- a/satd/Cargo.toml +++ b/satd/Cargo.toml @@ -52,6 +52,7 @@ node-index = { path = "../node-index" } node-filter-index = { path = "../node-filter-index" } esplora-handlers = { path = "../esplora-handlers" } satd-auth = { path = "../satd-auth" } +satd-alert = { path = "../satd-alert" } electrum-proto = { path = "../electrum-proto" } tls-config = { path = "../tls-config" } satd-events = { path = "../events", features = ["grpc", "zmq", "ws"] } diff --git a/satd/src/alert.rs b/satd/src/alert.rs new file mode 100644 index 000000000..b56a9ff2d --- /dev/null +++ b/satd/src/alert.rs @@ -0,0 +1,890 @@ +//! The alert webhook dispatcher. +//! +//! One fan-in task reads the event bus; one delivery task per configured hook +//! owns an outbound HTTP client and a bounded queue. `satd-alert` holds the +//! rules (what matches, how it is signed, when to retry); everything here is +//! the plumbing that runs them against a real socket. +//! +//! # Invariants +//! +//! **Nothing blocks consensus.** The fan-in `try_send`s into bounded queues and +//! drops on overflow; the delivery tasks run on the isolated API runtime. There +//! is no path from a slow endpoint back to block connection — the isolation is +//! structural, not a timeout budget. (The dispatcher this replaces ran its +//! outbound HTTP on the consensus runtime.) +//! +//! **A gap is never silent.** Both drop paths — a full hook queue and a lagged +//! broadcast receiver — set the hook's gap flag. Before its next delivery the +//! hook emits a synthesized `Lagged` body carrying the number of events lost +//! and the cursor to resume from, so a receiver that cares can go fetch the +//! span it missed. +//! +//! **Delivery is serial and ordered per hook.** One request in flight at a +//! time, so a receiver observes events in the order the node produced them and +//! a retry cannot be overtaken by the event behind it. +//! +//! **A dead endpoint degrades only itself.** Retries back off to a five-minute +//! ceiling and never give up on a transient failure, but a permanent 4xx skips +//! the event rather than pinning the head of the queue — a receiver returning +//! 404 forever must not convert every subsequent event into an overflow drop. + +use std::sync::Arc; +use std::sync::atomic::Ordering; + +use node::events::{Cursor, EventPublisher, NodeEvent, NodeEventBody}; +use node::metrics::{HookCounters, WebhookMetrics}; +use node::storage::Store; +use satd_alert::{AlertFile, Hook, HOOK_QUEUE_CAPACITY}; +use tokio::sync::{mpsc, watch}; + +/// Hook id the legacy `reorgwebhook=` alias reports under, in metric labels and +/// the `X-Satd-Hook` header. Namespaced so it can never collide with an +/// operator-chosen alertfile id (`-` is allowed in ids, but this exact string +/// is documented as reserved). +pub const LEGACY_REORG_HOOK_ID: &str = "reorg-legacy"; + +/// Per-attempt HTTP timeout. Matches the shipped reorg webhook. +const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + +/// How far back a hook's startup catch-up may replay. Shares the streaming +/// API's clamp so a webhook receiver and a streaming client see the same +/// "resync below this height yourself" boundary. +const MAX_CATCHUP_BLOCKS: u32 = node::events::MAX_REPLAY_BLOCKS; + +/// Composite stop signal: the process-wide shutdown, plus a per-generation +/// channel the SIGHUP reload flips to retire the previous dispatcher. +/// +/// Two channels rather than one because a reload must stop *this* generation's +/// tasks without touching the global signal every other subsystem watches. +#[derive(Clone)] +pub struct Stop { + global: watch::Receiver, + generation: watch::Receiver, +} + +impl Stop { + fn stopped(&self) -> bool { + *self.global.borrow() || *self.generation.borrow() + } + + /// Resolve once either channel signals. Cancel-safe: `changed()` only + /// marks a value seen when it completes, and the loop re-checks both + /// borrows, so being dropped inside a `select!` loses nothing. + async fn wait(&mut self) { + loop { + if self.stopped() { + return; + } + tokio::select! { + _ = self.global.changed() => {} + _ = self.generation.changed() => {} + } + } + } +} + +/// What a hook's delivery task receives. +enum Delivery { + /// An event to deliver, pre-rendered to the exact bytes that will be + /// signed and sent. Rendering happens once in the fan-in rather than per + /// hook so a body is never serialized differently for two receivers. + Event { + body: Arc>, + delivery_id: String, + /// Cursor to persist once the receiver acks, if this event carries one. + cursor: Option, + }, +} + +struct HookChannel { + id: String, + hook: Hook, + tx: mpsc::Sender, + counters: Arc, + /// Set when an event was dropped for this hook; cleared when the resulting + /// `Lagged` notice has been queued. + gap: Arc, +} + +/// Accumulated drop state for one hook. +#[derive(Default)] +struct GapState { + dropped: std::sync::atomic::AtomicU64, + /// Position of the last event successfully queued before the gap — the + /// anchor a receiver resumes from. + resume_height: std::sync::atomic::AtomicU32, +} + +/// Spawn the dispatcher for a parsed alertfile. +/// +/// Must be called from within the API runtime — every task spawned here does +/// outbound HTTP, which is exactly what must never share the consensus runtime. +/// Returns `None` when the file configures no hooks, so a node with an empty +/// alertfile starts no tasks at all. +fn spawn_with_metrics( + file: &AlertFile, + publisher: Arc, + store: Arc, + block_source: Option>, + metrics: Arc, + stop: Stop, +) { + if file.hooks.is_empty() { + return; + } + let mut hooks = Vec::new(); + for hook in &file.hooks { + let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); + let counters = metrics.hook(&hook.id); + let gap = Arc::new(GapState::default()); + tokio::spawn(deliver_loop( + hook.clone(), + rx, + counters.clone(), + store.clone(), + stop.clone(), + )); + hooks.push(HookChannel { + id: hook.id.clone(), + hook: hook.clone(), + tx, + counters, + gap, + }); + } + tokio::spawn(fan_in(hooks, publisher, store, block_source, stop)); +} + +/// Fan-in: one broadcast receiver, filtered and enqueued per hook. +async fn fan_in( + hooks: Vec, + publisher: Arc, + store: Arc, + block_source: Option>, + mut stop: Stop, +) { + // Subscribe BEFORE the catch-up replay, so an event published while the + // replay is being built is buffered rather than lost in the seam. + let mut rx = publisher.subscribe(); + + for hook in &hooks { + catch_up(hook, &publisher, store.as_ref(), block_source.as_deref()).await; + } + + // Per-hook heartbeat downsampling state (D11): last forwarded instant. + let mut last_heartbeat: Vec> = vec![None; hooks.len()]; + + tracing::info!( + target: "alert", + hooks = hooks.len(), + "alert webhook dispatcher started", + ); + + loop { + tokio::select! { + _ = stop.wait() => break, + recv = rx.recv() => match recv { + Ok(env) => { + // Render once for every hook: the body a receiver verifies + // must be the bytes a WS subscriber would have seen, and + // rendering per hook risks two receivers disagreeing. + let Ok(bytes) = serde_json::to_vec(&env) else { + tracing::warn!(target: "alert", "skipping event: serialization failed"); + continue; + }; + let body = Arc::new(bytes); + let delivery_id = satd_alert::delivery_id( + &hex::encode(env.stamp.node_id), + publisher.instance_id(), + env.stamp.seq, + ); + for (i, hook) in hooks.iter().enumerate() { + if !accepts(&hook.hook, &env, &mut last_heartbeat[i]) { + continue; + } + enqueue(hook, &publisher, Delivery::Event { + body: body.clone(), + delivery_id: delivery_id.clone(), + cursor: env.cursor, + }, env.cursor); + } + } + Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => { + // The dispatcher itself fell behind the bus. Every hook + // missed the same span, so every hook is told. + tracing::warn!(target: "alert", dropped = n, "alert dispatcher lagged the event bus"); + for hook in &hooks { + hook.gap.dropped.fetch_add(n, Ordering::Relaxed); + hook.counters.dropped.fetch_add(n, Ordering::Relaxed); + } + } + Err(tokio::sync::broadcast::error::RecvError::Closed) => break, + }, + } + } + tracing::info!(target: "alert", "alert webhook dispatcher stopped"); +} + +/// Queue an event for one hook, converting an overflow into a recorded gap. +fn enqueue(hook: &HookChannel, publisher: &EventPublisher, item: Delivery, cursor: Option) { + // Emit the pending gap notice ahead of the event that follows it, so the + // receiver learns about the hole before it sees the data after it. + flush_gap(hook, publisher); + match hook.tx.try_send(item) { + Ok(()) => { + hook.counters + .queue_depth + .store(queue_depth(&hook.tx) as u64, Ordering::Relaxed); + if let Some(c) = cursor { + hook.gap.resume_height.store(c.height, Ordering::Relaxed); + } + } + Err(mpsc::error::TrySendError::Full(_)) => { + // A receiver that cannot keep up degrades to "you missed N" rather + // than back-pressuring the bus. + hook.gap.dropped.fetch_add(1, Ordering::Relaxed); + hook.counters.dropped.fetch_add(1, Ordering::Relaxed); + } + Err(mpsc::error::TrySendError::Closed(_)) => {} + } +} + +/// If events were dropped for this hook, queue a `Lagged` notice describing the +/// gap. Best-effort: if the queue is still full the notice waits for the next +/// opportunity, and the drop count keeps accumulating in the meantime. +fn flush_gap(hook: &HookChannel, publisher: &EventPublisher) { + let dropped = hook.gap.dropped.swap(0, Ordering::Relaxed); + if dropped == 0 { + return; + } + let resume = publisher.resume_cursor(hook.gap.resume_height.load(Ordering::Relaxed), 0); + let env = node::events::lagged_event(publisher, dropped, resume); + let Ok(bytes) = serde_json::to_vec(&env) else { + return; + }; + let item = Delivery::Event { + body: Arc::new(bytes), + delivery_id: satd_alert::delivery_id( + &hex::encode(env.stamp.node_id), + publisher.instance_id(), + env.stamp.seq, + ), + cursor: None, + }; + if hook.tx.try_send(item).is_err() { + // Still full — put the count back so the notice is not lost. + hook.gap.dropped.fetch_add(dropped, Ordering::Relaxed); + } +} + +fn queue_depth(tx: &mpsc::Sender) -> usize { + HOOK_QUEUE_CAPACITY.saturating_sub(tx.capacity()) +} + +/// Whether a hook wants this event, applying category, kind, severity, and +/// heartbeat-downsampling filters in that order. +fn accepts(hook: &Hook, env: &NodeEvent, last_heartbeat: &mut Option) -> bool { + match &env.body { + NodeEventBody::Status(s) => hook.filter.accepts_status(s.kind, s.severity), + NodeEventBody::Heartbeat { .. } => { + let Some(interval) = hook.heartbeat_interval_secs else { + return false; + }; + if !hook.filter.categories.contains(node::events::CATEGORY_HEARTBEAT) { + return false; + } + // The bus beats at 1 Hz; a dead-man's-switch receiver wants one + // ping per interval, not sixty. + let now = std::time::Instant::now(); + let due = last_heartbeat + .map(|t| now.duration_since(t).as_secs() >= interval) + .unwrap_or(true); + if due { + *last_heartbeat = Some(now); + } + due + } + // A lag notice is a control signal: it reaches every hook regardless of + // filter, exactly as it reaches every streaming subscriber. + NodeEventBody::Lagged { .. } => true, + other => { + // Tweaks are refused at parse (never in a hook's mask), and the + // remaining bodies map to their category bit. + let bit = match other { + NodeEventBody::Mempool(_) => node::events::CATEGORY_MEMPOOL, + NodeEventBody::Chain(_) => node::events::CATEGORY_CHAIN, + _ => return false, + }; + hook.filter.categories.contains(bit) + } + } +} + +/// Replay the confirmed events a hook missed while the daemon was down. +/// +/// Only the chain category has durable history to replay; status is re-raised +/// by the detectors instead, and mempool events are ephemeral by construction. +/// A cursor older than the clamp yields a leading `Lagged` notice — the same +/// deterministic "resync below this height yourself" signal a streaming client +/// gets. +async fn catch_up( + hook: &HookChannel, + publisher: &EventPublisher, + store: &dyn Store, + block_source: Option<&dyn node::events::BlockCursorSource>, +) { + if !hook + .hook + .filter + .categories + .contains(node::events::CATEGORY_CHAIN) + { + return; + } + let Some(src) = block_source else { + return; + }; + let Some(cursor) = read_cursor(store, &hook.hook) else { + // A hook that has never delivered starts at the live head rather than + // replaying history nobody asked for. + return; + }; + + let replay = node::events::build_cursor_replay( + src, + publisher, + cursor, + node::events::CATEGORY_CHAIN, + MAX_CATCHUP_BLOCKS, + None, + ); + if replay.clamped { + let dropped = u64::from(replay.earliest_replayed.saturating_sub(cursor.height)); + hook.gap.dropped.fetch_add(dropped, Ordering::Relaxed); + hook.counters.dropped.fetch_add(dropped, Ordering::Relaxed); + hook.gap.resume_height.store(cursor.height, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + from = cursor.height, + earliest = replay.earliest_replayed, + "webhook catch-up clamped; receiver must resync the older span itself", + ); + } + let count = replay.events.len(); + for env in replay.events { + let Ok(bytes) = serde_json::to_vec(&env) else { + continue; + }; + let cursor = env.cursor; + enqueue( + hook, + publisher, + Delivery::Event { + body: Arc::new(bytes), + delivery_id: satd_alert::delivery_id( + &hex::encode(env.stamp.node_id), + publisher.instance_id(), + env.stamp.seq, + ), + cursor, + }, + cursor, + ); + } + if count > 0 { + tracing::info!( + target: "alert", + hook = %hook.id, + events = count, + from = cursor.height, + "webhook catch-up queued events missed while the daemon was down", + ); + } +} + +fn read_cursor(store: &dyn Store, hook: &Hook) -> Option { + let raw = store.read_alert_cursor(&hook.cursor_key())?; + decode_cursor(&raw) +} + +/// Cursors are stored as a fixed 24-byte little-endian record rather than JSON: +/// the format is written and read only here, and a fixed layout cannot acquire +/// a parse failure mode as the `Cursor` type grows fields. +fn encode_cursor(c: &Cursor) -> [u8; 24] { + let mut out = [0u8; 24]; + out[0..4].copy_from_slice(&c.height.to_le_bytes()); + out[4..8].copy_from_slice(&c.tx_index.to_le_bytes()); + out[8..16].copy_from_slice(&c.mempool_seq.to_le_bytes()); + out[16..24].copy_from_slice(&c.instance_id.to_le_bytes()); + out +} + +fn decode_cursor(raw: &[u8]) -> Option { + if raw.len() != 24 { + return None; + } + Some(Cursor { + height: u32::from_le_bytes(raw[0..4].try_into().ok()?), + tx_index: u32::from_le_bytes(raw[4..8].try_into().ok()?), + mempool_seq: u64::from_le_bytes(raw[8..16].try_into().ok()?), + instance_id: u64::from_le_bytes(raw[16..24].try_into().ok()?), + }) +} + +/// One hook's delivery loop: serial, in-order, retrying with backoff. +async fn deliver_loop( + hook: Hook, + mut rx: mpsc::Receiver, + counters: Arc, + store: Arc, + mut stop: Stop, +) { + let client = match reqwest::Client::builder().timeout(REQUEST_TIMEOUT).build() { + Ok(c) => c, + Err(e) => { + tracing::error!(target: "alert", hook = %hook.id, error = %e, "failed to build webhook HTTP client; this hook will not deliver"); + return; + } + }; + // Only persist a cursor when it actually moves forward a block: the cursor + // is a resume hint, not a ledger, and one RocksDB write per delivered event + // would be pure write amplification. + let mut persisted_height: Option = None; + + loop { + let item = tokio::select! { + _ = stop.wait() => return, + item = rx.recv() => match item { + Some(i) => i, + None => return, + }, + }; + let Delivery::Event { + body, + delivery_id, + cursor, + } = item; + + counters + .queue_depth + .store(rx.len() as u64, Ordering::Relaxed); + + let signature = satd_alert::sign_body(&hook.secret, &body); + let mut attempt: u32 = 0; + loop { + attempt = attempt.saturating_add(1); + let result = client + .post(&hook.url) + .header(reqwest::header::CONTENT_TYPE, "application/json") + .header(satd_alert::SIGNATURE_HEADER, &signature) + .header(satd_alert::DELIVERY_HEADER, &delivery_id) + .header(satd_alert::HOOK_HEADER, &hook.id) + .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) + .header(satd_alert::WEBHOOK_VERSION_HEADER, satd_alert::WEBHOOK_VERSION) + .body(body.as_ref().clone()) + .send() + .await; + let status = match &result { + Ok(r) => Some(r.status().as_u16()), + Err(_) => None, + }; + match satd_alert::classify_response(status) { + satd_alert::Disposition::Delivered => { + counters.delivered.fetch_add(1, Ordering::Relaxed); + counters + .last_success_unix + .store(unix_secs(), Ordering::Relaxed); + if let Some(c) = cursor + && persisted_height != Some(c.height) + { + if let Err(e) = store.write_alert_cursor(&hook.cursor_key(), &encode_cursor(&c)) { + tracing::warn!(target: "alert", hook = %hook.id, error = %e, "failed to persist webhook cursor"); + } else { + persisted_height = Some(c.height); + } + } + break; + } + satd_alert::Disposition::Drop => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + status = ?status, + delivery = %delivery_id, + "webhook receiver rejected the delivery permanently; skipping this event", + ); + break; + } + satd_alert::Disposition::Retry => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + match &result { + Ok(r) => tracing::warn!(target: "alert", hook = %hook.id, status = %r.status(), attempt, "webhook delivery failed; retrying"), + Err(e) => tracing::warn!(target: "alert", hook = %hook.id, error = %e, attempt, "webhook request failed; retrying"), + } + let delay = satd_alert::retry::jitter( + satd_alert::retry_delay(attempt), + rand::random::(), + ); + tokio::select! { + _ = tokio::time::sleep(delay) => {} + _ = stop.wait() => return, + } + } + } + } + } +} + +fn unix_secs() -> u64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map(|d| d.as_secs()) + .unwrap_or(0) +} + +/// Owns the running dispatcher generation and re-spawns it on SIGHUP. +/// +/// `alertfile=` follows the `authfile=` model: the *path* is restart-only, the +/// *contents* are re-read on every SIGHUP even when no `bitcoin.conf` key +/// changed — editing a hook in place is the whole point. A parse or permission +/// error keeps the last-good dispatcher running, because the failure mode of +/// "your alerting silently stopped after a typo" is worse than the failure mode +/// of "your edit did not take effect and the log says why". +pub struct AlertReloader { + path: std::path::PathBuf, + /// The API runtime. The reload runs on the consensus runtime's signal loop, + /// so the new generation's tasks must be spawned onto the API runtime + /// explicitly rather than inherited from the caller. + api_handle: tokio::runtime::Handle, + publisher: Arc, + store: Arc, + block_source: Option>, + metrics: Arc, + global_stop: watch::Receiver, + /// Stop signal for the generation currently running, if any. + current: parking_lot::Mutex>>, +} + +impl AlertReloader { + #[allow(clippy::too_many_arguments)] + pub fn new( + path: std::path::PathBuf, + api_handle: tokio::runtime::Handle, + publisher: Arc, + store: Arc, + block_source: Option>, + metrics: Arc, + global_stop: watch::Receiver, + ) -> Self { + Self { + path, + api_handle, + publisher, + store, + block_source, + metrics, + global_stop, + current: parking_lot::Mutex::new(None), + } + } + + pub fn path(&self) -> &std::path::Path { + &self.path + } + + /// Load the alertfile and start a dispatcher generation, replacing any + /// generation already running. + /// + /// Returns the hook count on success. On failure the previous generation is + /// left untouched and the error is returned for the caller to log with the + /// right severity (fatal at startup, warn-and-continue on reload). + pub fn apply(&self) -> Result { + let file = AlertFile::load(&self.path)?; + let ids: Vec = file.hooks.iter().map(|h| h.id.clone()).collect(); + + // Retire the previous generation only once the new file has parsed, so + // a bad edit never leaves the node with no dispatcher at all. + let (gen_tx, gen_rx) = watch::channel(false); + let stop = Stop { + global: self.global_stop.clone(), + generation: gen_rx, + }; + let publisher = self.publisher.clone(); + let store = self.store.clone(); + let block_source = self.block_source.clone(); + let metrics = self.metrics.clone(); + let hook_count = file.hooks.len(); + { + let _guard = self.api_handle.enter(); + spawn_with_metrics(&file, publisher, store, block_source, metrics, stop); + } + if let Some(old) = self.current.lock().replace(gen_tx) { + // Draining is deliberate rather than graceful: the retired + // generation's queued events are dropped. A reload is an operator + // changing where alerts go, and delivering the backlog to the *old* + // endpoint after they redirected it is not what they asked for. + let _ = old.send(true); + } + // Stop exporting counters for hooks that are no longer configured, + // rather than freezing their series at the last value forever. + self.metrics.retain(&ids); + Ok(hook_count) + } +} + +/// Deliver legacy `reorgwebhook=` records. +/// +/// Absorbed into this module so it shares the API runtime and the HTTP client +/// shape, but **the body stays the shipped `ReorgRecord` JSON, byte for byte**. +/// A `ChainEvent::Reorg` envelope does not carry `depth`, `fork_height`, or the +/// disconnected/reconnected hash lists, so switching this hook to the envelope +/// schema would silently break every deployed receiver. Operators who want the +/// envelope shape configure a new-style hook with `categories = ["chain"]`. +pub async fn legacy_reorg_dispatcher( + webhook: crate::reload::SharedWebhook, + mut rx: mpsc::Receiver, + counters: Arc, +) { + let client = match reqwest::Client::builder().timeout(REQUEST_TIMEOUT).build() { + Ok(c) => c, + Err(e) => { + tracing::warn!(target: "alert", error = %e, "failed to build reorg webhook HTTP client"); + return; + } + }; + tracing::info!(target: "alert", "reorg webhook dispatcher started (legacy alias)"); + while let Some(record) = rx.recv().await { + // Re-read the live target per record so a SIGHUP that changes or + // removes the URL takes effect without a restart. The guard is dropped + // before any await. + let Some(target) = webhook.read().clone() else { + continue; + }; + let Ok(body) = serde_json::to_vec(&record) else { + tracing::warn!(target: "alert", "failed to serialize reorg record for webhook"); + continue; + }; + let mut attempt: u32 = 0; + loop { + attempt += 1; + let mut req = client + .post(&target.url) + .header(reqwest::header::CONTENT_TYPE, "application/json") + .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) + .header(satd_alert::HOOK_HEADER, LEGACY_REORG_HOOK_ID) + .header(satd_alert::WEBHOOK_VERSION_HEADER, satd_alert::WEBHOOK_VERSION); + if let Some(secret) = &target.secret { + req = req.header(satd_alert::SIGNATURE_HEADER, satd_alert::sign_body(secret, &body)); + } + let result = req.body(body.clone()).send().await; + let status = match &result { + Ok(r) => Some(r.status().as_u16()), + Err(_) => None, + }; + match satd_alert::classify_response(status) { + satd_alert::Disposition::Delivered => { + counters.delivered.fetch_add(1, Ordering::Relaxed); + counters.last_success_unix.store(unix_secs(), Ordering::Relaxed); + break; + } + d => { + counters.failed_attempts.fetch_add(1, Ordering::Relaxed); + match &result { + Ok(r) => tracing::warn!(target: "alert", status = %r.status(), attempt, "reorg webhook returned non-2xx"), + Err(e) => tracing::warn!(target: "alert", error = %e, attempt, "reorg webhook request failed"), + } + // Bounded retries, unchanged from the shipped behavior: the + // legacy hook has no queue to fall behind on, so a failing + // endpoint is given three tries and the record is dropped. + if d == satd_alert::Disposition::Drop || attempt >= 3 { + counters.dropped.fetch_add(1, Ordering::Relaxed); + break; + } + tokio::time::sleep(std::time::Duration::from_millis( + 200u64 * (1 << (attempt - 1)), + )) + .await; + } + } + } + } + tracing::info!(target: "alert", "reorg webhook dispatcher stopped"); +} + +#[cfg(test)] +mod tests { + use super::*; + use node::events::{StatusEvent, StatusKind, StatusSeverity}; + + fn hook_from(toml: &str) -> Hook { + AlertFile::parse(std::path::Path::new("/test"), toml) + .expect("valid alertfile") + .hooks + .remove(0) + } + + fn status_env(kind: StatusKind, severity: StatusSeverity) -> NodeEvent { + let mut ev = StatusEvent::raised(StatusKind::TipStall, "x"); + ev.kind = kind; + ev.severity = severity; + NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Status(ev), + ) + } + + const STATUS_HOOK: &str = r#" +version = 1 +[[webhook]] +id = "ops" +url = "https://x.example/h" +secret = "s" +categories = ["status"] +min_severity = "warning" +"#; + + #[test] + fn cursor_round_trips_through_its_fixed_encoding() { + let c = Cursor { + height: 812_345, + tx_index: 7, + mempool_seq: 0xDEAD_BEEF_CAFE, + instance_id: 0x0102_0304_0506_0708, + }; + assert_eq!(decode_cursor(&encode_cursor(&c)), Some(c)); + } + + #[test] + fn a_truncated_or_garbage_cursor_is_ignored_not_misread() { + // A corrupt cursor must degrade to "start at the live head", never to a + // wrong height that would replay or skip history. + assert_eq!(decode_cursor(&[]), None); + assert_eq!(decode_cursor(&[0u8; 23]), None); + assert_eq!(decode_cursor(&[0u8; 25]), None); + } + + #[test] + fn status_filter_is_applied_per_hook() { + let hook = hook_from(STATUS_HOOK); + let mut hb = None; + assert!(accepts( + &hook, + &status_env(StatusKind::DiskLow, StatusSeverity::Critical), + &mut hb + )); + // Below the severity floor. + assert!(!accepts( + &hook, + &status_env(StatusKind::IbdComplete, StatusSeverity::Info), + &mut hb + )); + } + + #[test] + fn a_status_only_hook_does_not_receive_chain_events() { + let hook = hook_from(STATUS_HOOK); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Chain(node::chain::events::ChainEvent::BlockConnected { + hash: bitcoin::BlockHash::from_raw_hash( + bitcoin::hashes::Hash::from_byte_array([3u8; 32]), + ), + height: 5, + }), + ); + let mut hb = None; + assert!(!accepts(&hook, &env, &mut hb)); + } + + #[test] + fn lag_notices_bypass_every_filter() { + // A receiver must learn it missed events even if the events it missed + // were in a category it does not subscribe to — otherwise its cursor + // silently diverges. + let hook = hook_from(STATUS_HOOK); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Lagged { + dropped_count: 3, + resume_cursor: Cursor { + height: 1, + tx_index: 0, + mempool_seq: 0, + instance_id: 1, + }, + }, + ); + let mut hb = None; + assert!(accepts(&hook, &env, &mut hb)); + } + + #[test] + fn heartbeats_are_downsampled_to_the_configured_interval() { + let hook = hook_from( + r#" +version = 1 +[[webhook]] +id = "deadman" +url = "https://x.example/h" +secret = "s" +categories = ["heartbeat"] +heartbeat_interval_secs = 3600 +"#, + ); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Heartbeat { uptime_ns: 1 }, + ); + let mut hb = None; + // The bus beats at 1 Hz; the first one goes through and the rest of the + // hour's worth do not. + assert!(accepts(&hook, &env, &mut hb)); + for _ in 0..100 { + assert!(!accepts(&hook, &env, &mut hb)); + } + } + + #[test] + fn a_hook_without_the_heartbeat_interval_gets_no_heartbeats() { + let hook = hook_from(STATUS_HOOK); + let env = NodeEvent::new( + node::events::EdgeStamp { + node_id: [7; 16], + region: None, + edge_seen_at_ns: 0, + edge_wall_ns: 0, + seq: 1, + }, + NodeEventBody::Heartbeat { uptime_ns: 1 }, + ); + let mut hb = None; + assert!(!accepts(&hook, &env, &mut hb)); + } +} diff --git a/satd/src/config.rs b/satd/src/config.rs index 050ca497c..9c80f4914 100644 --- a/satd/src/config.rs +++ b/satd/src/config.rs @@ -1008,6 +1008,10 @@ pub struct Config { /// node warning, with `%s` replaced by the warning text. `None` = no /// dispatcher. Read once at startup (restart to change), like Core. pub alert_notify: Option, + /// Path to the TOML alertfile describing outbound webhooks. `None` = no + /// dispatcher. The path is restart-only; the file's *contents* are re-read + /// on every SIGHUP (the `authfile` model). + pub alertfile: Option, /// Health detector: raise `tip_stall` after this many seconds with no /// connected block. Deliberately not gated on IBD — see /// `node::health::check_tip_stall_values`. `0` disables. SIGHUP-live. @@ -3273,6 +3277,10 @@ impl Config { .or_else(|| file_get("reorgwebhooksecret")), block_notify: cli.blocknotify.clone().or_else(|| file_get("blocknotify")), alert_notify: cli.alertnotify.clone().or_else(|| file_get("alertnotify")), + alertfile: cli + .alertfile + .clone() + .or_else(|| file_get("alertfile").map(PathBuf::from)), alert_tip_stall_seconds: cli .alert_tip_stall_seconds .or_else(|| file_get("alerttipstallseconds").and_then(|v| v.parse().ok())) @@ -5155,6 +5163,13 @@ pub struct CliArgs { )] pub reorg_webhook_secret: Option, + #[arg( + long = "alertfile", + value_name = "PATH", + help = "TOML file describing outbound alert webhooks (see the Operator Manual). Must be mode 0600 — it holds signing secrets." + )] + pub alertfile: Option, + #[arg( long = "alerttipstallseconds", value_name = "SECS", @@ -6213,7 +6228,8 @@ pub const KNOWN_CONFIG_KEYS: &[&str] = &[ "shutdownnotify", "reorgwebhook", "reorgwebhooksecret", - // Health-alert detector thresholds (A3) + // Health alerts + webhook dispatcher (A3) + "alertfile", "alerttipstallseconds", "alertdiskfreemb", "alertmempoolfullpct", @@ -7505,6 +7521,7 @@ rpcport=8332 shutdownnotify: None, reorg_webhook: None, reorg_webhook_secret: None, + alertfile: None, alert_tip_stall_seconds: None, alert_disk_free_mb: None, alert_mempool_full_pct: None, @@ -7789,6 +7806,7 @@ rpcport=8332 shutdownnotify: None, reorg_webhook: None, reorg_webhook_secret: None, + alertfile: None, alert_tip_stall_seconds: None, alert_disk_free_mb: None, alert_mempool_full_pct: None, diff --git a/satd/src/main.rs b/satd/src/main.rs index a7a972e94..976ebf8e5 100644 --- a/satd/src/main.rs +++ b/satd/src/main.rs @@ -1,3 +1,4 @@ +mod alert; mod config; mod fast_start; mod notify; @@ -685,6 +686,12 @@ async fn main() { // of claiming success. Handed to `ReloadHandles` further down. let mut reorg_webhook_handle: Option = None; + // Shared per-hook delivery counters, rendered on `/metrics`. Created here + // because both the legacy reorg hook (below) and the alertfile dispatcher + // (further down) register into it; a node with neither configured leaves it + // empty, and an empty registry renders nothing. + let webhook_metrics = Arc::new(node::metrics::WebhookMetrics::new()); + // Open reorg log + webhook dispatcher. Failure is non-fatal — the node // still runs, just without persistent reorg history (and, since reorgs are // then not recorded, without webhook delivery either). @@ -704,7 +711,18 @@ async fn main() { Arc::new(parking_lot::RwLock::new(reload::webhook_target_from(&config))); let (tx, rx) = tokio::sync::mpsc::channel::(64); reorg_log.set_webhook_sender(tx); - tokio::spawn(reorg_webhook_dispatcher(shared.clone(), rx)); + // On the API runtime, not the consensus core: this task does + // outbound HTTP to an endpoint satd does not control, and the + // previous placement (a plain `tokio::spawn` here) put that on the + // same runtime as block connection. + { + let _api_guard = api_handle.enter(); + tokio::spawn(alert::legacy_reorg_dispatcher( + shared.clone(), + rx, + webhook_metrics.hook(alert::LEGACY_REORG_HOOK_ID), + )); + } chain_state.attach_reorg_log(reorg_log); reorg_webhook_handle = Some(shared); } @@ -2164,6 +2182,43 @@ async fn main() { } }; + // Outbound alert webhooks (`alertfile=`). The dispatcher is a plain event-bus + // consumer: it starts only when a file is configured, so a node without one + // spawns no tasks and its `/metrics` page is unchanged. + // + // A bad alertfile is fatal AT STARTUP (the operator explicitly configured + // alerting; coming up "successfully" with none is the failure they would + // notice last), but only a warning on SIGHUP, where the last-good set keeps + // running. + let alert_reloader = config.alertfile.clone().map(|path| { + std::sync::Arc::new(alert::AlertReloader::new( + path, + api_handle.clone(), + event_publisher.clone(), + chain_state.store_ref().clone() as std::sync::Arc, + // Block source for a hook's startup catch-up replay: read-only + // active-chain access, the same handle the streaming carriers use. + Some(chain_state.clone() as std::sync::Arc), + webhook_metrics.clone(), + shutdown_rx.clone(), + )) + }); + if let Some(reloader) = &alert_reloader { + match reloader.apply() { + Ok(hooks) => tracing::info!( + target: "alert", + hooks, + path = %reloader.path().display(), + "alertfile loaded", + ), + Err(e) => { + tracing::error!(target: "alert", error = %e, "alertfile is unusable"); + auth.cleanup(); + std::process::exit(1); + } + } + } + // Start MCP server if enabled if config.mcp { let mcp_ctx = std::sync::Arc::new(satd_mcp::McpContext { @@ -2178,6 +2233,7 @@ async fn main() { addr_enabled: config.addressindex, addr_subs: Some(address_index_concrete.subscription_registry()), health: health_state.clone(), + webhooks: Some(webhook_metrics.clone()), }); // `--mcp` only enables the feature; `--mcpport` provides the transport. @@ -2288,6 +2344,7 @@ async fn main() { addr_subs: Some(address_index_concrete.subscription_registry()), addr_enabled: config.addressindex, health: health_state.clone(), + webhooks: Some(webhook_metrics.clone()), }; let rx = shutdown_rx.clone(); api_handle.spawn(async move { @@ -3088,6 +3145,7 @@ async fn main() { rpc_auth: auth.clone(), token_store: token_store.clone(), alert_thresholds: alert_thresholds.clone(), + alert_reloader: alert_reloader.clone(), }; loop { tokio::select! { @@ -3427,92 +3485,6 @@ fn reload_tls_certificates() { tracing::info!(reloaded, failed, "TLS certificate reload complete"); } -/// Forwards reorg records to the configured HTTP webhook. Best effort — -/// failures are logged and dropped. Never blocks the consensus path: -/// the only backpressure is the channel itself, which `ReorgLog::record` -/// `try_send`s into (full queue = silent drop, counted). -/// -/// The target (URL + optional signing secret) is read from `webhook` per -/// record, so a SIGHUP reload that changes or removes it takes effect on the -/// next reorg without restarting this task. When the target is `None` the -/// record is drained and dropped (no webhook configured). -async fn reorg_webhook_dispatcher( - webhook: reload::SharedWebhook, - mut rx: tokio::sync::mpsc::Receiver, -) { - use reqwest::header::{HeaderMap, HeaderValue}; - let client = match reqwest::Client::builder() - .timeout(std::time::Duration::from_secs(10)) - .build() - { - Ok(c) => c, - Err(e) => { - tracing::warn!(error = %e, "Failed to build reorg webhook HTTP client"); - return; - } - }; - tracing::info!("Reorg webhook dispatcher started"); - while let Some(record) = rx.recv().await { - // Snapshot the live target. `None` => no webhook configured right now; - // drop the record. The guard is released before any `.await`. - let target = match webhook.read().clone() { - Some(t) => t, - None => continue, - }; - let url = target.url; - let secret = target.secret; - - let body = match serde_json::to_vec(&record) { - Ok(b) => b, - Err(e) => { - tracing::warn!(error = %e, "Failed to serialize reorg record for webhook"); - continue; - } - }; - - let mut headers = HeaderMap::new(); - headers.insert( - reqwest::header::CONTENT_TYPE, - HeaderValue::from_static("application/json"), - ); - if let Some(ref key) = secret { - let sig = hmac_sha256_hex(key.as_bytes(), &body); - if let Ok(h) = HeaderValue::from_str(&format!("sha256={}", sig)) { - headers.insert("X-Satd-Signature", h); - } - } - - // Simple retry loop: 3 attempts with jittered backoff. A failing - // webhook must not back up consensus, so we stop after 3 and move - // on to the next record. - let mut attempt = 0u32; - loop { - attempt += 1; - match client - .post(&url) - .headers(headers.clone()) - .body(body.clone()) - .send() - .await - { - Ok(r) if r.status().is_success() => break, - Ok(r) => { - tracing::warn!(status = %r.status(), attempt, "Reorg webhook returned non-2xx"); - } - Err(e) => { - tracing::warn!(error = %e, attempt, "Reorg webhook request failed"); - } - } - if attempt >= 3 { - break; - } - let backoff_ms = 200u64 * (1 << (attempt - 1)); - tokio::time::sleep(std::time::Duration::from_millis(backoff_ms)).await; - } - } - tracing::info!("Reorg webhook dispatcher stopped"); -} - /// Address-index backfill supervisor. Owns serialization (one runner /// at a time) and crash recovery (auto-respawns on startup if the /// persisted cursor state is Running and the index is enabled). @@ -3912,10 +3884,3 @@ async fn persist_sp_failed_with_cleanup( } } -fn hmac_sha256_hex(key: &[u8], msg: &[u8]) -> String { - use bitcoin::hashes::{Hash, HashEngine, Hmac, HmacEngine, sha256}; - let mut hmac: HmacEngine = HmacEngine::new(key); - hmac.input(msg); - let out = Hmac::::from_engine(hmac); - hex::encode(out.to_byte_array()) -} diff --git a/satd/src/reload.rs b/satd/src/reload.rs index 6532afe42..bcdefc9d9 100644 --- a/satd/src/reload.rs +++ b/satd/src/reload.rs @@ -181,6 +181,10 @@ pub struct ReloadHandles { /// are rotatable live (the cookie is preserved). Shared with every RPC /// listener surface, so a reload covers all of them at once. pub rpc_auth: Arc, + /// Alertfile dispatcher, present only when `alertfile=` is set. Re-read on + /// every SIGHUP independently of the rest of the config, so editing a hook + /// in place takes effect — the `TokenStore` precedent. + pub alert_reloader: Option>, /// Health-detector raise thresholds. Always present (the detector task is /// always configured, even when every threshold is 0), so a threshold /// change always has somewhere to land. @@ -416,8 +420,13 @@ const LOAD_ONLY_KEYS: &[&str] = &[ /// /// `policyfile`: the transaction-filtering ruleset — re-read, recompiled, swapped, /// and re-placed live (path change, in-place content edit, and removal all live). +/// +/// `alertfile`: the outbound webhook set — re-read, re-validated, and the +/// dispatcher generation replaced. A parse/permission error keeps the last-good +/// dispatcher running (silently losing alerting after a typo is the worse +/// failure); the *path* itself is restart-only. #[allow(dead_code)] -const HANDLER_RELOAD_KEYS: &[&str] = &["policyfile"]; +const HANDLER_RELOAD_KEYS: &[&str] = &["policyfile", "alertfile"]; /// Build the field-disposition table. Constructed at call time (once per /// reload) so the non-capturing `diff`/`apply` closures coerce to `fn` @@ -990,6 +999,25 @@ pub fn reload_from_sighup(handles: &ReloadHandles, running: &Config) -> Config { } } + // Outbound alert webhooks, same precedent: re-read the file's *contents* + // on every SIGHUP, even when the path is unchanged. A failure keeps the + // last-good dispatcher — an operator whose edit did not apply reads it in + // the log, whereas an operator whose alerting silently stopped finds out + // the next time something breaks. + if let Some(reloader) = &handles.alert_reloader { + match reloader.apply() { + Ok(hooks) => tracing::info!( + hooks, + path = %reloader.path().display(), + "alertfile reloaded" + ), + Err(e) => tracing::error!( + error = %e, + "alertfile reload failed — keeping the last-good webhook set" + ), + } + } + // Transaction-filtering policy file (§8), the `TokenStore` precedent: // re-read the external file's *contents* on every SIGHUP — even when the // path is unchanged — recompile, swap, and run the lossless re-placement diff --git a/satd/tests/e2e.rs b/satd/tests/e2e.rs index e7f95716e..15803a478 100644 --- a/satd/tests/e2e.rs +++ b/satd/tests/e2e.rs @@ -23,6 +23,10 @@ mod streaming; // against a live `satd`. #[path = "e2e/sdk.rs"] mod sdk; +// Alert webhook dispatcher E2E: a real satd delivering signed POSTs to a +// scriptable loopback receiver. +#[path = "e2e/webhooks.rs"] +mod webhooks; use common::{ DeterministicWallet, TestNode, build_signed_p2wpkh_spend_from_block1_coinbase, diff --git a/satd/tests/e2e/streaming.rs b/satd/tests/e2e/streaming.rs index c3273e044..6e6fb9c17 100644 --- a/satd/tests/e2e/streaming.rs +++ b/satd/tests/e2e/streaming.rs @@ -44,12 +44,23 @@ fn mine_addr(seed: u8) -> String { /// readiness with `reqwest::blocking`, whose internal runtime panics if dropped /// on a tokio worker thread — so the (blocking) startup runs on a blocking /// task. -async fn start_streaming_async(args: Vec<&'static str>) -> StreamingNode { +pub(crate) async fn start_streaming_async(args: Vec<&'static str>) -> StreamingNode { tokio::task::spawn_blocking(move || StreamingNode::start(&args)) .await .unwrap() } +/// Like [`start_streaming_async`] but for arguments computed at runtime (a +/// temp path, an allocated port), which cannot be `&'static str` literals. +pub(crate) async fn start_streaming_owned(args: Vec) -> StreamingNode { + tokio::task::spawn_blocking(move || { + let refs: Vec<&str> = args.iter().map(String::as_str).collect(); + StreamingNode::start(&refs) + }) + .await + .unwrap() +} + /// `getserverstatus` reports the runtime-bound gRPC + streamws listener /// addresses, and the harness discovers them. This exercises the /// `ServerListenerStatus` wiring (no client needed). @@ -288,7 +299,7 @@ async fn coinbase1(sn: &StreamingNode) -> String { .unwrap() } -async fn mine_n(sn: &StreamingNode, n: u32) { +pub(crate) async fn mine_n(sn: &StreamingNode, n: u32) { let rpc = sn.node.rpc_handle(); let addr = mine_addr(0x99); tokio::task::spawn_blocking(move || rpc.mine(n, &addr)) @@ -1792,8 +1803,16 @@ async fn grpc_watch_txid_unconfirmed_on_invalidateblock() { /// authoritative), so the value is written there — which means it must not have /// been passed on the command line as anything other than its startup default. async fn set_alert_threshold(sn: &StreamingNode, key: &str, value: &str) { + sighup_with_conf(sn, &format!("{key}={value}\n")).await; +} + +/// Replace the node's `bitcoin.conf` `[regtest]` section and SIGHUP, waiting +/// for the reload to land. Only the config *file* is re-read on SIGHUP (startup +/// CLI args stay authoritative), so a key set on the command line cannot be +/// changed this way. +pub(crate) async fn sighup_with_conf(sn: &StreamingNode, body: &str) { let conf = sn.node.datadir.join("bitcoin.conf"); - std::fs::write(&conf, format!("[regtest]\n{key}={value}\n")).expect("write bitcoin.conf"); + std::fs::write(&conf, format!("[regtest]\n{body}")).expect("write bitcoin.conf"); let pid = sn.node.process.id().to_string(); let status = tokio::task::spawn_blocking(move || { std::process::Command::new("kill") diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs new file mode 100644 index 000000000..30fd73e9e --- /dev/null +++ b/satd/tests/e2e/webhooks.rs @@ -0,0 +1,343 @@ +//! Alert webhook dispatcher E2E: a real `satd` POSTing to a real socket. +//! +//! The dispatcher's rules (matching, signing, retry classification) are +//! unit-tested in `satd-alert`. What only an end-to-end test can establish is +//! that a condition detected inside the daemon reaches an external HTTP +//! receiver with the right bytes and headers, that a failing receiver is +//! retried rather than dropped, and that a permanently-broken one cannot wedge +//! the queue behind it. +//! +//! The receiver is a raw TCP listener rather than an HTTP framework: the tests +//! need to script exact status codes per request and hold a connection open +//! without responding, which is easier to do with 20 lines of socket code than +//! to coax out of a server library. + +use std::collections::HashMap; +use std::sync::Arc; +use std::time::Duration; + +use tokio::io::{AsyncReadExt as _, AsyncWriteExt as _}; +use tokio::net::TcpListener; +use tokio::sync::Mutex; + +use crate::common::StreamingNode; + +/// One captured POST. +#[derive(Debug, Clone)] +pub struct Received { + pub headers: HashMap, + pub body: String, +} + +impl Received { + fn json(&self) -> serde_json::Value { + serde_json::from_str(&self.body).expect("webhook body is JSON") + } + + fn header(&self, name: &str) -> &str { + self.headers + .get(&name.to_ascii_lowercase()) + .map(String::as_str) + .unwrap_or_default() + } + + /// Verify `X-Satd-Signature` over the raw body, exactly as a third-party + /// receiver would. + fn signature_valid(&self, secret: &str) -> bool { + self.header("x-satd-signature") == satd_alert::sign_body(secret, self.body.as_bytes()) + } +} + +/// How the mock receiver answers the next request. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Behavior { + Ok, + /// Respond with this status for the first `n` requests, then 200. + FailFirst(u16, usize), +} + +struct Inner { + behavior: Behavior, + seen: usize, + received: Vec, +} + +/// A scriptable HTTP receiver on loopback. +pub struct MockReceiver { + port: u16, + inner: Arc>, +} + +impl MockReceiver { + async fn start(behavior: Behavior) -> Self { + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let port = listener.local_addr().unwrap().port(); + let inner = Arc::new(Mutex::new(Inner { + behavior, + seen: 0, + received: Vec::new(), + })); + let task_inner = inner.clone(); + tokio::spawn(async move { + loop { + let Ok((mut sock, _)) = listener.accept().await else { + return; + }; + let inner = task_inner.clone(); + tokio::spawn(async move { + let Some(req) = read_request(&mut sock).await else { + return; + }; + let status = { + let mut g = inner.lock().await; + g.seen += 1; + let status = match g.behavior { + Behavior::Ok => 200, + Behavior::FailFirst(code, n) if g.seen <= n => code, + Behavior::FailFirst(..) => 200, + }; + // Only a request the receiver actually accepted counts + // as received; a 503'd attempt is a retry, not a + // delivery. + if status == 200 { + g.received.push(req); + } + status + }; + let reason = if status == 200 { "OK" } else { "Error" }; + let _ = sock + .write_all( + format!("HTTP/1.1 {status} {reason}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n") + .as_bytes(), + ) + .await; + let _ = sock.flush().await; + }); + } + }); + Self { port, inner } + } + + pub async fn ok() -> Self { + Self::start(Behavior::Ok).await + } + + /// Reject the first `n` requests with `code`, then accept. + pub async fn failing_first(code: u16, n: usize) -> Self { + Self::start(Behavior::FailFirst(code, n)).await + } + + pub fn url(&self) -> String { + format!("http://127.0.0.1:{}/hook", self.port) + } + + /// Wait for a delivery matching `pred`, up to `secs`. + async fn wait_for( + &self, + secs: u64, + pred: impl Fn(&Received) -> bool, + ) -> Received { + let deadline = std::time::Instant::now() + Duration::from_secs(secs); + loop { + if let Some(r) = self + .inner + .lock() + .await + .received + .iter() + .find(|r| pred(r)) + .cloned() + { + return r; + } + assert!( + std::time::Instant::now() < deadline, + "timed out waiting for a matching webhook delivery; got: {:?}", + self.inner.blocking_lock_bodies() + ); + tokio::time::sleep(Duration::from_millis(100)).await; + } + } + + async fn attempts(&self) -> usize { + self.inner.lock().await.seen + } +} + +trait BodiesForPanic { + fn blocking_lock_bodies(&self) -> Vec; +} + +impl BodiesForPanic for Mutex { + fn blocking_lock_bodies(&self) -> Vec { + self.try_lock() + .map(|g| g.received.iter().map(|r| r.body.clone()).collect()) + .unwrap_or_default() + } +} + +/// Read one HTTP request (headers + Content-Length body) off a socket. +async fn read_request(sock: &mut tokio::net::TcpStream) -> Option { + let mut buf = Vec::new(); + let mut tmp = [0u8; 4096]; + // Headers first. + let head_end = loop { + let n = sock.read(&mut tmp).await.ok()?; + if n == 0 { + return None; + } + buf.extend_from_slice(&tmp[..n]); + if let Some(pos) = find_double_crlf(&buf) { + break pos; + } + }; + let head = String::from_utf8_lossy(&buf[..head_end]).to_string(); + let mut headers = HashMap::new(); + for line in head.lines().skip(1) { + if let Some((k, v)) = line.split_once(':') { + headers.insert(k.trim().to_ascii_lowercase(), v.trim().to_string()); + } + } + let len: usize = headers + .get("content-length") + .and_then(|v| v.parse().ok()) + .unwrap_or(0); + let mut body = buf[head_end + 4..].to_vec(); + while body.len() < len { + let n = sock.read(&mut tmp).await.ok()?; + if n == 0 { + break; + } + body.extend_from_slice(&tmp[..n]); + } + Some(Received { + headers, + body: String::from_utf8_lossy(&body).to_string(), + }) +} + +fn find_double_crlf(buf: &[u8]) -> Option { + buf.windows(4).position(|w| w == b"\r\n\r\n") +} + +const SECRET: &str = "an-operator-chosen-signing-secret"; + +/// Start a node whose `alertfile=` points at a one-hook file delivering +/// `categories` to `receiver`. +/// +/// The alertfile *path* is restart-only (the `authfile` model — the dispatcher +/// binds to a path at startup), so it must be on the command line rather than +/// SIGHUP'd in. The file lives in its own temp dir, which the returned guard +/// keeps alive for the test's duration. +async fn node_with_hook( + receiver: &MockReceiver, + id: &str, + categories: &str, + extra: Vec, +) -> (StreamingNode, tempfile::TempDir) { + use std::os::unix::fs::PermissionsExt as _; + let dir = tempfile::tempdir().expect("tempdir"); + let path = dir.path().join("alertfile.toml"); + std::fs::write( + &path, + format!( + r#"version = 1 +[[webhook]] +id = "{id}" +url = "{}" +secret = "{SECRET}" +categories = [{categories}] +"#, + receiver.url() + ), + ) + .expect("write alertfile"); + // The dispatcher refuses a group/world-readable file; it holds the signing + // secret. + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + + let mut args = vec![format!("--alertfile={}", path.display())]; + args.extend(extra); + let sn = crate::streaming::start_streaming_owned(args).await; + (sn, dir) +} + +// =========================================================================== + +/// A detected condition reaches the receiver as a correctly-signed POST whose +/// body is the same JSON a streaming subscriber would have received. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_delivers_a_signed_status_event() { + let receiver = MockReceiver::ok().await; + let (sn, _dir) = node_with_hook(&receiver, "ops", "\"status\"", vec![]).await; + + // Raise `disk_low` *after* the dispatcher is attached, by moving the + // threshold live rather than starting with it tripped: a status event is + // not replayable, so a condition raised during startup would never reach a + // hook that had not finished subscribing. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + + assert!( + got.signature_valid(SECRET), + "X-Satd-Signature must verify over the raw body", + ); + assert_eq!(got.header("x-satd-hook"), "ops"); + assert_eq!(got.header("x-satd-webhook-version"), "1"); + assert_eq!(got.header("x-satd-attempt"), "1"); + assert!( + !got.header("x-satd-delivery").is_empty(), + "an idempotency key is always present", + ); + assert_eq!(got.header("content-type"), "application/json"); + + let body = got.json(); + assert_eq!(body["body"]["kind"], "disk_low", "body: {body}"); + assert_eq!(body["body"]["state"], "raised"); + // The same envelope a WS subscriber sees — schema version and stamp too. + assert_eq!(body["schema_version"], 1); + assert!(body["stamp"]["seq"].is_number()); +} + +/// A receiver that is down comes back to find the event still waiting: the +/// dispatcher retries with backoff instead of dropping. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_retries_until_the_receiver_recovers() { + // 503 the first two attempts, then accept. Backoff is 1s then 2s, so the + // delivery lands a few seconds in. + let receiver = MockReceiver::failing_first(503, 2).await; + let (sn, _dir) = node_with_hook(&receiver, "flaky", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 1).await; + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "chain") + .await; + assert!(got.signature_valid(SECRET)); + // The attempt counter proves this was a retry rather than a fresh event — + // and it is what lets a receiver tell the two apart without keeping state. + let attempt: u32 = got.header("x-satd-attempt").parse().unwrap_or(0); + assert!(attempt >= 3, "expected the 3rd attempt to succeed, got {attempt}"); + assert!(receiver.attempts().await >= 3); +} + +/// A receiver that permanently rejects one event must not pin the queue: the +/// event is skipped and later events still arrive. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_permanent_rejection_does_not_wedge_the_queue() { + // 404 the first request only. If a permanent 4xx were retried forever the + // second event would never be delivered. + let receiver = MockReceiver::failing_first(404, 1).await; + let (sn, _dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 2).await; + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "chain") + .await; + assert_eq!(got.json()["body"]["kind"], "block_connected"); + assert!(got.signature_valid(SECRET)); + // Exactly one rejection, and it did not stall what followed. + assert!(receiver.attempts().await >= 2); +} From 87e53033c19829178fe7c231227c896f9019b94f Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Fri, 24 Jul 2026 16:27:08 -0600 Subject: [PATCH 02/11] alerting: satd-alert crate + webhook dispatcher, absorbing reorgwebhook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-3 review fixes folded in. Two of these change the wire contract; both are pre-publication, which is the cheapest moment to change them. - **Signature now covers the delivery metadata, not just the body (v2).** The contract tells receivers to deduplicate on `X-Satd-Delivery`, but v1 signed the body alone, leaving that header unauthenticated — and its value is fully predictable from any single observed delivery, since `seq` is dense. Anyone holding one valid (body, signature) pair could replay it under forged future ids, filling a receiver's dedup cache so the genuine alerts bearing those ids were discarded on arrival, while satd counted them delivered and advanced its cursor. The operator's pager simply never fires and nothing logs a problem. v2 signs "2\n\n\n\n" and adds `X-Satd-Timestamp`, which also bounds replay of the capture itself. Golden vectors computed independently rather than captured from this implementation. The legacy `-reorgwebhook` alias stays on v1 byte-for-byte — it shipped, and deployed receivers verify exactly that; the version header distinguishes them. - **Catch-up replay gave every event the same idempotency key.** Replayed envelopes are stamped `seq: 0` by the replay builder, and the delivery id was minted from that stamp — so a node down for 100 blocks replayed 100 events that a conforming receiver (including our own relay) collapses into one. The durability feature was delivering 1% of what it advertises, silently. Same collision swallowed every gap notice after the first, so "a gap is never silent" held exactly once per process. Synthesized deliveries now get their own `-r` id space, disjoint from the bus and watch spaces. - **No IBD gate.** The dispatcher starts ~600 lines before P2P, so a fresh node with an alertfile POSTed its entire initial sync — one delivery per historical block. The manual promises the opposite. Suppress the firehose during IBD; status events stay exempt, since "this node is unhealthy" is as true during a sync. - **A SIGHUP unrelated to the alertfile destroyed queued deliveries.** `apply()` ran on every SIGHUP whatever key was edited, and retiring a generation drops its queue. Chain events recover via catch-up, but status events have no replay and the detectors are edge-triggered against a `HealthState` that outlives the reload — so a `disk_low` in retry backoff was lost permanently and never re-raised. An unchanged alertfile is now a no-op. - **Catch-up discarded the snapshot->live boundary dedup** that both shipped carriers keep, so a block connecting during the replay window was delivered twice — with two different delivery ids, which a receiver cannot collapse. - `metrics.retain` was built from alertfile ids only, evicting the legacy reorg hook's counters at startup and leaving reorg delivery permanently unobservable on any node configuring both. - The fan-in serialized and minted an id for every envelope *before* checking whether any hook wanted it — including `BlockTweaks`, hundreds of KB per block, which is refused at parse and can never match. - A removed hook's durable cursor was never deleted, so reusing a short id (`pager`, `ops`) greeted a brand-new endpoint with its predecessor's history. - Catch-up logged the 10k-block clamp but not the 1024-deep queue truncation, which is the binding limit for any realistic outage; the success line then reported the pre-truncation count. Also an off-by-one in the skipped count. - Webhook metrics emitted a `# HELP`/`# TYPE` pair per hook rather than per family — well-formed with one hook, invalid with two, and a strict parser rejects the entire page. - `persist_cursor` had no monotonicity guard, so an in-flight delivery from a retired generation could rewind the durable cursor past what its successor had already advanced. - A closed hook channel was a silent no-op: a dead delivery task left the hook looking healthy on /metrics while delivering nothing, forever. - `is_local_target` split the authority on the last colon, so `http://127.0.0.1:8332@evil.example/` parsed as loopback and skipped the plaintext-HTTP acknowledgement while sending cleartext to a public host. - `Hook`'s derived `Debug` rendered `secret` in full; one `debug!(?hook)` added later would put a signing key in the log. - The legacy reorg alias retried only on the new `Disposition` classification, turning what was "retry any non-2xx three times" into a one-shot drop for 4xx on a flag operators already depend on. Restored. (Redirect-following is deliberately *not* restored — it is an SSRF vector for a signed request.) Each behavioral fix has a test verified to fail without it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH --- CHANGELOG.md | 6 +- Cargo.lock | 1 + docs/manual/src/observability.md | 10 +- docs/release-notes/0.5.0-pre.md | 11 +- node/src/events/replay.rs | 15 ++ node/src/metrics.rs | 85 +++++--- node/src/storage/coin_cache.rs | 4 + node/src/storage/mod.rs | 13 ++ node/src/storage/rocksdb_store.rs | 10 + node/src/storage/split_store.rs | 4 + satd-alert/src/config.rs | 107 ++++++++-- satd-alert/src/contract.rs | 146 ++++++++++++- satd-alert/src/lib.rs | 5 +- satd-alert/src/retry.rs | 15 +- satd/Cargo.toml | 4 + satd/src/alert.rs | 332 +++++++++++++++++++++++++++--- satd/tests/e2e/webhooks.rs | 305 ++++++++++++++++++++++++--- 17 files changed, 966 insertions(+), 107 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 40c7f8f5e..831164058 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -159,8 +159,10 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). metadata rides in headers (`X-Satd-Signature`, `X-Satd-Delivery`, `X-Satd-Hook`, `X-Satd-Attempt`). Delivery is serial and in-order per hook, retried with exponential backoff on transient failures and skipped on a - permanent 4xx, with a bounded queue whose overflow is reported in-band as a - `lagged` event rather than silently. Confirmed chain events are at-least-once + permanent 4xx (which still advances the resume position), with a bounded + queue whose overflow is reported in-band as a `lagged` event rather than + silently. Redirects are never followed — a 3xx is a permanent drop, so a + signed body cannot be steered to a host the alertfile never named. Confirmed chain events are at-least-once across a restart via a durable per-hook cursor. Hooks reload on SIGHUP (keep-last-good on error); per-hook counters are exported as `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with diff --git a/Cargo.lock b/Cargo.lock index 48ac6d827..64fef9dde 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4089,6 +4089,7 @@ dependencies = [ "sd-notify", "serde", "serde_json", + "subtle", "tempfile", "tikv-jemallocator", "tls-config", diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index e5ccb133b..e6b963597 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -208,7 +208,15 @@ acknowledge. - **Retried with backoff** on 5xx, 408, 429, timeouts, and connection failures: 1 s doubling to a 5-minute ceiling, indefinitely. Any *other* 4xx is treated as permanent, counted, and skipped — a receiver answering 404 forever must - not pin the queue and turn every later event into a drop. + not pin the queue and turn every later event into a drop. A skipped event + still advances the hook's resume position, so a hard-rejecting endpoint does + not turn every restart into a replay of the same refused span. +- **Redirects are not followed.** A 3xx is a permanent drop. The URL in the + alertfile is where the signed body goes; following a redirect would move it — + signature, hook identity and all — to a host you never named, and the useful + targets for that are exactly the ones you cannot see: a cloud metadata + endpoint, an RFC1918 admin port, the node's own RPC. If your receiver moves, + update the alertfile. - **Bounded queue.** A hook that falls far enough behind drops events, and the next delivery is preceded by a `lagged` body carrying how many were lost and the cursor to resume from. A gap is never silent. diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index 0851940b4..e7e7022a0 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -406,13 +406,16 @@ kinds = ["tip_stall", "disk_low"] `X-Satd-Hook`, `X-Satd-Attempt`) rides in headers and never in the body — which is what keeps the signature stable across retries and makes `X-Satd-Delivery` usable as an idempotency key. -- **Honest failure behavior.** Delivery is serial and in-order per hook, with +- **Failure behavior.** Delivery is serial and in-order per hook, with exponential backoff to a five-minute ceiling on transient failures. A permanent 4xx is counted and skipped rather than retried forever — a receiver answering 404 must not pin the queue and convert every later event into a - drop. A hook that falls far enough behind drops events and the next delivery - is preceded by a `lagged` body carrying the count and a resume cursor: a gap - is never silent. + drop — and the skip advances the resume position, so a hard-rejecting + endpoint does not rebuild the same refused span on every restart. Redirects + are never followed: a 3xx is a permanent drop, because chasing one would + carry the signed body to a host the alertfile never named. A hook that falls + far enough behind drops events and the next delivery is preceded by a + `lagged` body carrying the count and a resume cursor: a gap is never silent. - **At-least-once for confirmed chain events across a restart.** Each hook's resume position is persisted with the chainstate, and on startup the hook replays what it missed, bounded by the same 10 000-block window the streaming diff --git a/node/src/events/replay.rs b/node/src/events/replay.rs index 961a9b309..6a33b1ea0 100644 --- a/node/src/events/replay.rs +++ b/node/src/events/replay.rs @@ -42,6 +42,17 @@ pub trait BlockCursorSource: Send + Sync { /// Current active-chain tip height. fn current_tip_height(&self) -> u32; + /// Whether the node is still in initial block download. + /// + /// Not used by replay itself — it is here because the webhook dispatcher + /// needs to suppress its firehose during IBD and already holds this handle; + /// threading a second chain reference through the dispatcher's five + /// constructor layers to answer one boolean is worse. Defaults to `false` + /// so test doubles and any other implementor are unaffected. + fn in_initial_block_download(&self) -> bool { + false + } + /// Active-chain block hashes for the heights in `[from, to]` (inclusive), /// height-ascending. Resolved by walking the active chain back from the tip /// **once** (O(tip − from), not O(span²)) so it is reorg-safe: it follows @@ -60,6 +71,10 @@ impl BlockCursorSource for crate::chain::state::ChainState { self.tip_height() } + fn in_initial_block_download(&self) -> bool { + self.is_initial_block_download() + } + fn active_chain_range(&self, from: u32, to: u32) -> Vec<(u32, BlockHash)> { let (tip_hash, tip_height) = self.tip_snapshot(); let hi = to.min(tip_height); diff --git a/node/src/metrics.rs b/node/src/metrics.rs index cc0fd5f6c..89610ed19 100644 --- a/node/src/metrics.rs +++ b/node/src/metrics.rs @@ -490,53 +490,59 @@ fn render_webhook_metrics(out: &mut String, metrics: Option<&WebhookMetrics>) { .duration_since(std::time::UNIX_EPOCH) .map(|d| d.as_secs()) .unwrap_or(0); - for (id, c) in metrics.snapshot() { - let labels = [("hook", id.as_str())]; - metric( - out, + // Family headers once, then one labelled sample per hook. Emitting a + // header per hook (as a per-sample `metric` call in this loop does) is + // well-formed with a single webhook configured and invalid with two — the + // kind of bug that passes every test written against one hook and breaks + // an operator's whole scrape page the day they add a second. + let snapshot = metrics.snapshot(); + for (name, help, kind) in [ + ( "satd_alertwebhook_delivered_total", "Webhook events acknowledged (2xx) by the receiver.", "counter", - &labels, - c.delivered.load(Ordering::Relaxed), - ); - metric( - out, + ), + ( "satd_alertwebhook_failed_attempts_total", "Webhook delivery attempts that failed (retried or dropped).", "counter", - &labels, - c.failed_attempts.load(Ordering::Relaxed), - ); - metric( - out, + ), + ( "satd_alertwebhook_dropped_total", "Webhook events never delivered (queue overflow, broadcast lag, or a permanent 4xx).", "counter", - &labels, - c.dropped.load(Ordering::Relaxed), - ); - metric( - out, + ), + ( "satd_alertwebhook_queue_depth", "Events currently queued for this webhook.", "gauge", - &labels, - c.queue_depth.load(Ordering::Relaxed), - ); + ), // Age rather than a timestamp: an alerting rule wants "no successful // delivery in N minutes", which is awkward to express against an // absolute epoch value. 0 before the first success, which reads as // "fresh" — so pair it with `delivered_total > 0` in a rule. - let last = c.last_success_unix.load(Ordering::Relaxed); - metric( - out, + ( "satd_alertwebhook_last_success_age_seconds", "Seconds since this webhook's last acknowledged delivery (0 if none yet).", "gauge", - &labels, - if last == 0 { 0 } else { now.saturating_sub(last) }, - ); + ), + ] { + metric_header(out, name, help, kind); + for (id, c) in &snapshot { + let labels = [("hook", id.as_str())]; + let last = c.last_success_unix.load(Ordering::Relaxed); + let value = match name { + "satd_alertwebhook_delivered_total" => c.delivered.load(Ordering::Relaxed), + "satd_alertwebhook_failed_attempts_total" => { + c.failed_attempts.load(Ordering::Relaxed) + } + "satd_alertwebhook_dropped_total" => c.dropped.load(Ordering::Relaxed), + "satd_alertwebhook_queue_depth" => c.queue_depth.load(Ordering::Relaxed), + _ if last == 0 => 0, + _ => now.saturating_sub(last), + }; + metric_sample(out, name, &labels, value); + } } } @@ -932,6 +938,29 @@ mod tests { assert_eq!(scope_label(false, false), "none"); } + #[test] + fn webhook_metrics_are_valid_exposition_format_with_several_hooks() { + // Two hooks, because one hook cannot expose this: a header-per-sample + // renderer is well-formed with a single series and invalid with two. + // A strict parser rejects the entire page on the duplicate `# HELP`, + // so an operator adding a second webhook would lose every metric satd + // exports, not just these. + let m = WebhookMetrics::default(); + let _ = m.hook("pager"); + let _ = m.hook("deadman"); + let _ = m.hook("relay"); + let mut out = String::new(); + render_webhook_metrics(&mut out, Some(&m)); + assert_one_header_per_family(&out); + // And every hook is still represented. + for id in ["pager", "deadman", "relay"] { + assert!( + out.contains(&format!("satd_alertwebhook_delivered_total{{hook=\"{id}\"}}")), + "missing series for {id}:\n{out}" + ); + } + } + #[test] fn webhook_metrics_invisible_until_a_hook_is_configured() { use std::sync::atomic::Ordering; diff --git a/node/src/storage/coin_cache.rs b/node/src/storage/coin_cache.rs index 39bf17d8f..56e40b7fc 100644 --- a/node/src/storage/coin_cache.rs +++ b/node/src/storage/coin_cache.rs @@ -1292,6 +1292,10 @@ impl Store for CoinCache { self.inner.write_alert_cursor(key, value) } + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { + self.inner.delete_alert_cursor(key) + } + fn lookup_spend( &self, outpoint: &OutPoint, diff --git a/node/src/storage/mod.rs b/node/src/storage/mod.rs index 60b5807a0..ffebaddb6 100644 --- a/node/src/storage/mod.rs +++ b/node/src/storage/mod.rs @@ -865,4 +865,17 @@ pub trait Store: Send + Sync { "write_alert_cursor not supported on this backend".into(), )) } + + /// Forget a hook's resume cursor. + /// + /// Called when a hook disappears from the alertfile. Without it the key + /// outlives the hook forever, and hook ids are reused (`pager`, `alerts`, + /// `ops`) — so pointing a *new* endpoint at an old id would greet it with + /// the entire replay window of history belonging to its predecessor, with + /// no way to reset short of a reindex. + fn delete_alert_cursor(&self, _key: &[u8]) -> Result<(), StoreError> { + Err(StoreError::Database( + "delete_alert_cursor not supported on this backend".into(), + )) + } } diff --git a/node/src/storage/rocksdb_store.rs b/node/src/storage/rocksdb_store.rs index 9746ef2e8..0ee2a24a5 100644 --- a/node/src/storage/rocksdb_store.rs +++ b/node/src/storage/rocksdb_store.rs @@ -1266,6 +1266,16 @@ impl Store for RocksDbStore { .map_err(|e| StoreError::Database(e.to_string())) } + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { + let cf = self + .db + .cf_handle(CF_METADATA) + .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; + self.db + .delete_cf(&cf, key) + .map_err(|e| StoreError::Database(e.to_string())) + } + fn get_block_index(&self, hash: &BlockHash) -> Option { let cf = self.cf(CF_BLOCK_INDEX); let value = self.db.get_cf(&cf, hash_bytes(hash)).ok()??; diff --git a/node/src/storage/split_store.rs b/node/src/storage/split_store.rs index 264121bbc..2211217e7 100644 --- a/node/src/storage/split_store.rs +++ b/node/src/storage/split_store.rs @@ -100,6 +100,10 @@ impl Store for SplitStore { self.block_store.write_alert_cursor(key, value) } + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { + self.block_store.delete_alert_cursor(key) + } + fn chain_tx_backfill_complete(&self) -> bool { self.block_store.chain_tx_backfill_complete() } diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs index efbe3594a..cb0c5d6bb 100644 --- a/satd-alert/src/config.rs +++ b/satd-alert/src/config.rs @@ -85,7 +85,7 @@ impl CategoryMask { /// A hook's event filter: categories, then (for status events) kinds and a /// severity floor. -#[derive(Debug, Clone, Default)] +#[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct HookFilter { pub categories: CategoryMask, /// `None` = every kind in the subscribed categories. Only meaningful for @@ -118,7 +118,11 @@ impl HookFilter { } /// One configured webhook. -#[derive(Debug, Clone)] +/// +/// `Debug` is hand-written: a derived one renders `secret` in full, and a +/// single `tracing::debug!(?hook)` added later would put a signing key in the +/// log. The crate already takes this posture for watch-set scan keys. +#[derive(Clone, PartialEq, Eq)] pub struct Hook { pub id: String, pub url: String, @@ -137,6 +141,19 @@ pub struct Hook { pub allow_insecure_http: bool, } +impl std::fmt::Debug for Hook { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("Hook") + .field("id", &self.id) + .field("url", &self.url) + .field("secret", &"") + .field("filter", &self.filter) + .field("heartbeat_interval_secs", &self.heartbeat_interval_secs) + .field("allow_insecure_http", &self.allow_insecure_http) + .finish() + } +} + impl Hook { /// The `CF_METADATA` key holding this hook's durable resume cursor. pub fn cursor_key(&self) -> Vec { @@ -145,7 +162,7 @@ impl Hook { } /// A parsed, validated alertfile. -#[derive(Debug, Clone, Default)] +#[derive(Debug, Clone, Default, PartialEq, Eq)] pub struct AlertFile { pub hooks: Vec, } @@ -459,15 +476,27 @@ fn validate_url( /// plaintext is normal (a relay on the same host, a receiver inside a private /// network) and demanding TLS would just push operators to set the override. fn is_local_target(rest: &str) -> bool { - let host = rest - .split('/') - .next() - .unwrap_or("") - .rsplit_once(':') - .map(|(h, _)| h) - .unwrap_or_else(|| rest.split('/').next().unwrap_or("")) - .trim_start_matches('[') - .trim_end_matches(']'); + // Authority is everything before the path, query, or fragment. + let authority = rest.split(['/', '?', '#']).next().unwrap_or(""); + // Strip userinfo — everything through the *last* `@`. + // + // Without this, `http://127.0.0.1:8332@evil.example/hook` reads as host + // `127.0.0.1`, is judged loopback, and is accepted with no + // `allow_insecure_http` acknowledgement — while the request actually goes + // in cleartext to `evil.example` with `127.0.0.1:8332` as userinfo. The + // operator is not the adversary here; the gate exists to make them + // consciously accept cleartext to a public host, and this skipped it. + let host_port = authority.rsplit_once('@').map_or(authority, |(_, h)| h); + // An IPv6 literal is bracketed, and its address contains colons — so the + // port must be split on the bracket, not on the last colon. + let host = if let Some(after) = host_port.strip_prefix('[') { + match after.split_once(']') { + Some((h, _)) => h, + None => return false, + } + } else { + host_port.split_once(':').map_or(host_port, |(h, _)| h) + }; if host == "localhost" || host == "::1" { return true; } @@ -693,6 +722,31 @@ categories = ["chain"] assert!(parse(&opted).is_ok()); } + #[test] + fn userinfo_cannot_impersonate_a_local_host() { + // The authority's host is what comes after the last `@`. Reading the + // userinfo as the host would let a config pasted from a third-party + // setup guide send signed cleartext to a public endpoint while + // *looking* like a loopback target, silently skipping the acknowledgement + // the plaintext gate exists to force. + for sneaky in [ + "http://127.0.0.1:8332@evil.example/hook", + "http://localhost@evil.example/hook", + "http://10.0.0.1@evil.example:8080/hook", + "http://user:127.0.0.1@evil.example/hook", + ] { + let text = MINIMAL.replace("https://alerts.example/satd", sneaky); + assert!( + parse(&text).is_err(), + "{sneaky} resolves to a public host and must need the opt-in" + ); + } + + // A genuine loopback target with userinfo is still local. + let text = MINIMAL.replace("https://alerts.example/satd", "http://user:pw@127.0.0.1:9000/h"); + assert!(parse(&text).is_ok(), "userinfo on a real loopback host is fine"); + } + #[test] fn non_http_schemes_are_rejected() { for url in ["ftp://x/h", "file:///etc/passwd", "x.example/h", ""] { @@ -725,6 +779,35 @@ categories = ["chain"] assert!(parse(&zero).is_err()); } + #[test] + fn debug_never_renders_the_signing_secret() { + // A derived `Debug` prints `secret` in full, so one + // `tracing::debug!(?hook)` added later would write a signing key to the + // log — and to anywhere the log is shipped. Assert on the secret's + // *value*, not on the presence of "redacted", so the test cannot pass + // by rendering both. + let f = parse( + "version = 1\n\ + [[webhook]]\n\ + id = \"pager\"\n\ + url = \"https://example.invalid/h\"\n\ + secret = \"correct-horse-battery-staple\"\n\ + categories = [\"status\"]\n", + ) + .unwrap(); + let hook_dbg = format!("{:?}", f.hooks[0]); + assert!( + !hook_dbg.contains("correct-horse-battery-staple"), + "Hook Debug leaked the secret: {hook_dbg}" + ); + // The whole file renders through the same impl. + let file_dbg = format!("{f:?}"); + assert!( + !file_dbg.contains("correct-horse-battery-staple"), + "AlertFile Debug leaked the secret: {file_dbg}" + ); + } + #[test] fn empty_file_with_only_a_version_is_valid() { // "configured but no hooks" is a legitimate intermediate state during diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs index f6f4cf7cb..c9cc86d96 100644 --- a/satd-alert/src/contract.rs +++ b/satd-alert/src/contract.rs @@ -34,7 +34,20 @@ pub const ATTEMPT_HEADER: &str = "X-Satd-Attempt"; /// signature scheme — the *body* schema is versioned independently by the /// event envelope's `schema_version`. pub const WEBHOOK_VERSION_HEADER: &str = "X-Satd-Webhook-Version"; -pub const WEBHOOK_VERSION: &str = "1"; +/// Contract version for alertfile hooks: signature covers the delivery +/// metadata, not just the body, and `X-Satd-Timestamp` is present. +pub const WEBHOOK_VERSION: &str = "2"; +/// Contract version the legacy `-reorgwebhook` alias keeps emitting. +/// +/// That surface shipped before this crate existed: body-only signature, no +/// delivery id, `ReorgRecord` rather than the event envelope. Deployed +/// receivers verify it as-is, so it is frozen — the version header is how a +/// receiver tells the two apart. +pub const LEGACY_WEBHOOK_VERSION: &str = "1"; + +/// Unix seconds at which a delivery was signed. Covered by the v2 signature; +/// a receiver rejects a delivery outside [`MAX_TIMESTAMP_SKEW_SECS`]. +pub const TIMESTAMP_HEADER: &str = "X-Satd-Timestamp"; /// Sign a raw body with a hook secret, rendering the `X-Satd-Signature` value. /// @@ -58,6 +71,87 @@ pub fn delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { format!("{node_id_hex}-{instance_id}-{seq}") } +/// Build the idempotency key for a **synthesized** envelope — a catch-up replay +/// event or a gap notice. +/// +/// These do not come off the live bus. They are stamped by the replay builder, +/// which has no sequence to assign and writes `seq: 0` into every one of them; +/// live bus events start at 1. So `seq` is not merely a weak discriminator +/// here — it is one constant shared by every replayed event and every lag +/// notice for the life of the process. +/// +/// That matters more than it looks. Catch-up exists precisely so a hook that +/// was down does not miss what happened, and the contract tells receivers to +/// deduplicate on this header. Minting these from `seq` would mean a node down +/// for 100 blocks replays 100 events that a conforming receiver collapses into +/// one — the durability feature delivering 1% of what it advertises, silently. +/// The same collision would swallow every gap notice after the first, breaking +/// "a gap is never silent" from the second gap onward. +/// +/// The `r` prefix keeps this space disjoint from the bus (``) and watch +/// (`w`) spaces. +pub fn replay_delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { + format!("{node_id_hex}-{instance_id}-r{seq}") +} + +/// Maximum age a receiver should accept for a v2 delivery, in seconds. +/// +/// Normative for receivers, advisory here: satd stamps `X-Satd-Timestamp` and +/// signs it, but only the receiver can enforce freshness. Wide enough to +/// tolerate ordinary clock skew and a retry backoff (which caps at 300 s), tight +/// enough that a captured delivery is not a permanent replay token. +pub const MAX_TIMESTAMP_SKEW_SECS: u64 = 600; + +/// Build the canonical string a v2 signature covers. +/// +/// ```text +/// "2" LF LF LF LF +/// ``` +/// +/// Unambiguous without escaping because every field before the body is +/// constrained to a character set that excludes LF: the version is a literal, +/// the timestamp is decimal digits, the delivery id is hex plus `-` and an +/// optional `w`/`r` tag, and the hook id is restricted to `[A-Za-z0-9_-]` at +/// parse time. The body is last, so its content cannot be confused with a +/// preceding field however it is shaped. +pub fn v2_signing_string(timestamp: u64, delivery_id: &str, hook_id: &str, body: &[u8]) -> Vec { + let mut buf = + Vec::with_capacity(body.len() + delivery_id.len() + hook_id.len() + 32); + buf.extend_from_slice(b"2\n"); + buf.extend_from_slice(timestamp.to_string().as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(delivery_id.as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(hook_id.as_bytes()); + buf.push(b'\n'); + buf.extend_from_slice(body); + buf +} + +/// Sign a v2 delivery: HMAC-SHA256 over [`v2_signing_string`]. +/// +/// v1 (see [`sign_body`]) covers the body and nothing else, which leaves the +/// delivery id — the value the contract instructs receivers to deduplicate +/// on — unauthenticated *and* predictable, since `seq` is dense and every +/// component of the id is visible in any single captured delivery. Anyone +/// holding one valid `(body, signature)` pair could therefore replay it under +/// forged future ids, filling a receiver's dedup cache so that the genuine +/// alerts bearing those ids are discarded on arrival — while satd counts them +/// delivered and advances its cursor. Signing the id closes that, and signing a +/// timestamp bounds replay of the capture itself. +pub fn sign_v2( + secret: &str, + timestamp: u64, + delivery_id: &str, + hook_id: &str, + body: &[u8], +) -> String { + let mut mac = as Mac>::new_from_slice(secret.as_bytes()) + .expect("HMAC accepts a key of any length"); + mac.update(&v2_signing_string(timestamp, delivery_id, hook_id, body)); + format!("sha256={}", hex::encode(mac.finalize().into_bytes())) +} + #[cfg(test)] mod tests { use super::*; @@ -67,6 +161,56 @@ mod tests { // independent receiver implementation can verify its HMAC without running // a node. Changing any expected value here is a breaking protocol change // and must come with a `WEBHOOK_VERSION` bump. + /// v2 golden vectors. Computed independently (Python `hmac`) against the + /// canonical string, not captured from this implementation's own output — + /// a vector derived from the code under test only proves the code agrees + /// with itself. Reproduced verbatim in `docs/api/webhooks.md` and in the + /// reference relay's tests. + #[test] + fn v2_signature_golden_vectors() { + let node = "ab".repeat(16); + let did = delivery_id(&node, 7, 42); + assert_eq!(did, "abababababababababababababababab-7-42"); + assert_eq!( + sign_v2("hunter2", 1_753_400_000, &did, "pager", br#"{"hello":"world"}"#), + "sha256=0dcd7bcc563327beab8a0ec4464a261288b43825415ae4c1ebdc91e79c83e031", + ); + assert_eq!( + sign_v2("hunter2", 1_753_400_000, &did, "pager", b""), + "sha256=abb61065799427bc98456c493fe12ac9adec92fd10bebbc1bd00720becb69b6c", + ); + } + + #[test] + fn v2_signature_covers_every_metadata_field() { + let node = "ab".repeat(16); + let did = delivery_id(&node, 7, 42); + let base = sign_v2("s", 1_000, &did, "pager", b"body"); + // Each of these is a field v1 left unauthenticated. Changing any one + // must change the signature, or the field is not really covered. + assert_ne!(base, sign_v2("s", 1_001, &did, "pager", b"body"), "timestamp"); + assert_ne!( + base, + sign_v2("s", 1_000, &delivery_id(&node, 7, 43), "pager", b"body"), + "delivery id — the dedup key an attacker would forge" + ); + assert_ne!(base, sign_v2("s", 1_000, &did, "other", b"body"), "hook id"); + assert_ne!(base, sign_v2("s", 1_000, &did, "pager", b"body2"), "body"); + assert_ne!(base, sign_v2("t", 1_000, &did, "pager", b"body"), "secret"); + } + + #[test] + fn v2_canonical_string_is_unambiguous() { + // The delimiter is LF and no field before the body may contain one, so + // no reshuffling of field contents can produce the same signing string. + // Concatenation without a delimiter would let ("ab","c") collide with + // ("a","bc"); this asserts it does not. + assert_ne!( + v2_signing_string(1, "ab", "c", b"x"), + v2_signing_string(1, "a", "bc", b"x"), + ); + } + #[test] fn signature_golden_vectors() { for (secret, body, expected) in [ diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs index a5f9b1925..d93401077 100644 --- a/satd-alert/src/lib.rs +++ b/satd-alert/src/lib.rs @@ -27,7 +27,8 @@ pub mod retry; pub use config::{AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY}; pub use contract::{ - delivery_id, sign_body, ATTEMPT_HEADER, DELIVERY_HEADER, HOOK_HEADER, SIGNATURE_HEADER, - WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, + delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, ATTEMPT_HEADER, + DELIVERY_HEADER, HOOK_HEADER, LEGACY_WEBHOOK_VERSION, MAX_TIMESTAMP_SKEW_SECS, + SIGNATURE_HEADER, TIMESTAMP_HEADER, WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, }; pub use retry::{classify_response, retry_delay, Disposition, MAX_BACKOFF}; diff --git a/satd-alert/src/retry.rs b/satd-alert/src/retry.rs index b1a64f3f4..253297146 100644 --- a/satd-alert/src/retry.rs +++ b/satd-alert/src/retry.rs @@ -69,7 +69,11 @@ pub fn classify_response(status: Option) -> Disposition { Some(408) | Some(429) => Disposition::Retry, Some(s) if (500..600).contains(&s) => Disposition::Retry, // Every other 4xx is the receiver saying this request is wrong, and it - // will be just as wrong next time. + // will be just as wrong next time. 3xx lands here too: the delivery + // client does not follow redirects (a followed redirect would carry the + // signed body to a host the alertfile never named), so a redirecting + // endpoint is a misconfiguration to fix in the alertfile, not something + // to retry into. Some(_) => Disposition::Drop, } } @@ -108,6 +112,15 @@ mod tests { } } + #[test] + fn redirects_are_dropped_because_they_are_never_followed() { + // The delivery client sets `Policy::none()`, so a 3xx arrives here + // rather than being chased to a host the alertfile never named. + for s in [301, 302, 303, 307, 308] { + assert_eq!(classify_response(Some(s)), Disposition::Drop, "{s}"); + } + } + #[test] fn backoff_doubles_then_caps() { assert_eq!(retry_delay(1), Duration::from_secs(1)); diff --git a/satd/Cargo.toml b/satd/Cargo.toml index 91e7c032d..811831bbe 100644 --- a/satd/Cargo.toml +++ b/satd/Cargo.toml @@ -82,6 +82,10 @@ hex = { workspace = true } base64 = { workspace = true } tempfile = "3" rcgen = { workspace = true } +# Constant-time signature comparison in the webhook e2e helper: that helper is +# the most likely thing a third party copies when writing a receiver, and the +# spec requires a constant-time compare. +subtle = { workspace = true } # Streaming E2E harness (tests/e2e/streaming.rs, folded into the `e2e` test # target): a real gRPC client (tonic, # pinned to the workspace version so it matches the `satd_events::proto` diff --git a/satd/src/alert.rs b/satd/src/alert.rs index b56a9ff2d..737fca8e5 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -51,6 +51,13 @@ const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); /// "resync below this height yourself" boundary. const MAX_CATCHUP_BLOCKS: u32 = node::events::MAX_REPLAY_BLOCKS; +/// Numbers synthesized deliveries — catch-up replay events and gap notices. +/// +/// Process-wide rather than per-generation, for the same reason the watch +/// counter is: a SIGHUP reload keeps the same `instance_id`, so a per-generation +/// counter would restart at zero and re-mint ids the receiver has already seen. +static SYNTH_DELIVERY_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + /// Composite stop signal: the process-wide shutdown, plus a per-generation /// channel the SIGHUP reload flips to retire the previous dispatcher. /// @@ -167,8 +174,18 @@ async fn fan_in( // replay is being built is buffered rather than lost in the seam. let mut rx = publisher.subscribe(); + // Per-hook snapshot→live boundary dedup, exactly as the gRPC and WS + // carriers keep. A block connecting between `subscribe()` and the tip + // snapshot taken inside `catch_up` is *both* buffered on the broadcast and + // inside the replayed span, so without this it is delivered twice — and + // because the replayed copy is re-synthesized with its own delivery id, the + // two carry different idempotency keys and a conforming receiver cannot + // collapse them. A duplicate deposit alert is exactly the failure the + // idempotency contract exists to prevent. + let mut dedup: Vec> = + Vec::with_capacity(hooks.len()); for hook in &hooks { - catch_up(hook, &publisher, store.as_ref(), block_source.as_deref()).await; + dedup.push(catch_up(hook, &publisher, store.as_ref(), block_source.as_deref()).await); } // Per-hook heartbeat downsampling state (D11): last forwarded instant. @@ -185,6 +202,41 @@ async fn fan_in( _ = stop.wait() => break, recv = rx.recv() => match recv { Ok(env) => { + // Suppress the firehose during initial block download. + // + // The dispatcher is started long before P2P, so without + // this a brand-new node with an alertfile POSTs its entire + // sync — one `block_connected` per historical block, plus a + // watch-match per historical transaction touching a watched + // script, for as long as IBD takes. That is the opposite of + // what the manual promises, and the failure mode is a + // multi-day firehose at the receiver rather than anything + // that looks like a bug locally. + // + // Status events are exempt: "this node is unhealthy" is + // exactly as true during IBD, and the detectors already + // suppress the conditions that are meaningless while + // syncing. + let syncing = block_source + .as_deref() + .is_some_and(|s| s.in_initial_block_download()); + if syncing && !matches!(env.body, NodeEventBody::Status(_)) { + continue; + } + // Decide who wants this *before* rendering it. Serializing + // first meant a `BlockTweaks` envelope — hundreds of KB of + // per-block silent-payment rows — was rendered in full and + // then discarded, since tweaks are refused at parse and can + // never be in any hook's mask. + let wanted: Vec = hooks + .iter() + .enumerate() + .filter(|(i, hook)| accepts(&hook.hook, &env, &mut last_heartbeat[*i])) + .map(|(i, _)| i) + .collect(); + if wanted.is_empty() { + continue; + } // Render once for every hook: the body a receiver verifies // must be the bytes a WS subscriber would have seen, and // rendering per hook risks two receivers disagreeing. @@ -198,8 +250,19 @@ async fn fan_in( publisher.instance_id(), env.stamp.seq, ); - for (i, hook) in hooks.iter().enumerate() { - if !accepts(&hook.hook, &env, &mut last_heartbeat[i]) { + for i in wanted { + let hook = &hooks[i]; + // A live block identical to one already replayed for + // this hook. A reorg replacement at the same height has + // a different hash and is forwarded, which is why the + // check is on the hash and not the height alone. + if !dedup[i].is_empty() + && let NodeEventBody::Chain( + node::chain::events::ChainEvent::BlockConnected { height, hash }, + ) = &env.body + && dedup[i].get(height) == Some(hash) + { + dedup[i].remove(height); continue; } enqueue(hook, &publisher, Delivery::Event { @@ -245,7 +308,21 @@ fn enqueue(hook: &HookChannel, publisher: &EventPublisher, item: Delivery, curso hook.gap.dropped.fetch_add(1, Ordering::Relaxed); hook.counters.dropped.fetch_add(1, Ordering::Relaxed); } - Err(mpsc::error::TrySendError::Closed(_)) => {} + Err(mpsc::error::TrySendError::Closed(_)) => { + // The delivery task is gone: it failed to build its HTTP client, or + // it panicked. Silently ignoring this makes the hook look perfectly + // healthy on `/metrics` — `dropped_total` flat, `queue_depth` flat, + // no `Lagged` ever synthesized — while it delivers nothing at all, + // forever. Count it like any other loss so the existing + // "no successful delivery in N minutes" rule fires. + hook.gap.dropped.fetch_add(1, Ordering::Relaxed); + hook.counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + "hook delivery task is gone; event dropped", + ); + } } } @@ -264,10 +341,15 @@ fn flush_gap(hook: &HookChannel, publisher: &EventPublisher) { }; let item = Delivery::Event { body: Arc::new(bytes), - delivery_id: satd_alert::delivery_id( + // Synthesized envelope: `stamp.seq` is 0 for every one of these, so it + // gets an id from the replay space instead. Without this every gap + // notice in the process would carry the same idempotency key and a + // conforming receiver would discard all but the first — "a gap is never + // silent" would hold exactly once. + delivery_id: satd_alert::replay_delivery_id( &hex::encode(env.stamp.node_id), publisher.instance_id(), - env.stamp.seq, + SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), ), cursor: None, }; @@ -332,22 +414,22 @@ async fn catch_up( publisher: &EventPublisher, store: &dyn Store, block_source: Option<&dyn node::events::BlockCursorSource>, -) { +) -> std::collections::HashMap { if !hook .hook .filter .categories .contains(node::events::CATEGORY_CHAIN) { - return; + return Default::default(); } let Some(src) = block_source else { - return; + return Default::default(); }; let Some(cursor) = read_cursor(store, &hook.hook) else { // A hook that has never delivered starts at the live head rather than // replaying history nobody asked for. - return; + return Default::default(); }; let replay = node::events::build_cursor_replay( @@ -359,7 +441,11 @@ async fn catch_up( None, ); if replay.clamped { - let dropped = u64::from(replay.earliest_replayed.saturating_sub(cursor.height)); + // Replay begins at `cursor.height + 1`, so the number of blocks skipped + // is the distance to the first one actually replayed, not to the cursor + // itself — the latter counts the already-delivered block at the cursor. + let dropped = + u64::from(replay.earliest_replayed.saturating_sub(cursor.height.saturating_add(1))); hook.gap.dropped.fetch_add(dropped, Ordering::Relaxed); hook.counters.dropped.fetch_add(dropped, Ordering::Relaxed); hook.gap.resume_height.store(cursor.height, Ordering::Relaxed); @@ -372,6 +458,7 @@ async fn catch_up( ); } let count = replay.events.len(); + let before = hook.counters.dropped.load(Ordering::Relaxed); for env in replay.events { let Ok(bytes) = serde_json::to_vec(&env) else { continue; @@ -382,10 +469,15 @@ async fn catch_up( publisher, Delivery::Event { body: Arc::new(bytes), - delivery_id: satd_alert::delivery_id( + // Every replayed envelope is stamped `seq: 0`, so minting from + // it would give a node that was down for 100 blocks 100 + // deliveries sharing one idempotency key — a conforming + // receiver keeps the first and discards the other 99, which is + // the whole point of catch-up defeated silently. + delivery_id: satd_alert::replay_delivery_id( &hex::encode(env.stamp.node_id), publisher.instance_id(), - env.stamp.seq, + SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), ), cursor, }, @@ -393,14 +485,36 @@ async fn catch_up( ); } if count > 0 { + // Report what was *queued*, not what was built. The 10k-block replay + // clamp warns loudly, but the hook queue holds 1024 — and for any + // realistic outage the queue is the binding limit, since 1024 blocks is + // about a week and 10k is about ten. Overflow inside this loop is + // counted but was otherwise unlogged, so the success line reported the + // pre-truncation figure and read as if everything had been recovered. + let overflowed = hook + .counters + .dropped + .load(Ordering::Relaxed) + .saturating_sub(before); + if overflowed > 0 { + tracing::warn!( + target: "alert", + hook = %hook.id, + built = count, + dropped = overflowed, + "webhook catch-up exceeded the hook queue; the excess is lost \ + and reported to the receiver as a gap", + ); + } tracing::info!( target: "alert", hook = %hook.id, - events = count, + events = count as u64 - overflowed, from = cursor.height, "webhook catch-up queued events missed while the daemon was down", ); } + replay.confirmed_dedup } fn read_cursor(store: &dyn Store, hook: &Hook) -> Option { @@ -432,6 +546,65 @@ fn decode_cursor(raw: &[u8]) -> Option { }) } +/// The HTTP client every webhook delivery goes through. +/// +/// Redirects are **not** followed. The alertfile URL is validated once, at load +/// (scheme, and a warning for a non-loopback plaintext target), and a followed +/// redirect would silently move the request — body, `X-Satd-Signature`, and the +/// hook's identity — to a host that never passed that check. The interesting +/// destinations are exactly the ones an operator cannot see: a cloud metadata +/// endpoint, an RFC1918 admin port, the node's own RPC. HTTPS does not help; +/// a 302 to `http://169.254.169.254/` is a perfectly valid response. +/// +/// A receiver that wants to move must publish a stable final URL and have the +/// operator update the alertfile. A 3xx classifies as a permanent drop, so the +/// misconfiguration shows up in the logs rather than as silent non-delivery. +fn webhook_client() -> reqwest::Result { + reqwest::Client::builder() + .timeout(REQUEST_TIMEOUT) + .redirect(reqwest::redirect::Policy::none()) + .build() +} + +/// Persist a hook's resume position, at most once per block height. +/// +/// The cursor is a resume hint, not a ledger: one RocksDB write per delivered +/// event would be pure write amplification for a value only read at startup. +fn persist_cursor( + hook: &Hook, + store: &dyn Store, + cursor: Option, + persisted_height: &mut Option, +) { + let Some(c) = cursor else { return }; + if *persisted_height == Some(c.height) { + return; + } + // Never move a hook's durable cursor backwards. + // + // `persisted_height` is this task's own view and starts empty for a fresh + // generation, so it cannot see what a *retired* generation is still doing. + // A reload spawns the new generation before signalling the old one to stop, + // and an in-flight POST is not inside the stop `select!` — so a delivery + // retired mid-request can land up to `REQUEST_TIMEOUT` later and write a + // cursor the new generation has already advanced past. Rewinding it means + // the next restart replays a span the receiver already acked. + if let Some(existing) = store + .read_alert_cursor(&hook.cursor_key()) + .and_then(|raw| decode_cursor(&raw)) + && existing.height > c.height + { + *persisted_height = Some(existing.height); + return; + } + match store.write_alert_cursor(&hook.cursor_key(), &encode_cursor(&c)) { + Ok(()) => *persisted_height = Some(c.height), + Err(e) => { + tracing::warn!(target: "alert", hook = %hook.id, error = %e, "failed to persist webhook cursor") + } + } +} + /// One hook's delivery loop: serial, in-order, retrying with backoff. async fn deliver_loop( hook: Hook, @@ -440,7 +613,7 @@ async fn deliver_loop( store: Arc, mut stop: Stop, ) { - let client = match reqwest::Client::builder().timeout(REQUEST_TIMEOUT).build() { + let client = match webhook_client() { Ok(c) => c, Err(e) => { tracing::error!(target: "alert", hook = %hook.id, error = %e, "failed to build webhook HTTP client; this hook will not deliver"); @@ -470,7 +643,17 @@ async fn deliver_loop( .queue_depth .store(rx.len() as u64, Ordering::Relaxed); - let signature = satd_alert::sign_body(&hook.secret, &body); + // Signed once, outside the retry loop, and reused for every attempt: + // the signature must be stable across retries of one event (the + // attempt counter rides in a header, deliberately not in the body or + // the signed material), and the timestamp records when satd *signed* + // this delivery, not when it last retried it. A receiver enforcing a + // freshness window therefore sees a delivery age out if it is still + // being retried after the window, which is the intended behavior: a + // 20-minute-old "disk is filling" alert is not worth acting on. + let signed_at = unix_secs(); + let signature = + satd_alert::sign_v2(&hook.secret, signed_at, &delivery_id, &hook.id, &body); let mut attempt: u32 = 0; loop { attempt = attempt.saturating_add(1); @@ -478,6 +661,7 @@ async fn deliver_loop( .post(&hook.url) .header(reqwest::header::CONTENT_TYPE, "application/json") .header(satd_alert::SIGNATURE_HEADER, &signature) + .header(satd_alert::TIMESTAMP_HEADER, signed_at.to_string()) .header(satd_alert::DELIVERY_HEADER, &delivery_id) .header(satd_alert::HOOK_HEADER, &hook.id) .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) @@ -495,15 +679,7 @@ async fn deliver_loop( counters .last_success_unix .store(unix_secs(), Ordering::Relaxed); - if let Some(c) = cursor - && persisted_height != Some(c.height) - { - if let Err(e) = store.write_alert_cursor(&hook.cursor_key(), &encode_cursor(&c)) { - tracing::warn!(target: "alert", hook = %hook.id, error = %e, "failed to persist webhook cursor"); - } else { - persisted_height = Some(c.height); - } - } + persist_cursor(&hook, store.as_ref(), cursor, &mut persisted_height); break; } satd_alert::Disposition::Drop => { @@ -516,6 +692,16 @@ async fn deliver_loop( delivery = %delivery_id, "webhook receiver rejected the delivery permanently; skipping this event", ); + // Advance the cursor anyway. A permanent rejection is the + // receiver's decision about this event, and it will decide + // the same way next time — leaving the cursor parked would + // make every restart rebuild and re-queue the same span + // against an endpoint that has already refused it, forever. + // The event is lost either way; the difference is whether + // the hook makes progress past it. The drop is counted + // (`satd_alertwebhook_dropped_total`) and logged, so the loss is + // visible rather than silent. + persist_cursor(&hook, store.as_ref(), cursor, &mut persisted_height); break; } satd_alert::Disposition::Retry => { @@ -566,6 +752,9 @@ pub struct AlertReloader { global_stop: watch::Receiver, /// Stop signal for the generation currently running, if any. current: parking_lot::Mutex>>, + /// The last successfully-applied alertfile, so a SIGHUP that did not change + /// it can be a no-op instead of destroying in-flight deliveries. + last_applied: parking_lot::Mutex>, } impl AlertReloader { @@ -588,6 +777,7 @@ impl AlertReloader { metrics, global_stop, current: parking_lot::Mutex::new(None), + last_applied: parking_lot::Mutex::new(None), } } @@ -605,6 +795,30 @@ impl AlertReloader { let file = AlertFile::load(&self.path)?; let ids: Vec = file.hooks.iter().map(|h| h.id.clone()).collect(); + // A SIGHUP that did not change the alertfile must not disturb the + // dispatcher. `reload_from_sighup` calls this on *every* SIGHUP, + // whatever key the operator actually edited, and retiring a generation + // destroys its queued deliveries. For chain events that is recoverable + // — the cursor did not advance, so the next generation's catch-up + // re-queues them — but a status event has no replay by design, and the + // detectors are edge-triggered against a `HealthState` that outlives + // the reload. So a `disk_low` sitting in retry backoff when the + // operator SIGHUPs to change `maxconnections` would be dropped, never + // replayed, and never re-raised: the page simply never arrives. + // + // Comparing the parsed file rather than the file bytes means + // reformatting or a comment edit is also a no-op. + { + let last = self.last_applied.lock(); + if last.as_ref() == Some(&file) { + tracing::debug!( + target: "alert", + "alertfile unchanged; keeping the running dispatcher generation", + ); + return Ok(file.hooks.len()); + } + } + // Retire the previous generation only once the new file has parsed, so // a bad edit never leaves the node with no dispatcher at all. let (gen_tx, gen_rx) = watch::channel(false); @@ -630,7 +844,44 @@ impl AlertReloader { } // Stop exporting counters for hooks that are no longer configured, // rather than freezing their series at the last value forever. - self.metrics.retain(&ids); + // + // The legacy reorg alias is preserved explicitly: it registers its + // counters outside the alertfile, so retaining only alertfile ids would + // evict them the first time `apply` runs — and it runs at startup — + // leaving the still-running legacy dispatcher incrementing a snapshot + // nothing renders. Reorg-webhook delivery would become permanently + // unobservable, and only on nodes that configure both. + let mut keep = ids; + keep.push(LEGACY_REORG_HOOK_ID.to_string()); + self.metrics.retain(&keep); + + // Drop the durable cursor of any hook this reload removed. Hook ids are + // short and reused — `pager`, `alerts`, `ops` — so leaving the key + // behind means a later hook that happens to reuse the id inherits a + // stale resume position and greets a brand-new endpoint with the whole + // replay window of its predecessor's history. + if let Some(prev) = self.last_applied.lock().as_ref() { + for gone in prev + .hooks + .iter() + .filter(|h| !keep.iter().any(|k| k == &h.id)) + { + match self.store.delete_alert_cursor(&gone.cursor_key()) { + Ok(()) => tracing::info!( + target: "alert", + hook = %gone.id, + "hook removed from the alertfile; dropped its resume cursor", + ), + Err(e) => tracing::warn!( + target: "alert", + hook = %gone.id, + error = %e, + "failed to drop the removed hook's resume cursor", + ), + } + } + } + *self.last_applied.lock() = Some(file); Ok(hook_count) } } @@ -648,7 +899,7 @@ pub async fn legacy_reorg_dispatcher( mut rx: mpsc::Receiver, counters: Arc, ) { - let client = match reqwest::Client::builder().timeout(REQUEST_TIMEOUT).build() { + let client = match webhook_client() { Ok(c) => c, Err(e) => { tracing::warn!(target: "alert", error = %e, "failed to build reorg webhook HTTP client"); @@ -675,7 +926,14 @@ pub async fn legacy_reorg_dispatcher( .header(reqwest::header::CONTENT_TYPE, "application/json") .header(satd_alert::ATTEMPT_HEADER, attempt.to_string()) .header(satd_alert::HOOK_HEADER, LEGACY_REORG_HOOK_ID) - .header(satd_alert::WEBHOOK_VERSION_HEADER, satd_alert::WEBHOOK_VERSION); + // v1, frozen: this surface shipped with a body-only signature, + // no delivery id, and a `ReorgRecord` body. Deployed receivers + // verify exactly that, so it does not move to the v2 contract; + // the version header is how a receiver tells them apart. + .header( + satd_alert::WEBHOOK_VERSION_HEADER, + satd_alert::LEGACY_WEBHOOK_VERSION, + ); if let Some(secret) = &target.secret { req = req.header(satd_alert::SIGNATURE_HEADER, satd_alert::sign_body(secret, &body)); } @@ -696,10 +954,20 @@ pub async fn legacy_reorg_dispatcher( Ok(r) => tracing::warn!(target: "alert", status = %r.status(), attempt, "reorg webhook returned non-2xx"), Err(e) => tracing::warn!(target: "alert", error = %e, attempt, "reorg webhook request failed"), } - // Bounded retries, unchanged from the shipped behavior: the - // legacy hook has no queue to fall behind on, so a failing - // endpoint is given three tries and the record is dropped. - if d == satd_alert::Disposition::Drop || attempt >= 3 { + // Bounded retries, matching the shipped behavior: the legacy + // hook has no queue to fall behind on, so a failing endpoint + // is given three tries and the record is dropped. + // + // Deliberately NOT keyed on `Disposition` the way alertfile + // hooks are. The shipped dispatcher retried *any* non-2xx + // three times, so classifying 4xx as a one-shot drop here + // would quietly change behavior on a flag operators already + // depend on. (The redirect change is not reverted: the + // shipped client followed 30x, and that is an SSRF vector + // for a signed request, so 3xx now fails — documented as a + // breaking change in the release notes.) + let _ = d; + if attempt >= 3 { counters.dropped.fetch_add(1, Ordering::Relaxed); break; } diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index 30fd73e9e..3233347d1 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -41,19 +41,48 @@ impl Received { .unwrap_or_default() } - /// Verify `X-Satd-Signature` over the raw body, exactly as a third-party - /// receiver would. + /// Verify `X-Satd-Signature` exactly as a third-party receiver would. + /// + /// v2: the HMAC covers the timestamp, the delivery id, the hook id, and the + /// body — not the body alone. Signing only the body would leave the + /// delivery id unauthenticated, and since the contract tells receivers to + /// deduplicate on that header, anyone holding one captured delivery could + /// replay it under forged ids and pre-empt the genuine alerts. + /// + /// The comparison is constant-time. It is not a real secret-dependent + /// branch here — the node is the signer — but this helper is the most + /// likely thing someone copies when writing a receiver against satd, and + /// the spec requires constant-time comparison. fn signature_valid(&self, secret: &str) -> bool { - self.header("x-satd-signature") == satd_alert::sign_body(secret, self.body.as_bytes()) + use subtle::ConstantTimeEq as _; + let Ok(ts) = self.header("x-satd-timestamp").parse::() else { + return false; + }; + let expected = satd_alert::sign_v2( + secret, + ts, + self.header("x-satd-delivery"), + self.header("x-satd-hook"), + self.body.as_bytes(), + ); + let got = self.header("x-satd-signature"); + got.len() == expected.len() + && got.as_bytes().ct_eq(expected.as_bytes()).into() } } /// How the mock receiver answers the next request. -#[derive(Clone, Copy, PartialEq, Eq)] +#[derive(Clone, PartialEq, Eq)] enum Behavior { Ok, /// Respond with this status for the first `n` requests, then 200. FailFirst(u16, usize), + /// Answer every request with `302 Location: `. + Redirect(String), + /// Permanently reject (404) any `block_connected` at one of these heights, + /// accept everything else. Keyed on content rather than request ordinal so + /// the script does not depend on how many events a block produces. + RejectHeights(Vec), } struct Inner { @@ -88,13 +117,21 @@ impl MockReceiver { let Some(req) = read_request(&mut sock).await else { return; }; - let status = { + let (status, location) = { let mut g = inner.lock().await; g.seen += 1; - let status = match g.behavior { - Behavior::Ok => 200, - Behavior::FailFirst(code, n) if g.seen <= n => code, - Behavior::FailFirst(..) => 200, + let (status, location) = match &g.behavior { + Behavior::Ok => (200, None), + Behavior::FailFirst(code, n) if g.seen <= *n => (*code, None), + Behavior::FailFirst(..) => (200, None), + Behavior::Redirect(to) => (302, Some(to.clone())), + Behavior::RejectHeights(hs) => { + let h = req.json()["body"]["height"].as_u64(); + match h { + Some(h) if hs.contains(&h) => (404, None), + _ => (200, None), + } + } }; // Only a request the receiver actually accepted counts // as received; a 503'd attempt is a retry, not a @@ -102,12 +139,15 @@ impl MockReceiver { if status == 200 { g.received.push(req); } - status + (status, location) }; let reason = if status == 200 { "OK" } else { "Error" }; + let loc = location + .map(|l| format!("Location: {l}\r\n")) + .unwrap_or_default(); let _ = sock .write_all( - format!("HTTP/1.1 {status} {reason}\r\nContent-Length: 0\r\nConnection: close\r\n\r\n") + format!("HTTP/1.1 {status} {reason}\r\n{loc}Content-Length: 0\r\nConnection: close\r\n\r\n") .as_bytes(), ) .await; @@ -127,6 +167,28 @@ impl MockReceiver { Self::start(Behavior::FailFirst(code, n)).await } + /// Answer every request with a 302 pointing at `to`. + pub async fn redirecting_to(to: String) -> Self { + Self::start(Behavior::Redirect(to)).await + } + + /// Permanently reject `block_connected` at these heights; accept the rest. + pub async fn rejecting_heights(heights: Vec) -> Self { + Self::start(Behavior::RejectHeights(heights)).await + } + + /// Heights of every accepted `block_connected` delivery, in arrival order. + async fn accepted_heights(&self) -> Vec { + self.inner + .lock() + .await + .received + .iter() + .filter(|r| r.json()["body"]["kind"] == "block_connected") + .filter_map(|r| r.json()["body"]["height"].as_u64()) + .collect() + } + pub fn url(&self) -> String { format!("http://127.0.0.1:{}/hook", self.port) } @@ -162,6 +224,11 @@ impl MockReceiver { async fn attempts(&self) -> usize { self.inner.lock().await.seen } + + /// Every accepted delivery, in arrival order. + async fn all(&self) -> Vec { + self.inner.lock().await.received.clone() + } } trait BodiesForPanic { @@ -222,22 +289,25 @@ fn find_double_crlf(buf: &[u8]) -> Option { const SECRET: &str = "an-operator-chosen-signing-secret"; -/// Start a node whose `alertfile=` points at a one-hook file delivering -/// `categories` to `receiver`. +/// Write (or rewrite) the alertfile for a test node. /// -/// The alertfile *path* is restart-only (the `authfile` model — the dispatcher -/// binds to a path at startup), so it must be on the command line rather than -/// SIGHUP'd in. The file lives in its own temp dir, which the returned guard -/// keeps alive for the test's duration. -async fn node_with_hook( +/// `nudge` exists so a test can produce a *materially different* file: the +/// dispatcher deliberately treats a SIGHUP that did not change the alertfile as +/// a no-op, so a test that needs a fresh generation — and therefore a fresh +/// catch-up from the persisted cursor — has to actually change something. It +/// sets `allow_insecure_http`, which parses under every category and is a +/// no-op for the loopback URLs these tests use, so it changes the parsed hook +/// without changing behavior. +fn write_alertfile( + dir: &tempfile::TempDir, receiver: &MockReceiver, id: &str, categories: &str, - extra: Vec, -) -> (StreamingNode, tempfile::TempDir) { + nudge: bool, +) { use std::os::unix::fs::PermissionsExt as _; - let dir = tempfile::tempdir().expect("tempdir"); let path = dir.path().join("alertfile.toml"); + let hb = if nudge { "allow_insecure_http = true\n" } else { "" }; std::fs::write( &path, format!( @@ -247,7 +317,7 @@ id = "{id}" url = "{}" secret = "{SECRET}" categories = [{categories}] -"#, +{hb}"#, receiver.url() ), ) @@ -255,6 +325,24 @@ categories = [{categories}] // The dispatcher refuses a group/world-readable file; it holds the signing // secret. std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); +} + +/// Start a node whose `alertfile=` points at a one-hook file delivering +/// `categories` to `receiver`. +/// +/// The alertfile *path* is restart-only (the `authfile` model — the dispatcher +/// binds to a path at startup), so it must be on the command line rather than +/// SIGHUP'd in. The file lives in its own temp dir, which the returned guard +/// keeps alive for the test's duration. +async fn node_with_hook( + receiver: &MockReceiver, + id: &str, + categories: &str, + extra: Vec, +) -> (StreamingNode, tempfile::TempDir) { + let dir = tempfile::tempdir().expect("tempdir"); + write_alertfile(&dir, receiver, id, categories, false); + let path = dir.path().join("alertfile.toml"); let mut args = vec![format!("--alertfile={}", path.display())]; args.extend(extra); @@ -283,15 +371,38 @@ async fn webhook_delivers_a_signed_status_event() { assert!( got.signature_valid(SECRET), - "X-Satd-Signature must verify over the raw body", + "X-Satd-Signature must verify over the v2 signing string", ); assert_eq!(got.header("x-satd-hook"), "ops"); - assert_eq!(got.header("x-satd-webhook-version"), "1"); + assert_eq!(got.header("x-satd-webhook-version"), "2"); assert_eq!(got.header("x-satd-attempt"), "1"); assert!( !got.header("x-satd-delivery").is_empty(), "an idempotency key is always present", ); + // The timestamp is part of the signed material, so a receiver can bound + // replay of a captured delivery. + let ts: u64 = got + .header("x-satd-timestamp") + .parse() + .expect("X-Satd-Timestamp is present and numeric"); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + assert!(now.abs_diff(ts) < 300, "timestamp should be recent, got {ts} vs {now}"); + // Tampering with any signed field must invalidate the signature. The + // delivery id is the one that matters most: the contract tells receivers to + // deduplicate on it, so if it were unsigned an attacker holding one valid + // delivery could pre-poison a receiver's dedup cache against future alerts. + let mut forged = got.clone(); + forged + .headers + .insert("x-satd-delivery".into(), format!("{}-forged", got.header("x-satd-delivery"))); + assert!( + !forged.signature_valid(SECRET), + "the delivery id must be covered by the signature", + ); assert_eq!(got.header("content-type"), "application/json"); let body = got.json(); @@ -323,6 +434,87 @@ async fn webhook_retries_until_the_receiver_recovers() { assert!(receiver.attempts().await >= 3); } +/// A permanently-rejected event still advances the hook's resume cursor. +/// +/// Otherwise a receiver that hard-rejects one body shape turns every reload and +/// every restart into a rebuild of the same refused span — the events are lost +/// either way, and the only question is whether the hook makes progress past +/// them. Here blocks 2 and 3 are 404'd; the reload that follows re-runs catch-up +/// from the stored cursor, and must not replay them. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_permanent_rejection_advances_the_cursor() { + let receiver = MockReceiver::rejecting_heights(vec![2, 3]).await; + let (sn, _dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 3).await; + // Block 1 is accepted; 2 and 3 are refused. Wait for the last one to have + // been attempted, so the cursor writes have happened. + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 1) + .await; + tokio::time::sleep(Duration::from_secs(2)).await; + let before = receiver.attempts().await; + assert!(before >= 3, "expected all three blocks attempted, got {before}"); + + // A SIGHUP retires the generation and starts a new one, which re-runs + // catch-up from the persisted cursor. + // + // The alertfile must actually change: an unchanged one is deliberately a + // no-op, because retiring a generation destroys its queued deliveries and a + // status event has no replay to recover it. Without a real edit here this + // test would pass without ever re-running catch-up — the exact thing it + // exists to check. + write_alertfile(&_dir, &receiver, "chain", "\"chain\"", true); + crate::streaming::sighup_with_conf(&sn, "").await; + tokio::time::sleep(Duration::from_secs(3)).await; + + // Counting *attempts*, not accepted deliveries: a replayed block would be + // refused again and so would never show up as accepted. What matters is + // whether the request was made at all. + assert_eq!( + receiver.attempts().await, + before, + "a refused span must not be rebuilt and re-sent on reload", + ); + let heights = receiver.accepted_heights().await; + assert_eq!(heights, vec![1], "only block 1 was ever accepted; got {heights:?}"); + + // The hook is not wedged: the next block still arrives. + crate::streaming::mine_n(&sn, 1).await; + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 4) + .await; +} + +/// A redirect is never followed: the signed body does not go to a host the +/// alertfile never named. +/// +/// The configured URL is validated once, at load. If a 302 moved the request, +/// that check would be advisory only — and the interesting destinations are the +/// ones an operator cannot see from the outside: a cloud metadata endpoint, an +/// RFC1918 admin port, the node's own RPC. Here the redirect target is a second +/// mock receiver, which must never be touched. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn webhook_does_not_follow_redirects() { + let elsewhere = MockReceiver::ok().await; + let receiver = MockReceiver::redirecting_to(elsewhere.url()).await; + let (sn, _dir) = node_with_hook(&receiver, "redirector", "\"chain\"", vec![]).await; + + crate::streaming::mine_n(&sn, 2).await; + // Give the dispatcher room to deliver, retry, and give up. + tokio::time::sleep(Duration::from_secs(5)).await; + + assert!( + receiver.attempts().await >= 1, + "the configured endpoint should have been contacted", + ); + assert_eq!( + elsewhere.attempts().await, + 0, + "the redirect target must never receive the signed body", + ); +} + /// A receiver that permanently rejects one event must not pin the queue: the /// event is skipped and later events still arrive. #[tokio::test(flavor = "multi_thread", worker_threads = 2)] @@ -341,3 +533,68 @@ async fn webhook_permanent_rejection_does_not_wedge_the_queue() { // Exactly one rejection, and it did not stall what followed. assert!(receiver.attempts().await >= 2); } + +/// A restart-style catch-up must not hand every replayed block the same +/// idempotency key. +/// +/// Replayed envelopes are synthesized by the replay builder, which stamps every +/// one of them `seq: 0`. Minting the delivery id from that stamp gave a node +/// that had been down for N blocks N deliveries sharing one +/// `X-Satd-Delivery` — so a receiver following the contract ("deduplicate on +/// it") keeps the first and silently discards the rest. Catch-up exists +/// precisely to close that gap, so the bug quietly reduced it to delivering one +/// block per outage. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn catch_up_replay_gives_every_event_a_distinct_delivery_id() { + let receiver = MockReceiver::start(Behavior::Ok).await; + let (sn, dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; + + // Establish a cursor: one block delivered and acked. + crate::streaming::mine_n(&sn, 1).await; + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 1) + .await; + tokio::time::sleep(Duration::from_secs(2)).await; + + // Simulate the outage: retire the dispatcher by pointing the hook at a + // black hole, mine past it, then point it back. The blocks mined while the + // hook was dead are exactly the span catch-up must replay. + // Always-500 stands in for "the receiver is down": nothing is ever acked, + // so the shared cursor stays at block 1 and the blocks mined below are the + // span catch-up has to replay. + let sink = MockReceiver::start(Behavior::FailFirst(500, usize::MAX)).await; + write_alertfile(&dir, &sink, "chain", "\"chain\"", false); + crate::streaming::sighup_with_conf(&sn, "").await; + tokio::time::sleep(Duration::from_secs(2)).await; + crate::streaming::mine_n(&sn, 4).await; + tokio::time::sleep(Duration::from_secs(2)).await; + + let before = receiver.all().await.len(); + write_alertfile(&dir, &receiver, "chain", "\"chain\"", false); + crate::streaming::sighup_with_conf(&sn, "").await; + + // Wait for the replayed span to arrive. + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 5) + .await; + tokio::time::sleep(Duration::from_secs(2)).await; + + let replayed: Vec = receiver + .all() + .await + .into_iter() + .skip(before) + .map(|r| r.header("x-satd-delivery").to_string()) + .collect(); + assert!( + replayed.len() >= 3, + "expected a multi-block replay, got {replayed:?}" + ); + let unique: std::collections::HashSet<&String> = replayed.iter().collect(); + assert_eq!( + unique.len(), + replayed.len(), + "replayed events shared an idempotency key; a conforming receiver \ + would drop all but one: {replayed:?}" + ); +} From 4f2b74400d8bb60a468210bee36de4d426582bc8 Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Sat, 25 Jul 2026 16:02:01 -0600 Subject: [PATCH 03/11] alerting: fix the dispatcher's own review-round regressions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 4 of external review, scoped to the ~966 lines this branch added during round 3 — code written as fixes and never itself reviewed. The IBD gate inverted the feature's purpose. `is_initial_block_download()` is not a sync flag; it is `tip.header.time + 86400 < now`. A node whose tip stops advancing re-enters "IBD" a day later, so the entire alert path went dark during exactly the incident chain hooks exist to report — and the suppression was silent: no `gap.dropped`, no `Lagged`, and the first post-gate delivery then advanced the durable cursor across the whole suppressed span, making it unrecoverable on the next restart while the module claimed at-least-once. `Heartbeat` was not exempt either, so a multi-day sync made an external dead-man's switch declare a healthy node dead. Now latched on first leaving IBD, heartbeat-exempt, and suppressed events are counted as a gap so the receiver is told. The latch also keeps a per-event block-index lookup off the fan-in task once it fires. `catch_up` computed `count as u64 - overflowed` from a *shared* counter: `metrics.hook(id)` is process-lived and `apply()` spawns the new generation before retiring the old, so a retired generation still shedding events could drive the subtraction negative — a panic under debug_assertions, a wrap to 1.8e19 in release. The task is spawned bare with no supervisor, so that killed the dispatcher, all hooks, for the life of the process. `enqueue` now returns whether it queued and callers count locally. `Disposition::Drop` was the one drop path that never set `gap.dropped`, and it is the one most likely to fire on a routine misconfiguration — a 3xx from an `http://` URL behind a TLS-terminating proxy, or a 401 from a rotated secret or a delivery that aged past the receiver's freshness window mid-retry. Combined with the cursor advancing past it, an hour of chain history could be discarded with nothing said on either end. The replay dedup map held every replayed height, but the replay builds up to 10,000 events into a queue holding 1024. The overflowing tail is exactly the span still buffered on the broadcast, so those blocks' live copies were then suppressed as "already replayed" — dropped twice. Now restricted to heights actually queued. Replayed blocks take a delivery id derived from their height (`block_delivery_id`, `b`). Restart and reload-overlap replays are the only duplicates this design actually produces, and every counter-minted id embeds a random per-process `instance_id` — so "deduplicate on X-Satd-Delivery" was unimplementable for the only case that needs it. Also: cursor GC now reconciles against the stored keyspace rather than the previous generation, so removing a hook by editing the file and *restarting* — the ordinary way — no longer leaks its cursor forever (new `Store::list_alert_cursor_keys`); gap state is process-lived, so a reload no longer discards a pending `Lagged`; a gap notice is flushed on a timer, so it cannot wait forever on a hook that went quiet; the resume anchor is seeded from the durable cursor rather than advertising height 0; and the "delivery task is gone" warning is logged once per generation instead of once per event, which on a busy mempool was thousands of lines a second filling the disk `disk_low` watches. Security: the userinfo SSRF fix was bypassable with a backslash. `http://evil.example\@127.0.0.1/hook` — the WHATWG parser treats `\` as a path separator for special schemes, so reqwest resolves `evil.example` while `is_local_target` saw a loopback address, waived `allow_insecure_http`, and posted the signed body in cleartext to the attacker's host. The Operator Manual still documented the v1 body-only signature and version 1, with no timestamp header — anyone writing a receiver from it, which is most of them, would have rejected 100% of deliveries. Rewritten for v2 with the verification steps, the freshness requirement, and the delivery-id namespaces. Tests: the SIGHUP no-op fix had no regression test at all — reverting it left the whole suite green. Added one that keeps a status event in retry backoff across an unrelated reload. The retry test now pins the sign-once-per-event invariant across attempts (it previously asserted only on the accepted attempt, so moving the signing back inside the retry loop passed unchanged). Two tests no longer depend on the host's free space: they drive the threshold to a known-cleared state first, through the conf file, since a CLI value would win over every later SIGHUP. Each new guard verified to fail against the code it guards. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH --- docs/manual/src/observability.md | 69 +++++-- node/src/storage/coin_cache.rs | 4 + node/src/storage/mod.rs | 13 ++ node/src/storage/rocksdb_store.rs | 24 +++ node/src/storage/split_store.rs | 4 + satd-alert/src/config.rs | 53 +++++- satd-alert/src/contract.rs | 56 ++++++ satd-alert/src/lib.rs | 3 +- satd/src/alert.rs | 286 ++++++++++++++++++++++++------ satd/tests/e2e/webhooks.rs | 150 +++++++++++++++- 10 files changed, 588 insertions(+), 74 deletions(-) diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index e6b963597..5070e82f8 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -182,11 +182,12 @@ worse failure. ```http POST /your/endpoint HTTP/1.1 Content-Type: application/json -X-Satd-Signature: sha256= +X-Satd-Signature: sha256= +X-Satd-Timestamp: 1753400000 X-Satd-Delivery: -- X-Satd-Hook: pager X-Satd-Attempt: 1 -X-Satd-Webhook-Version: 1 +X-Satd-Webhook-Version: 2 {"schema_version":1,"stamp":{...},"body":{"category":"status", ...}} ``` @@ -196,10 +197,45 @@ the same event, so a receiver parses webhook bodies and streaming frames with one code path. Delivery metadata rides in headers and never in the body — which is what makes the signature stable across retries. -Verify the signature over the **raw body, before parsing**. Deduplicate on -`X-Satd-Delivery`: it is stable across retries of one event and unique across -restarts, so a retry and a genuine repeat are distinguishable. Reply `2xx` to -acknowledge. +### Verifying a delivery + +The signature covers a **canonical string**, not the bare body — five fields +joined by newlines, with the raw body last: + +``` +"2" LF LF LF LF +``` + +Signing only the body would leave `X-Satd-Delivery` unauthenticated *and* +predictable, and that header is the one this page tells you to deduplicate on. +One captured `(body, signature)` pair could then be replayed under forged future +delivery ids, poisoning your dedup cache so the real alerts were discarded on +arrival while satd counted them delivered. Binding the id, the hook, and a +timestamp into the signed material closes that. + +So a receiver must: + +1. Read `X-Satd-Timestamp` and reject anything older than **600 seconds**. This + is not optional — it is what stops a captured delivery being a permanent + replay token. A delivery still being retried after the window ages out by + design; a 20-minute-old "disk is filling" alert is not worth acting on. +2. Rebuild the canonical string above from the raw body, **before parsing it**. +3. Compare the HMAC in constant time. +4. Deduplicate on `X-Satd-Delivery` — stable across retries of one event, and + unique across restarts, so a retry and a genuine repeat are distinguishable. +5. Reply `2xx` to acknowledge. + +> **Upgrading from the pre-release v1 scheme.** Earlier drafts signed the raw +> body alone and sent `X-Satd-Webhook-Version: 1`. The legacy `reorgwebhook=` +> keys still use exactly that, unchanged, and still report version `1` — branch +> on the version header rather than assuming one scheme. + +The `X-Satd-Delivery` value is opaque; do not parse it. Its suffix distinguishes +how the delivery arose — a bare counter for a live event, `w` for a watch match, +`r` for a synthesized notice such as `lagged`, and `b` for a replayed +block. That last one is deliberately derived from the height rather than a +counter, so the same block replayed after a restart carries the same id and your +dedup actually collapses it. ### Delivery behavior @@ -222,12 +258,23 @@ acknowledge. the cursor to resume from. A gap is never silent. - **Nothing reaches consensus.** Deliveries run on the isolated API runtime; a stalled endpoint cannot affect block connection. -- **At-least-once for confirmed chain events across a restart.** Each hook's - resume position is persisted, and on startup the hook replays what it missed - (bounded by the same 10 000-block window the streaming API uses; beyond that - you get a `lagged` notice instead). Health events are re-raised by +- **At-least-once for confirmed chain events across a restart, or you are told.** + Each hook's resume position is persisted, and on startup the hook replays what + it missed (bounded by the same 10 000-block window the streaming API uses; + beyond that you get a `lagged` notice instead). Health events are re-raised by re-evaluation rather than replayed, and mempool events are best-effort — the - same contract the event bus itself offers. + same contract the event bus itself offers. Every way an event can be skipped — + queue overflow, a permanent rejection, suppression during a sync — increments + the hook's drop count and produces a `lagged` body, so the guarantee is + precisely "delivered, or reported missing"; it is never silent. +- **Suppressed during initial block download.** A node syncing from scratch does + not POST its entire history: while it is catching up, only `status` and + `heartbeat` events are delivered. Health alerts stay live because "this node is + unhealthy" is exactly as true mid-sync, and the heartbeat keeps flowing so an + external dead-man's switch does not declare a syncing node dead. Everything + suppressed is counted and reported in the next `lagged` body. The suppression + is latched on leaving IBD once, so a node whose tip later goes stale keeps + alerting — which is the whole point of a stalled-tip alert. Plaintext `http://` is accepted for loopback and private-network targets. For a public host, use `https://` or set `allow_insecure_http = true` on the hook. diff --git a/node/src/storage/coin_cache.rs b/node/src/storage/coin_cache.rs index 56e40b7fc..4df597d09 100644 --- a/node/src/storage/coin_cache.rs +++ b/node/src/storage/coin_cache.rs @@ -1292,6 +1292,10 @@ impl Store for CoinCache { self.inner.write_alert_cursor(key, value) } + fn list_alert_cursor_keys(&self) -> Result>, StoreError> { + self.inner.list_alert_cursor_keys() + } + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { self.inner.delete_alert_cursor(key) } diff --git a/node/src/storage/mod.rs b/node/src/storage/mod.rs index ffebaddb6..d0d87216d 100644 --- a/node/src/storage/mod.rs +++ b/node/src/storage/mod.rs @@ -878,4 +878,17 @@ pub trait Store: Send + Sync { "delete_alert_cursor not supported on this backend".into(), )) } + + /// Every persisted alert-cursor key, for reconciling against the configured + /// hooks. + /// + /// Removing a hook from the alertfile and restarting is the ordinary way + /// operators change this configuration, and a reload-time diff cannot see + /// it: at startup there is no previous generation to compare against. So + /// the dispatcher enumerates what is actually stored and drops whatever no + /// longer has a hook, which also cleans up cursors left behind by earlier + /// versions that had no GC at all. + fn list_alert_cursor_keys(&self) -> Result>, StoreError> { + Ok(Vec::new()) + } } diff --git a/node/src/storage/rocksdb_store.rs b/node/src/storage/rocksdb_store.rs index 0ee2a24a5..5d68cf988 100644 --- a/node/src/storage/rocksdb_store.rs +++ b/node/src/storage/rocksdb_store.rs @@ -1276,6 +1276,30 @@ impl Store for RocksDbStore { .map_err(|e| StoreError::Database(e.to_string())) } + fn list_alert_cursor_keys(&self) -> Result>, StoreError> { + const PREFIX: &[u8] = b"alertwebhook.cursor."; + let cf = self + .db + .cf_handle(CF_METADATA) + .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; + let mut keys = Vec::new(); + // The metadata CF is shared with index markers, so the iteration is + // prefix-bounded and stops at the first key outside the namespace + // rather than walking the whole CF. + let iter = self.db.iterator_cf( + &cf, + rocksdb::IteratorMode::From(PREFIX, rocksdb::Direction::Forward), + ); + for item in iter { + let (key, _) = item.map_err(|e| StoreError::Database(e.to_string()))?; + if !key.starts_with(PREFIX) { + break; + } + keys.push(key.to_vec()); + } + Ok(keys) + } + fn get_block_index(&self, hash: &BlockHash) -> Option { let cf = self.cf(CF_BLOCK_INDEX); let value = self.db.get_cf(&cf, hash_bytes(hash)).ok()??; diff --git a/node/src/storage/split_store.rs b/node/src/storage/split_store.rs index 2211217e7..85a309214 100644 --- a/node/src/storage/split_store.rs +++ b/node/src/storage/split_store.rs @@ -100,6 +100,10 @@ impl Store for SplitStore { self.block_store.write_alert_cursor(key, value) } + fn list_alert_cursor_keys(&self) -> Result>, StoreError> { + self.block_store.list_alert_cursor_keys() + } + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { self.block_store.delete_alert_cursor(key) } diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs index cb0c5d6bb..14b470db2 100644 --- a/satd-alert/src/config.rs +++ b/satd-alert/src/config.rs @@ -477,7 +477,16 @@ fn validate_url( /// network) and demanding TLS would just push operators to set the override. fn is_local_target(rest: &str) -> bool { // Authority is everything before the path, query, or fragment. - let authority = rest.split(['/', '?', '#']).next().unwrap_or(""); + // + // A backslash terminates the authority too. The WHATWG URL parser — which + // is what `reqwest` resolves this string with — treats `\` as a path + // separator for special schemes, so `http://evil.example\@127.0.0.1/hook` + // has host `evil.example`, while splitting on `/` alone leaves an authority + // of `evil.example\@127.0.0.1` whose last `@` yields `127.0.0.1`. That + // reads as loopback, waives the `allow_insecure_http` gate, and posts the + // signed body in cleartext to the attacker's host — the same bypass the + // userinfo rule below closes, through a different separator. + let authority = rest.split(['/', '\\', '?', '#']).next().unwrap_or(""); // Strip userinfo — everything through the *last* `@`. // // Without this, `http://127.0.0.1:8332@evil.example/hook` reads as host @@ -747,6 +756,48 @@ categories = ["chain"] assert!(parse(&text).is_ok(), "userinfo on a real loopback host is fine"); } + #[test] + fn a_backslash_cannot_smuggle_a_public_host_past_the_local_check() { + // The WHATWG URL parser (what `reqwest` resolves these with) treats `\` + // as a path separator for special schemes, so the host here is + // `evil.example` — but splitting the authority on `/` alone leaves + // `evil.example\@127.0.0.1`, whose last `@` yields a loopback address. + // Same bypass as the userinfo case, through a different separator. + // Written as TOML *literal* strings (single-quoted). A backslash in a + // basic string is an escape, so `"...\@..."` fails at the TOML layer + // and would make this test pass without ever reaching the URL check. + for sneaky in [ + r"http://evil.example\@127.0.0.1/hook", + r"http://evil.example\@10.0.0.1/hook", + r"http://evil.example\@localhost:8080/hook", + ] { + let text = format!( + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = '{sneaky}'\n\ + secret = \"s3cret\"\n\ + categories = [\"status\"]\n" + ); + assert!( + parse(&text).is_err(), + "{sneaky} resolves to a public host and must need the opt-in" + ); + } + + // A backslash in the *path* of a genuinely local target is unaffected. + let text = format!( + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = '{}'\n\ + secret = \"s3cret\"\n\ + categories = [\"status\"]\n", + r"http://127.0.0.1:9000/a\b" + ); + assert!(parse(&text).is_ok()); + } + #[test] fn non_http_schemes_are_rejected() { for url in ["ftp://x/h", "file:///etc/passwd", "x.example/h", ""] { diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs index c9cc86d96..c1495ad84 100644 --- a/satd-alert/src/contract.rs +++ b/satd-alert/src/contract.rs @@ -94,6 +94,26 @@ pub fn replay_delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> Stri format!("{node_id_hex}-{instance_id}-r{seq}") } +/// Delivery id for a replayed **confirmed block**, derived from its height. +/// +/// Deterministic on purpose, and the only id in the scheme that is. Every other +/// form embeds this process's random `instance_id` and a running counter, which +/// is right for an event that happens once — but a replayed block is precisely +/// the event this design can deliver twice: a restart replays from the durable +/// cursor, and a reload can leave two generations briefly overlapping. Under a +/// counter-minted id those duplicates arrive with *different* headers, which +/// makes "deduplicate on `X-Satd-Delivery`" unimplementable for the only +/// duplicate that actually occurs. Keyed on height, both copies carry the same +/// id and a conforming receiver collapses them, which is the contract. +/// +/// The instance component is fixed at `0` — a real `instance_id` is a random +/// `u64` and would reintroduce the per-process variation this exists to remove. +/// The `b` prefix keeps the space disjoint from the bus (``), watch +/// (`w`), and synthesized (`r`) spaces. +pub fn block_delivery_id(node_id_hex: &str, height: u32) -> String { + format!("{node_id_hex}-0-b{height}") +} + /// Maximum age a receiver should accept for a v2 delivery, in seconds. /// /// Normative for receivers, advisory here: satd stamps `X-Satd-Timestamp` and @@ -256,4 +276,40 @@ mod tests { // changed. Without this a receiver would dedupe away real events. assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 8, 42)); } + + /// The id spaces must not overlap. They are minted from independent + /// counters — the bus `seq`, a synth counter, and a block height — so + /// without the prefixes a replayed block at height 500 and a bus event with + /// `seq` 500 would collide, and a conforming receiver would silently + /// discard the second. (The watch space, `w`, is added with the + /// watch-hook feature and is covered alongside it.) + #[test] + fn the_delivery_id_spaces_are_disjoint() { + let node = "ab".repeat(16); + let n = 500u64; + let ids = [ + delivery_id(&node, 7, n), + replay_delivery_id(&node, 7, n), + block_delivery_id(&node, n as u32), + ]; + for (i, a) in ids.iter().enumerate() { + for b in ids.iter().skip(i + 1) { + assert_ne!(a, b, "delivery id spaces collide at the same counter value"); + } + } + } + + /// A replayed block's id is derived from its height alone, so the same + /// block delivered twice — a restart replaying from the durable cursor, or + /// a reload whose generations briefly overlap — carries the same + /// idempotency key. Every other id embeds a random per-process + /// `instance_id`, which would make those duplicates undedupable. + #[test] + fn a_replayed_block_id_is_stable_across_restarts() { + let node = "ab".repeat(16); + assert_eq!(block_delivery_id(&node, 840_000), block_delivery_id(&node, 840_000)); + assert_ne!(block_delivery_id(&node, 840_000), block_delivery_id(&node, 840_001)); + // No instance component to vary: this is the whole point. + assert!(block_delivery_id(&node, 840_000).ends_with("-0-b840000")); + } } diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs index d93401077..a2eeea1e7 100644 --- a/satd-alert/src/lib.rs +++ b/satd-alert/src/lib.rs @@ -27,7 +27,8 @@ pub mod retry; pub use config::{AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY}; pub use contract::{ - delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, ATTEMPT_HEADER, + block_delivery_id, delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, + ATTEMPT_HEADER, DELIVERY_HEADER, HOOK_HEADER, LEGACY_WEBHOOK_VERSION, MAX_TIMESTAMP_SKEW_SECS, SIGNATURE_HEADER, TIMESTAMP_HEADER, WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, }; diff --git a/satd/src/alert.rs b/satd/src/alert.rs index 737fca8e5..98b21a0d3 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -51,6 +51,9 @@ const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); /// "resync below this height yourself" boundary. const MAX_CATCHUP_BLOCKS: u32 = node::events::MAX_REPLAY_BLOCKS; +/// How often a pending gap notice is flushed when no event is driving the hook. +const GAP_FLUSH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15); + /// Numbers synthesized deliveries — catch-up replay events and gap notices. /// /// Process-wide rather than per-generation, for the same reason the watch @@ -111,6 +114,11 @@ struct HookChannel { /// Set when an event was dropped for this hook; cleared when the resulting /// `Lagged` notice has been queued. gap: Arc, + /// Whether the "delivery task is gone" warning has already been logged for + /// this generation. Per-generation on purpose: a reload should re-report a + /// genuinely dead hook once, not stay quiet because a retired generation + /// already said so. + reported_closed: std::sync::atomic::AtomicBool, } /// Accumulated drop state for one hook. @@ -122,6 +130,33 @@ struct GapState { resume_height: std::sync::atomic::AtomicU32, } +/// Per-hook gap state, keyed by hook id, living as long as the process. +/// +/// Deliberately not per-generation. A reload retires the generation that +/// accumulated a drop count, and per-generation state would take the pending +/// `Lagged` notice down with it — the receiver would never learn about a hole +/// that satd had already recorded. `HookCounters` is process-lived for the same +/// reason; this is the half that was missed. +static GAP_STATE: std::sync::LazyLock< + parking_lot::Mutex>>, +> = std::sync::LazyLock::new(|| parking_lot::Mutex::new(std::collections::HashMap::new())); + +fn gap_state_for(id: &str) -> Arc { + let mut map = GAP_STATE.lock(); + Arc::clone(map.entry(id.to_string()).or_default()) +} + +/// Whether this node has ever been observed out of initial block download. +/// +/// `is_initial_block_download()` is a function of the tip header's age, not of +/// whether a sync is in progress: a node whose tip stops advancing crosses back +/// into "IBD" 24 hours later. Without this latch the alert path would go dark +/// during exactly the incident it exists to report — a stalled tip — and, worse, +/// resume by advancing the durable cursor across everything it suppressed. +/// Once a node has been caught up, it is never "syncing" again for the +/// dispatcher's purposes. +static LEFT_IBD: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + /// Spawn the dispatcher for a parsed alertfile. /// /// Must be called from within the API runtime — every task spawned here does @@ -143,11 +178,12 @@ fn spawn_with_metrics( for hook in &file.hooks { let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); let counters = metrics.hook(&hook.id); - let gap = Arc::new(GapState::default()); + let gap = gap_state_for(&hook.id); tokio::spawn(deliver_loop( hook.clone(), rx, counters.clone(), + gap.clone(), store.clone(), stop.clone(), )); @@ -157,6 +193,7 @@ fn spawn_with_metrics( tx, counters, gap, + reported_closed: std::sync::atomic::AtomicBool::new(false), }); } tokio::spawn(fan_in(hooks, publisher, store, block_source, stop)); @@ -197,9 +234,22 @@ async fn fan_in( "alert webhook dispatcher started", ); + // A gap notice is otherwise only emitted ahead of the *next* event for that + // hook, so a hook whose traffic stops right after a drop holds it + // indefinitely — and traffic stopping is correlated with the drop, not + // independent of it. The one message that must not wait is the one saying + // data was lost. + let mut gap_flush = tokio::time::interval(GAP_FLUSH_INTERVAL); + gap_flush.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + loop { tokio::select! { _ = stop.wait() => break, + _ = gap_flush.tick() => { + for hook in &hooks { + flush_gap(hook, &publisher); + } + } recv = rx.recv() => match recv { Ok(env) => { // Suppress the firehose during initial block download. @@ -213,16 +263,31 @@ async fn fan_in( // multi-day firehose at the receiver rather than anything // that looks like a bug locally. // - // Status events are exempt: "this node is unhealthy" is - // exactly as true during IBD, and the detectors already - // suppress the conditions that are meaningless while - // syncing. - let syncing = block_source - .as_deref() - .is_some_and(|s| s.in_initial_block_download()); - if syncing && !matches!(env.body, NodeEventBody::Status(_)) { - continue; - } + // Latched (see `LEFT_IBD`), because the predicate is the + // tip's age rather than a sync flag. Once latched the + // `block_source` call is skipped entirely, which also keeps + // a per-event RocksDB block-index lookup off this task. + // + // Status and heartbeat events are exempt. "This node is + // unhealthy" is exactly as true during IBD, and the + // detectors already suppress the conditions that are + // meaningless while syncing. A heartbeat is a dead-man's + // switch: suppressing it during a multi-day sync makes an + // external watchdog declare a healthy node dead, which + // inverts the signal it exists to carry. + let syncing = !LEFT_IBD.load(Ordering::Relaxed) && { + let in_ibd = block_source + .as_deref() + .is_some_and(|s| s.in_initial_block_download()); + if !in_ibd { + LEFT_IBD.store(true, Ordering::Relaxed); + } + in_ibd + }; + let exempt = matches!( + env.body, + NodeEventBody::Status(_) | NodeEventBody::Heartbeat { .. } + ); // Decide who wants this *before* rendering it. Serializing // first meant a `BlockTweaks` envelope — hundreds of KB of // per-block silent-payment rows — was rendered in full and @@ -237,6 +302,22 @@ async fn fan_in( if wanted.is_empty() { continue; } + if syncing && !exempt { + // Record the suppression as a gap rather than dropping + // it silently. A silent skip would leave no counter + // moved and no `lagged` body emitted, and the first + // post-IBD delivery would then advance the durable + // cursor straight across everything suppressed — making + // the span unrecoverable on the next restart while the + // module claims at-least-once. Counting it means the + // receiver is told, and the resume anchor still points + // before the hole. + for i in wanted { + hooks[i].gap.dropped.fetch_add(1, Ordering::Relaxed); + hooks[i].counters.dropped.fetch_add(1, Ordering::Relaxed); + } + continue; + } // Render once for every hook: the body a receiver verifies // must be the bytes a WS subscriber would have seen, and // rendering per hook risks two receivers disagreeing. @@ -289,7 +370,17 @@ async fn fan_in( } /// Queue an event for one hook, converting an overflow into a recorded gap. -fn enqueue(hook: &HookChannel, publisher: &EventPublisher, item: Delivery, cursor: Option) { +/// +/// Returns whether the event was queued. Callers that need a count must use +/// this rather than diffing `counters.dropped`: those counters are process-lived +/// and shared across generations, so a retired generation still shedding events +/// perturbs the delta — which, on a subtraction, underflowed. +fn enqueue( + hook: &HookChannel, + publisher: &EventPublisher, + item: Delivery, + cursor: Option, +) -> bool { // Emit the pending gap notice ahead of the event that follows it, so the // receiver learns about the hole before it sees the data after it. flush_gap(hook, publisher); @@ -301,12 +392,14 @@ fn enqueue(hook: &HookChannel, publisher: &EventPublisher, item: Delivery, curso if let Some(c) = cursor { hook.gap.resume_height.store(c.height, Ordering::Relaxed); } + true } Err(mpsc::error::TrySendError::Full(_)) => { // A receiver that cannot keep up degrades to "you missed N" rather // than back-pressuring the bus. hook.gap.dropped.fetch_add(1, Ordering::Relaxed); hook.counters.dropped.fetch_add(1, Ordering::Relaxed); + false } Err(mpsc::error::TrySendError::Closed(_)) => { // The delivery task is gone: it failed to build its HTTP client, or @@ -317,11 +410,19 @@ fn enqueue(hook: &HookChannel, publisher: &EventPublisher, item: Delivery, curso // "no successful delivery in N minutes" rule fires. hook.gap.dropped.fetch_add(1, Ordering::Relaxed); hook.counters.dropped.fetch_add(1, Ordering::Relaxed); - tracing::warn!( - target: "alert", - hook = %hook.id, - "hook delivery task is gone; event dropped", - ); + // Logged once per hook, not once per event. This arm is reached on + // every event once a delivery task is gone — including for a + // generation being retired by a reload, where it is expected — and + // an unrated line here is thousands per second on a busy mempool, + // filling the very disk `disk_low` is watching. + if !hook.reported_closed.swap(true, Ordering::Relaxed) { + tracing::warn!( + target: "alert", + hook = %hook.id, + "hook delivery task is gone; events for this hook are being dropped", + ); + } + false } } } @@ -431,6 +532,14 @@ async fn catch_up( // replaying history nobody asked for. return Default::default(); }; + // Seed the gap anchor from the durable cursor before any replay is queued. + // `resume_height` is otherwise only set by a successful enqueue, so a hook + // that lags before its first one — a fresh generation whose catch-up burst + // overflows immediately — would emit a `Lagged` advertising height 0, which + // reads as "resume from genesis" rather than "resume from where you were". + hook.gap + .resume_height + .store(cursor.height, Ordering::Relaxed); let replay = node::events::build_cursor_replay( src, @@ -458,31 +567,58 @@ async fn catch_up( ); } let count = replay.events.len(); - let before = hook.counters.dropped.load(Ordering::Relaxed); + let mut queued = 0u64; + let mut overflowed = 0u64; + let mut queued_heights: std::collections::HashSet = std::collections::HashSet::new(); for env in replay.events { + let height = match &env.body { + NodeEventBody::Chain(node::chain::events::ChainEvent::BlockConnected { + height, + .. + }) => Some(*height), + _ => None, + }; let Ok(bytes) = serde_json::to_vec(&env) else { continue; }; let cursor = env.cursor; - enqueue( + // Every replayed envelope is stamped `seq: 0`, so minting from it would + // give a node that was down for 100 blocks 100 deliveries sharing one + // idempotency key — the whole point of catch-up defeated silently. + // + // A replayed block takes an id derived from its height rather than a + // process counter. This is the one event the design really can deliver + // twice (a restart, or a reload whose generations overlap), and every + // counter-minted id embeds a per-process random `instance_id`, so the + // duplicate would arrive under a fresh id and "deduplicate on + // X-Satd-Delivery" would be unimplementable for the only case that + // needs it. Non-block replays keep the counter — they carry no durable + // position to key on. + let delivery_id = match height { + Some(h) => satd_alert::block_delivery_id(&hex::encode(env.stamp.node_id), h), + None => satd_alert::replay_delivery_id( + &hex::encode(env.stamp.node_id), + publisher.instance_id(), + SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), + ), + }; + if enqueue( hook, publisher, Delivery::Event { body: Arc::new(bytes), - // Every replayed envelope is stamped `seq: 0`, so minting from - // it would give a node that was down for 100 blocks 100 - // deliveries sharing one idempotency key — a conforming - // receiver keeps the first and discards the other 99, which is - // the whole point of catch-up defeated silently. - delivery_id: satd_alert::replay_delivery_id( - &hex::encode(env.stamp.node_id), - publisher.instance_id(), - SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), - ), + delivery_id, cursor, }, cursor, - ); + ) { + queued += 1; + if let Some(h) = height { + queued_heights.insert(h); + } + } else { + overflowed += 1; + } } if count > 0 { // Report what was *queued*, not what was built. The 10k-block replay @@ -491,11 +627,6 @@ async fn catch_up( // about a week and 10k is about ten. Overflow inside this loop is // counted but was otherwise unlogged, so the success line reported the // pre-truncation figure and read as if everything had been recovered. - let overflowed = hook - .counters - .dropped - .load(Ordering::Relaxed) - .saturating_sub(before); if overflowed > 0 { tracing::warn!( target: "alert", @@ -509,12 +640,20 @@ async fn catch_up( tracing::info!( target: "alert", hook = %hook.id, - events = count as u64 - overflowed, + events = queued, from = cursor.height, "webhook catch-up queued events missed while the daemon was down", ); } - replay.confirmed_dedup + // Only suppress the live copy of a block this hook actually received. The + // replay builds up to 10,000 events into a queue that holds 1024, so on a + // long outage most of the tail overflows — and the tail is exactly the span + // still buffered on the broadcast, i.e. the blocks whose live copies are + // about to arrive. Keeping a dropped block in the dedup map suppresses that + // live copy too, dropping the same block twice. + let mut dedup = replay.confirmed_dedup; + dedup.retain(|height, _| queued_heights.contains(height)); + dedup } fn read_cursor(store: &dyn Store, hook: &Hook) -> Option { @@ -610,6 +749,7 @@ async fn deliver_loop( hook: Hook, mut rx: mpsc::Receiver, counters: Arc, + gap: Arc, store: Arc, mut stop: Stop, ) { @@ -692,6 +832,18 @@ async fn deliver_loop( delivery = %delivery_id, "webhook receiver rejected the delivery permanently; skipping this event", ); + // Record it as a gap, so the receiver is told in-band. + // + // This is the one drop path that did not, and it is the one + // most likely to fire on a routine misconfiguration: a 3xx + // (an `http://` URL behind a proxy that redirects to HTTPS) + // and a 401 (a rotated secret, or a delivery that aged past + // the receiver's freshness window while being retried) are + // both permanent by this classification. Without a `lagged` + // body the receiver has no way to learn which events it + // lost, and — because the cursor advances below — no way to + // recover them after the misconfiguration is fixed. + gap.dropped.fetch_add(1, Ordering::Relaxed); // Advance the cursor anyway. A permanent rejection is the // receiver's decision about this event, and it will decide // the same way next time — leaving the cursor parked would @@ -699,8 +851,9 @@ async fn deliver_loop( // against an endpoint that has already refused it, forever. // The event is lost either way; the difference is whether // the hook makes progress past it. The drop is counted - // (`satd_alertwebhook_dropped_total`) and logged, so the loss is - // visible rather than silent. + // (`satd_alertwebhook_dropped_total`), logged, and now + // announced to the receiver, so the loss is visible on both + // ends rather than silent. persist_cursor(&hook, store.as_ref(), cursor, &mut persisted_height); break; } @@ -860,26 +1013,45 @@ impl AlertReloader { // behind means a later hook that happens to reuse the id inherits a // stale resume position and greets a brand-new endpoint with the whole // replay window of its predecessor's history. - if let Some(prev) = self.last_applied.lock().as_ref() { - for gone in prev - .hooks - .iter() - .filter(|h| !keep.iter().any(|k| k == &h.id)) - { - match self.store.delete_alert_cursor(&gone.cursor_key()) { - Ok(()) => tracing::info!( - target: "alert", - hook = %gone.id, - "hook removed from the alertfile; dropped its resume cursor", - ), - Err(e) => tracing::warn!( - target: "alert", - hook = %gone.id, - error = %e, - "failed to drop the removed hook's resume cursor", - ), + // + // Reconciled against what is actually stored rather than against the + // previous generation. A reload-time diff cannot see the ordinary case: + // an operator who removes a hook by editing the file and *restarting* + // leaves no previous generation to compare against, so the cursor + // survived forever. Enumerating the keyspace also cleans up whatever + // earlier versions left behind. + let live: std::collections::HashSet> = file + .hooks + .iter() + .map(|h| h.cursor_key()) + .chain(std::iter::once( + format!("alertwebhook.cursor.{LEGACY_REORG_HOOK_ID}").into_bytes(), + )) + .collect(); + match self.store.list_alert_cursor_keys() { + Ok(stored) => { + for key in stored.into_iter().filter(|k| !live.contains(k)) { + let label = String::from_utf8_lossy(&key).to_string(); + match self.store.delete_alert_cursor(&key) { + Ok(()) => tracing::info!( + target: "alert", + key = %label, + "no hook owns this resume cursor; dropped it", + ), + Err(e) => tracing::warn!( + target: "alert", + key = %label, + error = %e, + "failed to drop an orphaned resume cursor", + ), + } } } + Err(e) => tracing::warn!( + target: "alert", + error = %e, + "could not enumerate stored resume cursors; orphans were not reclaimed", + ), } *self.last_applied.lock() = Some(file); Ok(hook_count) diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index 3233347d1..2ca7695df 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -89,6 +89,8 @@ struct Inner { behavior: Behavior, seen: usize, received: Vec, + /// Every request, accepted or refused — what `received` deliberately is not. + attempted: Vec, } /// A scriptable HTTP receiver on loopback. @@ -105,6 +107,7 @@ impl MockReceiver { behavior, seen: 0, received: Vec::new(), + attempted: Vec::new(), })); let task_inner = inner.clone(); tokio::spawn(async move { @@ -135,7 +138,10 @@ impl MockReceiver { }; // Only a request the receiver actually accepted counts // as received; a 503'd attempt is a retry, not a - // delivery. + // delivery. Every attempt is kept separately, so a test + // can assert on what changes (and what must not) across + // the retries of one event. + g.attempted.push(req.clone()); if status == 200 { g.received.push(req); } @@ -225,6 +231,11 @@ impl MockReceiver { self.inner.lock().await.seen } + /// Every request the receiver saw, accepted or not, in arrival order. + async fn all_attempts(&self) -> Vec { + self.inner.lock().await.attempted.clone() + } + /// Every accepted delivery, in arrival order. async fn all(&self) -> Vec { self.inner.lock().await.received.clone() @@ -359,6 +370,19 @@ async fn webhook_delivers_a_signed_status_event() { let receiver = MockReceiver::ok().await; let (sn, _dir) = node_with_hook(&receiver, "ops", "\"status\"", vec![]).await; + // Drive `disk_low` to a known-cleared state first, so the raise below is + // guaranteed to be an edge. + // + // At the 10 GiB default this test otherwise depends on the host's free + // space: on a machine under the floor the condition is already raised + // before the dispatcher attaches, the raise SIGHUP hits `raise_if_new`'s + // no-op, and the whole v2-signature guard dies as a bare 60 s timeout with + // nothing to say why. A 1 MiB floor is below any filesystem that can hold a + // datadir, so this clears on every host. It has to go through the conf file + // rather than the command line — a CLI value wins over the conf, so a CLI + // threshold would make every later SIGHUP retune a no-op. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + // Raise `disk_low` *after* the dispatcher is attached, by moving the // threshold live rather than starting with it tripped: a status event is // not replayable, so a condition raised during startup would never reach a @@ -432,6 +456,96 @@ async fn webhook_retries_until_the_receiver_recovers() { let attempt: u32 = got.header("x-satd-attempt").parse().unwrap_or(0); assert!(attempt >= 3, "expected the 3rd attempt to succeed, got {attempt}"); assert!(receiver.attempts().await >= 3); + + // One event is signed exactly once, and every attempt carries that same + // signature, timestamp, and delivery id — only `X-Satd-Attempt` varies. + // + // This is the invariant, not an implementation detail. Re-signing per + // attempt would refresh `X-Satd-Timestamp`, which is what makes a delivery + // age out of the receiver's freshness window while it is still being + // retried — deliberate, since a 20-minute-old alert is not worth acting on. + // Asserting only on the accepted attempt (as this test used to) passes just + // as happily with the signing moved inside the retry loop. + let attempts = receiver.all_attempts().await; + let chain: Vec<_> = attempts + .iter() + .filter(|r| r.json()["body"]["category"] == "chain") + .collect(); + assert!(chain.len() >= 3, "expected 3 attempts at one event, got {}", chain.len()); + let first = chain[0]; + for (i, r) in chain.iter().enumerate() { + assert_eq!( + r.header("x-satd-timestamp"), + first.header("x-satd-timestamp"), + "attempt {} re-stamped the timestamp; the event must be signed once", + i + 1 + ); + assert_eq!(r.header("x-satd-signature"), first.header("x-satd-signature")); + assert_eq!(r.header("x-satd-delivery"), first.header("x-satd-delivery")); + assert_eq!(r.body, first.body); + } + // ...and the attempt counter is the one thing that does move. + assert_ne!(chain[0].header("x-satd-attempt"), chain[2].header("x-satd-attempt")); +} + +/// A SIGHUP that did not touch the alertfile must not destroy a queued status +/// event. +/// +/// `reload_from_sighup` calls `AlertReloader::apply()` on *every* SIGHUP, +/// whatever key the operator actually edited, and retiring a generation drops +/// everything queued in it. For chain events that is survivable — the cursor +/// did not advance, so the next generation's catch-up re-queues them. A status +/// event has no replay by design, and the detectors are edge-triggered against +/// a `HealthState` that outlives the reload, so a `disk_low` sitting in retry +/// backoff when the operator SIGHUPs to change `maxconnections` would be +/// dropped, never replayed, and never re-raised. The page simply never arrives. +/// +/// The receiver 503s the first two attempts, so the event is provably still +/// in the dispatcher — mid-backoff — when the unrelated SIGHUP lands. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn an_unrelated_sighup_does_not_destroy_a_queued_status_event() { + // Refuse the first four attempts. Backoff is 1s, 2s, 4s, 8s, so the event + // stays in the dispatcher for ~15s — a wide window for the unrelated reload + // below to land inside. Too few refusals and the delivery could succeed + // before the SIGHUP, leaving the test green without ever exercising the + // reload path. + let receiver = MockReceiver::failing_first(503, 4).await; + let (sn, _dir) = node_with_hook(&receiver, "pager", "\"status\"", vec![]).await; + + // Known-cleared first, so the raise is an edge on any host (see + // `webhook_delivers_a_signed_status_event` for why this goes through the + // conf file rather than the command line). + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + + // Raise `disk_low`. The detectors re-evaluate on a 15s poll, so the raise + // does not follow the SIGHUP immediately — wait for the delivery to have + // been attempted rather than assuming a fixed delay. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + let deadline = std::time::Instant::now() + Duration::from_secs(45); + while receiver.attempts().await == 0 { + assert!( + std::time::Instant::now() < deadline, + "the status event was never attempted; the raise did not reach the hook" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + } + + // Now an unrelated reload, while that delivery is still being retried. The + // alertfile is untouched; only a daemon key changes. The threshold is + // carried over so the condition stays raised — a cleared-and-re-raised + // alert would produce a *new* event and mask the loss of the old one. + crate::streaming::sighup_with_conf( + &sn, + "alertdiskfreemb=17592186044416\nmaxconnections=42\n", + ) + .await; + + // The original event must still land. + let got = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + assert_eq!(got.json()["body"]["kind"], "disk_low"); + assert!(got.signature_valid(SECRET)); } /// A permanently-rejected event still advances the hook's resume cursor. @@ -453,8 +567,14 @@ async fn webhook_permanent_rejection_advances_the_cursor() { .wait_for(60, |r| r.json()["body"]["height"] == 1) .await; tokio::time::sleep(Duration::from_secs(2)).await; - let before = receiver.attempts().await; - assert!(before >= 3, "expected all three blocks attempted, got {before}"); + let block_attempts = |rs: Vec| -> Vec { + rs.iter() + .filter(|r| r.json()["body"]["kind"] == "block_connected") + .filter_map(|r| r.json()["body"]["height"].as_u64()) + .collect() + }; + let before = block_attempts(receiver.all_attempts().await); + assert_eq!(before, vec![1, 2, 3], "expected all three blocks attempted once"); // A SIGHUP retires the generation and starts a new one, which re-runs // catch-up from the persisted cursor. @@ -471,8 +591,14 @@ async fn webhook_permanent_rejection_advances_the_cursor() { // Counting *attempts*, not accepted deliveries: a replayed block would be // refused again and so would never show up as accepted. What matters is // whether the request was made at all. + // + // Scoped to `block_connected` rather than a raw request count, because a + // permanent rejection now also emits a `lagged` notice — the receiver is + // told which events it lost, since the cursor advancing past them means it + // can never fetch them later. A raw count would fold that in and read as a + // replayed block. assert_eq!( - receiver.attempts().await, + block_attempts(receiver.all_attempts().await), before, "a refused span must not be rebuilt and re-sent on reload", ); @@ -484,6 +610,22 @@ async fn webhook_permanent_rejection_advances_the_cursor() { receiver .wait_for(60, |r| r.json()["body"]["height"] == 4) .await; + + // ...and the refusal was announced rather than silent. A gap notice is + // emitted ahead of the next event for the hook, so this is asserted after + // block 4 rather than immediately after the rejections. + // + // It matters here more than on any other drop path: the cursor advances + // past a permanently-rejected event, so unlike a queue overflow the + // receiver cannot go back for it. Being told is all it gets. + let lagged = receiver + .wait_for(30, |r| r.json()["body"]["category"] == "lagged") + .await; + assert!( + lagged.json()["body"]["dropped_count"].as_u64().unwrap_or(0) >= 2, + "both refused blocks should be counted; got {}", + lagged.body, + ); } /// A redirect is never followed: the signed body does not go to a host the From d6d7ecac985f3fdd80edcb4ed7d01a977dc388df Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Sat, 25 Jul 2026 17:26:11 -0600 Subject: [PATCH 04/11] alert: sum lagged notices in the permanent-rejection e2e MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The assertion read `dropped_count` off the *first* `lagged` notice and required it to be 2. That held only because the two refusals happened to land inside a single gap-flush window and coalesce. Gap accounting also flushes on a timer, so a slower machine ticks between the two rejections and emits two notices of 1 rather than one of 2. The CI runner does exactly that: `wait_for` returned the first notice and the assertion failed on `dropped_count: 1`. Nothing was wrong with the dispatcher — announcing the loss promptly rather than batching it is the better behaviour, and the test was pinning down a race it has no stake in. Sum across every `lagged` notice within a deadline instead. The property under test is that a permanently-refused span is announced at all, since the cursor advances past it and the receiver can never go back for it; how the count is split across notices is not part of that. Negative control: disabling the gap attribution at the permanent-rejection site fails the test with "got 0 across 0 notice(s)". Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH --- satd/tests/e2e/webhooks.rs | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index 2ca7695df..fd26445a2 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -618,14 +618,33 @@ async fn webhook_permanent_rejection_advances_the_cursor() { // It matters here more than on any other drop path: the cursor advances // past a permanently-rejected event, so unlike a queue overflow the // receiver cannot go back for it. Being told is all it gets. - let lagged = receiver - .wait_for(30, |r| r.json()["body"]["category"] == "lagged") - .await; - assert!( - lagged.json()["body"]["dropped_count"].as_u64().unwrap_or(0) >= 2, - "both refused blocks should be counted; got {}", - lagged.body, - ); + // Summed across notices rather than read off the first one. The gap + // accounting also flushes on a timer, so whether the two refusals coalesce + // into a single notice of 2 or arrive as two notices of 1 depends only on + // which side of a tick they land on — a race this test has no reason to + // pin down. Only the total is meaningful. + let deadline = std::time::Instant::now() + Duration::from_secs(30); + loop { + let attempts = receiver.all_attempts().await; + let lagged: Vec<_> = attempts + .iter() + .filter(|r| r.json()["body"]["category"] == "lagged") + .collect(); + let total: u64 = lagged + .iter() + .filter_map(|r| r.json()["body"]["dropped_count"].as_u64()) + .sum(); + if total >= 2 { + break; + } + assert!( + std::time::Instant::now() < deadline, + "both refused blocks should be counted; got {total} across {} notice(s): {:?}", + lagged.len(), + lagged.iter().map(|r| r.body.clone()).collect::>(), + ); + tokio::time::sleep(Duration::from_millis(200)).await; + } } /// A redirect is never followed: the signed body does not go to a host the From 122e93a691fe0619a4730890f90800f2671f19da Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Sat, 25 Jul 2026 20:08:50 -0600 Subject: [PATCH 05/11] alerting: webhooks are realtime + a gap notice, not a replay log MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the D6 revision. The dispatcher had a cursor-driven startup catch-up that rebuilt and re-delivered the missed span. It is gone. The per-hook cursor stays, as a resume *marker*: read once at startup, compared against the tip, reported as one `Lagged`, then advanced past the span. Deleted with it: the `b` delivery-id namespace, the replay-to-live boundary dedup, and the catch-up clamp. `block_delivery_id` was keyed on height alone, so a block and its post-reorg replacement minted the same `X-Satd-Delivery` and a conforming receiver discarded the replacement — the one event class the feature exists to report. Its two tests could not have caught that: one asserted a pure function is pure, the other varied only the counter and never block identity. Gap counts are now conserved. `flush_gap` retired the count the moment a notice was *queued*, so a reload that retires the generation destroyed the queue and the count with it — leaving the process-lived `GapState`, which exists precisely to survive a reload, reading zero. The count now travels on the `Delivery` and is only retired on a 2xx; teardown drains the queue and hands weights back, including one already in hand mid-backoff. A permanent rejection re-adds what the delivery carried instead of adding 1, so rejecting a `Lagged` carrying 500 no longer shrinks it to 1. The IBD gate is scoped to chain events, which is what actually closes the restart hole. `is_initial_block_download()` is tip age, not a sync flag, so a node restarted while already wedged reads as syncing from its first event and never latches — and the old gate then suppressed *everything*: status, mempool, and watch matches, which have no replay to recover them. A stalled node is not producing chain events, so gating only those costs nothing in that state and still stops the firehose it was written for. Also: the gap map is pruned on reload alongside `metrics.retain` and the cursor GC, so a re-added hook id no longer inherits its predecessor's drop count; cursors are keyed on id + URL hash so repointing starts clean; and `InMemoryStore` implements the four alert-cursor methods, without which the trait defaults (`Ok(vec![])`, `Err`) make every cursor and GC test pass vacuously. E2E `catch_up_replay_gives_every_event_a_distinct_delivery_id` is replaced by `a_missed_span_is_announced_as_a_gap_and_never_replayed`, negative control verified. Workspace clippy clean; node, satd and satd-alert suites green. --- node/src/storage/db.rs | 33 +++ satd-alert/src/config.rs | 51 ++++- satd-alert/src/contract.rs | 65 ++---- satd-alert/src/lib.rs | 2 +- satd/src/alert.rs | 426 +++++++++++++++++-------------------- satd/tests/e2e/webhooks.rs | 120 +++++++---- 6 files changed, 381 insertions(+), 316 deletions(-) diff --git a/node/src/storage/db.rs b/node/src/storage/db.rs index ddb2e3d34..9b24f4f8c 100644 --- a/node/src/storage/db.rs +++ b/node/src/storage/db.rs @@ -25,6 +25,11 @@ pub struct InMemoryStore { tx_index: parking_lot::RwLock>, chain_tx: parking_lot::RwLock>, chain_tx_backfill_complete: parking_lot::RwLock, + /// Alert-webhook resume markers. Present so that cursor and GC tests run + /// against real behaviour: the `Store` defaults return `Ok(vec![])` from + /// `list_alert_cursor_keys` and `Err` from `write_alert_cursor`, which + /// would make every such test pass without exercising anything. + alert_cursors: parking_lot::RwLock, Vec>>, addr_funding: parking_lot::RwLock>, addr_spending: parking_lot::RwLock>, outpoint_spend: parking_lot::RwLock>, @@ -68,6 +73,7 @@ impl InMemoryStore { // chains populate chain_tx directly, so the backfill is a no-op // there regardless. chain_tx_backfill_complete: parking_lot::RwLock::new(false), + alert_cursors: parking_lot::RwLock::new(std::collections::BTreeMap::new()), addr_funding: parking_lot::RwLock::new(Vec::new()), addr_spending: parking_lot::RwLock::new(Vec::new()), outpoint_spend: parking_lot::RwLock::new(std::collections::HashMap::new()), @@ -106,6 +112,33 @@ impl InMemoryStore { } impl Store for InMemoryStore { + fn read_alert_cursor(&self, key: &[u8]) -> Option> { + self.alert_cursors.read().get(key).cloned() + } + + fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { + self.alert_cursors + .write() + .insert(key.to_vec(), value.to_vec()); + Ok(()) + } + + fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { + self.alert_cursors.write().remove(key); + Ok(()) + } + + fn list_alert_cursor_keys(&self) -> Result>, StoreError> { + const PREFIX: &[u8] = b"alertwebhook.cursor."; + Ok(self + .alert_cursors + .read() + .keys() + .filter(|k| k.starts_with(PREFIX)) + .cloned() + .collect()) + } + fn flush_durable(&self) -> Result<(), StoreError> { self.flush_durable_calls .fetch_add(1, std::sync::atomic::Ordering::Relaxed); diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs index 14b470db2..9ab3f7abd 100644 --- a/satd-alert/src/config.rs +++ b/satd-alert/src/config.rs @@ -155,9 +155,18 @@ impl std::fmt::Debug for Hook { } impl Hook { - /// The `CF_METADATA` key holding this hook's durable resume cursor. + /// The `CF_METADATA` key holding this hook's durable resume marker. + /// + /// Keyed on the id *and* a hash of the endpoint URL. On id alone, + /// repointing a hook at a new endpoint — which an operator does precisely + /// when the old one is dead and its marker is far behind — would hand the + /// new endpoint its predecessor's position and announce a gap it never had. + /// Including the URL means a repoint starts clean; the stale key is then + /// garbage-collected on the next reload like any other orphan. pub fn cursor_key(&self) -> Vec { - format!("alertwebhook.cursor.{}", self.id).into_bytes() + use sha2::{Digest as _, Sha256}; + let url_hash = hex::encode(Sha256::digest(self.url.as_bytes())); + format!("alertwebhook.cursor.{}-{}", self.id, &url_hash[..16]).into_bytes() } } @@ -579,7 +588,43 @@ categories = ["status"] assert!(h.filter.kinds.is_none()); assert!(h.filter.min_severity.is_none()); assert_eq!(h.heartbeat_interval_secs, None); - assert_eq!(h.cursor_key(), b"alertwebhook.cursor.ops".to_vec()); + let key = String::from_utf8(h.cursor_key()).unwrap(); + assert!(key.starts_with("alertwebhook.cursor.ops-"), "got {key}"); + } + + /// Repointing a hook at a new endpoint must not hand it the old one's + /// resume marker. + /// + /// An operator changes a URL precisely when the old endpoint is dead and + /// its marker has fallen far behind. Keyed on the id alone, the new + /// endpoint would inherit that position and be told it had missed a span it + /// was never subscribed to. + #[test] + fn repointing_a_hook_changes_its_cursor_key() { + let a = AlertFile::parse( + Path::new("/tmp/a.toml"), + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = \"https://a.example/hook\"\n\ + secret = \"0123456789abcdef0123456789abcdef\"\n\ + categories = [\"status\"]\n", + ) + .unwrap(); + let b = AlertFile::parse( + Path::new("/tmp/b.toml"), + "version = 1\n\ + [[webhook]]\n\ + id = \"ops\"\n\ + url = \"https://b.example/hook\"\n\ + secret = \"0123456789abcdef0123456789abcdef\"\n\ + categories = [\"status\"]\n", + ) + .unwrap(); + assert_ne!(a.hooks[0].cursor_key(), b.hooks[0].cursor_key()); + // Same URL, same key: a reload that changes nothing must not orphan the + // marker and restart the hook from the live head. + assert_eq!(a.hooks[0].cursor_key(), a.hooks[0].cursor_key()); } #[test] diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs index c1495ad84..70e3e01b4 100644 --- a/satd-alert/src/contract.rs +++ b/satd-alert/src/contract.rs @@ -94,25 +94,6 @@ pub fn replay_delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> Stri format!("{node_id_hex}-{instance_id}-r{seq}") } -/// Delivery id for a replayed **confirmed block**, derived from its height. -/// -/// Deterministic on purpose, and the only id in the scheme that is. Every other -/// form embeds this process's random `instance_id` and a running counter, which -/// is right for an event that happens once — but a replayed block is precisely -/// the event this design can deliver twice: a restart replays from the durable -/// cursor, and a reload can leave two generations briefly overlapping. Under a -/// counter-minted id those duplicates arrive with *different* headers, which -/// makes "deduplicate on `X-Satd-Delivery`" unimplementable for the only -/// duplicate that actually occurs. Keyed on height, both copies carry the same -/// id and a conforming receiver collapses them, which is the contract. -/// -/// The instance component is fixed at `0` — a real `instance_id` is a random -/// `u64` and would reintroduce the per-process variation this exists to remove. -/// The `b` prefix keeps the space disjoint from the bus (``), watch -/// (`w`), and synthesized (`r`) spaces. -pub fn block_delivery_id(node_id_hex: &str, height: u32) -> String { - format!("{node_id_hex}-0-b{height}") -} /// Maximum age a receiver should accept for a v2 delivery, in seconds. /// @@ -278,38 +259,36 @@ mod tests { } /// The id spaces must not overlap. They are minted from independent - /// counters — the bus `seq`, a synth counter, and a block height — so - /// without the prefixes a replayed block at height 500 and a bus event with - /// `seq` 500 would collide, and a conforming receiver would silently - /// discard the second. (The watch space, `w`, is added with the - /// watch-hook feature and is covered alongside it.) + /// counters — the bus `seq` and a synth counter — so without the prefix a + /// synthesized notice at counter 500 and a bus event at `seq` 500 would + /// collide, and a conforming receiver would silently discard the second. + /// (The watch space, `w<...>`, is added with the watch-hook feature and is + /// covered alongside it.) + /// + /// Swept across the counter range rather than sampled at one value: a + /// collision that only appears at a boundary is exactly the kind a single + /// sample misses. #[test] fn the_delivery_id_spaces_are_disjoint() { let node = "ab".repeat(16); - let n = 500u64; - let ids = [ - delivery_id(&node, 7, n), - replay_delivery_id(&node, 7, n), - block_delivery_id(&node, n as u32), - ]; - for (i, a) in ids.iter().enumerate() { - for b in ids.iter().skip(i + 1) { - assert_ne!(a, b, "delivery id spaces collide at the same counter value"); + for n in [0u64, 1, 9, 10, 500, u32::MAX as u64, u64::MAX] { + let ids = [delivery_id(&node, 7, n), replay_delivery_id(&node, 7, n)]; + for (i, a) in ids.iter().enumerate() { + for b in ids.iter().skip(i + 1) { + assert_ne!(a, b, "delivery id spaces collide at counter {n}"); + } } } } - /// A replayed block's id is derived from its height alone, so the same - /// block delivered twice — a restart replaying from the durable cursor, or - /// a reload whose generations briefly overlap — carries the same - /// idempotency key. Every other id embeds a random per-process - /// `instance_id`, which would make those duplicates undedupable. + /// Ids are unique per event within a process, across every field that + /// varies: node, instance, and counter. #[test] - fn a_replayed_block_id_is_stable_across_restarts() { + fn a_delivery_id_varies_with_every_component() { let node = "ab".repeat(16); - assert_eq!(block_delivery_id(&node, 840_000), block_delivery_id(&node, 840_000)); - assert_ne!(block_delivery_id(&node, 840_000), block_delivery_id(&node, 840_001)); - // No instance component to vary: this is the whole point. - assert!(block_delivery_id(&node, 840_000).ends_with("-0-b840000")); + let other = "cd".repeat(16); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&other, 7, 1)); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&node, 8, 1)); + assert_ne!(delivery_id(&node, 7, 1), delivery_id(&node, 7, 2)); } } diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs index a2eeea1e7..18cfc05c5 100644 --- a/satd-alert/src/lib.rs +++ b/satd-alert/src/lib.rs @@ -27,7 +27,7 @@ pub mod retry; pub use config::{AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY}; pub use contract::{ - block_delivery_id, delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, + delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, ATTEMPT_HEADER, DELIVERY_HEADER, HOOK_HEADER, LEGACY_WEBHOOK_VERSION, MAX_TIMESTAMP_SKEW_SECS, SIGNATURE_HEADER, TIMESTAMP_HEADER, WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, diff --git a/satd/src/alert.rs b/satd/src/alert.rs index 98b21a0d3..9787ffd6e 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -46,11 +46,6 @@ pub const LEGACY_REORG_HOOK_ID: &str = "reorg-legacy"; /// Per-attempt HTTP timeout. Matches the shipped reorg webhook. const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); -/// How far back a hook's startup catch-up may replay. Shares the streaming -/// API's clamp so a webhook receiver and a streaming client see the same -/// "resync below this height yourself" boundary. -const MAX_CATCHUP_BLOCKS: u32 = node::events::MAX_REPLAY_BLOCKS; - /// How often a pending gap notice is flushed when no event is driving the hook. const GAP_FLUSH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15); @@ -103,6 +98,14 @@ enum Delivery { delivery_id: String, /// Cursor to persist once the receiver acks, if this event carries one. cursor: Option, + /// How many lost events this delivery announces, if it is a `Lagged` + /// notice. The count travels *with* the delivery rather than being + /// retired when the notice is queued: a queued notice can still be + /// destroyed — by a reload retiring the generation, or by a permanent + /// rejection — and the one message that must not be lost is the one + /// saying data was lost. Anything that fails to deliver hands the + /// weight back to `GapState`. + gap_weight: u64, }, } @@ -130,32 +133,54 @@ struct GapState { resume_height: std::sync::atomic::AtomicU32, } -/// Per-hook gap state, keyed by hook id, living as long as the process. +/// Process-lived dispatcher state that must outlive any single generation. +/// +/// A reload retires the generation that accumulated a drop count, and +/// per-generation state would take the pending `Lagged` notice down with it — +/// the receiver would never learn about a hole satd had already recorded. +/// `HookCounters` is process-lived for the same reason. /// -/// Deliberately not per-generation. A reload retires the generation that -/// accumulated a drop count, and per-generation state would take the pending -/// `Lagged` notice down with it — the receiver would never learn about a hole -/// that satd had already recorded. `HookCounters` is process-lived for the same -/// reason; this is the half that was missed. -static GAP_STATE: std::sync::LazyLock< - parking_lot::Mutex>>, -> = std::sync::LazyLock::new(|| parking_lot::Mutex::new(std::collections::HashMap::new())); +/// Injected rather than reached for as a `static` so a test can hand in a fresh +/// one. With a bare static the first test to latch `left_ibd` makes every later +/// test of the gate vacuous, non-deterministically, since they share a process. +#[derive(Default)] +struct DispatcherState { + /// Whether this node has ever been observed out of initial block download. + /// + /// `is_initial_block_download()` is the tip header's age, not a sync flag, + /// so a node whose tip stops advancing crosses back into "IBD" a day later. + /// This keeps a flapping predicate from re-suppressing a caught-up node. + /// + /// It is deliberately not load-bearing. The latch lives in this process, so + /// a node restarted while already wedged never observes a non-IBD tip and + /// never arms it — and an earlier version of this gate went permanently + /// silent in exactly that case. What makes that harmless now is the *scope* + /// of the gate: only chain events are suppressed, and chain events are by + /// definition not arriving on a node whose tip has stopped. + left_ibd: std::sync::atomic::AtomicBool, + gaps: parking_lot::Mutex>>, +} + +impl DispatcherState { + fn gap_for(&self, id: &str) -> Arc { + Arc::clone(self.gaps.lock().entry(id.to_string()).or_default()) + } -fn gap_state_for(id: &str) -> Arc { - let mut map = GAP_STATE.lock(); - Arc::clone(map.entry(id.to_string()).or_default()) + /// Drop gap state for hooks no longer named in the alertfile. + /// + /// The mirror of `metrics.retain` and the cursor GC, which already run on + /// reload. Hook ids are short and reused — `pager`, `ops`, `alerts` — so a + /// re-added id would otherwise inherit its predecessor's pending drop count + /// and resume anchor, and announce a hole to a brand-new endpoint that + /// never had one. + fn retain_gaps(&self, keep: &std::collections::HashSet) { + self.gaps.lock().retain(|id, _| keep.contains(id)); + } } -/// Whether this node has ever been observed out of initial block download. -/// -/// `is_initial_block_download()` is a function of the tip header's age, not of -/// whether a sync is in progress: a node whose tip stops advancing crosses back -/// into "IBD" 24 hours later. Without this latch the alert path would go dark -/// during exactly the incident it exists to report — a stalled tip — and, worse, -/// resume by advancing the durable cursor across everything it suppressed. -/// Once a node has been caught up, it is never "syncing" again for the -/// dispatcher's purposes. -static LEFT_IBD: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); +/// The process-wide instance. Tests construct their own. +static DISPATCHER_STATE: std::sync::LazyLock> = + std::sync::LazyLock::new(|| Arc::new(DispatcherState::default())); /// Spawn the dispatcher for a parsed alertfile. /// @@ -169,6 +194,7 @@ fn spawn_with_metrics( store: Arc, block_source: Option>, metrics: Arc, + state: Arc, stop: Stop, ) { if file.hooks.is_empty() { @@ -178,7 +204,7 @@ fn spawn_with_metrics( for hook in &file.hooks { let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); let counters = metrics.hook(&hook.id); - let gap = gap_state_for(&hook.id); + let gap = state.gap_for(&hook.id); tokio::spawn(deliver_loop( hook.clone(), rx, @@ -196,7 +222,7 @@ fn spawn_with_metrics( reported_closed: std::sync::atomic::AtomicBool::new(false), }); } - tokio::spawn(fan_in(hooks, publisher, store, block_source, stop)); + tokio::spawn(fan_in(hooks, publisher, store, block_source, state, stop)); } /// Fan-in: one broadcast receiver, filtered and enqueued per hook. @@ -205,24 +231,16 @@ async fn fan_in( publisher: Arc, store: Arc, block_source: Option>, + state: Arc, mut stop: Stop, ) { - // Subscribe BEFORE the catch-up replay, so an event published while the - // replay is being built is buffered rather than lost in the seam. let mut rx = publisher.subscribe(); - // Per-hook snapshot→live boundary dedup, exactly as the gRPC and WS - // carriers keep. A block connecting between `subscribe()` and the tip - // snapshot taken inside `catch_up` is *both* buffered on the broadcast and - // inside the replayed span, so without this it is delivered twice — and - // because the replayed copy is re-synthesized with its own delivery id, the - // two carry different idempotency keys and a conforming receiver cannot - // collapse them. A duplicate deposit alert is exactly the failure the - // idempotency contract exists to prevent. - let mut dedup: Vec> = - Vec::with_capacity(hooks.len()); + // Announce whatever each hook missed while the daemon was down. Nothing is + // replayed — there is no snapshot-to-live seam to dedupe, because there is + // no snapshot. for hook in &hooks { - dedup.push(catch_up(hook, &publisher, store.as_ref(), block_source.as_deref()).await); + announce_gap(hook, store.as_ref(), block_source.as_deref()).await; } // Per-hook heartbeat downsampling state (D11): last forwarded instant. @@ -252,42 +270,47 @@ async fn fan_in( } recv = rx.recv() => match recv { Ok(env) => { - // Suppress the firehose during initial block download. + // Suppress the block firehose during initial block + // download. // - // The dispatcher is started long before P2P, so without - // this a brand-new node with an alertfile POSTs its entire - // sync — one `block_connected` per historical block, plus a - // watch-match per historical transaction touching a watched - // script, for as long as IBD takes. That is the opposite of - // what the manual promises, and the failure mode is a - // multi-day firehose at the receiver rather than anything - // that looks like a bug locally. + // The dispatcher starts long before P2P, so without this a + // brand-new node with an alertfile POSTs its entire sync — + // one `block_connected` per historical block, for as long as + // IBD takes. The failure mode is a multi-day firehose at the + // receiver rather than anything that looks like a bug + // locally. // - // Latched (see `LEFT_IBD`), because the predicate is the - // tip's age rather than a sync flag. Once latched the - // `block_source` call is skipped entirely, which also keeps - // a per-event RocksDB block-index lookup off this task. + // Scoped to chain events, and that scope is what makes the + // gate safe. `is_initial_block_download()` is the tip + // header's age, not a sync flag, so a node that is fully + // caught up and then *stops* reads as "syncing" a day later + // — and a node restarted while already wedged reads that way + // from its first event, with no chance to latch. An earlier + // version gated everything on that predicate and so went + // totally dark on a stalled node: no status, no mempool, and + // watch matches destroyed outright since they have no replay + // to recover them. The one thing a stalled node is *not* + // producing is chain events, so suppressing only those costs + // nothing in that state and still stops the firehose in the + // state it was written for. // - // Status and heartbeat events are exempt. "This node is - // unhealthy" is exactly as true during IBD, and the - // detectors already suppress the conditions that are - // meaningless while syncing. A heartbeat is a dead-man's - // switch: suppressing it during a multi-day sync makes an - // external watchdog declare a healthy node dead, which - // inverts the signal it exists to carry. - let syncing = !LEFT_IBD.load(Ordering::Relaxed) && { - let in_ibd = block_source - .as_deref() - .is_some_and(|s| s.in_initial_block_download()); - if !in_ibd { - LEFT_IBD.store(true, Ordering::Relaxed); - } - in_ibd - }; - let exempt = matches!( - env.body, - NodeEventBody::Status(_) | NodeEventBody::Heartbeat { .. } - ); + // Status and heartbeat therefore always pass, which they + // must: "this node is unhealthy" is exactly as true during + // IBD, and a heartbeat is a dead-man's switch — suppressing + // it during a multi-day sync makes an external watchdog + // declare a healthy node dead. + let suppressible = matches!(env.body, NodeEventBody::Chain(_)); + let syncing = suppressible + && !state.left_ibd.load(Ordering::Relaxed) + && { + let in_ibd = block_source + .as_deref() + .is_some_and(|s| s.in_initial_block_download()); + if !in_ibd { + state.left_ibd.store(true, Ordering::Relaxed); + } + in_ibd + }; // Decide who wants this *before* rendering it. Serializing // first meant a `BlockTweaks` envelope — hundreds of KB of // per-block silent-payment rows — was rendered in full and @@ -302,7 +325,7 @@ async fn fan_in( if wanted.is_empty() { continue; } - if syncing && !exempt { + if syncing { // Record the suppression as a gap rather than dropping // it silently. A silent skip would leave no counter // moved and no `lagged` body emitted, and the first @@ -333,23 +356,11 @@ async fn fan_in( ); for i in wanted { let hook = &hooks[i]; - // A live block identical to one already replayed for - // this hook. A reorg replacement at the same height has - // a different hash and is forwarded, which is why the - // check is on the hash and not the height alone. - if !dedup[i].is_empty() - && let NodeEventBody::Chain( - node::chain::events::ChainEvent::BlockConnected { height, hash }, - ) = &env.body - && dedup[i].get(height) == Some(hash) - { - dedup[i].remove(height); - continue; - } enqueue(hook, &publisher, Delivery::Event { body: body.clone(), delivery_id: delivery_id.clone(), cursor: env.cursor, + gap_weight: 0, }, env.cursor); } } @@ -453,6 +464,7 @@ fn flush_gap(hook: &HookChannel, publisher: &EventPublisher) { SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), ), cursor: None, + gap_weight: dropped, }; if hook.tx.try_send(item).is_err() { // Still full — put the count back so the notice is not lost. @@ -510,150 +522,67 @@ fn accepts(hook: &Hook, env: &NodeEvent, last_heartbeat: &mut Option, -) -> std::collections::HashMap { - if !hook - .hook - .filter - .categories - .contains(node::events::CATEGORY_CHAIN) - { - return Default::default(); - } +) { let Some(src) = block_source else { - return Default::default(); + return; }; + let tip = src.current_tip_height(); let Some(cursor) = read_cursor(store, &hook.hook) else { - // A hook that has never delivered starts at the live head rather than - // replaying history nobody asked for. - return Default::default(); + // No stored marker: this hook is forward-only from now. Anchor the + // resume height at the live tip anyway. It is otherwise only set by a + // successful enqueue, so a hook that lags before its first one — or one + // that never carries a cursor at all, like a status-only or watch-only + // hook — would emit a `Lagged` advertising height 0. A receiver reading + // that as "resume from genesis" would resync the whole chain over + // having missed nothing. + hook.gap.resume_height.store(tip, Ordering::Relaxed); + return; }; - // Seed the gap anchor from the durable cursor before any replay is queued. - // `resume_height` is otherwise only set by a successful enqueue, so a hook - // that lags before its first one — a fresh generation whose catch-up burst - // overflows immediately — would emit a `Lagged` advertising height 0, which - // reads as "resume from genesis" rather than "resume from where you were". - hook.gap - .resume_height - .store(cursor.height, Ordering::Relaxed); - - let replay = node::events::build_cursor_replay( - src, - publisher, - cursor, - node::events::CATEGORY_CHAIN, - MAX_CATCHUP_BLOCKS, - None, + hook.gap.resume_height.store(cursor.height, Ordering::Relaxed); + let missed = u64::from(tip.saturating_sub(cursor.height)); + if missed == 0 { + return; + } + hook.gap.dropped.fetch_add(missed, Ordering::Relaxed); + tracing::info!( + target: "alert", + hook = %hook.id, + from = cursor.height, + tip, + missed, + "webhook missed a span while the daemon was down; announcing it as a gap", ); - if replay.clamped { - // Replay begins at `cursor.height + 1`, so the number of blocks skipped - // is the distance to the first one actually replayed, not to the cursor - // itself — the latter counts the already-delivered block at the cursor. - let dropped = - u64::from(replay.earliest_replayed.saturating_sub(cursor.height.saturating_add(1))); - hook.gap.dropped.fetch_add(dropped, Ordering::Relaxed); - hook.counters.dropped.fetch_add(dropped, Ordering::Relaxed); - hook.gap.resume_height.store(cursor.height, Ordering::Relaxed); + // Advance the marker past the announced span. Without this a restart before + // the next delivery re-announces the same gap, and on a quiet chain that + // repeats on every restart. + let advanced = Cursor { height: tip, ..cursor }; + if let Err(e) = store.write_alert_cursor(&hook.hook.cursor_key(), &encode_cursor(&advanced)) { tracing::warn!( target: "alert", hook = %hook.id, - from = cursor.height, - earliest = replay.earliest_replayed, - "webhook catch-up clamped; receiver must resync the older span itself", + error = %e, + "failed to advance the webhook resume marker past an announced gap", ); } - let count = replay.events.len(); - let mut queued = 0u64; - let mut overflowed = 0u64; - let mut queued_heights: std::collections::HashSet = std::collections::HashSet::new(); - for env in replay.events { - let height = match &env.body { - NodeEventBody::Chain(node::chain::events::ChainEvent::BlockConnected { - height, - .. - }) => Some(*height), - _ => None, - }; - let Ok(bytes) = serde_json::to_vec(&env) else { - continue; - }; - let cursor = env.cursor; - // Every replayed envelope is stamped `seq: 0`, so minting from it would - // give a node that was down for 100 blocks 100 deliveries sharing one - // idempotency key — the whole point of catch-up defeated silently. - // - // A replayed block takes an id derived from its height rather than a - // process counter. This is the one event the design really can deliver - // twice (a restart, or a reload whose generations overlap), and every - // counter-minted id embeds a per-process random `instance_id`, so the - // duplicate would arrive under a fresh id and "deduplicate on - // X-Satd-Delivery" would be unimplementable for the only case that - // needs it. Non-block replays keep the counter — they carry no durable - // position to key on. - let delivery_id = match height { - Some(h) => satd_alert::block_delivery_id(&hex::encode(env.stamp.node_id), h), - None => satd_alert::replay_delivery_id( - &hex::encode(env.stamp.node_id), - publisher.instance_id(), - SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), - ), - }; - if enqueue( - hook, - publisher, - Delivery::Event { - body: Arc::new(bytes), - delivery_id, - cursor, - }, - cursor, - ) { - queued += 1; - if let Some(h) = height { - queued_heights.insert(h); - } - } else { - overflowed += 1; - } - } - if count > 0 { - // Report what was *queued*, not what was built. The 10k-block replay - // clamp warns loudly, but the hook queue holds 1024 — and for any - // realistic outage the queue is the binding limit, since 1024 blocks is - // about a week and 10k is about ten. Overflow inside this loop is - // counted but was otherwise unlogged, so the success line reported the - // pre-truncation figure and read as if everything had been recovered. - if overflowed > 0 { - tracing::warn!( - target: "alert", - hook = %hook.id, - built = count, - dropped = overflowed, - "webhook catch-up exceeded the hook queue; the excess is lost \ - and reported to the receiver as a gap", - ); - } - tracing::info!( - target: "alert", - hook = %hook.id, - events = queued, - from = cursor.height, - "webhook catch-up queued events missed while the daemon was down", - ); - } - // Only suppress the live copy of a block this hook actually received. The - // replay builds up to 10,000 events into a queue that holds 1024, so on a - // long outage most of the tail overflows — and the tail is exactly the span - // still buffered on the broadcast, i.e. the blocks whose live copies are - // about to arrive. Keeping a dropped block in the dedup map suppresses that - // live copy too, dropping the same block twice. - let mut dedup = replay.confirmed_dedup; - dedup.retain(|height, _| queued_heights.contains(height)); - dedup } fn read_cursor(store: &dyn Store, hook: &Hook) -> Option { @@ -744,6 +673,22 @@ fn persist_cursor( } } +/// Hand back the gap weight of every delivery this loop will never send. +/// +/// Retiring a generation drops its queue, and `flush_gap` has already zeroed +/// `GapState` on the assumption the queued notice would go out. Without this, +/// a reload silently destroys the record of a hole — and the process-lived +/// `GapState` that exists precisely to survive a reload reads zero afterwards. +fn drain_owed(rx: &mut mpsc::Receiver, gap: &GapState) { + let mut owed = 0u64; + while let Ok(Delivery::Event { gap_weight, .. }) = rx.try_recv() { + owed = owed.saturating_add(gap_weight); + } + if owed > 0 { + gap.dropped.fetch_add(owed, Ordering::Relaxed); + } +} + /// One hook's delivery loop: serial, in-order, retrying with backoff. async fn deliver_loop( hook: Hook, @@ -767,16 +712,17 @@ async fn deliver_loop( loop { let item = tokio::select! { - _ = stop.wait() => return, + _ = stop.wait() => return drain_owed(&mut rx, &gap), item = rx.recv() => match item { Some(i) => i, - None => return, + None => return drain_owed(&mut rx, &gap), }, }; let Delivery::Event { body, delivery_id, cursor, + gap_weight, } = item; counters @@ -843,7 +789,14 @@ async fn deliver_loop( // body the receiver has no way to learn which events it // lost, and — because the cursor advances below — no way to // recover them after the misconfiguration is fixed. - gap.dropped.fetch_add(1, Ordering::Relaxed); + // + // `gap_weight` is what this delivery was itself announcing: + // rejecting a `Lagged` that carried 500 must not shrink the + // count to 1. Under a receiver that rejects every request — + // a proxy 302ing the lot — a collapsing count would grind + // gap accounting down to 1 on every cycle indefinitely. + gap.dropped + .fetch_add(gap_weight.saturating_add(1), Ordering::Relaxed); // Advance the cursor anyway. A permanent rejection is the // receiver's decision about this event, and it will decide // the same way next time — leaving the cursor parked would @@ -869,7 +822,15 @@ async fn deliver_loop( ); tokio::select! { _ = tokio::time::sleep(delay) => {} - _ = stop.wait() => return, + // Retired mid-backoff. This delivery is already out of + // the queue, so `drain_owed` cannot see it — hand its + // weight back explicitly before dropping it. + _ = stop.wait() => { + if gap_weight > 0 { + gap.dropped.fetch_add(gap_weight, Ordering::Relaxed); + } + return drain_owed(&mut rx, &gap); + } } } } @@ -986,7 +947,15 @@ impl AlertReloader { let hook_count = file.hooks.len(); { let _guard = self.api_handle.enter(); - spawn_with_metrics(&file, publisher, store, block_source, metrics, stop); + spawn_with_metrics( + &file, + publisher, + store, + block_source, + metrics, + Arc::clone(&DISPATCHER_STATE), + stop, + ); } if let Some(old) = self.current.lock().replace(gen_tx) { // Draining is deliberate rather than graceful: the retired @@ -1008,6 +977,13 @@ impl AlertReloader { keep.push(LEGACY_REORG_HOOK_ID.to_string()); self.metrics.retain(&keep); + // Reclaim gap state for the same set. `metrics.retain` and the cursor + // GC below both prune on hook removal; the process-lived gap map has to + // prune with them, or a re-added id inherits its predecessor's pending + // drop count and resume anchor and announces a hole its new endpoint + // never had. + DISPATCHER_STATE.retain_gaps(&keep.iter().cloned().collect()); + // Drop the durable cursor of any hook this reload removed. Hook ids are // short and reused — `pager`, `alerts`, `ops` — so leaving the key // behind means a later hook that happens to reuse the id inherits a diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index fd26445a2..c61de77d7 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -79,6 +79,12 @@ enum Behavior { FailFirst(u16, usize), /// Answer every request with `302 Location: `. Redirect(String), + /// Answer with `code` for any `block_connected` at one of these heights, + /// 200 for everything else. Unlike `RejectHeights` the status is the + /// caller's, so a 5xx leaves the delivery in retry and the hook's resume + /// marker parked — which is what an endpoint outage looks like from the + /// node's side. + FailHeights(Vec, u16), /// Permanently reject (404) any `block_connected` at one of these heights, /// accept everything else. Keyed on content rather than request ordinal so /// the script does not depend on how many events a block produces. @@ -128,6 +134,13 @@ impl MockReceiver { Behavior::FailFirst(code, n) if g.seen <= *n => (*code, None), Behavior::FailFirst(..) => (200, None), Behavior::Redirect(to) => (302, Some(to.clone())), + Behavior::FailHeights(hs, code) => { + let h = req.json()["body"]["height"].as_u64(); + match h { + Some(h) if hs.contains(&h) => (*code, None), + _ => (200, None), + } + } Behavior::RejectHeights(hs) => { let h = req.json()["body"]["height"].as_u64(); match h { @@ -173,6 +186,11 @@ impl MockReceiver { Self::start(Behavior::FailFirst(code, n)).await } + /// Answer `code` for `block_connected` at these heights, 200 otherwise. + pub async fn failing_heights(heights: Vec, code: u16) -> Self { + Self::start(Behavior::FailHeights(heights, code)).await + } + /// Answer every request with a 302 pointing at `to`. pub async fn redirecting_to(to: String) -> Self { Self::start(Behavior::Redirect(to)).await @@ -237,6 +255,11 @@ impl MockReceiver { } /// Every accepted delivery, in arrival order. + /// + /// Unused on this branch — the dispatcher tests assert on *attempts*, since + /// a refused delivery is still evidence of what was sent. The watch-hook + /// tests stacked above do use it, and each branch is linted on its own. + #[allow(dead_code)] async fn all(&self) -> Vec { self.inner.lock().await.received.clone() } @@ -695,67 +718,76 @@ async fn webhook_permanent_rejection_does_not_wedge_the_queue() { assert!(receiver.attempts().await >= 2); } -/// A restart-style catch-up must not hand every replayed block the same -/// idempotency key. +/// A span missed while the hook was dead is announced, not replayed. +/// +/// Webhooks are realtime (design D6): the durable cursor is a resume marker, +/// not a replay log. On restart the dispatcher compares it against the tip and +/// emits one `Lagged` naming what was missed; the events themselves are gone, +/// and a receiver that needs them fetches them over the streaming API from the +/// advertised `resume_cursor`. /// -/// Replayed envelopes are synthesized by the replay builder, which stamps every -/// one of them `seq: 0`. Minting the delivery id from that stamp gave a node -/// that had been down for N blocks N deliveries sharing one -/// `X-Satd-Delivery` — so a receiver following the contract ("deduplicate on -/// it") keeps the first and silently discards the rest. Catch-up exists -/// precisely to close that gap, so the bug quietly reduced it to delivering one -/// block per outage. +/// This replaces a test of the old catch-up replay. That replay was built up to +/// 10,000 blocks into a queue holding 1,024, so any outage long enough to +/// matter had its "guaranteed" recovery converted straight back into an +/// overflow gap — and its per-height delivery ids collided between a block and +/// its post-reorg replacement, so a conforming receiver dropped the +/// replacement. #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn catch_up_replay_gives_every_event_a_distinct_delivery_id() { - let receiver = MockReceiver::start(Behavior::Ok).await; +async fn a_missed_span_is_announced_as_a_gap_and_never_replayed() { + // 503 blocks 2..=5 so they stay in retry and never advance the marker. + // Everything else — block 1, and the gap notice itself — is accepted. + let receiver = MockReceiver::failing_heights(vec![2, 3, 4, 5], 503).await; let (sn, dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; - // Establish a cursor: one block delivered and acked. + // Establish a marker: one block delivered and acked. crate::streaming::mine_n(&sn, 1).await; receiver .wait_for(60, |r| r.json()["body"]["height"] == 1) .await; tokio::time::sleep(Duration::from_secs(2)).await; - // Simulate the outage: retire the dispatcher by pointing the hook at a - // black hole, mine past it, then point it back. The blocks mined while the - // hook was dead are exactly the span catch-up must replay. - // Always-500 stands in for "the receiver is down": nothing is ever acked, - // so the shared cursor stays at block 1 and the blocks mined below are the - // span catch-up has to replay. - let sink = MockReceiver::start(Behavior::FailFirst(500, usize::MAX)).await; - write_alertfile(&dir, &sink, "chain", "\"chain\"", false); - crate::streaming::sighup_with_conf(&sn, "").await; - tokio::time::sleep(Duration::from_secs(2)).await; + // The outage. These four are queued and retrying; none is acked, so the + // marker stays at 1 while the tip moves to 5. crate::streaming::mine_n(&sn, 4).await; - tokio::time::sleep(Duration::from_secs(2)).await; + tokio::time::sleep(Duration::from_secs(3)).await; - let before = receiver.all().await.len(); - write_alertfile(&dir, &receiver, "chain", "\"chain\"", false); + let missed = |r: &Received| (2..=5).contains(&r.json()["body"]["height"].as_u64().unwrap_or(0)); + let attempts_before = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); + assert!(attempts_before > 0, "the outage span should have been attempted at least once"); + + // Retire the generation with a same-URL alertfile edit — the same code path + // a restart takes. The queued, still-retrying blocks go with it, which is + // exactly why the receiver has to be told. + write_alertfile(&dir, &receiver, "chain", "\"chain\"", true); crate::streaming::sighup_with_conf(&sn, "").await; - // Wait for the replayed span to arrive. - receiver - .wait_for(60, |r| r.json()["body"]["height"] == 5) + let lagged = receiver + .wait_for(60, |r| r.json()["body"]["category"] == "lagged") .await; - tokio::time::sleep(Duration::from_secs(2)).await; - - let replayed: Vec = receiver - .all() - .await - .into_iter() - .skip(before) - .map(|r| r.header("x-satd-delivery").to_string()) - .collect(); assert!( - replayed.len() >= 3, - "expected a multi-block replay, got {replayed:?}" + lagged.json()["body"]["dropped_count"].as_u64().unwrap_or(0) >= 4, + "the whole missed span must be counted; got {}", + lagged.body, + ); + // The anchor points at or before the last block the receiver actually got, + // never past the hole it is announcing. + assert!( + lagged.json()["body"]["resume_cursor"]["height"].as_u64().unwrap_or(u64::MAX) <= 1, + "resume anchor is past the gap it announces; got {}", + lagged.body, ); - let unique: std::collections::HashSet<&String> = replayed.iter().collect(); + tokio::time::sleep(Duration::from_secs(3)).await; + + // ...and nothing from the missed span was re-sent. + let attempts_after = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); assert_eq!( - unique.len(), - replayed.len(), - "replayed events shared an idempotency key; a conforming receiver \ - would drop all but one: {replayed:?}" + attempts_after, attempts_before, + "the missed span was replayed; a gap notice replaces it, it does not precede it" ); + + // The hook is live: the next block still arrives. + crate::streaming::mine_n(&sn, 1).await; + receiver + .wait_for(60, |r| r.json()["body"]["height"] == 6) + .await; } From 9886f673bb64145235041f68f368590f8efd239c Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Wed, 29 Jul 2026 23:22:31 -0600 Subject: [PATCH 06/11] alerting: hand the dispatcher over without dropping events MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two P0s in the SIGHUP handover, both of which silently destroy status events — the one category with no replay behind it. The bus subscription was taken inside the spawned fan-in, so it did not exist until the executor first polled that task. Everything published between retiring the outgoing generation and that first poll reached nobody: not delayed, gone. The detectors are edge-triggered against a HealthState that outlives the reload, so a disk_low landing in the window is never re-raised and the page never arrives. apply() now subscribes synchronously, before anything is retired, and hands the receiver to the task. Both generations are briefly subscribed, which is the safe direction: a bus delivery id is node-instance-, so they mint the same id and a receiver deduplicating on X-Satd-Delivery collapses them. A reload also rebuilt every hook's delivery task, discarding the queue and retry backoff of hooks the operator never touched. The unchanged-file early return covered the case where nothing changed at all; editing one stanza took every other hook's in-flight event down with it. The reloader now owns the delivery tasks and reconciles them per hook: a stanza that compares equal keeps its task, an edited or removed one is retired (and hands its undelivered count back to the process-lived GapState, so the hole is announced rather than swallowed). A carried-over hook also skips the startup gap announcement — its queue survived, so the span between its cursor and the tip is a backlog about to be delivered, not a hole. Stop's second channel is now per-task rather than per-generation, and a dropped sender counts as stopped: watch::Receiver::changed() returns Err forever once the sender is gone, which would have spun the loop at 100% CPU the moment a stop sender was dropped without sending. Four tests, each verified to fail against the pre-fix code: the subscriber-count handover invariant, hook carry-over and its mirror for an edited hook, and an e2e case where editing one hook must not destroy another's queued disk_low. --- node/src/events/publisher.rs | 10 + satd/src/alert.rs | 469 ++++++++++++++++++++++++++++------- satd/tests/e2e/webhooks.rs | 81 ++++++ 3 files changed, 473 insertions(+), 87 deletions(-) diff --git a/node/src/events/publisher.rs b/node/src/events/publisher.rs index 7e9a3cb6f..39bc4b9cc 100644 --- a/node/src/events/publisher.rs +++ b/node/src/events/publisher.rs @@ -204,6 +204,16 @@ impl EventPublisher { self.out.subscribe() } + /// How many receivers are currently subscribed to the envelope stream. + /// + /// A published event reaches exactly these; anything published while this + /// is zero is not buffered anywhere and cannot be recovered. That makes it + /// the observable a handover has to keep above zero — see + /// `satd::alert::AlertReloader::apply`. + pub fn subscriber_count(&self) -> usize { + self.out.receiver_count() + } + /// Edge identity stamped on every envelope. pub fn edge(&self) -> &EdgeIdentity { &self.edge diff --git a/satd/src/alert.rs b/satd/src/alert.rs index 9787ffd6e..a0a0dffae 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -56,33 +56,43 @@ const GAP_FLUSH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(1 /// counter would restart at zero and re-mint ids the receiver has already seen. static SYNTH_DELIVERY_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); -/// Composite stop signal: the process-wide shutdown, plus a per-generation -/// channel the SIGHUP reload flips to retire the previous dispatcher. +/// Composite stop signal: the process-wide shutdown, plus a narrower channel +/// the SIGHUP reload flips to retire one task. /// -/// Two channels rather than one because a reload must stop *this* generation's -/// tasks without touching the global signal every other subsystem watches. +/// Two channels rather than one because a reload must stop *some* of the +/// dispatcher's tasks without touching the global signal every other subsystem +/// watches. The scope of `local` differs by task on purpose: a fan-in is +/// retired per *generation*, because every reload rebuilds the hook list it +/// iterates; a delivery task is retired per *hook*, because a reload that +/// leaves a hook's stanza untouched must leave its queue alone (see +/// `AlertReloader::apply`). #[derive(Clone)] pub struct Stop { global: watch::Receiver, - generation: watch::Receiver, + local: watch::Receiver, } impl Stop { fn stopped(&self) -> bool { - *self.global.borrow() || *self.generation.borrow() + *self.global.borrow() || *self.local.borrow() } /// Resolve once either channel signals. Cancel-safe: `changed()` only /// marks a value seen when it completes, and the loop re-checks both /// borrows, so being dropped inside a `select!` loses nothing. + /// + /// A dropped sender counts as stopped. `changed()` returns `Err` forever + /// once the sender is gone, so ignoring it would spin this loop at 100% + /// CPU — and the reading is right anyway: nothing that could retire this + /// task still exists. async fn wait(&mut self) { loop { if self.stopped() { return; } tokio::select! { - _ = self.global.changed() => {} - _ = self.generation.changed() => {} + r = self.global.changed() => if r.is_err() { return }, + r = self.local.changed() => if r.is_err() { return }, } } } @@ -122,6 +132,15 @@ struct HookChannel { /// genuinely dead hook once, not stay quiet because a retired generation /// already said so. reported_closed: std::sync::atomic::AtomicBool, + /// Whether this hook's delivery task was started by the same `apply` that + /// built this channel, and so should be told what it missed while nothing + /// was delivering for it. + /// + /// False for a hook carried across a reload. Its queue survived, so the + /// span between its durable cursor and the tip is not a hole — it is a + /// backlog about to be delivered, and announcing it as a gap tells the + /// receiver to go rescan a range it is about to be sent anyway. + fresh: bool, } /// Accumulated drop state for one hook. @@ -182,64 +201,75 @@ impl DispatcherState { static DISPATCHER_STATE: std::sync::LazyLock> = std::sync::LazyLock::new(|| Arc::new(DispatcherState::default())); -/// Spawn the dispatcher for a parsed alertfile. +/// A delivery task the reloader keeps alive across reloads. /// -/// Must be called from within the API runtime — every task spawned here does -/// outbound HTTP, which is exactly what must never share the consensus runtime. -/// Returns `None` when the file configures no hooks, so a node with an empty -/// alertfile starts no tasks at all. -fn spawn_with_metrics( - file: &AlertFile, - publisher: Arc, +/// Held by `AlertReloader` rather than by the fan-in, so a reload can decide +/// per hook whether to keep the task — and its queue of pending deliveries — +/// or retire it. +struct RunningHook { + /// The stanza this task was started for. A reload reuses the task when the + /// new stanza compares equal, and retires it when it does not. + config: Hook, + tx: mpsc::Sender, + counters: Arc, + gap: Arc, + /// Retires this hook's delivery task alone. + stop: watch::Sender, +} + +/// Start one hook's delivery task. +/// +/// Must be called from within the API runtime — the task does outbound HTTP, +/// which is exactly what must never share the consensus runtime. +fn start_hook( + hook: &Hook, + metrics: &WebhookMetrics, + state: &DispatcherState, store: Arc, - block_source: Option>, - metrics: Arc, - state: Arc, - stop: Stop, -) { - if file.hooks.is_empty() { - return; - } - let mut hooks = Vec::new(); - for hook in &file.hooks { - let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); - let counters = metrics.hook(&hook.id); - let gap = state.gap_for(&hook.id); - tokio::spawn(deliver_loop( - hook.clone(), - rx, - counters.clone(), - gap.clone(), - store.clone(), - stop.clone(), - )); - hooks.push(HookChannel { - id: hook.id.clone(), - hook: hook.clone(), - tx, - counters, - gap, - reported_closed: std::sync::atomic::AtomicBool::new(false), - }); + global_stop: watch::Receiver, +) -> RunningHook { + let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); + let (stop_tx, stop_rx) = watch::channel(false); + let counters = metrics.hook(&hook.id); + let gap = state.gap_for(&hook.id); + tokio::spawn(deliver_loop( + hook.clone(), + rx, + counters.clone(), + gap.clone(), + store, + Stop { + global: global_stop, + local: stop_rx, + }, + )); + RunningHook { + config: hook.clone(), + tx, + counters, + gap, + stop: stop_tx, } - tokio::spawn(fan_in(hooks, publisher, store, block_source, state, stop)); } /// Fan-in: one broadcast receiver, filtered and enqueued per hook. +/// +/// The bus receiver is created by the caller, not here. When it is created +/// relative to retiring the previous generation is the whole handover protocol; +/// see `AlertReloader::apply`. async fn fan_in( hooks: Vec, + mut rx: tokio::sync::broadcast::Receiver, publisher: Arc, store: Arc, block_source: Option>, state: Arc, mut stop: Stop, ) { - let mut rx = publisher.subscribe(); - // Announce whatever each hook missed while the daemon was down. Nothing is // replayed — there is no snapshot-to-live seam to dedupe, because there is // no snapshot. - for hook in &hooks { + for hook in hooks.iter().filter(|h| h.fresh) { announce_gap(hook, store.as_ref(), block_source.as_deref()).await; } @@ -864,13 +894,27 @@ pub struct AlertReloader { block_source: Option>, metrics: Arc, global_stop: watch::Receiver, - /// Stop signal for the generation currently running, if any. - current: parking_lot::Mutex>>, + /// Everything a handover has to hand over. + running: parking_lot::Mutex, /// The last successfully-applied alertfile, so a SIGHUP that did not change /// it can be a no-op instead of destroying in-flight deliveries. last_applied: parking_lot::Mutex>, } +/// The live dispatcher, as much of it as a reload has to reason about. +/// +/// Under one lock because the handover in `apply` is a sequence over these +/// fields whose *order* is the correctness argument; splitting them would turn +/// that order into an unstated convention between independent critical +/// sections. +#[derive(Default)] +struct Running { + /// Retires the fan-in of the generation currently running. + fan_in_stop: Option>, + /// Live delivery tasks by hook id, carried across reloads. + hooks: std::collections::HashMap, +} + impl AlertReloader { #[allow(clippy::too_many_arguments)] pub fn new( @@ -890,7 +934,7 @@ impl AlertReloader { block_source, metrics, global_stop, - current: parking_lot::Mutex::new(None), + running: parking_lot::Mutex::new(Running::default()), last_applied: parking_lot::Mutex::new(None), } } @@ -909,16 +953,15 @@ impl AlertReloader { let file = AlertFile::load(&self.path)?; let ids: Vec = file.hooks.iter().map(|h| h.id.clone()).collect(); - // A SIGHUP that did not change the alertfile must not disturb the - // dispatcher. `reload_from_sighup` calls this on *every* SIGHUP, - // whatever key the operator actually edited, and retiring a generation - // destroys its queued deliveries. For chain events that is recoverable - // — the cursor did not advance, so the next generation's catch-up - // re-queues them — but a status event has no replay by design, and the - // detectors are edge-triggered against a `HealthState` that outlives - // the reload. So a `disk_low` sitting in retry backoff when the - // operator SIGHUPs to change `maxconnections` would be dropped, never - // replayed, and never re-raised: the page simply never arrives. + // A SIGHUP that did not change the alertfile is a no-op. + // `reload_from_sighup` calls this on *every* SIGHUP, whatever key the + // operator actually edited, so without this an unrelated + // `maxconnections` edit would churn the whole dispatcher. + // + // Belt and braces rather than the only defence: the handover below + // carries each hook's delivery task across a reload when that hook's + // stanza is unchanged, so even a real edit leaves the untouched hooks' + // queues and retry backoff intact. // // Comparing the parsed file rather than the file bytes means // reformatting or a comment edit is also a no-op. @@ -933,37 +976,118 @@ impl AlertReloader { } } - // Retire the previous generation only once the new file has parsed, so - // a bad edit never leaves the node with no dispatcher at all. - let (gen_tx, gen_rx) = watch::channel(false); - let stop = Stop { - global: self.global_stop.clone(), - generation: gen_rx, - }; - let publisher = self.publisher.clone(); - let store = self.store.clone(); - let block_source = self.block_source.clone(); - let metrics = self.metrics.clone(); let hook_count = file.hooks.len(); + + // ---- Handover ---------------------------------------------------- + // + // Subscribe the incoming generation to the bus here, synchronously, + // before anything is retired — and hand the receiver to the task rather + // than letting it subscribe for itself. + // + // A `broadcast::Receiver` only sees what is published after it is + // created. Subscribing inside the spawned fan-in means the subscription + // does not exist until the executor first polls that task, so every + // event published between retiring the outgoing generation and that + // first poll reaches nobody. Not delayed: gone. Status events have no + // replay by design and the detectors that raise them are edge-triggered + // against a `HealthState` that outlives the reload, so a `disk_low` + // that lands in the window is never re-raised and the page never + // arrives. The window is short but it is scheduler latency, which is + // longest exactly when the node is loaded enough to be raising alerts. + // + // Holding both subscriptions open for a moment is the safe direction: a + // bus delivery id is `node-instance-`, so the two + // generations mint the *same* id for the same event and a receiver + // deduplicating on `X-Satd-Delivery`, as the contract instructs, + // collapses them. + let bus_rx = self.publisher.subscribe(); + + let mut running = self.running.lock(); + + // Reconcile the delivery tasks. A hook whose stanza is unchanged keeps + // the task it already has, queue and retry backoff included — a reload + // is an operator editing one stanza, and it must not destroy pending + // deliveries for every *other* hook in the file. A status event has no + // replay behind it, so one sitting in backoff for an untouched hook is + // lost outright. Only hooks actually edited or removed are retired. + let mut next_hooks: std::collections::HashMap = + std::collections::HashMap::with_capacity(file.hooks.len()); + let mut channels: Vec = Vec::with_capacity(file.hooks.len()); { let _guard = self.api_handle.enter(); - spawn_with_metrics( - &file, - publisher, - store, - block_source, - metrics, - Arc::clone(&DISPATCHER_STATE), - stop, - ); + for hook in &file.hooks { + let kept = match running.hooks.remove(&hook.id) { + Some(r) if r.config == *hook => Some(r), + Some(edited) => { + // This one's queue is dropped on purpose: the operator + // changed where or how it delivers, and flushing the + // backlog to the superseded endpoint is not what they + // asked for. `deliver_loop` hands the undelivered count + // back to the process-lived `GapState` on its way out, + // so the hole is announced rather than swallowed. + let _ = edited.stop.send(true); + None + } + None => None, + }; + let fresh = kept.is_none(); + let running_hook = kept.unwrap_or_else(|| { + start_hook( + hook, + &self.metrics, + &DISPATCHER_STATE, + self.store.clone(), + self.global_stop.clone(), + ) + }); + channels.push(HookChannel { + id: hook.id.clone(), + hook: hook.clone(), + tx: running_hook.tx.clone(), + counters: running_hook.counters.clone(), + gap: running_hook.gap.clone(), + reported_closed: std::sync::atomic::AtomicBool::new(false), + fresh, + }); + next_hooks.insert(hook.id.clone(), running_hook); + } + // Whatever is left was removed from the file. + for (_, gone) in running.hooks.drain() { + let _ = gone.stop.send(true); + } } - if let Some(old) = self.current.lock().replace(gen_tx) { - // Draining is deliberate rather than graceful: the retired - // generation's queued events are dropped. A reload is an operator - // changing where alerts go, and delivering the backlog to the *old* - // endpoint after they redirected it is not what they asked for. + running.hooks = next_hooks; + + // Retire the outgoing fan-in. Its queued deliveries are not lost with + // it: the queues belong to the delivery tasks, which this handover has + // already decided the fate of, one hook at a time. + if let Some(old) = running.fan_in_stop.take() { let _ = old.send(true); } + + if channels.is_empty() { + // An alertfile with no hooks runs no tasks at all. + drop(bus_rx); + } else { + let (gen_tx, gen_rx) = watch::channel(false); + { + let _guard = self.api_handle.enter(); + tokio::spawn(fan_in( + channels, + bus_rx, + self.publisher.clone(), + self.store.clone(), + self.block_source.clone(), + Arc::clone(&DISPATCHER_STATE), + Stop { + global: self.global_stop.clone(), + local: gen_rx, + }, + )); + } + running.fan_in_stop = Some(gen_tx); + } + drop(running); // Stop exporting counters for hooks that are no longer configured, // rather than freezing their series at the last value forever. // @@ -1303,4 +1427,175 @@ heartbeat_interval_secs = 3600 let mut hb = None; assert!(!accepts(&hook, &env, &mut hb)); } + + // === Handover ========================================================== + + /// A reloader over a one-line alertfile in `dir`, plus the publisher it + /// dispatches from. + /// + /// The URL is a discard port on loopback: these tests are about what the + /// reloader does to its own tasks, not about delivery, and nothing here + /// waits on a response. + fn handover_fixture(dir: &std::path::Path) -> (AlertReloader, Arc) { + let publisher = EventPublisher::new( + node::events::EdgeIdentity::new([9; 16], None).expect("edge identity"), + 64, + ); + let (_tx, global_stop) = watch::channel(false); + // `_tx` must outlive the reloader or every `Stop` reads "sender gone" + // and the tasks retire themselves immediately. Leak it: the fixture + // owns nothing else that could hold it for the test's duration. + std::mem::forget(_tx); + let reloader = AlertReloader::new( + dir.join("alertfile.toml"), + tokio::runtime::Handle::current(), + publisher.clone(), + Arc::new(node::storage::db::InMemoryStore::new()), + None, + Arc::new(WebhookMetrics::new()), + global_stop, + ); + (reloader, publisher) + } + + fn write_hooks(dir: &std::path::Path, stanzas: &str) { + use std::os::unix::fs::PermissionsExt as _; + let path = dir.join("alertfile.toml"); + std::fs::write(&path, format!("version = 1\n{stanzas}")).expect("write alertfile"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); + } + + fn stanza(id: &str, categories: &str) -> String { + format!( + "\n[[webhook]]\nid = \"{id}\"\nurl = \"http://127.0.0.1:9/{id}\"\n\ + secret = \"{}\"\ncategories = [{categories}]\n", + "s".repeat(32) + ) + } + + /// `apply` must take the incoming generation's bus subscription itself, + /// not leave it to the task it spawns. + /// + /// A `broadcast::Receiver` only sees what is published after it is created. + /// If the fan-in subscribes for itself, the subscription does not exist + /// until the executor first polls it — and every event published between + /// retiring the outgoing generation and that first poll reaches nobody. + /// A status event lost there is lost for good: there is no replay, and the + /// detectors are edge-triggered against a `HealthState` that outlives the + /// reload, so the condition is never re-raised. + /// + /// This is a single-threaded runtime and nothing is awaited across the + /// second `apply`, so no spawned task has run: the subscriber count is + /// exactly what `apply` did synchronously. Deferring the subscription to + /// the task leaves it at 1 and fails here. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_subscribes_the_incoming_generation_before_retiring_the_old_one() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + // Let generation one's fan-in reach its select loop. + for _ in 0..8 { + tokio::task::yield_now().await; + } + assert_eq!( + publisher.subscriber_count(), + 1, + "one generation running should mean exactly one bus subscriber" + ); + + write_hooks( + dir.path(), + &format!( + "{}{}", + stanza("pager", "\"status\""), + stanza("ops", "\"chain\"") + ), + ); + reloader.apply().expect("second apply"); + + assert_eq!( + publisher.subscriber_count(), + 2, + "apply() must subscribe the incoming generation itself; deferring it \ + to the spawned task leaves the bus unsubscribed for as long as the \ + executor takes to poll, and events published in that window are gone" + ); + } + + /// Editing one hook must not destroy another hook's pending deliveries. + /// + /// A reload used to retire the whole generation, taking every hook's queue + /// and retry backoff with it. For chain events that is survivable — the + /// durable cursor did not advance. A status event has no replay by design, + /// so one sitting in backoff for a hook the operator never touched is + /// simply lost, and the edge-triggered detector will not raise it again. + /// + /// Identity of the `mpsc::Sender` is the observable: a carried-over hook + /// keeps the same channel, and therefore the same queue. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_carries_over_the_delivery_task_of_an_untouched_hook() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is running"); + + // Add an unrelated second hook. `pager`'s stanza is byte-identical. + write_hooks( + dir.path(), + &format!( + "{}{}", + stanza("pager", "\"status\""), + stanza("ops", "\"chain\"") + ), + ); + reloader.apply().expect("second apply"); + + let after = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is still running"); + assert!( + before.same_channel(&after), + "an untouched hook must keep its delivery task across a reload; a \ + fresh channel means its queued deliveries were destroyed" + ); + assert!( + !before.is_closed(), + "the carried-over hook's delivery task must still be running" + ); + } + + /// The mirror: a hook the operator *did* edit is retired, so the new + /// endpoint does not inherit a backlog addressed to the old one. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_retires_the_delivery_task_of_an_edited_hook() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader.running.lock().hooks["pager"].tx.clone(); + + write_hooks(dir.path(), &stanza("pager", "\"status\", \"chain\"")); + reloader.apply().expect("second apply"); + + let after = reloader.running.lock().hooks["pager"].tx.clone(); + assert!( + !before.same_channel(&after), + "an edited hook must get a fresh delivery task" + ); + } } diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index c61de77d7..8152547c6 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -361,6 +361,30 @@ categories = [{categories}] std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); } +/// Write a two-hook alertfile. `cats_b` is the only thing callers vary between +/// two writes, so hook `a`'s stanza stays byte-identical across a reload. +fn write_two_hook_alertfile( + dir: &tempfile::TempDir, + a: (&MockReceiver, &str, &str), + b: (&MockReceiver, &str, &str), +) { + use std::os::unix::fs::PermissionsExt as _; + let path = dir.path().join("alertfile.toml"); + let stanza = |(r, id, cats): (&MockReceiver, &str, &str)| { + format!( + "\n[[webhook]]\nid = \"{id}\"\nurl = \"{}\"\nsecret = \"{SECRET}\"\n\ + categories = [{cats}]\n", + r.url() + ) + }; + std::fs::write( + &path, + format!("version = 1\n{}{}", stanza(a), stanza(b)), + ) + .expect("write alertfile"); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap(); +} + /// Start a node whose `alertfile=` points at a one-hook file delivering /// `categories` to `receiver`. /// @@ -571,6 +595,63 @@ async fn an_unrelated_sighup_does_not_destroy_a_queued_status_event() { assert!(got.signature_valid(SECRET)); } +/// Editing one hook must not destroy a *different* hook's queued event. +/// +/// The companion to `an_unrelated_sighup_does_not_destroy_a_queued_status_event`, +/// and the harder half. That one is protected by the unchanged-file early +/// return in `apply`; here the alertfile genuinely changes, so the reload runs +/// the whole handover. A reload that rebuilt every delivery task would take +/// `pager`'s in-flight `disk_low` down with it even though the operator only +/// touched `ops` — and a status event has no replay, so it is lost outright and +/// the edge-triggered detector never raises it again. +/// +/// `pager` 503s the first four attempts, so the event is provably still in its +/// queue, mid-backoff, when the reload lands. +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn editing_one_hook_does_not_destroy_another_hooks_queued_event() { + let pager = MockReceiver::failing_first(503, 4).await; + let ops = MockReceiver::ok().await; + + let dir = tempfile::tempdir().expect("tempdir"); + write_two_hook_alertfile( + &dir, + (&pager, "pager", "\"status\""), + (&ops, "ops", "\"status\""), + ); + let path = dir.path().join("alertfile.toml"); + let sn = + crate::streaming::start_streaming_owned(vec![format!("--alertfile={}", path.display())]) + .await; + + // Known-cleared, then raised, so the detector fires an edge on any host. + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=1\n").await; + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let deadline = std::time::Instant::now() + Duration::from_secs(45); + while pager.attempts().await == 0 { + assert!( + std::time::Instant::now() < deadline, + "the status event was never attempted; the raise did not reach the hook" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + } + + // Edit `ops` only. `pager`'s stanza is byte-identical, so its delivery task + // — and the `disk_low` still in its queue — must survive. + write_two_hook_alertfile( + &dir, + (&pager, "pager", "\"status\""), + (&ops, "ops", "\"status\", \"chain\""), + ); + crate::streaming::sighup_with_conf(&sn, "alertdiskfreemb=17592186044416\n").await; + + let got = pager + .wait_for(60, |r| r.json()["body"]["category"] == "status") + .await; + assert_eq!(got.json()["body"]["kind"], "disk_low"); + assert!(got.signature_valid(SECRET)); +} + /// A permanently-rejected event still advances the hook's resume cursor. /// /// Otherwise a receiver that hard-rejects one body shape turns every reload and From b4bdaef954f73e3953b479ee201c8cee07547a8e Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Thu, 30 Jul 2026 08:31:41 -0600 Subject: [PATCH 07/11] alerting: correct the docs for realtime delivery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The manual, changelog and release notes still described the replay machinery that D6 removed: "at-least-once ... on startup the hook replays what it missed", and a `b` delivery-id space for replayed blocks that no longer exists. Both now say what the code does — at-most-once with every gap announced, and a resume marker rather than a replay cursor — and point at RescanBlocks or the JSON-RPC history calls for operators who actually want the missing span. Also documents the reload handover: an untouched hook keeps its delivery task, so editing one stanza no longer discards another hook's pending page. --- CHANGELOG.md | 6 ++++-- docs/manual/src/observability.md | 33 +++++++++++++++++--------------- docs/release-notes/0.5.0-pre.md | 21 ++++++++++++++------ 3 files changed, 37 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 831164058..533ac0b5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -162,8 +162,10 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). permanent 4xx (which still advances the resume position), with a bounded queue whose overflow is reported in-band as a `lagged` event rather than silently. Redirects are never followed — a 3xx is a permanent drop, so a - signed body cannot be steered to a host the alertfile never named. Confirmed chain events are at-least-once - across a restart via a durable per-hook cursor. Hooks reload on SIGHUP + signed body cannot be steered to a host the alertfile never named. Delivery + is at-most-once with every gap announced: the durable per-hook cursor is a + resume *marker*, so a hook coming back is told what it missed with one + `lagged` body rather than being re-sent the span. Hooks reload on SIGHUP (keep-last-good on error); per-hook counters are exported as `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with their original payload, now delivered by the same dispatcher — which also diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index 5070e82f8..b5281ff6d 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -232,10 +232,8 @@ So a receiver must: The `X-Satd-Delivery` value is opaque; do not parse it. Its suffix distinguishes how the delivery arose — a bare counter for a live event, `w` for a watch match, -`r` for a synthesized notice such as `lagged`, and `b` for a replayed -block. That last one is deliberately derived from the height rather than a -counter, so the same block replayed after a restart carries the same id and your -dedup actually collapses it. +and `r` for a synthesized notice such as `lagged`. Deduplicate on the whole +header; it is unique per event and stable across the retries of one delivery. ### Delivery behavior @@ -245,8 +243,9 @@ dedup actually collapses it. 1 s doubling to a 5-minute ceiling, indefinitely. Any *other* 4xx is treated as permanent, counted, and skipped — a receiver answering 404 forever must not pin the queue and turn every later event into a drop. A skipped event - still advances the hook's resume position, so a hard-rejecting endpoint does - not turn every restart into a replay of the same refused span. + still advances the hook's resume position, so a hard-rejecting endpoint makes + progress instead of announcing the same refused span as a gap after every + restart. The skip is counted and reported in the next `lagged` body. - **Redirects are not followed.** A 3xx is a permanent drop. The URL in the alertfile is where the signed body goes; following a redirect would move it — signature, hook identity and all — to a host you never named, and the useful @@ -258,15 +257,19 @@ dedup actually collapses it. the cursor to resume from. A gap is never silent. - **Nothing reaches consensus.** Deliveries run on the isolated API runtime; a stalled endpoint cannot affect block connection. -- **At-least-once for confirmed chain events across a restart, or you are told.** - Each hook's resume position is persisted, and on startup the hook replays what - it missed (bounded by the same 10 000-block window the streaming API uses; - beyond that you get a `lagged` notice instead). Health events are re-raised by - re-evaluation rather than replayed, and mempool events are best-effort — the - same contract the event bus itself offers. Every way an event can be skipped — - queue overflow, a permanent rejection, suppression during a sync — increments - the hook's drop count and produces a `lagged` body, so the guarantee is - precisely "delivered, or reported missing"; it is never silent. +- **At-most-once, and a gap is announced.** Webhooks deliver what is happening + now; they are not a log you can rewind. Each hook's resume position is + persisted, but it is a *marker*, not a replay cursor: on startup the hook is + told what it missed while the daemon was down — one `lagged` body carrying the + count and the height to resume from — and then goes live. It does not re-send + the span. Health events are re-raised by re-evaluation, and mempool events are + best-effort, the same contract the event bus itself offers. Every way an event + can be skipped — the daemon being down, queue overflow, a permanent rejection, + suppression during a sync — increments the hook's drop count and produces a + `lagged` body. So the guarantee is precisely "delivered, or reported missing", + never both and never silent. If you need the span itself, the `lagged` body + carries the position to fetch it from: use `RescanBlocks` on the streaming API + or the JSON-RPC history calls. - **Suppressed during initial block download.** A node syncing from scratch does not POST its entire history: while it is catching up, only `status` and `heartbeat` events are delivered. Health alerts stay live because "this node is diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index e7e7022a0..f87bd7337 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -416,16 +416,25 @@ kinds = ["tip_stall", "disk_low"] carry the signed body to a host the alertfile never named. A hook that falls far enough behind drops events and the next delivery is preceded by a `lagged` body carrying the count and a resume cursor: a gap is never silent. -- **At-least-once for confirmed chain events across a restart.** Each hook's - resume position is persisted with the chainstate, and on startup the hook - replays what it missed, bounded by the same 10 000-block window the streaming - API uses. Health events come back by re-evaluation instead; mempool events - stay best-effort, which is the contract the bus itself offers. +- **At-most-once, with every gap announced.** Webhooks carry what is happening + now; they are not a log you can rewind. Each hook's resume position is + persisted with the chainstate, but it is a marker rather than a replay + cursor: a hook coming back after a restart, an outage, or an overflow gets + one `lagged` body carrying how much it missed and the height to resume from, + and then goes live. It is never sent the span twice, and never left to + discover the hole itself. Health events come back by re-evaluation instead; + mempool events stay best-effort, which is the contract the bus itself offers. + When you do need the missing span, fetch it deliberately — `RescanBlocks` on + the streaming API, or the JSON-RPC history calls. - **Reloadable.** Hook definitions are re-read on every SIGHUP even when no `bitcoin.conf` key changed, so a hook can be added, edited, or removed live. A parse error keeps the last-good set — alerting that silently stopped after a typo is the worse failure. The file must be mode 0600; it holds signing - secrets. + secrets. A reload is a handover, not a restart: a hook whose stanza you did + not touch keeps its delivery task, so its queue and retry backoff survive. + Editing one hook does not discard a page that another hook was still + retrying — which matters because a status event has no replay behind it and + the detector that raised it will not raise it twice. - Per-hook counters are exported (`satd_alertwebhook_delivered_total`, `_failed_attempts_total`, `_dropped_total`, `_queue_depth`, `_last_success_age_seconds`), and a node with no alertfile exports none of From 490ac766632d9ee4244399b18104637060b3662f Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Thu, 30 Jul 2026 09:44:22 -0600 Subject: [PATCH 08/11] alerting: webhooks are best-effort; delete the durability machinery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scope correction per the amended design D6. Webhooks are the basic way to automate off chain and mempool events; the streaming API is the canonical integration surface and already does resumable consumption properly. Every durability feature here was a worse restatement of that. Gone: the durable resume cursor and with it the Store trait extension (read/write/delete/list_alert_cursor_keys) across all four backends, the cursor GC pass, URL-derived key rebinding, the fixed 24-byte cursor codec, and per-height write throttling. Gone with it: GapState, gap_weight conservation through overflow and permanent rejection, drain_owed, announce_gap, flush_gap, the 15s flush timer, the synthesized delivery-id space, and the Lagged body. A drop is now a counter and a log line. The Lagged filter arm went too. lagged_event is built per-connection by the streaming carriers and never published to the shared bus, so the dispatcher could not receive one — the arm was unreachable and its test vacuous. satd/src/alert.rs 1601 → 1142 lines; node/src/storage/ back to master. The e2e suite asserts the new contract directly: a hook that was down resumes at the live head, re-sends nothing, and synthesizes no notice. --- CHANGELOG.md | 9 +- docs/manual/src/observability.md | 50 ++- docs/release-notes/0.5.0-pre.md | 23 +- node/src/storage/coin_cache.rs | 16 - node/src/storage/db.rs | 33 -- node/src/storage/mod.rs | 46 --- node/src/storage/rocksdb_store.rs | 51 --- node/src/storage/split_store.rs | 19 -- satd-alert/src/config.rs | 61 +--- satd/src/alert.rs | 539 +++--------------------------- satd/src/main.rs | 5 +- satd/tests/e2e/webhooks.rs | 210 ++---------- 12 files changed, 125 insertions(+), 937 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 533ac0b5f..0c52aef00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -162,10 +162,11 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). permanent 4xx (which still advances the resume position), with a bounded queue whose overflow is reported in-band as a `lagged` event rather than silently. Redirects are never followed — a 3xx is a permanent drop, so a - signed body cannot be steered to a host the alertfile never named. Delivery - is at-most-once with every gap announced: the durable per-hook cursor is a - resume *marker*, so a hook coming back is told what it missed with one - `lagged` body rather than being re-sent the span. Hooks reload on SIGHUP + signed body cannot be steered to a host the alertfile never named. Delivery is + **best-effort**: nothing is persisted, a hook that was down resumes at the + live head, and drops are counted in `satd_alertwebhook_dropped_total`. The + Streaming Consumption API remains the surface for guaranteed, resumable + consumption. Hooks reload on SIGHUP (keep-last-good on error); per-hook counters are exported as `satd_alertwebhook_*`. The existing `reorgwebhook=` keys keep working with their original payload, now delivered by the same dispatcher — which also diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index b5281ff6d..f84d84c84 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -230,10 +230,9 @@ So a receiver must: > keys still use exactly that, unchanged, and still report version `1` — branch > on the version header rather than assuming one scheme. -The `X-Satd-Delivery` value is opaque; do not parse it. Its suffix distinguishes -how the delivery arose — a bare counter for a live event, `w` for a watch match, -and `r` for a synthesized notice such as `lagged`. Deduplicate on the whole -header; it is unique per event and stable across the retries of one delivery. +The `X-Satd-Delivery` value is opaque; do not parse it. Deduplicate on the whole +header: it is unique per event and stable across the retries of one delivery, so +the only duplicate you can receive is a retry of something you already saw. ### Delivery behavior @@ -243,40 +242,39 @@ header; it is unique per event and stable across the retries of one delivery. 1 s doubling to a 5-minute ceiling, indefinitely. Any *other* 4xx is treated as permanent, counted, and skipped — a receiver answering 404 forever must not pin the queue and turn every later event into a drop. A skipped event - still advances the hook's resume position, so a hard-rejecting endpoint makes - progress instead of announcing the same refused span as a gap after every - restart. The skip is counted and reported in the next `lagged` body. + The skip is counted in `satd_alertwebhook_dropped_total` and logged. - **Redirects are not followed.** A 3xx is a permanent drop. The URL in the alertfile is where the signed body goes; following a redirect would move it — signature, hook identity and all — to a host you never named, and the useful targets for that are exactly the ones you cannot see: a cloud metadata endpoint, an RFC1918 admin port, the node's own RPC. If your receiver moves, update the alertfile. -- **Bounded queue.** A hook that falls far enough behind drops events, and the - next delivery is preceded by a `lagged` body carrying how many were lost and - the cursor to resume from. A gap is never silent. +- **Bounded queue.** A hook that falls far enough behind drops events. They are + counted in `satd_alertwebhook_dropped_total` and logged; nothing is held for + later and nothing is inserted into the stream to tell the receiver. - **Nothing reaches consensus.** Deliveries run on the isolated API runtime; a stalled endpoint cannot affect block connection. -- **At-most-once, and a gap is announced.** Webhooks deliver what is happening - now; they are not a log you can rewind. Each hook's resume position is - persisted, but it is a *marker*, not a replay cursor: on startup the hook is - told what it missed while the daemon was down — one `lagged` body carrying the - count and the height to resume from — and then goes live. It does not re-send - the span. Health events are re-raised by re-evaluation, and mempool events are - best-effort, the same contract the event bus itself offers. Every way an event - can be skipped — the daemon being down, queue overflow, a permanent rejection, - suppression during a sync — increments the hook's drop count and produces a - `lagged` body. So the guarantee is precisely "delivered, or reported missing", - never both and never silent. If you need the span itself, the `lagged` body - carries the position to fetch it from: use `RescanBlocks` on the streaming API - or the JSON-RPC history calls. +- **Best-effort, and that is the whole contract.** Nothing is persisted. A + webhook fires when something happens and is retried while your endpoint is + briefly unreachable; that is all it promises. A node that was down did not + deliver those events and will not go back for them — when it comes up, its + hooks resume at the live head. Health alerts are the exception, and they get + it for free: the detectors re-evaluate at startup and re-raise anything still + true (see **Durability** above), so a standing problem still reaches you. + + If you need guaranteed delivery, resumability across downtime, or history, + use the [Streaming Consumption API](streaming.md). That is the recommended + way to integrate with satd and it does all three properly — real cursors, + backpressure, and a bounded `RescanBlocks`. Webhooks are for automation you + are happy to miss occasionally: page me, poke a script, ping a dead-man's + switch. - **Suppressed during initial block download.** A node syncing from scratch does not POST its entire history: while it is catching up, only `status` and `heartbeat` events are delivered. Health alerts stay live because "this node is unhealthy" is exactly as true mid-sync, and the heartbeat keeps flowing so an - external dead-man's switch does not declare a syncing node dead. Everything - suppressed is counted and reported in the next `lagged` body. The suppression - is latched on leaving IBD once, so a node whose tip later goes stale keeps + external dead-man's switch does not declare a syncing node dead. What was + suppressed is counted in `satd_alertwebhook_dropped_total`. The suppression is + latched on leaving IBD once, so a node whose tip later goes stale keeps alerting — which is the whole point of a stalled-tip alert. Plaintext `http://` is accepted for loopback and private-network targets. For a diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index f87bd7337..25850903f 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -416,16 +416,19 @@ kinds = ["tip_stall", "disk_low"] carry the signed body to a host the alertfile never named. A hook that falls far enough behind drops events and the next delivery is preceded by a `lagged` body carrying the count and a resume cursor: a gap is never silent. -- **At-most-once, with every gap announced.** Webhooks carry what is happening - now; they are not a log you can rewind. Each hook's resume position is - persisted with the chainstate, but it is a marker rather than a replay - cursor: a hook coming back after a restart, an outage, or an overflow gets - one `lagged` body carrying how much it missed and the height to resume from, - and then goes live. It is never sent the span twice, and never left to - discover the hole itself. Health events come back by re-evaluation instead; - mempool events stay best-effort, which is the contract the bus itself offers. - When you do need the missing span, fetch it deliberately — `RescanBlocks` on - the streaming API, or the JSON-RPC history calls. +- **Best-effort by design.** Webhooks are the basic way to automate off chain + and mempool events — page me, poke a script, ping a dead-man's switch. They + are not an integration API and deliberately do not try to be one: nothing is + persisted, a hook that was down resumes at the live head, and events dropped + by a full queue or a lagging bus are counted in + `satd_alertwebhook_dropped_total` and logged. Retries cover the one failure + the surface promises to survive, a receiver that is briefly unreachable. + Health alerts are the exception and get it for free: the detectors re-evaluate + at startup and re-raise anything still standing, so a live problem reaches you + across a restart without any delivery machinery. Anything needing guaranteed + delivery, resumability, or history uses the **Streaming Consumption API**, + which is the canonical way to integrate with satd and already does all of it + properly — real cursors, backpressure, and a bounded `RescanBlocks`. - **Reloadable.** Hook definitions are re-read on every SIGHUP even when no `bitcoin.conf` key changed, so a hook can be added, edited, or removed live. A parse error keeps the last-good set — alerting that silently stopped after diff --git a/node/src/storage/coin_cache.rs b/node/src/storage/coin_cache.rs index 4df597d09..90f352b57 100644 --- a/node/src/storage/coin_cache.rs +++ b/node/src/storage/coin_cache.rs @@ -1284,22 +1284,6 @@ impl Store for CoinCache { self.inner.write_sp_backfill_last_error(msg) } - fn read_alert_cursor(&self, key: &[u8]) -> Option> { - self.inner.read_alert_cursor(key) - } - - fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { - self.inner.write_alert_cursor(key, value) - } - - fn list_alert_cursor_keys(&self) -> Result>, StoreError> { - self.inner.list_alert_cursor_keys() - } - - fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { - self.inner.delete_alert_cursor(key) - } - fn lookup_spend( &self, outpoint: &OutPoint, diff --git a/node/src/storage/db.rs b/node/src/storage/db.rs index 9b24f4f8c..ddb2e3d34 100644 --- a/node/src/storage/db.rs +++ b/node/src/storage/db.rs @@ -25,11 +25,6 @@ pub struct InMemoryStore { tx_index: parking_lot::RwLock>, chain_tx: parking_lot::RwLock>, chain_tx_backfill_complete: parking_lot::RwLock, - /// Alert-webhook resume markers. Present so that cursor and GC tests run - /// against real behaviour: the `Store` defaults return `Ok(vec![])` from - /// `list_alert_cursor_keys` and `Err` from `write_alert_cursor`, which - /// would make every such test pass without exercising anything. - alert_cursors: parking_lot::RwLock, Vec>>, addr_funding: parking_lot::RwLock>, addr_spending: parking_lot::RwLock>, outpoint_spend: parking_lot::RwLock>, @@ -73,7 +68,6 @@ impl InMemoryStore { // chains populate chain_tx directly, so the backfill is a no-op // there regardless. chain_tx_backfill_complete: parking_lot::RwLock::new(false), - alert_cursors: parking_lot::RwLock::new(std::collections::BTreeMap::new()), addr_funding: parking_lot::RwLock::new(Vec::new()), addr_spending: parking_lot::RwLock::new(Vec::new()), outpoint_spend: parking_lot::RwLock::new(std::collections::HashMap::new()), @@ -112,33 +106,6 @@ impl InMemoryStore { } impl Store for InMemoryStore { - fn read_alert_cursor(&self, key: &[u8]) -> Option> { - self.alert_cursors.read().get(key).cloned() - } - - fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { - self.alert_cursors - .write() - .insert(key.to_vec(), value.to_vec()); - Ok(()) - } - - fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { - self.alert_cursors.write().remove(key); - Ok(()) - } - - fn list_alert_cursor_keys(&self) -> Result>, StoreError> { - const PREFIX: &[u8] = b"alertwebhook.cursor."; - Ok(self - .alert_cursors - .read() - .keys() - .filter(|k| k.starts_with(PREFIX)) - .cloned() - .collect()) - } - fn flush_durable(&self) -> Result<(), StoreError> { self.flush_durable_calls .fetch_add(1, std::sync::atomic::Ordering::Relaxed); diff --git a/node/src/storage/mod.rs b/node/src/storage/mod.rs index d0d87216d..d467b73ba 100644 --- a/node/src/storage/mod.rs +++ b/node/src/storage/mod.rs @@ -845,50 +845,4 @@ pub trait Store: Send + Sync { "write_sp_backfill_last_error not supported on this backend".into(), )) } - - /// Read a webhook hook's durable resume cursor from metadata, by its - /// `alertwebhook.cursor.` key. `None` when the hook has never acked a - /// cursor-bearing delivery (a fresh hook starts at the live head). - /// - /// Deliberately narrow rather than a generic metadata get/put: the alert - /// dispatcher is the only writer, and a general "write any key into the - /// chainstate metadata CF" accessor is an invitation to collide with the - /// index markers that share it. - fn read_alert_cursor(&self, _key: &[u8]) -> Option> { - None - } - - /// Persist a webhook hook's resume cursor. Default: error, so a non-Rocks - /// backend fails loud rather than silently losing at-least-once delivery. - fn write_alert_cursor(&self, _key: &[u8], _value: &[u8]) -> Result<(), StoreError> { - Err(StoreError::Database( - "write_alert_cursor not supported on this backend".into(), - )) - } - - /// Forget a hook's resume cursor. - /// - /// Called when a hook disappears from the alertfile. Without it the key - /// outlives the hook forever, and hook ids are reused (`pager`, `alerts`, - /// `ops`) — so pointing a *new* endpoint at an old id would greet it with - /// the entire replay window of history belonging to its predecessor, with - /// no way to reset short of a reindex. - fn delete_alert_cursor(&self, _key: &[u8]) -> Result<(), StoreError> { - Err(StoreError::Database( - "delete_alert_cursor not supported on this backend".into(), - )) - } - - /// Every persisted alert-cursor key, for reconciling against the configured - /// hooks. - /// - /// Removing a hook from the alertfile and restarting is the ordinary way - /// operators change this configuration, and a reload-time diff cannot see - /// it: at startup there is no previous generation to compare against. So - /// the dispatcher enumerates what is actually stored and drops whatever no - /// longer has a hook, which also cleans up cursors left behind by earlier - /// versions that had no GC at all. - fn list_alert_cursor_keys(&self) -> Result>, StoreError> { - Ok(Vec::new()) - } } diff --git a/node/src/storage/rocksdb_store.rs b/node/src/storage/rocksdb_store.rs index 5d68cf988..cb435721a 100644 --- a/node/src/storage/rocksdb_store.rs +++ b/node/src/storage/rocksdb_store.rs @@ -1249,57 +1249,6 @@ impl RocksDbStore { } impl Store for RocksDbStore { - /// Read a hook's `alertwebhook.cursor.` blob. The key namespace is - /// disjoint from the index markers that share this CF. - fn read_alert_cursor(&self, key: &[u8]) -> Option> { - let cf = self.db.cf_handle(CF_METADATA)?; - self.db.get_cf(&cf, key).ok().flatten() - } - - fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { - let cf = self - .db - .cf_handle(CF_METADATA) - .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; - self.db - .put_cf(&cf, key, value) - .map_err(|e| StoreError::Database(e.to_string())) - } - - fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { - let cf = self - .db - .cf_handle(CF_METADATA) - .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; - self.db - .delete_cf(&cf, key) - .map_err(|e| StoreError::Database(e.to_string())) - } - - fn list_alert_cursor_keys(&self) -> Result>, StoreError> { - const PREFIX: &[u8] = b"alertwebhook.cursor."; - let cf = self - .db - .cf_handle(CF_METADATA) - .ok_or_else(|| StoreError::Database("metadata CF missing".into()))?; - let mut keys = Vec::new(); - // The metadata CF is shared with index markers, so the iteration is - // prefix-bounded and stops at the first key outside the namespace - // rather than walking the whole CF. - let iter = self.db.iterator_cf( - &cf, - rocksdb::IteratorMode::From(PREFIX, rocksdb::Direction::Forward), - ); - for item in iter { - let (key, _) = item.map_err(|e| StoreError::Database(e.to_string()))?; - if !key.starts_with(PREFIX) { - break; - } - keys.push(key.to_vec()); - } - Ok(keys) - } - fn get_block_index(&self, hash: &BlockHash) -> Option { let cf = self.cf(CF_BLOCK_INDEX); let value = self.db.get_cf(&cf, hash_bytes(hash)).ok()??; diff --git a/node/src/storage/split_store.rs b/node/src/storage/split_store.rs index 85a309214..41ca0970f 100644 --- a/node/src/storage/split_store.rs +++ b/node/src/storage/split_store.rs @@ -89,25 +89,6 @@ impl Store for SplitStore { self.block_store.get_cumulative_tx_count(hash) } - // Webhook resume cursors live with the block index in the shared store, for - // the same reason cumulative tx counts do: they must survive the - // snapshot→background handoff, which replaces the coins store. - fn read_alert_cursor(&self, key: &[u8]) -> Option> { - self.block_store.read_alert_cursor(key) - } - - fn write_alert_cursor(&self, key: &[u8], value: &[u8]) -> Result<(), StoreError> { - self.block_store.write_alert_cursor(key, value) - } - - fn list_alert_cursor_keys(&self) -> Result>, StoreError> { - self.block_store.list_alert_cursor_keys() - } - - fn delete_alert_cursor(&self, key: &[u8]) -> Result<(), StoreError> { - self.block_store.delete_alert_cursor(key) - } - fn chain_tx_backfill_complete(&self) -> bool { self.block_store.chain_tx_backfill_complete() } diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs index 9ab3f7abd..31169b579 100644 --- a/satd-alert/src/config.rs +++ b/satd-alert/src/config.rs @@ -4,7 +4,7 @@ //! version = 1 //! //! [[webhook]] -//! id = "ops" # unique; appears in headers and the cursor key +//! id = "ops" # unique; appears in headers and metric labels //! url = "https://alerts.example/satd" //! secret = "a-long-random-string" # required — it signs the body //! categories = ["status"] # status | chain | mempool | heartbeat @@ -154,21 +154,7 @@ impl std::fmt::Debug for Hook { } } -impl Hook { - /// The `CF_METADATA` key holding this hook's durable resume marker. - /// - /// Keyed on the id *and* a hash of the endpoint URL. On id alone, - /// repointing a hook at a new endpoint — which an operator does precisely - /// when the old one is dead and its marker is far behind — would hand the - /// new endpoint its predecessor's position and announce a gap it never had. - /// Including the URL means a repoint starts clean; the stale key is then - /// garbage-collected on the next reload like any other orphan. - pub fn cursor_key(&self) -> Vec { - use sha2::{Digest as _, Sha256}; - let url_hash = hex::encode(Sha256::digest(self.url.as_bytes())); - format!("alertwebhook.cursor.{}-{}", self.id, &url_hash[..16]).into_bytes() - } -} + /// A parsed, validated alertfile. #[derive(Debug, Clone, Default, PartialEq, Eq)] @@ -239,7 +225,7 @@ impl AlertFile { } // Ids name a hook in delivery headers, in metrics labels, and in the - // durable cursor key, so a duplicate would silently make two hooks + // metric label and header value, so a duplicate would conflate two hooks // share one resume position. let mut seen = BTreeSet::new(); for h in &hooks { @@ -278,7 +264,7 @@ fn parse_hook(path: &Path, t: &toml_edit::Table) -> Result if id.is_empty() || !id.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') { return Err(AlertFileError::invalid( path, - format!("webhook id `{id}` must be non-empty [A-Za-z0-9_-] (it is used in headers, metric labels, and the cursor key)"), + format!("webhook id `{id}` must be non-empty [A-Za-z0-9_-] (it is used in headers and metric labels)"), )); } let url = req_str(path, t, "url")?; @@ -588,43 +574,6 @@ categories = ["status"] assert!(h.filter.kinds.is_none()); assert!(h.filter.min_severity.is_none()); assert_eq!(h.heartbeat_interval_secs, None); - let key = String::from_utf8(h.cursor_key()).unwrap(); - assert!(key.starts_with("alertwebhook.cursor.ops-"), "got {key}"); - } - - /// Repointing a hook at a new endpoint must not hand it the old one's - /// resume marker. - /// - /// An operator changes a URL precisely when the old endpoint is dead and - /// its marker has fallen far behind. Keyed on the id alone, the new - /// endpoint would inherit that position and be told it had missed a span it - /// was never subscribed to. - #[test] - fn repointing_a_hook_changes_its_cursor_key() { - let a = AlertFile::parse( - Path::new("/tmp/a.toml"), - "version = 1\n\ - [[webhook]]\n\ - id = \"ops\"\n\ - url = \"https://a.example/hook\"\n\ - secret = \"0123456789abcdef0123456789abcdef\"\n\ - categories = [\"status\"]\n", - ) - .unwrap(); - let b = AlertFile::parse( - Path::new("/tmp/b.toml"), - "version = 1\n\ - [[webhook]]\n\ - id = \"ops\"\n\ - url = \"https://b.example/hook\"\n\ - secret = \"0123456789abcdef0123456789abcdef\"\n\ - categories = [\"status\"]\n", - ) - .unwrap(); - assert_ne!(a.hooks[0].cursor_key(), b.hooks[0].cursor_key()); - // Same URL, same key: a reload that changes nothing must not orphan the - // marker and restart the hook from the live head. - assert_eq!(a.hooks[0].cursor_key(), a.hooks[0].cursor_key()); } #[test] @@ -736,7 +685,7 @@ categories = ["chain"] #[test] fn duplicate_ids_are_rejected() { - // Two hooks sharing an id would share a durable cursor key. + // Two hooks sharing an id would be indistinguishable to a receiver. let two = format!("{MINIMAL}\n[[webhook]]\nid = \"ops\"\nurl = \"https://b.example/h\"\nsecret = \"t\"\ncategories = [\"chain\"]\n"); let err = parse(&two).unwrap_err(); assert!(err.to_string().contains("duplicate"), "{err}"); diff --git a/satd/src/alert.rs b/satd/src/alert.rs index a0a0dffae..fd354afef 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -13,11 +13,13 @@ //! structural, not a timeout budget. (The dispatcher this replaces ran its //! outbound HTTP on the consensus runtime.) //! -//! **A gap is never silent.** Both drop paths — a full hook queue and a lagged -//! broadcast receiver — set the hook's gap flag. Before its next delivery the -//! hook emits a synthesized `Lagged` body carrying the number of events lost -//! and the cursor to resume from, so a receiver that cares can go fetch the -//! span it missed. +//! **Best-effort, and deliberately so.** Nothing is persisted and nothing is +//! replayed. A full hook queue or a lagged broadcast receiver drops events, +//! moves `satd_alertwebhook_dropped_total`, and logs; a restart resumes at the +//! live head. Anything that needs to know what it missed wants the Streaming +//! Consumption API, which is the canonical integration surface and has real +//! cursors, backpressure and a bounded `RescanBlocks`. Adding durability here +//! would be a worse reimplementation of that one surface over. //! //! **Delivery is serial and ordered per hook.** One request in flight at a //! time, so a receiver observes events in the order the node produced them and @@ -31,9 +33,8 @@ use std::sync::Arc; use std::sync::atomic::Ordering; -use node::events::{Cursor, EventPublisher, NodeEvent, NodeEventBody}; +use node::events::{EventPublisher, NodeEvent, NodeEventBody}; use node::metrics::{HookCounters, WebhookMetrics}; -use node::storage::Store; use satd_alert::{AlertFile, Hook, HOOK_QUEUE_CAPACITY}; use tokio::sync::{mpsc, watch}; @@ -46,16 +47,6 @@ pub const LEGACY_REORG_HOOK_ID: &str = "reorg-legacy"; /// Per-attempt HTTP timeout. Matches the shipped reorg webhook. const REQUEST_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); -/// How often a pending gap notice is flushed when no event is driving the hook. -const GAP_FLUSH_INTERVAL: std::time::Duration = std::time::Duration::from_secs(15); - -/// Numbers synthesized deliveries — catch-up replay events and gap notices. -/// -/// Process-wide rather than per-generation, for the same reason the watch -/// counter is: a SIGHUP reload keeps the same `instance_id`, so a per-generation -/// counter would restart at zero and re-mint ids the receiver has already seen. -static SYNTH_DELIVERY_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); - /// Composite stop signal: the process-wide shutdown, plus a narrower channel /// the SIGHUP reload flips to retire one task. /// @@ -106,16 +97,6 @@ enum Delivery { Event { body: Arc>, delivery_id: String, - /// Cursor to persist once the receiver acks, if this event carries one. - cursor: Option, - /// How many lost events this delivery announces, if it is a `Lagged` - /// notice. The count travels *with* the delivery rather than being - /// retired when the notice is queued: a queued notice can still be - /// destroyed — by a reload retiring the generation, or by a permanent - /// rejection — and the one message that must not be lost is the one - /// saying data was lost. Anything that fails to deliver hands the - /// weight back to `GapState`. - gap_weight: u64, }, } @@ -124,40 +105,18 @@ struct HookChannel { hook: Hook, tx: mpsc::Sender, counters: Arc, - /// Set when an event was dropped for this hook; cleared when the resulting - /// `Lagged` notice has been queued. - gap: Arc, /// Whether the "delivery task is gone" warning has already been logged for /// this generation. Per-generation on purpose: a reload should re-report a /// genuinely dead hook once, not stay quiet because a retired generation /// already said so. reported_closed: std::sync::atomic::AtomicBool, - /// Whether this hook's delivery task was started by the same `apply` that - /// built this channel, and so should be told what it missed while nothing - /// was delivering for it. - /// - /// False for a hook carried across a reload. Its queue survived, so the - /// span between its durable cursor and the tip is not a hole — it is a - /// backlog about to be delivered, and announcing it as a gap tells the - /// receiver to go rescan a range it is about to be sent anyway. - fresh: bool, -} - -/// Accumulated drop state for one hook. -#[derive(Default)] -struct GapState { - dropped: std::sync::atomic::AtomicU64, - /// Position of the last event successfully queued before the gap — the - /// anchor a receiver resumes from. - resume_height: std::sync::atomic::AtomicU32, } /// Process-lived dispatcher state that must outlive any single generation. /// /// A reload retires the generation that accumulated a drop count, and -/// per-generation state would take the pending `Lagged` notice down with it — -/// the receiver would never learn about a hole satd had already recorded. -/// `HookCounters` is process-lived for the same reason. +/// `HookCounters` is process-lived so a reload does not reset an operator's +/// counters. /// /// Injected rather than reached for as a `static` so a test can hand in a fresh /// one. With a bare static the first test to latch `left_ibd` makes every later @@ -177,24 +136,6 @@ struct DispatcherState { /// of the gate: only chain events are suppressed, and chain events are by /// definition not arriving on a node whose tip has stopped. left_ibd: std::sync::atomic::AtomicBool, - gaps: parking_lot::Mutex>>, -} - -impl DispatcherState { - fn gap_for(&self, id: &str) -> Arc { - Arc::clone(self.gaps.lock().entry(id.to_string()).or_default()) - } - - /// Drop gap state for hooks no longer named in the alertfile. - /// - /// The mirror of `metrics.retain` and the cursor GC, which already run on - /// reload. Hook ids are short and reused — `pager`, `ops`, `alerts` — so a - /// re-added id would otherwise inherit its predecessor's pending drop count - /// and resume anchor, and announce a hole to a brand-new endpoint that - /// never had one. - fn retain_gaps(&self, keep: &std::collections::HashSet) { - self.gaps.lock().retain(|id, _| keep.contains(id)); - } } /// The process-wide instance. Tests construct their own. @@ -212,7 +153,6 @@ struct RunningHook { config: Hook, tx: mpsc::Sender, counters: Arc, - gap: Arc, /// Retires this hook's delivery task alone. stop: watch::Sender, } @@ -224,20 +164,15 @@ struct RunningHook { fn start_hook( hook: &Hook, metrics: &WebhookMetrics, - state: &DispatcherState, - store: Arc, global_stop: watch::Receiver, ) -> RunningHook { let (tx, rx) = mpsc::channel::(HOOK_QUEUE_CAPACITY); let (stop_tx, stop_rx) = watch::channel(false); let counters = metrics.hook(&hook.id); - let gap = state.gap_for(&hook.id); tokio::spawn(deliver_loop( hook.clone(), rx, counters.clone(), - gap.clone(), - store, Stop { global: global_stop, local: stop_rx, @@ -247,7 +182,6 @@ fn start_hook( config: hook.clone(), tx, counters, - gap, stop: stop_tx, } } @@ -261,18 +195,10 @@ async fn fan_in( hooks: Vec, mut rx: tokio::sync::broadcast::Receiver, publisher: Arc, - store: Arc, block_source: Option>, state: Arc, mut stop: Stop, ) { - // Announce whatever each hook missed while the daemon was down. Nothing is - // replayed — there is no snapshot-to-live seam to dedupe, because there is - // no snapshot. - for hook in hooks.iter().filter(|h| h.fresh) { - announce_gap(hook, store.as_ref(), block_source.as_deref()).await; - } - // Per-hook heartbeat downsampling state (D11): last forwarded instant. let mut last_heartbeat: Vec> = vec![None; hooks.len()]; @@ -282,22 +208,9 @@ async fn fan_in( "alert webhook dispatcher started", ); - // A gap notice is otherwise only emitted ahead of the *next* event for that - // hook, so a hook whose traffic stops right after a drop holds it - // indefinitely — and traffic stopping is correlated with the drop, not - // independent of it. The one message that must not wait is the one saying - // data was lost. - let mut gap_flush = tokio::time::interval(GAP_FLUSH_INTERVAL); - gap_flush.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - loop { tokio::select! { _ = stop.wait() => break, - _ = gap_flush.tick() => { - for hook in &hooks { - flush_gap(hook, &publisher); - } - } recv = rx.recv() => match recv { Ok(env) => { // Suppress the block firehose during initial block @@ -356,17 +269,10 @@ async fn fan_in( continue; } if syncing { - // Record the suppression as a gap rather than dropping - // it silently. A silent skip would leave no counter - // moved and no `lagged` body emitted, and the first - // post-IBD delivery would then advance the durable - // cursor straight across everything suppressed — making - // the span unrecoverable on the next restart while the - // module claims at-least-once. Counting it means the - // receiver is told, and the resume anchor still points - // before the hole. + // Counted rather than skipped silently, so an operator + // can see on `/metrics` that the sync suppressed + // deliveries rather than the hook being broken. for i in wanted { - hooks[i].gap.dropped.fetch_add(1, Ordering::Relaxed); hooks[i].counters.dropped.fetch_add(1, Ordering::Relaxed); } continue; @@ -386,20 +292,18 @@ async fn fan_in( ); for i in wanted { let hook = &hooks[i]; - enqueue(hook, &publisher, Delivery::Event { + enqueue(hook, Delivery::Event { body: body.clone(), delivery_id: delivery_id.clone(), - cursor: env.cursor, - gap_weight: 0, - }, env.cursor); + }); } } Err(tokio::sync::broadcast::error::RecvError::Lagged(n)) => { - // The dispatcher itself fell behind the bus. Every hook - // missed the same span, so every hook is told. + // The dispatcher itself fell behind the bus. Best-effort + // means those events are simply gone: the counter and this + // line are the whole record. tracing::warn!(target: "alert", dropped = n, "alert dispatcher lagged the event bus"); for hook in &hooks { - hook.gap.dropped.fetch_add(n, Ordering::Relaxed); hook.counters.dropped.fetch_add(n, Ordering::Relaxed); } } @@ -410,46 +314,34 @@ async fn fan_in( tracing::info!(target: "alert", "alert webhook dispatcher stopped"); } -/// Queue an event for one hook, converting an overflow into a recorded gap. +/// Queue an event for one hook. /// -/// Returns whether the event was queued. Callers that need a count must use -/// this rather than diffing `counters.dropped`: those counters are process-lived -/// and shared across generations, so a retired generation still shedding events -/// perturbs the delta — which, on a subtraction, underflowed. -fn enqueue( - hook: &HookChannel, - publisher: &EventPublisher, - item: Delivery, - cursor: Option, -) -> bool { - // Emit the pending gap notice ahead of the event that follows it, so the - // receiver learns about the hole before it sees the data after it. - flush_gap(hook, publisher); +/// Returns whether the event was queued. An overflow drops the event and moves +/// `satd_alertwebhook_dropped_total`; nothing is held for later and nothing is +/// synthesized into the stream. Webhooks are best-effort by design — a consumer +/// that needs to know what it missed wants the streaming API, which has real +/// cursors and backpressure. +fn enqueue(hook: &HookChannel, item: Delivery) -> bool { match hook.tx.try_send(item) { Ok(()) => { hook.counters .queue_depth .store(queue_depth(&hook.tx) as u64, Ordering::Relaxed); - if let Some(c) = cursor { - hook.gap.resume_height.store(c.height, Ordering::Relaxed); - } true } Err(mpsc::error::TrySendError::Full(_)) => { - // A receiver that cannot keep up degrades to "you missed N" rather - // than back-pressuring the bus. - hook.gap.dropped.fetch_add(1, Ordering::Relaxed); + // A receiver that cannot keep up loses events rather than + // back-pressuring the bus. Consensus is never the thing that waits. hook.counters.dropped.fetch_add(1, Ordering::Relaxed); false } Err(mpsc::error::TrySendError::Closed(_)) => { // The delivery task is gone: it failed to build its HTTP client, or // it panicked. Silently ignoring this makes the hook look perfectly - // healthy on `/metrics` — `dropped_total` flat, `queue_depth` flat, - // no `Lagged` ever synthesized — while it delivers nothing at all, - // forever. Count it like any other loss so the existing - // "no successful delivery in N minutes" rule fires. - hook.gap.dropped.fetch_add(1, Ordering::Relaxed); + // healthy on `/metrics` — `dropped_total` flat, `queue_depth` flat + // — while it delivers nothing at all, forever. Count it like any + // other loss so the existing "no successful delivery in N minutes" + // rule fires. hook.counters.dropped.fetch_add(1, Ordering::Relaxed); // Logged once per hook, not once per event. This arm is reached on // every event once a delivery task is gone — including for a @@ -468,40 +360,6 @@ fn enqueue( } } -/// If events were dropped for this hook, queue a `Lagged` notice describing the -/// gap. Best-effort: if the queue is still full the notice waits for the next -/// opportunity, and the drop count keeps accumulating in the meantime. -fn flush_gap(hook: &HookChannel, publisher: &EventPublisher) { - let dropped = hook.gap.dropped.swap(0, Ordering::Relaxed); - if dropped == 0 { - return; - } - let resume = publisher.resume_cursor(hook.gap.resume_height.load(Ordering::Relaxed), 0); - let env = node::events::lagged_event(publisher, dropped, resume); - let Ok(bytes) = serde_json::to_vec(&env) else { - return; - }; - let item = Delivery::Event { - body: Arc::new(bytes), - // Synthesized envelope: `stamp.seq` is 0 for every one of these, so it - // gets an id from the replay space instead. Without this every gap - // notice in the process would carry the same idempotency key and a - // conforming receiver would discard all but the first — "a gap is never - // silent" would hold exactly once. - delivery_id: satd_alert::replay_delivery_id( - &hex::encode(env.stamp.node_id), - publisher.instance_id(), - SYNTH_DELIVERY_SEQ.fetch_add(1, Ordering::Relaxed).wrapping_add(1), - ), - cursor: None, - gap_weight: dropped, - }; - if hook.tx.try_send(item).is_err() { - // Still full — put the count back so the notice is not lost. - hook.gap.dropped.fetch_add(dropped, Ordering::Relaxed); - } -} - fn queue_depth(tx: &mpsc::Sender) -> usize { HOOK_QUEUE_CAPACITY.saturating_sub(tx.capacity()) } @@ -529,9 +387,10 @@ fn accepts(hook: &Hook, env: &NodeEvent, last_heartbeat: &mut Option true, + // A `Lagged` body is built per-connection by the streaming carriers and + // is never published to the shared bus, so the dispatcher cannot see + // one. Treated as any other unsubscribed body rather than special-cased + // into a path that no event takes. other => { // Tweaks are refused at parse (never in a hook's mask), and the // remaining bodies map to their category bit. @@ -545,105 +404,6 @@ fn accepts(hook: &Hook, env: &NodeEvent, last_heartbeat: &mut Option, -) { - let Some(src) = block_source else { - return; - }; - let tip = src.current_tip_height(); - let Some(cursor) = read_cursor(store, &hook.hook) else { - // No stored marker: this hook is forward-only from now. Anchor the - // resume height at the live tip anyway. It is otherwise only set by a - // successful enqueue, so a hook that lags before its first one — or one - // that never carries a cursor at all, like a status-only or watch-only - // hook — would emit a `Lagged` advertising height 0. A receiver reading - // that as "resume from genesis" would resync the whole chain over - // having missed nothing. - hook.gap.resume_height.store(tip, Ordering::Relaxed); - return; - }; - hook.gap.resume_height.store(cursor.height, Ordering::Relaxed); - let missed = u64::from(tip.saturating_sub(cursor.height)); - if missed == 0 { - return; - } - hook.gap.dropped.fetch_add(missed, Ordering::Relaxed); - tracing::info!( - target: "alert", - hook = %hook.id, - from = cursor.height, - tip, - missed, - "webhook missed a span while the daemon was down; announcing it as a gap", - ); - // Advance the marker past the announced span. Without this a restart before - // the next delivery re-announces the same gap, and on a quiet chain that - // repeats on every restart. - let advanced = Cursor { height: tip, ..cursor }; - if let Err(e) = store.write_alert_cursor(&hook.hook.cursor_key(), &encode_cursor(&advanced)) { - tracing::warn!( - target: "alert", - hook = %hook.id, - error = %e, - "failed to advance the webhook resume marker past an announced gap", - ); - } -} - -fn read_cursor(store: &dyn Store, hook: &Hook) -> Option { - let raw = store.read_alert_cursor(&hook.cursor_key())?; - decode_cursor(&raw) -} - -/// Cursors are stored as a fixed 24-byte little-endian record rather than JSON: -/// the format is written and read only here, and a fixed layout cannot acquire -/// a parse failure mode as the `Cursor` type grows fields. -fn encode_cursor(c: &Cursor) -> [u8; 24] { - let mut out = [0u8; 24]; - out[0..4].copy_from_slice(&c.height.to_le_bytes()); - out[4..8].copy_from_slice(&c.tx_index.to_le_bytes()); - out[8..16].copy_from_slice(&c.mempool_seq.to_le_bytes()); - out[16..24].copy_from_slice(&c.instance_id.to_le_bytes()); - out -} - -fn decode_cursor(raw: &[u8]) -> Option { - if raw.len() != 24 { - return None; - } - Some(Cursor { - height: u32::from_le_bytes(raw[0..4].try_into().ok()?), - tx_index: u32::from_le_bytes(raw[4..8].try_into().ok()?), - mempool_seq: u64::from_le_bytes(raw[8..16].try_into().ok()?), - instance_id: u64::from_le_bytes(raw[16..24].try_into().ok()?), - }) -} - /// The HTTP client every webhook delivery goes through. /// /// Redirects are **not** followed. The alertfile URL is validated once, at load @@ -664,68 +424,11 @@ fn webhook_client() -> reqwest::Result { .build() } -/// Persist a hook's resume position, at most once per block height. -/// -/// The cursor is a resume hint, not a ledger: one RocksDB write per delivered -/// event would be pure write amplification for a value only read at startup. -fn persist_cursor( - hook: &Hook, - store: &dyn Store, - cursor: Option, - persisted_height: &mut Option, -) { - let Some(c) = cursor else { return }; - if *persisted_height == Some(c.height) { - return; - } - // Never move a hook's durable cursor backwards. - // - // `persisted_height` is this task's own view and starts empty for a fresh - // generation, so it cannot see what a *retired* generation is still doing. - // A reload spawns the new generation before signalling the old one to stop, - // and an in-flight POST is not inside the stop `select!` — so a delivery - // retired mid-request can land up to `REQUEST_TIMEOUT` later and write a - // cursor the new generation has already advanced past. Rewinding it means - // the next restart replays a span the receiver already acked. - if let Some(existing) = store - .read_alert_cursor(&hook.cursor_key()) - .and_then(|raw| decode_cursor(&raw)) - && existing.height > c.height - { - *persisted_height = Some(existing.height); - return; - } - match store.write_alert_cursor(&hook.cursor_key(), &encode_cursor(&c)) { - Ok(()) => *persisted_height = Some(c.height), - Err(e) => { - tracing::warn!(target: "alert", hook = %hook.id, error = %e, "failed to persist webhook cursor") - } - } -} - -/// Hand back the gap weight of every delivery this loop will never send. -/// -/// Retiring a generation drops its queue, and `flush_gap` has already zeroed -/// `GapState` on the assumption the queued notice would go out. Without this, -/// a reload silently destroys the record of a hole — and the process-lived -/// `GapState` that exists precisely to survive a reload reads zero afterwards. -fn drain_owed(rx: &mut mpsc::Receiver, gap: &GapState) { - let mut owed = 0u64; - while let Ok(Delivery::Event { gap_weight, .. }) = rx.try_recv() { - owed = owed.saturating_add(gap_weight); - } - if owed > 0 { - gap.dropped.fetch_add(owed, Ordering::Relaxed); - } -} - /// One hook's delivery loop: serial, in-order, retrying with backoff. async fn deliver_loop( hook: Hook, mut rx: mpsc::Receiver, counters: Arc, - gap: Arc, - store: Arc, mut stop: Stop, ) { let client = match webhook_client() { @@ -735,25 +438,16 @@ async fn deliver_loop( return; } }; - // Only persist a cursor when it actually moves forward a block: the cursor - // is a resume hint, not a ledger, and one RocksDB write per delivered event - // would be pure write amplification. - let mut persisted_height: Option = None; loop { let item = tokio::select! { - _ = stop.wait() => return drain_owed(&mut rx, &gap), + _ = stop.wait() => return, item = rx.recv() => match item { Some(i) => i, - None => return drain_owed(&mut rx, &gap), + None => return, }, }; - let Delivery::Event { - body, - delivery_id, - cursor, - gap_weight, - } = item; + let Delivery::Event { body, delivery_id } = item; counters .queue_depth @@ -795,7 +489,6 @@ async fn deliver_loop( counters .last_success_unix .store(unix_secs(), Ordering::Relaxed); - persist_cursor(&hook, store.as_ref(), cursor, &mut persisted_height); break; } satd_alert::Disposition::Drop => { @@ -808,36 +501,6 @@ async fn deliver_loop( delivery = %delivery_id, "webhook receiver rejected the delivery permanently; skipping this event", ); - // Record it as a gap, so the receiver is told in-band. - // - // This is the one drop path that did not, and it is the one - // most likely to fire on a routine misconfiguration: a 3xx - // (an `http://` URL behind a proxy that redirects to HTTPS) - // and a 401 (a rotated secret, or a delivery that aged past - // the receiver's freshness window while being retried) are - // both permanent by this classification. Without a `lagged` - // body the receiver has no way to learn which events it - // lost, and — because the cursor advances below — no way to - // recover them after the misconfiguration is fixed. - // - // `gap_weight` is what this delivery was itself announcing: - // rejecting a `Lagged` that carried 500 must not shrink the - // count to 1. Under a receiver that rejects every request — - // a proxy 302ing the lot — a collapsing count would grind - // gap accounting down to 1 on every cycle indefinitely. - gap.dropped - .fetch_add(gap_weight.saturating_add(1), Ordering::Relaxed); - // Advance the cursor anyway. A permanent rejection is the - // receiver's decision about this event, and it will decide - // the same way next time — leaving the cursor parked would - // make every restart rebuild and re-queue the same span - // against an endpoint that has already refused it, forever. - // The event is lost either way; the difference is whether - // the hook makes progress past it. The drop is counted - // (`satd_alertwebhook_dropped_total`), logged, and now - // announced to the receiver, so the loss is visible on both - // ends rather than silent. - persist_cursor(&hook, store.as_ref(), cursor, &mut persisted_height); break; } satd_alert::Disposition::Retry => { @@ -852,15 +515,7 @@ async fn deliver_loop( ); tokio::select! { _ = tokio::time::sleep(delay) => {} - // Retired mid-backoff. This delivery is already out of - // the queue, so `drain_owed` cannot see it — hand its - // weight back explicitly before dropping it. - _ = stop.wait() => { - if gap_weight > 0 { - gap.dropped.fetch_add(gap_weight, Ordering::Relaxed); - } - return drain_owed(&mut rx, &gap); - } + _ = stop.wait() => return, } } } @@ -890,7 +545,6 @@ pub struct AlertReloader { /// explicitly rather than inherited from the caller. api_handle: tokio::runtime::Handle, publisher: Arc, - store: Arc, block_source: Option>, metrics: Arc, global_stop: watch::Receiver, @@ -921,7 +575,6 @@ impl AlertReloader { path: std::path::PathBuf, api_handle: tokio::runtime::Handle, publisher: Arc, - store: Arc, block_source: Option>, metrics: Arc, global_stop: watch::Receiver, @@ -930,7 +583,6 @@ impl AlertReloader { path, api_handle, publisher, - store, block_source, metrics, global_stop, @@ -1030,24 +682,15 @@ impl AlertReloader { } None => None, }; - let fresh = kept.is_none(); let running_hook = kept.unwrap_or_else(|| { - start_hook( - hook, - &self.metrics, - &DISPATCHER_STATE, - self.store.clone(), - self.global_stop.clone(), - ) + start_hook(hook, &self.metrics, self.global_stop.clone()) }); channels.push(HookChannel { id: hook.id.clone(), hook: hook.clone(), tx: running_hook.tx.clone(), counters: running_hook.counters.clone(), - gap: running_hook.gap.clone(), reported_closed: std::sync::atomic::AtomicBool::new(false), - fresh, }); next_hooks.insert(hook.id.clone(), running_hook); } @@ -1076,7 +719,6 @@ impl AlertReloader { channels, bus_rx, self.publisher.clone(), - self.store.clone(), self.block_source.clone(), Arc::clone(&DISPATCHER_STATE), Stop { @@ -1101,58 +743,6 @@ impl AlertReloader { keep.push(LEGACY_REORG_HOOK_ID.to_string()); self.metrics.retain(&keep); - // Reclaim gap state for the same set. `metrics.retain` and the cursor - // GC below both prune on hook removal; the process-lived gap map has to - // prune with them, or a re-added id inherits its predecessor's pending - // drop count and resume anchor and announces a hole its new endpoint - // never had. - DISPATCHER_STATE.retain_gaps(&keep.iter().cloned().collect()); - - // Drop the durable cursor of any hook this reload removed. Hook ids are - // short and reused — `pager`, `alerts`, `ops` — so leaving the key - // behind means a later hook that happens to reuse the id inherits a - // stale resume position and greets a brand-new endpoint with the whole - // replay window of its predecessor's history. - // - // Reconciled against what is actually stored rather than against the - // previous generation. A reload-time diff cannot see the ordinary case: - // an operator who removes a hook by editing the file and *restarting* - // leaves no previous generation to compare against, so the cursor - // survived forever. Enumerating the keyspace also cleans up whatever - // earlier versions left behind. - let live: std::collections::HashSet> = file - .hooks - .iter() - .map(|h| h.cursor_key()) - .chain(std::iter::once( - format!("alertwebhook.cursor.{LEGACY_REORG_HOOK_ID}").into_bytes(), - )) - .collect(); - match self.store.list_alert_cursor_keys() { - Ok(stored) => { - for key in stored.into_iter().filter(|k| !live.contains(k)) { - let label = String::from_utf8_lossy(&key).to_string(); - match self.store.delete_alert_cursor(&key) { - Ok(()) => tracing::info!( - target: "alert", - key = %label, - "no hook owns this resume cursor; dropped it", - ), - Err(e) => tracing::warn!( - target: "alert", - key = %label, - error = %e, - "failed to drop an orphaned resume cursor", - ), - } - } - } - Err(e) => tracing::warn!( - target: "alert", - error = %e, - "could not enumerate stored resume cursors; orphans were not reclaimed", - ), - } *self.last_applied.lock() = Some(file); Ok(hook_count) } @@ -1292,26 +882,6 @@ categories = ["status"] min_severity = "warning" "#; - #[test] - fn cursor_round_trips_through_its_fixed_encoding() { - let c = Cursor { - height: 812_345, - tx_index: 7, - mempool_seq: 0xDEAD_BEEF_CAFE, - instance_id: 0x0102_0304_0506_0708, - }; - assert_eq!(decode_cursor(&encode_cursor(&c)), Some(c)); - } - - #[test] - fn a_truncated_or_garbage_cursor_is_ignored_not_misread() { - // A corrupt cursor must degrade to "start at the live head", never to a - // wrong height that would replay or skip history. - assert_eq!(decode_cursor(&[]), None); - assert_eq!(decode_cursor(&[0u8; 23]), None); - assert_eq!(decode_cursor(&[0u8; 25]), None); - } - #[test] fn status_filter_is_applied_per_hook() { let hook = hook_from(STATUS_HOOK); @@ -1351,34 +921,6 @@ min_severity = "warning" assert!(!accepts(&hook, &env, &mut hb)); } - #[test] - fn lag_notices_bypass_every_filter() { - // A receiver must learn it missed events even if the events it missed - // were in a category it does not subscribe to — otherwise its cursor - // silently diverges. - let hook = hook_from(STATUS_HOOK); - let env = NodeEvent::new( - node::events::EdgeStamp { - node_id: [7; 16], - region: None, - edge_seen_at_ns: 0, - edge_wall_ns: 0, - seq: 1, - }, - NodeEventBody::Lagged { - dropped_count: 3, - resume_cursor: Cursor { - height: 1, - tx_index: 0, - mempool_seq: 0, - instance_id: 1, - }, - }, - ); - let mut hb = None; - assert!(accepts(&hook, &env, &mut hb)); - } - #[test] fn heartbeats_are_downsampled_to_the_configured_interval() { let hook = hook_from( @@ -1450,7 +992,6 @@ heartbeat_interval_secs = 3600 dir.join("alertfile.toml"), tokio::runtime::Handle::current(), publisher.clone(), - Arc::new(node::storage::db::InMemoryStore::new()), None, Arc::new(WebhookMetrics::new()), global_stop, diff --git a/satd/src/main.rs b/satd/src/main.rs index 976ebf8e5..fe3b62f29 100644 --- a/satd/src/main.rs +++ b/satd/src/main.rs @@ -2195,9 +2195,8 @@ async fn main() { path, api_handle.clone(), event_publisher.clone(), - chain_state.store_ref().clone() as std::sync::Arc, - // Block source for a hook's startup catch-up replay: read-only - // active-chain access, the same handle the streaming carriers use. + // Read-only active-chain access, used only to answer "is this node + // still in initial block download" for the firehose gate. Some(chain_state.clone() as std::sync::Arc), webhook_metrics.clone(), shutdown_rx.clone(), diff --git a/satd/tests/e2e/webhooks.rs b/satd/tests/e2e/webhooks.rs index 8152547c6..709a696bf 100644 --- a/satd/tests/e2e/webhooks.rs +++ b/satd/tests/e2e/webhooks.rs @@ -80,15 +80,11 @@ enum Behavior { /// Answer every request with `302 Location: `. Redirect(String), /// Answer with `code` for any `block_connected` at one of these heights, - /// 200 for everything else. Unlike `RejectHeights` the status is the - /// caller's, so a 5xx leaves the delivery in retry and the hook's resume - /// marker parked — which is what an endpoint outage looks like from the - /// node's side. + /// 200 for everything else. Keyed on content rather than request ordinal so + /// a script does not depend on how many events a block produces; a 5xx + /// leaves the delivery in retry, which is what an endpoint outage looks + /// like from the node's side. FailHeights(Vec, u16), - /// Permanently reject (404) any `block_connected` at one of these heights, - /// accept everything else. Keyed on content rather than request ordinal so - /// the script does not depend on how many events a block produces. - RejectHeights(Vec), } struct Inner { @@ -141,13 +137,6 @@ impl MockReceiver { _ => (200, None), } } - Behavior::RejectHeights(hs) => { - let h = req.json()["body"]["height"].as_u64(); - match h { - Some(h) if hs.contains(&h) => (404, None), - _ => (200, None), - } - } }; // Only a request the receiver actually accepted counts // as received; a 503'd attempt is a retry, not a @@ -196,23 +185,6 @@ impl MockReceiver { Self::start(Behavior::Redirect(to)).await } - /// Permanently reject `block_connected` at these heights; accept the rest. - pub async fn rejecting_heights(heights: Vec) -> Self { - Self::start(Behavior::RejectHeights(heights)).await - } - - /// Heights of every accepted `block_connected` delivery, in arrival order. - async fn accepted_heights(&self) -> Vec { - self.inner - .lock() - .await - .received - .iter() - .filter(|r| r.json()["body"]["kind"] == "block_connected") - .filter_map(|r| r.json()["body"]["height"].as_u64()) - .collect() - } - pub fn url(&self) -> String { format!("http://127.0.0.1:{}/hook", self.port) } @@ -652,105 +624,6 @@ async fn editing_one_hook_does_not_destroy_another_hooks_queued_event() { assert!(got.signature_valid(SECRET)); } -/// A permanently-rejected event still advances the hook's resume cursor. -/// -/// Otherwise a receiver that hard-rejects one body shape turns every reload and -/// every restart into a rebuild of the same refused span — the events are lost -/// either way, and the only question is whether the hook makes progress past -/// them. Here blocks 2 and 3 are 404'd; the reload that follows re-runs catch-up -/// from the stored cursor, and must not replay them. -#[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn webhook_permanent_rejection_advances_the_cursor() { - let receiver = MockReceiver::rejecting_heights(vec![2, 3]).await; - let (sn, _dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; - - crate::streaming::mine_n(&sn, 3).await; - // Block 1 is accepted; 2 and 3 are refused. Wait for the last one to have - // been attempted, so the cursor writes have happened. - receiver - .wait_for(60, |r| r.json()["body"]["height"] == 1) - .await; - tokio::time::sleep(Duration::from_secs(2)).await; - let block_attempts = |rs: Vec| -> Vec { - rs.iter() - .filter(|r| r.json()["body"]["kind"] == "block_connected") - .filter_map(|r| r.json()["body"]["height"].as_u64()) - .collect() - }; - let before = block_attempts(receiver.all_attempts().await); - assert_eq!(before, vec![1, 2, 3], "expected all three blocks attempted once"); - - // A SIGHUP retires the generation and starts a new one, which re-runs - // catch-up from the persisted cursor. - // - // The alertfile must actually change: an unchanged one is deliberately a - // no-op, because retiring a generation destroys its queued deliveries and a - // status event has no replay to recover it. Without a real edit here this - // test would pass without ever re-running catch-up — the exact thing it - // exists to check. - write_alertfile(&_dir, &receiver, "chain", "\"chain\"", true); - crate::streaming::sighup_with_conf(&sn, "").await; - tokio::time::sleep(Duration::from_secs(3)).await; - - // Counting *attempts*, not accepted deliveries: a replayed block would be - // refused again and so would never show up as accepted. What matters is - // whether the request was made at all. - // - // Scoped to `block_connected` rather than a raw request count, because a - // permanent rejection now also emits a `lagged` notice — the receiver is - // told which events it lost, since the cursor advancing past them means it - // can never fetch them later. A raw count would fold that in and read as a - // replayed block. - assert_eq!( - block_attempts(receiver.all_attempts().await), - before, - "a refused span must not be rebuilt and re-sent on reload", - ); - let heights = receiver.accepted_heights().await; - assert_eq!(heights, vec![1], "only block 1 was ever accepted; got {heights:?}"); - - // The hook is not wedged: the next block still arrives. - crate::streaming::mine_n(&sn, 1).await; - receiver - .wait_for(60, |r| r.json()["body"]["height"] == 4) - .await; - - // ...and the refusal was announced rather than silent. A gap notice is - // emitted ahead of the next event for the hook, so this is asserted after - // block 4 rather than immediately after the rejections. - // - // It matters here more than on any other drop path: the cursor advances - // past a permanently-rejected event, so unlike a queue overflow the - // receiver cannot go back for it. Being told is all it gets. - // Summed across notices rather than read off the first one. The gap - // accounting also flushes on a timer, so whether the two refusals coalesce - // into a single notice of 2 or arrive as two notices of 1 depends only on - // which side of a tick they land on — a race this test has no reason to - // pin down. Only the total is meaningful. - let deadline = std::time::Instant::now() + Duration::from_secs(30); - loop { - let attempts = receiver.all_attempts().await; - let lagged: Vec<_> = attempts - .iter() - .filter(|r| r.json()["body"]["category"] == "lagged") - .collect(); - let total: u64 = lagged - .iter() - .filter_map(|r| r.json()["body"]["dropped_count"].as_u64()) - .sum(); - if total >= 2 { - break; - } - assert!( - std::time::Instant::now() < deadline, - "both refused blocks should be counted; got {total} across {} notice(s): {:?}", - lagged.len(), - lagged.iter().map(|r| r.body.clone()).collect::>(), - ); - tokio::time::sleep(Duration::from_millis(200)).await; - } -} - /// A redirect is never followed: the signed body does not go to a host the /// alertfile never named. /// @@ -799,71 +672,60 @@ async fn webhook_permanent_rejection_does_not_wedge_the_queue() { assert!(receiver.attempts().await >= 2); } -/// A span missed while the hook was dead is announced, not replayed. +/// A hook that was not running does not go back for what it missed. /// -/// Webhooks are realtime (design D6): the durable cursor is a resume marker, -/// not a replay log. On restart the dispatcher compares it against the tip and -/// emits one `Lagged` naming what was missed; the events themselves are gone, -/// and a receiver that needs them fetches them over the streaming API from the -/// advertised `resume_cursor`. +/// This is the whole durability contract, asserted from the outside. Webhooks +/// are best-effort (design D6): there is no cursor, no replay and no gap +/// notice. A hook that comes back resumes at the live head, and the events that +/// happened while it was down are simply not delivered. /// -/// This replaces a test of the old catch-up replay. That replay was built up to -/// 10,000 blocks into a queue holding 1,024, so any outage long enough to -/// matter had its "guaranteed" recovery converted straight back into an -/// overflow gap — and its per-height delivery ids collided between a block and -/// its post-reorg replacement, so a conforming receiver dropped the -/// replacement. +/// The assertion is deliberately that the node does *not* do something. Earlier +/// revisions delivered the missed span, then announced it — each of which cost +/// a durable keyspace, a `Store` trait extension across four backends, a GC +/// pass and a synthesized delivery-id space, to serve a use case the streaming +/// API already serves properly with real cursors and backpressure. #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn a_missed_span_is_announced_as_a_gap_and_never_replayed() { - // 503 blocks 2..=5 so they stay in retry and never advance the marker. - // Everything else — block 1, and the gap notice itself — is accepted. +async fn a_hook_that_was_down_resumes_at_the_live_head() { + // 503 blocks 2..=5 so they stay queued and un-acked when the hook is + // retired. Everything else is accepted. let receiver = MockReceiver::failing_heights(vec![2, 3, 4, 5], 503).await; let (sn, dir) = node_with_hook(&receiver, "chain", "\"chain\"", vec![]).await; - // Establish a marker: one block delivered and acked. crate::streaming::mine_n(&sn, 1).await; receiver .wait_for(60, |r| r.json()["body"]["height"] == 1) .await; - tokio::time::sleep(Duration::from_secs(2)).await; - // The outage. These four are queued and retrying; none is acked, so the - // marker stays at 1 while the tip moves to 5. + // The outage: four blocks queued and retrying, none acked. crate::streaming::mine_n(&sn, 4).await; tokio::time::sleep(Duration::from_secs(3)).await; let missed = |r: &Received| (2..=5).contains(&r.json()["body"]["height"].as_u64().unwrap_or(0)); - let attempts_before = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); - assert!(attempts_before > 0, "the outage span should have been attempted at least once"); + let before = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); + assert!(before > 0, "the outage span should have been attempted at least once"); // Retire the generation with a same-URL alertfile edit — the same code path - // a restart takes. The queued, still-retrying blocks go with it, which is - // exactly why the receiver has to be told. + // a restart takes. The queued blocks go with it. write_alertfile(&dir, &receiver, "chain", "\"chain\"", true); crate::streaming::sighup_with_conf(&sn, "").await; - - let lagged = receiver - .wait_for(60, |r| r.json()["body"]["category"] == "lagged") - .await; - assert!( - lagged.json()["body"]["dropped_count"].as_u64().unwrap_or(0) >= 4, - "the whole missed span must be counted; got {}", - lagged.body, - ); - // The anchor points at or before the last block the receiver actually got, - // never past the hole it is announcing. - assert!( - lagged.json()["body"]["resume_cursor"]["height"].as_u64().unwrap_or(u64::MAX) <= 1, - "resume anchor is past the gap it announces; got {}", - lagged.body, - ); tokio::time::sleep(Duration::from_secs(3)).await; - // ...and nothing from the missed span was re-sent. - let attempts_after = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); + // Nothing from the missed span comes back... + let after = receiver.all_attempts().await.iter().filter(|r| missed(r)).count(); assert_eq!( - attempts_after, attempts_before, - "the missed span was replayed; a gap notice replaces it, it does not precede it" + after, before, + "the missed span was re-sent; a best-effort hook resumes at the head and \ + does not carry history across a reload" + ); + + // ...and no in-band notice is synthesized about it either. + assert!( + !receiver + .all_attempts() + .await + .iter() + .any(|r| r.json()["body"]["category"] == "lagged"), + "a lagged body was delivered; gap accounting was removed with the cursor" ); // The hook is live: the next block still arrives. From 6da95f63edd486e1ee2a67f2e2d0d2082ddb777d Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Thu, 30 Jul 2026 12:58:49 -0600 Subject: [PATCH 09/11] alerting: parse hook URLs at load; stop leaking secrets and credentials into logs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep-review fixes for the dispatcher and the `satd-alert` contract. Two reviewers found the URL bug independently. **A hook URL was validated by prefix match, never parsed — so an unresolvable URL was accepted at load and then retried forever, silently.** `validate_url` tested `starts_with("https://")`. `https://`, `https://:8080/h`, `https://alerts.example:99999/h`, `https://[::1`, and a template placeholder left as `https:///hook` all pass that and all fail the WHATWG parser `reqwest` actually resolves with. The daemon logged `alertfile loaded, hooks=1` and started clean; then `reqwest` failed in the builder with no HTTP status, `classify_response(None)` read that as *transient*, and the hook retried every five minutes for the life of the process — never delivering, never dropping, pinning the head of its serial queue until the 1024-slot queue overflowed and took every real alert with it. Neither documented health signal fires in that state: `dropped_total` stays 0 for months on a `status`-only hook, and `last_success_age_seconds` reports 0 ("fresh") for a hook that has *never* succeeded. `validate_url` now parses, and `deliver_loop` treats a builder error as `Drop` rather than retrying something that cannot ever work. **A malformed alertfile printed hook secrets to the log.** `toml_edit`'s error `Display` quotes the offending source line verbatim, and the line most likely to be mid-edit when TOML syntax breaks is `secret = "..."` — an operator rotating a key drops the closing quote. Both callers log this with `error = %e`, so the plaintext HMAC signing key landed in `debug.log` and anywhere that log ships. The crate hand-writes `Debug` for `Hook` precisely to keep that key out of logs; this defeated it from the other direction. The error now carries message and line number only. Control-verified by `a_toml_syntax_error_never_echoes_the_secret`. **A webhook URL is frequently the credential itself** (Slack, Discord, PagerDuty), and reqwest appends the full URL — userinfo included — to transport and timeout errors. One refused connection wrote it to the log. Now logged through `without_url()`. **An event retried past the published freshness window is futile by construction.** `signed_at` is fixed per event and `MAX_TIMESTAMP_SKEW_SECS` is 600, but the doubling curve reaches 600 s of cumulative age around attempt 10 and then retries at the 300 s cap indefinitely — against a conforming receiver that is required to refuse every one of them. Worse, a gateway answering 503 rather than 4xx keeps it classified transient, so it never reaches `Drop` and pins the queue permanently. The dispatcher now abandons an event once it ages out. **`reorg-legacy` was reserved in prose only.** An alertfile hook could take the id, be handed the *same* `Arc` as the built-in `reorgwebhook=` dispatcher (metrics keys on the id string), and send `X-Satd-Hook: reorg-legacy` under a different secret and a different contract version — so a receiver keying its secret lookup off that header verifies with the wrong key. The parser now refuses it, and a test in `satd` pins the two crates' constants together. Also: `allow_insecure_http` was the one field that accepted-and-ignored a wrong-typed value (`"true"` read as `false`, then rejecting the very hook the operator opted in for); hook ids are capped at 64 chars since each becomes an HTTP header value and a Prometheus label on every delivery; deleted `replay_delivery_id`, dead since the durability machinery was removed, along with three stale doc blocks that described a `GapState`, a `Lagged` notice and a "resume position" that no longer exist. Fixed the manual's claim that IBD suppresses everything but `status`/`heartbeat` — only `chain` is suppressed, so a `mempool` hook keeps firing throughout a multi-day sync. `unknown_keys_are_rejected_not_ignored` was also not testing what it named: appending a key to the fixture puts it *inside* `[[webhook]]`, so both assertions exercised the per-hook list and the top-level check had zero coverage. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH --- Cargo.lock | 1 + docs/manual/src/observability.md | 17 +- satd-alert/Cargo.toml | 4 + satd-alert/src/config.rs | 296 ++++++++++++++++++++++--------- satd-alert/src/contract.rs | 60 ++----- satd-alert/src/lib.rs | 7 +- satd-alert/src/retry.rs | 2 +- satd/src/alert.rs | 76 +++++++- 8 files changed, 313 insertions(+), 150 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 64fef9dde..1a92c50a0 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4115,6 +4115,7 @@ dependencies = [ "thiserror 2.0.18", "toml_edit", "tracing", + "url", ] [[package]] diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index f84d84c84..47faa6b89 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -155,7 +155,7 @@ each matching event to your endpoint. version = 1 [[webhook]] -id = "pager" # unique; appears in headers, metrics, and the cursor key +id = "pager" # unique; appears in the X-Satd-Hook header and metric labels url = "https://alerts.example/satd" secret = "a-long-random-string" # required — it signs every delivery categories = ["status"] # status | chain | mempool | heartbeat @@ -268,11 +268,13 @@ the only duplicate you can receive is a retry of something you already saw. backpressure, and a bounded `RescanBlocks`. Webhooks are for automation you are happy to miss occasionally: page me, poke a script, ping a dead-man's switch. -- **Suppressed during initial block download.** A node syncing from scratch does - not POST its entire history: while it is catching up, only `status` and - `heartbeat` events are delivered. Health alerts stay live because "this node is - unhealthy" is exactly as true mid-sync, and the heartbeat keeps flowing so an - external dead-man's switch does not declare a syncing node dead. What was +- **`chain` alerts are suppressed during initial block download.** A node + syncing from scratch does not POST its entire block history. `status`, + `heartbeat` **and `mempool`** keep flowing: health alerts stay live because + "this node is unhealthy" is exactly as true mid-sync, and the heartbeat keeps + flowing so an external dead-man's switch does not declare a syncing node dead. + Note the consequence for `mempool` hooks — a mainnet mempool subscription is + thousands of events a minute, and a multi-day sync does not quiet it. What was suppressed is counted in `satd_alertwebhook_dropped_total`. The suppression is latched on leaving IBD once, so a node whose tip later goes stale keeps alerting — which is the whole point of a stalled-tip alert. @@ -281,7 +283,8 @@ Plaintext `http://` is accepted for loopback and private-network targets. For a public host, use `https://` or set `allow_insecure_http = true` on the hook. Per-hook counters are exported: `satd_alertwebhook_delivered_total`, -`_failed_attempts_total`, `_dropped_total` (the dead-letter count), +`_failed_attempts_total`, `_dropped_total` (events lost, not held — +there is no dead-letter queue), `_queue_depth`, and `_last_success_age_seconds`, all labelled `hook=""`. Nothing is exported when no hook is configured. diff --git a/satd-alert/Cargo.toml b/satd-alert/Cargo.toml index f03cfda17..34efcde93 100644 --- a/satd-alert/Cargo.toml +++ b/satd-alert/Cargo.toml @@ -27,6 +27,10 @@ tracing = { workspace = true } # Parse-only, like satd-auth: the `display` default feature pulls an # unpublished toml_writer and nothing here round-trips the file. toml_edit = { version = "0.25", default-features = false, features = ["parse"] } +# The same WHATWG parser `reqwest` resolves a hook URL with. Validating against +# it rather than against a prefix check is what makes "the alertfile loaded" and +# "this hook can actually deliver" the same statement. +url = "2" [dev-dependencies] tempfile = "3" diff --git a/satd-alert/src/config.rs b/satd-alert/src/config.rs index 31169b579..c8581461d 100644 --- a/satd-alert/src/config.rs +++ b/satd-alert/src/config.rs @@ -13,8 +13,8 @@ //! heartbeat_interval_secs = 60 # optional dead-man ping; default off //! ``` //! -//! Why a file rather than flat `bitcoin.conf` keys: a hook has a secret, a -//! filter, and (from the watch-set work) a watch-set. Core's config format is +//! Why a file rather than flat `bitcoin.conf` keys: a hook has a secret and a +//! filter. Core's config format is //! flat and first-wins, so several hooks cannot be expressed in it without //! inventing an index syntax. The precedent is `authfile=`, which is the same //! shape for the same reason — down to the 0600 permission check, since both @@ -31,12 +31,33 @@ use std::path::{Path, PathBuf}; use node::events::{StatusKind, StatusSeverity}; /// Per-hook outbound queue depth. Delivery is serial per hook, so this is how -/// far a hook may fall behind before events are dropped and the gap is reported -/// to the receiver as a `Lagged` notice. 1024 is the same order as the -/// publisher's own replay ring: enough to ride out a receiver restart, small -/// enough that a permanently dead endpoint cannot grow memory without bound. +/// far a hook may fall behind before events are dropped. Nothing is announced +/// to the receiver when that happens — webhooks are best-effort, and the loss +/// is visible only on `satd_alertwebhook_dropped_total`. 1024 is the same order +/// as the publisher's own replay ring: enough to ride out a receiver restart, +/// small enough that a permanently dead endpoint cannot grow memory without +/// bound. pub const HOOK_QUEUE_CAPACITY: usize = 1024; +/// The hook id the built-in `reorgwebhook=` dispatcher registers under. +/// +/// Reserved rather than merely documented as reserved: hook ids key the metrics +/// registry, so an alertfile hook sharing this id would be handed the *same* +/// `HookCounters` as the legacy dispatcher and silently sum two unrelated +/// series. Both would also send `X-Satd-Hook: reorg-legacy` under different +/// secrets and different contract versions, so a receiver keying its secret +/// lookup off that header would verify with the wrong key. The crate already +/// hard-errors on duplicate ids for this exact reason; this is the one +/// collision the duplicate check cannot see. +/// +/// Kept in step with `satd::alert::LEGACY_REORG_HOOK_ID` by a test there — +/// `satd-alert` cannot depend on the binary crate. +pub const RESERVED_LEGACY_REORG_ID: &str = "reorg-legacy"; + +/// Cap on a hook id's length. It becomes an HTTP header value and a Prometheus +/// label on every single delivery, so an unbounded id is paid for forever. +pub const MAX_HOOK_ID_LEN: usize = 64; + /// The only alertfile schema version this build understands. pub const ALERTFILE_VERSION: u64 = 1; @@ -53,11 +74,23 @@ pub enum AlertFileError { secrets — run: chmod 600 {path}" )] Permissions { path: PathBuf, mode: u32 }, - #[error("alertfile {path}: not valid TOML: {source}")] + /// A TOML syntax error, rendered as **message and line only**. + /// + /// `toml_edit::TomlError`'s own `Display` quotes the offending source line + /// back verbatim. That is a good diagnostic for a config file and a bad one + /// for this file: the most likely line to be mid-edit when the syntax + /// breaks is `secret = "..."` — an operator rotating a key drops the + /// closing quote — and both callers log this with `error = %e` + /// (`satd/src/main.rs`, `satd/src/reload.rs`), which would put the + /// plaintext HMAC signing key in `debug.log` and anywhere that log is + /// shipped. The whole point of `Hook`'s hand-written `Debug` below is to + /// keep that key out of logs; rendering the snippet here would defeat it + /// from a different direction. + #[error("alertfile {path}: not valid TOML at line {line}: {message}")] Toml { path: PathBuf, - #[source] - source: toml_edit::TomlError, + line: usize, + message: String, }, #[error("alertfile {path}: {message}")] Invalid { path: PathBuf, message: String }, @@ -181,10 +214,16 @@ impl AlertFile { /// so the rules are testable without a file on disk. pub fn parse(path: &Path, text: &str) -> Result { let doc: toml_edit::DocumentMut = - text.parse().map_err(|source| AlertFileError::Toml { - path: path.to_path_buf(), - source, - })?; + text.parse() + .map_err(|source: toml_edit::TomlError| AlertFileError::Toml { + path: path.to_path_buf(), + // `span()` is a byte range into `text`; turn it into a line + // number without carrying any of the text along with it. + line: source.span().map_or(0, |s| { + text[..s.start.min(text.len())].bytes().filter(|b| *b == b'\n').count() + 1 + }), + message: source.message().to_string(), + })?; match doc.get("version").and_then(|v| v.as_integer()) { Some(v) if v as u64 == ALERTFILE_VERSION => {} @@ -224,9 +263,9 @@ impl AlertFile { } } - // Ids name a hook in delivery headers, in metrics labels, and in the - // metric label and header value, so a duplicate would conflate two hooks - // share one resume position. + // An id is a hook's identity in delivery headers and in metric labels, + // so two hooks sharing one would conflate two unrelated series and make + // `X-Satd-Hook` ambiguous at the receiver. let mut seen = BTreeSet::new(); for h in &hooks { if !seen.insert(h.id.clone()) { @@ -267,6 +306,25 @@ fn parse_hook(path: &Path, t: &toml_edit::Table) -> Result format!("webhook id `{id}` must be non-empty [A-Za-z0-9_-] (it is used in headers and metric labels)"), )); } + if id == RESERVED_LEGACY_REORG_ID { + return Err(AlertFileError::invalid( + path, + format!( + "webhook id `{id}` is reserved for the built-in `reorgwebhook=` dispatcher; \ + pick another id" + ), + )); + } + if id.len() > MAX_HOOK_ID_LEN { + return Err(AlertFileError::invalid( + path, + format!( + "webhook id is {} characters; the limit is {MAX_HOOK_ID_LEN} (it becomes an HTTP \ + header value and a Prometheus label on every delivery)", + id.len() + ), + )); + } let url = req_str(path, t, "url")?; let secret = req_str(path, t, "secret")?; if secret.is_empty() { @@ -276,10 +334,19 @@ fn parse_hook(path: &Path, t: &toml_edit::Table) -> Result )); } - let allow_insecure_http = t - .get("allow_insecure_http") - .and_then(|v| v.as_bool()) - .unwrap_or(false); + // Typed strictly, like every sibling field. `allow_insecure_http = "true"` + // is a plausible thing to write, and silently reading it as `false` would + // reject the very hook the operator just opted in for — with an error + // message about plaintext that makes no sense next to the line they added. + let allow_insecure_http = match t.get("allow_insecure_http") { + None => false, + Some(v) => v.as_bool().ok_or_else(|| { + AlertFileError::invalid( + path, + format!("webhook `{id}`: `allow_insecure_http` must be a bool (true or false)"), + ) + })?, + }; validate_url(path, &id, &url, allow_insecure_http)?; let categories = parse_categories(path, &id, t)?; @@ -440,76 +507,67 @@ fn parse_kinds( Ok(Some(out)) } +/// Validate a hook URL against the parser that will actually resolve it. +/// +/// This parses rather than prefix-matching, because the two disagree on strings +/// an operator really types. `https://alerts.example:99999/satd`, +/// `https://[::1`, and `https://` all pass a `starts_with("https://")` test and +/// all fail `Url::parse`. Accepting one is not a cosmetic slip: the dispatcher +/// only discovers it at send time, where `reqwest` reports a builder error with +/// no status, `classify_response(None)` reads that as *transient*, and the hook +/// retries a permanently-invalid URL every five minutes forever — filling its +/// queue and dropping every real alert behind it, on a node that logged +/// `alertfile loaded` at startup. An accepted-but-undeliverable hook is the +/// "accepted-but-ignored rule" this module's posture exists to prevent. fn validate_url( path: &Path, id: &str, url: &str, allow_insecure_http: bool, ) -> Result<(), AlertFileError> { - if url.starts_with("https://") { - return Ok(()); - } - let Some(rest) = url.strip_prefix("http://") else { - return Err(AlertFileError::invalid( + let parsed = url::Url::parse(url).map_err(|e| { + AlertFileError::invalid(path, format!("webhook `{id}`: url is not a valid URL: {e}")) + })?; + match parsed.scheme() { + "https" => Ok(()), + "http" => { + if allow_insecure_http || is_local_host(parsed.host()) { + Ok(()) + } else { + Err(AlertFileError::invalid( + path, + format!( + "webhook `{id}`: plaintext http:// to a non-local address; use https://, \ + or set allow_insecure_http = true to accept it" + ), + )) + } + } + other => Err(AlertFileError::invalid( path, - format!("webhook `{id}`: url must start with https:// or http://"), - )); - }; - if allow_insecure_http || is_local_target(rest) { - return Ok(()); - } - Err(AlertFileError::invalid( - path, - format!( - "webhook `{id}`: plaintext http:// to a non-local address; use https://, or set \ - allow_insecure_http = true to accept it" - ), - )) + format!("webhook `{id}`: url scheme must be https:// or http:// (got `{other}://`)"), + )), + } } -/// Whether an `http://` authority is loopback or RFC1918 — the cases where -/// plaintext is normal (a relay on the same host, a receiver inside a private -/// network) and demanding TLS would just push operators to set the override. -fn is_local_target(rest: &str) -> bool { - // Authority is everything before the path, query, or fragment. - // - // A backslash terminates the authority too. The WHATWG URL parser — which - // is what `reqwest` resolves this string with — treats `\` as a path - // separator for special schemes, so `http://evil.example\@127.0.0.1/hook` - // has host `evil.example`, while splitting on `/` alone leaves an authority - // of `evil.example\@127.0.0.1` whose last `@` yields `127.0.0.1`. That - // reads as loopback, waives the `allow_insecure_http` gate, and posts the - // signed body in cleartext to the attacker's host — the same bypass the - // userinfo rule below closes, through a different separator. - let authority = rest.split(['/', '\\', '?', '#']).next().unwrap_or(""); - // Strip userinfo — everything through the *last* `@`. - // - // Without this, `http://127.0.0.1:8332@evil.example/hook` reads as host - // `127.0.0.1`, is judged loopback, and is accepted with no - // `allow_insecure_http` acknowledgement — while the request actually goes - // in cleartext to `evil.example` with `127.0.0.1:8332` as userinfo. The - // operator is not the adversary here; the gate exists to make them - // consciously accept cleartext to a public host, and this skipped it. - let host_port = authority.rsplit_once('@').map_or(authority, |(_, h)| h); - // An IPv6 literal is bracketed, and its address contains colons — so the - // port must be split on the bracket, not on the last colon. - let host = if let Some(after) = host_port.strip_prefix('[') { - match after.split_once(']') { - Some((h, _)) => h, - None => return false, - } - } else { - host_port.split_once(':').map_or(host_port, |(h, _)| h) - }; - if host == "localhost" || host == "::1" { - return true; - } - let Ok(ip) = host.parse::() else { - return false; - }; - match ip { - std::net::IpAddr::V4(v4) => v4.is_loopback() || v4.is_private() || v4.is_link_local(), - std::net::IpAddr::V6(v6) => v6.is_loopback() || v6.is_unique_local(), +/// Whether a parsed host is loopback or RFC1918 — the cases where plaintext is +/// normal (a relay on the same host, a receiver inside a private network) and +/// demanding TLS would just push operators to set the override. +/// +/// Taking the host from `Url` rather than slicing the authority out by hand is +/// what closes the two smuggling tricks this check used to unpick manually. +/// `http://127.0.0.1:8332@evil.example/hook` has host `evil.example`, not +/// `127.0.0.1` — userinfo runs to the *last* `@`. And `\` terminates the +/// authority for a special scheme, so `http://evil.example\@127.0.0.1/hook` is +/// also `evil.example`. Both previously read as loopback, waiving the +/// `allow_insecure_http` gate and posting the signed body in cleartext to the +/// attacker's host. The parser gets both right by construction. +fn is_local_host(host: Option>) -> bool { + match host { + Some(url::Host::Domain(d)) => d == "localhost", + Some(url::Host::Ipv4(v4)) => v4.is_loopback() || v4.is_private() || v4.is_link_local(), + Some(url::Host::Ipv6(v6)) => v6.is_loopback() || v6.is_unique_local(), + None => false, } } @@ -654,10 +712,84 @@ categories = ["chain"] fn unknown_keys_are_rejected_not_ignored() { // An accepted-but-ignored alerting rule is worse than a refused one: // the operator believes they are covered when they are not. - assert!(parse(&format!("{MINIMAL}\nnot_a_key = 1\n")).is_err()); + // + // Both levels, placed deliberately. Appending to MINIMAL does *not* + // test the top-level check: TOML puts a trailing key inside the last + // open table, so `{MINIMAL}\nnot_a_key = 1` lands in `[[webhook]]` and + // exercises `parse_hook`'s KNOWN list twice over. The top-level loop + // needs a key written before the first `[[webhook]]` header. + let top_level = MINIMAL.replace("[[webhook]]", "not_a_key = 1\n[[webhook]]"); + let err = parse(&top_level).expect_err("a stray top-level key must be refused"); + assert!(err.to_string().contains("not_a_key"), "{err}"); assert!(parse(&MINIMAL.replace("categories = [\"status\"]", "categories = [\"status\"]\nkindz = [\"tip_stall\"]")).is_err()); } + /// A URL that satisfies the old `starts_with("https://")` check but that + /// `reqwest` cannot resolve. Accepting one is not cosmetic: the dispatcher + /// sees a builder error with no status, classifies it transient, and + /// retries forever without ever delivering — on a node whose startup log + /// says the alertfile loaded cleanly. + #[test] + fn a_well_prefixed_but_unparseable_url_is_refused_at_load() { + for bad in [ + "https://", // no host at all + "https://:8080/hook", // empty host with a port + "https://alerts.example:99999/hook", // port out of range + "https://[::1/hook", // unterminated IPv6 literal + "https:///hook", // template placeholder, left unedited + ] { + let f = MINIMAL.replace("https://alerts.example/satd", bad); + assert!( + parse(&f).is_err(), + "`{bad}` parses as an alertfile but cannot ever deliver", + ); + } + } + + #[test] + fn the_legacy_reorg_hook_id_is_reserved() { + // Sharing this id hands the alertfile hook the same metrics counters as + // the built-in `reorgwebhook=` dispatcher, and makes `X-Satd-Hook` + // ambiguous between two hooks with different secrets and different + // contract versions. + let f = MINIMAL.replace(r#"id = "ops""#, r#"id = "reorg-legacy""#); + let err = parse(&f).expect_err("the reserved id must be refused"); + assert!(err.to_string().contains("reserved"), "{err}"); + } + + #[test] + fn a_wrong_typed_allow_insecure_http_is_refused_not_ignored() { + // Reading `"true"` as `false` would reject the very hook the operator + // just opted in for, with an error about plaintext that makes no sense + // beside the line they added. + let f = MINIMAL.replace( + "https://alerts.example/satd", + "http://alerts.example/satd", + ) + "allow_insecure_http = \"true\"\n"; + let err = parse(&f).expect_err("a non-bool must be refused"); + assert!(err.to_string().contains("must be a bool"), "{err}"); + } + + #[test] + fn an_over_long_hook_id_is_refused() { + let f = MINIMAL.replace(r#"id = "ops""#, &format!(r#"id = "{}""#, "x".repeat(65))); + assert!(parse(&f).is_err(), "an id becomes a header value on every delivery"); + } + + /// A syntax error must not quote the offending line back, because the line + /// most likely to be mid-edit when the syntax breaks is the secret. + #[test] + fn a_toml_syntax_error_never_echoes_the_secret() { + let broken = MINIMAL.replace(r#"secret = "s3cret""#, r#"secret = "s3cret"#); + let err = parse(&broken).expect_err("unterminated string must fail"); + let rendered = err.to_string(); + assert!( + !rendered.contains("s3cret"), + "the signing secret reached the error text, and both callers log it: {rendered}", + ); + assert!(rendered.contains("line"), "should still locate the error: {rendered}"); + } + #[test] fn unknown_categories_and_kinds_are_rejected() { assert!(parse(&MINIMAL.replace("\"status\"", "\"blocks\"")).is_err()); diff --git a/satd-alert/src/contract.rs b/satd-alert/src/contract.rs index 70e3e01b4..3cae74628 100644 --- a/satd-alert/src/contract.rs +++ b/satd-alert/src/contract.rs @@ -71,36 +71,19 @@ pub fn delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { format!("{node_id_hex}-{instance_id}-{seq}") } -/// Build the idempotency key for a **synthesized** envelope — a catch-up replay -/// event or a gap notice. -/// -/// These do not come off the live bus. They are stamped by the replay builder, -/// which has no sequence to assign and writes `seq: 0` into every one of them; -/// live bus events start at 1. So `seq` is not merely a weak discriminator -/// here — it is one constant shared by every replayed event and every lag -/// notice for the life of the process. +/// Maximum age a receiver should accept for a v2 delivery, in seconds. /// -/// That matters more than it looks. Catch-up exists precisely so a hook that -/// was down does not miss what happened, and the contract tells receivers to -/// deduplicate on this header. Minting these from `seq` would mean a node down -/// for 100 blocks replays 100 events that a conforming receiver collapses into -/// one — the durability feature delivering 1% of what it advertises, silently. -/// The same collision would swallow every gap notice after the first, breaking -/// "a gap is never silent" from the second gap onward. +/// Normative for receivers, and now enforced on the sending side too: a +/// delivery older than this is refused by a conforming receiver, so continuing +/// to retry it is guaranteed-futile work that pins the head of a serial queue. +/// The dispatcher abandons an event once it crosses this age. /// -/// The `r` prefix keeps this space disjoint from the bus (``) and watch -/// (`w`) spaces. -pub fn replay_delivery_id(node_id_hex: &str, instance_id: u64, seq: u64) -> String { - format!("{node_id_hex}-{instance_id}-r{seq}") -} - - -/// Maximum age a receiver should accept for a v2 delivery, in seconds. +/// Note the bound that matters is the *total* retry span, not the 300 s cap on +/// any one backoff interval — a doubling curve reaches 600 s of cumulative age +/// around the tenth attempt. /// -/// Normative for receivers, advisory here: satd stamps `X-Satd-Timestamp` and -/// signs it, but only the receiver can enforce freshness. Wide enough to -/// tolerate ordinary clock skew and a retry backoff (which caps at 300 s), tight -/// enough that a captured delivery is not a permanent replay token. +/// Wide enough to tolerate ordinary clock skew, tight enough that a captured +/// delivery is not a permanent replay token. pub const MAX_TIMESTAMP_SKEW_SECS: u64 = 600; /// Build the canonical string a v2 signature covers. @@ -258,29 +241,6 @@ mod tests { assert_ne!(delivery_id(&node, 7, 42), delivery_id(&node, 8, 42)); } - /// The id spaces must not overlap. They are minted from independent - /// counters — the bus `seq` and a synth counter — so without the prefix a - /// synthesized notice at counter 500 and a bus event at `seq` 500 would - /// collide, and a conforming receiver would silently discard the second. - /// (The watch space, `w<...>`, is added with the watch-hook feature and is - /// covered alongside it.) - /// - /// Swept across the counter range rather than sampled at one value: a - /// collision that only appears at a boundary is exactly the kind a single - /// sample misses. - #[test] - fn the_delivery_id_spaces_are_disjoint() { - let node = "ab".repeat(16); - for n in [0u64, 1, 9, 10, 500, u32::MAX as u64, u64::MAX] { - let ids = [delivery_id(&node, 7, n), replay_delivery_id(&node, 7, n)]; - for (i, a) in ids.iter().enumerate() { - for b in ids.iter().skip(i + 1) { - assert_ne!(a, b, "delivery id spaces collide at counter {n}"); - } - } - } - } - /// Ids are unique per event within a process, across every field that /// varies: node, instance, and counter. #[test] diff --git a/satd-alert/src/lib.rs b/satd-alert/src/lib.rs index 18cfc05c5..3ce6d1cad 100644 --- a/satd-alert/src/lib.rs +++ b/satd-alert/src/lib.rs @@ -25,9 +25,12 @@ pub mod config; pub mod contract; pub mod retry; -pub use config::{AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY}; +pub use config::{ + AlertFile, AlertFileError, Hook, HookFilter, HOOK_QUEUE_CAPACITY, MAX_HOOK_ID_LEN, + RESERVED_LEGACY_REORG_ID, +}; pub use contract::{ - delivery_id, replay_delivery_id, sign_body, sign_v2, v2_signing_string, + delivery_id, sign_body, sign_v2, v2_signing_string, ATTEMPT_HEADER, DELIVERY_HEADER, HOOK_HEADER, LEGACY_WEBHOOK_VERSION, MAX_TIMESTAMP_SKEW_SECS, SIGNATURE_HEADER, TIMESTAMP_HEADER, WEBHOOK_VERSION, WEBHOOK_VERSION_HEADER, diff --git a/satd-alert/src/retry.rs b/satd-alert/src/retry.rs index 253297146..a726539d9 100644 --- a/satd-alert/src/retry.rs +++ b/satd-alert/src/retry.rs @@ -11,7 +11,7 @@ use std::time::Duration; /// What to do with an attempt's outcome. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Disposition { - /// 2xx — the receiver acked. Advance the cursor. + /// 2xx — the receiver acked. Delivered, /// Transient: retry this same event after a backoff. Retry, diff --git a/satd/src/alert.rs b/satd/src/alert.rs index fd354afef..99f3f728b 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -118,9 +118,11 @@ struct HookChannel { /// `HookCounters` is process-lived so a reload does not reset an operator's /// counters. /// -/// Injected rather than reached for as a `static` so a test can hand in a fresh -/// one. With a bare static the first test to latch `left_ibd` makes every later -/// test of the gate vacuous, non-deterministically, since they share a process. +/// Held in a process `static` (`DISPATCHER_STATE`). Note the consequence for +/// testing: `left_ibd` is a latch, so the first test in a process to set it +/// makes every later test of the IBD gate vacuous, non-deterministically. The +/// gate is covered end-to-end rather than by unit tests for that reason; a unit +/// test of it would need this state threaded through `AlertReloader::new`. #[derive(Default)] struct DispatcherState { /// Whether this node has ever been observed out of initial block download. @@ -505,9 +507,56 @@ async fn deliver_loop( } satd_alert::Disposition::Retry => { counters.failed_attempts.fetch_add(1, Ordering::Relaxed); - match &result { + // A builder error is not transient. reqwest defers URL + // parsing into the builder, so a URL that satisfies the + // alertfile's checks but that the WHATWG parser rejects + // fails here with no status — which `classify_response` + // reads as a transient network problem. `validate_url` + // parses at load precisely so this cannot happen, but if + // one ever slips through, retrying it every five minutes + // forever would pin the head of a serial queue and drop + // every real alert behind it. + if result.as_ref().err().is_some_and(reqwest::Error::is_builder) { + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::error!( + target: "alert", + hook = %hook.id, + "webhook url cannot be resolved; dropping this event (fix the \ + alertfile — retrying cannot help)", + ); + break; + } + // Stop once the delivery is older than the freshness window + // the contract publishes. `signed_at` is fixed for the life + // of the event, so past this age a conforming receiver is + // required to refuse every remaining attempt. Continuing + // would be guaranteed-futile work — and if the rejection + // arrives as a 503 from a gateway rather than a 4xx, the + // event would never reach `Drop` and would pin the queue + // permanently. + if unix_secs().saturating_sub(signed_at) > satd_alert::MAX_TIMESTAMP_SKEW_SECS { + counters.dropped.fetch_add(1, Ordering::Relaxed); + tracing::warn!( + target: "alert", + hook = %hook.id, + delivery = %delivery_id, + attempt, + "webhook delivery aged past the freshness window; giving up on it", + ); + break; + } + match result { Ok(r) => tracing::warn!(target: "alert", hook = %hook.id, status = %r.status(), attempt, "webhook delivery failed; retrying"), - Err(e) => tracing::warn!(target: "alert", hook = %hook.id, error = %e, attempt, "webhook request failed; retrying"), + // `without_url` because a webhook URL is frequently the + // credential itself (Slack, Discord, PagerDuty) and may + // carry userinfo; reqwest's `Display` appends it + // verbatim to transport and timeout errors, so the + // plain form writes that credential to the log on every + // endpoint blip. + Err(e) => { + let e = e.without_url(); + tracing::warn!(target: "alert", hook = %hook.id, error = %e, attempt, "webhook request failed; retrying"); + } } let delay = satd_alert::retry::jitter( satd_alert::retry_delay(attempt), @@ -674,9 +723,9 @@ impl AlertReloader { // This one's queue is dropped on purpose: the operator // changed where or how it delivers, and flushing the // backlog to the superseded endpoint is not what they - // asked for. `deliver_loop` hands the undelivered count - // back to the process-lived `GapState` on its way out, - // so the hole is announced rather than swallowed. + // asked for. Nothing is announced — webhooks are + // best-effort, and the loss shows up only on + // `satd_alertwebhook_dropped_total`. let _ = edited.stop.send(true); None } @@ -856,6 +905,17 @@ mod tests { .remove(0) } + /// The alertfile parser reserves this id so an operator hook cannot + /// collide with the built-in `reorgwebhook=` dispatcher. The two constants + /// live in different crates — `satd-alert` cannot depend on this binary — + /// so nothing but this assertion keeps them in step. If they drift, the + /// reservation silently protects the wrong string and both hooks share one + /// set of metrics counters again. + #[test] + fn the_reserved_hook_id_matches_the_one_the_parser_refuses() { + assert_eq!(LEGACY_REORG_HOOK_ID, satd_alert::RESERVED_LEGACY_REORG_ID); + } + fn status_env(kind: StatusKind, severity: StatusSeverity) -> NodeEvent { let mut ev = StatusEvent::raised(StatusKind::TipStall, "x"); ev.kind = kind; From fdd0ca69527343d3b8bc11b356b9c8e33380526a Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Fri, 31 Jul 2026 15:40:57 -0600 Subject: [PATCH 10/11] alerting: strip the URL from legacy reorg-webhook failure logs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The alertfile dispatcher applies `without_url()` to reqwest errors and says why: a webhook URL is frequently the credential itself (Slack, Discord, PagerDuty) and may carry userinfo, and reqwest's `Display` appends it verbatim to transport and timeout errors. The legacy `reorgwebhook` path, rewritten a few hundred lines below in the same change, did not. So `reorgwebhook=https://hooks.slack.com/services/T00/B00/` plus one endpoint blip during a reorg wrote the full credential to stdout, three times per record, from where journald and any log shipper pick it up. The behavior was inherited from the deleted `reorg_webhook_dispatcher`, but this change rewrote the line and fixed the identical problem in its sibling. Takes `result` by value in the failure arm — nothing below reads it — since `without_url` consumes the error. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01GJqZbJtTUo7K7G9wJcrvHv --- satd/src/alert.rs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/satd/src/alert.rs b/satd/src/alert.rs index 99f3f728b..7b93cbdce 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -861,9 +861,19 @@ pub async fn legacy_reorg_dispatcher( } d => { counters.failed_attempts.fetch_add(1, Ordering::Relaxed); - match &result { + // Consumes `result` (nothing below reads it) so the error + // can be stripped of its URL before it is formatted. + match result { Ok(r) => tracing::warn!(target: "alert", status = %r.status(), attempt, "reorg webhook returned non-2xx"), - Err(e) => tracing::warn!(target: "alert", error = %e, attempt, "reorg webhook request failed"), + // `without_url` for the same reason the alertfile + // dispatcher applies it: a webhook URL is frequently + // the credential itself (Slack, Discord, PagerDuty) + // and may carry userinfo, and reqwest's `Display` + // appends it verbatim to transport and timeout errors. + Err(e) => { + let e = e.without_url(); + tracing::warn!(target: "alert", error = %e, attempt, "reorg webhook request failed") + } } // Bounded retries, matching the shipped behavior: the legacy // hook has no queue to fall behind on, so a failing endpoint From 0c4c1b735550eb8e86039739237a6b9d0f6a3c5a Mon Sep 17 00:00:00 2001 From: Benjamen Keroack Date: Sat, 1 Aug 2026 10:40:23 -0600 Subject: [PATCH 11/11] alerting: sign at enqueue, restart dead hooks, and stop overpromising MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review findings on the dispatcher. **`signed_at` was stamped at dequeue**, so the freshness window bounded the retry span rather than the event's age. Delivery is serial per hook and a hard-down endpoint burns its full retry budget on each event in turn, so a `tip_stall` raised at T+0 could be popped at T+2h, signed with `now`, sail through the receiver's 600s check, and page on-call for a condition that raised and cleared two hours earlier. The signed timestamp is the ONLY staleness signal the documented receiver algorithm checks, so this defeated it entirely. The stamp is now taken once in the fan-in, when the event leaves the bus — one clock read for every hook it fans out to, so an event is never stale for one receiver and fresh for another. **A dead delivery task was re-adopted on every SIGHUP.** The keep-the-task rule compared only the stanza, so a loop that returned early left a closed channel that each reload happily kept — the config compares equal forever, so nothing restarted it and the hook stayed dark for the life of the process. The liveness check alone would not have been reachable: `apply()` short-circuits when the alertfile is unchanged, which is the common case for a SIGHUP — an operator reloads precisely BECAUSE something is wrong. So the short-circuit now makes an exception for a closed sender. Without that, "edit the stanza to force a mismatch" remained the only recovery, which is exactly what the finding described. **Docs claimed things the code does not do.** The changelog and release notes promised a `lagged` in-band gap notice; nothing synthesizes one and the branch's own e2e asserts its absence, so an integrator treating silence as proof of no gap would miss every drop. The manual said retries continue "indefinitely" when they stop at the 600s freshness bound, around attempt 10 — an operator reading that would think a 15-minute relay redeploy was covered. And it still told `reorgwebhook=` users they "need no edits" after redirect-following was removed, so a receiver answering 301/302 now silently loses every reorg record while the canonical doc says nothing changed. The dead-task test was run against the previous behavior and fails there. --- CHANGELOG.md | 6 +- docs/manual/src/observability.md | 33 ++++++-- docs/release-notes/0.5.0-pre.md | 7 +- satd/src/alert.rs | 129 ++++++++++++++++++++++++++++--- 4 files changed, 154 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0c52aef00..740b7eaec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -159,9 +159,9 @@ layout) per [`STABILITY_POLICY.md`](STABILITY_POLICY.md). metadata rides in headers (`X-Satd-Signature`, `X-Satd-Delivery`, `X-Satd-Hook`, `X-Satd-Attempt`). Delivery is serial and in-order per hook, retried with exponential backoff on transient failures and skipped on a - permanent 4xx (which still advances the resume position), with a bounded - queue whose overflow is reported in-band as a `lagged` event rather than - silently. Redirects are never followed — a 3xx is a permanent drop, so a + permanent 4xx. The per-hook queue is bounded and overflow **drops silently** + — there is no in-band gap notice; the record is + `satd_alertwebhook_dropped_total` and a log line. Redirects are never followed — a 3xx is a permanent drop, so a signed body cannot be steered to a host the alertfile never named. Delivery is **best-effort**: nothing is persisted, a hook that was down resumes at the live head, and drops are counted in `satd_alertwebhook_dropped_total`. The diff --git a/docs/manual/src/observability.md b/docs/manual/src/observability.md index 47faa6b89..1ebf88287 100644 --- a/docs/manual/src/observability.md +++ b/docs/manual/src/observability.md @@ -239,10 +239,20 @@ the only duplicate you can receive is a retry of something you already saw. - **Serial and in-order per hook** — one request in flight at a time, so events arrive in the order the node produced them. - **Retried with backoff** on 5xx, 408, 429, timeouts, and connection failures: - 1 s doubling to a 5-minute ceiling, indefinitely. Any *other* 4xx is treated - as permanent, counted, and skipped — a receiver answering 404 forever must - not pin the queue and turn every later event into a drop. A skipped event - The skip is counted in `satd_alertwebhook_dropped_total` and logged. + 1 s doubling to a 5-minute ceiling — but **not forever**. A delivery is + abandoned once it ages past the 600 s freshness window it was signed with, + which lands somewhere around the tenth or eleventh attempt. That is + deliberate: the signed timestamp is the only staleness signal a receiver + checks, so a delivery that could no longer pass that check is not worth + sending. The practical consequence is that a receiver down for longer than + ten minutes — a relay redeploy, a restart that takes a while — loses the + events raised during the outage. Alert on + `satd_alertwebhook_dropped_total` if that matters to you; the detectors + re-raise standing health conditions, so those recover on their own, but + chain and mempool events do not. + Any *other* 4xx is treated as permanent, counted, and skipped — a receiver + answering 404 forever must not pin the queue and turn every later event into + a drop. The skip is counted in `satd_alertwebhook_dropped_total` and logged. - **Redirects are not followed.** A 3xx is a permanent drop. The URL in the alertfile is where the signed body goes; following a redirect would move it — signature, hook identity and all — to a host you never named, and the useful @@ -290,9 +300,18 @@ Nothing is exported when no hook is configured. > **Note.** The older `reorgwebhook=` / `reorgwebhooksecret=` keys still work > and are now served by this dispatcher, with their original `ReorgRecord` -> payload unchanged — existing receivers need no edits. New deployments should -> prefer an `alertfile` hook with `categories = ["chain"]`, which delivers the -> standard event envelope. +> payload and v1 body-only signature unchanged. +> +> **One behavior did change:** redirects are no longer followed. A receiver +> that answers 301/302 — an `http`→`https` proxy hop, a trailing-slash +> redirect, a load balancer that relocates — used to be chased and now +> classifies as a permanent drop. If your reorg endpoint relies on a redirect, +> point `reorgwebhook=` at the final URL; otherwise every reorg record is +> silently discarded. Everything else about the payload and signature is +> byte-identical, so a receiver on a stable URL needs no edits. +> +> New deployments should prefer an `alertfile` hook with +> `categories = ["chain"]`, which delivers the standard event envelope. ## Structured JSON Logging diff --git a/docs/release-notes/0.5.0-pre.md b/docs/release-notes/0.5.0-pre.md index 25850903f..57929d552 100644 --- a/docs/release-notes/0.5.0-pre.md +++ b/docs/release-notes/0.5.0-pre.md @@ -414,8 +414,11 @@ kinds = ["tip_stall", "disk_low"] endpoint does not rebuild the same refused span on every restart. Redirects are never followed: a 3xx is a permanent drop, because chasing one would carry the signed body to a host the alertfile never named. A hook that falls - far enough behind drops events and the next delivery is preceded by a - `lagged` body carrying the count and a resume cursor: a gap is never silent. + far enough behind **drops events silently** — there is no in-band gap notice + and no resume cursor, because there is no durable log to resume from. Watch + `satd_alertwebhook_dropped_total` if you need to know. Do not read the + absence of a notice as the absence of a gap; if you need that guarantee, the + Streaming Consumption API is the surface that offers it. - **Best-effort by design.** Webhooks are the basic way to automate off chain and mempool events — page me, poke a script, ping a dead-man's switch. They are not an integration API and deliberately do not try to be one: nothing is diff --git a/satd/src/alert.rs b/satd/src/alert.rs index 7b93cbdce..3cae8d1ce 100644 --- a/satd/src/alert.rs +++ b/satd/src/alert.rs @@ -97,6 +97,20 @@ enum Delivery { Event { body: Arc>, delivery_id: String, + /// When the dispatcher took this event off the bus — NOT when the + /// delivery task got round to it. + /// + /// This is what `X-Satd-Timestamp` carries, and the spec makes it the + /// only staleness signal a receiver checks. Stamping it at dequeue + /// instead meant the freshness window bounded the *retry span* rather + /// than the event's actual age: delivery is serial per hook and a + /// hard-down endpoint burns the full retry budget on each event in + /// turn, so a `tip_stall` raised at T+0 could be popped at T+2h, + /// signed with `now`, sail through the receiver's 600 s check, and + /// page on-call for a condition that raised and cleared two hours + /// earlier. Stamped here, that delivery is correctly rejected as + /// stale. + queued_at: u64, }, } @@ -287,6 +301,10 @@ async fn fan_in( continue; }; let body = Arc::new(bytes); + // One clock read for every hook this event fans out to, so + // the same event is never stale for one receiver and fresh + // for another. + let queued_at = unix_secs(); let delivery_id = satd_alert::delivery_id( &hex::encode(env.stamp.node_id), publisher.instance_id(), @@ -297,6 +315,7 @@ async fn fan_in( enqueue(hook, Delivery::Event { body: body.clone(), delivery_id: delivery_id.clone(), + queued_at, }); } } @@ -449,7 +468,7 @@ async fn deliver_loop( None => return, }, }; - let Delivery::Event { body, delivery_id } = item; + let Delivery::Event { body, delivery_id, queued_at } = item; counters .queue_depth @@ -458,12 +477,15 @@ async fn deliver_loop( // Signed once, outside the retry loop, and reused for every attempt: // the signature must be stable across retries of one event (the // attempt counter rides in a header, deliberately not in the body or - // the signed material), and the timestamp records when satd *signed* - // this delivery, not when it last retried it. A receiver enforcing a - // freshness window therefore sees a delivery age out if it is still - // being retried after the window, which is the intended behavior: a - // 20-minute-old "disk is filling" alert is not worth acting on. - let signed_at = unix_secs(); + // the signed material). + // + // The timestamp is the moment the event left the bus, not the moment + // this task reached it — see `Delivery::Event::queued_at`. So the + // freshness window bounds the age of the *alert*, which is what a + // receiver actually wants to reason about, rather than the length of + // one retry chain. A 20-minute-old "disk is filling" alert is not + // worth acting on however long it spent queued versus retrying. + let signed_at = queued_at; let signature = satd_alert::sign_v2(&hook.secret, signed_at, &delivery_id, &hook.id, &body); let mut attempt: u32 = 0; @@ -666,15 +688,33 @@ impl AlertReloader { // // Comparing the parsed file rather than the file bytes means // reformatting or a comment edit is also a no-op. + // + // The one exception is a hook whose delivery task is no longer running. + // "Unchanged" is the common case for a SIGHUP — the operator is + // reloading precisely *because* something is wrong — so short-circuiting + // on it unconditionally would make the restart below unreachable on the + // only path an operator would take to trigger it, leaving "edit the + // stanza to force a mismatch" as the sole recovery. { let last = self.last_applied.lock(); - if last.as_ref() == Some(&file) { + let dead = { + let running = self.running.lock(); + running.hooks.values().any(|h| h.tx.is_closed()) + }; + if last.as_ref() == Some(&file) && !dead { tracing::debug!( target: "alert", "alertfile unchanged; keeping the running dispatcher generation", ); return Ok(file.hooks.len()); } + if dead { + tracing::warn!( + target: "alert", + "at least one webhook delivery task is no longer running; \ + rebuilding the dispatcher generation to restart it", + ); + } } let hook_count = file.hooks.len(); @@ -718,7 +758,26 @@ impl AlertReloader { let _guard = self.api_handle.enter(); for hook in &file.hooks { let kept = match running.hooks.remove(&hook.id) { - Some(r) if r.config == *hook => Some(r), + // An unchanged stanza keeps its task AND its queued + // backlog — but only if that task is still alive. A + // delivery loop that returned early (its `webhook_client()` + // failed to build at startup, say) leaves a closed channel + // behind, and re-adopting it on every SIGHUP made the hook + // permanently dark: the config compares equal forever, so + // nothing ever restarted it, and the operator's only + // recoveries were a full restart or editing the stanza to + // force a mismatch. A closed sender means the task is gone, + // so fall through and start a fresh one. + Some(r) if r.config == *hook && !r.tx.is_closed() => Some(r), + Some(dead) if dead.config == *hook => { + tracing::warn!( + target: "alert", + hook = %hook.id, + "webhook delivery task is no longer running; restarting it", + ); + let _ = dead.stop.send(true); + None + } Some(edited) => { // This one's queue is dropped on purpose: the operator // changed where or how it delivers, and flushing the @@ -1189,6 +1248,58 @@ heartbeat_interval_secs = 3600 ); } + /// A hook whose delivery task has died must be restarted by the next + /// reload, not re-adopted forever. + /// + /// The keep-the-task rule compared only the stanza, so a task that exited + /// early — `webhook_client()` failing to build at startup is the realistic + /// way — left a closed channel that every subsequent SIGHUP happily + /// re-adopted, because an unchanged stanza compares equal every time. The + /// hook was dark for the life of the process and the only recoveries were a + /// restart or editing the stanza to force a mismatch. Reloading is exactly + /// what an operator does to fix things, so it must actually fix this. + #[tokio::test(flavor = "current_thread")] + async fn a_reload_restarts_a_hook_whose_delivery_task_has_died() { + let dir = tempfile::tempdir().expect("tempdir"); + write_hooks(dir.path(), &stanza("pager", "\"status\"")); + let (reloader, _publisher) = handover_fixture(dir.path()); + + reloader.apply().expect("first apply"); + let before = reloader.running.lock().hooks["pager"].tx.clone(); + + // Kill this hook's delivery task the way an early return would, and let + // it observe the stop so the receiver drops and the channel closes. + reloader.running.lock().hooks["pager"] + .stop + .send(true) + .expect("stop the delivery task"); + for _ in 0..100 { + if before.is_closed() { + break; + } + tokio::task::yield_now().await; + } + assert!(before.is_closed(), "fixture must actually kill the task"); + + // Same file, same stanza — the exact case that used to re-adopt it. + reloader.apply().expect("second apply"); + let after = reloader + .running + .lock() + .hooks + .get("pager") + .map(|h| h.tx.clone()) + .expect("pager is still registered"); + assert!( + !after.is_closed(), + "a reload must restart a dead delivery task, not re-adopt it", + ); + assert!( + !before.same_channel(&after), + "and that means a fresh channel, not the corpse", + ); + } + /// The mirror: a hook the operator *did* edit is retired, so the new /// endpoint does not inherit a backlog addressed to the old one. #[tokio::test(flavor = "current_thread")]