alerting: normative webhook spec + stalled-endpoint measurement by bkeroack · Pull Request #484 · epochbtc/satd

bkeroack · 2026-07-24T19:56:53Z

PR 5 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §6.3, §7). Stacked on #483 → #482 → #481 → #480 — merge bottom-up.

`docs/api/webhooks.md`

The wire contract for anyone writing a receiver: every header, the signature scheme, retry classification, gap semantics, and per-event-class durability. Includes HMAC test vectors (the same ones satd-alert's unit tests assert) plus reference verification snippets in Python and Rust, so an independent implementation can be checked for agreement without running a node.

It's deliberately explicit about non-goals — no exactly-once, no batching, no cross-hook ordering, no hosted relay — so nobody builds around a guarantee that was never offered.

The stalled-endpoint measurement

The design's release criterion 3b says a stalled endpoint must not affect consensus. That's structurally true (deliveries run on the isolated API runtime; the fan-in only try_sends), but "structurally true" is what everyone says right before it isn't, so this measures it.

The receiver used is the worst case for a dispatcher: it accepts TCP and never answers, so every attempt burns the full 10 s timeout rather than failing fast on a refused connection.

Connecting 20 regtest blocks: 11.23 ms with no webhook configured, 11.24 ms with every event going to that receiver.

If delivery were coupled to block connection, each block would have waited out the timeout — 200 s for the run — so the test's ceiling catches a regression by two orders of magnitude while staying wide enough not to flake on shared CI hardware. (A tight ratio between two wall-clock measurements on a loaded runner would flake; a regression here would not be marginal.)

Also

CORE_DIFFERENCES.md: entries for node-health alerts (Core ships -alertnotify but raises almost nothing through it, so most operators have never seen it fire) and the alertfile webhook surface, which has no Core counterpart.
The streaming spec and the manual link the new document; the manual records the measured numbers.

Acceptance (design §10 PR 5)

Latency harness recorded in the PR and the manual.
docs/api/webhooks.md normative spec + vectors; the vectors match satd-alert's tests by construction (same values, asserted in both places — and again, independently, by the push relay in PR 7).
Manual + config reference + CORE_DIFFERENCES.md.
clippy clean; the 6 webhook E2E tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

…nts) Round-3 review fixes folded in: - **A silent-payment scan key may now belong to only one hook.** The watch registry keys SP targets by `scan_pubkey` alone, so a second hook claiming the same key *replaced* the first — the replacement's labels and spend pubkey won — while routing (also scan-pubkey-only) delivered the survivor's matches to both hooks. Two silent failures: the losing hook stopped receiving the labels it asked for (change outputs, say) with no error and no log, and with differing spend pubkeys one hook's endpoint received payment details — txid, vout, amount, tweak — for a wallet it does not own. That is the one match shape where the per-hook re-filter was weaker than what was registered. The configuration is ambiguous, so it is refused at parse. - **Silent-payment targets are capped per hook**, matching the cap the streaming surface already enforces per connection. Scripts, outpoints, and txids are inverted-index lookups; SP matching has no index and costs one ECDH per target per eligible transaction, so an unbounded alertfile list put thousands of EC operations on the matcher for every taproot-bearing transaction in a block. - `SpWatchTarget` gained an identity `PartialEq` over its public fields only. `scan_pubkey` is `b_scan·G`, so comparing it covers the secret without comparing secret bytes; spend pubkey and labels are part of the identity too, since two targets sharing a scan key but differing in either derive different outputs. - The `watch_rx` closed arm warned and fell through, where the sibling bus arm breaks. A closed mpsc returns `None` immediately and forever, so that would spin at 100% CPU with a log flood. Unreachable today; the asymmetry was a trap. Known and deliberately not changed here, documented in the spec (#484): the reload hand-off between dispatcher generations is not atomic with respect to the watch registry, so a match landing in the millisecond-scale window can be missed or delivered twice. Fixing it properly means registering synchronously in `apply()` and threading the handle into `fan_in`; that is a larger change than this review round should carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

PR 5 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §6.3, §7). `docs/api/webhooks.md` is the wire contract for anyone writing a receiver: every header, the signature scheme, retry classification, gap semantics, and per-event-class durability — with HMAC test vectors (asserted by the satd-alert unit tests) and reference verification snippets, so an implementation can be checked for agreement without running a node. It is deliberately explicit about non-goals — no exactly-once, no batching, no cross-hook ordering, no hosted relay — so nobody builds around a guarantee that was never offered. Also adds the release criterion's measurement rather than asserting the property from architecture alone. A receiver that accepts TCP and never answers is the worst case for a dispatcher: not a refused connection it can fail fast on, but an open socket that burns the full 10s timeout on every attempt. Connecting 20 regtest blocks: 11.23ms with no webhook configured, 11.24ms with every event going to that receiver. If delivery were coupled to block connection, each block would have waited out the timeout — 200s for the run — so the test's ceiling catches a regression by two orders of magnitude while staying wide enough not to flake on shared CI hardware. CORE_DIFFERENCES.md gains entries for node-health alerts (Core ships -alertnotify but raises almost nothing through it) and the alertfile webhook surface, which has no Core counterpart.

Round-3 review fixes folded in. The spec is a contract, and every one of these was a place where it did not describe the code: - **The hash byte-order rule was backwards for the firehose.** §2 claimed all hashes are internal (consensus) order. That holds for watch matches, which are rendered by `watch_json`, but `chain` and `mempool` bodies derive `Serialize` over rust-bitcoin types, which render display order — and those are exactly the bodies whose fields are named `txid` and `block_hash`. An integrator following the spec would byte-reverse a `mempool.enter` txid and look up a transaction that does not exist, with no error to tell them why. Replaced with a per-body-family table; verified empirically against a serialized event, not inferred from the derive chain. - **§2's envelope does not describe watch matches.** They carry no `stamp` at all, their `cursor` is always *present* (null while unconfirmed), and it omits `instance_id`. One deserializer written against §2 would hard-fail on the first deposit alert. - **v2 signing** (see the dispatcher commit): §3 rewritten for the canonical signing string and `X-Satd-Timestamp`, with a mandated freshness window, independently-computed vectors, an `openssl` one-liner, and receiver reference code in Python and Rust. §6 consolidated onto the v2 vectors. - **The legacy `reorgwebhook=` alias is now explicitly out of scope.** It sends no delivery id, signs only when a secret is set (§3 claimed every delivery is signed), sends a `ReorgRecord` rather than an envelope, and retries on a different schedule. A receiver written to this spec and pointed at both surfaces would either reject its unsigned deliveries as forgeries or key on an absent header. - **§7's TLS claim was inverted.** reqwest is built with `rustls-tls`, which is webpki-roots — the bundled Mozilla set, not the platform trust store. An operator with a private CA installed system-wide would have followed the spec into a receiver that fails verification and retries forever. - **§5.4 overclaimed at-least-once.** A permanent non-retryable rejection is counted and advances the cursor but never produces a `lagged` body, which contradicts §5.3's "a gap is never silent". Both sides now say so. - §5.3: a `lagged` notice's resume cursor recovers chain and mempool history only — watch matches are forward-only and are not replayed by a cursor resume. - §4: documents all three delivery-id spaces, including the new `r` space for synthesized replay and gap events. - §2's worked example showed a `status` body carrying a `cursor`, contradicting the line immediately below it and the code. - New: reorgs re-emit confirmed watch matches under a *new* delivery id, so idempotency will not collapse them — and a watch-only hook receives no chain event and cannot detect the rollback at all. The manual's byte-order paragraph and the changelog/release-notes entries are corrected to match, and the `reorgwebhook=` redirect change is called out as a breaking change for that key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

Round 4 of external review. A reviewer checked every normative claim in the spec against the implementation; these are the mismatches. The document declared itself "normative for X-Satd-Webhook-Version: 1" and its one worked request example sent `1`, while §3 specified the v2 canonical signing string. The document tells receivers to branch on that header before applying anything below it, so an integrator following it would have routed legacy reorg deliveries into the v2 path — where they fail, being body-only signed with no timestamp — and had no branch at all for the `2` that actually arrives. The example also omitted `X-Satd-Timestamp` entirely, which is a field of the signing string: a receiver built from it could not verify anything. Corrected against source: `output_key` is `output_pubkey`; the `lagged` example was missing the top-level `cursor` the envelope carries, which a receiver using `deny_unknown_fields` would reject — and that is the one message class that must never be dropped; "chain events carry a cursor" is true of `block_connected` only, so a receiver persisting `event.cursor` on any chain body stalls on the first reorg. Documented what shipped but was written down nowhere: IBD suppression and its latch; the `b<height>` delivery-id space and why a replayed block's id is deliberately the one deterministic id in the scheme; that webhook watch bodies always carry `raw_tx: null` and `descriptor_matches: []` because those are per-subscription streaming knobs with no alertfile equivalent, and §2 points integrators at `streaming.md` where both are populated; and that `reorg-legacy` is a reserved hook id, which the code calls "documented as reserved" while `docs/` had zero hits for it. §5.4 claimed a permanently-rejected event produces no `lagged` body "because you were told about it at the time by being asked". That was the wrong call and the code now does emit one: the cursor advances past such an event, so unlike a queue overflow the receiver cannot go back for it, and being told is all it gets. The release note said a redirecting legacy endpoint is dropped "after the first attempt"; the legacy path deliberately retries any non-2xx three times. Finally, the manual now documents the alertfile's byte-order trap, which is inverted from the body convention *within one TOML table*: `scripts` are internal order, `outpoints` and `txids` parse as display order. Copying a txid out of a `script_matched` body into `[webhook.watch] txids` yields a valid-looking entry that watches a transaction which does not exist. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

The normative spec still documented the `b<height>` delivery-id space for replayed blocks and promised at-least-once chain delivery with a replay on startup. Neither exists: D6 made webhooks realtime with an announced gap. The four sequence spaces are three. The durability table says what the cursor is — a marker naming where a hole starts, not a log to rewind — and points at RescanBlocks or the JSON-RPC history calls for the contents. IBD suppression is documented as scoped to confirmed events, with the reason: the predicate is the tip's age, so an unscoped gate silences a stalled node on exactly the matches that have no replay behind them.

The normative contract still described watch-set bodies, a `w` delivery-id space, in-band gap notices and a durable resume position — none of which exist. A receiver written against it would have handled three id spaces for one, waited for a `lagged` body that never arrives, and persisted a cursor satd does not use. Rewritten for what the code does: one id space, no cursor, no gap notice, best-effort delivery with retries covering a briefly-unreachable receiver. §5.3 says plainly that drops are visible as an operator-side counter and not in-band, and why that is the shape rather than an omission. The scope note is now at the top rather than implied, and the non-goals list names the four things people will ask for — guaranteed delivery, gap notices, resume-after-downtime, per-address matching — and points each at the streaming API. Choosing webhooks for an integration that needs those is choosing the wrong surface, and the spec should say so before someone builds on it. Byte order simplifies with the watch bodies gone: everything a webhook delivers is RPC display order.

…ly does `docs/api/webhooks.md` is normative — third parties hardcode against it — so each of these is a receiver implemented wrong. **`CORE_DIFFERENCES.md` advertised both rejected features as shipped.** It described hooks "optionally carrying a watch-set of scripts / outpoints / txids / silent-payment scan keys" and promised "gaps are reported in-band rather than silently". Neither exists: the alertfile parser has eight known keys and none of them is a watch-set, and a full queue bumps a counter with no notice to the receiver. This is the paragraph a reader hits first, and it directly contradicts the non-goals section of the spec it links to. **§4 claimed satd never re-sends a delivered event.** A `SIGHUP` reload can send one twice, both stamped `X-Satd-Attempt: 1`. That is deliberate — the incoming generation subscribes to the bus before the outgoing one is retired, so nothing falls between them, and the delivery id is content-derived so both copies carry the same one. But a receiver following the old text ("deduplicating on this header is sufficient", alongside a promise that no duplicate is generated) would reasonably lean on the attempt counter and double-fire the page. Dedupe is now stated as required, with the reload case spelled out. **§7 said IBD suppresses everything but `status` and `heartbeat`.** Only `chain` is suppressed — `mempool` keeps flowing throughout, which on mainnet is thousands of deliveries a minute for as long as a multi-day sync takes. A receiver author sizing for silence during sync gets the opposite. Also: a `[webhook.watch]` table is refused as a plain unknown key, not with a message naming the streaming API (only the `tweaks` *category* gets a pointer); and the signing-string section described delivery ids as carrying an optional `w`/`r` tag, which no id has had since the watch and replay id spaces were removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

Three places said retries continue indefinitely. They do not: a delivery is abandoned once it ages past the 600s freshness window it was signed with, around attempt 10 or 11. §4 requires the receiver to reject anything outside that window, so retrying past it cannot succeed. The operational consequence was the part actually missing, so it is now stated: a receiver unreachable for more than ~10 minutes loses the events raised during the outage. A relay redeploy is long enough. An operator reading "indefinitely" would conclude the opposite and not alert on `satd_alertwebhook_dropped_total`. The private-CA note said such a receiver is "retried forever", which understated it in the same direction — that hook is dark, not delayed.

bkeroack mentioned this pull request Jul 24, 2026

alerting: SDK typed Event::Status + health_watch example #485

Merged

3 tasks

bkeroack force-pushed the feature/a3-watch-hooks branch from cecaf09 to 47d0811 Compare July 24, 2026 22:39

bkeroack force-pushed the feature/a3-webhooks-spec branch 2 times, most recently from 8c23001 to e1238a4 Compare July 24, 2026 22:53

bkeroack mentioned this pull request Jul 24, 2026

alerting: webhook watch-sets (scripts, outpoints, txids, silent payments) #483

Closed

8 tasks

bkeroack force-pushed the feature/a3-watch-hooks branch from d74fd4c to e8994da Compare July 25, 2026 01:27

bkeroack force-pushed the feature/a3-webhooks-spec branch from e1238a4 to d8344e7 Compare July 25, 2026 01:27

bkeroack force-pushed the feature/a3-watch-hooks branch from e8994da to a54e20b Compare July 25, 2026 15:46

bkeroack force-pushed the feature/a3-webhooks-spec branch from d8344e7 to 1c0ab06 Compare July 25, 2026 15:46

bkeroack force-pushed the feature/a3-watch-hooks branch from a54e20b to d416d3c Compare July 25, 2026 22:15

bkeroack force-pushed the feature/a3-webhooks-spec branch from 1c0ab06 to 29d1817 Compare July 25, 2026 22:15

bkeroack force-pushed the feature/a3-watch-hooks branch from d416d3c to bd0c28b Compare July 25, 2026 23:27

bkeroack force-pushed the feature/a3-webhooks-spec branch from 29d1817 to 4867fcd Compare July 25, 2026 23:27

bkeroack force-pushed the feature/a3-watch-hooks branch from bd0c28b to 68d95af Compare July 30, 2026 00:51

bkeroack force-pushed the feature/a3-webhooks-spec branch from 4867fcd to 6d430ff Compare July 30, 2026 00:51

bkeroack force-pushed the feature/a3-watch-hooks branch from 68d95af to 949965c Compare July 30, 2026 02:05

bkeroack force-pushed the feature/a3-webhooks-spec branch from 6d430ff to 12e7b8d Compare July 30, 2026 02:05

bkeroack force-pushed the feature/a3-watch-hooks branch from 949965c to f00cf2a Compare July 30, 2026 05:36

bkeroack force-pushed the feature/a3-webhooks-spec branch from 12e7b8d to 1a52afa Compare July 30, 2026 05:36

bkeroack force-pushed the feature/a3-watch-hooks branch from f00cf2a to f0d9935 Compare July 30, 2026 14:37

bkeroack force-pushed the feature/a3-webhooks-spec branch from 1a52afa to 62e9d27 Compare July 30, 2026 14:37

bkeroack changed the base branch from feature/a3-watch-hooks to feature/a3-webhook-dispatcher July 30, 2026 15:54

bkeroack force-pushed the feature/a3-webhooks-spec branch from 62e9d27 to 90e36f9 Compare July 30, 2026 15:54

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from 37a26e3 to fbbf5d4 Compare July 30, 2026 19:10

bkeroack force-pushed the feature/a3-webhooks-spec branch from 90e36f9 to ef73e3c Compare July 30, 2026 19:10

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from fbbf5d4 to 050aaf8 Compare July 30, 2026 19:38

bkeroack force-pushed the feature/a3-webhooks-spec branch from ef73e3c to d729fac Compare July 30, 2026 19:38

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from 050aaf8 to 1f0db76 Compare July 30, 2026 19:56

bkeroack force-pushed the feature/a3-webhooks-spec branch from d729fac to 1f18a9b Compare July 30, 2026 19:56

bkeroack mentioned this pull request Jul 31, 2026

alerting: satd-alert crate + webhook dispatcher, absorbing reorgwebhook #482

Merged

5 tasks

bkeroack force-pushed the feature/a3-webhooks-spec branch from 1f18a9b to e492b00 Compare July 31, 2026 21:41

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from 2948bcd to 001966e Compare August 1, 2026 04:06

bkeroack force-pushed the feature/a3-webhooks-spec branch from e492b00 to 08da44a Compare August 1, 2026 04:06

bkeroack mentioned this pull request Aug 1, 2026

alerting: node-health detectors, warnings mirroring, and gauges #481

Merged

5 tasks

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from 001966e to 1cd27d0 Compare August 1, 2026 14:09

bkeroack force-pushed the feature/a3-webhooks-spec branch from 08da44a to 38b7ad1 Compare August 1, 2026 14:09

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from 1cd27d0 to beceb29 Compare August 1, 2026 17:01

bkeroack force-pushed the feature/a3-webhooks-spec branch from 38b7ad1 to d28d014 Compare August 1, 2026 17:01

bkeroack changed the base branch from feature/a3-webhook-dispatcher to master August 2, 2026 00:02

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from beceb29 to 0c4c1b7 Compare August 2, 2026 00:02

bkeroack and others added 7 commits August 1, 2026 18:02

bkeroack force-pushed the feature/a3-webhooks-spec branch from d28d014 to fe430c1 Compare August 2, 2026 00:03

bkeroack merged commit eacadea into master Aug 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

alerting: normative webhook spec + stalled-endpoint measurement - #484

alerting: normative webhook spec + stalled-endpoint measurement#484
bkeroack merged 7 commits into
masterfrom
feature/a3-webhooks-spec

bkeroack commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

bkeroack commented Jul 24, 2026

docs/api/webhooks.md

The stalled-endpoint measurement

Also

Acceptance (design §10 PR 5)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`docs/api/webhooks.md`