alerting: webhook watch-sets (scripts, outpoints, txids, silent payments) by bkeroack · Pull Request #483 · epochbtc/satd

bkeroack · 2026-07-24T19:56:24Z

PR 4 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §4.5). Stacked on #482 → #481 → #480 — merge bottom-up.

A hook can now carry a watch-set and receive the same match events the streaming Watch stream delivers:

[webhook.watch]
scripts   = ["<32-byte scripthash hex>"]
outpoints = ["<txid>:<vout>"]
txids     = ["<txid>"]

[[webhook.watch.silent_payments]]
scan_key     = "<32-byte hex>"
spend_pubkey = "<33-byte compressed hex>"
labels       = [0]

Decisions worth reviewing

watch_match_json moves out of the WebSocket carrier into node::events. The WS firehose and the webhook dispatcher now render matches with the same code, so a receiver written against the streaming spec parses webhook matches unchanged and the two surfaces cannot drift. This is the substance of the PR; the rest is plumbing.
One union registry subscriber for all hooks, not one per hook. The matcher's cost is per watched entry per transaction, so registering the same script twice would double it. Routing each match back to the hooks that asked for it is a cheap membership test on the delivery side — and getting it wrong would leak one operator's deposit activity into an unrelated endpoint, so it's asserted in both directions (unit + an E2E with a watching hook and a bystander hook).
The alertfile is the durable source of truth for the set. Watch state is per-subscriber and memory-only, so each dispatcher generation re-registers from the file — the same "rebuild from durable truth on reconnect" shape ResilientWatch uses, with the config file as the loader. A SIGHUP reload therefore re-registers for free.
Silent-payment scan keys in an alertfile are a deliberate exception to "satd never persists a scan key" (SP design D3, amended for this case). Over the streaming API the client and the operator are different parties, so a key lives in memory for one connection and is never written. A webhook consumer is the operator, alerting on their own wallet on their own node: the key is watch-only — payment detection, no spend authority — and the file already holds signing secrets at 0600. Guardrails: zeroized in memory, WatchSet's Debug renders counts only, and a malformed-key error deliberately does not echo the value (config errors end up in bug reports). Both are asserted by tests. Entries require silentpaymentindex=1.
Validation stays recognize-and-reject: a short scripthash, an outpoint without a :vout, a bad txid, or an unknown key under [webhook.watch] is an error rather than a watch that silently matches nothing.

Semantics worth stating

A watch-set is forward-only from the moment it is registered. Adding an entry does not replay history for it: you are told about payments from now on, not about the ones you already reconciled. That is the intended behavior for an alerting surface — a backfill would fire one notification per historical transaction touching the entry, which is not what "tell me when my wallet gets a deposit" means. getaddresshistory and the streaming API's RescanBlocks are how you ask for history.

A restart is not a gap: the watch-set is re-registered before P2P starts, so blocks arriving during catch-up are matched normally, and a -reindex replays before the dispatcher exists so it does not re-fire years of alerts. The one case that does lose a match is a crash between a block connecting and the receiver acknowledging — the block's chain event comes back from the hook's cursor, but the match does not, because the block is already connected and is not rescanned. Normally milliseconds; wider if the receiver is down and the queue has backed up. Documented in the manual, the spec's durability table, and the release notes.

Acceptance

E2E deposit-watch webhook: fund a watched script → signed script_matched at mempool → confirmed re-emit after mining.
E2E routing isolation: a second hook with no watch-set receives its chain events but never the first hook's matches.
Unit: script and silent-payment routing by identity, in both directions.
Unit: scan key absent from Debug output and from error messages; malformed targets rejected.
SP entries refused without silentpaymentindex=1.
Watch-set re-registered before P2P accepts connections, so a restart is not a delivery gap (verified against startup ordering in main.rs; the remaining crash-between-connect-and-ack window is documented).
E2E: every delivery on one hook carries a distinct X-Satd-Delivery, including two matches produced by a single transaction — the case where a shared bus counter collided.
clippy --workspace --all-targets --all-features clean; suite green except the 10 pre-existing test_address_index_backfill_* disk-guard failures on this machine.

🤖 Generated with Claude Code

https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

bkeroack · 2026-07-24T22:40:34Z

Round-1 review fix (High — data loss under receiver-side dedupe). Confirmed and fixed.

Watch-match deliveries minted X-Satd-Delivery from publisher.published() — a read of the bus sequence counter. Bus events take theirs from stamp.seq, which fetch_adds. Matches never advanced it, so every match emitted between two bus publishes shared one id, and that id also collided with the bus event that last moved the counter. A receiver following the contract (including the reference relay in #486) ACKs the second as a duplicate and drops it.

Watch matches now draw from their own process-wide counter and render through watch_delivery_id(), which w-prefixes the sequence so the two spaces cannot overlap at any value. Process-wide rather than per-generation: a SIGHUP reload keeps the same instance_id, so a per-generation counter would restart at 1 and re-mint seen ids. One id per match, shared across hooks — same rule the bus path follows.

Worth flagging about the test: my first version asserted uniqueness over a single watched payment and passed against the buggy code, because each of its two matches happened to land after a different bus publish. The test now pays the watched script twice in one transaction, which puts two matches between one pair of bus publishes. Against the old scheme it fails with the duplicate pair visible in the output (…-103, …-103, …-104, …-104).

Also corrected the watch-durability wording here and in the manual, spec, and release notes — see the updated PR body. It claimed a match occurring while the daemon was down is not re-delivered; the watch-set is in fact re-registered before P2P starts, so a restart is not a gap.

bkeroack · 2026-07-24T22:53:20Z

Documented, per the follow-up question about categories.

A watch-set is not filtered by categories — they are independent subscriptions on one hook. categories selects from the node's firehose (what the node and the chain are doing); the watch-set selects your own addresses, coins, and transactions. Both phases of a match arrive whatever the categories say, because a deposit watcher wants the pending credit whether or not it wants every transaction on the network.

The trap this prevents is adding "mempool" to a hook in order to see unconfirmed matches: you already see them, and what you gain is thousands of transactions a minute against an endpoint that wanted to hear about one wallet.

The manual also gained the table of what each category delivers and at what rate — it never had one — plus a note that kinds/min_severity narrow status alone and are checked after the category bit, so kinds without "status" matches nothing. Same rule stated normatively in docs/api/webhooks.md §2 (in #484).

The e2e deposit-watch test already configured categories = ["chain"] and asserted the unconfirmed sighting arrives, so it was the guard for this by accident; its doc comment now says so on purpose.

bkeroack · 2026-07-25T01:27:55Z

Round-2 blocker — confirmed and fixed. My error, and worth stating exactly how it happened.

When I added the categories documentation commit I used git add -A. The /contrib/push-relay/target ignore rule lives in #486's commit — five branches above this one — so on #483 those paths were not ignored, and a stale build directory left behind by running the relay's tests got swept into the commit. From there it cascaded into #484–#486, and #486's .gitignore could not help: ignore rules do not untrack files that are already tracked below them.

Fixed by amending the doc commit to drop them (git rm -r --cached) and rebasing the four branches above. The commit is now the three files it should always have been. Verified across the whole stack:

feature/a3-status-events           target=0   contrib total=0
feature/a3-health-detectors        target=0   contrib total=0
feature/a3-webhook-dispatcher      target=0   contrib total=0
feature/a3-watch-hooks             target=0   contrib total=0
feature/a3-webhooks-spec           target=0   contrib total=0
feature/a3-sdk-status              target=0   contrib total=0
feature/a3-push-relay              target=0   contrib total=9   ← source only

PR file counts: #483 16, #484 8, #485 8, #486 15.

I also deleted the build directory from the worktree so it cannot be swept up again, and will point CARGO_TARGET_DIR outside the repo for future relay builds. The ignore rule stays in #486, where the crate that produces the output is introduced.

Separately: #483's previous CI "Tests" job reported failure with no failed step and no retrievable log, which reads like the runner dying rather than a real failure — plausibly on the 2119-file checkout. The re-run on the purged tip will settle that; I'll report it.

…nts) PR 4 of the A3 alerting stack (SATD_ALERTING_DESIGN.md §4.5). A hook can now carry a `[webhook.watch]` table and receive the same match events the streaming `Watch` stream delivers. The shared-rendering point is the substance here: `watch_match_json` moves out of the WebSocket carrier into `node::events`, so the WS firehose and the webhook dispatcher emit *the same bytes* for the same match. A receiver written against the streaming spec parses webhook matches unchanged, and the two surfaces cannot drift apart. Design points: - One union registry subscriber for all hooks, not one per hook. The matcher's cost is per watched entry per transaction, so registering the same script twice would double it. Routing a match back to the hooks that actually asked for it is a cheap membership test on the delivery side — and getting it wrong would leak one operator's deposit activity into an unrelated endpoint, so it is unit-tested in both directions. - The alertfile is the durable source of truth for the set. Watch state is per-subscriber and lives only in memory, so each dispatcher generation re-registers from the file — the same "rebuild from durable truth on reconnect" shape `ResilientWatch` uses, with the config file as the loader. A reload therefore re-registers automatically. - Silent-payment scan keys in an alertfile are a deliberate exception to "satd never persists a scan key". Over the streaming API the client and the operator are different parties, so a key is held for one connection and never written. A webhook consumer IS the operator, alerting on their own wallet on their own node: the key is watch-only (payment detection, no spend authority) and the file already holds signing secrets at 0600. Keys are zeroized in memory, `WatchSet`'s Debug renders counts only, and a malformed-key error deliberately does not echo the value — config errors end up in bug reports. Entries require silentpaymentindex=1. - Validation stays recognize-and-reject: a short scripthash, an outpoint without a vout, a bad txid, or an unknown key under `[webhook.watch]` is an error rather than a watch that silently matches nothing. Left out, deliberately and stated in the docs: historical catch-up for watch *matches* after a restart. A hook's confirmed chain events are replayed from its cursor (PR 3), but a match that occurred while the daemon was down is not re-delivered — that needs the bounded-rescan driver currently embedded in the gRPC carrier to become shared machinery, which is a refactor of a hot, carefully-tuned path and belongs in its own change. Live matching is unaffected.

Watch-match deliveries minted X-Satd-Delivery from publisher.published() — a *read* of the bus sequence counter, not an allocation. Bus events take their id from stamp.seq, which fetch_adds. Matches never advanced it, so every match emitted between two bus publishes shared one id, and that id also collided with the bus event that last moved the counter. The contract makes this header the receiver's idempotency key, and the reference push relay (and any receiver following the spec) dedupes on it. Two watched outputs in one transaction produced two alerts carrying the same id: a correct receiver ACKs the second as a duplicate and drops it. Silent data loss in the feature this branch exists to ship. Watch matches now draw from their own process-wide counter and render through watch_delivery_id(), which `w`-prefixes the sequence so the two id spaces cannot overlap at any value. Process-wide rather than per-generation because a SIGHUP reload keeps the same instance_id — a per-generation counter would restart at 1 and re-mint ids the receiver has already seen. One id per match, shared by every hook that wants it, matching how the bus path already works. The e2e test now pays the watched script twice in a single transaction, which is what puts two matches between one pair of bus publishes. It fails against the old scheme with the duplicate pair visible in the assertion output; an earlier single-output version of the same test did not, because each match happened to land after a different bus publish. Also corrects the watch-set durability wording in the manual, release notes, and changelog. It said a match occurring while the daemon was down is not re-delivered; the watch-set is in fact re-registered before P2P starts, so a restart is not a gap and -reindex does not re-fire history. Forward-only from registration is the intended semantic. The one real hole is a crash between a block connecting and the receiver acknowledging. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

…nts) Round-3 review fixes folded in: - **A silent-payment scan key may now belong to only one hook.** The watch registry keys SP targets by `scan_pubkey` alone, so a second hook claiming the same key *replaced* the first — the replacement's labels and spend pubkey won — while routing (also scan-pubkey-only) delivered the survivor's matches to both hooks. Two silent failures: the losing hook stopped receiving the labels it asked for (change outputs, say) with no error and no log, and with differing spend pubkeys one hook's endpoint received payment details — txid, vout, amount, tweak — for a wallet it does not own. That is the one match shape where the per-hook re-filter was weaker than what was registered. The configuration is ambiguous, so it is refused at parse. - **Silent-payment targets are capped per hook**, matching the cap the streaming surface already enforces per connection. Scripts, outpoints, and txids are inverted-index lookups; SP matching has no index and costs one ECDH per target per eligible transaction, so an unbounded alertfile list put thousands of EC operations on the matcher for every taproot-bearing transaction in a block. - `SpWatchTarget` gained an identity `PartialEq` over its public fields only. `scan_pubkey` is `b_scan·G`, so comparing it covers the secret without comparing secret bytes; spend pubkey and labels are part of the identity too, since two targets sharing a scan key but differing in either derive different outputs. - The `watch_rx` closed arm warned and fell through, where the sibling bus arm breaks. A closed mpsc returns `None` immediately and forever, so that would spin at 100% CPU with a log flood. Unreachable today; the asymmetry was a trap. Known and deliberately not changed here, documented in the spec (#484): the reload hand-off between dispatcher generations is not atomic with respect to the watch registry, so a match landing in the millisecond-scale window can be missed or delivered twice. Fixing it properly means registering synchronously in `apply()` and threading the handle into `fan_in`; that is a larger change than this review round should carry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

Round 4 of external review, scoped to the ~185 lines this branch added during round 3. The IBD gate was missing from the watch arm — three reviewers found it independently. `run_watch_matcher` is spawned unconditionally and scans every connected block, so a fresh node with a watched busy address POSTed one signed delivery per historical matching transaction for the whole sync. The bus arm's own comment names this as half of what it prevents, and the manual promises watch-sets are forward-only from registration. A reload registered the new generation's watch-set before retiring the old one, so both were live registry subscribers for the overlap and each minted its own `WATCH_DELIVERY_SEQ`. Two POSTs, same body, same hook, *different* `X-Satd-Delivery` — a receiver deduplicating on that header, as the contract instructs, credits the deposit twice. The window is sub-millisecond normally but stretches to seconds while the new generation's catch-up replays, i.e. right after a restart. Retiring now precedes spawning: a momentary gap instead of an undedupable duplicate, which is the better trade because a chain-event gap is recovered by the cursor and watch matches are already documented as best-effort. `break` on a closed watch channel exited `fan_in` entirely, so status, chain, and heartbeat deliveries would all stop on the strength of the watch channel closing — one warn line, no metric moving. Unreachable today, but reachable the moment registry-side subscriber pruning lands, which the original comment anticipates. The arm is now guarded out of the `select!` instead, which fixes the 100% CPU spin it was written for without the blast radius. `MAX_SP_TARGETS_PER_HOOK` bounded one stanza while the cost is per union subscriber: matching is one ECDH per registered target per eligible transaction whether the targets came from one hook or two hundred, and nothing capped hook count. Added `MAX_SP_TARGETS_TOTAL`. The collision error named only the losing hook (a `BTreeSet` discarded the first claimant) and described a duplicate *within* one hook as "already claimed by another hook", sending the operator to look for a stanza that does not exist. The cross-hook guard lived only inside `parse`, while `AlertFile` has public fields and derives `Default`. Extracted as `AlertFile::validate`, so a future runtime "add a webhook" surface or an SDK consumer cannot rebuild the cross-tenant leak by constructing one directly. Docs: both new limits are hard startup failures and were documented nowhere — an operator with two teams' hooks on a shared corporate wallet would have hit an `exit(1)` explained by no changelog, manual chapter, or release note. Also documented that a `lagged` covering watch matches is weaker than one covering chain events: its resume cursor is a chain position, so replaying from it re-delivers blocks, not the lost matches. Each new guard verified to fail against the code it guards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018AfAm3VVN8gBeYcTL5BJqH

The watch half of the handover. The fan-in registered the union watch-set for itself and held the handle, so deregistration happened whenever the retired task next got scheduled — which meant the incoming generation could register while the outgoing one was still live. Both then receive every match in the window, and each mints its own WATCH_DELIVERY_SEQ: two POSTs, same body, same hook, different X-Satd-Delivery. A receiver deduplicating on that header, as the contract instructs, credits the deposit twice. The previous fix for this was to retire the old generation before spawning the new one, which bought watch correctness at the cost of a window with no bus subscriber at all — the P0 the parent commit fixes. The two resources want opposite orderings, so they now get them: the bus is overlapped (its ids are derived from the event's own seq, so duplicates collapse), the watch registration is gapped (its ids come from a process counter, so duplicates do not). apply() owns both ends and does the swap itself. register_watch_sets takes stanzas rather than live channels and returns the handle to the caller. The fan-in's "watch channel closed" error is now reserved for a channel that closed on its own: deregistration is a handover, and logging it as a total alerting outage on every reload would train operators to ignore the line that means it really happened. WatchRegistry::watched_items counts entries across all subscribers, which is what makes the swap observable. The test registers different numbers of scripts across the two applies so a deferred swap reads 1 where a completed one reads 2; verified to fail against a fan-in that registers for itself.

The IBD gate is scoped to confirmed matches — an unconfirmed one comes from the mempool and cannot be historical, and the predicate is the tip's age, which reads "syncing" on a stalled node. The manual still said watch alerts were suppressed outright. The lagged-notice and unclean-shutdown paragraphs also still described a cursor that re-delivers the span. It announces it.

bkeroack · 2026-07-30T15:54:56Z

Closing: webhook watch-sets are cut from the design.

Webhooks had grown into a second integration API. The streaming API is the canonical way to do sophisticated integration with satd, and its Watch stream already does per-address, per-outpoint, per-txid and per-scan-key matching properly — registration control messages, per-connection watch-sets, depth alarms, RescanBlocks for history, backpressure. This PR reimplemented a subset of that behind an HTTP POST for ~1,670 lines, and pulled in a second delivery-id space, a confirmed/unconfirmed IBD gate, per-hook match routing, and a registration-handover ordering problem to go with it.

Design D8 now reads "rejected" rather than "SP scan keys included", and a [webhook.watch] table is a named parse error pointing at the streaming API. That also reinstates SP design D3 unamended: scan keys arrive over the streaming API only and the node never accepts one from an alertfile.

#484 has been retargeted onto #482, so the rest of the stack is unaffected. The branch is left in place if any of this is wanted later.

This was referenced Jul 24, 2026

alerting: normative webhook spec + stalled-endpoint measurement #484

Merged

alerting: SDK typed Event::Status + health_watch example #485

Merged

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from eb6b6d2 to a2d97b0 Compare July 24, 2026 22:39

bkeroack force-pushed the feature/a3-watch-hooks branch from cecaf09 to 47d0811 Compare July 24, 2026 22:39

bkeroack mentioned this pull request Jul 24, 2026

alerting: reference APNs/FCM push relay in contrib/ #486

Merged

3 tasks

bkeroack force-pushed the feature/a3-watch-hooks branch from d74fd4c to e8994da Compare July 25, 2026 01:27

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from a2d97b0 to fb24c8c Compare July 25, 2026 15:46

bkeroack force-pushed the feature/a3-watch-hooks branch from e8994da to a54e20b Compare July 25, 2026 15:46

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from fb24c8c to 8c4e12f Compare July 25, 2026 22:15

bkeroack force-pushed the feature/a3-watch-hooks branch 2 times, most recently from d416d3c to bd0c28b Compare July 25, 2026 23:27

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from d29f5f9 to aa6573d Compare July 30, 2026 00:51

bkeroack force-pushed the feature/a3-watch-hooks branch from bd0c28b to 68d95af Compare July 30, 2026 00:51

bkeroack force-pushed the feature/a3-webhook-dispatcher branch from aa6573d to 6addf3e Compare July 30, 2026 02:05

bkeroack force-pushed the feature/a3-watch-hooks branch 2 times, most recently from 949965c to f00cf2a Compare July 30, 2026 05:36

bkeroack and others added 6 commits July 30, 2026 08:31

bkeroack force-pushed the feature/a3-watch-hooks branch from f00cf2a to f0d9935 Compare July 30, 2026 14:37

bkeroack closed this Jul 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

alerting: webhook watch-sets (scripts, outpoints, txids, silent payments) - #483

alerting: webhook watch-sets (scripts, outpoints, txids, silent payments)#483
bkeroack wants to merge 6 commits into
feature/a3-webhook-dispatcherfrom
feature/a3-watch-hooks

bkeroack commented Jul 24, 2026 •

edited

Loading

Uh oh!

bkeroack commented Jul 24, 2026

Uh oh!

bkeroack commented Jul 24, 2026

Uh oh!

bkeroack commented Jul 25, 2026

Uh oh!

bkeroack commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

bkeroack commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Decisions worth reviewing

Semantics worth stating

Acceptance

Uh oh!

bkeroack commented Jul 24, 2026

Uh oh!

bkeroack commented Jul 24, 2026

Uh oh!

bkeroack commented Jul 25, 2026

Uh oh!

bkeroack commented Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bkeroack commented Jul 24, 2026 •

edited

Loading