Skip to content

Report a refused watch add in-band (WatchAddRejected) - #899

Merged
bkeroack merged 9 commits into
release/0.6from
feat/streaming-watch-add-rejected
Oct 6, 2026
Merged

bkeroack merged 9 commits into
release/0.6from
feat/streaming-watch-add-rejected

Conversation

@bkeroack

@bkeroack bkeroack commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The streaming docs promised that an over-quota watch add is rejected with RESOURCE_EXHAUSTED on gRPC or 429 on WebSocket. The server sent neither. An incremental Add* it refused was logged on the node and dropped (events/src/watchset.rs, add_items_priced), and the client was never told. The same was true of an add the per-add rate limit throttled, an add over the 16-target silent-payment cap or the 256-descriptor cap, an add on WebSocket past streamwsmaxsubscriptions, an add from a token without stream:watch, and an add malformed as a whole. A wallet that hit its quota could believe it was watching an address and miss a payment to it, and ResilientWatch kept re-sending the refused adds on every reconnect.

The spec's literal promise could not be kept on gRPC: a status ends the RPC, and the spec also says the subscription stays up. So the rejection is an in-band event, the way SetWatchSet, SetCursor and RescanBlocks already report their outcomes.

Fix

  • New event WatchAddRejected (NodeEvent tag 32, additive). It names the kind of add, the reason, the numbers behind it (required/held/quota for the quota, retry_after_secs for the rate limit, required/quota for a cap) and the refused items, as the client named them. Reasons: QUOTA_EXCEEDED, RATE_LIMITED, CAP_EXCEEDED, PERMISSION_DENIED, MALFORMED. A refused AddDescriptor carries the window it asked for and descriptor_kept, which says whether an earlier window stays watched. Silent-payment targets are named by scan pubkey, never the secret. An add that registers gets no event.
  • The watch-set reports, the carriers deliver. WatchSet's add paths return Result<(), AddRejected> with the net-new items they refused. Re-asserted items are never named, since they stay watched. The gRPC and WS inbound readers pass the refusal to the outbound task over a blocking channel, the same bridge as WatchSetResult, which emits the event (JSON watch_add_rejected on WS, with txids in the display hex a WS add uses).
  • Silent-payment label updates are free. A silent-payment add that only re-asserts held targets (a label update) used to spend a per-add rate token, and so could be throttled. Re-asserted targets are now free and always applied, like a re-asserted script's floor; only net-new targets are charged, and a refusal names only them. Their label updates apply even when the same message's net-new targets are refused.
  • The WS entry cap moves into the watch-set (WatchSet::with_entry_cap), so a capped add is reported like any other. It also stops shedding an add that only re-asserts held items, since that grows nothing. SetWatchSet reads the same cap.
  • SDKs. Rust Event::WatchAddRejected and Go *WatchAddRejected decode the event. An older SDK maps the unknown body to Unknown and ignores it.
  • ResilientWatch on a refusal. A RATE_LIMITED refusal is transient. The items stay in the mirror and are re-sent after the node's retry_after_secs or the backoff delay, whichever is later, and the event is absorbed. Each item has its own budget (max_retries). The re-send is built from the mirror at send time, so a removal since the refusal wins. Any other refusal, or a rate limit past the budget, drops the items from the mirror and hands the event to the caller, so a reconnect re-registers only what the node holds. A refused descriptor slide falls back to the latest earlier window that was not itself refused; each SDK keeps a short history of windows per descriptor for this. Rust drives retries from next(), racing the read against the earliest deadline (EventStream::message is cancel-safe). Go uses a timer that re-checks it is still on the same stream and sends under mu, like caller edits.
  • Docs. docs/api/streaming.md gains §7.3.2. §9 and the manual (streaming, authentication, Rust and Go SDK chapters) stop promising RESOURCE_EXHAUSTED/429 for an add; those codes come only at stream open. Both SDKs' QuotaExhausted docs now name only its real causes. The CHANGELOG bullet and the release-notes write-up open the 0.6.1 cycle (docs/release-notes/0.6.1-pre.md).

Tests

  • watchset: the quota, rate-limit, prefix, descriptor and silent-payment refusal tests now assert the returned reason and items. New tests cover a refused add naming only its net-new items, PERMISSION_DENIED without stream:watch, the entry cap refusing growth but not re-asserts, and a refused descriptor slide keeping the earlier window. The old SP-cap test's vacuous registered = true marker is replaced by a real assertion.
  • grpc::tests::watch_over_quota_add_is_rejected_in_band: over the authenticated wire, a 2-script add against a 1-unit quota is refused with both scripthashes. An add that fits produces no event: the next event is the following add's refusal, with held = 1. watch_add_rejected_encodes_each_kind pins the encoding: internal-order txids, a masked prefix, the descriptor window, scan pubkeys.
  • ws::tests: the entry-cap test now asserts the in-band refusal and its JSON. A new test covers each kind's JSON and a malformed min_values add.
  • Rust SDK: event mapping for every field, unknown codes, mirror pruning (scripts, outpoints, a masked prefix), and the descriptor fallback, including a refusal for a window the caller already slid past. Rate-limit retries: a refused add is re-sent from the mirror with its current floor and without an item removed since; once the budget is spent it is dropped and surfaced; a reconnect resets pending retries.
  • Go SDK: the exhaustive decoder test caught the new arm before its fixture existed. Scripted-server tests check mirror pruning and the descriptor fallback, and that a rate-limited add is re-sent while Next sees only the next real event.
  • e2e over the real binary: grpc_watch_over_quota_add_is_rejected_in_band (authfile token, watch_quota = 1), ws_add_over_entry_cap_is_rejected_in_band (--streamws-max-subscriptions=1, txid echoed in display hex), and sdk_resilient_watch_re_sends_a_rate_limited_add. In the last, a rate_limit = "1/s" token spends its only token opening the Watch, so the replayed add is throttled. The SDK re-sends it on its own, a payment to the script is matched, and the refusal never reaches the caller. The parity harness renders the event identically from both SDKs.

Each guard is pinned by a test that fails without it:

  • Letting a quota refusal return Ok fails over_quota_add_is_all_or_nothing and the gRPC wire test, which times out waiting for the event.
  • Letting a rate-limited add return Ok fails rate_limited_add_is_shed_without_dropping.
  • Disabling the entry cap fails entry_cap_refuses_growth_but_not_reasserts and the WS test on "an add at the cap must be reported".
  • Not forwarding the refusal to the gRPC outbound task fails the wire test.
  • Not pruning the Rust mirror fails refused_adds_are_pruned_from_the_mirror on "a refused script is dropped". Not falling back fails the descriptor test on "replays the held window".
  • Not pruning the Go mirror fails TestRefusedAddsArePrunedFromTheMirror on both the script and the descriptor window.
  • Not retrying a rate-limited add in Rust fails rate_limited_add_is_re_sent_from_the_mirror. Never driving the retry from next() fails the e2e test, whose payment never matches. Re-sending an item the caller removed fails the unit test on "only what the caller still holds".
  • Not retrying in Go fails TestRateLimitedAddIsReSent: the refusal reaches Next instead of the next real event.

Ran locally: cargo clippy --all-targets --all-features -- -D warnings, cargo clippy -p satd-events-client --no-default-features --all-targets -- -D warnings, the satd-events tests (182), the satd-events-client tests with all features (154) and with none (134), the streaming::, parity:: and sdk:: e2e tests (66, --features e2e), clients/go/lint.sh, go test ./..., clients/go/gen.sh with no diff in eventspb, and mdbook build docs/manual.

Notes

  • Targets release/0.6, for 0.6.1. It was first opened against master. The branch forked at the release/0.6 cut (a7a7a87), so its commits replay onto release/0.6 with the same patch; only the notes moved. This PR opens the 0.6.1 cycle: [Unreleased] in CHANGELOG is now bound for 0.6.1, and 0.6.1-pre.md is new. A forward-port to master follows once this merges.
  • Individually unparseable items inside an otherwise valid add (a scripthash that is not 32 bytes, a prefix outside the bit range, an invalid silent-payment key) are still skipped without a rejection. The spec now says so.

🤖 Generated with Claude Code

bkeroack and others added 9 commits October 6, 2026 07:06
An incremental Add* that the watch-set refused (over quota, over the
per-add rate limit, over a per-connection cap, missing stream:watch, or
malformed as a whole) was logged on the node and dropped. The client got
no signal and could believe it was watching addresses it was not.

The WatchSet add paths now return the refusal and the net-new items it
covered. Both carriers hand it to the outbound task over a blocking
channel, like the SetWatchSet result, and emit a WatchAddRejected event
(NodeEvent tag 32) on gRPC and its JSON mirror on WS. The event names
the kind, the reason with the numbers behind it, and the refused items;
a refused descriptor slide says whether the earlier window is kept.

The WS per-connection entry cap moves into the watch-set
(WatchSet::with_entry_cap), so a capped add is reported like any other
and an add that only re-asserts held items is no longer shed at the cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both SDKs decode the new event (Rust Event::WatchAddRejected, Go
*WatchAddRejected) with its kind, reason, numbers and items. When it
arrives, ResilientWatch drops the refused items from its mirror so a
reconnect re-registers only what the node holds; a refused descriptor
slide falls back to the window the node kept. The event is still
handed to the caller.

The parity harness renders the event the same way from both SDKs, and
two e2e tests drive it over the real binary: an over-quota gRPC add and
a WS add over the per-connection entry cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… open

The streaming spec gains §7.3.2 on the new event; the quota section,
the operator manual (streaming, authentication, Rust and Go SDK
chapters) and both SDKs' QuotaExhausted docs stop promising
RESOURCE_EXHAUSTED / 429 for an over-quota add and point at the event.
CHANGELOG and the release notes describe the fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A rate limit is transient, so ResilientWatch keeps the items of a
RATE_LIMITED WatchAddRejected in its mirror and re-sends them after the
node's retry_after_secs or the backoff delay, whichever is later,
absorbing the event. Each item has its own retry budget (the backoff's
max_retries); once one is spent the add is handled like any other
refusal: dropped from the mirror and handed to the caller. The re-send
is built from the mirror at send time, so a removal since the refusal
wins. A reconnect re-sends the whole mirror and resets the budgets.

Rust drives the retries from next(), racing the stream read against the
earliest deadline (EventStream::message is cancel-safe). Go schedules a
timer that re-checks it is still on the same stream and sends under mu,
as caller edits do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…efusal

Review round 1:

- A silent-payment add that only re-asserted held targets (a label
  update) spent a rate token and could come back RATE_LIMITED naming no
  items. Re-asserted targets are now free and always applied, like a
  re-asserted script's floor in add_items_priced; only net-new targets
  are charged, and a refusal names only them. Their label updates apply
  even when the net-new targets are refused.
- ResilientWatch kept one previous window per descriptor, so two refused
  slides in flight restored the first refused window. Both SDKs now keep
  a short history of earlier windows: a refused window leaves it, and a
  refused latest window falls back to the latest earlier one that was
  not refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fix ships in 0.6.1 from release/0.6, so its changelog bullet and
write-up go into the 0.6.1 cycle rather than the 0.6.0 notes it was
first written against.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@bkeroack
bkeroack changed the base branch from master to release/0.6 October 6, 2026 13:16
@bkeroack
bkeroack force-pushed the feat/streaming-watch-add-rejected branch from 42d0bc6 to f9b9e20 Compare October 6, 2026 13:16
@bkeroack
bkeroack merged commit e10c19e into release/0.6 Oct 6, 2026
49 checks passed
@bkeroack
bkeroack deleted the feat/streaming-watch-add-rejected branch October 6, 2026 15:55
bkeroack added a commit that referenced this pull request Oct 6, 2026
* events: report a refused watch add in-band with WatchAddRejected

An incremental Add* that the watch-set refused (over quota, over the
per-add rate limit, over a per-connection cap, missing stream:watch, or
malformed as a whole) was logged on the node and dropped. The client got
no signal and could believe it was watching addresses it was not.

The WatchSet add paths now return the refusal and the net-new items it
covered. Both carriers hand it to the outbound task over a blocking
channel, like the SetWatchSet result, and emit a WatchAddRejected event
(NodeEvent tag 32) on gRPC and its JSON mirror on WS. The event names
the kind, the reason with the numbers behind it, and the refused items;
a refused descriptor slide says whether the earlier window is kept.

The WS per-connection entry cap moves into the watch-set
(WatchSet::with_entry_cap), so a capped add is reported like any other
and an add that only re-asserts held items is no longer shed at the cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* SDKs surface WatchAddRejected and drop refused items from the mirror

Both SDKs decode the new event (Rust Event::WatchAddRejected, Go
*WatchAddRejected) with its kind, reason, numbers and items. When it
arrives, ResilientWatch drops the refused items from its mirror so a
reconnect re-registers only what the node holds; a refused descriptor
slide falls back to the window the node kept. The event is still
handed to the caller.

The parity harness renders the event the same way from both SDKs, and
two e2e tests drive it over the real binary: an over-quota gRPC add and
a WS add over the per-connection entry cap.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: describe WatchAddRejected; RESOURCE_EXHAUSTED is only at stream open

The streaming spec gains §7.3.2 on the new event; the quota section,
the operator manual (streaming, authentication, Rust and Go SDK
chapters) and both SDKs' QuotaExhausted docs stop promising
RESOURCE_EXHAUSTED / 429 for an over-quota add and point at the event.
CHANGELOG and the release notes describe the fix.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* satd-events-client: build the rejected descriptor with then_some

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* SDKs re-send a rate-limited watch add instead of dropping it

A rate limit is transient, so ResilientWatch keeps the items of a
RATE_LIMITED WatchAddRejected in its mirror and re-sends them after the
node's retry_after_secs or the backoff delay, whichever is later,
absorbing the event. Each item has its own retry budget (the backoff's
max_retries); once one is spent the add is handled like any other
refusal: dropped from the mirror and handed to the caller. The re-send
is built from the mirror at send time, so a removal since the refusal
wins. A reconnect re-sends the whole mirror and resets the budgets.

Rust drives the retries from next(), racing the stream read against the
earliest deadline (EventStream::message is cancel-safe). Go schedules a
timer that re-checks it is still on the same stream and sends under mu,
as caller edits do.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* e2e: ResilientWatch re-sends a watch add the node throttled

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* SP label re-asserts are free; a refused slide falls back past every refusal

Review round 1:

- A silent-payment add that only re-asserted held targets (a label
  update) spent a rate token and could come back RATE_LIMITED naming no
  items. Re-asserted targets are now free and always applied, like a
  re-asserted script's floor in add_items_priced; only net-new targets
  are charged, and a refusal names only them. Their label updates apply
  even when the net-new targets are refused.
- ResilientWatch kept one previous window per descriptor, so two refused
  slides in flight restored the first refused window. Both SDKs now keep
  a short history of earlier windows: a refused window leaves it, and a
  refused latest window falls back to the latest earlier one that was
  not refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* clients/go: regenerate bindings for the WatchAddRejected comment

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: open the 0.6.1 notes with the WatchAddRejected fix

The fix ships in 0.6.1 from release/0.6, so its changelog bullet and
write-up go into the 0.6.1 cycle rather than the 0.6.0 notes it was
first written against.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Forward-port to master: the CHANGELOG bullet and docs/release-notes/0.6.1-pre.md
stay on release/0.6 and come to master with the 0.6.1 cut.
(cherry picked from commit e10c19e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant