Repository navigation
Report a refused watch add in-band (WatchAddRejected) - #899
Merged
Merged
Conversation
This was referenced Oct 5, 2026
An incremental Add* that the watch-set refused (over quota, over the per-add rate limit, over a per-connection cap, missing stream:watch, or malformed as a whole) was logged on the node and dropped. The client got no signal and could believe it was watching addresses it was not. The WatchSet add paths now return the refusal and the net-new items it covered. Both carriers hand it to the outbound task over a blocking channel, like the SetWatchSet result, and emit a WatchAddRejected event (NodeEvent tag 32) on gRPC and its JSON mirror on WS. The event names the kind, the reason with the numbers behind it, and the refused items; a refused descriptor slide says whether the earlier window is kept. The WS per-connection entry cap moves into the watch-set (WatchSet::with_entry_cap), so a capped add is reported like any other and an add that only re-asserts held items is no longer shed at the cap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both SDKs decode the new event (Rust Event::WatchAddRejected, Go *WatchAddRejected) with its kind, reason, numbers and items. When it arrives, ResilientWatch drops the refused items from its mirror so a reconnect re-registers only what the node holds; a refused descriptor slide falls back to the window the node kept. The event is still handed to the caller. The parity harness renders the event the same way from both SDKs, and two e2e tests drive it over the real binary: an over-quota gRPC add and a WS add over the per-connection entry cap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… open The streaming spec gains §7.3.2 on the new event; the quota section, the operator manual (streaming, authentication, Rust and Go SDK chapters) and both SDKs' QuotaExhausted docs stop promising RESOURCE_EXHAUSTED / 429 for an over-quota add and point at the event. CHANGELOG and the release notes describe the fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A rate limit is transient, so ResilientWatch keeps the items of a RATE_LIMITED WatchAddRejected in its mirror and re-sends them after the node's retry_after_secs or the backoff delay, whichever is later, absorbing the event. Each item has its own retry budget (the backoff's max_retries); once one is spent the add is handled like any other refusal: dropped from the mirror and handed to the caller. The re-send is built from the mirror at send time, so a removal since the refusal wins. A reconnect re-sends the whole mirror and resets the budgets. Rust drives the retries from next(), racing the stream read against the earliest deadline (EventStream::message is cancel-safe). Go schedules a timer that re-checks it is still on the same stream and sends under mu, as caller edits do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…efusal Review round 1: - A silent-payment add that only re-asserted held targets (a label update) spent a rate token and could come back RATE_LIMITED naming no items. Re-asserted targets are now free and always applied, like a re-asserted script's floor in add_items_priced; only net-new targets are charged, and a refusal names only them. Their label updates apply even when the net-new targets are refused. - ResilientWatch kept one previous window per descriptor, so two refused slides in flight restored the first refused window. Both SDKs now keep a short history of earlier windows: a refused window leaves it, and a refused latest window falls back to the latest earlier one that was not refused. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fix ships in 0.6.1 from release/0.6, so its changelog bullet and write-up go into the 0.6.1 cycle rather than the 0.6.0 notes it was first written against. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bkeroack
force-pushed
the
feat/streaming-watch-add-rejected
branch
from
October 6, 2026 13:16
42d0bc6 to
f9b9e20
Compare
bkeroack
added a commit
that referenced
this pull request
Oct 6, 2026
* events: report a refused watch add in-band with WatchAddRejected An incremental Add* that the watch-set refused (over quota, over the per-add rate limit, over a per-connection cap, missing stream:watch, or malformed as a whole) was logged on the node and dropped. The client got no signal and could believe it was watching addresses it was not. The WatchSet add paths now return the refusal and the net-new items it covered. Both carriers hand it to the outbound task over a blocking channel, like the SetWatchSet result, and emit a WatchAddRejected event (NodeEvent tag 32) on gRPC and its JSON mirror on WS. The event names the kind, the reason with the numbers behind it, and the refused items; a refused descriptor slide says whether the earlier window is kept. The WS per-connection entry cap moves into the watch-set (WatchSet::with_entry_cap), so a capped add is reported like any other and an add that only re-asserts held items is no longer shed at the cap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * SDKs surface WatchAddRejected and drop refused items from the mirror Both SDKs decode the new event (Rust Event::WatchAddRejected, Go *WatchAddRejected) with its kind, reason, numbers and items. When it arrives, ResilientWatch drops the refused items from its mirror so a reconnect re-registers only what the node holds; a refused descriptor slide falls back to the window the node kept. The event is still handed to the caller. The parity harness renders the event the same way from both SDKs, and two e2e tests drive it over the real binary: an over-quota gRPC add and a WS add over the per-connection entry cap. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: describe WatchAddRejected; RESOURCE_EXHAUSTED is only at stream open The streaming spec gains §7.3.2 on the new event; the quota section, the operator manual (streaming, authentication, Rust and Go SDK chapters) and both SDKs' QuotaExhausted docs stop promising RESOURCE_EXHAUSTED / 429 for an over-quota add and point at the event. CHANGELOG and the release notes describe the fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * satd-events-client: build the rejected descriptor with then_some Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * SDKs re-send a rate-limited watch add instead of dropping it A rate limit is transient, so ResilientWatch keeps the items of a RATE_LIMITED WatchAddRejected in its mirror and re-sends them after the node's retry_after_secs or the backoff delay, whichever is later, absorbing the event. Each item has its own retry budget (the backoff's max_retries); once one is spent the add is handled like any other refusal: dropped from the mirror and handed to the caller. The re-send is built from the mirror at send time, so a removal since the refusal wins. A reconnect re-sends the whole mirror and resets the budgets. Rust drives the retries from next(), racing the stream read against the earliest deadline (EventStream::message is cancel-safe). Go schedules a timer that re-checks it is still on the same stream and sends under mu, as caller edits do. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * e2e: ResilientWatch re-sends a watch add the node throttled Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * SP label re-asserts are free; a refused slide falls back past every refusal Review round 1: - A silent-payment add that only re-asserted held targets (a label update) spent a rate token and could come back RATE_LIMITED naming no items. Re-asserted targets are now free and always applied, like a re-asserted script's floor in add_items_priced; only net-new targets are charged, and a refusal names only them. Their label updates apply even when the net-new targets are refused. - ResilientWatch kept one previous window per descriptor, so two refused slides in flight restored the first refused window. Both SDKs now keep a short history of earlier windows: a refused window leaves it, and a refused latest window falls back to the latest earlier one that was not refused. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * clients/go: regenerate bindings for the WatchAddRejected comment Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: open the 0.6.1 notes with the WatchAddRejected fix The fix ships in 0.6.1 from release/0.6, so its changelog bullet and write-up go into the 0.6.1 cycle rather than the 0.6.0 notes it was first written against. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Forward-port to master: the CHANGELOG bullet and docs/release-notes/0.6.1-pre.md stay on release/0.6 and come to master with the 0.6.1 cut. (cherry picked from commit e10c19e)
This was referenced Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The streaming docs promised that an over-quota watch add is rejected with
RESOURCE_EXHAUSTEDon gRPC or429on WebSocket. The server sent neither. An incrementalAdd*it refused was logged on the node and dropped (events/src/watchset.rs,add_items_priced), and the client was never told. The same was true of an add the per-add rate limit throttled, an add over the 16-target silent-payment cap or the 256-descriptor cap, an add on WebSocket paststreamwsmaxsubscriptions, an add from a token withoutstream:watch, and an add malformed as a whole. A wallet that hit its quota could believe it was watching an address and miss a payment to it, andResilientWatchkept re-sending the refused adds on every reconnect.The spec's literal promise could not be kept on gRPC: a status ends the RPC, and the spec also says the subscription stays up. So the rejection is an in-band event, the way
SetWatchSet,SetCursorandRescanBlocksalready report their outcomes.Fix
WatchAddRejected(NodeEventtag 32, additive). It names the kind of add, the reason, the numbers behind it (required/held/quotafor the quota,retry_after_secsfor the rate limit,required/quotafor a cap) and the refused items, as the client named them. Reasons:QUOTA_EXCEEDED,RATE_LIMITED,CAP_EXCEEDED,PERMISSION_DENIED,MALFORMED. A refusedAddDescriptorcarries the window it asked for anddescriptor_kept, which says whether an earlier window stays watched. Silent-payment targets are named by scan pubkey, never the secret. An add that registers gets no event.WatchSet's add paths returnResult<(), AddRejected>with the net-new items they refused. Re-asserted items are never named, since they stay watched. The gRPC and WS inbound readers pass the refusal to the outbound task over a blocking channel, the same bridge asWatchSetResult, which emits the event (JSONwatch_add_rejectedon WS, with txids in the display hex a WS add uses).WatchSet::with_entry_cap), so a capped add is reported like any other. It also stops shedding an add that only re-asserts held items, since that grows nothing.SetWatchSetreads the same cap.Event::WatchAddRejectedand Go*WatchAddRejecteddecode the event. An older SDK maps the unknown body toUnknownand ignores it.ResilientWatchon a refusal. ARATE_LIMITEDrefusal is transient. The items stay in the mirror and are re-sent after the node'sretry_after_secsor the backoff delay, whichever is later, and the event is absorbed. Each item has its own budget (max_retries). The re-send is built from the mirror at send time, so a removal since the refusal wins. Any other refusal, or a rate limit past the budget, drops the items from the mirror and hands the event to the caller, so a reconnect re-registers only what the node holds. A refused descriptor slide falls back to the latest earlier window that was not itself refused; each SDK keeps a short history of windows per descriptor for this. Rust drives retries fromnext(), racing the read against the earliest deadline (EventStream::messageis cancel-safe). Go uses a timer that re-checks it is still on the same stream and sends undermu, like caller edits.docs/api/streaming.mdgains §7.3.2. §9 and the manual (streaming, authentication, Rust and Go SDK chapters) stop promisingRESOURCE_EXHAUSTED/429for an add; those codes come only at stream open. Both SDKs'QuotaExhausteddocs now name only its real causes. The CHANGELOG bullet and the release-notes write-up open the 0.6.1 cycle (docs/release-notes/0.6.1-pre.md).Tests
watchset: the quota, rate-limit, prefix, descriptor and silent-payment refusal tests now assert the returned reason and items. New tests cover a refused add naming only its net-new items,PERMISSION_DENIEDwithoutstream:watch, the entry cap refusing growth but not re-asserts, and a refused descriptor slide keeping the earlier window. The old SP-cap test's vacuousregistered = truemarker is replaced by a real assertion.grpc::tests::watch_over_quota_add_is_rejected_in_band: over the authenticated wire, a 2-script add against a 1-unit quota is refused with both scripthashes. An add that fits produces no event: the next event is the following add's refusal, withheld = 1.watch_add_rejected_encodes_each_kindpins the encoding: internal-order txids, a masked prefix, the descriptor window, scan pubkeys.ws::tests: the entry-cap test now asserts the in-band refusal and its JSON. A new test covers each kind's JSON and a malformedmin_valuesadd.Nextsees only the next real event.grpc_watch_over_quota_add_is_rejected_in_band(authfile token,watch_quota = 1),ws_add_over_entry_cap_is_rejected_in_band(--streamws-max-subscriptions=1, txid echoed in display hex), andsdk_resilient_watch_re_sends_a_rate_limited_add. In the last, arate_limit = "1/s"token spends its only token opening the Watch, so the replayed add is throttled. The SDK re-sends it on its own, a payment to the script is matched, and the refusal never reaches the caller. The parity harness renders the event identically from both SDKs.Each guard is pinned by a test that fails without it:
Okfailsover_quota_add_is_all_or_nothingand the gRPC wire test, which times out waiting for the event.Okfailsrate_limited_add_is_shed_without_dropping.entry_cap_refuses_growth_but_not_reassertsand the WS test on "an add at the cap must be reported".refused_adds_are_pruned_from_the_mirroron "a refused script is dropped". Not falling back fails the descriptor test on "replays the held window".TestRefusedAddsArePrunedFromTheMirroron both the script and the descriptor window.rate_limited_add_is_re_sent_from_the_mirror. Never driving the retry fromnext()fails the e2e test, whose payment never matches. Re-sending an item the caller removed fails the unit test on "only what the caller still holds".TestRateLimitedAddIsReSent: the refusal reachesNextinstead of the next real event.Ran locally:
cargo clippy --all-targets --all-features -- -D warnings,cargo clippy -p satd-events-client --no-default-features --all-targets -- -D warnings, thesatd-eventstests (182), thesatd-events-clienttests with all features (154) and with none (134), thestreaming::,parity::andsdk::e2e tests (66,--features e2e),clients/go/lint.sh,go test ./...,clients/go/gen.shwith no diff ineventspb, andmdbook build docs/manual.Notes
release/0.6, for 0.6.1. It was first opened against master. The branch forked at therelease/0.6cut (a7a7a87), so its commits replay ontorelease/0.6with the same patch; only the notes moved. This PR opens the 0.6.1 cycle:[Unreleased]in CHANGELOG is now bound for 0.6.1, and0.6.1-pre.mdis new. A forward-port to master follows once this merges.🤖 Generated with Claude Code