Tracking: transactional rate-limit admission — charge/refund lifecycle + capacity path

[erskingardner]

Root cause

Rate-limit charging is non-transactional: budget is spent at check_and_increment time, but admission is only decided later — and the refund logic only covers some failure paths. The limiter's capacity handling also serializes the hot path.

• In process_inner (src/nostr/events.rs), the encrypted-token limiter is charged before the device-token limiter is checked; a device-reject just continues, stranding the encrypted charge with no refund and no rollback record — a later legitimate redelivery reads as a replay ([LOW] Encrypted-token rate-limit increment is never refunded when the paired device token is rate-limited (asymmetric with the dispatch-failure rollback) #170).
• Tokens the dispatcher drops after admission (unconfigured platform, non-UTF8 FCM token — src/push/dispatcher.rs:397-425) keep their spent budget and are counted by no metric; the drop is invisible below trace! ([LOW] A decrypted token dropped inside dispatch() (unconfigured platform / non-UTF8 FCM) keeps its rate-limit budget and is recorded by no metric #177).
• At capacity, check_and_increment runs a 1000-entry eviction scan (with per-entry prunes) while holding the limiter's single global write lock — a cardinality flood converts into a server-wide stall of the very machinery meant to shed it ([MEDIUM] RateLimiter::check_and_increment runs an O(1000) eviction scan while holding the global write lock, serializing all rate-limit checks under cache-cardinality pressure #123).
• transponder_rate_limit_cache_size is only written from the 60s cleanup tick, so saturation onset is invisible for up to a minute ([LOW] transponder_rate_limit_cache_size gauge is only updated during the 60s cleanup, never on insertion — cache saturation / capacity-limit onset is invisible for up to a minute #125).
• Same hot path, adjacent lock waste: the dedup cache takes its global write lock twice per successful event (try_reserve then mark_seen) ([LOW] Successful event path acquires the dedup write lock twice (try_reserve then mark_seen), doubling lock contention and gauge updates on the hot path #197).

Refactor design

1. AdmissionGuard (RAII): check_and_increment returns a guard; commit() on successful hand-off to the dispatcher, drop-without-commit refunds the charge. Device-reject then automatically refunds the encrypted charge ([LOW] Encrypted-token rate-limit increment is never refunded when the paired device token is rate-limited (asymmetric with the dispatch-failure rollback) #170). The decrypt-failure path keeps its charge intentionally (documented replay defense) via explicit commit(). Builds on the reservation-aware rollback semantics from PR fix: make rate-limit rollback reservation-aware #206.
2. Filter before charging: the dispatcher exposes accepts(platform) / token-encoding checks so unconfigured-platform and non-UTF8 tokens are rejected before any limiter charge, with a real record_push_failed(platform, "unconfigured"/"invalid_encoding")-style drop metric ([LOW] A decrypted token dropped inside dispatch() (unconfigured platform / non-UTF8 FCM) keeps its rate-limit budget and is recorded by no metric #177).
3. Capacity path off the global lock: shard the limiter (N × RwLock<LruCache> by key hash) and/or replace the 1000-entry scan with small-K candidate sampling so a full cache costs bounded work per check ([MEDIUM] RateLimiter::check_and_increment runs an O(1000) eviction scan while holding the global write lock, serializing all rate-limit checks under cache-cardinality pressure #123).
4. Truthful gauges: the limiter publishes its real len() on insertion/eviction (throttled), not just at cleanup ([LOW] transponder_rate_limit_cache_size gauge is only updated during the 60s cleanup, never on insertion — cache saturation / capacity-limit onset is invisible for up to a minute #125) — extends the admission-eviction counters from PR fix: surface rate-limit admission evictions #210.
5. Single-critical-section dedup: fold try_reserve/mark_seen into one lock acquisition or reuse the reservation entry at completion ([LOW] Successful event path acquires the dedup write lock twice (try_reserve then mark_seen), doubling lock contention and gauge updates on the hot path #197).

Members

• [LOW] Encrypted-token rate-limit increment is never refunded when the paired device token is rate-limited (asymmetric with the dispatch-failure rollback) #170 — refund encrypted-token charge when device limiter rejects
• [LOW] A decrypted token dropped inside dispatch() (unconfigured platform / non-UTF8 FCM) keeps its rate-limit budget and is recorded by no metric #177 — pre-charge filtering + drop metrics for dispatch-level drops
• [MEDIUM] RateLimiter::check_and_increment runs an O(1000) eviction scan while holding the global write lock, serializing all rate-limit checks under cache-cardinality pressure #123 — bounded/sharded capacity eviction (no O(1000) scan under global write lock)
• [LOW] transponder_rate_limit_cache_size gauge is only updated during the 60s cleanup, never on insertion — cache saturation / capacity-limit onset is invisible for up to a minute #125 — cache-size gauge updated on growth, not only cleanup
• [LOW] Successful event path acquires the dedup write lock twice (try_reserve then mark_seen), doubling lock contention and gauge updates on the hot path #197 — single write-lock acquisition on the dedup success path

In-flight PRs (merge first — this tracker builds on their semantics and tests)

• PR fix: make rate-limit rollback reservation-aware #206 (fixes [LOW] RateLimiter::rollback_increment is identity-unaware (pop_back), so under concurrent processing of the same key it can remove the wrong hit #87) — reservation-aware rollback; its mechanism is superseded by AdmissionGuard but its tests carry forward.
• PR fix: keep per-token rate-limited events retryable #203 (fixes [MEDIUM] Event with all tokens shed by the per-token rate limiters is marked permanently "seen" and counted as Processed, inconsistent with the transient-shed semantics of the global limiter #195) — all-tokens-shed events stay retryable; the guard design must preserve this.
• PR fix: surface rate-limit admission evictions #210 (fixes Observability: surface rate-limiter admission-pressure evictions + ExceededCapacityLimit rate #81) — admission-eviction/capacity counters; extend, don't rename.
• [LOW] RateLimiter::rollback_increment is identity-unaware (pop_back), so under concurrent processing of the same key it can remove the wrong hit #87 — identity-aware rollback (point fix in flight)
• [MEDIUM] Event with all tokens shed by the per-token rate limiters is marked permanently "seen" and counted as Processed, inconsistent with the transient-shed semantics of the global limiter #195 — shed ≠ seen/Processed (point fix in flight)
• Observability: surface rate-limiter admission-pressure evictions + ExceededCapacityLimit rate #81 — admission-pressure observability (point fix in flight)

Sequencing

Starts after the config tracker's typed NonZeroUsize configs land (soft dependency). Overlaps the push-provider tracker only in dispatcher.rs, in different functions (dispatch() filtering loop vs send_push) — safe to run in parallel with minor merge friction.

[erskingardner]

Completed: all member issues resolved via PR #234 (merged to master). See roadmap #222.

[erskingardner]

V2 follow-up trackers opened: #283 for event admission/replay semantics and #287 for post-sharding rate-limiter behavior. Keeping this V1 tracker closed.