Tracking: retry/backoff policy — Retry-After cap + live-task bound

[erskingardner]

Root cause

Retry policy decisions are scattered across retry.rs constants, header parsing, and permit mechanics, and two of them break the system's own stated invariants under provider stress:

• A provider-supplied Retry-After is honored verbatim with no ceiling (retry_wait_duration, src/push/retry.rs:219-224) — an untrusted header can pin a send task (holding a decrypted device token, and an InFlightGuard that blocks drain) for an arbitrary duration ([LOW] Provider-supplied Retry-After is honored verbatim with no upper bound — an untrusted header can pin a send task (and its decrypted token) for an arbitrary duration #162).
• BackoffPermit::release() frees the concurrency slot during backoff sleeps while the task stays alive, so under a provider 429/5xx storm the "max 100 concurrent sends" bound silently becomes "up to the full 10k queue of live tasks, each holding a decrypted token, thundering-herd re-competing on reacquire" ([MEDIUM] Backoff permit release lets live send-task (and decrypted-token) count exceed the 100 concurrency bound up to the full 10k queue #160).

Refactor design

On top of the push-provider tracker's rewritten retry engine (hard dependency — same retry.rs machinery):

1. Cap honored Retry-After at a small policy ceiling (e.g. 60–120s): server_delay.min(MAX_RETRY_AFTER) in retry_wait_duration/parse_retry_after ([LOW] Provider-supplied Retry-After is honored verbatim with no upper bound — an untrusted header can pin a send task (and its decrypted token) for an arbitrary duration #162).
2. Bound total live tasks (active + sleeping-in-backoff) independently of active-HTTP concurrency: a second semaphore gating task spawn/dequeue, or a cap on how many tasks may hold a released permit at once — so backoff can no longer balloon live decrypted-token residency to queue size ([MEDIUM] Backoff permit release lets live send-task (and decrypted-token) count exceed the 100 concurrency bound up to the full 10k queue #160).
3. Carry forward PR fix: desynchronize push retry backoff #204's floor + jitter as first-class policy in the new engine (single RetryPolicy struct owning floor, cap, jitter, budget), so the lower bound ([LOW] Retry backoff has no minimum floor and no jitter: Retry-After: 0 yields immediate back-to-back retries and concurrent sends retry in synchronized waves #96), upper bound ([LOW] Provider-supplied Retry-After is honored verbatim with no upper bound — an untrusted header can pin a send task (and its decrypted token) for an arbitrary duration #162), and concurrency interaction ([MEDIUM] Backoff permit release lets live send-task (and decrypted-token) count exceed the 100 concurrency bound up to the full 10k queue #160) live in one reviewed place.

Members

• [LOW] Provider-supplied Retry-After is honored verbatim with no upper bound — an untrusted header can pin a send task (and its decrypted token) for an arbitrary duration #162 — upper-bound the honored Retry-After
• [MEDIUM] Backoff permit release lets live send-task (and decrypted-token) count exceed the 100 concurrency bound up to the full 10k queue #160 — bound live send tasks during backoff

In-flight PRs

• PR fix: desynchronize push retry backoff #204 (fixes [LOW] Retry backoff has no minimum floor and no jitter: Retry-After: 0 yields immediate back-to-back retries and concurrent sends retry in synchronized waves #96, backoff floor + jitter): merge first; this tracker absorbs its policy into the unified RetryPolicy.
• [LOW] Retry backoff has no minimum floor and no jitter: Retry-After: 0 yields immediate back-to-back retries and concurrent sends retry in synchronized waves #96 — backoff floor + jitter (point fix in flight)

Sequencing

Wave 2 — hard-blocked behind the push-provider tracker (both rewrite with_retry/BackoffPermit in retry.rs). The optional dispatcher InFlightTracker → TaskTracker unification (from the lifecycle tracker) also waits for this.

[erskingardner]

Completed: #162 and #160 resolved via PR #233 (merged; retry policy was folded into the push-provider unification since both rewrite retry.rs). See roadmap #222.

[erskingardner]

V2 follow-up tracker opened: #284. Keeping this V1 tracker closed; the new issue owns residual retry-residency and backoff observability questions after the first retry-policy wave.