Tracking: unify APNs/FCM push-provider machinery — shared auth cache + rich PushOutcome

[erskingardner]

Root cause

ApnsClient and FcmClient are ~95%-identical copy-pastes that have already diverged, and the send path collapses every outcome into Result<bool>, destroying the information downstream consumers need.

• Duplicated auth machinery: identical CachedToken { token: Zeroizing<String>, expires_at } behind Arc<RwLock<Option<CachedToken>>> with the same double-checked read/write-lock pattern in src/push/apns.rs:206-389 and src/push/fcm.rs:214-455. The copies have already drifted (transport-retry budgets, log levels, FCM-only aud/endpoint mismatch) — a fix in one provider does not reach the other.
• Lossy outcome type: SendAttemptResult::Success(bool) conflates "provider says the token is dead" with "retries exhausted" (with_retry returns Ok(false) on exhaustion, src/push/retry.rs:197-204) and "unknown status" (APNs catch-all _ => Success(false), src/push/apns.rs:495-502). The dispatcher then labels every Ok(false) as reason="invalid_token" (src/push/dispatcher.rs:258-293) — the one metric whose downstream use (dead-token pruning) is destructive.

Every member issue below is a symptom of one of these two defects. Fixing them individually re-patches the same duplicated code paths; fixing the architecture closes them all.

Refactor design

New src/push/auth.rs — shared credential machinery:

• trait AuthTokenGenerator { async fn mint(&self) -> Result<MintedToken>; } with MintedToken { token: Zeroizing<String>, expires_at: SystemTime }.
  • FCM generator: honors provider expires_in instead of the hardcoded 50-min TTL ([MEDIUM] FCM caches OAuth access token for a hardcoded 50 min, ignoring the server-provided expires_in #88); POSTs to sa.token_uri so the request target matches the signed aud ([LOW] FCM OAuth JWT aud is signed with service_account.token_uri but the request is always POSTed to the hardcoded OAUTH_TOKEN_URL — a non-standard token_uri causes a silent total FCM outage #153); reads the token response body with a byte cap ([LOW] FCM OAuth token success response body is read with unbounded response.json() #154).
  • APNs generator: local ES256 signing, unchanged semantics.
• TokenCache<G: AuthTokenGenerator>: read-lock fast path; refresh is single-flight and runs outside the write lock so concurrent senders keep using the still-valid token during a refresh ([LOW] FCM access-token refresh holds the cache write lock across the OAuth HTTP round-trip, stalling all concurrent senders during refresh #86); invalidate_if(reason) gates eviction on parsed auth-reject reasons (ExpiredProviderToken / InvalidProviderToken), not blanket HTTP 403 — ends the per-push ES256 re-sign stampede under BadCertificateEnvironment-class 403s ([MEDIUM] APNs evicts the cached JWT on every HTTP 403 reason, causing a per-notification ES256 re-sign stampede under non-JWT 403s (e.g. BadCertificateEnvironment) #145).

Rich outcome enum, replacing Result<bool> / Success(bool) end-to-end:

enum PushOutcome {
    Delivered,
    TokenInvalid(InvalidReason),        // only provider-confirmed dead tokens
    RetriesExhausted { last_status: u16 },
    Shed(ShedReason),                   // shutdown, etc.
    Failed(ProviderError),              // unexpected status, config fault, transport
}

• APNs unexpected statuses (404/405/413/…) map to Failed(UnexpectedStatus), never token-invalid ([LOW] APNs unexpected-status catch-all (_ => Success(false)) misclassifies server/config errors (404/405/413/…) as a dead device token #169); FCM/APNs asymmetry removed.
• Dispatcher matches exhaustively → failure metrics get truthful labels (retries_exhausted, unexpected_status, …) instead of blanket invalid_token ([LOW] Dispatcher labels every Ok(false) as reason="invalid_token", conflating dead tokens with retries-exhausted and unknown-status failures (and dropping retriable 408s) #90).
• Metric layering fixed with the same change: per-attempt counters stay in handle_response; per-logical-push duration/status recorded once in send() after retries resolve, so histogram counts equal push volume ([LOW] push_duration and push_response_status are recorded once per HTTP attempt, so retried pushes over-count samples (attempts ≠ logical pushes) #168).
• Provider errors are passed through reqwest::Error::without_url() (or an equivalent redacting wrapper) before reaching any log sink, so the APNs device-token-bearing URL can never leak at WARN/DEBUG ([MEDIUM] APNs device token leaks into WARN/DEBUG logs via the reqwest::Error URL on transport failures #172 — acceptance test owned by the secret-hygiene tracker).
• Both providers build the request once per send as a template borrowed by the retry closure — removes FCM's per-attempt HashMap/FcmRequest/token-string rebuild ([LOW] FCM rebuilds the entire request (HashMap + FcmRequest + device_token.to_string()) on every transport-retry attempt; APNs hoists it out of the retry closure — divergence + extra non-zeroized token copy #198) and the extra non-zeroized token copy; where feasible assemble the body via a zeroizing buffer (the [LOW] FCM/APNs copy the decrypted device token into non-zeroized String/JSON body inside the request builder, defeating the upstream Zeroizing wrapper #126 remainder — also tracked by the secret-hygiene tracker).
• Delete the unreachable is_valid_device_token branches (or demote to debug_assert!) and document TokenPayload::from_decrypted + the dispatcher's UTF-8 gate as the single source of truth for token well-formedness ([LOW] is_valid_device_token odd-length / non-hex / empty branches are unreachable on the production dispatch path — dead defensive code that reads as meaningful validation #199).

Members

• [LOW] Refactor: APNs and FCM duplicate their entire cached-token/auth machinery, and the copies have already diverged #158 — the refactor seed: extract shared cached-token/auth machinery
• [MEDIUM] APNs evicts the cached JWT on every HTTP 403 reason, causing a per-notification ES256 re-sign stampede under non-JWT 403s (e.g. BadCertificateEnvironment) #145 — reason-gated JWT invalidation (no eviction on non-JWT 403s)
• [MEDIUM] FCM caches OAuth access token for a hardcoded 50 min, ignoring the server-provided expires_in #88 — honor OAuth expires_in instead of hardcoded 50 min
• [LOW] FCM access-token refresh holds the cache write lock across the OAuth HTTP round-trip, stalling all concurrent senders during refresh #86 — OAuth refresh outside the cache write lock (single-flight)
• [LOW] FCM OAuth JWT aud is signed with service_account.token_uri but the request is always POSTed to the hardcoded OAUTH_TOKEN_URL — a non-standard token_uri causes a silent total FCM outage #153 — POST to sa.token_uri so aud and endpoint agree
• [LOW] FCM OAuth token success response body is read with unbounded response.json() #154 — bounded read of the OAuth success body
• [LOW] FCM rebuilds the entire request (HashMap + FcmRequest + device_token.to_string()) on every transport-retry attempt; APNs hoists it out of the retry closure — divergence + extra non-zeroized token copy #198 — build FCM request once, outside the transport-retry closure
• [LOW] APNs unexpected-status catch-all (_ => Success(false)) misclassifies server/config errors (404/405/413/…) as a dead device token #169 — APNs catch-all must not classify server faults as dead tokens
• [LOW] Dispatcher labels every Ok(false) as reason="invalid_token", conflating dead tokens with retries-exhausted and unknown-status failures (and dropping retriable 408s) #90 — dispatcher label taxonomy from exhaustive PushOutcome match
• [LOW] push_duration and push_response_status are recorded once per HTTP attempt, so retried pushes over-count samples (attempts ≠ logical pushes) #168 — per-attempt vs per-push metric separation
• [LOW] is_valid_device_token odd-length / non-hex / empty branches are unreachable on the production dispatch path — dead defensive code that reads as meaningful validation #199 — remove dead token-validation branches; single source of truth
• [MEDIUM] APNs device token leaks into WARN/DEBUG logs via the reqwest::Error URL on transport failures #172 — (rider, tracked by secret-hygiene tracker) redact provider errors before logging
• [LOW] FCM/APNs copy the decrypted device token into non-zeroized String/JSON body inside the request builder, defeating the upstream Zeroizing wrapper #126 — (rider, remainder after PR fix: zeroize derived device token strings #205; tracked by secret-hygiene tracker) zeroizing request body

In-flight PRs

• PR fix: retry transient FCM OAuth token failures #209 (fixes [MEDIUM] FCM OAuth token-acquisition failures are classified Permanent — a transient Google OAuth blip drops every FCM push (retry bypassed) #83, retry transient OAuth failures): merge first — its logic lives in refresh_token_inner, which lifts wholesale into the FCM AuthTokenGenerator.
• PR fix: retry push auth rejections once #207 (fixes [MEDIUM] APNs 403 / FCM 401 invalidates the cached auth token but returns Permanent for the current request — drops one notification per device instead of retrying with a fresh token #85, retry once after auth invalidation): closed as superseded — this control flow is exactly what PushOutcome + TokenCache's retry-once policy expresses. Its tests are spec material for this tracker; [MEDIUM] APNs 403 / FCM 401 invalidates the cached auth token but returns Permanent for the current request — drops one notification per device instead of retrying with a fresh token #85 stays open as a member here.
• [MEDIUM] APNs 403 / FCM 401 invalidates the cached auth token but returns Permanent for the current request — drops one notification per device instead of retrying with a fresh token #85 — retry the in-hand notification once after auth invalidation (via the new outcome/cache design)

Sequencing

• Wave 1; parallel with the config, lifecycle, secret-hygiene, NIP-59, and health trackers (disjoint files except dispatcher.rs, where the admission tracker touches different functions).
• The retry-policy tracker (Retry-After cap, live-task bound) is hard-blocked behind this one — same retry.rs machinery.

[erskingardner]

Completed: all member issues resolved via PR #233 (merged to master). See roadmap #222.

[erskingardner]

V2 follow-up tracker opened: #284. Keeping this V1 tracker closed; the new issue owns residual push-provider outcome, OAuth minting, retry-residency, and delivery-observability gaps.