[Due for payment 2026-08-18] [Performance] App hangs when OpenApp / ReconnectApp / GetMissingOnyxMessages fire in rapid stampedes

[mountiny]

If you haven't already, check out our contributing guidelines.

Version Number: Multiple production versions observed (e.g. 9.3.54–9.3.64)
Reproducible in staging?: Unknown
Reproducible in production?: Yes
Email or phone of affected tester (no customers): N/A — reproduced via production telemetry, not a single test account
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL: https://github.com/Expensify/Expensify/issues/644858
Issue reported by: Internal engineering investigation

What performance issue do we need to solve?

The JS thread is blocked for 10+ seconds (dead clicks / fully unresponsive UI) when large OpenApp, ReconnectApp, and GetMissingOnyxMessages responses arrive in rapid succession. Sentry traces show this drives a storm of Onyx updates and deep equality work across subscribers.

Symptom tracking (aggregate): https://expensify.sentry.io/issues/7323151341/ (~14k events / 90 days)

What is the impact on end-users?

• App appears frozen after navigation, reconnect, or opening reports/search/settings
• Affects iOS, Android, Windows, and Mac (not platform-specific)
• Both new and established accounts

Benchmarks / evidence (from production traces)

Payload sizes observed: ~0.5 MB – 4 MB per response on sessions that should be lightweight.

Pattern A — Same-endpoint stampede

One session fires the same command many times in minutes:

• 40× ReconnectApp in ~10 min (iOS 9.3.54-7, scan flow, every 8–30s) — https://expensify.sentry.io/explore/traces/trace/6c175d912b7d4bf5b238d131fdcd10a7
• 21× ReconnectApp in ~5 min (iOS, route change) — https://expensify.sentry.io/explore/traces/trace/688bf8372e4744f3b9fc57afb30eda21
• 13× GetMissingOnyxMessages in ~28s (iOS, Report page) — https://expensify.sentry.io/explore/traces/trace/61d5b4e4d45f4fea979548f86352c912
• 20× ReconnectApp in ~10s (Windows, ~146 KB each, ~2 calls/sec) — https://expensify.sentry.io/explore/traces/trace/298128ff3d274b70860b2060c9fc3581
• 27× ReconnectApp in ~9s (Windows, /settings/profile/contact-methods) — https://expensify.sentry.io/explore/traces/trace/e3d86d00ed3148bf85d85162c4218e57

Pattern B — Byte-identical duplicate responses (≤3s apart)

Server returns the same payload twice; client processes duplicate Onyx data:

• 2× ReconnectApp, 516,004 / 516,004 bytes identical, 1s apart (Windows) — https://expensify.sentry.io/explore/traces/trace/61bc0c35ab24458fac3e76592dbcc28b
• 2× OpenApp, ~1 MB each, ~90s apart (iOS) — https://expensify.sentry.io/explore/traces/trace/44c8c38f52a941f7bf6cd4549fb1bbfd
• 4× byte-identical ~224 KB ReconnectApp in 2s on /search (Windows) — https://expensify.sentry.io/explore/traces/trace/9d20e02d47d047aeaad3f0afcc82fa1d

Pattern C — Cross-command overlap (≤10s)

• GetMissingOnyxMessages 475 KB → ReconnectApp 475 KB, 1s apart (Mac, Report) — https://expensify.sentry.io/explore/traces/trace/1d72808fd7c8414e8b86bdc38d349ed1
• ReconnectApp → GetMissingOnyxMessages, byte-identical 250,893 B, 1s apart (Windows, Home) — https://expensify.sentry.io/explore/traces/trace/da50a56c2ec4443685eada0f86e486e2

Example hang timeline (iOS HybridApp 9.3.64-31)

1. OpenApp on Home — https://expensify.sentry.io/explore/traces/trace/940b5545caec4d5ba1ac8ce1c0b8738b
2. ReconnectApp on Onboarding_Purpose ~32s later — https://expensify.sentry.io/explore/traces/trace/0df88e7f01bc4a1ea048f491b87f021d
3. APP HANG ~52s after open — https://expensify.sentry.io/issues/7323151341/events/8677a76e805d4948bd37c2412b90e3fd/

Proposed investigation / solution direction

Open questions (from internal triage):

1. Why are payloads so large? — Possible whole-collection responses; correlate server output.size in logs for OpenApp / ReconnectApp.
2. Why are commands fired repeatedly? — Network flapping vs uncoordinated client callers (e.g. subscribeToFullReconnect.ts and other call sites).
3. Sentry ↔ server correlation — Confirm requestID still appears in HTTP breadcrumbs for recent sessions.
4. Delegate sessions — Check overlap with known GetMissingOnyxMessages over-fetch in delegate mode (PR in review internally).
5. Mitigation — Client in-flight dedup by updateIDFrom/updateIDTo may help, but root cause (why duplicate/large calls happen) is priority.

Sentry traces above are shareable; customer emails and FullStory remain on the internal issue only.

Platforms

• iOS: App
• Android: App (inferred from cross-platform scope)
• Windows: Chrome
• MacOS: Chrome / Safari

Issue Owner

Current Issue Owner: @mallenexpensify

[MelvinBot]

Issue Analysis

Root cause (client side): The hang is amplified by two missing dedup guards in App — but the trigger (why duplicate/large responses are produced at all) is server side and needs an internal backend investigation.

1. Stampede amplifier (Pattern A) — no in-flight dedup. ReconnectApp/OpenApp/GetMissingOnyxMessages have ~6 uncoordinated trigger paths (network reachability, app-foreground, Pusher multi-event, auth-screen mount, subscribeToFullReconnect, delegate/priority flows) with no throttle, debounce, or in-flight guard. The only dedup — writeWithNoDuplicatesConflictAction → RequestConflictUtils.ts:57-73 — matches only against the queued requests returned by PersistedRequests.getAll(), which excludes the in-flight ongoingRequest. So while one ReconnectApp is mid-flight, every new trigger pushes a fresh copy that fires the instant the in-flight one returns → N calls per stampede.

2. Duplicate-apply amplifier (Patterns B & C) — large payloads reapplied in full. OnyxUpdates.apply() has an isUpdateOld guard that normally skips reapplying onyxData when lastUpdateID <= lastUpdateIDAppliedToClient — but it is explicitly bypassed for OpenApp and full ReconnectApp at OnyxUpdates.ts:133-136 (these commands also skip gap-detection via the bypass set in SaveResponseInOnyx.ts:12-18). Result: a byte-identical 0.5–4 MB duplicate OpenApp/full-ReconnectApp payload is written to Onyx again in full, firing subscriber callbacks + deep-equality work across every subscriber — the 10 s+ JS-thread block.

Recommended fix (App-side mitigation): These two guards are exactly the "client in-flight dedup" the issue calls out. They're implementable in App and will collapse most of the storm, but they are a mitigation, not the root cause — they reduce the cost of duplicate/large responses without stopping them from being produced.

• In-flight dedup: make the duplicate-conflict matcher also consult ongoingRequest (or add an in-flight no-op guard in reconnectApp/openApp) so a second identical command attaches to the in-flight one instead of queueing a fresh copy.
• Duplicate-payload guard: for OpenApp/full-ReconnectApp, skip reapplying onyxData when lastUpdateID hasn't advanced past lastUpdateIDAppliedToClient (revisit the bypass at OnyxUpdates.ts:133-134) so byte-identical duplicates don't re-storm Onyx.

Needs internal backend investigation (outside the App repo): The priority questions from your triage — (1) why payloads are 0.5–4 MB (whole-collection responses?) and (2) why the server returns byte-identical duplicates ≤3 s apart — live in Auth/Web-Expensify and can't be diagnosed or fixed from App. These should be escalated to an internal engineer with backend access; correlating server output.size and requestID against the Sentry traces is the next step there.

Investigation details & evidence

In-flight dedup gap (Pattern A):

• reconnectApp() and openApp() have no in-flight flag/Promise guard; only conditionals are hasLoadedApp (init) and isUsingImportedState. getMissingOnyxUpdates() uses makeRequestWithSideEffects with no conflict checker at all.
• Trigger paths with no dedup: Reconnect.ts (network-restored and app-foreground), User.ts (Pusher multi-event), AuthScreensInitHandler.tsx (per-mount), subscribeToFullReconnect.ts.
• Partial flap protection exists only on the sustained-failure path: a 0–5 s RECONNECT_STAMPEDE_JITTER_MS jitter in NetworkState.ts:127-147, whose comment even asserts "Duplicate reconnectApp() calls are safe — SQ deduplicates them" — which is false for the in-flight case because of the getAll()/ongoingRequest gap above.
• The flow: first call → queue empty → push → processNextRequest moves it to ongoingRequest, leaving persistedRequests empty → next call sees empty queue → pushes another fresh copy → repeats.

Duplicate-apply gap (Patterns B & C):

• The isUpdateOld guard (OnyxUpdates.ts:132-159) strips onyxData for ordinary duplicates but exempts OpenApp and full ReconnectApp (no updateIDFrom). When not exempted-but-old, only successData/failureData are reapplied; when exempted, the full onyxData is handed to Onyx.update(...).
• There is no content-hash / requestID / "already applied" dedup anywhere in this pipeline — the only dedup key is lastUpdateID, and it's intentionally disabled for the two biggest-payload commands.
• GetMissingOnyxMessages / partial ReconnectApp duplicates can be caught by the guard, but only if the async lastUpdateIDAppliedToClient cache (OnyxUpdates.ts:17-23) has propagated before the duplicate is processed — a race when two land ≤3 s apart.

Alternative explanations considered: Could the hang be pure render cost from one large legitimate payload (no duplication)? Partly — a single 4 MB OpenApp is expensive regardless. But that doesn't explain the repeated traces showing byte-identical duplicates and 20–40× same-command bursts, which the two gaps above directly account for. The deep-equality "storm across subscribers" in the Sentry traces is consistent with full-payload reapplication, not a one-shot apply.

Confidence: High that both client gaps exist and amplify the hang (verified at file:line). Medium on attribution — whether these client amplifiers vs. the server over-fetch/duplication are the dominant driver needs the log correlation below.

Logs that would confirm dominance (internal): count, per affected session, [OnyxUpdateManager] Applying update type: https with lastUpdateID: <N> (OnyxUpdates.ts:130) for the same lastUpdateID within 3 s broken out by command; and [SequentialQueue] Conflict action determined with command: ReconnectApp and conflictType: push vs replace. Correlate against server ReconnectApp/OpenApp output.size and requestID.

---

Next Steps: Reply with @MelvinBot implement this to draft a PR for the two App-side mitigations (in-flight dedup + duplicate-payload guard), or @MelvinBot <feedback> to refine. The backend over-fetch/duplication questions (priority root cause) need an internal engineer with Auth/Web-Expensify access.

[melvin-bot]

@adhorodyski Eep! 4 days overdue now. Issues have feelings too...

[adhorodyski]

First, I wrote down this PRD which sums up the problem I saw in more detail, it was AI-assisted and some context might be off for the model but issues are real. This includes proposed solutions.

---

Reconnect Storm Prevention — PRD

Status: Draft · Date: 2026-06-05 · Branch: audit-reconnect-patterns
Source investigation: project_reconnect_app_storm_investigation (memory)

Problem Statement

Users experience "dead clicks" and app hangs — taps that do nothing, a UI frozen for seconds to minutes. Over the last 90 days there are ~14k such reports. The hangs trace to the JS thread being saturated by a flood of large and/or rapid HTTP calls in the reconnect family: ReconnectApp, OpenApp, and GetMissingOnyxMessages.

Production telemetry (Sentry, prod host, 14 days) shows the scale of the storm:

• ReconnectApp: 816,044 calls. p50 328ms, p95 2,223ms, max 544,963ms (9 minutes). One trace fired 31 ReconnectApp calls in 9 seconds (~3.4/s), each ~285ms, all succeeding fast.
• GetMissingOnyxMessages: 486,535 calls. p50 185ms, p95 1,744ms, max 81,676ms (82s). One trace fired 28 calls in 9 seconds with zero ReconnectApp.
• OpenApp: 213,265 calls. p50 329ms, p95 1,325ms, max 382,956ms (6.4min). Bursts cap around 9.
• Response bodies confirmed up to 22.8 MB (also 13.8 / 9.4 / 8.2 / 7.4 MB) via http.response_content_length.

Total: ~1.5M reconnect-family calls per 14 days. The leadership ask (tgolen, ionatan) is explicit: find and fix the root cause of why so many fire, not add another dedup band-aid.

The investigation isolated three independent failure modes plus one amplifier:

1. ReconnectApp loop — a self-re-arming full-reconnect trigger.
2. GetMissingOnyxMessages loop — the OnyxUpdateManager gap-detection path re-firing, bypassing the SequentialQueue entirely so nothing dedups it.
3. Slow giants — single 7–22MB full-payload reconnects stalled on a flaky network or thermally/memory-constrained device, hanging the thread for tens of seconds to minutes. Not a loop.
4. Amplifier — the SequentialQueue no-duplicates resolver only scans the waiting queue, so a duplicate enqueued while an identical request is in flight is never collapsed. This lets every loop run unthrottled.

Solution

From the user's perspective: the app stops freezing. Taps register. Coming back online, returning from background, or receiving a server "please reconnect" signal triggers at most one reconnect cycle, not a storm. A slow reconnect on a bad network degrades gracefully instead of locking the UI.

We achieve this by closing the re-arm hazards at each loop's source and closing the dedup gap that lets duplicates escape:

• Mode 1 — Make the full-reconnect trigger idempotent: it must not re-arm itself off its own response, and it must not fire a second reconnect while one is already in flight.
• Mode 2 — Give the missing-updates fetch a real in-flight guard so concurrent gap detections collapse onto a single outstanding request.
• Amplifier — Extend the no-duplicates resolver to also consider the in-flight (ongoing) request, so duplicate ReconnectApp / OpenApp / GetMissingOnyxMessages collapse against it.
• Mode 3 — Add client-side resilience for the slow-giant case (single-flight + observability), and file the payload-size reduction as a server-side dependency.

The unifying principle: every reconnect entry point converges on a single in-flight cycle. A trigger that fires while a cycle is active is a no-op (or a coalesced replace), never a new request.

---

Root cause detail (code-verified)

Mode 1 — the subscribeToFullReconnect re-arm hazard. subscribeToFullReconnect.ts subscribes to two Onyx keys — LAST_FULL_RECONNECT_TIME and NVP_RECONNECT_APP_IF_FULL_RECONNECT_BEFORE — and on any change calls doFullReconnectIfNecessary(), which fires reconnectApp() whenever lastFullReconnectTime < reconnectAppIfFullReconnectBefore. There is no in-flight guard and no debounce.

The loop: every OpenApp / full-ReconnectApp response re-delivers NVP_RECONNECT_APP_IF_FULL_RECONNECT_BEFORE in response.onyxData. applyHTTPSOnyxUpdates applies response.onyxData before successData. The only writer of LAST_FULL_RECONNECT_TIME = now (the loop-breaker) is successData (App.ts:381-387, only on isOpenApp || isFullReconnect). So when the NVP callback runs, the timestamp is still stale → comparison says "reconnect needed" → fires another reconnectApp() → whose response re-delivers the NVP → repeat. The burst trace 7d45b533… (31 bare ReconnectApp in 9s, kicked off by Ping/PusherPing, zero GetMissingOnyxMessages) is this loop.

The team already patched the clear-Onyx / delegate / supportal paths by pre-seeding the timestamp (clearOnyxAndSeedFullReconnect.ts). The steady-state path (network reconnect, Pusher RECONNECT_APP) was never seeded or guarded.

Mode 2 — the OnyxUpdateManager gap re-detect. getMissingOnyxUpdates() calls API.makeRequestWithSideEffects, which bypasses the SequentialQueue, so there is no dedup at all. OnyxUpdateManager has partial guards (getMissingOnyxUpdatesQueryPromise() in the !lastUpdateIDFromClient branch; areDeferredUpdatesQueued || isFetchingForPendingUpdates in the gap branch) but the gap branch's guard is defeated under concurrent Pusher updates, producing 28-in-9s stampedes (trace ff14342c…).

Amplifier — SQ in-flight dedup gap. The no-duplicates resolver (resolveDuplicationConflictAction, in RequestConflictUtils.ts:57) runs findIndex over persistedRequests only — the waiting queue. processNextRequest() removes the in-flight request from that array (slice(1)) before sending. The deepEqual ongoing-guard (PersistedRequests.ts:150) misses it because the duplicate enqueued mid-flight has a different requestID. So a reconnect enqueued while another is in flight is never collapsed.

Mode 3 — slow giants. Trace a389b884…: a lone ReconnectApp of 544,963ms (9 min) running in parallel with an AuthenticatePusher of 544,926ms from the same instant — socket open, no progress. This is a network/device stall returning a 7–22MB full payload, matching tgolen's flaky-net reports and vit's thermal/low-mem reports. Distinct from the burst loops.

---

User Stories

1. As a user returning to the app from background, I want a single reconnect cycle, so that the UI is responsive instead of frozen while dozens of identical requests pile up.
2. As a user whose network just came back online, I want exactly one reconnect to fire, so that I don't trigger a storm of ReconnectApp calls.
3. As a user, I want the app to ignore its own reconnect response as a reason to reconnect again, so that I never enter a self-perpetuating reconnect loop.
4. As a user on a flaky connection, I want a slow in-progress reconnect to not be joined by 30 more, so that one slow request doesn't multiply into a multi-minute hang.
5. As a user, I want a "please reconnect" signal from the server (the NVP) to result in at most one reconnect, so that repeated delivery of that signal doesn't re-arm the loop.
6. As a user with the app open in multiple tabs, I want only the leader tab to drive reconnects, so that N tabs don't each fire their own storm.
7. As a user who just received several Pusher updates with update-ID gaps, I want a single GetMissingOnyxMessages fetch to cover them, so that I don't fire one request per gap detection.
8. As a user, I want a duplicate reconnect-family request enqueued while an identical one is in flight to be collapsed, so that the in-flight request isn't shadowed by a redundant follow-up.
9. As a user on a constrained device (low memory, thermal throttling), I want a stalled multi-megabyte reconnect to fail or time out gracefully, so that the JS thread is freed rather than held open indefinitely.
10. As a user, I want the app to never freeze on tap during reconnect, so that I can keep working offline-first as designed.
11. As an engineer, I want the full-reconnect decision logic extracted into a pure, unit-testable function, so that I can prove the re-arm hazard is closed without spinning up the whole app.
12. As an engineer, I want a regression test that reproduces the 31-in-9s burst and asserts it collapses to one call, so that the loop cannot silently return.
13. As an engineer, I want a test that concurrent gap detections produce a single GetMissingOnyxMessages, so that Mode 2 stays fixed.
14. As an engineer, I want a test that a duplicate command enqueued during an in-flight identical command is collapsed, so that the amplifier stays closed.
15. As an on-call engineer, I want reconnect triggers and suppressions logged with their reason, so that I can confirm in Sentry/logs that storms stopped after deploy.
16. As an on-call engineer, I want a Sentry-visible signal (log/metric) when a reconnect is suppressed by the in-flight guard, so that I can measure how often the guard fires in production.
17. As a product owner, I want the ~1.5M/14d reconnect-family call volume to drop materially, so that backend load and the dead-click report rate both fall.
18. As an engineer, I want the existing seed-the-timestamp pattern reused for the steady-state path, so that the fix is consistent with the already-patched clear/delegate/supportal paths.
19. As an engineer, I want the slow-giant payload-size reduction tracked as an explicit server-side dependency, so that the client-side mitigation isn't mistaken for a complete fix.
20. As a delegate user transitioning from OldDot, I want the existing seeded-reconnect behavior preserved, so that this change doesn't regress the already-fixed delegate/supportal paths.

Implementation Decisions

Module 1 — Full-reconnect decision (Mode 1)

Extract the decision out of the Onyx-subscription side effects in subscribeToFullReconnect into a pure, deep module with a tiny interface:

// pure — no Onyx, no I/O
function shouldTriggerFullReconnect(state: {
    lastFullReconnectTime: string;
    reconnectAppIfFullReconnectBefore: string;
    isReconnectInFlight: boolean;
}): boolean {
    if (state.isReconnectInFlight) {
        return false; // a cycle is already running — never stack a second
    }
    return state.lastFullReconnectTime < state.reconnectAppIfFullReconnectBefore;
}

The subscription file becomes a thin shell that owns the mutable state (the two timestamps + an isReconnectInFlight flag), calls the pure decision, and on a true result:

1. Seeds LAST_FULL_RECONNECT_TIME = now optimistically before firing, so the NVP re-delivered in the response can't re-arm the loop in the steady-state path (mirrors clearOnyxAndSeedFullReconnect).
2. Sets isReconnectInFlight = true, fires reconnectApp(), and clears the flag when the cycle settles (success or failure).

let isReconnectInFlight = false;

function doFullReconnectIfNecessary() {
    if (!shouldTriggerFullReconnect({lastFullReconnectTime, reconnectAppIfFullReconnectBefore, isReconnectInFlight})) {
        return;
    }
    isReconnectInFlight = true;
    Onyx.merge(ONYXKEYS.LAST_FULL_RECONNECT_TIME, DateUtils.getDBTime()); // optimistic seed / loop-breaker
    Log.info('Full reconnect triggered', false, {lastFullReconnectTime, reconnectAppIfFullReconnectBefore});
    reconnectApp().finally(() => { isReconnectInFlight = false; });
}

Decision: prefer the in-flight flag + optimistic seed over a time-based debounce. A debounce papers over the loop with a delay; the flag + seed removes the re-arm cause. reconnectApp() must return/expose a settle signal so the flag can be cleared (today it returns void on the writeWithNoDuplicatesConflictAction path — we thread a promise through, or resolve the flag off the successData/finallyData write).

Module 2 — Missing-updates single-flight guard (Mode 2)

getMissingOnyxUpdates bypasses the SequentialQueue, so it needs its own guard. Introduce an explicit single-flight wrapper keyed on "a missing-updates fetch is outstanding":

let missingUpdatesInFlight: Promise<void> | null = null;

function fetchMissingOnyxUpdatesOnce(from, to) {
    if (missingUpdatesInFlight) {
        return missingUpdatesInFlight; // coalesce concurrent gap detections
    }
    missingUpdatesInFlight = getMissingOnyxUpdates(from, to).finally(() => { missingUpdatesInFlight = null; });
    return missingUpdatesInFlight;
}

The OnyxUpdateManager gap branch routes through this wrapper instead of calling getMissingOnyxUpdates directly. This makes the areDeferredUpdatesQueued || isFetchingForPendingUpdates guard authoritative even under concurrent Pusher updates — the inferred guard-defeat path is closed structurally rather than by reasoning about ordering.

Note (honest gap): the exact guard-defeat sequence inside OnyxUpdateManager is inferred, not line-traced. The single-flight wrapper is defensive — it holds regardless of which intermediate guard leaks.

Module 3 — SequentialQueue ongoing-request dedup (Amplifier)

Close the gap so the no-duplicates resolver considers the in-flight request, not just the waiting queue. Two candidate shapes:

• Option A (resolver-level): pass the ongoing request into the dedup check so resolveDuplicationConflictAction can return noAction when the new request matches the ongoing one. Requires writeWithNoDuplicatesConflictAction to surface getOngoingRequest() to the matcher.
• Option B (queue-level): in PersistedRequests.save(), after the waiting-queue dedup, also compare the incoming request against ongoingRequest by command + matcher (not requestID, not deepEqual — the in-flight copy has a different requestID).

Decision: Option B, scoped to the reconnect-family commands (RECONNECT_APP, OPEN_APP) that use the no-duplicates conflict action, to avoid changing dedup semantics for unrelated commands. The matcher must compare on command identity, since the whole point is that requestID/deepEqual differ.

Module 4 — Slow-giant resilience (Mode 3, client side)

• A reconnect already in flight (via Module 1's flag) prevents new reconnects from stacking on top of a stalled giant — this is the primary client mitigation and it falls out of Module 1.
• Add observability: log/emit when a reconnect exceeds a threshold (e.g. p95) so the slow-giant population is measurable distinctly from the loops.
• Server-side dependency (out of scope for this PRD's code): reduce full-payload size / stream incremental updates so a single reconnect can't return 7–22MB. File as a follow-up; the client fix alone does not eliminate the multi-minute hang on a true stall.

Cross-cutting decisions

• Leader-gating: reconnect triggers are currently not leader-gated (NetworkState is per-tab); only SequentialQueue.flush() is. Mode 2's path already leader-gates inside OnyxUpdateManager (isClientTheLeader()). For Mode 1, evaluate adding leader-gating to doFullReconnectIfNecessary so N tabs don't each seed+fire. The NVP and timestamp are shared Onyx keys, so the optimistic seed already dampens cross-tab fan-out, but explicit leader-gating is the clean fix. (Multi-tab is a contributing factor to the exact 31-count, which remains not fully explained.)
• Reuse the seed pattern: the steady-state seed reuses the exact reasoning documented in clearOnyxAndSeedFullReconnect.ts (response.onyxData applies before successData). Keep the comment lineage so the why is preserved.
• No manual memoization / no extracted JSX: N/A here (non-UI modules), but standard project rules apply to any touched component.

Testing Decisions

A good test here asserts external behavior — "how many requests fired" / "was the duplicate collapsed" — not internal flag values. Tests drive the public entry points (Onyx key changes, network transitions, enqueue calls) and assert on the resulting API calls via the existing API/network mocks. Prior art: the existing SequentialQueue and PersistedRequests test suites, and OnyxUpdateManager tests, already mock API and assert on emitted commands — follow that style.

Tests to write (all chosen modules):

1. Full-reconnect decision module (shouldTriggerFullReconnect, pure):

   • Returns true when stale timestamp < NVP and no reconnect in flight.
   • Returns false when a reconnect is in flight (the re-arm guard).
   • Returns false when timestamp ≥ NVP.
   • Integration: simulate an OpenApp response re-delivering the NVP with a stale timestamp → assert exactly one reconnectApp() fires (reproduces the 31-in-9s burst, asserts it collapses to 1). This is the headline regression test for the leadership ask.
   • Optimistic seed: assert LAST_FULL_RECONNECT_TIME is written before the request, so a re-delivered NVP in the same response does not re-trigger.

2. Missing-updates single-flight guard:

   • Two concurrent gap detections → assert a single GetMissingOnyxMessages request.
   • After the in-flight fetch settles, a subsequent genuine gap → assert a new fetch is allowed (guard releases).

3. SQ ongoing-request dedup:

   • Enqueue a RECONNECT_APP, mark it in flight (processNextRequest), enqueue an identical RECONNECT_APP → assert the second is collapsed (noAction) and not sent twice.
   • Control: a different command enqueued during an in-flight RECONNECT_APP is not collapsed (no over-dedup regression).

4. Slow-giant / observability:

   • With a reconnect in flight, a new reconnect trigger is suppressed (covered by Module 1's in-flight test; assert no second request even when the first is artificially slow/pending).
   • Threshold log/metric emits when a (mocked) reconnect exceeds the duration threshold.

Verification & Success Metrics

Principle: every fix is provable against production numbers before and after. A fix is not "done" because the code looks right — it is done when a specific Sentry query that showed the storm before deploy shows it gone after. We cannot claim "the API attack is over" from aggregate volume drift alone (volume moves for many reasons); each fix therefore emits its own suppression counter so we can attribute the drop directly to the guard that caused it.

Measurement protocol

1. Pin the baseline. Snapshot the queries below over a fixed 14-day window before rollout (the numbers in this PRD are that snapshot: prod host, 14d, captured 2026-06). Record exact query + window so post-deploy is an apples-to-apples re-run.
2. Roll out behind attribution. Each guard logs both when it fires a reconnect and when it suppresses one. Even if all four fixes ship together, the per-guard suppression counts attribute the storm reduction to the specific fix — no feature flag required for attribution (though staged rollout is still preferred for blast-radius safety).
3. Re-run identical queries at 7d and 14d post-rollout. Compare against the pinned baseline. The burst-signature queries are the bright line: they must go to zero.

Per-fix metric contract

Fix	Baseline (14d, prod)	Target	Primary query / signal
Mode 1 — ReconnectApp loop	816,044 calls; max 31/trace, ~3.4/s in burst trace 7d45b533…	Max ReconnectApp-per-trace within any 10s window ≤ 2; bursts (≥5/trace) → 0; overall call volume materially down	span.description:*ReconnectApp* aggregated count per trace_id; assert the distribution's tail collapses. Plus Full reconnect suppressed (in-flight) log count > 0 in the storm population.
Mode 2 — GetMissingOnyxMessages loop	486,535 calls; max 28/trace, burst trace ff14342c…	Max GMOM-per-trace within any 10s window ≤ 2; bursts → 0	span.description:*GetMissingOnyxMessages* count per trace_id. Plus Missing-updates fetch coalesced (single-flight) log count.
Amplifier — SQ ongoing dedup	In-flight duplicates never collapsed (0 collapses logged today)	Duplicate reconnect-family requests collapsed against ongoing > 0; contributes to Mode 1/2 burst collapse	Collapsed duplicate against ongoing request log count, broken down by command.
Mode 3 — slow giants	max 544,963ms (9min); payloads to 22.8MB; trace a389b884… had reconnect ‖ AuthenticatePusher hanging from same instant	Concurrent reconnects from the same instant → ≤ 1 (no stacking on a stall). Duration p95/p99/max and payload size are server-dependent — tracked, not committed	Count of reconnects with overlapping time ranges per trace; http.response_content_length distribution; Slow reconnect exceeded threshold log count.

North-star metrics (end-to-end)

• Dead-click / app-hang reports: ~14k / 90d baseline → target a material reduction over the 90d post-rollout window. This is the user-facing proof.
• Total reconnect-family call volume: ReconnectApp + GetMissingOnyxMessages + OpenApp ≈ 1.5M / 14d baseline → target a material drop. Watch that OpenApp doesn't absorb the reduction (a reconnect downgraded to openApp is still a giant).

Instrumentation requirements (implementation must add these)

• Module 1 emits a structured log on every full-reconnect fire and every suppression by the in-flight guard, with the decision inputs (lastFullReconnectTime, reconnectAppIfFullReconnectBefore, isReconnectInFlight). Reuse the existing Log.info('Full reconnect triggered', …) line and add a paired Full reconnect suppressed (in-flight) line.
• Module 2 emits a log when a gap detection is coalesced onto an in-flight fetch.
• Module 3 (amplifier) emits a log on each ongoing-request collapse, tagged with command, so we count amplifier activations independently.
• Module 4 emits a log when a reconnect exceeds a duration threshold (≈ current p95, 2.2s, or a round 10s — TBD), to size the slow-giant population separately from the loops.
• All four logs must carry enough cardinality to be count-aggregated in Sentry by command and grouped per trace.

[adhorodyski]

I'd just treat the above as context (it's too content heavy for a human I) and the implementation might differ, but even proposed solutions tackle the real problem, we're missing a single source of truth for when/when to not reconnect, it's not a single module that owns this.

[adhorodyski]

Also, here's a verification session I ran for this later:

---

Reconnect Storm — Verification State

Date: 2026-06-05 · Branch: audit-reconnect-patterns
Companion to: 2026-06-05-reconnect-storm-prevention-prd.md
Repo: https://github.com/Expensify/App · Code anchored to d8f64dd837ebec0c1b30062a91cac7b78b552c12 (origin/main HEAD = merge of PR #91368)

Every failure mode in the PRD is confirmed on two independent axes: live code (GitHub permalinks below) and live production telemetry (Sentry, 14d, environment:production, org expensify / project app).

---

Production telemetry (Sentry, 14d, prod)

Command	Calls	p50	p95	max
ReconnectApp	819,920	329ms	2,227ms	544,963ms (9.1 min)
GetMissingOnyxMessages	494,733	185ms	1,727ms	81,676ms (82s)
OpenApp	215,153	331ms	1,333ms	382,956ms (6.4 min)

Total reconnect-family ≈ 1.53M calls / 14d.
ReconnectApp response bodies reach 22,843,956 bytes (22.8 MB) via http.response_content_length (p95 of measured bodies = 137 KB; 342k of 820k spans carry the attribute).

Dashboard: https://expensify.sentry.io/explore/traces/?query=span.description%3A*ReconnectApp*+environment%3Aproduction&statsPeriod=14d
Query form: use plain span.description:*ReconnectApp* — free-text gets coerced to exact-match and misses.

Signature traces

• Mode 1 loop — trace 7d45b533b0a5490481b09871a2c886c5: 31 ReconnectApp spans, top of the per-trace distribution.
• Mode 2 loop — trace ff14342c65c54ff48df07eb4a8b23df9: 28 GetMissingOnyxMessages spans, zero ReconnectApp.
• Mode 3 slow giant — trace a389b884c52f4c11a46c6a6024052255 (2026-05-28 12:34:27): a 544,963ms ReconnectApp running parallel to a 544,926ms AuthenticatePusher from the same instant — socket open, no progress. The same trace holds a 545,538ms "Route Change" / "Navigation … cancelled or screen mounted" and ~13 ActivityIndicator spinners each ~544,992ms — the multi-minute frozen UI behind a dead-click report.

---

Code

Mode 1 — subscribeToFullReconnect re-arm hazard

• subscribeToFullReconnect.ts L6–35 — subscribes to LAST_FULL_RECONNECT_TIME and NVP_RECONNECT_APP_IF_FULL_RECONNECT_BEFORE; doFullReconnectIfNecessary() fires reconnectApp() on lastFullReconnectTime < reconnectAppIfFullReconnectBefore with no in-flight guard, no debounce.
• App.ts L381–387 — successData is the only writer of LAST_FULL_RECONNECT_TIME=now, gated on isOpenApp || isFullReconnect (the loop-breaker).
• App.ts L511 — const isFullReconnect = !updateIDFrom (arg-less ⇒ full dump).
• clearOnyxAndSeedFullReconnect.ts L6–21 — the comment states applyHTTPSOnyxUpdates applies response.onyxData before successData, and the path pre-seeds the timestamp; the steady-state path is left unseeded.
• User.ts L989–991 — Pusher RECONNECT_APP → bare reconnectApp().

Mode 2 — OnyxUpdateManager gap re-detect, bypasses SequentialQueue

• OnyxUpdateManager/index.ts L152–191 — !lastUpdateIDFromClient diverts to full ReconnectApp; the gap branch is guarded only by areDeferredUpdatesQueued || isFetchingForPendingUpdates, then calls getMissingOnyxUpdates.
• App.ts L547–560 — getMissingOnyxUpdates → API.makeRequestWithSideEffects (bypasses SequentialQueue, no in-flight guard).
• App.ts L528–540 — finalReconnectAppAfterActivatingReliableUpdates (declared L528, call at L538) also uses makeRequestWithSideEffects, bypassing the SequentialQueue.

Amplifier — SequentialQueue in-flight dedup gap

• RequestConflictUtils.ts L57–73 — resolveDuplicationConflictAction does findIndex over the waiting persisted requests only; wired into the queue via API.writeWithNoDuplicatesConflictAction → checkAndFixConflictingRequest (API/index.ts L193).
• PersistedRequests.ts L384 — processNextRequest removes the in-flight request via slice(1).
• PersistedRequests.ts L150 — deepEqual(nextRequestToProcess, ongoingRequest) guard misses the mid-flight duplicate (different requestID).
• SequentialQueue.ts L336–342 — flush() is leader-gated via isClientTheLeader(); reconnect triggers are not.

---

State

All four failure modes are real and live in production. The code paths exist exactly as the PRD describes them, and the production telemetry — the call volumes, the byte-level payloads, and the three signature traces (7d45b533, ff14342c, a389b884) — confirms each mode is firing in the field.