Skip to content

test: CI network-conditioned getOfferings perf regression gate#3762

Draft
vegaro wants to merge 12 commits into
mainfrom
cesar/perf-ci-network-test
Draft

test: CI network-conditioned getOfferings perf regression gate#3762
vegaro wants to merge 12 commits into
mainfrom
cesar/perf-ci-network-test

Conversation

@vegaro

@vegaro vegaro commented Jul 14, 2026

Copy link
Copy Markdown
Member

Description

Adds a CI-friendly performance regression gate for getOfferings(), living entirely under purchases/src/test/ as Robolectric unit tests — no emulator, runs on every PR in the existing testDefaultsDebugUnitTest lane.

The problem it solves: the workflow/config-path performance checks were manual (run an app repeatedly, sit a proxy in the request path, read medians off logcat). That can't gate CI, and the obvious automation — asserting an absolute millisecond threshold — flakes, because CI machine speed and network jitter dominate the signal.

The approach here removes both variables. The SDK is driven end-to-end (Purchases.configure()getOfferings()) against a local MockWebServer via Purchases.proxyURL (which redirects both the main API and /v1/config), so every byte comes from localhost and the only variable is a delay we inject ourselves. The legacy path (DangerousSettings()) and the workflow path (DangerousSettings.forWorkflows()) run back-to-back in the same test run, and the assertions are things that don't move with machine speed.

What it asserts

Tier Gate Why it doesn't flake
Primary Round-trip count / structure of the getOfferings critical path Machine-independent. Adding a serial round-trip fails it on any CI regardless of speed.
Secondary Same-run feature / baseline latency ratio < 3.0 under injected delay Both arms run on the same machine under the same delay, so speed cancels. Catches "we made this 3× slower."
Behavior Warm path still returns offerings under bad network; a failing (non-critical) /config sync still returns offerings Correctness under adversity, not timing.

Confirmed request sequences (the primary gate's anchors):

Path Requests /config
Baseline (DangerousSettings()) /offerings, /products 0
Feature (forWorkflows()) /offerings, /products, /config/app 1

So the gate asserts feature.requests == baseline.requests + 1 and exactly one /config on the workflow path.

Components

  • PerfFixtures — loads JSON fixtures (perf-fixtures/) into a MockWebServer dispatcher; rewrites a placeholder asset host to the live mock URL.
  • NetworkProfileGOOD / BAD / FLAKY delay + failure injection over any dispatcher.
  • PerfHarness — one cold/warm measured cycle → CycleResult(elapsedMs, offeringsCount, requestPaths, error).
  • GetOfferingsPerfTest — the three gate tiers.
  • perf-fixtures/CAPTURE.md and a README.md in the test package document how it works and how to refresh fixtures.

Notes

  • Test-only. No production or public-API changes.
  • Deliberately no absolute-ms threshold — round-trip counts and the same-run ratio are the machine-speed-independent signals; a timing threshold would only add flakiness.
  • The warm assertion checks resilience, not speed: this SDK's warm path revalidates via conditional requests (same round-trip shape as cold), so a "warm is faster" comparison would compare two near-equal, high-variance timings and flake.
  • Complementary to the separate on-demand getOfferings benchmark (test: add on-demand getOfferings benchmark (remote config off vs on) #3745), which hits the real backend for absolute-latency deep dives.

Testing

12 tests across 5 classes, ~1 min, no emulator:

./gradlew :purchases:testDefaultsDebugUnitTest --tests "com.revenuecat.purchases.perf.*"

vegaro and others added 12 commits July 14, 2026 11:01
Make the placeholder-host rewrite assertion non-vacuous (adds blob.json
fixture that actually contains PERF_MOCK_HOST), removes dead imports,
and adds 404/ambiguous-path coverage for the dispatcher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the zero-timeout request drain in PerfHarness.runCycle with a
bounded-grace-wait loop so the async workflow /config POST is captured
deterministically instead of racing a zero-timeout poll. Also remove a
vacuous fixture assertion, assert offerings are actually returned in the
median-latency sampling loop, and prefer imports over inline
fully-qualified okhttp3 references.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant