test: CI network-conditioned getOfferings perf regression gate#3762
Draft
vegaro wants to merge 12 commits into
Draft
test: CI network-conditioned getOfferings perf regression gate#3762vegaro wants to merge 12 commits into
vegaro wants to merge 12 commits into
Conversation
Make the placeholder-host rewrite assertion non-vacuous (adds blob.json fixture that actually contains PERF_MOCK_HOST), removes dead imports, and adds 404/ambiguous-path coverage for the dispatcher.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the zero-timeout request drain in PerfHarness.runCycle with a bounded-grace-wait loop so the async workflow /config POST is captured deterministically instead of racing a zero-timeout poll. Also remove a vacuous fixture assertion, assert offerings are actually returned in the median-latency sampling loop, and prefer imports over inline fully-qualified okhttp3 references.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a CI-friendly performance regression gate for
getOfferings(), living entirely underpurchases/src/test/as Robolectric unit tests — no emulator, runs on every PR in the existingtestDefaultsDebugUnitTestlane.The problem it solves: the workflow/config-path performance checks were manual (run an app repeatedly, sit a proxy in the request path, read medians off logcat). That can't gate CI, and the obvious automation — asserting an absolute millisecond threshold — flakes, because CI machine speed and network jitter dominate the signal.
The approach here removes both variables. The SDK is driven end-to-end (
Purchases.configure()→getOfferings()) against a localMockWebServerviaPurchases.proxyURL(which redirects both the main API and/v1/config), so every byte comes from localhost and the only variable is a delay we inject ourselves. The legacy path (DangerousSettings()) and the workflow path (DangerousSettings.forWorkflows()) run back-to-back in the same test run, and the assertions are things that don't move with machine speed.What it asserts
getOfferingscritical pathfeature / baselinelatency ratio < 3.0 under injected delay/configsync still returns offeringsConfirmed request sequences (the primary gate's anchors):
/configDangerousSettings())/offerings,/productsforWorkflows())/offerings,/products,/config/appSo the gate asserts
feature.requests == baseline.requests + 1and exactly one/configon the workflow path.Components
PerfFixtures— loads JSON fixtures (perf-fixtures/) into aMockWebServerdispatcher; rewrites a placeholder asset host to the live mock URL.NetworkProfile—GOOD/BAD/FLAKYdelay + failure injection over any dispatcher.PerfHarness— one cold/warm measured cycle →CycleResult(elapsedMs, offeringsCount, requestPaths, error).GetOfferingsPerfTest— the three gate tiers.perf-fixtures/CAPTURE.mdand aREADME.mdin the test package document how it works and how to refresh fixtures.Notes
getOfferingsbenchmark (test: add on-demand getOfferings benchmark (remote config off vs on) #3745), which hits the real backend for absolute-latency deep dives.Testing
12 tests across 5 classes, ~1 min, no emulator: