Skip to content

test: add offerings startup perf tester app#3743

Draft
vegaro wants to merge 9 commits into
mainfrom
cesar/perf-tester-app
Draft

test: add offerings startup perf tester app#3743
vegaro wants to merge 9 commits into
mainfrom
cesar/perf-tester-app

Conversation

@vegaro

@vegaro vegaro commented Jul 9, 2026

Copy link
Copy Markdown
Member

New test-apps:perftester app to measure how long configure() + getOfferings() takes at startup, baseline vs workflows. It's the realistic counterpart to the instrumentation harness: a real app, a button press, timestamps around the two calls, one JSON line appended per press to results.jsonl on device.

Why an app and not just the instrumentation test: the harness runs in-process with mocked billing and a simulated lifecycle, so it can't show real process cold-start ordering. This can. The had_* cache flags in each row say whether a sample was cold or warm, so run order doesn't have to be tracked separately.

How it works:

  • Toggle picks the arm: baseline or DangerousSettings.forWorkflows().
  • "Configure + getOfferings" runs one measured cycle: close() if configured (so configure() doesn't short-circuit on an equal config), snapshot the cache flags, configure(), getOfferings(), then on the workflows arm ui_config + first workflow body readiness.
  • "Clear SDK caches" wipes the SDK prefs/dirs for a cold sample without reinstalling.
  • -e autorun true -e workflows true drives the same cycle for scripted am start loops.

A Test Store key is recommended (PERF_TESTER_API_KEY in local.properties): products resolve through the simulated store, no Play setup, and billing is the same in both arms. No changes to :purchases.

Known limitation: the render-readiness withTimeout can't actually bound awaitGetWorkflow/awaitGetUiConfig, because those SDK APIs use non-cancellable suspendCoroutine. A hung fetch on the workflows arm keeps the cycle waiting past the guard. The getOfferings measurement itself is properly bounded. Left as-is for now, a proper fix needs a detached-async workaround.

Verification

  • :test-apps:perftester:testDebugUnitTest (3/3)
  • :test-apps:perftester:assembleDebug
  • detektAll
  • on-device measurement runs: needs a Test Store key. The app builds, installs, launches and renders clean on a device, but I couldn't run real measurements here.

@vegaro vegaro added the pr:other label Jul 9, 2026
@RevenueCat-Danger-Bot

RevenueCat-Danger-Bot commented Jul 9, 2026

Copy link
Copy Markdown
1 Message
📖 This PR changes 441 lines of production Kotlin/Java (limit 300); skipped via skip-pr-lines-changed-check label.

Generated by 🚫 Danger

vegaro and others added 8 commits July 10, 2026 11:20
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#3742 made the UiConfig constructor require app/localizations/variableConfig,
so PerfCycleRunner's `uiConfig == UiConfig()` no longer compiles. Drop the
uiConfigIsDefault diagnostic (keep the ui_config timing) so the app builds
against current main.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.31%. Comparing base (4766536) to head (1ed047c).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3743   +/-   ##
=======================================
  Coverage   80.31%   80.31%           
=======================================
  Files         402      402           
  Lines       16452    16452           
  Branches     2351     2351           
=======================================
  Hits        13214    13214           
  Misses       2305     2305           
  Partials      933      933           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vegaro

vegaro commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Added post-readiness memory measurements to this perftester PR. Each JSONL row now records before/after/delta RSS, RSS anon/file breakdown, and Java/native heap. The after sample is taken only after getOfferings plus the existing ui_config/workflow readiness sequence completes, so this measures retained memory after workflow loading. It is intentionally not a peak-memory metric; RSS values are read from /proc/self/status and are reported in KiB. Unit tests, assembleDebug, and detekt passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants