Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 40 additions & 4 deletions .github/PERFORMANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,33 @@ Changed benchmark definitions run two head-only measurements as a new baseline,
without executing the old base app or publishing an invented paired baseline.
This also handles the first rollout of a new runner protocol.

## Saved apps and measurement reruns

The workflow separates preparation, platform builds, platform measurements and
collection. Each measurement job downloads the immutable app artifact ID produced
by its build job; base and head still run together on one machine. A changed suite
builds only head. An identical base/head SHA reuses the same binary for both sides.
Otherwise each revision is built from its own checkout with the same build script.

Use GitHub's **Re-run job and dependent jobs** on `measure-android` or
`measure-ios` to repeat measurements without rebuilding successful ancestors.
Collection downloads the exact result IDs from those jobs. An untouched platform
keeps its original attempt; the report records and links both platform measurement
attempts and their app build attempts. Rerunning all jobs intentionally rebuilds.
Apps and results expire after 30 days; an expired artifact requires a new build.

The app artifact includes base/head SHAs, suite hashes, Release configuration,
architecture and toolchain metadata. iOS apps are tar archives to preserve
permissions and symlinks. Gradle's basic cache is the sole Android cache owner;
Gradle still checks source/task inputs, while exact app reuse is by artifact ID.
There is no new iOS compiler cache. First-run speed or CI stability improvements
have not been measured; app reuse specifically avoids build work on manual reruns.

## Artifacts and publishing

The canonical artifact is `performance-report-<attempt>`: raw JSON for every
measured process and discarded calibration plan, plus `performance-report.json` with repository, revisions,
workflow run and attempt provenance. Artifacts remain available for 30 days.
workflow run and attempt provenance, original build metadata and measurement artifact IDs. Artifacts remain available for 30 days.
The PR comment links its exact immutable artifact ID; downloads require GitHub
access. An agent can inspect the JSON instead of scraping the rendered table.

Expand All @@ -46,17 +68,17 @@ forks and main runs. It downloads the exact artifact from the triggering attempt
validates bounded JSON against GitHub's run and current PR metadata, and computes
the comparison, Markdown and Bencher values from raw samples. It never installs
or executes PR code or app artifacts. Docs-only and cancelled runs skip
publication. Relevant failures remain failures. Stale PR results are skipped.
publication. Markdown/MDX-only edits also skip measurements inside package/app directories. Relevant failures remain failures. Stale PR results are skipped.

The trusted publisher uses `BENCHER_KEY` as an Actions secret. Its CLI version and
binary digest are pinned. Bencher receives median latency values without invented
bounds; its JSON adapter requires only `value`. PR publications seed both measured
platform baselines at `baseline-<base SHA>` before recording the head at
`pr-<number>`. Main runs record main history. User comments are never edited.
`pr-<number>`. Main runs record main history. Bencher receives history only: it does not post a second GitHub comment or create alert-driven checks. The trusted renderer owns the one PR comment. User comments are never edited.

The raw-manifest publisher and producer land together in the first cleanup PR.
Until that PR reaches the default branch, the previous trusted reporter cannot
consume its new manifest. Merge that PR before running later stack revisions;
consume its new manifest. Merge the complete producer/publisher stack before relying on its new provenance contract;
there is no permanent old-schema reporting path. Existing raw app results may
contain extra summary fields, which the raw parser ignores.

Expand All @@ -71,3 +93,17 @@ bound boot time to five minutes and individual install/launch commands to two.
The benchmark app shares real Nitro test packages without Harness/navigation UI.
Correctness remains in `apps/example`. See the [benchmark app README](../apps/benchmark/README.md)
for local build and run commands.

## Remaining upstream warnings

The benchmark now uses the supported ReactModuleInfo constructor and matching
`reactContext` parameter name. Its Gradle property assignments use current syntax.
AGP/RN and Nitro package Gradle warnings remain outside that focused app edit.
Android uses supported `-gpu swiftshader` with required KVM CPU acceleration.

The emulator action hardcodes `cmdline-tools/latest`. Older hosted SDK managers
can still emit the XML v3/v4 metadata warning. Updating only the action does not
replace an installed SDK manager; latest command-line tools 23 also deprecates
`sdkmanager` in favor of the Android CLI. This change logs the selected SDK manager
and version, and leaves provisioning to the runner/action rather than patching its
SDK layout. This inherited warning is not suppressed.
7 changes: 3 additions & 4 deletions .github/workflows/performance-report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ on:

permissions:
actions: read
checks: write
contents: read
pull-requests: write

Expand All @@ -30,7 +29,7 @@ jobs:
- name: Download performance report
id: download
if: steps.select.outputs.artifact_id != ''
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ steps.select.outputs.artifact_id }}
path: untrusted-artifact
Expand Down Expand Up @@ -60,6 +59,7 @@ jobs:
--expected-repository "$GITHUB_REPOSITORY" \
--trusted-workflow-event "$GITHUB_EVENT_PATH" \
--artifact-id "${{ steps.select.outputs.artifact_id }}" \
--trusted-artifacts trusted-artifacts.json \
"${TRUSTED_PR_ARGUMENTS[@]}"

- name: Post paired comparison to the PR
Expand All @@ -79,10 +79,9 @@ jobs:
run: |
printf '%s %s\n' 'c2d3a6a7fae654246134e5ced1408bdb9ba4e198b0ac3b903af17a06574a7e08' "$(command -v bencher)" | sha256sum --check -

- name: Publish to Bencher and GitHub
- name: Publish Bencher history
if: steps.download.outcome == 'success' && steps.validate.outputs.stale != 'true'
env:
BENCHER_API_KEY: ${{ secrets.BENCHER_KEY }}
BENCHER_PROJECT: nitro
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: bun scripts/performance/publish.ts --directory validated-report
Loading