perf: simplify raw performance reporting

.github/PERFORMANCE.md

@@ -1,95 +1,69 @@
 # Release performance CI
 
-`Nitro Performance` builds `apps/benchmark` in its ordinary Release/Hermes configuration,
-then runs base → head → head → base on one booted target. A noisy comparison can
-run one additional head/base pair. Each suite measures 40 cases with five warmup
-batches and twenty samples, calibrated toward 150 ms (roughly 100–200 ms) of
-timed work per sample. Iteration counts are rounded to two significant digits
-and frozen after warmup; measured outliers are retained. Allocation-heavy samples
-sum bounded timed chunks with explicit GC and native-cleanup yields between chunks excluded from timing,
-so memory-safety limits do not shorten the sample. See `iterations` and
-`chunkIterations` in raw results; sample milliseconds are `ns/op * iterations / 1e6`.
-Calibration changes and version-2 benchmark definitions require a new baseline.
-Each binary is installed once per suite run. Each case uses a fresh app process
-so the runtime-scoped JSI reference cache does not accumulate millions of weak
-reference records across unrelated cases. Startup is outside measured work;
-the host validates and combines the per-case results, preserving their raw samples.
-The host computes confidence intervals by resampling whole matched base/head
-process runs, then batches within each selected run. Bencher median bounds use
-the same run-aware approach. Treating all batches from multiple processes as
-independent gave a false-positive control result in the initial A/A validation;
-process-level variability must be preserved in the uncertainty estimate.
-Robust CV remains visible and causes a neutral result when uncertainty overlaps
-zero. It is not an unconditional veto: an effect beyond the budget whose full
-run-aware confidence interval excludes zero is still reported in its direction.
+`Nitro Performance` builds the dedicated `apps/benchmark` Release/Hermes app.
+Each platform runs base → head → head → base on the same machine, reversing case
+order for the second pair. Each case gets a fresh app process; installation,
+startup, transport and process restarts are outside timing. There is no automatic
+third pair. Manual reruns are retained as identifiable workflow attempts.
 
-The benchmark app is separate from `apps/example`, which keeps the demos and
-Harness correctness tests. It shares the real Nitro test packages, but has no
-Harness, navigation, screens, or safe-area dependencies. The TurboModule control
-and all benchmark cases belong to the benchmark app. Android permits
-cleartext only to loopback; the example app does not contain the CI entrypoint.
-See [`apps/benchmark/README.md`](../apps/benchmark/README.md) for local commands.
+Each case records twenty ordered batch averages after five warmup batches.
+Calibration targets 150 ms of timed work; this target does not establish steady
+state or erase drift. Allocation-heavy cases sum bounded timed chunks with
+explicit cleanup outside timing. Raw `iterations`, `chunkIterations` and ordered
+`samplesNsPerOp` describe the work; timed sample milliseconds are
+`ns/op * iterations / 1e6`. Slow samples are retained.
 
-## Android host requirements
+## Reading results
+
+The main score is the median (p50) of batch averages in ns/op, not individual-call
+tail latency. The report shows every observed change of at least 5%, including
+Promise cases. This is a presentation threshold, not a calibrated regression
+budget. Expand the report for all metrics, individual process medians, matched
+pair changes, and sample MAD relative to p50. Matching pooled medians do not prove
+equal performance. Two process pairs do not justify confidence intervals.
 
-The API 36 x86_64 emulator **must use KVM CPU acceleration**. The workflow grants
-the ephemeral runner access to `/dev/kvm`, verifies acceleration before boot, and
-launches with `-accel on`. Missing acceleration fails the job rather than silently
-falling back to software CPU emulation. Software GPU rendering is separate and is
-still used on the headless runner.
+Performance is currently report-only. Build, execution and malformed-result
+failures still fail CI. Turning observed differences into a regression gate needs
+empirical validation on unchanged commits and intentional slowdowns on each
+unchanged suite/testbed. No Promise case is permanently exempt. Scheduled/manual
+runs with the same base and head SHA measure baseline variation explicitly.
+Changed benchmark definitions require a new baseline and are not compared.
 
-The initial bootstrap run accidentally used `-accel off`: boot alone took 8½
-minutes and six suites took another 32½ minutes. Those Android timings are not a
-performance baseline. Accelerated results use a distinct `...-kvm` Bencher testbed.
+## Artifacts and publishing
 
-Host-side log lines show each suite's start, completion, measurement duration,
-total wall time (including app installation/launch), and why a repeat pair ran.
-These logs are outside the app's timed regions. Raw JSON, BMF, and comparison
-Markdown are retained in the workflow artifacts for 30 days.
-Both targets have a five-minute boot limit; individual device installation and
-launch commands have a two-minute limit. Android app exits also retain logcat and
-process-exit diagnostics, so native crashes do not disappear at emulator teardown.
+The canonical artifact is `performance-report-<attempt>`: raw JSON for every
+base/head process plus `performance-report.json` with repository, revisions,
+workflow run and attempt provenance. Artifacts remain available for 30 days.
+The PR comment links its exact immutable artifact ID; downloads require GitHub
+access. An agent can inspect the JSON instead of scraping the rendered table.
 
-Same-repository PRs publish from a separate clean job after both device jobs
-finish. This works before merge: it checks out an immutable, reviewed reporting
-commit, downloads only result data, validates it against GitHub API metadata,
-and rebuilds the table/BMF without installing or executing either app checkout.
-Fork PRs never enter this privileged job; their `workflow_run` reporter becomes
-active once its definition reaches the default branch. That reporter skips
-same-repository PRs to avoid duplicate uploads. Both paths post the rebuilt
-paired-comparison table as one updatable PR comment, independently of Bencher's
-historical comparison. Stale results do
-not overwrite a newer PR revision, and user-authored comments are never edited.
-Revoke the previously exposed credential and replace the
-repository's `BENCHER_KEY` secret before enabling publishing. PR jobs never receive
-that secret in device/build jobs or any fork job. After rotation, set the repository variable `NITRO_BENCHER_ENABLED`
-to `true` to enable Bencher uploads. The paired PR comment does not need that key.
-Verdicts remain advisory during noise calibration.
+One default-branch `Publish Nitro Performance` workflow handles internal PRs,
+forks and main runs. It downloads the exact artifact from the triggering attempt,
+validates bounded JSON against GitHub's run and current PR metadata, and computes
+the comparison, Markdown and Bencher values from raw samples. It never installs
+or executes PR code or app artifacts. Docs-only and cancelled runs skip
+publication. Relevant failures remain failures. Stale PR results are skipped.
 
-The pre-merge publisher requires a real base benchmark app. The infrastructure
-PR's bootstrap A/A runs remain diagnostic artifacts, not Bencher baselines.
-For paired PR reports, first upload the measured base to `baseline-<full-base-sha>`
-on each testbed, then upload `pr-<number>` with that exact baseline as its start
-point. This works with an empty Bencher project and with stacked PRs without
-pretending their base is `main`. Pushes/scheduled main runs record main history.
-Both platform baselines are uploaded before either head. The publisher does not
-reset the PR branch for each platform, preserving the other testbed's reports;
-a changed base SHA naturally selects a different start point.
-The Bencher action and downloaded CLI version are both pinned. Both publishers
-verify the reviewed Linux CLI SHA-256 before the step receiving the API key.
+The trusted publisher uses `BENCHER_KEY` as an Actions secret. Its CLI version and
+binary digest are pinned. Bencher receives median latency values without invented
+bounds; its JSON adapter requires only `value`. PR publications seed both measured
+platform baselines at `baseline-<base SHA>` before recording the head at
+`pr-<number>`. Main runs record main history. User comments are never edited.
 
-## Promoting performance verdicts to a gate
+The raw-manifest publisher and producer land together in the first cleanup PR.
+Until that PR reaches the default branch, the previous trusted reporter cannot
+consume its new manifest. Merge that PR before running later stack revisions;
+there is no permanent old-schema reporting path. Existing raw app results may
+contain extra summary fields, which the raw parser ignores.
+
+## Android host requirements
 
-This initial workflow always passes `--mode advisory`; merging it does **not**
-enable performance enforcement. Collect at least 30 successful main/no-change
-runs on each unchanged suite and testbed before a separate reviewed promotion.
-Use only same-commit comparisons to estimate noise; ordinary base/head deltas
-may include real code changes and must not inflate the noise allowance.
+The API 36 x86_64 emulator requires KVM. CI checks `/dev/kvm`, verifies acceleration
+before boot and uses `-accel on`; it must not silently use software CPU emulation.
+Software GPU rendering is separate. Base/head measurements stay on that machine.
+Android process failures retain logcat and process-exit diagnostics. Both platforms
+bound boot time to five minutes and individual install/launch commands to two.
 
-For each synchronous case, the promotion must set a per-case budget of
-`max(5%, 1.5 × p95(abs(no-change delta)))`. Cases requiring over 10%, Promise
-metrics, and inconclusive comparisons remain advisory. The current 5% table
-threshold is provisional, not a calibrated gate. Do not simply change `--mode`
-to `enforce`: the initial comparator uses that single provisional threshold, and
-the trusted reporter deliberately rejects PR attempts to enable enforcement.
-Promotion needs the reviewed per-case policy and matching reporter support.
+The benchmark app shares real Nitro test packages without Harness/navigation UI.
+Correctness remains in `apps/example`. See the [benchmark app README](../apps/benchmark/README.md)
+for local build and run commands.

.github/workflows/performance-report.yml

@@ -13,11 +13,6 @@ permissions:
 
 jobs:
   publish:
-    # Same-repository PRs publish in a separate job of Nitro Performance, which
-    # also works before this workflow reaches the default branch.
-    if: >-
-      github.event.workflow_run.event != 'pull_request' ||
-      github.event.workflow_run.head_repository.full_name != github.repository
     runs-on: ubuntu-24.04
     steps:
       - name: Checkout trusted reporting code
@@ -26,17 +21,24 @@ jobs:
         with:
           bun-version: 1.3.14
 
+      - name: Select exact report artifact
+        id: select
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: bun scripts/performance/select-report.ts
+
       - name: Download performance report
         id: download
-        continue-on-error: true
+        if: steps.select.outputs.artifact_id != ''
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
         with:
-          name: performance-report
+          artifact-ids: ${{ steps.select.outputs.artifact_id }}
           path: untrusted-artifact
           github-token: ${{ secrets.GITHUB_TOKEN }}
           run-id: ${{ github.event.workflow_run.id }}
 
       - name: Validate untrusted report
+        id: validate
         if: steps.download.outcome == 'success'
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -57,27 +59,28 @@ jobs:
             --output-directory validated-report \
             --expected-repository "$GITHUB_REPOSITORY" \
             --trusted-workflow-event "$GITHUB_EVENT_PATH" \
+            --artifact-id "${{ steps.select.outputs.artifact_id }}" \
             "${TRUSTED_PR_ARGUMENTS[@]}"
 
       - name: Post paired comparison to the PR
-        if: steps.download.outcome == 'success'
+        if: steps.download.outcome == 'success' && steps.validate.outputs.stale != 'true'
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: bun scripts/performance/github-report.ts --directory validated-report
 
       - name: Install Bencher CLI
-        if: steps.download.outcome == 'success' && vars.NITRO_BENCHER_ENABLED == 'true'
+        if: steps.download.outcome == 'success' && steps.validate.outputs.stale != 'true'
         uses: bencherdev/bencher@8d75325c3bc59403a2186a056b472c4f49d42838 # v0.6.12
         with:
           version: '0.6.12'
 
       - name: Verify pinned Bencher binary
-        if: steps.download.outcome == 'success' && vars.NITRO_BENCHER_ENABLED == 'true'
+        if: steps.download.outcome == 'success' && steps.validate.outputs.stale != 'true'
         run: |
           printf '%s  %s\n' 'c2d3a6a7fae654246134e5ced1408bdb9ba4e198b0ac3b903af17a06574a7e08' "$(command -v bencher)" | sha256sum --check -
 
       - name: Publish to Bencher and GitHub
-        if: steps.download.outcome == 'success' && vars.NITRO_BENCHER_ENABLED == 'true'
+        if: steps.download.outcome == 'success' && steps.validate.outputs.stale != 'true'
         env:
           BENCHER_API_KEY: ${{ secrets.BENCHER_KEY }}
           BENCHER_PROJECT: nitro