perf: simplify raw performance reporting - #1604
Draft
mrousavy wants to merge 1 commit into
Draft
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Performance reports currently derive confidence intervals from only two or three process pairs, automatically add a third pair, and hide Promise results regardless of the observed change. This keeps a fixed ABBA sequence and reports p50 batch-average ns/op, base/head change, process medians, paired changes, and descriptive MAD. Every observed change of at least 5% is visible; 5% is a presentation threshold and all performance results remain report-only.
Raw JSON plus run provenance is now the canonical artifact. One default-branch publisher validates downloaded data and computes the single Markdown report and median-only Bencher values. It selects the triggering attempt's immutable artifact ID, links the raw artifact in the PR comment, skips docs-only/cancelled/stale runs, and still surfaces relevant failures. The obsolete pre-merge publisher and key-rotation gate are removed.
Validation: 53 host tests; host and benchmark app TypeScript checks; benchmark app lint; actionlint on both workflows; full local package build. Tests cover raw-data/provenance rejection, fork reports, stale PRs, cancelled/docs-only runs, and exact artifact-attempt selection. Bencher's documented JSON adapter requires only
value; no synthetic production data was published.Rollout: merge this PR before running later stack revisions. The raw-manifest producer and trusted publisher land together; the old default-branch publisher cannot consume the new manifest before this merges. Changed suite definitions require a new baseline. Shared-work calibration and immutable build/measure job separation follow in separate stacked PRs.
CI is intentionally cancelled immediately after PR creation, as requested to conserve Actions minutes. Remote CI is not claimed to have passed. Do not merge until reviewed.