Skip to content

chore(ci): run the performance gate only when a change can move it - #34

Closed
chh-ay wants to merge 1 commit into
developfrom
chore/ci-performance-gate-scope
Closed

chore(ci): run the performance gate only when a change can move it#34
chh-ay wants to merge 1 commit into
developfrom
chore/ci-performance-gate-scope

Conversation

@chh-ay

@chh-ay chh-ay commented Jul 26, 2026

Copy link
Copy Markdown
Owner

What

The matched zero-regression benchmark occupies the single self-hosted runner for up to 90 minutes and every other pull request queues behind it — yet it ran for changeset-only and docs-only diffs. This gates it on whether the diff can move a measurement, using the same path-classification mechanism docs_required already uses.

While gating it, two holes turned up and are closed here.

Changes

Scope the benchmark to relevant diffs. requiresPerformanceRun() joins requiresDocumentationBuild() in scripts/ci-paths.ts, preflight emits perf_required, and controlled-performance is gated on it. Unknown, mixed, and empty change sets keep taking the expensive lane: a false negative ships a regression, a false positive costs one benchmark run. bench/results/render-baseline.json deliberately requires a run — it is the reference the gate compares against, so changing it must re-measure.

Install one browser instead of three. unit-coverage installed Chromium, Firefox and WebKit with system dependencies. Its only browser consumer is the bench:verify render smoke, which runs render-driver.ts --smoke and launches Chromium alone.

Refuse fork code on the self-hosted runner. controlled-performance is the only non-ephemeral runner in the workflow. A persistent host exposes its filesystem, caches and network position regardless of a read-only token, so the job now requires a same-repository event. Required CI fails a fork pull request that touches performance-sensitive paths rather than accepting the skip — deferring to a post-merge run would let a fork alter the baseline and be validated against it after the merge already happened. The message tells a maintainer to replay the commits on a same-repository branch.

Fix a latent always-skip. preflight joins the job's needs. The needs context exposes direct dependencies only, so the new condition would otherwise evaluate against an undefined context and skip on every run.

Contract tests

  • Trigger branch lists are pinned exactly, so adding a release branch is a deliberate edit
  • No workflow may use pull_request_target
  • No non-ephemeral runner may execute fork pull-request code
  • Any job whose if: reads needs.<job>.outputs must list <job> in needs — the generic form of the always-skip bug, which also protects docs-build
  • Required CI rejects, rather than skips, a fork pull request that needs a measurement
  • scripts/ci-paths.test.ts joins test:tooling

Verification

Local, on this branch:

bun run test:tooling          146 pass, 0 fail
bun run lint                  1 warning (pre-existing: 2 MiB conformance corpus exceeds Biome's file limit)
bun run typecheck:verification  clean
bun run changeset:ci          passes — no publishable package changes, so no changeset

Not in scope

The benchmark's thresholds, sample counts, timeout, and runner are untouched. This changes when the gate runs, never what it asserts.

The matched zero-regression benchmark occupies the single self-hosted
runner for up to 90 minutes and every other pull request queues behind
it, yet it ran for changeset-only and docs-only diffs. Classify
performance-relevant paths the way `docs_required` already classifies
documentation, and gate the job on the result. Unknown, mixed, and empty
change sets keep taking the expensive lane: a false negative ships a
regression, a false positive costs one benchmark run.

`unit-coverage` installed Chromium, Firefox and WebKit with system
dependencies, but its only browser consumer is the `bench:verify` render
smoke, which launches Chromium alone.

Also close two holes found while gating:

- The job now refuses fork pull requests. It is the only non-ephemeral
  runner in the workflow, and a persistent host exposes its filesystem,
  caches and network position regardless of a read-only token. Required
  CI fails such a pull request rather than accepting the skip, so a fork
  cannot alter `bench/results/render-baseline.json` and be validated
  against its own baseline after merge.
- `preflight` joins the job's `needs`, because the `needs` context
  exposes direct dependencies only and the condition would otherwise
  evaluate against an undefined context and skip on every run.

Contract tests cover the trigger lists, the dependency requirement for
any job reading another job's outputs, the fork rejection, and the
absence of `pull_request_target`.
@vercel

vercel Bot commented Jul 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
sheetwrite Ready Ready Preview, Comment Jul 26, 2026 9:33am

@chh-ay

chh-ay commented Jul 26, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #35. The repository has zero registered runners for the self-hosted label, so scoping the job still leaves performance-sensitive PRs — including this workflow change itself — permanently queued. #35 removes the unreliable required job while preserving the matched protocol as a fail-closed local command.

@chh-ay chh-ay closed this Jul 26, 2026
@chh-ay chh-ay mentioned this pull request Jul 26, 2026
36 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant