Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .changeset/fuzzy-mails-march.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
---

Remove the unavailable self-hosted benchmark from required CI while retaining its local protocol.
50 changes: 0 additions & 50 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -405,53 +405,6 @@ jobs:
playwright-report/
if-no-files-found: warn

controlled-performance:
name: Matched Zero-Regression Performance
needs: artifact-build
runs-on: sheetwrite-perf-i9-12900h-cachyos
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Setup Node
uses: actions/setup-node@249970729cb0ef3589644e2896645e5dc5ba9c38 # v6
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@735343b667d3e6f658f44d0eca948eb6282f2b76 # v2.0.2
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install exact dependencies and benchmark browser
run: |
bun install --frozen-lockfile
bunx playwright install chromium
- name: Build controlled render harness
run: bun run --filter '@sheetwrite/bench' bench:render:prepare
- name: Capture repeated full matched samples
working-directory: bench
run: |
bun run src/render-driver.ts \
--rounds 10 \
--output results/render-fresh.json \
--markdown-output results/render-fresh.md
- name: Require matched zero-regression decision
working-directory: bench
run: |
bun run src/check.ts \
--baseline results/render-baseline.json \
--fresh results/render-fresh.json \
--power-mode balanced \
--concurrency 1
- name: Upload controlled raw samples
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: controlled-performance-raw-${{ github.sha }}
path: |
bench/results/render-fresh.json
bench/results/render-fresh.md
if-no-files-found: error
retention-days: 30


required:
Expand All @@ -466,7 +419,6 @@ jobs:
- delivery-size
- docs-build
- browser-smoke
- controlled-performance
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
Expand All @@ -480,7 +432,6 @@ jobs:
SIZE: ${{ needs.delivery-size.result }}
DOCS: ${{ needs.docs-build.result }}
BROWSER: ${{ needs.browser-smoke.result }}
PERFORMANCE: ${{ needs.controlled-performance.result }}
DOCS_REQUIRED: ${{ needs.preflight.outputs.docs_required }}
run: |
test "$PREFLIGHT" = success
Expand All @@ -489,7 +440,6 @@ jobs:
test "$PACKED" = success
test "$BUNDLERS" = success
test "$SIZE" = success
test "$PERFORMANCE" = success
if [ "$DOCS_REQUIRED" = "true" ]; then
test "$DOCS" = success
test "$BROWSER" = success
Expand Down
4 changes: 3 additions & 1 deletion bench/test/matched-formula-evidence.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,11 @@ function validate(value: MatchedFormulaEvidence): void {
}

describe("matched formula evidence contract", () => {
// The success path validates the complete bootstrap artifact. Tamper cases
// below fail early, but this one deliberately traverses every sample.
test("accepts the freshly generated canonical artifact", () => {
expect(() => validate(copy())).not.toThrow();
});
}, 60_000);

for (const [label, mutate] of [
[
Expand Down
23 changes: 13 additions & 10 deletions bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions docs/src/content/docs/guides/performance-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ description: "Freshness-gated benchmark and package-size evidence for Sheetwrite
---
Every number on this page comes from a validated local protocol artifact captured on a clean tree; nothing is published from an unvalidated or protocol-mismatched artifact. Every expected cell carries either a validated timing or its recorded failure - a run that did not complete is shown as a failure, never converted into a timing.

## Matched regression gate
## Matched local regression check

The release gate does not treat a competitor comparison or a smoke ceiling as regression evidence. On the controlled performance runner it captures ten fresh matched rounds, retains every raw sample, and compares the fresh artifact with the committed baseline. Any unapproved slowdown fails the required CI job.
Timing comparisons run deliberately on a controlled local machine, not as a required CI job. Capture ten fresh matched rounds, retain every raw sample, and compare the fresh artifact with the committed baseline. Any unapproved slowdown fails the local command.

```sh verify title="Zero-regression benchmark"
bun run --filter @sheetwrite/bench bench:render:prepare
Expand Down
5 changes: 4 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -92,5 +92,8 @@
},
"trustedDependencies": [
"@biomejs/biome"
]
],
"overrides": {
"postcss": "^8.5.18"
}
}
4 changes: 2 additions & 2 deletions scripts/docs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1632,9 +1632,9 @@ export async function renderEvidencePage(sizeHistoryOverride?: SizeHistory): Pro
).trimEnd(),
"Every number on this page comes from a validated local protocol artifact captured on a clean tree; nothing is published from an unvalidated or protocol-mismatched artifact. Every expected cell carries either a validated timing or its recorded failure - a run that did not complete is shown as a failure, never converted into a timing.",
"",
"## Matched regression gate",
"## Matched local regression check",
"",
"The release gate does not treat a competitor comparison or a smoke ceiling as regression evidence. On the controlled performance runner it captures ten fresh matched rounds, retains every raw sample, and compares the fresh artifact with the committed baseline. Any unapproved slowdown fails the required CI job.",
"Timing comparisons run deliberately on a controlled local machine, not as a required CI job. Capture ten fresh matched rounds, retain every raw sample, and compare the fresh artifact with the committed baseline. Any unapproved slowdown fails the local command.",
"",
'```sh verify title="Zero-regression benchmark"',
"bun run --filter @sheetwrite/bench bench:render:prepare",
Expand Down
20 changes: 0 additions & 20 deletions scripts/toolchain-contract.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -158,26 +158,6 @@ describe("contributor and CI toolchain contract", () => {
expect(commands).toContain("test:browser");
});

it("requires a full matched comparison on the pinned performance runner", () => {
const jobs = parsedWorkflow.jobs ?? {};
const controlled = jobs["controlled-performance"];
expect(controlled?.["runs-on"]).toBe("sheetwrite-perf-i9-12900h-cachyos");
expect(controlled?.needs).toBe("artifact-build");
const commands = controlled?.steps?.flatMap((step) => (step.run ? [step.run] : [])).join("\n");
expect(commands).toContain("--rounds 10");
expect(commands).toContain("src/check.ts");
expect(commands).toContain("--power-mode balanced");
expect(commands).toContain("--concurrency 1");
expect(commands).not.toContain("--report-only");
expect(commands).not.toContain("bench:verify");
const upload = controlled?.steps?.find((step) =>
step.uses?.startsWith("actions/upload-artifact@"),
);
expect(upload?.if).toBe("always()");
expect(upload?.with?.path).toContain("render-fresh.json");
expect(JSON.stringify(jobs.required)).toContain("controlled-performance");
});

it("gates the exact docs artifact without weakening Required CI", () => {
const jobs = parsedWorkflow.jobs ?? {};
const preflight = jobs.preflight;
Expand Down
4 changes: 0 additions & 4 deletions scripts/workflow-contract.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,10 +179,6 @@ describe("CI and release workflow contracts", () => {
expect(commands(workflows().ci.jobs[jobName]!)).toContain("bun run browser:install:chromium");
}

const controlledPerformance = workflows().ci.jobs["controlled-performance"]!;
expect(controlledPerformance["runs-on"]).toBe("sheetwrite-perf-i9-12900h-cachyos");
expect(commands(controlledPerformance)).toContain("bunx playwright install chromium");

const projectByName = Object.fromEntries(
(playwrightConfig.projects ?? []).map((project) => [project.name, project]),
);
Expand Down
Loading