Add runtime performance benchmarks (#32)

Author: SandroMaglione

.changeset/fast-machines-benchmark.md

@@ -0,0 +1,5 @@
+---
+"@typeonce/effect-machine": patch
+---
+
+Add local and pull request runtime benchmark reporting for pure planning, end-to-end event drainage, machine lifecycle throughput, and idle-machine memory growth against the compiled package, XState 5, and the published XState 6 alpha.

.github/pull_request_template.md

@@ -12,8 +12,9 @@
 - [ ] `pnpm check`
 - [ ] Relevant example checks, when examples changed
 - [ ] Reviewed the automated type-performance report, when the public TypeScript API or inference changed
+- [ ] Reviewed the automated runtime-performance report, when runtime behavior changed
 
 <!--
-The type-performance workflow posts and updates the base-versus-PR comparison automatically.
+The type- and runtime-performance workflows post and update their base-versus-PR comparisons automatically.
 Do not copy a manually measured table into this description.
 -->

.github/workflows/runtime-performance-comment.yml

@@ -0,0 +1,102 @@
+name: Runtime performance comment
+
+on:
+  workflow_run:
+    workflows: [Runtime performance]
+    types: [completed]
+
+concurrency:
+  group: runtime-performance-comment-${{ github.event.workflow_run.head_repository.id }}-${{ github.event.workflow_run.head_branch }}
+  cancel-in-progress: true
+
+permissions:
+  actions: read
+  contents: read
+  issues: write
+  pull-requests: write
+
+jobs:
+  comment:
+    if: >-
+      github.event.workflow_run.event == 'pull_request' &&
+      github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out trusted reporting code
+        uses: actions/checkout@v7
+
+      - name: Download performance report
+        uses: actions/download-artifact@v8
+        with:
+          name: runtime-performance-report
+          path: report
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          run-id: ${{ github.event.workflow_run.id }}
+
+      - name: Render report from validated benchmark data
+        run: >-
+          node scripts/compare-runtime-performance.mjs
+          report/base
+          report/head
+          > "$RUNNER_TEMP/report.md"
+
+      - name: Create or update pull request comment
+        uses: actions/github-script@v9
+        env:
+          REPORT_PATH: ${{ runner.temp }}/report.md
+        with:
+          script: |
+            const fs = require("node:fs")
+            const marker = "<!-- effect-machine-runtime-performance -->"
+            const report = fs.readFileSync(process.env.REPORT_PATH, "utf8")
+
+            if (Buffer.byteLength(report, "utf8") > 60_000) {
+              core.setFailed("Runtime-performance report exceeds the safe comment size")
+              return
+            }
+
+            const run = context.payload.workflow_run
+            let pull = run.pull_requests?.[0]
+
+            if (pull === undefined) {
+              try {
+                const associated = await github.rest.repos.listPullRequestsAssociatedWithCommit({
+                  ...context.repo,
+                  commit_sha: run.head_sha
+                })
+                pull = associated.data.find((candidate) =>
+                  candidate.base.repo.full_name === `${context.repo.owner}/${context.repo.repo}`
+                )
+              } catch (error) {
+                core.warning(`Unable to look up a pull request for ${run.head_sha}: ${error.message}`)
+              }
+            }
+
+            if (pull === undefined) {
+              core.notice("No pull request is associated with this workflow run")
+              return
+            }
+
+            const body = `${marker}\n${report}`
+            const comments = await github.paginate(github.rest.issues.listComments, {
+              ...context.repo,
+              issue_number: pull.number,
+              per_page: 100
+            })
+            const previous = comments.find((comment) =>
+              comment.user?.type === "Bot" && comment.body?.startsWith(marker)
+            )
+
+            if (previous === undefined) {
+              await github.rest.issues.createComment({
+                ...context.repo,
+                issue_number: pull.number,
+                body
+              })
+            } else {
+              await github.rest.issues.updateComment({
+                ...context.repo,
+                comment_id: previous.id,
+                body
+              })
+            }

.github/workflows/runtime-performance.yml

@@ -0,0 +1,96 @@
+name: Runtime performance
+
+on:
+  pull_request:
+
+concurrency:
+  group: runtime-performance-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  runtime-performance:
+    name: runtime-performance
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Check out base
+        uses: actions/checkout@v7
+        with:
+          path: base
+          ref: ${{ github.event.pull_request.base.sha }}
+
+      - name: Check out pull request
+        uses: actions/checkout@v7
+        with:
+          path: head
+          repository: ${{ github.event.pull_request.head.repo.full_name }}
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - uses: pnpm/action-setup@v6
+        with:
+          package_json_file: head/package.json
+
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 24
+          cache: pnpm
+          cache-dependency-path: |
+            base/pnpm-lock.yaml
+            head/pnpm-lock.yaml
+
+      - name: Install and build pull request
+        run: |
+          pnpm --dir head install --frozen-lockfile
+          pnpm --dir head build
+
+      - name: Install and build base when benchmarkable
+        if: ${{ hashFiles('base/scripts/runtime-performance.mjs') != '' }}
+        run: |
+          pnpm --dir base install --frozen-lockfile
+          pnpm --dir base build
+
+      - name: Measure base and pull request
+        shell: bash
+        run: |
+          set -euo pipefail
+          reports="$RUNNER_TEMP/runtime-performance"
+          mkdir -p "$reports/base" "$reports/head"
+
+          measure() {
+            local checkout="$1"
+            local output="$2"
+            (
+              cd "$checkout"
+              node --expose-gc scripts/runtime-performance.mjs --json
+            ) > "$output"
+          }
+
+          if [[ -f base/scripts/runtime-performance.mjs ]]; then
+            measure base "$reports/base/1.json"
+            measure head "$reports/head/1.json"
+            measure head "$reports/head/2.json"
+            measure base "$reports/base/2.json"
+            measure base "$reports/base/3.json"
+            measure head "$reports/head/3.json"
+          else
+            measure head "$reports/head/1.json"
+            measure head "$reports/head/2.json"
+            measure head "$reports/head/3.json"
+          fi
+
+          node head/scripts/compare-runtime-performance.mjs \
+            "$reports/base" \
+            "$reports/head" \
+            > "$RUNNER_TEMP/runtime-performance-report.md"
+          cat "$RUNNER_TEMP/runtime-performance-report.md" >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload report for the comment workflow
+        uses: actions/upload-artifact@v7
+        with:
+          name: runtime-performance-report
+          path: ${{ runner.temp }}/runtime-performance
+          if-no-files-found: error
+          retention-days: 7

package.json

@@ -50,6 +50,7 @@
     "test:types": "tstyche",
     "typecheck": "tsc -p tsconfig.json --noEmit",
     "perf:types": "pnpm build && node scripts/type-performance.mjs",
+    "perf:runtime": "pnpm build && node --expose-gc scripts/runtime-performance.mjs",
     "format": "dprint fmt",
     "format:check": "dprint check",
     "test:consumer": "node scripts/test-consumer.mjs",
@@ -68,9 +69,12 @@
     "@types/node": "25.7.0",
     "dprint": "0.55.2",
     "effect": "4.0.0-beta.102",
+    "tinybench": "2.9.0",
     "tstyche": "7.2.1",
     "typescript": "6.0.3",
-    "vitest": "4.1.10"
+    "vitest": "4.1.10",
+    "xstate-v5": "npm:xstate@5.32.5",
+    "xstate-v6": "npm:xstate@6.0.0-alpha.27"
   },
   "packageManager": "pnpm@10.17.1",
   "engines": {

perf/runtime/README.md

@@ -0,0 +1,82 @@
+# Runtime performance
+
+Run the local runtime benchmark suite against the compiled package and the
+pinned XState comparison versions:
+
+```sh
+pnpm perf:runtime
+```
+
+The command reports:
+
+- pure `Machine.plan` counter-transition throughput;
+- end-to-end useful-increment throughput for a burst sent to one running machine;
+- machine start-and-stop throughput;
+- idle heap and resident-memory growth at 100, 500, and 1,000 live machines.
+
+The comparison dependencies use package aliases, so XState 5 and 6 can be
+loaded by the same process:
+
+- `xstate-v5`: `xstate@5.32.5`, the stable v5 baseline;
+- `xstate-v6`: `xstate@6.0.0-alpha.27`, the latest published v6 alpha available
+  when the harness was added.
+
+All implementations use the same flat counter topology, immutable events, and
+terminal fence. The XState adapter uses `assign` in v5 and the v6 transition
+function API because `assign` is not exported by that alpha.
+
+The burst benchmark reports useful counter increments per second. It enqueues
+one final fence event after all counter events and awaits the machine's terminal
+output, so the measured duration also amortizes that fence and terminal
+cleanup. This measures complete queue drainage, not only the enqueue time
+returned by `MachineRef.send`.
+
+Results are informational. Compare runs on the same machine while it is idle,
+using the same Node.js and dependency versions. Tinybench warms each scenario
+before collecting samples, and the memory measurements force garbage
+collection before every observation. Each implementation's memory curve runs
+in a fresh child process so garbage from one library cannot distort another
+library's baseline.
+
+These scenarios compare observable work, not identical internals. Effect
+Machine plans through an `Effect`, validates schema-backed state and events,
+and its running machine provisions Effect queues, fibers, synchronization,
+change publication, and child/invoke lifecycle machinery. XState's counter is
+a smaller synchronous actor. Treat the comparison as an application-level
+cost baseline, not a claim that the libraries provide the same runtime
+guarantees.
+
+The fitted heap slope is the primary idle-capacity metric. Resident memory is
+reported as a raw diagnostic because V8 and the operating-system allocator can
+reuse already committed pages. The capacity-per-GiB value is a linear estimate
+that excludes shared process overhead; it is not a run-until-OOM limit.
+
+Use a shorter smoke run while changing the harness:
+
+```sh
+pnpm perf:runtime -- --quick
+```
+
+Display the complete versioned report as JSON, or write a machine-readable JSON
+file alongside the terminal report:
+
+```sh
+pnpm perf:runtime -- --json
+pnpm perf:runtime -- --output runtime-performance.json
+```
+
+Prefer `--output` for scripts: pnpm and the preceding build may add their own
+lines to standard output before `--json` is printed.
+
+Pull requests run the suite three times for both the base and pull request
+revisions on the same GitHub-hosted runner. The workflow publishes the median
+of those process-level results to the job summary and a sticky pull request
+comment. The benchmark workflow has read-only repository access; a separate
+trusted `workflow_run` workflow validates the uploaded JSON before receiving
+permission to update the comment.
+
+The implementation lives in `scripts/runtime-performance.mjs`; the Effect
+Machine fixture is in `perf/runtime/counter.mjs`, and the comparison adapter is
+in `perf/runtime/xstate.mjs`. Add new scenarios only when every implementation
+performs equivalent observable work and the result is consumed and checked so
+the JavaScript engine cannot discard it.