Skip to content

Commit fa7eef0

Browse files
Strengthen child runtime benchmarks
1 parent 2a8602d commit fa7eef0

8 files changed

Lines changed: 531 additions & 75 deletions

File tree

.github/workflows/runtime-performance.yml

Lines changed: 14 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
runtime-performance:
1515
name: runtime-performance
1616
runs-on: ubuntu-latest
17-
timeout-minutes: 15
17+
timeout-minutes: 25
1818
steps:
1919
- name: Check out base
2020
uses: actions/checkout@v7
@@ -46,8 +46,7 @@ jobs:
4646
pnpm --dir head install --frozen-lockfile
4747
pnpm --dir head build
4848
49-
- name: Install and build base when benchmarkable
50-
if: ${{ hashFiles('base/scripts/runtime-performance.mjs') != '' }}
49+
- name: Install and build base
5150
run: |
5251
pnpm --dir base install --frozen-lockfile
5352
pnpm --dir base build
@@ -57,29 +56,28 @@ jobs:
5756
run: |
5857
set -euo pipefail
5958
reports="$RUNNER_TEMP/runtime-performance"
59+
harness="$GITHUB_WORKSPACE/head/scripts/runtime-performance.mjs"
6060
mkdir -p "$reports/base" "$reports/head"
6161
6262
measure() {
6363
local checkout="$1"
6464
local output="$2"
6565
(
6666
cd "$checkout"
67-
node --expose-gc scripts/runtime-performance.mjs --json
67+
EFFECT_MACHINE_BENCHMARK_ROOT="$PWD" node --expose-gc "$harness" --json
6868
) > "$output"
6969
}
7070
71-
if [[ -f base/scripts/runtime-performance.mjs ]]; then
72-
measure base "$reports/base/1.json"
73-
measure head "$reports/head/1.json"
74-
measure head "$reports/head/2.json"
75-
measure base "$reports/base/2.json"
76-
measure base "$reports/base/3.json"
77-
measure head "$reports/head/3.json"
78-
else
79-
measure head "$reports/head/1.json"
80-
measure head "$reports/head/2.json"
81-
measure head "$reports/head/3.json"
82-
fi
71+
measure base "$reports/base/1.json"
72+
measure head "$reports/head/1.json"
73+
measure head "$reports/head/2.json"
74+
measure base "$reports/base/2.json"
75+
measure base "$reports/base/3.json"
76+
measure head "$reports/head/3.json"
77+
measure head "$reports/head/4.json"
78+
measure base "$reports/base/4.json"
79+
measure base "$reports/base/5.json"
80+
measure head "$reports/head/5.json"
8381
8482
node head/scripts/compare-runtime-performance.mjs \
8583
"$reports/base" \

perf/runtime/README.md

Lines changed: 14 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,12 @@ The command reports:
1111

1212
- pure `Machine.plan` counter-transition throughput;
1313
- end-to-end useful-increment throughput for a burst sent to one running machine;
14+
- the same running burst while a consumer drains every published snapshot;
15+
- repeated child lookup and delivery to one running child;
1416
- machine start-and-stop throughput;
15-
- idle heap and resident-memory growth at 100, 500, and 1,000 live machines.
17+
- parent-with-child start-and-stop throughput;
18+
- heap and resident-memory growth for both idle machines and idle parents with
19+
one child, at 100, 500, and 1,000 live units.
1620

1721
The comparison dependencies use package aliases, so XState 5 and 6 can be
1822
loaded by the same process:
@@ -68,12 +72,15 @@ pnpm perf:runtime -- --output runtime-performance.json
6872
Prefer `--output` for scripts: pnpm and the preceding build may add their own
6973
lines to standard output before `--json` is printed.
7074

71-
Pull requests run the suite three times for both the base and pull request
72-
revisions on the same GitHub-hosted runner. The workflow publishes the median
73-
of those process-level results to the job summary and a sticky pull request
74-
comment. The benchmark workflow has read-only repository access; a separate
75-
trusted `workflow_run` workflow validates the uploaded JSON before receiving
76-
permission to update the comment.
75+
Pull requests run the pull request's benchmark harness five times against both
76+
the base and pull request library revisions on the same GitHub-hosted runner.
77+
Using one harness revision means a newly added scenario can compare both
78+
implementations immediately. The runs are interleaved to reduce time-dependent
79+
machine drift. The workflow publishes the process-level median and its median
80+
absolute deviation to the job summary and a sticky pull request comment. The
81+
benchmark workflow has read-only repository access; a separate trusted
82+
`workflow_run` workflow validates the uploaded JSON before receiving permission
83+
to update the comment.
7784

7885
The implementation lives in `scripts/runtime-performance.mjs`; the Effect
7986
Machine fixture is in `perf/runtime/counter.mjs`, and the comparison adapter is

perf/runtime/counter.mjs

Lines changed: 132 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
1-
import { Effect, Schema } from "effect"
2-
import { Machine } from "../../dist/index.js"
1+
import { readFileSync } from "node:fs"
2+
import { createRequire } from "node:module"
3+
import { dirname, join, resolve } from "node:path"
4+
import { fileURLToPath, pathToFileURL } from "node:url"
5+
6+
const implementationRoot = resolve(
7+
process.env.EFFECT_MACHINE_BENCHMARK_ROOT ?? fileURLToPath(new URL("../..", import.meta.url))
8+
)
9+
const implementationRequire = createRequire(pathToFileURL(join(implementationRoot, "package.json")))
10+
const effectPackagePath = implementationRequire.resolve("effect/package.json")
11+
const effectPackage = JSON.parse(readFileSync(effectPackagePath, "utf8"))
12+
const effect = await import(pathToFileURL(resolve(dirname(effectPackagePath), effectPackage.exports["."])).href)
13+
const { Machine } = await import(pathToFileURL(join(implementationRoot, "dist/index.js")).href)
14+
const { Effect, Fiber, Option, Schema, Stream } = effect
315

416
const CounterState = Schema.TaggedUnion({
517
Count: {
@@ -42,6 +54,20 @@ export const counterMachine = Machine.make({
4254
}
4355
})
4456

57+
const ParentState = Schema.TaggedUnion({ Active: {} })
58+
const ParentStates = Machine.defineStates({ Active: ParentState.cases.Active })
59+
const CounterChild = Machine.child("counter", counterMachine)
60+
const counterParentMachine = Machine.make({
61+
id: "RuntimeBenchmarkCounterParent",
62+
states: ParentStates.states,
63+
events: [],
64+
initial: () => ParentStates.initial.Active(ParentState.cases.Active.make({}))
65+
}).handle({
66+
Active: {
67+
invoke: Machine.invokeMachine({ child: CounterChild })
68+
}
69+
})
70+
4571
export const incrementEvent = CounterEvent.cases.Increment.make({})
4672
const finishEvent = CounterEvent.cases.Finish.make({})
4773

@@ -67,6 +93,76 @@ export const startCounter = () => Effect.runPromise(Machine.start(counterMachine
6793

6894
export const stopCounter = (ref) => Effect.runPromise(ref.stop)
6995

96+
export const startObservedCounter = () =>
97+
Effect.runPromise(
98+
Effect.gen(function*() {
99+
const ref = yield* Machine.start(counterMachine)
100+
const observer = yield* ref.changes.pipe(Stream.runDrain, Effect.forkDetach)
101+
yield* Effect.yieldNow
102+
return { ref, observer }
103+
})
104+
)
105+
106+
export const stopObservedCounter = ({ ref, observer }) =>
107+
Effect.runPromise(
108+
ref.stop.pipe(Effect.ensuring(Fiber.interrupt(observer)))
109+
)
110+
111+
export const runObservedCounterBurst = ({ ref, observer }, size) =>
112+
Effect.runPromise(
113+
Effect.gen(function*() {
114+
for (let index = 0; index < size; index += 1) {
115+
yield* ref.send(incrementEvent)
116+
}
117+
yield* ref.send(finishEvent)
118+
const value = yield* ref.join
119+
yield* Fiber.join(observer)
120+
return value
121+
})
122+
)
123+
124+
const waitForCounterChild = (parent) =>
125+
Effect.gen(function*() {
126+
for (let attempt = 0; attempt < 1_000; attempt += 1) {
127+
const child = yield* parent.child(CounterChild)
128+
if (Option.isSome(child)) {
129+
return child.value
130+
}
131+
yield* Effect.yieldNow
132+
}
133+
return yield* Effect.dieMessage("Effect Machine child did not become ready")
134+
})
135+
136+
export const startChildCounter = () =>
137+
Effect.runPromise(
138+
Effect.gen(function*() {
139+
const parent = yield* Machine.start(counterParentMachine)
140+
yield* waitForCounterChild(parent)
141+
return parent
142+
})
143+
)
144+
145+
export const stopChildCounter = (parent) => Effect.runPromise(parent.stop)
146+
147+
export const runChildCounterBurst = (parent, size) =>
148+
Effect.runPromise(
149+
Effect.gen(function*() {
150+
for (let index = 0; index < size; index += 1) {
151+
const child = yield* parent.child(CounterChild)
152+
if (Option.isNone(child)) {
153+
return yield* Effect.dieMessage("Effect Machine child disappeared during the benchmark")
154+
}
155+
yield* child.value.send(incrementEvent)
156+
}
157+
const child = yield* parent.child(CounterChild)
158+
if (Option.isNone(child)) {
159+
return yield* Effect.dieMessage("Effect Machine child disappeared before the terminal fence")
160+
}
161+
yield* child.value.send(finishEvent)
162+
return yield* child.value.join
163+
})
164+
)
165+
70166
export const runCounterBurst = (ref, size) =>
71167
Effect.runPromise(
72168
Effect.gen(function*() {
@@ -95,13 +191,31 @@ export const stopCounters = (refs) =>
95191
})
96192
)
97193

194+
export const startChildCounters = (count) =>
195+
Effect.runPromise(
196+
Effect.forEach(
197+
Array.from({ length: count }),
198+
() =>
199+
Effect.gen(function*() {
200+
const parent = yield* Machine.start(counterParentMachine)
201+
yield* waitForCounterChild(parent)
202+
return parent
203+
}),
204+
{ concurrency: 1 }
205+
)
206+
)
207+
208+
export const stopChildCounters = stopCounters
209+
98210
export const effectMachineAdapter = {
99211
implementation: "effect-machine",
100212
label: "Effect Machine",
101213
version: undefined,
102214
async: true,
103215
planCounterBatch,
104216
runCounterBurst,
217+
runObservedCounterBurst,
218+
runChildCounterBurst,
105219
runLifecycle: async () => {
106220
const ref = await startCounter()
107221
try {
@@ -112,8 +226,24 @@ export const effectMachineAdapter = {
112226
await stopCounter(ref)
113227
}
114228
},
229+
runChildLifecycle: async () => {
230+
const parent = await startChildCounter()
231+
try {
232+
if (!parent.sessionId.startsWith("machine:")) {
233+
throw new Error(`Child lifecycle benchmark produced invalid session id ${parent.sessionId}`)
234+
}
235+
} finally {
236+
await stopChildCounter(parent)
237+
}
238+
},
115239
startCounter,
240+
startObservedCounter,
241+
startChildCounter,
116242
startCounters,
243+
startChildCounters,
117244
stopCounter,
245+
stopChildCounter,
246+
stopChildCounters,
247+
stopObservedCounter,
118248
stopCounters
119249
}

perf/runtime/implementations.mjs

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,19 @@
11
import { readFileSync } from "node:fs"
2+
import { resolve } from "node:path"
3+
import { fileURLToPath } from "node:url"
24
import * as XStateV5 from "xstate-v5"
35
import * as XStateV6 from "xstate-v6"
46
import { effectMachineAdapter } from "./counter.mjs"
57
import { makeXStateAdapter } from "./xstate.mjs"
68

79
const readPackageVersion = (path) => JSON.parse(readFileSync(path, "utf8")).version
10+
const implementationRoot = resolve(
11+
process.env.EFFECT_MACHINE_BENCHMARK_ROOT ?? fileURLToPath(new URL("../..", import.meta.url))
12+
)
813

914
export const packageVersions = {
10-
effectMachine: readPackageVersion(new URL("../../package.json", import.meta.url)),
11-
effect: readPackageVersion(new URL("../../node_modules/effect/package.json", import.meta.url)),
15+
effectMachine: readPackageVersion(resolve(implementationRoot, "package.json")),
16+
effect: readPackageVersion(resolve(implementationRoot, "node_modules/effect/package.json")),
1217
tinybench: readPackageVersion(new URL("../../node_modules/tinybench/package.json", import.meta.url)),
1318
xstateV5: readPackageVersion(new URL("../../node_modules/xstate-v5/package.json", import.meta.url)),
1419
xstateV6: readPackageVersion(new URL("../../node_modules/xstate-v6/package.json", import.meta.url))

perf/runtime/memory-worker.mjs

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,12 +8,33 @@ if (typeof globalThis.gc !== "function") {
88

99
const implementationId = process.argv[2]
1010
const counts = JSON.parse(process.argv[3] ?? "[]")
11+
const profileId = process.argv[4] ?? "idle"
1112
const implementation = implementations.find((candidate) => candidate.implementation === implementationId)
1213

1314
if (implementation === undefined) {
1415
throw new Error(`Unknown runtime benchmark implementation: ${implementationId}`)
1516
}
1617

18+
const profile = profileId === "idle"
19+
? {
20+
id: "idle",
21+
label: "Idle machine",
22+
start: implementation.startCounters,
23+
stop: implementation.stopCounters
24+
}
25+
: profileId === "parent-with-child"
26+
? {
27+
id: "parent-with-child",
28+
label: "Idle parent with one child",
29+
start: implementation.startChildCounters,
30+
stop: implementation.stopChildCounters
31+
}
32+
: undefined
33+
34+
if (profile === undefined) {
35+
throw new Error(`Unknown runtime memory profile: ${profileId}`)
36+
}
37+
1738
const collectGarbage = async () => {
1839
for (let index = 0; index < 3; index += 1) {
1940
globalThis.gc()
@@ -35,8 +56,8 @@ const linearSlope = (points, value) => {
3556
}
3657

3758
// Trigger implementation-specific lazy initialization before taking the baseline.
38-
const warmup = await implementation.startCounter()
39-
await implementation.stopCounter(warmup)
59+
const warmup = await profile.start(1)
60+
await profile.stop(warmup)
4061
await collectGarbage()
4162

4263
const baseline = process.memoryUsage()
@@ -45,7 +66,7 @@ const refs = []
4566

4667
try {
4768
for (const count of counts) {
48-
refs.push(...await implementation.startCounters(count - refs.length))
69+
refs.push(...await profile.start(count - refs.length))
4970
await collectGarbage()
5071
const usage = process.memoryUsage()
5172
points.push({
@@ -57,14 +78,16 @@ try {
5778
})
5879
}
5980
} finally {
60-
await implementation.stopCounters(refs)
81+
await profile.stop(refs)
6182
await collectGarbage()
6283
}
6384

6485
process.stdout.write(JSON.stringify({
6586
implementation: implementation.implementation,
6687
implementationLabel: implementation.label,
6788
implementationVersion: implementation.version,
89+
id: profile.id,
90+
label: profile.label,
6891
baseline: {
6992
heapUsedBytes: baseline.heapUsed,
7093
rssBytes: baseline.rss

0 commit comments

Comments
 (0)