chore(vitest): run tests from the repo root with balanced shards - #6477
Draft
kanadgupta wants to merge 6 commits into
Draft
chore(vitest): run tests from the repo root with balanced shards#6477kanadgupta wants to merge 6 commits into
kanadgupta wants to merge 6 commits into
Conversation
kanadgupta
added
the
run-ci
kanadgupta
force-pushed
the
kanad-claude/root-test-runs
branch
from
September 5, 2026 00:23
84b29a8 to
ed8a45e
Compare
Contributor
Supabase CLI previewnpx --yes https://pkg.pr.new/supabase/cli/supabase@221aa9eccaab90c5d9d72f0ced0fe731ac7a1978Preview package for commit |
Unit and integration no longer pass through Turbo: the root test, test:unit, and test:integration scripts run one Vitest process over every package's projects, and package scripts call Vitest directly with no :run layer. e2e stays a Turbo task because it depends on the built CLI, but as one root task (//#test:e2e:run, dependsOn supabase#build) that runs every package's e2e projects in one process. The per-package test Turbo tasks and the temporary test:vitest script are gone; the Go workspace's test:unit calls go test directly. The root config gains a sequencer that deals e2e-stack files round-robin by sorted path and then the stackless files, so shards get the same number of each class and compute the same partition independently; Vitest's default hash slicing put the serial stack-backed files on shards by luck. Sorting within a project is lexicographic, which the compatibility suite relied on from its own sequencer (still applied to standalone runs). The root config also carries the merged coverage options for the develop coverage workflow, with repo-relative include patterns. fsModuleCache is on in the shared preset, and the preset is now .mts so Vite stops warning about loading ESM as CommonJS on every run. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Unit and integration become one two-shard matrix job through the root test script; the summary job keeps the required check name. e2e runs through the root Turbo task with --shard=N/3, so the balanced sequencer sees every package's e2e files at once. Both jobs restore node_modules/.vitest-cache so the file-system module cache has something to hit. --coverage.enabled leaves the PR jobs: nothing consumed the per-package tables and instrumentation taxed every run. A develop-push workflow produces one merged report and keeps it as an artifact. Vitest flags are passed straight after the script name. pnpm forwards a literal -- to the script and Vitest treats everything after -- as file filters, which silently dropped --shard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The first CI run of the root sequencer split unit and integration roughly 27 seconds against six minutes. Sorting every file by path puts x.integration.test.ts immediately before x.unit.test.ts, so alternating positions handed one shard nearly all integration files and the other all unit files. Dealing round-robin within each test project, with a running offset across projects, halves every project instead: 284/283 files with each project split evenly, and the e2e-stack projects, dealt first, still spread 5/4/4 across three shards. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kanadgupta
force-pushed
the
kanad-claude/root-test-runs
branch
from
September 5, 2026 00:51
245f1a0 to
1967ead
Compare
Neither file count nor file size predicts test duration here: dealing by count left the worst e2e shard at 305s of serial stack-backed work against a 195s ideal, and size weighting made it worse (a 1.2 KB start test takes 42s, a 20 KB native createStack test 9s). Only measured durations balance it, so they are taken from the run that already exists. develop-tests.yml (replacing coverage.yml) runs unit, integration, and the three e2e shards on every develop push with Vitest's json reporter, then tools/test-shard-weights.ts merges the per-file durations across shards into .vitest/shard-weights.json, keeping the previous value for files that did not run, and saves it to the Actions cache. The coverage report moves into the same workflow. In test.yml the gate job restores that file once and publishes it as a run artifact that every shard downloads: shards must partition from identical input, and a prefix cache restore in each shard could pick up different generations and silently drop files. Caches saved from PR or merge-queue runs are invisible to other branches, which is why the writer runs on develop. The root sequencer assigns files largest-first to the least-loaded shard within each class, stack-backed first, with the class median for files without a recorded duration; without the file it keeps dealing by count. Simulated on this branch's CI durations, stack-backed e2e lands at 196/195/195s per shard. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… CONTRIBUTING Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kanadgupta
force-pushed
the
kanad-claude/root-test-runs
branch
from
September 5, 2026 02:00
02c2254 to
221aa9e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #6473 (PR B), which is stacked on #6472 (PR A). The base is the B branch so only PR C's changes show here; retarget to
developonce B merges. Draft withrun-ci.Summary
Third PR from
docs/superpowers/plans/2026-09-04-test-execution-topology.mdand the one that changes how tests run. ADR 0024 records the decisions; this PR implements them.One root Vitest run is the unit of execution. The root
test,test:unit, andtest:integrationscripts run a single Vitest process over every package's projects, and CI runs that process split two ways with--shard.test:e2eis the one script that still goes through Turbo: it is a root task,//#test:e2e:run, that depends onsupabase#build, so Turbo keeps the part it is good at, the build graph, and one root Vitest run then covers every package's e2e projects. The per-packagetest:*:runTurbo tasks, the:runscript layer, and the temporarytest:vitestscript are gone; package scripts call Vitest directly. The Go workspace'stest:unitcallsgo testdirectly.Balanced shards, from develop's own durations. Vitest's default sharding sorts files by path hash and cuts contiguous slices, which is why the stack-backed files landed on shards by luck (7.9 / 5.6 / 3.3 minutes of test time on develop, and still 5.0 / 3.8 / 6.0 after #6472). Neither file count nor file size predicts duration here, so the sequencer uses measured durations:
develop-tests.ymlruns the full suite on every develop push with Vitest'sjsonreporter,tools/test-shard-weights.tsmerges the per-file durations across shards into.vitest/shard-weights.json, and the file is saved to the Actions cache. A PR run's gate job restores it once and publishes it as a run artifact, so every shard partitions from byte-identical input (a prefix cache restore in each shard could pick up different generations and silently drop files). The sequencer assigns files largest-first to the least-loaded shard within each class, stack-backed first, and files without a recorded duration take the class median. Simulated on this branch's own CI durations the stack-backed e2e work lands at 196 / 195 / 195 seconds per shard against 305 today. With no weights file, which is the case locally and until develop has run once, files are dealt round-robin within each test project, which still halves every project across shards. Sorting within a project stays lexicographic for the compatibility suite inapps/cli-e2e. Caches saved from PR or merge-queue runs are invisible to other branches, which is why the writer has to be the develop workflow, at the cost of one extra e2e run per merge.CI shape. Unit and integration become one two-shard matrix job through the root
testscript; the summary job keeps the required check name. E2e runs through the root task with--shard=N/3. The--coverage.enabledflags leave the PR jobs. Both test jobs restorenode_modules/.vitest-cachebetween runs so the file-system module cache, now on in the shared preset, has something to hit; transform and import were about a third of the CLI unit run.Coverage moves to develop. The same
develop-tests.ymlruns the roottestwith--coverage.enabledand uploads the merged lcov as an artifact. The root config carries the coverage options with repo-relative include patterns, since Vitest 5 matches coverage globs without a "contains" fallback, and the CLI's exclude list prefixed accordingly.Verified locally.
pnpm run test --shard=1/2and--shard=2/2partition the 567 unit and integration files 284/283 with every project halved and no overlap; the root e2e task honours file filters through Turbo; both CI shards of the unit-plus-integration run pass. One flake surfaced once in a full run,supervisor.integration.test.tsbounding attached-owner recovery to one startup deadline, and passed on the other run; it is unrelated to this change but worth an eye. The native PostgREST e2e failure seen in this PR's first two CI runs was a gateway bug that PR B's preset uncovered; it is fixed in #6473 and this branch is rebased on that fix.Small things.
vitest.shared.tsbecomes.mtsso Vite stops warning about loading ESM as CommonJS on every run. Docs:CONTRIBUTING.md,AGENTS.md,apps/cli/AGENTS.md, andapps/cli-e2e/AGENTS.mddescribe the new script layout and the root-run sharding.vitest doctoronsupabase (unit)Run once on this branch (Vitest 5.0.0, Bun 1.4.1, Apple Silicon), against the CLI package's unit project:
pool: forks,isolate: true)pool: 'threads'Setting process.umask() is not supported in workersinlegacy-upgrade-notice.unit.test.tsisolate: falsemaxWorkers: 4Doctor's own recommendation: keep the current configuration; no candidate was more than 10% faster.
No isolation or pool changes are made in this PR; about 40 test files depend on per-file isolation today (see ADR 0024).
🤖 Generated with Claude Code