Repository navigation
ci: parallelize and shard the test suite, restore Go caches - #32
Draft
paulbalaji wants to merge 3 commits into
Draft
paulbalaji wants to merge 3 commits into
paulbalaji wants to merge 3 commits into
Conversation
internal/compiler ran its ~270 top-level tests serially, mostly process-bound differential and SBF runs, and took 8-9 minutes of each CI job. Tests that use only their own temp dirs and read-only inputs now call t.Parallel(). These stay serial: - tests that set environment variables or can rewrite tracked files (GOSVM_UPDATE_*); - the buildMemoryVariant callers, which set the package-level memorySyscallMin that every concurrent build reads.
setup-go looked for a root go.sum, which this module does not have, so every run started with empty module and build caches. Key the cache on go.mod and the example go.sum files, and pass -count=1 to the test steps so a restored cache never replaces a test run with a cached result. A Go test process starts child processes one at a time on macOS, so t.Parallel alone barely moved the darwin job. scripts/test_shards.py runs the make test suite with internal/compiler split round-robin across one go test process per CPU, checks that the shards cover the package's test list exactly, and runs the other packages alongside.
Shard the make test suite across four runners per platform, packed by the recorded per-test durations in scripts/test_timings.json, and move vet, the example modules, walkthroughs and the reference ELF rebuild into a checks job per platform that runs alongside. ci-ok aggregates every job for branch protection.
paulbalaji
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
CI takes 13-14 minutes. Two causes:
internal/compilerruns ~270 tests serially, about 500 s of each job.setup-gonever restores its cache, because there's no rootgo.sum.Solution
t.Parallel()on hermetic compiler tests. The fourbuildMemoryVariantcallers stay serial: they set the package-levelmemorySyscallMin, which every concurrent build reads, so running them in parallel produced different ELFs for the same source.scripts/test_shards.pysplits the wholemake testsuite into shards using per-test durations measured in CI. It fails unless every test lands in exactly one shard. Inside a shard it runs onego testprocess per CPU, because Go launches child processes one at a time on macOS.checksjob per platform (vet, examples, walkthroughs, reference ELF rebuild) runs alongside them, so macOS uses GitHub's limit of 5 concurrent jobs.-count=1.ci-okdepends on every job, so branch protection only needs that one check.Benchmarks
Fork CI,
main(939b215) vs this branch, warm caches, mean of 2 runs:A fully cold run takes 312 s and 48.7 runner-minutes (95% more), mostly because every job downloads platform-tools.
About 2 minutes is the floor here. On a loaded runner, single tests take 60-105 s (
TestRealLibrariesSBF,TestMapSBF) and setup takes 15-35 s. Getting lower would mean splitting subtests across shards or getting more macOS concurrency.Tests
Only tests, the workflow and the script changed. The compiler itself is untouched. The reference ELF rebuild, corpus test, SBF on both profiles and the example suites all still run. All green on the fork and here.
Linked work
None.