CI: parallelize wheel builds per Python version + sccache caching#117
Merged
Conversation
Split the build_wheels matrix so each CPython version (cp310–cp314) is its own job per architecture instead of one serial cibuildwheel loop per runner — 4 arch × 5 versions = 20 parallel jobs. Since the extension is not abi3, each version is a distinct wheel, so the split is a real ~5x wall-clock cut on the per-runner build. Add sccache (compiler-level cache) backed by the GitHub Actions cache so the parallel jobs don't each recompile dependencies from scratch: - RUSTC_WRAPPER=sccache + SCCACHE_GHA_ENABLED on all platforms; sccache keys on compiler inputs, so dependency crates are reused across runs and across the parallel per-version jobs. - CARGO_INCREMENTAL=0 so artifacts are cacheable. - Linux wheels build inside the manylinux container, which shares neither the host filesystem nor host env, so: install the sccache binary in the container via CIBW_BEFORE_ALL_LINUX, and forward the sccache config plus the GHA cache credentials (exported by sccache-action) into the container via CIBW_ENVIRONMENT_PASS_LINUX. CIBW_BUILD pins each job to its one version; sdist is emitted once (ubuntu-latest + cp313); artifacts renamed to include the python tag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Linux wheel jobs failed: sccache aborted with "environment variable not found" on `rustc -vV`. GitHub runners now use the cache service v2 (ACTIONS_CACHE_SERVICE_V2=on, ACTIONS_RESULTS_URL), but the binary installed inside the manylinux container was sccache 0.8.2, which only speaks the retired v1 service (ACTIONS_CACHE_URL) and exits when it's absent. macOS/Windows passed because they use the host's 0.15.0 from sccache-action. Pin the container install to 0.15.0 to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Speeds up the PyPI wheel build in the Release workflow two ways:
Parallel matrix — split
build_wheelsso each CPython version (cp310–cp314) is its own job per architecture, instead of one runner serially looping through all five via cibuildwheel. Matrix is now4 arch × 5 versions = 20 parallel jobs. Since the extension is notabi3, each version is a distinct wheel, so the split is a real ~5× wall-clock cut on the per-runner build.sccache caching — compiler-level cache (
RUSTC_WRAPPER=sccache,SCCACHE_GHA_ENABLED) backed by the GitHub Actions cache, withCARGO_INCREMENTAL=0so artifacts are cacheable. sccache keys on compiler inputs, so dependency crates (nalgebra, pyo3, …) are reused across runs and across the parallel per-version jobs — which is what pays back the redundant dependency compilation the matrix split would otherwise introduce.Linux container plumbing
Linux wheels build inside the manylinux container, which shares neither the host filesystem nor host env, so:
mozilla-actions/sccache-actioninstalls sccache on the host and exports theACTIONS_*cache creds into the step env.CIBW_BEFORE_ALL_LINUXinstalls the sccache binary inside the container (pinned to 0.15.0, which speaks the current GHA cache service v2 /ACTIONS_RESULTS_URL; older sccache only knows the retired v1 /ACTIONS_CACHE_URLand aborts with "environment variable not found").CIBW_ENVIRONMENT_PASS_LINUXforwards the sccache config + GHA cache credentials into the container.CIBW_BUILDpins each job to its one version; sdist is emitted once (ubuntu-latest + cp313); artifacts renamed to include the python tag.Validation
Triggered via
workflow_dispatchon this branch (run 26852358627) — all 20 wheel jobs green, sdist built,publish_pypicorrectly skipped (gated onrefs/tags/v*). sccache reported a 94% hit rate on a warm macOS job (deps + C/assembler fully cached; only satkit's own crates miss).🤖 Generated with Claude Code