perf(ci): cache smoke downloads, dedup portable, narrow detect#56
Merged
Conversation
… detect
Three independent levers to cut smoke-CI time (was: every run re-downloads
xim:python/make, gcc/llvm toolchains, and every compat source archive — only
~/.mcpp/registry was cached).
1. Cross-run download cache: add actions/cache for the smoke download dir
(${runner.temp}/mcpp-smoke-cache) in smoke-full-linux AND smoke-portable,
keyed by MCPP_VERSION + a restore-keys prefix so a partial hit survives any
pkg/test edit (only changed archives re-download).
2. Portable within-run dedup: smoke-portable never set MCPP_INDEX_SMOKE_CACHE_DIR
and smoke_compat_portable.sh had only a restore half — so its 5 projects each
re-downloaded the full llvm toolchain + archives within one run. Add the
save_smoke_cache half (+ pbuild wrapper routing all 5 builds through it) and
wire MCPP_INDEX_SMOKE_CACHE_DIR in the portable job.
3. Narrow detect: smoke-full-linux and smoke-portable run disjoint scripts on
disjoint platforms, so split the single full=true into full_linux/full_portable.
A change to one platform's harness no longer triggers the other's suite
(a smoke_compat_portable.sh edit skips the ~12-min Linux job; a Linux smoke
script edit skips win+mac). Common harness, unclassified test changes, and
pkgs without an example still fall back to BOTH — no coverage is dropped.
Classification verified against 9 representative changed-file sets.
Diagnosis: see memory mcpp-index-ci-repeated-downloads. Python is pulled only by
compat.xcb (build-dep, transitively via x11/glfw), Linux-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts smoke-CI time. Today every run re-downloads
xim:python/xim:make, the gcc/llvm toolchains, and every compat source archive — the onlyactions/cachepath is~/.mcpp/registry(just the static prebuilt registry from the mcpp tarball). Three independent levers:1. Cross-run download cache. Add
actions/cachefor the smoke download dir (${{ runner.temp }}/mcpp-smoke-cache) in smoke-full-linux and smoke-portable, keyedsmoke-dl-<os>-<mcppver>-<hash(pkgs)>with asmoke-dl-<os>-<mcppver>-restore-keys prefix → a partial hit survives any pkg/test edit; only changed archives re-download.2. Portable within-run dedup.
smoke-portablenever setMCPP_INDEX_SMOKE_CACHE_DIR, andsmoke_compat_portable.shhad only a restore half — so its 5 projects each re-downloaded the full llvm toolchain + archives within one run. Added thesave_smoke_cachehalf (+ apbuildwrapper routing all 5 builds through it) and wired the env var in the job.3. Narrow
detect.smoke-full-linuxandsmoke-portablerun disjoint scripts on disjoint platforms, so the singlefull=trueis split intofull_linux/full_portable. A change to one platform's harness no longer triggers the other's suite (editingsmoke_compat_portable.shskips the ~12-min Linux job; editing a Linux smoke script skips win+mac). Common harness, unclassifiedtests/changes, and pkgs without an example still fall back to BOTH — no coverage is silently dropped. Classification was unit-tested against 9 representative changed-file sets.Python is pulled only by
compat.xcb(build-depxim:python@latest, transitively via x11/glfw); Linux-only.compat.xorgproto/compat.x11do not shell out to Python.Note: this PR touches
validate.yml+ a smoke script, so per lever 3 it correctly runs BOTH suites (validate.yml = common harness).