ci(wheels): cache SHA-verified source distfiles + vcpkg assets on every build - #970
Merged
Conversation
…ry build Every bundle run — releases included — re-downloaded all source tarballs from ~30 upstream hosts, so a single 503/429 from any one failed the build (the lcms2 GitHub 503 in run 31621639572; the earlier DKRZ libaec 429). The existing caches store COMPILED binaries and are (correctly) skipped on the release path to avoid cache-poisoning-to-release. Source distfiles are a different, safe thing to cache: they are integrity-verified before use, so a tampered/corrupt cache entry is rejected and re-downloaded rather than baked into a wheel. - Linux from-source: build-gdal-stack.sh exports a DISTFILES_DIR (a sibling of the compiled-stack cache on the /host mount); config.sh's fetch() reuses a cached tarball only when it matches the pinned SHA256, else downloads and persists it. A new, UN-gated actions/cache (arch-independent key on config.sh) keeps the tarballs across runs — including releases — so the common path makes zero upstream requests. - win_arm64 vcpkg: an UN-gated actions/cache of C:\vcpkg\downloads persists the SHA512-verified port distfiles (vcpkg re-verifies on use), the same safety property. Both invalidate only when the pins change (config.sh / vcpkg.json hash). GHA cache is best-effort (branch scope, eviction), so a first run after a bump still downloads; a persistently-down host still needs a mirror/overlay.
…autysh beautysh could not parse the backslash-continued [[ ]] condition (indent/outdent mismatch). Compute the cached tarball's SHA into a variable first, then branch on a single-line condition — identical behavior, no continuation.
…sent fetch()'s persist guard wrote the cache only when the cached file was absent. A cached tarball whose SHA no longer matches its pin (upstream re-upload of the same version, mirror swap, corrected pin, or a partial write) correctly falls through to a fresh wget + SHA-verify, but the guard then skipped the write-back, so the stale file stayed cached forever and that dep was re-downloaded from its (possibly flaky) upstream on every build — the exact fragility this cache removes, silently defeated for that dep. Since the tarball is SHA-verified immediately above, overwrite unconditionally so the cache self-heals.
…can't seed a partial actions/cache saves at post-job even when the build failed, and the four Linux jobs share one write-once distfiles key — so a job aborting mid-fetch (the flaky-host case this cache targets) would seed the shared key with a partial tarball set that never self-heals until config.sh's hash rotates. The same partial-save-on-failure applies to the win_arm64 vcpkg downloads cache (which is why its seed never took today — every failed run saved a partial). Replace the combined actions/cache with actions/cache/restore (before the build) + actions/cache/save gated on success() and a non-exact-key restore, for all three new caches. A failed or partial fetch no longer seeds the immutable key; only a fully-built run does.
The vcpkg downloads cache keyed on ci/vcpkg.json only, but the overlay port ci/vcpkg-ports/libaec redirects libaec off the DKRZ host that the cache's own comment cites as its reason to exist. If that portfile's URL/version/SHA512 changes, the download set changes but the key would not rotate. Add ci/vcpkg-ports/** to the key hash (restore + save kept in step) so the cache tracks the overlay. Self-healing via vcpkg's SHA512 re-verify regardless, so this is a freshness fix, not a correctness one.
…che-hit The distfile/downloads dir is only populated by config.sh's fetch() loop (or vcpkg install), which build-gdal-stack.sh skips on a compiled-stack cache hit (mkdir -p still leaves the dir empty). In the reachable state 'compiled cache hit + distfile exact-key miss' (the two entries evict independently under GitHub's 10 GB LRU pool), the save fired on the restore-keys prefix match (cache-hit != 'true') and persisted an empty/stale set under the immutable exact key. The release path (compiled cache gated off) then restored that exact key with cache-hit == 'true', skipped its own save, and was forced to re-download every tarball from its upstream forever — silently reverting releases to the flaky-download behavior this cache exists to eliminate. Give each compiled/installed cache step an id and AND its cache-hit != 'true' into the distfile/downloads save condition, so the save fires only when the source build actually ran and populated the dir. On the release path the compiled step is skipped, so its empty cache-hit output keeps the distfile save active there, as intended. No wheel-correctness change (the SHA re-verify already guaranteed that); this restores the resilience benefit on the release path.
…the SHA pin The distfile cache key hashes config.sh only, while the GDAL tarball name also depends on GDAL_VERSION defaulted in build-gdal-stack.sh. This is safe only because the GDAL SHA is pinned inline in config.sh (a bump rotates the key); record that in the key comment so a future maintainer who decouples GDAL_VERSION into a workflow input remembers to add build-gdal-stack.sh to the key.
…ing dirs The from-source GDAL build persists both dirs into the runner workspace (the compiled-stack tar and the SHA-verified source distfiles). Like the existing un-ignored .srcbuild-cache, they were untracked but not ignored; ignore both so neither can be swept into an sdist or trip a future working-tree-cleanliness gate.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Every bundle run — releases included — re-downloads all source tarballs from ~30 upstream hosts, so a single 503/429
from any one fails the whole build (the lcms2 GitHub 503 in run 31621639572; the earlier DKRZ libaec 429). The
existing caches store compiled binaries and are (correctly) skipped on the release path to avoid
cache-poisoning-to-release. Source distfiles are a different, safe thing to cache: they are integrity-verified
before use, so a tampered/corrupt cache entry is rejected and re-downloaded rather than baked into a wheel.
build-gdal-stack.sh+config.sh): exports aDISTFILES_DIR(a sibling of thecompiled-stack cache on the
/hostmount);fetch()reuses a cached tarball only when it matches the pinnedSHA256, else downloads and persists it. A new un-gated
actions/cache(arch-independent key onconfig.sh,shared by the glibc + musl jobs) keeps the ~26 tarballs across runs — including releases — so the common path
makes zero upstream requests.
actions/cacheofC:\vcpkg\downloadspersists vcpkg's SHA512-verifiedport distfiles (vcpkg re-verifies on use).
Both invalidate only when the pins change (
config.sh/vcpkg.jsonhash). GHA cache is best-effort (branch scope,eviction), so a first run after a bump still downloads; a persistently-down host still needs a mirror/overlay.
Issues
Type of change
How Has This Been Tested?
bash -nclean on both shell scripts; workflow YAML parses; no lines over 120.Simulated
fetch(): cache-hit path taken on a SHA256 match; emptyDISTFILES_DIRreverts to the exact olddownload-only behavior (safe fallback).
A
bundle-pypi-wheelsdry-run seeds the caches on this run; a second dry-run confirms the Linux build reuses thecached distfiles ("using cached ... from ...") and makes no upstream downloads.
Shell syntax + YAML + fetch() cache logic validated locally
Dry-run seed + hit confirmed in CI
Checklist: