Context
The release wheel-bundling workflow (bundle-pypi-wheels.yml) builds GDAL from source on Linux and via
vcpkg on win_arm64. Both paths re-download every dependency on every run: the Linux from-source build
pulls ~26 SHA256-pinned tarballs from ~30 upstream hosts, and the win_arm64 vcpkg build re-fetches every
port's source. A single 503/429/outage from any one host (e.g. the DKRZ libaec host, lcms2, libwebp,
openjpeg) fails the whole build — this repeatedly broke releases during GitHub/CDN incidents.
Delivered on branch ci/cache-source-distfiles (PR #970).
Problem / Current Behaviour
The existing compiled-stack cache stores built binaries and is deliberately gated off the release
(workflow_run) path to avoid a cache-poisoning-to-PyPI vector — so releases always cold-build and
re-download everything, with no resilience against a flaky upstream host. There was no caching of the
downloaded source itself.
Affected locations
| File |
Symbol |
Notes |
.github/workflows/bundle-pypi-wheels.yml |
glibc / musl / win_arm64 build jobs |
new restore/save cache steps |
ci/source-build/config.sh |
fetch() |
cache-aware download + SHA-verify + persist |
ci/source-build/build-gdal-stack.sh |
DISTFILES_DIR plumbing |
export dir through the /host mount |
.gitignore |
— |
ignore the new working dirs |
Proposed Solution
Cache the downloaded source (not binaries), which is safe to keep on the release path because every
artifact is re-verified after restore:
- Linux source-distfile cache:
fetch() reuses a SHA256-matching cached tarball, else wgets it,
verifies either way, and persists the verified tarball to a DISTFILES_DIR plumbed through
cibuildwheel's /host mount. Arch-independent shared key seeds all four glibc/musl jobs.
- win_arm64 vcpkg downloaded-assets cache over
C:\vcpkg\downloads (vcpkg SHA512-verifies on use).
- Un-gated off
workflow_run: per-tarball SHA256/SHA512 re-verification means a stale or tampered
cache entry can never place non-pin bytes into a published wheel, so the optimization is release-safe.
- Cache-integrity guards: refresh on SHA mismatch (not only when absent) so a stale same-name distfile
self-heals; split each cache into restore + save gated on success() and a compiled/installed
cache miss, so neither a failed mid-fetch nor a build served entirely from the compiled cache can seed
the write-once key with a partial or empty set.
- Key the vcpkg downloads cache on the
ci/vcpkg-ports/** overlay so the libaec redirect rotates the key.
Out of Scope
- The compiled-stack / vcpkg-
installed caches keep their existing workflow_run gating (unchanged).
- No change to which wheels are built or published; the
DISTFILES_DIR-unset path is byte-for-byte the
old download-only behaviour.
Effort Estimate
Size: M
Rationale: touched four CI-infra files with subtle GitHub-Actions cache semantics (write-once keys,
restore-keys prefix fallback, restore/save split) and required a live dry-run to prove cache hits.
Definition of Done
Context
The release wheel-bundling workflow (
bundle-pypi-wheels.yml) builds GDAL from source on Linux and viavcpkg on win_arm64. Both paths re-download every dependency on every run: the Linux from-source build
pulls ~26 SHA256-pinned tarballs from ~30 upstream hosts, and the win_arm64 vcpkg build re-fetches every
port's source. A single 503/429/outage from any one host (e.g. the DKRZ libaec host, lcms2, libwebp,
openjpeg) fails the whole build — this repeatedly broke releases during GitHub/CDN incidents.
Delivered on branch
ci/cache-source-distfiles(PR #970).Problem / Current Behaviour
The existing compiled-stack cache stores built binaries and is deliberately gated off the release
(
workflow_run) path to avoid a cache-poisoning-to-PyPI vector — so releases always cold-build andre-download everything, with no resilience against a flaky upstream host. There was no caching of the
downloaded source itself.
Affected locations
.github/workflows/bundle-pypi-wheels.ymlci/source-build/config.shfetch()ci/source-build/build-gdal-stack.shDISTFILES_DIRplumbing/hostmount.gitignoreProposed Solution
Cache the downloaded source (not binaries), which is safe to keep on the release path because every
artifact is re-verified after restore:
fetch()reuses a SHA256-matching cached tarball, elsewgets it,verifies either way, and persists the verified tarball to a
DISTFILES_DIRplumbed throughcibuildwheel's
/hostmount. Arch-independent shared key seeds all four glibc/musl jobs.C:\vcpkg\downloads(vcpkg SHA512-verifies on use).workflow_run: per-tarball SHA256/SHA512 re-verification means a stale or tamperedcache entry can never place non-pin bytes into a published wheel, so the optimization is release-safe.
self-heals; split each cache into
restore+savegated onsuccess()and a compiled/installedcache miss, so neither a failed mid-fetch nor a build served entirely from the compiled cache can seed
the write-once key with a partial or empty set.
ci/vcpkg-ports/**overlay so the libaec redirect rotates the key.Out of Scope
installedcaches keep their existingworkflow_rungating (unchanged).DISTFILES_DIR-unset path is byte-for-byte theold download-only behaviour.
Effort Estimate
Size:
MRationale: touched four CI-infra files with subtle GitHub-Actions cache semantics (write-once keys,
restore-keys prefix fallback, restore/save split) and required a live dry-run to prove cache hits.
Definition of Done
(proven: run 31634004496 — restore=1, compiled-miss=1, 26
using cached, 0 downloads)C:\vcpkg\downloads, keyed on the ports overlay