Skip to content

Retry dave build on transient registry failures #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aa8y merged 4 commits into from
Aug 30, 2026
Merged

Retry dave build on transient registry failures #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
393fdf5
Retry dave build on transient registry failures
aa8y Aug 30, 2026
d820be5
Pre-pull base images once per shard to stay inside the pull quota
aa8y Aug 30, 2026
ff8e324
Serialize the PR cache export out of the build to stop mid-build 429s
aa8y Aug 30, 2026
aeb1928
Pin pagila to its pre-pgvector commit
aa8y Aug 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 80 additions & 10 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,16 +132,20 @@ jobs:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

# Opt in to exporting this shard's build cache under a PR scope
# (buildcache-<tag>-<arch>-pr), so trusted PR builds upload the layers
# they just built without ever clobbering the master-owned
# buildcache-<tag>-<arch> refs the import side trusts. Gated on the
# same-repo condition above: on fork PRs the guard is false, the
# variable stays unset, and the manifest's ${CACHE_TO_SCOPE:+...} guard
# erases the --cache-to flag entirely.
- name: Export PR build cache
# Opt in to importing the PR-scoped cache refs (buildcache-<tag>-<arch>-pr)
# during every build below, so a re-run attempt of this PR hits the cache
# its own earlier attempt exported. Import only: the matching EXPORT
# (CACHE_TO_SCOPE) is deliberately not set here but on the dedicated
# serialized step after the build -- a mode=max cache export is a heavy
# registry write, and with 10 shards building at -j 3 the fleet peaks at
# ~30 simultaneous writers, which trips Docker Hub's 429 rate limiting
# mid-build (the same arithmetic that forced `dave push` to -j 1). Gated
# on the same-repo condition above: on fork PRs the guard is false, the
# variable stays unset, and the manifest's ${CACHE_FROM_SCOPE:+...} guard
# erases the flag entirely.
- name: Import PR build cache
if: env.REGISTRY_AUTH == 'true' && github.event_name == 'pull_request'
run: echo "CACHE_TO_SCOPE=pr" >> "$GITHUB_ENV"
run: echo "CACHE_FROM_SCOPE=pr" >> "$GITHUB_ENV"

# Last-known-good upstream fingerprints, carried across runs. When a live
# metadata fetch fails, bin/dataset-checksum falls back to the newest
Expand Down Expand Up @@ -200,10 +204,76 @@ jobs:
run: |
echo "DATASET_CHECKSUM_CACHE=$DATASET_CHECKSUM_CACHE" >> "$GITHUB_ENV"
echo "DATASET_CHECKSUM_FALLBACK=$DATASET_CHECKSUM_FALLBACK" >> "$GITHUB_ENV"
# Both persisted dirs exist even if the shard dies before the step
# that would first write them, so the always() cache saves below
# never warn about a missing path.
mkdir -p "$DATASET_CHECKSUM_FALLBACK" "${{ runner.temp }}/itest-stamps"
bin/dataset-checksums -c ${{ matrix.context }}

# The builds themselves must not be what pulls the bases: each context
# has only two or three distinct FROM images, but ~21 tags per shard
# each resolve them against Docker Hub independently, and 10 shards
# doing that from one account exhausts the shared hourly pull quota --
# once it is gone, no amount of minutes-scale backoff brings it back
# within the job timeout. So pull each distinct base exactly once here
# (a handful of pulls per shard, well inside the quota) and let every
# build resolve FROM from the local image store. Stage aliases (`FROM
# engine`) are names defined by an earlier `FROM ... AS`, not registry
# refs, and are skipped.
- name: Pre-pull base images
run: |
aliases="$(awk 'toupper($1)=="FROM" && toupper($3)=="AS" {print $4}' ${{ matrix.context }}/Dockerfile | tr '\n' ' ')"
images="$(awk 'toupper($1)=="FROM" {print $2}' ${{ matrix.context }}/Dockerfile | sort -u)"
for img in $images; do
case " $aliases " in *" $img "*) continue ;; esac
for attempt in 1 2 3 4 5; do
if docker pull --platform "$PLATFORM" "$img"; then
break
fi
if [ "$attempt" -eq 5 ]; then
echo "pull of $img failed after 5 attempts" >&2
exit 1
fi
delay=$((attempt * 60))
echo "pull of $img attempt $attempt failed; retrying in ${delay}s" >&2
sleep "$delay"
done
done

# Retried like the push below, and for the same reason: with the bases
# pre-pulled above the remaining registry traffic is buildcache manifest
# resolution, which can still catch a transient 429. A second
# `dave build` finds every already-built layer in the local store, so a
# retry redoes only the tags a failure actually killed.
- name: Build images
run: dave build -c ${{ matrix.context }} -j "$DAVE_JOBS_BUILD"
run: |
for attempt in 1 2 3; do
if dave build -c ${{ matrix.context }} -j "$DAVE_JOBS_BUILD"; then
exit 0
fi
if [ "$attempt" -eq 3 ]; then
echo "build failed after 3 attempts" >&2
exit 1
fi
delay=$((attempt * 60))
echo "build attempt $attempt failed; retrying in ${delay}s" >&2
sleep "$delay"
done

# The PR cache export, moved out of the build itself (see the Import
# step above for the concurrency arithmetic). Re-running `dave build`
# with CACHE_TO_SCOPE set resolves every layer from the local store --
# the same full-cache-hit trick the Push step uses -- so this is
# upload-only, and -j 1 keeps the fleet at ~10 concurrent writers, the
# level the registry tolerates. continue-on-error because the cache is
# purely an optimization for the next attempt: a throttled export must
# never fail a shard whose images built and will be tested regardless.
- name: Export PR build cache
if: env.REGISTRY_AUTH == 'true' && github.event_name == 'pull_request'
continue-on-error: true
env:
CACHE_TO_SCOPE: pr
run: dave build -c ${{ matrix.context }} -j 1

- name: Test images (structure)
run: dave structure-test -c ${{ matrix.context }} -j "$DAVE_JOBS_ST"
Expand Down
44 changes: 28 additions & 16 deletions manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,15 +39,20 @@ templates:
# run still gets a warm cache from before the split.
# CACHE_TO_SCOPE build: additionally EXPORT the cache, under a scoped ref
# (`...-<scope>`). Absent unless set, so an ordinary local
# build still never writes to the registry. The build
# template also IMPORTS that same scoped ref when the
# variable is set: a same-repo PR exports to its own scope,
# and without importing it a re-run attempt of that PR could
# not hit the cache its own earlier attempt just wrote -- it
# would rebuild, land on a different image ID, and so miss
# the integration-test skip stamps keyed on that ID. As
# always, a ref that does not exist yet (the first attempt)
# is a cache miss, not an error.
# build still never writes to the registry. CI sets it only
# on a dedicated serialized re-build step, never on the
# parallel build itself: a mode=max export is a heavy
# registry write, and ten shards exporting at build
# concurrency are ~30 simultaneous writers -- the same
# arithmetic that forced `dave push` down to -j 1.
# CACHE_FROM_SCOPE build: additionally IMPORT the scoped ref
# (`...-<scope>`). Set alongside exports of that scope so a
# re-run attempt of a same-repo PR hits the cache its own
# earlier attempt wrote -- without it the retry would
# rebuild, land on a different image ID, and miss the
# integration-test skip stamps keyed on that ID. A ref that
# does not exist yet (the first attempt) is a cache miss,
# not an error.
# TAG_SUFFIX push: suffix for the published tag, so per-arch pushes
# land on `<tag>-amd64` / `<tag>-arm64` for
# bin/merge-manifests to join into one multi-arch tag.
Expand All @@ -65,7 +70,7 @@ templates:
{{^retagFrom}}docker buildx build --load ${PLATFORM:+--platform "$PLATFORM"}
--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}
${CACHE_ARCH:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}}
${CACHE_TO_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE}}
${CACHE_FROM_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_FROM_SCOPE}}
${CACHE_TO_SCOPE:+--cache-to type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE},mode=max,image-manifest=true}
--build-arg DATASET={{dataset}}
--build-arg DATASET_CHECKSUM="$(bin/dataset-checksum '{{{extractUrl}}}')"
Expand Down Expand Up @@ -194,8 +199,15 @@ contexts:
configs:
- test/config/openflights.yaml
pagila:
extractUrl: https://github.com/devrimgunduz/pagila.git
sqlFiles: pagila/pagila-schema.sql pagila/pagila-data.sql
# A pinned tarball, no longer a clone of upstream git HEAD: pagila
# added a pgvector example (CREATE EXTENSION vector) in 2026-07, which
# postgres:18-alpine cannot satisfy -- init aborts at that statement
# and the container never becomes ready. This commit is the pre-2026-07
# HEAD the recorded expected counts were measured against. Bump it
# deliberately: alongside pgvector support in the image and re-recorded
# expectations, not by tracking HEAD.
extractUrl: https://github.com/devrimgunduz/pagila/archive/e0e35a666f31a786b9d3c06cb83799fc46db25d4.tar.gz
sqlFiles: pagila-e0e35a666f31a786b9d3c06cb83799fc46db25d4/pagila-schema.sql pagila-e0e35a666f31a786b9d3c06cb83799fc46db25d4/pagila-data.sql
structureTest:
configs:
- test/config/pagila.yaml
Expand Down Expand Up @@ -348,7 +360,7 @@ contexts:
{{^retagFrom}}docker buildx build --load ${PLATFORM:+--platform "$PLATFORM"}
--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}
${CACHE_ARCH:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}}
${CACHE_TO_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE}}
${CACHE_FROM_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_FROM_SCOPE}}
${CACHE_TO_SCOPE:+--cache-to type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE},mode=max,image-manifest=true}
--build-arg DATASET={{dataset}}
--build-arg DATASET_CHECKSUM="$(bin/dataset-checksum '{{{extractUrl}}}')"
Expand Down Expand Up @@ -611,7 +623,7 @@ contexts:
{{^retagFrom}}docker buildx build --load ${PLATFORM:+--platform "$PLATFORM"}
--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}
${CACHE_ARCH:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}}
${CACHE_TO_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE}}
${CACHE_FROM_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_FROM_SCOPE}}
${CACHE_TO_SCOPE:+--cache-to type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE},mode=max,image-manifest=true}
--build-arg DATASET={{dataset}}
--build-arg DATASET_CHECKSUM="$(bin/dataset-checksum '{{{extractUrl}}}')"
Expand Down Expand Up @@ -853,7 +865,7 @@ contexts:
{{^retagFrom}}docker buildx build --load ${PLATFORM:+--platform "$PLATFORM"}
--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}
${CACHE_ARCH:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}}
${CACHE_TO_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE}}
${CACHE_FROM_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_FROM_SCOPE}}
${CACHE_TO_SCOPE:+--cache-to type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE},mode=max,image-manifest=true}
--build-arg DATASET={{dataset}}
--build-arg DATASET_CHECKSUM="$(bin/dataset-checksum '{{{extractUrl}}}')"
Expand Down Expand Up @@ -1123,7 +1135,7 @@ contexts:
{{^retagFrom}}docker buildx build --load ${PLATFORM:+--platform "$PLATFORM"}
--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}
${CACHE_ARCH:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}}
${CACHE_TO_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE}}
${CACHE_FROM_SCOPE:+--cache-from type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_FROM_SCOPE}}
${CACHE_TO_SCOPE:+--cache-to type=registry,ref={{{repository}}}:buildcache-{{tag}}${CACHE_ARCH:-}-${CACHE_TO_SCOPE},mode=max,image-manifest=true}
--build-arg DATASET={{dataset}}
--build-arg DATASET_CHECKSUM="$(bin/dataset-checksum '{{{extractUrl}}}')"
Expand Down
Loading