Retry dave build on transient registry failures - #8
Merged
Conversation
The first master run of the retry/skip redesign failed in dave build: with cold buildcache refs every shard really pulls its base images, and 10 shards at -j 3 resolving manifests near-simultaneously tripped Docker Hub's 429 rate limit -- the one pipeline stage the redesign left without a retry. Same bounded-retry shape as the push step; a second dave build finds every already-built layer in the local store, so a retry redoes only the tags the 429 killed. Also mkdir the two persisted cache dirs up front so the always() cache saves stop warning about a missing path when a shard dies early. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aa8y
enabled auto-merge (squash)
August 30, 2026 18:02
The build retry alone did not survive the second run: the 429s are not a short burst but an exhausted shared hourly pull quota, which minutes of backoff cannot restore. Each context only has two or three distinct FROM images, so pull them exactly once per shard (with retry) and let every build resolve FROM from the local image store instead of hammering Docker Hub ~21 times per shard. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rerun surfaced the real PR-run killer: with CACHE_TO_SCOPE set for the whole job, every build exports a mode=max cache at -j 3 across 10 shards -- ~30 concurrent registry writers, the same arithmetic that forced dave push to -j 1. The writes 429 and take the builds down with them. Split the knob: CACHE_FROM_SCOPE (job-wide on same-repo PRs) keeps the scoped-ref import so a retry still rebuilds to identical image IDs, while CACHE_TO_SCOPE now only exists on a dedicated post-build step that re-runs dave build at -j 1 -- a full local cache hit, so upload-only -- with continue-on-error, because a throttled cache export must never fail a shard whose images built and will be tested regardless. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Upstream pagila added a pgvector example (CREATE EXTENSION vector) in July 2026; postgres:18-alpine cannot satisfy it, so init aborts and the container never becomes ready -- a deterministic readiness timeout on both arches whenever the git HEAD checksum busts the cache. Switch the extract from a HEAD clone to the pinned pre-2026-07 commit tarball the recorded expected counts were measured against. Upgrading to current pagila is its own change: pgvector in the image plus re-recorded expectations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The first master run of #7's redesign failed in
dave buildwith Docker Hub429 Too Many Requestswhile resolving base-image manifests — the one pipeline stage the redesign left without a retry (tests and push got one). It was aggravated by coldbuildcache-*refs from the previously failed master run: nothing to import means every base image is really pulled, and 10 shards at-j 3resolving manifests near-simultaneously trip the rate limit.dave buildin the same bounded retry as the push step (3 attempts, 60s/120s backoff). A seconddave buildfinds every already-built layer in the local store, so a retry redoes only the tags the 429 actually killed.mkdir -pthe two persisted cache dirs up front so thealways()cache saves stop warning about a missing path when a shard dies before first writing them.Verification
🤖 Generated with Claude Code