Make CI retries cheap and integration tests less flaky - #7
Merged
Conversation
Two goals: a re-run of a failed shard should skip everything that already passed for the same commit, and the failures that force re-runs in the first place should happen less often. Skip-on-retry: persist the identical-image dedupe stamp dir across re-run attempts of a commit via actions/cache. The stamp key (image ID + exact expected-file bytes) already guarantees correctness, so persistence only adds the skip. The build template now also imports the PR-scoped buildcache ref it exports, so a retried PR shard rebuilds to identical image IDs and actually hits its stamps. Flakiness: run*.sh now exit 3 for deterministic assertion failures and other codes for infra trouble; the new with-retry.sh wrapper (wired into every manifest test template) retries only the latter, once. bin/dataset-checksum gains a persisted last-known-good fallback so a transient upstream-metadata blip no longer degrades the fingerprint to unknown and busts the EXTRACT cache. dave push retries up to 3 times with backoff on registry 429s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two goals: a re-run of a failed shard skips everything that already passed for the same commit, and the failures that force re-runs happen less often.
Skip-on-retry (same SHA)
DDS_TEST_CACHE) across re-run attempts viaactions/cache, keyeditest-stamps-<context>-<arch>-<sha>-<attempt>with a SHA-scoped restore prefix, savedif: always()so a partially failed shard still banks its passing tags. Correctness is inherent to the stamp key (image ID + exact expected-file bytes).buildcache-<tag>-<arch>-pr), so a retried PR shard rebuilds to identical image IDs and actually hits its stamps.Flakiness reduction
run*.shexit 3 for deterministic assertion failures (mismatched counts, missing expected file, corrupt sqlite db) and other codes for infra trouble (readiness timeout, docker/network errors). Newtest/integration/with-retry.sh— wired into every manifesttest:template — retries only the infra class, once (DDS_ITEST_RETRIES/DDS_ITEST_RETRY_DELAY).bin/dataset-checksumgains an opt-inDATASET_CHECKSUM_FALLBACKlast-known-good store (persisted viaactions/cache): a transient upstream-metadata blip no longer degrades the fingerprint tounknown, which used to bust the EXTRACT layer cache and force a full re-download from the very upstream that was flaking. Only clean fetches refresh the store.dave pushretries up to 3 times with 60s/120s backoff on registry 429s; retries are upload-only (full local cache hit).Verification
bash -n+ shellcheck clean on new/changed scripts; actionlint + YAML validation clean on the workflow; manifest parses with all ten templates rendering as intended.Docs:
docs/testing.mddocuments the exit-code contract, wrapper knobs, stamp persistence, and checksum fallback.🤖 Generated with Claude Code