ansible: cap Docker BuildKit cache at 3GB after every build#88
Open
magent-cryptograss wants to merge 1 commit into
Open
ansible: cap Docker BuildKit cache at 3GB after every build#88magent-cryptograss wants to merge 1 commit into
magent-cryptograss wants to merge 1 commit into
Conversation
delivery-kid's BuildKit cache grew to 28GB across 337 entries (2026-05-30) and took the host to 100% disk, killing the next deploy at the "Install required packages" task with an unhelpful "Failed to update apt cache: unknown reason". Root cause: every --rebuild deploy runs `docker compose build --no-cache`, and --no-cache bypasses cache *use* but still adds new BuildKit cache entries. With no prune step, those entries accumulate forever. The same pattern existed in hunter's base-image rebuild and pickipedia-vps's wiki build. Add a `docker builder prune -f --keep-storage 3GB` task right after each build site: - delivery-kid playbook: after Start services - hunter build-image role: after Verify image was built - pickipedia-vps docker-wiki tasks: after Build wiki Docker image - pickipedia-vps docker-wiki handlers (Rebuild wiki): chained inline -f (no -a) prunes dangling cache only, so active layers stay reusable for the next deploy. --keep-storage caps total cache at 3GB regardless of dangling status — a hard ceiling that can't be defeated by however many builds run between deploys. Idempotent: when cache is already under 3GB, the task is essentially a no-op (returns quickly with nothing to prune).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
delivery-kid's BuildKit cache grew to 28 GB across 337 entries on 2026-05-30 and took the host to 100% disk, killing the next deploy at the Install required packages task with the unhelpful `Failed to update apt cache: unknown reason`.
Root cause: every `--rebuild` deploy runs `docker compose build --no-cache`, and `--no-cache` bypasses cache use but still creates new BuildKit cache entries. With no prune step in the playbook, those entries accumulate forever. The same pattern existed in hunter's base-image rebuild and pickipedia-vps's wiki build — both would have hit this eventually.
What
A new `docker builder prune -f --keep-storage 3GB` task right after every build site:
Flag choices
Test plan
Followup (out of scope here)
The dedup check in `pickipedia_client.py` is dead code due to `snapshot_at` always changing — flagged in #87 review. Worth a separate small PR.