ci: reclaim docker disk and require headroom before a rig run - #512
Merged
Conversation
A full root filesystem does not fail a job on the rig box, it takes the runner offline: the systemd unit stays active while the agent can no longer write, so GitHub marks every instance offline at once and the queue stalls with nothing red to look at. That is what happened on 2026-08-28, and the nightlies had been dying the same way since the 25th before anyone noticed. Nothing reclaimed docker's churn. Containers were already handled twice (the reset step at the top of this job, and each suite's always-run destroy), and the deb cache and the CI log store both prune themselves. Images and build cache did not: every run rebuilds the osvbng, Kea and xl2tpd images, leaving the previous ones dangling with their layers, and BuildKit's cache grows without bound. On a development box carrying the same workload that is 7 GB of dangling images and 10 GB of build cache. So prune both once per run, before the images are rebuilt, and then refuse to start a sweep with less than 10 GB free rather than discovering it halfway through. The floor is a judgement call: a run needs room for an image rebuild plus every lab a parallel suite brings up. Volumes are deliberately not pruned. osvbng-vpp-work-debian12 and its ccache hold the VPP build tree and are mounted only while a build runs, so between builds they look unused and docker volume prune would take them, costing a clean rebuild and the whole ccache. Images are pruned dangling-only for the same reason: prune -a would drop osvbng:local, the builder image, frr, bngblaster and bngtester.
The first cut refused to start a sweep below 10G. The box does not sit at 10G: its own first run of this workflow reclaimed 2G to 5G and then blocked itself. A guard that stops work at the comfortable number is its own kind of outage, and it would have red-flagged every PR until somebody cleared space by hand. Split it. The hard floor is now just above what a run actually consumes, an image rebuild plus the labs a parallel suite brings up, because the failure being guarded against is the box reaching zero mid-sweep, which does not fail a job: it drops all five runners offline while their units stay active, so nothing goes red. The 10G case becomes a warning, loud in the log and in the step summary, so the trend is visible while there is still time to act. Both tiers print docker system df, and the hard failure also prints df, so the job log carries the diagnosis rather than pointing at a box the reader may not be able to reach.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A full root filesystem does not fail a job on the rig box, it takes the runner offline. The systemd units stay
active (running)while the agent can no longer write, so GitHub marks all five instances offline at once and the queue stalls with nothing red to look at.That is what happened on 2026-08-28:
/hit 100% with 0 bytes free, and the nightlies had been dying the same way since the 25th (cancelled on the 25th, 26th and 27th, the 28th's stuck queued for 11 hours). The last successfuldataplane-artifactsbuild before the box was cleared was 2026-08-17, so the rig had been installing a twelve-day-old dataplane without anyone seeing a failure.What was actually leaking
Not containers. Those are already cleaned twice: the
Reset host statestep at the top of this job removes everyclab-container box-wide, and each suite has analways()destroy.docker container pruneon the box reclaimed 0 B, which confirms it.Images and build cache. Every run rebuilds the osvbng, Kea and xl2tpd images, so the previous ones are left dangling with their layers, and BuildKit's cache grows without bound. Nothing prunes either. On a development box carrying the same workload that is 7.3 GB of dangling images and 10.1 GB of build cache — about 20 GB of pure churn on a 48 GB disk.
The deb cache (
keep 3) and the CI log store (mtime +7) already prune themselves and were not the problem.Change
One reclaim step per rig run, before the images are rebuilt, and a preflight that refuses to start a sweep below 10 GB free rather than discovering it halfway through. The floor is a judgement call: a run needs room for an image rebuild plus every lab a parallel suite brings up. Raise it if the image set grows.
Deliberately not pruned, both of which would be actively harmful here:
osvbng-vpp-work-debian12and its ccache hold the VPP build tree and are mounted only while a build runs. Between builds they look unused, sodocker volume prune(orsystem prune --volumes) would delete them, costing a clean rebuild and the entire ccache.image prune -awould droposvbng:local, the builder image, frr, bngblaster and bngtester, all of which then have to be pulled or rebuilt.Verification
suite-run.ymlparses and the step order is correct (Reset host state->Reclaim docker disk->Require free disk->Fetch dataplane artifacts). Both step scripts passbash -n, and thedf --output=availparsing was checked against a real filesystem.The reclaim itself is exercised by this PR's own rig run, which cannot start unless the preflight passes. I have not simulated the failure branch on the box; the error path is a plain
exit 1with the message naming what to look at next.