Skip to content

ci: reclaim docker disk and require headroom before a rig run - #512

Merged
BSpendlove merged 2 commits into
mainfrom
ci/reclaim-disk-before-rig-runs
Aug 29, 2026
Merged

ci: reclaim docker disk and require headroom before a rig run#512
BSpendlove merged 2 commits into
mainfrom
ci/reclaim-disk-before-rig-runs

Conversation

@BSpendlove

Copy link
Copy Markdown
Member

Problem

A full root filesystem does not fail a job on the rig box, it takes the runner offline. The systemd units stay active (running) while the agent can no longer write, so GitHub marks all five instances offline at once and the queue stalls with nothing red to look at.

That is what happened on 2026-08-28: / hit 100% with 0 bytes free, and the nightlies had been dying the same way since the 25th (cancelled on the 25th, 26th and 27th, the 28th's stuck queued for 11 hours). The last successful dataplane-artifacts build before the box was cleared was 2026-08-17, so the rig had been installing a twelve-day-old dataplane without anyone seeing a failure.

What was actually leaking

Not containers. Those are already cleaned twice: the Reset host state step at the top of this job removes every clab- container box-wide, and each suite has an always() destroy. docker container prune on the box reclaimed 0 B, which confirms it.

Images and build cache. Every run rebuilds the osvbng, Kea and xl2tpd images, so the previous ones are left dangling with their layers, and BuildKit's cache grows without bound. Nothing prunes either. On a development box carrying the same workload that is 7.3 GB of dangling images and 10.1 GB of build cache — about 20 GB of pure churn on a 48 GB disk.

The deb cache (keep 3) and the CI log store (mtime +7) already prune themselves and were not the problem.

Change

One reclaim step per rig run, before the images are rebuilt, and a preflight that refuses to start a sweep below 10 GB free rather than discovering it halfway through. The floor is a judgement call: a run needs room for an image rebuild plus every lab a parallel suite brings up. Raise it if the image set grows.

Deliberately not pruned, both of which would be actively harmful here:

  • Volumes. osvbng-vpp-work-debian12 and its ccache hold the VPP build tree and are mounted only while a build runs. Between builds they look unused, so docker volume prune (or system prune --volumes) would delete them, costing a clean rebuild and the entire ccache.
  • Non-dangling images. image prune -a would drop osvbng:local, the builder image, frr, bngblaster and bngtester, all of which then have to be pulled or rebuilt.

Verification

suite-run.yml parses and the step order is correct (Reset host state -> Reclaim docker disk -> Require free disk -> Fetch dataplane artifacts). Both step scripts pass bash -n, and the df --output=avail parsing was checked against a real filesystem.

The reclaim itself is exercised by this PR's own rig run, which cannot start unless the preflight passes. I have not simulated the failure branch on the box; the error path is a plain exit 1 with the message naming what to look at next.

A full root filesystem does not fail a job on the rig box, it takes
the runner offline: the systemd unit stays active while the agent can
no longer write, so GitHub marks every instance offline at once and
the queue stalls with nothing red to look at. That is what happened on
2026-08-28, and the nightlies had been dying the same way since the
25th before anyone noticed.

Nothing reclaimed docker's churn. Containers were already handled
twice (the reset step at the top of this job, and each suite's
always-run destroy), and the deb cache and the CI log store both prune
themselves. Images and build cache did not: every run rebuilds the
osvbng, Kea and xl2tpd images, leaving the previous ones dangling with
their layers, and BuildKit's cache grows without bound. On a
development box carrying the same workload that is 7 GB of dangling
images and 10 GB of build cache.

So prune both once per run, before the images are rebuilt, and then
refuse to start a sweep with less than 10 GB free rather than
discovering it halfway through. The floor is a judgement call: a run
needs room for an image rebuild plus every lab a parallel suite brings
up.

Volumes are deliberately not pruned. osvbng-vpp-work-debian12 and its
ccache hold the VPP build tree and are mounted only while a build
runs, so between builds they look unused and docker volume prune would
take them, costing a clean rebuild and the whole ccache. Images are
pruned dangling-only for the same reason: prune -a would drop
osvbng:local, the builder image, frr, bngblaster and bngtester.
The first cut refused to start a sweep below 10G. The box does not sit
at 10G: its own first run of this workflow reclaimed 2G to 5G and then
blocked itself. A guard that stops work at the comfortable number is
its own kind of outage, and it would have red-flagged every PR until
somebody cleared space by hand.

Split it. The hard floor is now just above what a run actually
consumes, an image rebuild plus the labs a parallel suite brings up,
because the failure being guarded against is the box reaching zero
mid-sweep, which does not fail a job: it drops all five runners
offline while their units stay active, so nothing goes red. The 10G
case becomes a warning, loud in the log and in the step summary, so
the trend is visible while there is still time to act.

Both tiers print docker system df, and the hard failure also prints
df, so the job log carries the diagnosis rather than pointing at a box
the reader may not be able to reach.
@BSpendlove
BSpendlove deployed to integration-rig August 29, 2026 19:56 — with GitHub Actions Active
@BSpendlove
BSpendlove merged commit bc3e15b into main Aug 29, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant