fix(runner): cap the journal and bound the go-build cache by size

[Benehiko]

What this is not

Worth stating up front, because it was my own starting assumption and it was wrong: the containerd/nerdctl/BuildKit prune already works. Measured on the live runner —

• an hourly GC run reclaimed images and took the disk to 12G used / 20%
• BuildKit sits at exactly 2.0G, the floor its --keep-storage 2048 ceiling defines (watched it reclaim 439MB down to precisely 2.00GB)

Steady-state after GC is roughly 10.4G on a 58G disk. No changes to that path.

The journal had no cap

journald.conf was empty, so systemd applied its default ceiling of 10% of the filesystem. That is a proportional limit that silently grows with the disk: resizing this runner from 20G to 60G raised the journal's allowance from ~1.9G to ~5.8G without anyone choosing it. It was sitting at 463M.

A journald drop-in now pins SystemMaxUse=200M with a one-week retention, and the GC vacuums as a backstop against bursts between rotations. Restarting journald in runcmd is required — without it the daemon keeps running with the compiled-in default. Applying the drop-in on the live runner dropped the journal 463.6M → 193.0M immediately.

The vacuum needed sudo

The GC service runs as User=runner, which cannot unlink files under /var/log/journal. An unprivileged journalctl --vacuum-size reports "Permission denied" per file, frees 0B, and still exits 0 — a silent no-op that would have looked like a working fix. A sudoers drop-in grants exactly that one command, at 0440 since sudo ignores group- or world-writable sudoers files.

The go-build cache was all-or-nothing

It was skipped entirely while a job ran and otherwise wiped with go clean -cache, so it grew unchecked on a busy runner and went fully cold on an idle one. It now evicts least-recently-used entries down to a 1536MB ceiling, keeping the cache warm below the limit.

This also removes the need for the Runner.Worker check: evicting a cold entry costs a recompile, never a failure, so there is no reason to skip the trim during a job.

The age floor has to be in minutes

This is the part worth reviewing. A day-granularity guard (-mtime +1) cannot work here — a busy runner rewrites its whole cache within a day, so nothing is ever a day old. Observed live before the fix:

total files:                    16584
older than 1 day (eligible):        0     <- ceiling unenforceable

Hourly mtime spread on the same cache showed the real working set:

older than  60min: 11280 files
older than 120min:  3643 files
older than 480min:     0 files

At a 120-minute floor the same cache went 2675MB → 1950MB, evicting 3643 entries in ~6s, stopping only because it exhausted entries older than the floor. 120 minutes comfortably exceeds the longest CI job.

Performance detail

Eviction subtracts each file's own size (find -printf '%T@ %k %p') rather than re-running du after every unlink. On a cache of tens of thousands of files, a du per deletion would mean thousands of full-tree stat walks.

Testing

go build ./..., make lint (0 issues) and the full go test ./... suite pass. The embedded script passes shellcheck and bash -n.

Verified against the live runner, plus standalone harnesses for:

• the eviction loop — 71MB → exactly the 20MB ceiling, oldest-first, all fresh files surviving
• an all-fresh over-ceiling cache — evicts nothing, exits immediately, does not hang
• a missing cache directory — skips cleanly

Tests assert the size ceiling, the minute-granularity age floor, the absence of a wholesale clear, the sudo vacuum, the journald cap, and the journald restart. Three mutations were checked and each fails the suite: dropping the sudo from the vacuum, reverting to -mtime +1, and removing the journald restart.

One existing assertion changed. TestGitHubRunnerCloudInit required the Runner.Worker check to survive specifically to gate the go-build clear; that guard is what this replaces, so the assertion now covers the stronger invariant (size ceiling + age floor + no wholesale clear) instead.

Note for existing runners

Templates apply at create time, so an existing runner VM does not pick this up automatically. The three files were installed on nyx-ci by hand to confirm the behaviour; other runners need a recreate or the same manual install.