fix(runner): harden disk GC to hourly + age-gated orphan reap#47
Merged
Conversation
The github-runner template's disk GC skipped its entire run whenever a job was in progress (pgrep Runner.Worker -> exit 0) and fired only daily. On a busy runner the daily trigger kept landing mid-job and skipping, so GC effectively never ran for days while canceled/crashed CI jobs leaked still-"Up" compose stacks that nerdctl's own `system prune` never reaps. The runner disk filled to 100%, actions-runner could no longer copy run-helper.sh, and the service crash-looped -> GitHub marked it offline. Replace the whole-run skip with per-action age-gating: force-remove any container older than a 90-minute ceiling even if still "Up". A real job's containers are always younger than the ceiling, so the live job is never touched and no name allow-list is needed (works for any repo's runner). The Runner.Worker check survives only to keep the go-build cache warm during a job. Timer cadence daily -> hourly so orphans can't accumulate for up to 24h between runs. Applies to every runner VM created or reinstalled from the template; no new flag (the GC was already unconditional).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
github-runnertemplate's disk GC (vee-runner-gc.sh+.timer) had two flaws that combined to knock a runner offline:pgrep Runner.Worker >/dev/null && exit 0. On a busy runner the timer kept firing while a job ran, GC kept skipping, and it effectively never executed for days.Meanwhile canceled/crashed CI jobs leak compose stacks that stay
Up, andnerdctl system pruneonly reaps stopped containers — so those leaked stacks (plus their volumes) were never collected. The runner disk filled to 100%,actions-runnercould no longer copyrun-helper.shon start, and the service crash-looped (observed restart counter >10,000) → GitHub marked the runner offline.Fix
Up(via.State.StartedAt). A real job's containers are always younger than the ceiling, so the in-flight job is never touched and no name allow-list is needed — this works for any repo's runner, not just one project's*-e2e-*stacks.Runner.Workercheck survives, but only to keep the go-build cache warm during a job (clearing it mid-build is a pointless cache-cold penalty, not a correctness issue — the age-gated reap is what protects the live job).Applies to every runner VM created or
--reinstalled from the template. No new flag — the GC was already unconditional; this just makes the default safe.Validation
go build ./...,go test -race ./internal/templates/— green (added asserts: no whole-run skip,ORPHAN_MAX_AGE_SECpresent,nerdctl rm -fpresent, hourly timer, not daily).make lint— 0 issues.shellcheck --shell=bashon the extracted embedded script — exit 0, no findings.The already-offline runner was recovered by hand-applying this same script; the template now matches what's running on it.
🤖 Generated with Claude Code