Skip to content

fix(runner): harden disk GC to hourly + age-gated orphan reap#47

Merged
Benehiko merged 1 commit into
mainfrom
fix/runner-gc-hourly-age-reap
Jul 22, 2026
Merged

fix(runner): harden disk GC to hourly + age-gated orphan reap#47
Benehiko merged 1 commit into
mainfrom
fix/runner-gc-hourly-age-reap

Conversation

@Benehiko

Copy link
Copy Markdown
Owner

Problem

The github-runner template's disk GC (vee-runner-gc.sh + .timer) had two flaws that combined to knock a runner offline:

  1. Whole-run skip on active job — the script did pgrep Runner.Worker >/dev/null && exit 0. On a busy runner the timer kept firing while a job ran, GC kept skipping, and it effectively never executed for days.
  2. Daily cadence — even without the skip, orphans could accumulate for up to 24h between runs.

Meanwhile canceled/crashed CI jobs leak compose stacks that stay Up, and nerdctl system prune only reaps stopped containers — so those leaked stacks (plus their volumes) were never collected. The runner disk filled to 100%, actions-runner could no longer copy run-helper.sh on start, and the service crash-looped (observed restart counter >10,000) → GitHub marked the runner offline.

Fix

  • Replace the whole-run skip with per-action age-gating. Force-remove any container older than a 90-minute ceiling even if still Up (via .State.StartedAt). A real job's containers are always younger than the ceiling, so the in-flight job is never touched and no name allow-list is needed — this works for any repo's runner, not just one project's *-e2e-* stacks.
  • The Runner.Worker check survives, but only to keep the go-build cache warm during a job (clearing it mid-build is a pointless cache-cold penalty, not a correctness issue — the age-gated reap is what protects the live job).
  • Timer cadence daily → hourly so orphans can't pile up between runs.

Applies to every runner VM created or --reinstalled from the template. No new flag — the GC was already unconditional; this just makes the default safe.

Validation

  • go build ./..., go test -race ./internal/templates/ — green (added asserts: no whole-run skip, ORPHAN_MAX_AGE_SEC present, nerdctl rm -f present, hourly timer, not daily).
  • make lint — 0 issues.
  • shellcheck --shell=bash on the extracted embedded script — exit 0, no findings.

The already-offline runner was recovered by hand-applying this same script; the template now matches what's running on it.

🤖 Generated with Claude Code

The github-runner template's disk GC skipped its entire run whenever a
job was in progress (pgrep Runner.Worker -> exit 0) and fired only daily.
On a busy runner the daily trigger kept landing mid-job and skipping, so
GC effectively never ran for days while canceled/crashed CI jobs leaked
still-"Up" compose stacks that nerdctl's own `system prune` never reaps.
The runner disk filled to 100%, actions-runner could no longer copy
run-helper.sh, and the service crash-looped -> GitHub marked it offline.

Replace the whole-run skip with per-action age-gating: force-remove any
container older than a 90-minute ceiling even if still "Up". A real job's
containers are always younger than the ceiling, so the live job is never
touched and no name allow-list is needed (works for any repo's runner).
The Runner.Worker check survives only to keep the go-build cache warm
during a job. Timer cadence daily -> hourly so orphans can't accumulate
for up to 24h between runs.

Applies to every runner VM created or reinstalled from the template; no
new flag (the GC was already unconditional).
@Benehiko
Benehiko merged commit 774a0a8 into main Jul 22, 2026
4 checks passed
@Benehiko
Benehiko deleted the fix/runner-gc-hourly-age-reap branch July 22, 2026 19:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant