Killed test runs leak their GOTMPDIR work dirs: 27 abandoned /var/tmp/gotmp/go-build* trees held 4.5G and ENOSPC-failed a push gate that passed 10/10 after reclaim (gc-68bao) by zook-bot · Pull Request #176 · zookanalytics/gascity

zook-bot · 2026-09-04T15:44:10Z

Summary

What changed

go removes its per-invocation compile/link work dir under $GOTMPDIR only on
a clean exit. A run the gate watchdog kills (SIGQUIT->SIGKILL), a signalled run,
or an OOM kill leaks the work-dir tree, and nothing sweeps it. One measured
hoard was 27 trees holding 4.5G that ENOSPC-failed a push gate which then passed
10/10 once they were reclaimed; the linker's go-link-* trees are the largest
single hoard. The trees appear under $GOTMPDIR, and directly under /var/tmp
when a run set no GOTMPDIR.

The change

scripts/lib/harness-reap.sh — new gc_harness_sweep_stale_build_dirs.
The gate's existing process sweep already reaps stranded test processes but
never their leaked work dirs; this reclaims them right after, on the same age
floor, so no in-flight build is ever in scope.
scripts/test-local-parallel — calls the new sweep after the process
sweep (the one funnel all four heavy gate targets pass through), over both
$GOTMPDIR and /var/tmp, so the reclaim runs on every gate with no new
order or command.
Safety bound. Reclaim is limited to go's exact generated work-dir shapes.
go names them with os.MkdirTemp(dir, "go-build") and
os.MkdirTemp(dir, "go-link-"), which append a decimal suffix, so the only
go-owned shapes are go-build<digits> and go-link-<digits>; the sweep
matches exactly those. The bare go-build*/go-link* prefix is deliberately
NOT used — it would also select human-named scratch such as go-build-base,
which this rm -rf path would then destroy. A go-owned work dir is never a
checkout, a cache-with-value, or a comparison base a human wants kept, and is
regenerated on the next build. A tree is removed only when it is owned by the
invoking user, older than the go test budget, and referenced by no live
process (the environ+cmdline+cwd /proc predicate). Every delete routes
through one gc_harness_reclaim_dir seam, so the destroy authority is
reviewable in isolation.
Tests. scripts/harness_reap_test.go's
TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected reclaims an
abandoned go-build<n>/go-link-<n> tree at either root and spares a
live-held tree, a too-young tree, a .git-bearing tree, a non-go-prefixed
cache dir, and — pinning the safety bound — go-prefixed but non-generated
shapes (go-build-base, go-link-base, go-build12ab) the bare-prefix glob
would have destroyed. Its one bash subprocess raises the resourcecensus
subprocess baseline by 1 across the three mirrors —
internal/testpolicy/resourcecensus/census.go, test/test-resources.toml,
and the regenerated TESTING.md.

Scope / follow-up

The larger /var/tmp agent-cache half (gocache-* and cold-build throwaways,
12.6G in one incident) is a policy call with an unrecoverable blast radius — a
naive age+liveness scan matched 662 owned dirs of mixed provenance, including
comparison bases — so it is filed separately as gc-ulys6, not shipped here.
The liveness check's grep -F is an unanchored substring match, so a leaked
dir whose path is a substring of a live one is spuriously spared (fails safe:
over-spares, never over-deletes). Filed as gc-3e7j3, out of scope here.

Dispatch — what this work was asked to do

What

go removes its per-invocation work dir under \$GOTMPDIR only on a clean
exit. When a test run is killed the dir survives, and nothing sweeps it.
Measured 2026-09-02 in /var/tmp/gotmp: 29 go-build* trees, of which 27
were referenced by no live process and 4.5G in total. The two survivors were
the live runs.

The killer is visible in the gate log itself. scripts/test-local-parallel
opens with its own reaper:

harness sweep: killing socket-parked pid 3408324 (age 2982s, run gct-3377763-4134694241 is gone)
... 4 more ...
harness sweep: reaped 4 stale test process(es)

Those kills free the process table and leave the work dir. Ages in the sweep
line (~2900-3000s) match the mtimes of the abandoned trees.

Why it is P1

Same symptom and same disk as gc-4guc2, different hoard. With / at 100%
(142M free) every unit-cmd-gc-* shard of my pre-push gate died before
running a test:

/usr/local/go/pkg/tool/linux_amd64/link: mapping output file failed: no space left on device
FAIL	github.com/gastownhall/gascity/cmd/gc [build failed]

and unit-core went red on shell-script tests writing to disk
(printf: I/O error, error: unable to create temporary file: No space left on device) rather than on any assertion. Zero EDQUOT in the log, so this is
host ENOSPC, not the per-agent quota shape in gc-6jye9.

That gate reads as a FAIL on the diff and is not one. Proof on an identical
tree: removing the 27 unreferenced work dirs took / from 142M to 4.4G free,
and the same commit re-ran 10/10 green with zero ENOSPC lines.

Reclaim predicate that proved safe

Union of /proc/*/environ, /proc/*/cmdline and /proc/*/cwd gives the set
of work dirs any live process still references. A dir absent from that set and
older than 30 minutes is garbage. That rule removed 27 trees during two
concurrent live gates without disturbing either.

{ for p in /proc/[0-9]*; do
    tr "\\0" "\\n" < "$p/environ"; tr "\\0" "\\n" < "$p/cmdline"; readlink "$p/cwd"
  done; } 2>/dev/null | grep -o "go-\(build\|link\)[0-9][0-9]*" | sort -u

Where the fix belongs

The harness sweep already knows the run id it is killing and kills the
process. Removing that run's work dir in the same place is the smallest fix.
A standalone sweep over \$GOTMPDIR using the liveness predicate above is the
fallback if the sweep cannot attribute a dir to a run.

Evidence

Observed while landing the rework on gc-oaq1l / anchor gc-ndns7, branch
polecat/gc-ndns7. First gate attempt: 3 ok, 7 red, all disk. After reclaim:
10/10 ok, push accepted at b4aaa39.

Refinery handoff

Issue: gc-68bao
Source branch: polecat/gc-68bao
Target: main
Gates codex signed off pre-open at ba8c006f; PR opened green.

…ss sweep (gc-68bao) go removes its per-invocation compile/link work dir under $GOTMPDIR only on a clean exit. A run the harness watchdog kills (SIGQUIT->SIGKILL), a signalled run, or an OOM kill leaks the tree, and nothing sweeps it. gc-68bao measured 27 such trees holding 4.5G that ENOSPC-failed a push gate which then passed 10/10 once they were reclaimed; the linker's go-link-* trees are the largest single hoard. The trees appear both under $GOTMPDIR and, when a run set no GOTMPDIR, directly under /var/tmp. The process sweep in scripts/lib/harness-reap.sh already reaps stranded test processes at gate start but never their leaked directories. This adds gc_harness_sweep_stale_build_dirs there, called from test-local-parallel (the one funnel all four heavy gate targets pass through) right after the process sweep and on the same age floor, so the reclaim runs on every gate without a new order or command. The reclaim is deliberately bounded to go's own go-build*/go-link* shapes. That bound is the safety argument: those prefixes are go-owned, never a checkout, a cache-with-value, or a comparison base a human wants kept, and are regenerated on the next build, so removing one needs no per-shape judgment. A tree is reclaimed only when it is owned by the invoking user, older than the go test budget, and referenced by no live process — the environ+cmdline+cwd /proc predicate proven safe in gc-68bao (it spared every live build across two concurrent gates). Every delete routes through the single gc_harness_reclaim_dir seam, mirroring gc_harness_kill_pid, so the destroy authority is reviewable in isolation and the self-test records decisions instead of removing real trees. Scope: gc-68bao's notes also describe a larger /var/tmp half — gocache-* and cold-build throwaway caches, 12.6G in one incident. A naive age+liveness scan of that class was measured against the real host and matched 662 owned dirs of mixed provenance, including comparison bases; encoding an unconditional rm -rf of that set on every fleet gate is an unrecoverable blast radius and a policy call, so it is filed as gc-ulys6 rather than shipped here. Validation: new self-test TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected (reclaims an abandoned tree at either root; spares a live-held tree, a too-young tree, a non-go shape, and a .git-bearing dir). go vet clean; scripts harness/budget/tmpdir suites green. A print-only dry run over the real host /var/tmp selected exactly the ~10 real leaked go trees and none of the 662 other dirs. Follow-up: gc-ulys6 (the /var/tmp agent-cache half). Claude-Session: https://claude.ai/code/session_01BnanGqoxonob7hhD37Jt1X

…t (gc-68bao) The fix commit added buildDirSweepHarness to scripts/harness_reap_test.go, a fifth exec.Command("bash", ...) call that shells out to exercise gc_harness_sweep_stale_build_dirs. That file is untagged (no //go:build constraint), so the new call lands in the untagged subprocess census and trips resourcecensus's "totals cannot grow" ratchet: TestRepositoryLedgerMatchesCensusAndDocumentation went red with the scanned count one above the checked baseline on three rows. The ledger, the bootstrap policy, and the rendered doc are three mirrors that comparePolicyFields/validateBaseline require to agree exactly, so the +1 is applied to all three: test/test-resources.toml and census.go's bootstrapPolicy (baseline_calls 649->650 all-source, 432->433 untagged source, 425->426 untagged Small), then TESTING.md regenerated via the -update flag. Only baseline_calls moves; baseline_files is unchanged because the call joined a file already counted, and the historical reported_* columns are left as-is. The new subprocess is a legitimate test harness that runs a shell function, so raising the ratcheted baseline is the sanctioned path (the fixed-sleep census is the one you shrink instead). Validation: full internal/testpolicy/resourcecensus package green without -update, go vet clean, and the deliverable's own TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected still passes.

The go work-dir sweep's living comments carried a bead id and incident history — measured hoard sizes, gate pass/fail counts, a follow-up policy note. Convention is that comments state what is true now and the constraints it rests on; history belongs in the bead and the commit log. Rewrite the four comment blocks in scripts/lib/harness-reap.sh, scripts/test-local-parallel, and scripts/harness_reap_test.go to state only the reclaim's behavior and safety contract: it reclaims stale same-UID go-build/go-link temp dirs after age and live-reference checks, scans each caller-named root at depth 1, and leaves broader cache-root cleanup out of scope. No behavior change. Addresses pre-open signoff finding (round 1) on polecat/gc-68bao. Claude-Session: https://claude.ai/code/session_01S8WGc6bV45qvCiKoFdR9GE

…(gc-68bao) The sweep matched the bare go-build*/go-link* prefix, so an old, unreferenced human-named scratch dir such as go-build-base was selected and rm -rf'd. go names its work dirs with os.MkdirTemp(dir, "go-build") and os.MkdirTemp(dir, "go-link-"), which append a decimal suffix, so the only go-owned shapes are go-build<digits> and go-link-<digits>. Match exactly those. The regression test renames its spare fixtures to valid go shapes so the liveness/age/.git predicates (not shape) do the sparing, and adds prefix-but-not-generated-shape cases the old glob would have reclaimed. Claude-Session: https://claude.ai/code/session_01YADk5PdK1sqev2okiEhutK

zook-bot · 2026-09-04T15:44:16Z

Pre-open signoff (comment-only — not an approval):

VERDICT: approve
Reviewed branch: polecat/gc-68bao
Reviewed base: main
Reviewed commit: ba8c006

Scope checked: Read the full three-dot diff from origin/main to ba8c006 across TESTING.md, internal/testpolicy/resourcecensus/census.go, scripts/harness_reap_test.go, scripts/lib/harness-reap.sh, scripts/test-local-parallel, and test/test-resources.toml. Checked anchor bead gc-68bao, review bead gc-knrhw, the mol-review step text, the pack work-quality fragment, and the learning-exemplar fragment. Checked Go's local os.MkdirTemp implementation plus the Go tool and linker call sites that create go-build and go-link temp dirs. This is pre-open, so there is no PR page to review. I did not run the full project matrix. shellcheck is not installed in this environment.

Findings: none.

Filed: none.

Verification: At the reviewed commit in detached worktree /tmp/gc-review-gc-knrhw.zpWSKW, go test ./scripts ./internal/testpolicy/resourcecensus -count=1 passed: scripts 219.914s, internal/testpolicy/resourcecensus 13.881s. go test ./scripts -run TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected -count=1 -v passed. bash -n scripts/lib/harness-reap.sh scripts/test-local-parallel passed. git diff --check origin/main...ba8c006fdd32c0b3cc027de215a4f77a83bcaaff passed.

Anchor: gc-68bao — check.codex @ ba8c006

…/gotmp/go-build* trees held 4.5G and ENOSPC-failed a push gate that passed 10/10 after reclaim (gc-68bao) (#176) * fix(scripts): reclaim leaked go-build/go-link temp trees in the harness sweep (gc-68bao) go removes its per-invocation compile/link work dir under $GOTMPDIR only on a clean exit. A run the harness watchdog kills (SIGQUIT->SIGKILL), a signalled run, or an OOM kill leaks the tree, and nothing sweeps it. gc-68bao measured 27 such trees holding 4.5G that ENOSPC-failed a push gate which then passed 10/10 once they were reclaimed; the linker's go-link-* trees are the largest single hoard. The trees appear both under $GOTMPDIR and, when a run set no GOTMPDIR, directly under /var/tmp. The process sweep in scripts/lib/harness-reap.sh already reaps stranded test processes at gate start but never their leaked directories. This adds gc_harness_sweep_stale_build_dirs there, called from test-local-parallel (the one funnel all four heavy gate targets pass through) right after the process sweep and on the same age floor, so the reclaim runs on every gate without a new order or command. The reclaim is deliberately bounded to go's own go-build*/go-link* shapes. That bound is the safety argument: those prefixes are go-owned, never a checkout, a cache-with-value, or a comparison base a human wants kept, and are regenerated on the next build, so removing one needs no per-shape judgment. A tree is reclaimed only when it is owned by the invoking user, older than the go test budget, and referenced by no live process — the environ+cmdline+cwd /proc predicate proven safe in gc-68bao (it spared every live build across two concurrent gates). Every delete routes through the single gc_harness_reclaim_dir seam, mirroring gc_harness_kill_pid, so the destroy authority is reviewable in isolation and the self-test records decisions instead of removing real trees. Scope: gc-68bao's notes also describe a larger /var/tmp half — gocache-* and cold-build throwaway caches, 12.6G in one incident. A naive age+liveness scan of that class was measured against the real host and matched 662 owned dirs of mixed provenance, including comparison bases; encoding an unconditional rm -rf of that set on every fleet gate is an unrecoverable blast radius and a policy call, so it is filed as gc-ulys6 rather than shipped here. Validation: new self-test TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected (reclaims an abandoned tree at either root; spares a live-held tree, a too-young tree, a non-go shape, and a .git-bearing dir). go vet clean; scripts harness/budget/tmpdir suites green. A print-only dry run over the real host /var/tmp selected exactly the ~10 real leaked go trees and none of the 662 other dirs. Follow-up: gc-ulys6 (the /var/tmp agent-cache half). * test(census): bump subprocess baselines for the new harness sweep test (gc-68bao) The fix commit added buildDirSweepHarness to scripts/harness_reap_test.go, a fifth exec.Command("bash", ...) call that shells out to exercise gc_harness_sweep_stale_build_dirs. That file is untagged (no //go:build constraint), so the new call lands in the untagged subprocess census and trips resourcecensus's "totals cannot grow" ratchet: TestRepositoryLedgerMatchesCensusAndDocumentation went red with the scanned count one above the checked baseline on three rows. The ledger, the bootstrap policy, and the rendered doc are three mirrors that comparePolicyFields/validateBaseline require to agree exactly, so the +1 is applied to all three: test/test-resources.toml and census.go's bootstrapPolicy (baseline_calls 649->650 all-source, 432->433 untagged source, 425->426 untagged Small), then TESTING.md regenerated via the -update flag. Only baseline_calls moves; baseline_files is unchanged because the call joined a file already counted, and the historical reported_* columns are left as-is. The new subprocess is a legitimate test harness that runs a shell function, so raising the ratcheted baseline is the sanctioned path (the fixed-sleep census is the one you shrink instead). Validation: full internal/testpolicy/resourcecensus package green without -update, go vet clean, and the deliverable's own TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected still passes. * docs(scripts): state current behavior in harness-sweep comments The go work-dir sweep's living comments carried a bead id and incident history — measured hoard sizes, gate pass/fail counts, a follow-up policy note. Convention is that comments state what is true now and the constraints it rests on; history belongs in the bead and the commit log. Rewrite the four comment blocks in scripts/lib/harness-reap.sh, scripts/test-local-parallel, and scripts/harness_reap_test.go to state only the reclaim's behavior and safety contract: it reclaims stale same-UID go-build/go-link temp dirs after age and live-reference checks, scans each caller-named root at depth 1, and leaves broader cache-root cleanup out of scope. No behavior change. Addresses pre-open signoff finding (round 1) on polecat/gc-68bao. * fix(scripts): reclaim only go's exact go-build<n>/go-link-<n> shapes (gc-68bao) The sweep matched the bare go-build*/go-link* prefix, so an old, unreferenced human-named scratch dir such as go-build-base was selected and rm -rf'd. go names its work dirs with os.MkdirTemp(dir, "go-build") and os.MkdirTemp(dir, "go-link-"), which append a decimal suffix, so the only go-owned shapes are go-build<digits> and go-link-<digits>. Match exactly those. The regression test renames its spare fixtures to valid go shapes so the liveness/age/.git predicates (not shape) do the sparing, and adds prefix-but-not-generated-shape cases the old glob would have reclaimed. --------- Co-authored-by: refinery costing <refinery@local> Rebased 2026-09-15 onto upstream 83c8270: the resource-census +1 hunks (census.go, test-resources.toml, TESTING.md) were deltas on the previous sync's baselines and are re-derived over the merged tree in the census commit at the end of this rebase; the reaper, its test and the test-local-parallel sweep call merged unchanged. Claude-Session: https://claude.ai/code/session_01XfZfLDuwcBfs3TXnn1udn9

refinery costing added 4 commits September 4, 2026 13:22

github-actions Bot added the status/needs-triage label Sep 4, 2026

zook-bot merged commit fca036a into main Sep 4, 2026
76 of 81 checks passed

Navigation Menu

Killed test runs leak their GOTMPDIR work dirs: 27 abandoned /var/tmp/gotmp/go-build* trees held 4.5G and ENOSPC-failed a push gate that passed 10/10 after reclaim (gc-68bao) - #176

Killed test runs leak their GOTMPDIR work dirs: 27 abandoned /var/tmp/gotmp/go-build* trees held 4.5G and ENOSPC-failed a push gate that passed 10/10 after reclaim (gc-68bao)#176
zook-bot merged 4 commits into
mainfrom
polecat/gc-68bao

zook-bot commented Sep 4, 2026

Uh oh!

zook-bot commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zook-bot commented Sep 4, 2026

Summary

What changed

The change

Scope / follow-up

What

Why it is P1

Reclaim predicate that proved safe

Where the fix belongs

Related

Evidence

Refinery handoff

Uh oh!

zook-bot commented Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant