Killed test runs leak their GOTMPDIR work dirs: 27 abandoned /var/tmp/gotmp/go-build* trees held 4.5G and ENOSPC-failed a push gate that passed 10/10 after reclaim (gc-68bao) (#176)

* fix(scripts): reclaim leaked go-build/go-link temp trees in the harness sweep (gc-68bao)

go removes its per-invocation compile/link work dir under $GOTMPDIR only on a
clean exit. A run the harness watchdog kills (SIGQUIT->SIGKILL), a signalled
run, or an OOM kill leaks the tree, and nothing sweeps it. gc-68bao measured 27
such trees holding 4.5G that ENOSPC-failed a push gate which then passed 10/10
once they were reclaimed; the linker's go-link-* trees are the largest single
hoard. The trees appear both under $GOTMPDIR and, when a run set no GOTMPDIR,
directly under /var/tmp.

The process sweep in scripts/lib/harness-reap.sh already reaps stranded test
processes at gate start but never their leaked directories. This adds
gc_harness_sweep_stale_build_dirs there, called from test-local-parallel (the
one funnel all four heavy gate targets pass through) right after the process
sweep and on the same age floor, so the reclaim runs on every gate without a
new order or command.

The reclaim is deliberately bounded to go's own go-build*/go-link* shapes. That
bound is the safety argument: those prefixes are go-owned, never a checkout, a
cache-with-value, or a comparison base a human wants kept, and are regenerated
on the next build, so removing one needs no per-shape judgment. A tree is
reclaimed only when it is owned by the invoking user, older than the go test
budget, and referenced by no live process — the environ+cmdline+cwd /proc
predicate proven safe in gc-68bao (it spared every live build across two
concurrent gates). Every delete routes through the single gc_harness_reclaim_dir
seam, mirroring gc_harness_kill_pid, so the destroy authority is reviewable in
isolation and the self-test records decisions instead of removing real trees.

Scope: gc-68bao's notes also describe a larger /var/tmp half — gocache-* and
cold-build throwaway caches, 12.6G in one incident. A naive age+liveness scan of
that class was measured against the real host and matched 662 owned dirs of
mixed provenance, including comparison bases; encoding an unconditional rm -rf
of that set on every fleet gate is an unrecoverable blast radius and a policy
call, so it is filed as gc-ulys6 rather than shipped here.

Validation: new self-test TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected
(reclaims an abandoned tree at either root; spares a live-held tree, a too-young
tree, a non-go shape, and a .git-bearing dir). go vet clean; scripts
harness/budget/tmpdir suites green. A print-only dry run over the real host
/var/tmp selected exactly the ~10 real leaked go trees and none of the 662 other
dirs.

Follow-up: gc-ulys6 (the /var/tmp agent-cache half).

* test(census): bump subprocess baselines for the new harness sweep test (gc-68bao)

The fix commit added buildDirSweepHarness to scripts/harness_reap_test.go,
a fifth exec.Command("bash", ...) call that shells out to exercise
gc_harness_sweep_stale_build_dirs. That file is untagged (no //go:build
constraint), so the new call lands in the untagged subprocess census and
trips resourcecensus's "totals cannot grow" ratchet:
TestRepositoryLedgerMatchesCensusAndDocumentation went red with the scanned
count one above the checked baseline on three rows.

The ledger, the bootstrap policy, and the rendered doc are three mirrors that
comparePolicyFields/validateBaseline require to agree exactly, so the +1 is
applied to all three: test/test-resources.toml and census.go's bootstrapPolicy
(baseline_calls 649->650 all-source, 432->433 untagged source, 425->426
untagged Small), then TESTING.md regenerated via the -update flag. Only
baseline_calls moves; baseline_files is unchanged because the call joined a
file already counted, and the historical reported_* columns are left as-is.

The new subprocess is a legitimate test harness that runs a shell function, so
raising the ratcheted baseline is the sanctioned path (the fixed-sleep census
is the one you shrink instead). Validation: full
internal/testpolicy/resourcecensus package green without -update, go vet clean,
and the deliverable's own TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected
still passes.

* docs(scripts): state current behavior in harness-sweep comments

The go work-dir sweep's living comments carried a bead id and incident
history — measured hoard sizes, gate pass/fail counts, a follow-up policy
note. Convention is that comments state what is true now and the
constraints it rests on; history belongs in the bead and the commit log.

Rewrite the four comment blocks in scripts/lib/harness-reap.sh,
scripts/test-local-parallel, and scripts/harness_reap_test.go to state
only the reclaim's behavior and safety contract: it reclaims stale
same-UID go-build/go-link temp dirs after age and live-reference checks,
scans each caller-named root at depth 1, and leaves broader cache-root
cleanup out of scope. No behavior change.

Addresses pre-open signoff finding (round 1) on polecat/gc-68bao.

* fix(scripts): reclaim only go's exact go-build<n>/go-link-<n> shapes (gc-68bao)

The sweep matched the bare go-build*/go-link* prefix, so an old,
unreferenced human-named scratch dir such as go-build-base was selected
and rm -rf'd. go names its work dirs with os.MkdirTemp(dir, "go-build")
and os.MkdirTemp(dir, "go-link-"), which append a decimal suffix, so the
only go-owned shapes are go-build<digits> and go-link-<digits>. Match
exactly those.

The regression test renames its spare fixtures to valid go shapes so the
liveness/age/.git predicates (not shape) do the sparing, and adds
prefix-but-not-generated-shape cases the old glob would have reclaimed.

---------

Co-authored-by: refinery costing <refinery@local>

Rebased 2026-09-15 onto upstream 83c8270: the resource-census +1 hunks
(census.go, test-resources.toml, TESTING.md) were deltas on the previous
sync's baselines and are re-derived over the merged tree in the census
commit at the end of this rebase; the reaper, its test and the
test-local-parallel sweep call merged unchanged.

Claude-Session: https://claude.ai/code/session_01XfZfLDuwcBfs3TXnn1udn9

Author: zook-bot

scripts/harness_reap_test.go

@@ -6,6 +6,7 @@ import (
 	"os"
 	"os/exec"
 	"path/filepath"
+	"runtime"
 	"strconv"
 	"strings"
 	"syscall"
@@ -459,3 +460,124 @@ func TestGoTestShardLeavesNothingHoldingTheCallersPipe(t *testing.T) {
 		t.Fatal("reading the shard runner's output blocked after the run finished: a descendant outlived the run holding the caller's pipe")
 	}
 }
+
+// buildDirSweepHarness runs gc_harness_sweep_stale_build_dirs against a fixture
+// scratch tree and returns its reclaim decisions. Two seams are overridden: the
+// liveness source (so the test supplies a process's referenced paths without a
+// real /proc) and the one function that deletes a tree (so decisions are
+// recorded instead of destroying real dirs), mirroring how the process sweep
+// overrides gc_harness_kill_pid.
+func buildDirSweepHarness(t *testing.T, minAge string, roots []string, referenced string) string {
+	t.Helper()
+
+	dir := t.TempDir()
+	refFixture := filepath.Join(dir, "referenced.txt")
+	if err := os.WriteFile(refFixture, []byte(referenced), 0o644); err != nil {
+		t.Fatalf("write referenced fixture: %v", err)
+	}
+	reclaimLog := filepath.Join(dir, "reclaims")
+
+	var rootArgs strings.Builder
+	for _, r := range roots {
+		fmt.Fprintf(&rootArgs, " %q", r)
+	}
+
+	script := fmt.Sprintf(`
+set -euo pipefail
+source %q
+gc_harness_referenced_paths() { cat %q ; }
+gc_harness_reclaim_dir() { printf 'RECLAIMED %%s\n' "$1" >> %q ; }
+gc_harness_sweep_stale_build_dirs %s%s
+`, filepath.Join(repoRoot(t), "scripts", "lib", "harness-reap.sh"),
+		refFixture, reclaimLog, minAge, rootArgs.String())
+
+	out, err := exec.Command("bash", "-c", script).CombinedOutput()
+	if err != nil {
+		t.Fatalf("build-dir sweep failed: %v\n%s", err, out)
+	}
+	reclaims, readErr := os.ReadFile(reclaimLog)
+	if readErr != nil && !os.IsNotExist(readErr) {
+		t.Fatalf("read reclaim log: %v", readErr)
+	}
+	return string(reclaims) + string(out)
+}
+
+// TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected verifies the
+// go work-dir sweep: go removes its work dir only on a clean exit, so a run the
+// harness watchdog or a signal kills leaks the tree and nothing else sweeps it.
+// The sweep must reclaim an abandoned go-build<n>/go-link-<n> tree at any of the
+// scratch roots it is given while sparing a tree a live build still holds, one
+// too young to be abandoned, anything carrying a .git pointer, and — because the
+// sweep rm -rf's whatever it matches — any dir that is not go's exact generated
+// shape, including a go-prefixed but human-named comparison base.
+func TestHarnessSweepReclaimsLeakedGoWorkDirsButSparesLiveAndProtected(t *testing.T) {
+	if runtime.GOOS != "linux" {
+		t.Skip("go work-dir reclaim reads /proc for liveness; Linux-only")
+	}
+
+	root := t.TempDir()                   // stands in for /var/tmp
+	gotmp := filepath.Join(root, "gotmp") // stands in for $GOTMPDIR
+
+	old := time.Now().Add(-time.Hour)
+	fresh := time.Now()
+
+	mk := func(parent, name string, mtime time.Time, dotGit bool) string {
+		p := filepath.Join(parent, name)
+		if err := os.MkdirAll(p, 0o755); err != nil {
+			t.Fatalf("mkdir %s: %v", p, err)
+		}
+		if dotGit {
+			if err := os.WriteFile(filepath.Join(p, ".git"), []byte("gitdir: /elsewhere\n"), 0o644); err != nil {
+				t.Fatalf("write .git in %s: %v", p, err)
+			}
+		}
+		if err := os.Chtimes(p, mtime, mtime); err != nil {
+			t.Fatalf("chtimes %s: %v", p, err)
+		}
+		return p
+	}
+
+	reclaimAbandoned := mk(root, "go-build2147480000", old, false)
+	reclaimLink := mk(root, "go-link-4242", old, false)
+	// Spared by a real predicate, not by shape: each carries a valid
+	// go-build<n>/go-link-<n> name so the shape gate passes and the
+	// liveness/age/.git predicate is what does the sparing.
+	spareLive := mk(root, "go-build770001", old, false)
+	spareFresh := mk(root, "go-build550002", fresh, false)
+	spareWorktree := mk(root, "go-build330003", old, true)
+	// Spared by shape. go builds its work dirs with os.MkdirTemp(dir,
+	// "go-build") and os.MkdirTemp(dir, "go-link-"), which append a decimal
+	// suffix, so the only go-owned shapes are go-build<digits> and
+	// go-link-<digits>. These share a go prefix but are not that shape — a
+	// human-named comparison base, or a digit-then-letter tail — and one carries
+	// no go prefix at all (a real cache dir). All are old and unreferenced, so
+	// shape is the only thing standing between them and an rm -rf.
+	spareHumanBuild := mk(root, "go-build-base", old, false)
+	spareHumanLink := mk(root, "go-link-base", old, false)
+	spareDigitThenLetters := mk(root, "go-build12ab", old, false)
+	spareNonGo := mk(root, "gocache-tk-nnx2gd", old, false)
+
+	// The /var/tmp scan does not descend, so a tree leaked under $GOTMPDIR is
+	// only reclaimed through gotmp being passed as its own root. gotmp itself is
+	// a child of root but is spared there: its name is not a go work-dir shape.
+	mk(root, "gotmp", old, false)
+	reclaimNested := mk(gotmp, "go-build9998887", old, false)
+
+	// A live build holds spareLive as its working directory; the seam feeds that
+	// path to the sweep as the sole referenced path.
+	got := buildDirSweepHarness(t, "60", []string{root, gotmp}, spareLive+"\n")
+
+	for _, want := range []string{reclaimAbandoned, reclaimLink, reclaimNested} {
+		if !strings.Contains(got, "RECLAIMED "+want+"\n") {
+			t.Errorf("sweep left an abandoned go work dir behind: %s\nlog:\n%s", want, got)
+		}
+	}
+	for _, spared := range []string{
+		spareLive, spareFresh, spareWorktree,
+		spareHumanBuild, spareHumanLink, spareDigitThenLetters, spareNonGo,
+	} {
+		if strings.Contains(got, "RECLAIMED "+spared+"\n") {
+			t.Errorf("sweep reclaimed a dir it must spare: %s\nlog:\n%s", spared, got)
+		}
+	}
+}

scripts/lib/harness-reap.sh

@@ -289,3 +289,108 @@ gc_harness_test_socket_holders() {
     }
   '
 }
+
+# gc_harness_referenced_paths prints every filesystem path a live process of the
+# invoking user still references — env values, argv, and cwd — one per line. It
+# is the liveness half of gc_harness_sweep_stale_build_dirs: a work dir whose
+# path appears here belongs to a running build and must be spared. Reading
+# /proc/<pid>/{environ,cmdline,cwd} is the liveness predicate. Only the invoking
+# user's process files are readable, which is exactly the set that can own a
+# 0700 go work dir, so a different user's live build can never be misread as
+# absent. Factored out as the one data source so the self-test can inject a
+# process table without a real /proc.
+#
+# stderr is discarded at the loop, not per command: opening another user's
+# /proc/<pid>/environ is denied, and that open failure is the shell's, emitted
+# before a trailing per-command 2>/dev/null on the same line can take effect —
+# so on a shared host it would otherwise flood every caller with one
+# "Permission denied" per foreign process. A denied entry is simply not one of
+# ours and contributes nothing to the set.
+gc_harness_referenced_paths() {
+  local p
+  for p in /proc/[0-9]*; do
+    tr '\0' '\n' < "$p/environ" || true
+    tr '\0' '\n' < "$p/cmdline" || true
+    readlink "$p/cwd" || true
+  done 2>/dev/null
+}
+
+# gc_harness_reclaim_dir removes one abandoned work-dir tree. Every reclaim goes
+# through here — the one place the sweep deletes a directory — mirroring
+# gc_harness_kill_pid for process kills: it keeps the destroy authority
+# reviewable in isolation and gives the self-test a seam to record decisions
+# instead of removing real trees.
+gc_harness_reclaim_dir() {
+  local dir="${1:-}"
+  [[ -n "$dir" ]] || return 0
+  rm -rf -- "$dir" 2>/dev/null || true
+}
+
+# gc_harness_sweep_stale_build_dirs reclaims go's per-invocation compile and
+# link temp trees that a killed run leaked. go removes its work dir only on a
+# clean exit, so a run ended by the watchdog above, a signal, or an OOM kill
+# leaves the tree behind and nothing else sweeps it. The trees appear both under
+# $GOTMPDIR and, when a run set no GOTMPDIR, directly under its TMPDIR, so every
+# scratch root the caller names is scanned at depth 1.
+#
+# go builds those dirs with os.MkdirTemp(dir, "go-build") and
+# os.MkdirTemp(dir, "go-link-"), each of which appends a decimal random suffix,
+# so the only go-owned shapes are go-build<digits> and go-link-<digits>. The
+# sweep matches exactly those, and that exactness is the whole safety argument:
+# a go-owned work dir is never a checkout, a cache-with-value, or a comparison
+# base a human wants kept, and is regenerated on the next build, so reclaiming
+# one is always safe and needs no per-shape judgment. Matching the bare
+# go-build/go-link prefix instead would also select human-named scratch such as
+# go-build-base, which this path would then rm -rf. Broader cache-root cleanup —
+# arbitrary scratch dirs of mixed provenance, which can include comparison bases
+# a human wants kept — is out of scope here. A tree is reclaimed only when it is
+# owned by the invoking user, older than min_age_seconds, and referenced by no
+# live process. The age floor is the caller's go test budget, so a concurrently
+# starting build is never in scope, and the liveness check spares an in-flight
+# build regardless of age. Set GC_TEST_NO_BUILD_DIR_SWEEP=1 to skip the reclaim
+# entirely.
+gc_harness_sweep_stale_build_dirs() {
+  local min_age_seconds="${1:-3600}"
+  if (( $# > 0 )); then shift; fi
+  [[ "${GC_TEST_NO_BUILD_DIR_SWEEP:-0}" != "1" ]] || return 0
+  [[ $# -gt 0 ]] || return 0
+  # Liveness assessment needs procfs. Without it a dir cannot be proven unused,
+  # and reclaiming on an empty liveness set would delete a live build, so the
+  # whole sweep skips rather than guess — the leak is a Linux-host condition.
+  [[ -e /proc/self/cmdline ]] || return 0
+  [[ "$min_age_seconds" =~ ^[0-9]+$ ]] || return 0
+
+  local uid now referenced
+  uid="$(id -u)"
+  now="$(date +%s)"
+  referenced="$(gc_harness_referenced_paths)"
+
+  local reclaimed=0
+  local root child base mtime
+  for root in "$@"; do
+    [[ -n "$root" && -d "$root" ]] || continue
+    while IFS= read -r child; do
+      [[ -n "$child" ]] || continue
+      base="${child##*/}"
+      # Match only go's generated shapes, not the bare go-build/go-link prefix:
+      # this path rm -rf's what it matches, and the prefix alone also selects
+      # human-named scratch such as go-build-base.
+      [[ "$base" =~ ^go-build[0-9]+$ || "$base" =~ ^go-link-[0-9]+$ ]] || continue
+      # A go work dir never carries a .git pointer; skipping one is defence in
+      # depth so a name collision can never turn the sweep on a checkout.
+      [[ -e "$child/.git" ]] && continue
+      mtime="$(stat -c %Y "$child" 2>/dev/null)" || continue
+      (( now - mtime >= min_age_seconds )) || continue
+      if [[ -n "$referenced" ]] && grep -qF -- "$child" <<< "$referenced"; then
+        continue
+      fi
+      printf 'harness sweep: reclaiming abandoned go work dir %s (age %ss, no live holder)\n' \
+        "$child" "$(( now - mtime ))" >&2
+      gc_harness_reclaim_dir "$child"
+      reclaimed=$(( reclaimed + 1 ))
+    done < <(find "$root" -maxdepth 1 -mindepth 1 -type d -uid "$uid" 2>/dev/null || true)
+  done
+
+  (( reclaimed == 0 )) || printf 'harness sweep: reclaimed %s abandoned go work dir(s)\n' "$reclaimed" >&2
+  return 0
+}

scripts/test-local-parallel

@@ -287,6 +287,16 @@ export TEST_LOCAL_NICE="$nice_prefix"
 sweep_min_age_seconds="$(gc_harness_duration_seconds "$go_test_timeout")" || sweep_min_age_seconds=3600
 gc_harness_sweep_stale_orphans "$sweep_min_age_seconds"
 
+# Reclaim go's leaked compile/link temp before adding load of our own. go
+# deletes its per-invocation work dir only on a clean exit, so a run the
+# watchdog above, a signal, or an OOM kill ends leaks go-build*/go-link* under
+# $GOTMPDIR — or under /var/tmp when the run set no GOTMPDIR — and nothing else
+# sweeps them. Same age floor as the process sweep, so an in-flight build is
+# never in scope; a bare session reports no GOTMPDIR, so fall back to the shim's
+# on-disk root.
+gc_harness_sweep_stale_build_dirs "$sweep_min_age_seconds" \
+  "${gotmpdir_val:-/var/tmp/gotmp}" /var/tmp
+
 echo "Running ${#jobspecs[@]} ${mode} job(s) with LOCAL_TEST_JOBS=${local_jobs} inner_p=${inner_p}"
 
 set +e