fix: Polecat worktrees for merged work beads are never reclaimed: 17 stale trees (~2.5G) in the gascity rig alone, and the resulting host ENOSPC false-FAILs every push gate at the cmd/gc link step (gc-4guc2) - #182
Conversation
…tree (gc-4guc2) Per-bead git worktrees were never reclaimed. Measured across a four-rig city: 748 worktrees held, every one of them on the git-safety gate, and 708 of those reporting stashes=true. In the gascity rig a single stash — "WIP on main", on the rig's own checkout — held all 31 of its worktrees. refs/stash is a single repository-global ref. Git records no worktree identity on a stash entry, so `git stash list` answers identically from every worktree of a repository: a per-worktree gate cannot be built on it, and one stash anywhere protects the whole rig for as long as it exists. Removal cannot lose the stash either — `git worktree remove` deletes the checkout, not refs/stash — which is the same argument the reaper already carries for declining to gate on push state. The gate was therefore both unimplementable and unnecessary, and it is gone from all three places that ran it: the closed-bead reaper, the worker_dir prune that AutoPruneWorkerDir drives by default, and doctor's nested-worktree check, which the other two cite as their reference gate. The uncommitted and unreachable-commit gates stay: removal really does destroy both. The operator-facing text that advertised the stash gate goes with it — doctor's --fix hint and the three config doc comments that feed docs/reference/config.md and the city schema. Against the live city the classification goes from 33 reclaimable / 748 protected to 278 / 502, with no protection mentioning stashes and the dirty and unreachable-commit trees still correctly held. Reclaiming still needed an operator to edit city.toml and restart the controller, because the patrol is the reaper's only caller and defaults to off. `gc worktree reap` runs the same classification on demand: it reports what it would remove and removes nothing until --apply. That is how the numbers above were measured. It joins `gc worktree` alongside ensure, verify and cleanup, which own single-worktree provisioning; reap is the bulk sweep driven by bead state. reapReport grows Errors and LivenessSource so a caller rendering the report rather than the stderr log still sees a rig it could not scan, a removal git refused, and a verdict reached on the fallback liveness scanner. The report is ordered by rig then path: the reaper walks rigs in Go map order, which is randomized per run, so without a sort the same city lists its rigs differently every invocation and two runs of the command cannot be diffed against each other. Command census: reap mints id 208 against next_id 208; upstream holds 203-205 for worktree cleanup/ensure/verify, and the generated catalog count goes 203 -> 204. Claude-Session: https://claude.ai/code/session_01BdbURuwqXGvCC3waJcyhzT
`gc worktree reap --json` advertised a machine-readable mode the root JSON
contract rejected. A built-in command with no schemas/<path>/result.schema.json
is answered with json_unsupported before its RunE runs, so the flag exited 1
and emitted no report, and the payload carried no result envelope either.
- Add schemas/worktree/reap/result.schema.json. Its presence is what makes the
command declare JSON support.
- Stamp schema_version, ok, command and action in writeWorktreeReapJSON, so
both producers describe the pass identically and ok tracks the exit code.
- Cover the real CLI path. TestCmdWorktreeReapStrictJSONContract runs
run([]string{"worktree", "reap", "--json"}) under GC_JSON_CONTRACT_STRICT=1
and validates the payload against the schema; the populated-verdict test
validates entry shape. The existing tests called runWorktreeReap directly and
so never met the contract gate.
The troubleshooting page said a reaped worktree reproduces with
`git worktree add <path> origin/<branch>`. The git gate allows removal when
HEAD is reachable from any branch, tag, or remote-tracking ref, so a local-only
branch is reapable while origin/<branch> does not exist. State the recovery
from the surviving local ref, and name origin/<branch> only when it exists.
Claude-Session: https://claude.ai/code/session_01XNQF4rpt8UbEa1vqthuj56
…mment (gc-4guc2) The pruneAgentHomeWorktreeIfSafeInfo call site in reconcileSessionBeadsTracedWithNamedDemand described the prune as "Skipped under safety gates (uncommitted, unpushed, stashed)", but this branch already removed the stashed gate from the helper: refs/stash is a single repository-global ref carrying no worktree identity, and git worktree remove cannot lose it. The helper now skips only on uncommitted or unpushed work, so the comment named a gate that no longer exists. Drop it so the comment states the current behavior. Addresses the sole pre-open signoff P1 (review gc-uhxwi) on branch polecat/gc-4guc2. Claude-Session: https://claude.ai/code/session_01VJTf66qCuhxWY5MfYVjWPv
…reap` --json (gc-4guc2) A reap pass that could not scan a rig or complete a removal returns exit 1, while writeWorktreeReapJSON derived ok from the same errors: the --json output was ok:false with the reap-specific fields and no error object. That payload matched neither the published result schema (ok is const true) nor the shared failure schema (which requires an error object), so a machine caller reading a partial pass got output that validated against no contract. Stamp the shared error object (code/message/exit_code) whenever the pass has errors, mirroring worktree cleanup. The reaped/protected verdicts stay in the envelope so a partial pass is still legible; failure.schema.json admits them via additionalProperties. Adds an error-path contract test alongside the success-path one and a shared-failure-schema validation helper. Claude-Session: https://claude.ai/code/session_01ApGo7VPkyHqmXjyiW9rFYn
|
Pre-open signoff (comment-only — not an approval): VERDICT: request-changes Scope checked: Read the full three-dot diff from origin/main to 3680eef across the reaper, worker-dir prune, worktree command, JSON schema, command census, config/doctor comments, generated docs, and troubleshooting docs. Checked anchor bead gc-4guc2, review bead gc-oieqs, the mol-review step text, the pack work-quality fragment, the learning-exemplar fragment, the JSON result/failure contract path, the event-provider call path, openStandaloneRigStores, reaper call sites, and the git helper safety semantics. This is pre-open, so there is no PR page to review. I did not run the full project matrix or dashboard checks; the diff does not touch internal/api, OpenAPI, dashboard code, or generated dashboard types. Findings: P1 REQUEST-CHANGES - internal/doctor/checks_semantic_test.go:1433 still says the fake worktrees have "no stashes." At this reviewed commit, the Filed: none. Verification: At the reviewed commit in detached worktree /tmp/gc-review-gc-oieqs.STWxJd, Anchor: gc-4guc2 — check.codex @ 3680eef |
|
Closing as not-needed: disposition recorded on anchor gc-4guc2 (successor gc-85pdo). The refinery disposes the anchor from this close; no rework-or-close decision is owed. Operator ruled against fork-local adoption (keep gascity aligned with upstream); acute ENOSPC already relieved by dropping the 4-month stale 'WIP on main' stash that had jammed the repo-global stash gate. The stash-check fix, if ever wanted, belongs upstream — not a local change. Fix preserved in closed PR #182. |
1 participant
Summary
The problem
Per-bead git worktrees were never reclaimed. Measured across a four-rig city: 748
worktrees held, every one of them on the git-safety gate, and 708 of those
reporting
stashes=true. In the gascity rig a single stash — "WIP on main", onthe rig's own checkout — held all 31 of its worktrees. The trees accumulate until
the host runs out of disk, and the resulting ENOSPC false-FAILs every push gate at
the
cmd/gclink step.Why the stash gate cannot be made to work
refs/stashis a single repository-global ref. Git records no worktree identity ona stash entry, so
git stash listanswers identically from every worktree of arepository: a per-worktree gate cannot be built on it, and one stash anywhere
protects the whole rig for as long as it exists.
Removal cannot lose the stash either —
git worktree removedeletes the checkout,not
refs/stash— which is the same argument the reaper already carries fordeclining to gate on push state. The gate was therefore both unimplementable and
unnecessary.
The change
The stash probe is gone from all three places that ran it: the closed-bead
reaper (
cmd/gc/bead_worktree_reaper.go), theworker_dirprune thatAutoPruneWorkerDirdrives by default (cmd/gc/session_worktree_prune.go), anddoctor's nested-worktree check (
internal/doctor/checks_semantic.go), which theother two cite as their reference gate. The uncommitted and unreachable-commit
gates stay: removal really does destroy both. The operator-facing text that
advertised the stash gate goes with it — doctor's
--fixhint and the threeconfig doc comments that feed
docs/reference/config.mdand the city schema.New
gc worktree reap. Reclaiming still needed an operator to editcity.tomland restart the controller, because the patrol is the reaper's onlycaller and defaults to off.
reapruns the same classification on demand: itreports what it would remove and removes nothing until
--apply. It joinsgc worktreealongsideensure,verifyandcleanup, which ownsingle-worktree provisioning;
reapis the bulk sweep driven by bead state.--jsonis schema-declared. The root JSON contract answers a built-incommand that has no
schemas/<path>/result.schema.jsonwithjson_unsupportedbefore its
RunEruns, so the flag alone does not make the mode usable.schemas/worktree/reap/result.schema.jsondeclares it, and the payload openswith the standard
schema_version/ok/command/actionenvelope, withokcarrying the same verdict as the exit code. The contract test drives
run([]string{"worktree", "reap", "--json"})underGC_JSON_CONTRACT_STRICT=1and validates the output against the published schema.
A pass that could not scan a rig or complete a removal exits nonzero, and the
published result schema pins
okto a consttrue, so a nonzero exit cannotsatisfy it. On such a pass
okis false and the sharederrorobject(
code,message,exit_code) is stamped, mirroring worktree cleanup, so theoutput meets the shared failure schema while
reapedandprotectedstillreport the partial pass. A second contract test drives that error path and
validates it against the failure schema.
reapReportgrowsErrorsandLivenessSource, so a caller rendering thereport rather than the stderr log still sees a rig it could not scan, a removal
git refused, and a verdict reached on the fallback liveness scanner.
The report is ordered by rig then path. The reaper walks rigs in Go map
order, which is randomized per run, so without a sort the same city lists its
rigs differently every invocation and two runs of the command cannot be diffed
against each other.
docs/troubleshooting/reclaim-worktrees.mdgives the operator the disk-fullcascade, the gate table, and the recovery. Recovery is stated from the ref that
survives removal: the git gate passes a worktree whose HEAD is reachable from
any branch, tag, or remote-tracking ref, so a local-only branch is reapable
while
origin/<branch>does not exist.Effect
Against the live city the classification goes from 33 reclaimable / 748 protected
to 278 / 502, with no protection mentioning stashes and the dirty and
unreachable-commit trees still correctly held. That is also how the numbers above
were measured.
Review notes
Command census:
reapmints id 208 againstnext_id208; upstream holds 203-205for worktree cleanup/ensure/verify, and the generated catalog count goes 203 -> 204.
Dispatch — what this work was asked to do
What
Every polecat worktree under
.gc/worktrees/<rig>/polecats/<slot>/worktrees/<bead>survives after its work bead closes. Measured 2026-09-02 in the gascity rig:
17 trees still on disk, ~2.5G, all for beads that are closed and in most cases
merge_result=mergedand landed on main..gc/worktreesacross all four rigsis 32G.
Why it is P1 rather than housekeeping
The host disk it sits on is the same one
go linkwrites to. With / at 99%(3.9G free) every
unit-cmd-gc-*shard of a pre-push gate dies before running asingle test:
All 6 cmd/gc shards plus unit-core failed that way on gc-09meh's gate, with zero
EDQUOT anywhere in the log, so this is real host ENOSPC and not the per-agent
quota shape tracked by gc-6jye9. That gate reads as a FAIL on the diff. It is
not one: internal/tomledit still produced a real
okline in the same run whileinternal/config read
[build failed].Reclaiming only provably-orphaned space — 6 stale
/var/tmp/go-link-*dirs withno live holder, stale
gc-local-tests.*roots, and the four finished worktrees inthis pool slot's own home — moved free space 3.9G -> 4.7G, and the identical gate
then passed 10/10 with zero
no space lefthits. So the accumulation is theproximate cause of a false-FAIL class that costs a full ~40-minute gate per
occurrence and invites blaming the diff.
Two candidate mechanisms, both worth checking
poured workflow (
gc bd dep list-> "no dependencies"), so its method camefrom metadata. It runs workspace-setup and creates a worktree, but there is no
cleanup-worktreestep in existence to remove it. Every rework round mintsone of these.
trees above are named for anchor ids (gc-c9upz, gc-dz64s, gc-gf1l6), i.e.
they came from the original
mol-polecat-workrun's workspace-setup, not froma rework. Those runs had a full molecule with a
cleanup-worktreestep, andthe tree is still there. gc-gfoc7 (closed) covered one way that step gets
stranded; this is evidence it is still not reliably reached.
What is safe to reclaim
git worktree removeon a clean tree loses nothing: every commit stays reachablethrough its branch ref. The gate is the existing one from the polecat-patterns
worktree-reclaim rule — owning session closed or absent, no process cwd'd in the
tree,
git status --porcelainempty — plus the bead being closed. All four treesremoved by hand here met it.
Do NOT reach for the 68G
~/.cache/go-build: AGENTS.md hard-bansgo clean -cachebecause it corrupts the shared fleet cache.Related
failure: this one has GOTMPDIR set and hits real ENOSPC on /.
Refinery handoff
gc-4guc2polecat/gc-4guc2maincodexsigned off pre-open at3680eef7; PR opened green.Beads on this branch
Every bead whose work is on
polecat/gc-4guc2. Approving this PR approves all of them.gc-4guc2— Polecat worktrees for merged work beads are never reclaimed: 17 stale trees (~2.5G) in the gascity rig alone, and the resulting host ENOSPC false-FAILs every pu (opener)gc-gq3o4— Rework branch polecat/gc-4guc2: address pre-open signoff findingsgc-tpgxm— Rework branch polecat/gc-4guc2: address pre-open signoff findingsgc-7li1z— Rework branch polecat/gc-4guc2: address pre-open signoff findings