gc session: a CLOSED session bead permanently reserves its alias — no CLI lever releases it, so an on_demand agent can never exist again (gc-5fdrr) by zook-bot · Pull Request #144 · zookanalytics/gascity

zook-bot · 2026-08-15T21:00:51Z

Summary

When a session bead is swept CLOSED it KEEPS canonical_instance_name / alias / session_name.
Name resolution still consults the closed bead and refuses to create a replacement, while every
command that could clear or reuse the name rejects it for being closed. The alias is permanently
bricked and the agent can NEVER be spawned again.

Live incident (2026-08-13 -> 2026-08-15, ~40h outage)

lx-mzfh — session bead for shutupandlisten/gc-toolkit.refinery, closed 2026-08-14T02:22:28Z,
close_reason="session swept: no assigned work in any rig". It retained:

agent_name              = shutupandlisten/gc-toolkit.refinery
alias                   = shutupandlisten/gc-toolkit.refinery
canonical_instance_name = shutupandlisten/gc-toolkit.refinery
session_name            = shutupandlisten--gc-toolkit__refinery

No tmux session by that name existed — the reservation lived purely in the session store.

The deadlock: every designed lever is blocked

command	result
`gc session nudge <alias>`	`session name already exists: "shutupandlisten--gc-toolkit__refinery" already belongs to lx-mzfh`
`gc session wake <alias>`	same
`gc session peek <alias>`	`not_found: session not found`
`gc session nudge lx-mzfh`	`session lx-mzfh is closed`
`gc session wake lx-mzfh`	`session lx-mzfh is closed`
`gc session kill lx-mzfh`	`session is closed: lx-mzfh` (no `--force` flag exists)
`gc session prune`	only suspended/asleep/drained; `--state` does not accept `closed`
`gc session new`	no reclaim/force flag

Diagnostic asymmetry that makes this hard to spot: peek by a bricked id exits 0 with EMPTY
output (a bogus id exits 1), and the bead is absent from gc session list --state all.

Impact

shutupandlisten/gc-toolkit.refinery is on_demand and could never wake. Work bead su-uzy9.6
(branch polecat/su-uzy9.6, pushed to origin at df67aaeb) sat unmergeable for ~40h. The rig's
witness escalated 20+ times; the shutdown-dance warrant (lx-ua2wf) returned EXECUTE_FAILED
because there was no agent to dance with — only a name reservation.

Manual workaround (what actually unblocked it)

Session-store surgery by the mayor — clear the reservation on the CLOSED bead:

gc bd update lx-mzfh --set-metadata canonical_instance_name= --set-metadata alias= --set-metadata session_name=
gc session wake shutupandlisten/gc-toolkit.refinery   # succeeds immediately afterward

This requires knowing the bead id, the three exact metadata keys, and that the operation is safe —
none of which is discoverable from any error message.

This is not a one-off — there is a live second instance

lx-36okl (signal-loom/gc-toolkit.refinery) is OPEN but already holds
canonical_instance_name=signal-loom/gc-toolkit.refinery while running under the MISMATCHED tmux
name gc-toolkit__refinery-lx-36okl. gc doctor reports it as
session-model: configured-named-conflict. When that bead is eventually swept closed it will brick
signal-loom's refinery alias in exactly the same way.

Asks (any one closes the outage class; 1 or 2 preferred)

Release the reservation on close. When a session bead is closed/swept, clear
canonical_instance_name / alias / session_name (keep agent_name for history). A closed
session should hold no runtime name.
Ignore closed beads in name resolution. Treat a closed bead's name as free; the store already
knows the bead is closed.
Give operators a lever — gc session kill --force <id> or gc session release-name <alias>
— and make the "name already exists" error name the holding bead AND the remedy.
Make gc doctor's configured-named-conflict finding actionable: state that a closed holder is
unrecoverable without metadata surgery.

Provenance

dog-1 EXECUTE_FAILED report (mail lx-wisp-buj0d), deacon escalation (lx-wisp-33o96), 20+
shutupandlisten witness QUEUE_HEALTH escalations, deacon doctor finding (lx-wisp-qb873).
Diagnosed and manually unblocked by mayor 2026-08-15T17:30Z.

Implementation notes

Implemented (polecat gc-toolkit.nux, branch polecat/gc-5fdrr, 2 commits).

Root cause was NOT "closed beads keep their metadata" in general — it was a
divergence between two checks of the same name. The named-session start path
pre-clears the name via the cfg-aware EnsureSessionNameAvailableWithConfigForOwner
(whose legacy bypass exists for exactly this closed-phantom shape), then
Manager.CreateSession re-checks it via the cfg-less
ensureSessionNameAvailableForSelfAndOwner. That inner check recognized a closed
configured-named holder only by the boolean flag or configured_named_identity —
both owner-AGNOSTIC — so lx-mzfh (identity carried only by alias/agent_name,
the ga-n2d Gap C shape) fell through and the inner check vetoed what the outer
check had just allowed.

Delivered against the asks:

Ask 2 (scoped): the inner check now also releases a CLOSED holder via
configuredNamedIdentitySignalsMatch — the owner-SCOPED recognizer already
used for this same trap in name_claim_sweep.go and the alias check. Only the
configured identity whose own recorded signals resolve to the holder reclaims
the name. A different claimant, an ownerless claimant, and any live holder
are all still rejected, so the deliberate permanent-identity rule for ad-hoc
explicit session names is untouched. (Ask 1 — clear the keys on close — was
not taken: it inverts that deliberate rule for every explicit name, not just
configured ones.)
Ask 3, both halves: gc session release-name <name-or-alias> clears
session_name / alias / canonical-identity from CLOSED holders (agent_name
survives as history), refuses all-or-nothing if any holder is live, and
no-ops cleanly when nothing holds the name. The ErrSessionNameExists message
now says the holder is (closed) and names that command, so the remedy is
readable off the error instead of requiring 40h of diagnosis.
Ask 4: not applicable as written. gc doctor's session-model check skips
closed beads outright (cmd/gc/doctor_session_model.go), so it never reported
a closed holder — and its premise ("unrecoverable without metadata surgery")
is now false. The live-conflict finding is left alone: a live conflict's
remedy is stopping/reconciling that session, not releasing a name.

Note for the second live instance called out on this bead: lx-36okl
(signal-loom/gc-toolkit.refinery) is OPEN, so it is unaffected either way
today; when it is eventually swept closed, its name is now releasable both
automatically (to its own identity) and manually.

Filed as discovered work, out of scope here: gc-tczq2 — gc session peek on a
CLOSED session exits 0 with empty output, indistinguishable from a live-but-
quiet session (the diagnostic asymmetry this bead describes).

Validation: internal/session and internal/productmetrics packages green; the
new Manager-level test was confirmed to FAIL without the fix with the exact
production error and pass with it. Full make test-fast-parallel push gate
passed on the final push ("All fast jobs passed"). An earlier sweep flaked on
TestWaitForSupervisorReadyUsesHookedTimeout; that is a load flake, not a
regression — it passes 3/3 standalone, its shard is green 2/2 in isolation on
both sides, and the SAME gate run at the merge-base (unmodified origin/main)
failed with a LARGER, disjoint set
{TestCityRuntimeForceShutdownTearsDownAfterLateAsyncSweep,
TestProvider_StartCancellationInterruptsForegroundChild}.
2026-08-15T20:50Z REFINERY pre-open dispatch (wisp gc-wisp-ehd). Fresh mr bead (merge_strategy unset -> city default mr; no existing_pr) => PRE-OPEN path. NOT a rebase bead (no molecule_id, no backup_ref).

BRANCH: polecat/gc-5fdrr @ 1637d66, 2 ahead / 0 BEHIND origin/main dbc0012 (merge-base IS current main). git merge-tree --write-tree rc=0. No rebase performed or owed — per the pre-open rule a self-inflicted rebase would only drag in the local pre-push gate for nothing.

CAP: branch-scoped prior rework rounds = 0 => review round 1 of 3 (GC_MAX_REVIEW_ROUNDS). No reviews were in flight for this branch (idempotency checked).

GATE (fresh + isolated, TEST_ENV shape, TMPDIR=/var/tmp/rp, worktree mutex held): build ./... rc=0; vet ./cmd/gc ./internal/session ./internal/productmetrics rc=0; internal/session 626 tests ok 1.851s; internal/productmetrics ok 19.722s; cmd/gc via all SIX make test-cmd-gc-unit-shard shards (never a bare go test ./cmd/gc/ — Footgun 29) headers 'shard N of 6' with 1482/1482/1482/1481/1481/1481 = 8889 tests and six DISTINCT durations 72.4/91.8/85.9/78.6/109.8/107.4s. GATE_OVERALL_RC=0, GATE_DISK_EVENT=0, 8 'ok' lines, 0 FAIL/panic, 0 'Couldn't get agent socket' (gitconfig isolation intact), 0 SKIP in the affected packages. Branch's OWN new tests confirmed selected+run: TestReleaseSessionNameClaim_* (5) and TestCmdSessionReleaseName* (3). Wall 20:39:47Z-20:48:46Z. Logs /var/tmp/refinery-gatelogs/gc-5fdrr/.

Command census (metrics_census_gen.go / productmetrics_command_census.json / command_ids_gen.go) is machine-generated by gen-command-census and validated by cmd/gc/metrics_census_test.go, which the shards exercise — faithfulness covered mechanically.

DISPATCHED codex signoff gc-xtpwm (anchor_bead=gc-5fdrr, review_branch/base, review_pool + gc.routed_to = gascity/gc-toolkit.polecat-codex), dep 'blocks gc-5fdrr' recorded, anchor_bead verified on BOTH read paths. Codex wake/nudge no-op'd 'session not found' as usual; supervisor auto-spawns from gc.routed_to. Review body frames two scope questions for codex WITHOUT deciding them (the polecat declined Ask 1 and declared Ask 4 moot, narrowing a 4-ask bead to a scoped Ask 2 + Ask 3; and the owner-scoped configuredNamedIdentitySignalsMatch boundary is the whole safety argument), plus the polecat's pre-existing-flake claim to sanity-check rather than re-derive. Refinery did NOT review the code.

Parked pre_open_gate; branch+target KEPT for pre-open-resolve.sh.

Refinery handoff

Issue: gc-5fdrr (bug, P2)
Source branch: polecat/gc-5fdrr
Target: main
Codex signed off pre-open at 1637d667; PR opened codex-green.

…ed identity (gc-5fdrr) A CLOSED session bead kept the runtime identifiers it ran under, and name resolution still consulted them, so an on_demand agent's alias could be reserved forever by a dead record. The live incident (gc-5fdrr) held `shutupandlisten/gc-toolkit.refinery` unspawnable for ~40h: every documented lever refused it — nudge/wake by alias hit "name already exists", nudge/wake/ kill by bead ID hit "session is closed", `prune` does not accept the closed state, and `session new` has no reclaim flag. It was broken only by hand metadata surgery on three undiscoverable keys. Root cause is a divergence between two checks of the same name. The named-session start path clears `EnsureSessionNameAvailableWithConfigForOwner`, whose legacy bypass exists precisely for pre-ga841 phantoms, and then `Manager.CreateSession` re-checks the same name through the cfg-less `ensureSessionNameAvailableForSelfAndOwner`. That inner check recognized a closed configured-named holder only by the boolean flag or the recorded identity — both owner-AGNOSTIC — so a phantom carrying its identity solely as alias/agent_name (ga-n2d Gap C shape) fell through and the inner check vetoed what the outer check had just allowed. The fix reuses `configuredNamedIdentitySignalsMatch`, the owner-SCOPED recognizer already used for this same trap in the startup sweep and the alias-availability check: a CLOSED holder releases the name only to the configured identity its own recorded signals resolve to. A different claimant, an ownerless claimant, and any live holder are all still rejected, so the permanent-identity rule for ad-hoc explicit names is untouched. Two things the automatic release deliberately does not cover — an ad-hoc explicit name, and a holder whose signals name no claimant — still left the operator with no lever, so this also adds `gc session release-name <name-or-alias>`: it clears session_name, alias, and the canonical-identity record from CLOSED holders (agent_name survives as history), refuses all-or-nothing when any holder is live, and no-ops cleanly when nothing holds the name. The ErrSessionNameExists message now says the holder is closed and names that command, so the next operator to hit this reads the remedy off the error instead of diagnosing for 40h. Validation: new unit tests in internal/session cover the release, the other-owner / ownerless / live-holder guards, the error text, and the release-name domain function; cmd/gc tests cover the command end to end. The generated command census, metrics catalog, example schema enum, and CLI reference were regenerated by their own generators; the productmetrics catalog-size baseline moves 199 -> 200 for the new command. On ask 4 of the bead (make gc doctor's `configured-named-conflict` finding say a closed holder needs metadata surgery): that check skips closed beads outright (`cmd/gc/doctor_session_model.go`), so it only ever reports a LIVE conflict, and its premise no longer holds — a closed holder is now either auto-released to its own identity or releasable with one command. The live finding is left as-is: a live conflict's remedy is stopping or reconciling that session, not releasing a name.

…pawn The helper-level tests cover ensureSessionNameAvailableForSelfAndOwner in isolation, but the outage happened one layer up: the named-session start path pre-cleared the name through the cfg-aware check and Manager.CreateSession then re-checked it through the cfg-less helper and refused. A test that never goes through CreateSession would not have caught that divergence. This adds the end-to-end guard — a bead-only CreateSession for a configured named identity over a CLOSED legacy holder whose identity survives only as alias/agent_name. Confirmed to fail without the fix with the exact production error ("already belongs to gc-1 (closed)") and pass with it.

zook-bot · 2026-08-15T21:00:55Z

Codex signoff (pre-open, comment-only — not an approval):

VERDICT: COMMENT
Reviewed branch: polecat/gc-5fdrr
Reviewed base: main
Reviewed commit: 1637d66

Scope checked: Full diff from origin/main to the pinned commit, covering cmd/gc release-name command wiring and JSON result schema, internal/session name availability and manual release paths, manager CreateSession regression coverage, productmetrics command census updates, generated schemas, and CLI docs. I explicitly checked the two scope questions: the scoped Ask 2 plus Ask 3 closes the outage class without clearing every closed session's identifiers on close, and the new legacy-holder bypass is owner-scoped through configuredNamedIdentitySignalsMatch while live holders, other owners, and ownerless callers remain rejected for the legacy shape under review.

Findings: None. No P0/P1 issues found.

Verification: git diff --check origin/main...1637d66 passed with no output. TMPDIR=/var/tmp/rp go test ./internal/session ./internal/productmetrics passed: internal/session 1.263s, internal/productmetrics 10.403s. GC_FAST_UNIT=1 GO_TEST_COUNT=1 GO_TEST_TIMEOUT=20m go test ./cmd/gc -run '^(TestCmdSessionReleaseNameClearsClosedHolder|TestCmdSessionReleaseNameRefusesLiveHolder|TestCmdSessionReleaseNameNoHolder|TestSessionMutationActionSchemasDeclared|TestProductMetricsCommandCensusMatchesProductionBuiltins)$' -count=1 passed: cmd/gc 1.752s. GC_FAST_UNIT=1 GO_TEST_COUNT=1 GO_TEST_TIMEOUT=20m go test ./cmd/gc -run '^(TestJSONSchemaManifestForSupportedCommand|TestActionResultSchemasAllowExtensionFields)$' -count=1 passed: cmd/gc 0.478s.

Not checked: I did not rerun the refinery's full mechanical gate because the review bead explicitly recorded it as already run and said not to redo it; CI remains authoritative after PR open.

zook-bot · 2026-08-15T23:20:47Z

Refinery CI attribution — all 12 failing checks are pre-existing on `main`

Comment-only. This is not an approval and does not release any merge gate.

This PR's base is main @ dbc0012ac. Comparing the failing-check sets directly:

	count
FAILURE on this PR	12
FAILURE on `main` @ `dbc0012ac` (same base)	25
Novel on this PR (failing here, passing on main)	0

Every failing check here is a strict subset of what main is already failing at the
same commit. Per-check:

check	attribution
`Go module vulnerabilities`	pre-existing on main
`Preflight / static checks`	pre-existing on main
`cmd/gc process / shard 1 of 12`	pre-existing on main — see note below
`Integration / packages-core-1-of-4`	pre-existing on main
`Integration / packages-core-2-of-4`	pre-existing on main
`Integration / packages-core-4-of-4`	pre-existing on main
`Integration / packages-cmd-gc-integration`	pre-existing on main
`Integration / rest-smoke-2-of-2`	pre-existing on main
`Check`, `CI / preflight`, `CI / integration`, `CI / required`	aggregators — downstream reporters of the above, not independent diagnoses

cmd/gc process / shard 1 of 12 was verified by TEST NAME, not by shard label.
Shard membership is computed over the package's test list, and this branch adds
cmd/gc tests, so the same label can hold a different test set on each side — a
label-only comparison would have been unsound here. Both sides fail on the identical
single test:

--- FAIL: TestFreshManagedBdCityInitSeedsPinnedHQDatabaseAndKeepsGCPrefix

PR job 95069359410 and main job 95045525975. That test is bd v1.2.1 fallout
already tracked in the ledger (gc-1kqz0, alongside gc-zfh0w / gc-6227x /
gc-14byf), and it cannot be reached by this diff.

Scope of this branch: the diff is confined to cmd/gc (session name-claim +
the new gc session release-name command), internal/session,
internal/productmetrics, docs/reference/cli.md and schemas/. It contains no
change that could affect module vulnerability scanning or the integration tiers.

Refinery gate on the branch head 1637d6679 (green, run fresh and isolated):
go build ./... and go vet on the three affected packages both clean;
internal/session 626 tests ok; internal/productmetrics ok; all six cmd/gc
fast-unit shards ok (8889 tests, no FAIL/panic, no skips in the affected packages).
The branch's own new tests were confirmed to execute rather than pass by absence —
TestReleaseSessionNameClaim_* (5) and TestCmdSessionReleaseName* (3).

One honest limitation: that gate runs the fast-unit tier (GC_FAST_UNIT=1), which
skips the process-tier tests, so it did not independently exercise
cmd/gc process. The attribution for that shard rests entirely on the main-vs-PR
comparison above, not on the local gate.

… CLI lever releases it, so an on_demand agent can never exist again (gc-5fdrr) (#144) * fix(session): release a closed bead's name claim for its own configured identity (gc-5fdrr) A CLOSED session bead kept the runtime identifiers it ran under, and name resolution still consulted them, so an on_demand agent's alias could be reserved forever by a dead record. The live incident (gc-5fdrr) held `shutupandlisten/gc-toolkit.refinery` unspawnable for ~40h: every documented lever refused it — nudge/wake by alias hit "name already exists", nudge/wake/ kill by bead ID hit "session is closed", `prune` does not accept the closed state, and `session new` has no reclaim flag. It was broken only by hand metadata surgery on three undiscoverable keys. Root cause is a divergence between two checks of the same name. The named-session start path clears `EnsureSessionNameAvailableWithConfigForOwner`, whose legacy bypass exists precisely for pre-ga841 phantoms, and then `Manager.CreateSession` re-checks the same name through the cfg-less `ensureSessionNameAvailableForSelfAndOwner`. That inner check recognized a closed configured-named holder only by the boolean flag or the recorded identity — both owner-AGNOSTIC — so a phantom carrying its identity solely as alias/agent_name (ga-n2d Gap C shape) fell through and the inner check vetoed what the outer check had just allowed. The fix reuses `configuredNamedIdentitySignalsMatch`, the owner-SCOPED recognizer already used for this same trap in the startup sweep and the alias-availability check: a CLOSED holder releases the name only to the configured identity its own recorded signals resolve to. A different claimant, an ownerless claimant, and any live holder are all still rejected, so the permanent-identity rule for ad-hoc explicit names is untouched. Two things the automatic release deliberately does not cover — an ad-hoc explicit name, and a holder whose signals name no claimant — still left the operator with no lever, so this also adds `gc session release-name <name-or-alias>`: it clears session_name, alias, and the canonical-identity record from CLOSED holders (agent_name survives as history), refuses all-or-nothing when any holder is live, and no-ops cleanly when nothing holds the name. The ErrSessionNameExists message now says the holder is closed and names that command, so the next operator to hit this reads the remedy off the error instead of diagnosing for 40h. Validation: new unit tests in internal/session cover the release, the other-owner / ownerless / live-holder guards, the error text, and the release-name domain function; cmd/gc tests cover the command end to end. The generated command census, metrics catalog, example schema enum, and CLI reference were regenerated by their own generators; the productmetrics catalog-size baseline moves 199 -> 200 for the new command. On ask 4 of the bead (make gc doctor's `configured-named-conflict` finding say a closed holder needs metadata surgery): that check skips closed beads outright (`cmd/gc/doctor_session_model.go`), so it only ever reports a LIVE conflict, and its premise no longer holds — a closed holder is now either auto-released to its own identity or releasable with one command. The live finding is left as-is: a live conflict's remedy is stopping or reconciling that session, not releasing a name. * test(session): pin the CreateSession layer that vetoed the gc-5fdrr spawn The helper-level tests cover ensureSessionNameAvailableForSelfAndOwner in isolation, but the outage happened one layer up: the named-session start path pre-cleared the name through the cfg-aware check and Manager.CreateSession then re-checked it through the cfg-less helper and refused. A test that never goes through CreateSession would not have caught that divergence. This adds the end-to-end guard — a bead-only CreateSession for a configured named identity over a CLOSED legacy holder whose identity survives only as alias/agent_name. Confirmed to fail without the fix with the exact production error ("already belongs to gc-1 (closed)") and pass with it.

zook-bot added 2 commits August 15, 2026 18:21

github-actions Bot added the status/needs-triage label Aug 15, 2026

johnzook approved these changes Aug 17, 2026

View reviewed changes

zook-bot merged commit f475b68 into main Aug 17, 2026
73 of 85 checks passed

Navigation Menu

Uh oh!

gc session: a CLOSED session bead permanently reserves its alias — no CLI lever releases it, so an on_demand agent can never exist again (gc-5fdrr) - #144

gc session: a CLOSED session bead permanently reserves its alias — no CLI lever releases it, so an on_demand agent can never exist again (gc-5fdrr)#144
zook-bot merged 2 commits into
mainfrom
polecat/gc-5fdrr

zook-bot commented Aug 15, 2026

Uh oh!

zook-bot commented Aug 15, 2026

Uh oh!

zook-bot commented Aug 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zook-bot commented Aug 15, 2026

Summary

Summary

Live incident (2026-08-13 -> 2026-08-15, ~40h outage)

The deadlock: every designed lever is blocked

Impact

Manual workaround (what actually unblocked it)

This is not a one-off — there is a live second instance

Asks (any one closes the outage class; 1 or 2 preferred)

Provenance

Implementation notes

Refinery handoff

Uh oh!

zook-bot commented Aug 15, 2026

Uh oh!

zook-bot commented Aug 15, 2026

Refinery CI attribution — all 12 failing checks are pre-existing on main

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Refinery CI attribution — all 12 failing checks are pre-existing on `main`