Queued nudges to always/fresh sessions fail on session fence mismatch — fix delivery + expose delivered/dropped (impl of lx-9foy) (gc-1kz4) - #105
Conversation
…ng the nudge (gc-1kz4) A queued nudge carries a fence -- the target's session id and continuation_epoch at enqueue time -- and delivery re-checks it. For a session configured mode=always/wake_mode=fresh that fence could not be satisfied: shouldBumpContinuationEpoch bumps continuation_epoch on EVERY wake after the first while the session bead id stays put, so any nudge that outlived one recycle met a moved epoch. An epoch mismatch is not a retry -- failedQueuedNudge special-cases errNudgeSessionFenceMismatch and dead-letters on the FIRST attempt, bypassing attempts and TTL. The nudge was destroyed, not deferred. Measured: of boot's five nudges to one such agent over 14 days, four never landed. lx-9foy marked that mechanism INFERRED; this confirms it against the code and sharpens it in two ways. It is the EPOCH half of the fence that fires, never the session-id half -- an item stamped for a different session is not claimed at all (it stays pending), so the failure is specific to same-session conversation recycling. And all three delivery paths -- hook drain, legacy poller, supervisor dispatcher -- share the one gate, so the fix lands once. An epoch drift that provably names the same session is now retargeted onto the live conversation. Two cases keep the strict fence: a wait-sourced nudge, whose stamped epoch is the wait's own registered_epoch (a real conversation-scoped guarantee -- the wait state machine independently cancels a stale-epoch wait as continuation-stale, and retargeting would contradict it), and an item naming no session, where identity cannot be proven at all. The wait source is now a named constant since it is load-bearing at two delivery gates. Second half of the bead: callers could not tell a delivered nudge from a dropped one, so silence was unreadable -- which matters because the shutdown dance kills a target that never answers N nudges, and against these sessions the nudge itself was what failed. `gc session nudge` now returns the queued nudge's id (nudge_id), and `gc nudge show <nudge-id>` resolves it to exactly one of pending / in_flight / delivered / dropped (with reason) / unknown. It reads the flock'd queue first and the durable shadow bead second -- the authority order, since a delivered item leaves the queue entirely and only the bead remembers. Reporting, not judging: a dropped outcome still exits 0; exit 1 means the id is unknown. The delivered-vs-dropped classification lives in nudgequeue.OutcomeForState, beside the state codes it reads, so a new state cannot silently read as delivered at a call site that never learned about it. Ratchets a new visible subcommand trips, all four required: the product-metrics command census (manifest entry + regenerated tables), the generated CLI reference, the packlint valid-subcommand allowlist, and the per-command JSON result schema -- without that last one the CLI contract gate rejects --json with json_unsupported before the command runs, which a unit test calling the renderer directly cannot see. A binary-level probe caught it; TestNudgeShowDeclaresJSONSupport now pins it. Validation: TDD throughout -- the five new fence tests failed for the right reason first, including an end-to-end delivery across a simulated fresh-wake epoch bump. One pre-existing test changed deliberately (TestTryDeliverQueuedNudgesByPollerDeliversDespiteStaleFenceBeadMarkFailure): its subject is bookkeeping resilience, so its stale item moved to the wait source that still fences. Full ./cmd/gc, ./internal/nudgequeue, ./internal/session, ./internal/productmetrics and ./test/packlint pass; go vet ./... clean; both `gc nudge show` arms verified against a throwaway city with the branch binary. internal/productmetrics' TestOpenProductionAndPreparation... fails identically on the pristine base commit under this host's GC_DISABLE_USAGE_METRICS=1 -- pre-existing and environmental, proven by a stashed-tree run. Follow-up left to the operator: the live positive control the bead asks for -- queue a nudge to the real deacon and watch it cross a recycle boundary -- needs this binary deployed to the city, so it belongs after the merge, not to the polecat. Items 3 and 4 of lx-9foy (re-arm boot-health escalation, audit mol-shutdown-dance) stay gated on lx-9foy as specified.
|
Codex signoff (pre-open, comment-only — not an approval): VERDICT: COMMENT Scope checked: Read the full diff for the queued nudge delivery/fence changes, Findings: No P0/P1 findings. P2: P2: Verification: |
2 participants
Summary
Queued nudges to mode=always/wake_mode=fresh named sessions mostly fail on
session fence mismatch. Fix delivery, and give callers a way to tell "delivered"
from "dropped".
Implementation bead for HQ tracker lx-9foy -- read lx-9foy for the full
evidence and the downstream consequences (cross-store, so no dep link is
possible; close this and the mayor closes lx-9foy).
Scope of THIS bead: lx-9foy items 1 and 2 only
retarget to the current fence rather than terminalizing, or deliver
synchronously when the target is live.
before they draw any conclusion from silence.
Items 3 (re-arm boot-health escalation) and 4 (audit mol-shutdown-dance) stay
on lx-9foy and are deliberately gated behind these two. Do NOT do them here.
Where it lives
Fork-local fix in this repo. Do not open an upstream PR.
Evidence
Of boot's five nudges to gc-toolkit.deacon in the 14 days to 2026-08-08, four
never landed. Verified directly on wisp lx-wisp-n79jc:
Also lx-wisp-f5top and lx-wisp-fn3sz (both 2026-08-02). lx-wisp-3d2v6 did land,
via provider injection. lx-wisp-myfeq queued and died.
Confirm the mechanism before changing queue semantics
lx-9foy's mechanism is explicitly marked INFERRED: the nudge is queued carrying
a session fence / continuation_epoch; these sessions recycle constantly (the
deacon cycles every ~4.6 min), so between queue and deliver_after the session is
torn down and re-created fresh, the fence no longer matches, and the queue
terminalizes delivery instead of retargeting it.
That is consistent with every observation above but has not been read out of the
code. Confirm it against cmd_nudge.go FIRST and write down what you actually
find. If the real mechanism differs, fix what is there and correct lx-9foy --
do not implement against the inference.
Affects every always/fresh named session: deacon, mayor, mechanik, boot, and
four witnesses.
Why item 2 matters as much as item 1
Silence from a nudged agent currently means either "healthy but didn't answer"
or "the nudge never arrived", and callers cannot tell which. mol-shutdown-dance
establishes liveness by nudging three times and killing the target if it never
replies ALIVE -- so against these sessions the nudge is what fails, the pardon
path cannot fire, and the dance would proceed to
gc session killon a healthyagent. Any caller that reasons from silence needs a delivery signal it can
trust. Make the failure legible at the call site, not just in the wisp.
The interrogation ladder is supposed to be the false-positive guard that makes
mechanical detection safe. Against these targets it is not one.
Verify
A positive control, not an absence: queue a nudge to a live always/fresh session
(the deacon is one), let it cross at least one recycle boundary, and confirm it
is DELIVERED -- not merely that no error was logged. Then confirm the
delivered/dropped signal from item 2 reports correctly in both directions,
including a genuinely undeliverable target.
Implementation notes
Mechanism CONFIRMED against the code (2026-08-09, polecat gc-toolkit.furiosa)
lx-9foy's INFERRED mechanism is correct. Read out of the code, with two
corrections/refinements that matter for the fix:
queuedNudgeOptionsFromTarget (cmd_nudge.go:554) -> {SessionID,
ContinuationEpoch} -> newQueuedNudgeWithOptions (:1703).
shouldBumpContinuationEpoch (session_wake.go:125) returns true when
WakeMode == "fresh" && LastWokeAt != ""; persisted by PreWakePatch(internal/session/lifecycle_transition.go:193). The session BEAD ID is
unchanged across a fresh wake -- only the epoch moves.
(nudge_dispatcher.go:167 resolveNudgeTargetFromSessionInfo), so
target.continuationEpoch is always current.
(:1748) compares SessionID but NOT the epoch (it only requires the
target's epoch to be non-empty).
requires exact epoch equality -> splitQueuedNudgesForTarget rejects ->
recordQueuedNudgeFailureWithStore(..., errNudgeSessionFenceMismatch).
FIRST attempt -- immediate dead-letter, bypassing attempts/TTL/retry.
Shadow bead closes state=failed, last_error="queued nudge session fence
mismatch", close_reason="nudge failed: queue terminalization rejected
delivery" (internal/nudgequeue/store.go:448). Matches the wisp evidence.
CORRECTION 1 (matters): it is the EPOCH half of the fence that fires, not the
session-id half. A nudge stamped for a DIFFERENT session id is never claimed
at all -- it stays pending (TestClaimDueQueuedNudgesForTargetLeavesSiblingFence
Pending). So the failure is specific to same-session conversation recycling,
which is exactly what mode=always/wake_mode=fresh does on every cycle.
CORRECTION 2 (matters): the drop is unconditional and immediate, not a
retry-exhaustion. Any nudge not delivered before the next recycle is
GUARANTEED destroyed. All three delivery paths share the gate: hook drain
(:484), legacy poller (:1353 via tryDeliverQueuedNudgesByPoller), and the
supervisor dispatcher (nudge_dispatcher.go:197 -> same helper).
WHY THE FENCE EXISTS (do not remove it wholesale): it arrived with durable
session waits (204c85b). For a
wait-sourced nudge the stamped epoch is thewait's own registered_epoch (cmd_wait.go:1195), i.e. a real conversation-scoped
guarantee -- and the wait state machine independently cancels a stale-epoch
wait as "continuation-stale" (cmd_wait.go:1040) whose queued nudge is then
withdrawn as wait-canceled (blockedQueuedNudgeReason :1575). Retargeting a
wait nudge would contradict that. For every other source the stamped epoch is
just "whichever conversation happened to be live when the sender typed the
command" -- the sender addressed the AGENT, not a conversation instance -- so
the fence guarantees nothing and only destroys the message.
FIX (item 1): same-session epoch drift retargets to the live epoch instead of
dead-lettering, EXCEPT for wait-sourced items, which keep the strict fence.
Cross-session and unprovable-identity (no session id on the item) stay strict.
--- SELF-REVIEW (step gc-yw74, polecat gc-toolkit.furiosa) ---
Branch polecat/gc-1kz4 @ 2c23aa7, rebased on live origin/main (3e629ad);
merge-base == origin/main, working tree clean, 1 commit, 18 files.
Gates run (rig configures no test/lint/build command, so scoped to the diff):
go build ./... clean
go vet ./cmd/gc/... ./internal/nudgequeue/...
./internal/productmetrics/... ./test/packlint/... clean
go test ./cmd/gc/ PASS (519.9s)
go test ./internal/nudgequeue/... PASS
go test ./test/packlint/... PASS
go test ./internal/productmetrics/... 1 FAIL, pre-existing
INVERTED-FIX CHECK (the fix is not inert): reverted the one behavioral line
(queuedNudgeMatchesTargetFence returning queuedNudgeEpochRetargetable(item) ->
false) in a throwaway worktree at the branch head. All four fence tests fail,
including the end-to-end TestTryDeliverQueuedNudgesByPollerDeliversAcrossFresh
WakeEpochBump ("delivered = false, want the pre-recycle nudge retargeted").
The tests genuinely pin the behavior change.
PRE-EXISTING FAILURE, filed as gc-1mtj (not caused by this diff):
internal/productmetrics TestOpenProductionAndPreparationAreLazyAndNonCreating
fails identically on a pristine origin/main worktree (3e629ad) with no local
changes -- this host exports GC_DISABLE_USAGE_METRICS=1 and the test reads it
instead of neutralizing it. Verified side-by-side, not inferred from the
earlier commit message.
Ratchets confirmed present: command census (manifest + metrics_census_gen.go +
command_ids_gen.go + the 192->193 catalog assertion), generated CLI reference,
packlint valid-subcommand allowlist, and schemas/nudge/show/result.schema.json.
--- HANDOFF TO REFINERY (step gc-hnba, polecat gc-toolkit.furiosa, 2026-08-09) ---
Implemented: queued-nudge session fence no longer dead-letters a same-session
epoch drift — it retargets onto the live conversation, so nudges survive an
always/wake_mode=fresh recycle. Strict fence kept for wait-sourced items (the
stamped epoch is the wait's own registered_epoch) and for items naming no
session (identity unprovable). Second half:
gc session nudgenow returnsnudge_id and
gc nudge show <id>resolves pending/in_flight/delivered/dropped(with reason)/unknown, queue first then durable shadow bead; classification
lives in nudgequeue.OutcomeForState. Design doc filed at
engdocs/design/queued-nudge-session-fence.md.
Branch polecat/gc-1kz4 @ 2c23aa7, pushed and verified on origin
(ls-remote == local HEAD). Rebased on origin/main 3e629ad: 1 commit ahead,
0 behind at handoff. 18 files, +1048/-18. Local branch deleted after the
verified push; origin ref is the record.
Gates (recorded in full in the SELF-REVIEW section above): go build ./...
clean, go vet on touched packages clean, go test ./cmd/gc/ PASS,
./internal/nudgequeue/... PASS, ./test/packlint/... PASS. One failure in
./internal/productmetrics is PRE-EXISTING — reproduced identically on a
pristine origin/main worktree and filed as gc-1mtj; it is not caused by this
diff. Inverted-fix check performed: reverting the one behavioral line fails all
four fence tests, so the tests genuinely pin the change.
Refinery handoff
gc-1kz4(bug, P2)polecat/gc-1kz4main2c23aa71; PR opened codex-green.