marvel stop --teardown prints "agents torn down", but at least once it left a workspace tmux session (and its windows) alive afterward. When that happens the teardown contract is violated: marvel-managed panes, and the agent processes in them, can outlive a teardown that reported success. This surfaced during a live demo run, so it is worth recording even though it is intermittent.
Expected: after stop --teardown returns, no marvel-<workspace> tmux session survives and no marvel-managed agent pane is left running.
Actual (observed once): after Act 1d of the demo (workspace health, team ward, three roles with restart_policy always/never/capped, none ever heartbeating, daemon started with MARVEL_SHIFT_TIMEOUT=15s, a stuck shift that timed out and rolled back to gen 1), a later stop --teardown printed "agents torn down" but left the marvel-health session alive with 6 windows, which had to be killed by hand.
Intermittent: not reproduced deterministically. A controlled A/B afterward was clean in every case:
- no shift (workspace
recover): clean.
- successful shift (
recover/line, process-alive healthcheck): clean.
- stuck-shift rollback (
health/ward, never-heartbeat, 10 to 15s timeout), three iterations with an 8s post-rollback churn window: clean every time. Window counts: before_shift=4, after_rollback=4 (the rollback did not leak a window), after_churn=5 (a normal restart_policy=always respawn), and teardown reaped all of it.
The one difference in the failing case: a longer wall-clock gap elapsed between the shift timeout and the teardown, during which the never-heartbeating always-restart roles kept respawning, so more churn had accumulated before teardown ran.
Hypothesis (hedged): stop --teardown reaps a snapshot of tracked panes while the reconcile loop is still spawning replacements for restart_policy=always roles, so panes and windows created inside the teardown race window, plus the workspace tmux session that contains them, can be orphaned. A rolled-back stuck shift may widen the window with gen-2 remnants.
Suggested deterministic repro to try: leave a never-heartbeat always-restart team running 60s or more after a stuck-shift rollback (or drive stop --teardown concurrently with an in-flight respawn), then teardown, and assert no marvel-<workspace> tmux session survives.
Could teardown drain and quiesce the reconcile loop before it snapshots panes, and then sweep any remaining marvel-<workspace> sessions by name as a backstop?
Relevant paths: internal/daemon (stop/teardown), internal/team/controller.go (reconcile + the shift-timeout rollback path), internal/session/manager.go and internal/tmux/driver.go (pane and session teardown).
Tracked internally as aae-orc-b6d3.
marvel stop --teardownprints "agents torn down", but at least once it left a workspace tmux session (and its windows) alive afterward. When that happens the teardown contract is violated: marvel-managed panes, and the agent processes in them, can outlive a teardown that reported success. This surfaced during a live demo run, so it is worth recording even though it is intermittent.Expected: after
stop --teardownreturns, nomarvel-<workspace>tmux session survives and no marvel-managed agent pane is left running.Actual (observed once): after Act 1d of the demo (workspace
health, teamward, three roles withrestart_policyalways/never/capped, none ever heartbeating, daemon started withMARVEL_SHIFT_TIMEOUT=15s, a stuck shift that timed out and rolled back to gen 1), a laterstop --teardownprinted "agents torn down" but left themarvel-healthsession alive with 6 windows, which had to be killed by hand.Intermittent: not reproduced deterministically. A controlled A/B afterward was clean in every case:
recover): clean.recover/line, process-alive healthcheck): clean.health/ward, never-heartbeat, 10 to 15s timeout), three iterations with an 8s post-rollback churn window: clean every time. Window counts: before_shift=4, after_rollback=4 (the rollback did not leak a window), after_churn=5 (a normalrestart_policy=alwaysrespawn), and teardown reaped all of it.The one difference in the failing case: a longer wall-clock gap elapsed between the shift timeout and the teardown, during which the never-heartbeating always-restart roles kept respawning, so more churn had accumulated before teardown ran.
Hypothesis (hedged):
stop --teardownreaps a snapshot of tracked panes while the reconcile loop is still spawning replacements forrestart_policy=alwaysroles, so panes and windows created inside the teardown race window, plus the workspace tmux session that contains them, can be orphaned. A rolled-back stuck shift may widen the window with gen-2 remnants.Suggested deterministic repro to try: leave a never-heartbeat always-restart team running 60s or more after a stuck-shift rollback (or drive
stop --teardownconcurrently with an in-flight respawn), then teardown, and assert nomarvel-<workspace>tmux session survives.Could teardown drain and quiesce the reconcile loop before it snapshots panes, and then sweep any remaining
marvel-<workspace>sessions by name as a backstop?Relevant paths:
internal/daemon(stop/teardown),internal/team/controller.go(reconcile + the shift-timeout rollback path),internal/session/manager.goandinternal/tmux/driver.go(pane and session teardown).Tracked internally as aae-orc-b6d3.