Skip to content

stop --teardown intermittently leaves an orphaned workspace tmux session #92

Description

@arcaven

marvel stop --teardown prints "agents torn down", but at least once it left a workspace tmux session (and its windows) alive afterward. When that happens the teardown contract is violated: marvel-managed panes, and the agent processes in them, can outlive a teardown that reported success. This surfaced during a live demo run, so it is worth recording even though it is intermittent.

Expected: after stop --teardown returns, no marvel-<workspace> tmux session survives and no marvel-managed agent pane is left running.

Actual (observed once): after Act 1d of the demo (workspace health, team ward, three roles with restart_policy always/never/capped, none ever heartbeating, daemon started with MARVEL_SHIFT_TIMEOUT=15s, a stuck shift that timed out and rolled back to gen 1), a later stop --teardown printed "agents torn down" but left the marvel-health session alive with 6 windows, which had to be killed by hand.

Intermittent: not reproduced deterministically. A controlled A/B afterward was clean in every case:

  • no shift (workspace recover): clean.
  • successful shift (recover/line, process-alive healthcheck): clean.
  • stuck-shift rollback (health/ward, never-heartbeat, 10 to 15s timeout), three iterations with an 8s post-rollback churn window: clean every time. Window counts: before_shift=4, after_rollback=4 (the rollback did not leak a window), after_churn=5 (a normal restart_policy=always respawn), and teardown reaped all of it.

The one difference in the failing case: a longer wall-clock gap elapsed between the shift timeout and the teardown, during which the never-heartbeating always-restart roles kept respawning, so more churn had accumulated before teardown ran.

Hypothesis (hedged): stop --teardown reaps a snapshot of tracked panes while the reconcile loop is still spawning replacements for restart_policy=always roles, so panes and windows created inside the teardown race window, plus the workspace tmux session that contains them, can be orphaned. A rolled-back stuck shift may widen the window with gen-2 remnants.

Suggested deterministic repro to try: leave a never-heartbeat always-restart team running 60s or more after a stuck-shift rollback (or drive stop --teardown concurrently with an in-flight respawn), then teardown, and assert no marvel-<workspace> tmux session survives.

Could teardown drain and quiesce the reconcile loop before it snapshots panes, and then sweep any remaining marvel-<workspace> sessions by name as a backstop?

Relevant paths: internal/daemon (stop/teardown), internal/team/controller.go (reconcile + the shift-timeout rollback path), internal/session/manager.go and internal/tmux/driver.go (pane and session teardown).

Tracked internally as aae-orc-b6d3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions