Gateway restart leaves every sandbox in Error (StartFailed); --restart-policy always not honored

User Story

As an operator restarting the local gateway (e.g. brew services restart openshell after a config change or upgrade), I want my sandboxes to come back — automatically per their restart policy, or at least be directly restartable — so that routine gateway maintenance doesn't take down every running workload.

Problem Statement

After a gateway restart, every sandbox transitions to Error:

Ready: False (StartFailed) - Failed to start sandbox during gateway startup:
Docker supervisor exited before becoming ready

No sandbox recovers on its own, including ones created with --restart-policy always (Restart count stays 0).
openshell sandbox start refuses from this state:

Error: × sandbox must be Stopped, Completed, or a failed main-process Error to start
  (current phase: Error); stop the sandbox before retrying start

The only recovery is a per-sandbox stop → start.

Impact / Why This Matters

Any gateway restart or upgrade is a fully destructive operation for all sandboxes, with per-sandbox manual recovery. --restart-policy always silently does not apply to this path, which is surprising given its name.

Acceptance Criteria

After a gateway restart, sandboxes with restart-policy: always return to Ready automatically, or at minimum sandbox start works directly from the post-restart state, or
the documentation clearly states that a gateway restart requires per-sandbox stop/start.

Reproduction Steps

OpenShell v0.1.5, docker compute driver:

openshell sandbox create --name repro-b1 --from <any image>                            # Ready
openshell sandbox create --name repro-b2 --from <any image> --restart-policy always    # Ready
brew services restart openshell        # gateway listener back in ~2s
openshell sandbox list                 # both sandboxes: Error
openshell sandbox get repro-b2
#   Phase: Error
#   Restart count: 0        <- policy=always never engaged

Recovery (works, ~6.5 s per sandbox; workspace and policy revisions fully preserved):

openshell sandbox stop repro-b2 && openshell sandbox start repro-b2   # back to Ready

Mechanism (from logs)

During gateway startup, all sandboxes are restored concurrently. Workload containers exit with code 1, logging:

Error: × sandbox boundary received SIGTERM before supervisor confirmation

The new supervisor's log tail ends with:

boundary terminated: boundary process leaf: sandbox session is terminal

Interpretation: only the same supervisor process can resume a frozen workload (by design), so after the gateway restart the old workload cannot be resumed; it is SIGTERMed, the new supervisor sees a terminal session and gives up with StartFailed. The restart-policy machinery never runs on this path.

Suggested UX (if applicable)

On gateway startup, either re-create the workload for sandboxes whose restart policy allows it, or leave them in a phase from which sandbox start works directly.

Environment

OpenShell v0.1.5 (Homebrew, macOS 26.6.2 Apple Silicon)
Docker compute driver; Docker provider: OrbStack 2.2.4 (Docker Engine 29.8.2). OrbStack is not in the support matrix, so we cannot fully rule out a provider-specific component in "supervisor exited before becoming ready" — but the SIGTERM-before-confirmation behavior looks driver-agnostic. Happy to retest on Docker Desktop if useful.
Reproduced across 4 independent gateway restarts with 2–4 sandboxes; restart policies tested: never (default) and always.

Logs

$ openshell sandbox get repro-b2
  Phase: Error
  Restart count: 0
  Conditions:
    - Ready: False (StartFailed) - Failed to start sandbox during gateway startup: Docker supervisor exited before becoming ready; log tail: … openshell_supervisor: Starting sandbox supervision … boundary terminated: boundary process leaf: sandbox session is terminal

$ docker logs <workload container>
Error: × sandbox boundary received SIGTERM before supervisor confirmation

After filing, we looked into the v0.1.5 source, and part of this behavior is documented design rather than a defect:

The workload dying across a gateway restart follows from the one-supervisor-per-sandbox design ("If the supervisor disconnects, the sandbox freezes the agent. Only the same supervisor process can reconnect and resume it. A new supervisor can't take over a running sandbox, even with valid credentials." — architecture docs).
The driver also anticipates this path: boundary_startup_termination_marker explicitly recognizes "sandbox boundary received SIGTERM before supervisor confirmation" (crates/openshell-driver-podman/src/watcher.rs, mirrored in the docker driver), and the readiness wait has a dedicated branch for "Docker supervisor exited before becoming ready".

We're not contesting the non-resumability itself. The parts we'd still like to see addressed — or documented — are:

--restart-policy always is not consulted on this path. The flag is documented as "Restart behavior after the canonical main process exits". A gateway restart does terminate the main process (SIGTERM before supervisor confirmation), yet Restart count stays 0: the gateway-startup restore reports StartFailed without evaluating the restart policy. If that's intentional, the flag/docs should state that the policy does not cover gateway restarts; if not, the restore path could fall back to the policy before parking the sandbox in Error.
sandbox start is blocked from the post-restart Error phase ("sandbox must be Stopped, Completed, or a failed main-process Error to start … stop the sandbox before retrying start"). Either allowing start directly from this state, or documenting the required stop → start recovery, would remove the surprise for operators.

Happy to reframe this issue to just those two points (or split them) if that helps triage.

Likely two separate failures stacked here, and the policy one is the easier to explain: restart policy is only ever evaluated by the supervisor loop that owns the sandbox, so if that loop dies with the gateway and nothing re-reads the persisted policy on startup, --restart-policy is dead config no matter what you pass. Worth confirming the policy field actually round-trips into the state store — if it isn't persisted, that alone explains "always not honored".

The StartFailed is a different bug, and you need to know which of two shapes you have before touching policy code:

openshell sandbox list -o json
docker ps -a

Run both right after the restart. If the sandbox record persists but the container is gone, the start path is trying to start something that isn't there — the reconciler needs a "runtime object missing → re-create from spec" branch. If the container is still alive and only the gateway process died, it's an adopt bug: the gateway is re-adopting a handle it no longer owns and failing on the stale handle.

I haven't run this on macOS with brew services, so treat the second half as what I'd check first rather than a diagnosis.

Ran both checks on v0.1.5 (macOS, brew services, docker driver).

Persistence: the policy field does round-trip. Same sandbox (--restart-policy always), immediately before and after brew services restart:

$ openshell sandbox get lab-h1 -o json | grep -E 'restart_'
"restart_count": 0
"restart_policy": "always"        # before

(gateway restart; listener back in ~2s; sandbox → Error)

"restart_count": 0
"restart_policy": "always"        # after — still present, still not acted on

So "dead config" doesn't explain this one: the restored record carries the policy; the restore path just never consults it.

Shape: docker ps -a right after the restart:

openshell-default--lab-h1-<uuid>    Exited (1) 1 second ago    # workload: present, killed during restore
openshell-default--lab-h2-<uuid>    Exited (1) 1 second ago
# no supervisor containers remain: the restore-started supervisors exited before becoming healthy and were removed

So it's neither shape exactly: the sandbox record persists, the workload container is present but freshly Exited (1) (the "boundary received SIGTERM before supervisor confirmation" exit), and the supervisor container is gone after failing readiness. sandbox start then refuses purely on the phase gate:

Error: sandbox must be Stopped, Completed, or a failed main-process Error to start
  (current phase: Error); stop the sandbox before retrying start

—the runtime objects a direct start would need (fresh supervisor + restarted workload) are exactly what the forced stop → start cycle creates anyway.

That makes it closest to your adopt/state-machine shape: restore attempts adoption → the new supervisor can't resume the frozen workload (by design) → everything unwinds to Error → and the restart policy is never evaluated even though the field survived the restart in the record.

feat(e2e): add a chaos experiment harness

@addozhang

Hi @addozhang,

I reproduced this on the current main branch (9f1cdb4, Docker driver) with an automated fault-injection experiment based on the chaos harness I am proposing in mpereira#1. The sandboxes survive the gateway outage intact. What kills them is the restarted gateway's recovery path, and on the Docker driver that path cannot succeed by construction. The graceful SIGTERM path on main works (stop, then relaunch). So, likely a bug in the Docker driver's startup recovery, not the fence working as intended.

How to reproduce

About 2 minutes after the first build, local Docker only:

git clone -b feat/chaos-experiment-harness/mpereira https://github.com/mpereira/OpenShell
cd OpenShell
mise run e2e:chaos e2e/chaos/experiments/gateway-kill-keeps-state.yaml

The branch is main 9f1cdb4 plus the harness (nothing under crates/ changes). The command builds the gateway, runs the experiment and prints the run directory, which holds report.html, the gateway and sandbox logs, docker events and the journals. The experiment lists this issue under expectations:, so the verdict reads XFAIL (an expected failure) today and XPASS (an unexpected pass) once the behaviour changes. The graceful counterpart is e2e/chaos/experiments/gateway-graceful-restart.yaml.

Result

Three sandboxes, each appending one line per second to a journal. The gateway gets SIGKILL 20 s in, stays down 15 s, and is started again. Times are seconds from run start for counter-2, the first sandbox the startup sweep reached; counter-1 and counter-0 go through the same steps 12 s and 24 s later, because the sweep is serial.

event	seconds from run start
Gateway SIGKILL	22.6
Last journal tick (tick 38, one per second since start)	39.7
Old supervisor container force-removed by the restarted gateway	40.4
Sandbox: "connection lost; workload frozen pending authenticated recovery"	40.5
Gateway listener bound again	40.8
Replacement supervisor attach denied	41.3
Sandbox container SIGTERM, then SIGKILL 10 s later	41.4 / 51.4
Marked `Error` / `StartFailed`	51.4

The journals hold ticks 1 to 38 at one-second spacing: the 18 s without a gateway cost nothing, and the ticks stop the second the restarted gateway removes the supervisor containers. A sandbox created after the sweep ran to exit 0.

gantt
  title gateway-kill-keeps-state · 20261029T120842Z-3b2c · XFAIL
  dateFormat mm:ss
  axisFormat %M:%S
  todayMarker off
  section flow
    counter :00:01, 01:18
    wait counter started :00:01, 00:03
    delay 20s :00:03, 00:23
    fault gateway-down (outage) :crit, 00:23, 00:41
    fresh-sandbox :01:18, 01:19
  section counter-0
    Created :milestone, 00:03, 00:03
    Running :active, 00:03, 00:24
    Unknown :crit, 00:24, 00:42
    Running :active, 00:42, 01:07
    Pending :01:07, 01:18
    Failed :milestone, crit, 01:18, 01:18
  section counter-1
    Created :milestone, 00:03, 00:03
    Running :active, 00:03, 00:24
    Unknown :crit, 00:24, 00:42
    Running :active, 00:42, 00:55
    Pending :00:55, 01:05
    Failed :milestone, crit, 01:05, 01:05
  section counter-2
    Created :milestone, 00:03, 00:03
    Running :active, 00:03, 00:24
    Unknown :crit, 00:24, 00:42
    Pending :00:42, 00:53
    Failed :milestone, crit, 00:53, 00:53
  section fresh-sandbox-0
    Created :milestone, 01:19, 01:19
    Succeeded :milestone, done, 01:19, 01:19

(Unknown is the harness getting no answer while the gateway is down; the Running after it is the store still saying Ready until the sweep reaches that sandbox.)

Trimmed log lines for counter-2, from the gateway, the sandbox and docker events in that order:

12:09:22.849 INFO  driver.initialize: Compute driver connected configured_driver=docker
12:09:23.156 INFO  ComputeDriver/StartSandbox (counter-2)
12:09:23.661 WARN  Docker supervisor exited before becoming ready supervisor_logs="... Error: × attachment denied: boundary process leaf: sandbox runtime is already bound to another supervisor process"
12:09:33.764 WARN  Failed to start sandbox during gateway startup error="Docker supervisor exited before becoming ready; log tail: ..."
12:09:33.773 INFO  ComputeDriver/StartSandbox (counter-1)

12:09:22.811 WARN  Sandbox Protocol connection lost; workload frozen pending authenticated recovery
12:09:23.581 WARN  Supervisor takeover denied [reason:sandbox runtime is already bound to another supervisor process]
12:09:23.581 OCSF  CONFIG:TAKEOVER_DENIED [MED] Supervisor takeover denied

12:09:22.670 container kill   role=supervisor signal=9      (old supervisor, by the restarted gateway's init)
12:09:23.335 container start  role=supervisor               (replacement)
12:09:23.636 container die    role=supervisor exitCode=1
12:09:23.678 container kill   role=sandbox    signal=15
12:09:33.720 container kill   role=sandbox    signal=9
12:09:33.722 container die    role=sandbox    exitCode=137

Mechanism

The gateway dies. The sandbox container, its dedicated supervisor container and the workload all keep running.
The restarted gateway's Docker driver init, reconcile_runtime_resources_at_startup, force-removes every managed supervisor, staging and identity container (L2135-L2166). That freezes the workload.
The startup sweep, start_persisted_sandboxes_with_authentication, calls StartSandbox with credentials from mint_persisted_authentication: same auth epoch, no takeover grant. Only the Kubernetes shared supervisor pool mints grants (supervisor_pool.rs#L2784).
start_sandbox_with_lifecycle_fence sees a running container plus credentials and adopts (L2562-L2572), launching a new supervisor container, which is a new supervisor instance.
The sandbox's plan_attach admits only the same instance or a grant with a higher epoch, so it answers WrongSupervisorInstance. That is the documented fence (docs/about/architecture.mdx: a new supervisor can't take over a running sandbox, even with valid credentials).
The driver reports "Docker supervisor exited before becoming ready", stops the sandbox container (SIGTERM, 10 s, SIGKILL), and the gateway marks StartFailed (mod.rs#L4313-L4320). restart_policy is only read in should_restart_main_process. The sweep is serial, about 11 s per sandbox, with the listener already bound, so sandbox list says Ready for frozen sandboxes until the sweep reaches them.

So the adopt path removes the only supervisor the sandbox would accept, offers one it will refuse, and has no grant to offer instead. I found nothing saying it is meant to work this way, and the two paths that do work both stop and relaunch:

Graceful restart: stop_persisted_sandboxes_on_shutdown stops every running-intent sandbox on SIGTERM and the sweep relaunches it (pinned by the e2e test docker_gateway_restart_preserves_running_and_stopped_intent). My gateway-graceful-restart experiment passes on the same build: both sandboxes Ready 6 s after SIGTERM, each script started a second time, exit 0.
VM driver: driver.rs restarts the VM on a non-empty bundle during startup recovery, "rather than leaving the old supervisor connected with invalid credentials."

The fix looks scoped: a running container plus credentials at Docker driver startup should mean stop and start fresh, not adopt. A crash, OOM kill or kill -9 of the gateway would then end like a graceful restart. docs/how-it-works/sandboxes/overview.mdx ("an existing sandbox can return to Provisioning temporarily while its supervisor reconnects") should then say the sandbox is restarted.

Your v0.1.5 report fits the same picture, with one difference. "sandbox session is terminal" is what a v0.1.5 sandbox sets 30 s after losing its supervisor (AUTHENTICATED_RECONNECT_TIMEOUT), so your containers ran without one for over 30 s, which means the shutdown sweep did not finish. launchd's default SIGKILL 20 s after SIGTERM, against a 10 s docker stop per sandbox, would explain it; I have not verified that. The "Stopped sandbox during gateway shutdown" lines in that gateway log would settle it. The Exited (1) containers are the driver's post-failure stop.

Not checked

I only ran this against the Docker driver, on macOS. I did not run v0.1.5, Podman, the VM driver or Kubernetes, and I did not verify the launchd timeout on the Homebrew service. The graceful run's scripts exit 0 on their second attempt, so it says nothing about restart_policy after the relaunch.

Two questions for OpenShell maintainers:

Should the Docker driver stop and start here, like the VM driver, instead of adopting?
Is resuming the workload's process tree across a gateway restart a goal for Docker at all? It would need a takeover grant on this path and contradicts the architecture page.

We see the same end state on Linux after a gateway crash (graceful restarts are fine for us), OpenShell v0.1.5, Docker driver, gateway as a systemd service on Ubuntu 24.04.

With 3 sandboxes running, systemctl kill -s KILL openshell-gateway (systemd restarts it).
For a few seconds the workloads keep running and ExecSandbox fails with "sandbox is owned by gateway replica … which advertises no peer endpoint; set OPENSHELL_PEER_ENDPOINT…".
The new gateway then starts each sandbox again and every one fails: "Docker supervisor exited before becoming ready … attachment denied: boundary process leaf: sandbox runtime is already bound to another supervisor process".
All sandboxes stay Error/StartFailed; we polled for 6 minutes with no recovery, and workload traffic stops. A per-sandbox StopSandbox + StartSandbox recovers each one, as described above.

A graceful systemctl restart behaves differently for us: the shutdown sweep stops every sandbox and the new gateway starts them again.

Navigation Menu

Repository navigation

Gateway restart leaves every sandbox in Error (StartFailed); --restart-policy always not honored #4910

User Story

Problem Statement

Impact / Why This Matters

Acceptance Criteria

Reproduction Steps

Mechanism (from logs)

Suggested UX (if applicable)

Environment

Logs

How to reproduce

Result

Mechanism

Not checked

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gateway restart leaves every sandbox in Error (StartFailed); --restart-policy always not honored #4910

Description

User Story

Problem Statement

Impact / Why This Matters

Acceptance Criteria

Reproduction Steps

Mechanism (from logs)

Suggested UX (if applicable)

Environment

Logs

Activity

addozhang commented on Oct 25, 2026

O96a commented on Oct 25, 2026

addozhang commented on Oct 26, 2026

mpereira commented on Oct 29, 2026

How to reproduce

Result

Mechanism

Not checked

thenashef commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions