feat(service): recover active work and reap restarted gateways

[hannesrudolph]

Summary

• make ocm service restart <env> restart immediately through OpenClaw's protocol-v1 recovery handoff so eligible interrupted work resumes after startup
• detach the restart helper so a restart requested from inside the target gateway does not deadlock on its own turn
• preserve the existing direct-supervisor restart for legacy and wrapper-backed gateways, with a warning instead of a new --force requirement
• prevent supervisor reconciliation from reloading sibling gateways when command callers inherit different ambient environments or select equivalent direct-versus-managed-Node launch shapes
• clean the full supervised process group after child exit and always wait on the process-group leader so a stopped gateway cannot remain as a zombie
• retain --force as an explicit emergency bypass when a gateway advertises recovery support but cannot accept the handoff

Why

The previous service restart path asked the supervisor to replace the child directly. An active agent turn could therefore be terminated mid-response and would only continue after another user message.

OpenClaw restart-handoff protocol v1 can record eligible interrupted work before an externally supervised restart and automatically continue it after startup. OCM now uses that negotiated path by default without delaying gateway development until active work finishes.

Supervisor state is also rebuilt by lifecycle commands outside the target gateway. Caller-only differences such as PATH, package-manager environment, proxy/CA settings, or choosing the same installed OpenClaw entrypoint directly versus through OCM's managed Node fallback must not be interpreted as gateway configuration changes. OCM now preserves the persisted child specification for those equivalent cases while still applying meaningful binding, runtime, port, argument, path, and OpenClaw environment changes.

Finally, a gateway child could exit after try_wait() returned None but before the process-group existence probe. The stop path then returned without a final wait(), leaving the process-group leader as a zombie. OpenClaw's single-instance check could treat that defunct PID as the active gateway owner even though no process was listening, causing an endless supervised startup loop. All stop paths now converge on an unconditional wait for the leader after process-group cleanup.

Existing users with older or wrapper-backed bindings keep the restart behavior they already have. They receive a warning that in-flight work may have been interrupted, but they do not need to add --force merely to preserve compatibility.

Behavior

• Protocol v1: request an immediate recovery-aware restart and observe the replacement child.
• Active eligible work: mark it for recovery, replace the gateway process, and continue it after startup.
• Self-restart: schedule the detached helper and return, avoiding a deadlock on the requesting turn.
• Missing protocol v1: use the existing direct-supervisor restart automatically and warn that recovery was unavailable.
• Advertised but unhealthy handoff: fail with actionable guidance rather than silently claiming recovery succeeded; --force remains available as the explicit emergency bypass.
• Ambient caller drift: preserve existing gateway child specifications, including equivalent direct openclaw.mjs and managed-Node launch forms.
• Meaningful spec changes: reload only affected children and log changed field names without exposing environment values.
• Child exit/stop: terminate the supervised process group, escalate to KILL when necessary, and always reap the leader before returning.
• --force: directly replace the supervised child and report that the OpenClaw recovery handoff was bypassed.

Upgrade transactions continue to request the existing synchronous forced restart explicitly in this PR. Recovery-aware upgrade orchestration remains a separate follow-up.

Validation

• cargo fmt --check
• git diff --check
• cargo test --test service_command_tests — 36 passed
• cargo test --test daemon_runtime_tests — 20 passed
• focused process-group leader reaping regression passed
• full test suite passed
• cargo clippy --all-targets completed with the same three baseline allowances

Live integration

Recovery-aware restart was tested with a disposable OCM environment bound to OpenClaw 2026.7.2-beta.3 during a real 30-second tool turn:

• the gateway process was replaced in 10 seconds instead of waiting for the turn to finish
• OpenClaw recorded the interrupted session and injected its built-in continuation prompt after startup
• the recovered run completed with OCM_IMMEDIATE_RECOVERY_OK
• the disposable environment was destroyed after verification

Supervisor reconciliation was tested against the live mixed-version setup:

• state was rebuilt across different ambient PATH values and both direct and managed-Node launch resolution
• the shared daemon PID and all five gateway PIDs remained unchanged after observation
• the persisted managed-Node launch specification was retained instead of cycling the fleet

The child-reaping repair was then exercised against an intentionally failing temporary gateway:

• supervised child PID 58301 exited with code 1
• OCM recorded the expected retry and the child disappeared completely
• no zombie, stale port lock, or blocked OpenClaw single-instance owner remained
• the temporary gateway was stopped again after the test
• all five production gateway routes returned HTTP 200 after final verification