Skip to content

fix: restart a wedged OpenCode, guard migrate bootstrap, deprecate in-memory MCP add - #2

Draft
andokivilaid wants to merge 4 commits into
mainfrom
runtime-hardening
Draft

andokivilaid wants to merge 4 commits into
mainfrom
runtime-hardening

Conversation

@andokivilaid

Copy link
Copy Markdown
Owner

Ports the parts of fork work (itsablabla/garza-suna, kageprime/dosco-suna) that apply to the current architecture, rewritten for our code.

Changes

  1. Wedged OpenCode is killed and respawned (apps/kortix-sandbox-agent-server/src/harness/open-code/lifecycle.ts).
    • Before: the liveness probe downgraded state to starting after 3 failures, and the proxy served 503. Nothing killed a wedged child (process alive, HTTP unresponsive). The session stayed broken until a manual refresh.
    • Now: nextWedgeAction sends SIGKILL to the process group when both /session and /kortix-liveness-probe fail for ≥60 s (≥3 probes), with a 5 min cooldown. The existing exit handler respawns the child and finalizes orphaned turns.
    • A probe gap >30 s resets the count (warm-fork restore clock jumps).
    • Only a child that has answered once is watched. Standby candidates are excluded.
    • Kill switch KORTIX_OPENCODE_WEDGE_KILL=0; threshold KORTIX_OPENCODE_WEDGE_KILL_AFTER_MS. Counters in GET /kortix/diag → opencode.wedge_kill.
  2. migrate bootstrap refuses non-local hosts unless --allow-remote is passed (packages/db/scripts/migration-target.ts, migrate.ts). Allowed: 127.0.0.1, localhost, [::1], supabase-db. Runbook updated.
  3. SDK: useAddMcpServer is marked @deprecated. It uses OpenCode POST /mcp, which is in memory only. The replacement is createConnector({ provider: 'mcp' }). JSDoc only; the public surface is unchanged.
  4. Mobile: removed the unused in-memory useAddMcpServer.

Not ported (with reasons)

  • Fork's stuck-session reaper: would abort healthy long tool calls.
  • Idle recycler: boxes auto-stop after 15 min.
  • Pooler-safe timeouts: conflict with a learnings rule on the audit-pool lock_timeout, and prod connects directly.
  • E2B webhook: the fork verifies with HMAC, but E2B signs with a plain SHA-256 hash, so real deliveries would be rejected.

Verification

  • apps/kortix-sandbox-agent-server: bun test src → 1284 pass / 1 fail.
    • The failure is a pre-existing flake in verified-reload-process.e2e.test.ts: 5/15 fails on this branch vs 4/15 with the main lifecycle.
    • New tests: opencode-wedge-kill.test.ts (10 unit tests), and opencode-wedge-kill-process.e2e.test.ts (2 tests; sends SIGSTOP to a real child, asserts kill → new pid → state ok; switch off → no kill).
  • packages/db: migration-target.test.ts 9/9 pass. A real run against a remote host exits 1 with the refusal message.
  • SDK: typecheck clean, 3668 pass / 0 fail, smoke:install passed.

Not verified

  • Wedge kill on a real Linux sandbox. To check: kill -STOP <opencode_pid> from a provider exec, then expect a [opencode] wedge-kill log line within ~70 s and opencode.wedge_kill.kills == 1 in /kortix/diag.
  • Root pnpm test.

🤖 Generated with Claude Code

andokivilaid and others added 4 commits September 24, 2026 11:11
The liveness hysteresis gated an HTTP-unresponsive opencode off (state
`starting`, proxy 503) but never replaced it. Respawn ran only on process
exit, so a child that stayed alive but stopped answering stayed wedged until
a manual /kortix/refresh or a box stop.

- nextWedgeAction (pure): kill when the /session probe AND the
  Instance-independent /kortix-liveness-probe both fail continuously for
  >= 60 s over >= 3 probes, with a 5 min cooldown between kills. A probe gap
  over 30 s (snapshot restore) restarts the run.
- The readiness loop SIGKILLs the child's process group on a kill decision
  and logs `[opencode] wedge-kill`. The existing exit handler respawns it and
  runs onUnplannedRespawn to finalize the orphaned turn.
- Only a supervised child that has already been heard from is watched;
  verification candidates are never watched.
- KORTIX_OPENCODE_WEDGE_KILL=0|false disables it;
  KORTIX_OPENCODE_WEDGE_KILL_AFTER_MS overrides the threshold.
- /kortix/diag reports opencode.wedge_kill {kills, lastKillAt,
  unresponsiveMs, enabled}.
- Tests: decision-function unit tests, and a process test that SIGSTOPs a
  real child and asserts kill, respawn, and the finalize hook.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`local-up` and `preview-up` already refuse the wrong host, but `bootstrap`
installed the non-kortix prerequisites (credit RPCs, an auth.users trigger,
a basejump stub) into any DATABASE_URL it was given.

- assertBootstrapTarget allows loopback and `supabase-db` only; any other
  host fails before a connection unless `--allow-remote` is passed.
- The self-host compose one-shot and the CI schema-bootstrap check use
  `supabase-db` and are unaffected.
- The prod us-east-2 runbook's fresh-project bootstrap now shows the
  explicit `--allow-remote` command.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`useAddMcpServer` calls OpenCode `POST /mcp`, which keeps the server in
process memory only. The registration is lost on the next OpenCode or
sandbox restart. It is public API, so it stays exported (surface snapshots
unchanged); its JSDoc now carries `@deprecated` and points to the persistent
replacement: `createConnector(projectId, { provider: 'mcp', ... })` /
`kortix.project(id).connectors.create`, with `discoverConnectorAuth` for
OAuth detection.

A source test pins the tag, the reason, and that both named replacements
are real root exports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The hook posted to OpenCode `POST /mcp`, which keeps the server in process
memory only, so a server added through it was lost on the next OpenCode or
sandbox restart. Nothing in apps/mobile imports it or AddMcpServerParams.
Persistent MCP servers are project connectors (`createConnector` with
`provider: 'mcp'` in @kortix/sdk).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant