Skip to content

fix: persist runtime-owned settled-worker resume fences #19344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Jinwoo-H wants to merge 7 commits into from
Closed

fix: persist runtime-owned settled-worker resume fences #19344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0a4916
fix(renderer): make the sleeping record the settled-worker fence's on…
Jinwoo-H Sep 7, 2026
1045909
fix(orchestration): publish the resume fence only from main's committ…
Jinwoo-H Sep 7, 2026
4203db2
refactor(orchestration): give the settled-worker fence one owner and …
Jinwoo-H Sep 7, 2026
57d0ef6
fix: close settled-worker fence publication and hydration gaps
Jinwoo-H Sep 7, 2026
6720908
fix: skip untouched resume-fence host partitions
Jinwoo-H Sep 7, 2026
7f36880
fix: hydrate worker records and resume fences atomically
Jinwoo-H Sep 7, 2026
2197d23
fix: supersede runtime fences only with supplied authority
Jinwoo-H Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions config/reliability-gates.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,89 @@
}
},
"gates": [
{
"id": "terminal-recovery.settled-worker-resume-fence",
"title": "Runtime-owned settled-worker resume fences survive publication and hydration",
"maturity": "experimental",
"protection": "partial",
"owner": "terminal-runtime",
"layer": "store-runtime-unit-and-electron-cdp",
"surfaces": ["workspace session hydration", "automatic agent resume"],
"platforms": ["macos", "linux", "windows"],
"providers": ["local", "daemon", "ssh", "remote-runtime"],
"coveredPlatforms": ["macos"],
"coveredProviders": ["local", "daemon"],
"coverageNotes": "Deterministic local/SSH partition and mixed-source tests; hidden desktop restart/reload and retirement E2Es. No live independently versioned paired-server or physical Windows/Linux claim.",
"motivatingLinks": ["https://github.com/stablyai/orca/pull/19344"],
"invariant": "A runtime-authored settled-worker fence survives old writers, absent old-source authority, overlapping hydration, and sibling host write failure; authoritative scoped retirement preserves sibling fences.",
"oracle": "Nine round-2 reviewer probes plus atomic pending-hydration and publication/read-boundary tests; queue-only-fence, absent-authority supersession, scope accumulation, and stale-read mutations must fail alongside the six round-3 fix mutations. Restart/reload does not resume a fenced worker, while explicit retirement permits resume.",
"commands": [
"ORCA_BACKGROUND_LAUNCH=1 npx vitest run --config config/vitest.config.ts src/renderer/src/lib/runtime-session-ssh-projection-authority.test.ts src/renderer/src/lib/runtime-session-pending-hydration.test.ts src/renderer/src/lib/runtime-session-application.test.ts src/main/persistence/loading-store/store-runtime-authored-session-fences.test.ts src/main/runtime/runtime-legacy-worker-session-publication.test.ts src/main/runtime/runtime-legacy-worker-host-write-publication.test.ts src/main/runtime/runtime-legacy-worker-recovery-publication-order.test.ts src/renderer/src/lib/legacy-worker-old-source-resume.test.ts src/renderer/src/lib/runtime-session-refresh-hydration-order.test.ts src/renderer/src/store/slices/agent-sleep-resume-fence-races.test.ts src/renderer/src/store/slices/workspace-scoped-resume-fence-hydration.test.ts src/renderer/src/components/terminal-pane/pty-connection/sleeping-record-access-recordless-fence.test.ts",
"ORCA_BACKGROUND_LAUNCH=1 SKIP_BUILD=1 npx playwright test tests/e2e/settled-worker-resume-fence-restart.spec.ts tests/e2e/completed-worker-retirement-resume.spec.ts --config tests/playwright.config.ts --project=electron-headless --workers=1"
],
"testFiles": [
"src/renderer/src/lib/runtime-session-ssh-projection-authority.test.ts",
"src/renderer/src/lib/runtime-session-pending-hydration.test.ts",
"src/renderer/src/lib/runtime-session-application.test.ts",
"src/main/persistence/loading-store/store-runtime-authored-session-fences.test.ts",
"src/main/runtime/runtime-legacy-worker-session-publication.test.ts",
"src/main/runtime/runtime-legacy-worker-host-write-publication.test.ts",
"src/main/runtime/runtime-legacy-worker-recovery-publication-order.test.ts",
"src/renderer/src/lib/legacy-worker-old-source-resume.test.ts",
"src/renderer/src/lib/runtime-session-refresh-hydration-order.test.ts",
"src/renderer/src/store/slices/agent-sleep-resume-fence-races.test.ts",
"src/renderer/src/store/slices/workspace-scoped-resume-fence-hydration.test.ts",
"src/renderer/src/components/terminal-pane/pty-connection/sleeping-record-access-recordless-fence.test.ts",
"tests/e2e/settled-worker-resume-fence-restart.spec.ts",
"tests/e2e/completed-worker-retirement-resume.spec.ts"
],
"assertionRefs": [
{
"file": "src/renderer/src/lib/runtime-session-application.test.ts",
"assertions": [
"applies startup runtime fields at read time, before delayed catalog hydration",
"orders a scoped retirement after an in-flight invalidation and retains its sibling",
"publishes sleeping records and their authority in a single subscriber notification",
"full supersession remains full after scoped hydration",
"two overlapping scoped hydrations retain both scopes and accept a third scope reply"
]
}
],
"evidenceRuns": [
{
"date": "2026-09-07",
"runner": "local",
Comment on lines +66 to +69

@coderabbitai coderabbitai Bot Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 6 '\bSKIP_BUILD\b|fresh build' . \
  --glob '!node_modules/**' \
  --glob '!dist/**'

Repository: stablyai/orca

Length of output: 50370

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- gate excerpt ---'
sed -n '1,95p' config/reliability-gates.jsonc

printf '%s\n' '--- SKIP_BUILD definitions and consumers outside the gate ledger ---'
rg -n -C 5 'SKIP_BUILD' . \
  --glob '!config/reliability-gates.jsonc' \
  --glob '!node_modules/**' \
  --glob '!dist/**' \
  --glob '!build/**' \
  --glob '!coverage/**' \
  | head -n 240

printf '%s\n' '--- relevant package scripts/config references ---'
rg -n -C 4 'playwright|electron-vite build|test:e2e|SKIP_BUILD' package.json tests/playwright.config.ts scripts . \
  --glob '!config/reliability-gates.jsonc' \
  --glob '!node_modules/**' \
  --glob '!dist/**' \
  --glob '!build/**' \
  --glob '!coverage/**' \
  | head -n 320

Repository: stablyai/orca

Length of output: 47971

🤖 get_repo_knowledge executed:

get_repo_knowledge stablyai/orca /tmp/coderabbit-repo-knowledge/stablyai-orca-89dc44e4/learnings /tmp/coderabbit-repo-knowledge/stablyai-orca-89dc44e4/conventions

Length of output: 10408

Align the build evidence with the command.

When out/main/index.js exists, SKIP_BUILD=1 makes globalSetup reuse it. The recorded command therefore does not prove a fresh build. Record an explicit pnpm exec electron-vite build --mode e2e step in the evidence run, or revise the summary to describe reuse of an existing build.

"platform": "macos",
"command": "ORCA_BACKGROUND_LAUNCH=1 SKIP_BUILD=1 npx playwright test tests/e2e/settled-worker-resume-fence-restart.spec.ts tests/e2e/completed-worker-retirement-resume.spec.ts --config tests/playwright.config.ts --project=electron-headless --workers=1",
"result": "passed",
"durationSeconds": 24.8,
"summary": "Three hidden Electron tests pass on a fresh build, with isolated homes/profiles."
}
],
"runtimeBudget": {
"p95Seconds": 120,
"scope": "Reviewer probes and paired hidden Electron specs."
},
"flakeHistory": {
"status": "not-started",
"evidence": "Local acceptance runs only; no CI soak claimed."
},
"redGreenEvidence": {
"status": "complete",
"evidence": "Baseline reviewer overlay: 7 failures / 29 tests. Candidate passes. All six fix mutations and two additional ordering/scoping mutations fail checked-in assertions."
},
"performanceBudget": {
"required": false,
"evidence": "One serialized session application lane and coalesced invalidations; no polling or per-pane subprocesses."
},
"knownGaps": [
"No live independently versioned paired runtime, physical Windows/Linux, or SSH transport journey."
],
"promotionCriteria": [
"Complete CI soak and cross-platform runtime validation without weakening the mutation oracles."
],
"demotionRule": "Keep experimental until the declared CI soak is complete."
},
{
"id": "terminal-performance.padded-fullscreen-redraw",
"title": "Fullscreen redraw padding does not stall terminal delivery",
Expand Down
7 changes: 7 additions & 0 deletions src/main/orca-profiles/profile-project-session-field-disposition.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,13 @@ export const WORKSPACE_SESSION_FIELD_DISPOSITION = {
// the record's worktreeId on worktree and project removal. Moving a project between profiles runs
// removeSourceRepo, which has no owner scan, so these records leak there.
sleepingAgentSessionsByPaneKey: { onRepoRemoval: 'notRepoScoped', onTransfer: 'notTransferred' },
// Why not transferred: the fence is a projection of the source profile's orchestration DB, and a
// transferred project carries no dispatch rows to re-derive it from. The destination runtime
// re-stamps whatever its own DB still claims on its first recovery pass.
legacyWorkerResumeFencesByPaneKey: {
onRepoRemoval: 'prunedByBespokeRule',

@pullfrog pullfrog Bot Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prunedByBespokeRule promises a removal rule that removeRepoFromWorkspaceSession never implements for this field. terminalPtyIncarnationsByPaneKey gets its pane-key filter rule, and this field is pane-keyed the same way, so removing a repo leaves its fence entries behind as inert residue — which contradicts the census system whose whole point is that a classified field is handled by construction.

Technical details 
# legacyWorkerResumeFencesByPaneKey has no bespoke repo-removal rule

## Affected sites
- src/main/orca-profiles/profile-project-session-field-disposition.ts:115 — classified `onRepoRemoval: 'prunedByBespokeRule'`
- src/main/orca-profiles/profile-project-session-state.ts — `removeRepoFromWorkspaceSession` has bespoke rules for `terminalSurfaceTombstonesByPaneKey` and `terminalPtyIncarnationsByPaneKey` but none for this field

## Required outcome
- A removed repo must not leave fence entries for its panes in the session, or the classification must say so honestly (`notRepoScoped` with a comment noting main's level-triggered rewrite owns the field's lifecycle).

## Suggested approach (optional)
- Mirror the `terminalPtyIncarnationsByPaneKey` rule: filter entries by `paneKey.slice(0, paneKey.lastIndexOf(':'))` against `removedTerminalTabIds`. Add a census test case so a future reclassification can't silently drop the rule.

onTransfer: 'notTransferred'
},
terminalPtyIncarnationsByPaneKey: {
onRepoRemoval: 'prunedByBespokeRule',
onTransfer: 'copiedByBespokeRule'
Expand Down
25 changes: 25 additions & 0 deletions src/main/persistence/legacy-worker-session-projection.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
import type { WorkspaceSessionState } from '../../shared/workspace-session-state-types'

// Old readers consume a derived record flag; only the runtime-authored set can retire it.
export function projectLegacyWorkerSession(session: WorkspaceSessionState): WorkspaceSessionState {
const fences = session.legacyWorkerResumeFencesByPaneKey
const records = session.sleepingAgentSessionsByPaneKey
if (fences === undefined || !records) {
return session
}
let projected: typeof records | undefined
for (const [paneKey, record] of Object.entries(records)) {
const fenced = fences[paneKey] === true
if (fenced === (record.automaticResumeBlockedBy === 'legacy-orchestration-worker')) {
continue
}
projected ??= { ...records }
if (fenced) {
projected[paneKey] = { ...record, automaticResumeBlockedBy: 'legacy-orchestration-worker' }
} else {
const { automaticResumeBlockedBy: _retired, ...unfenced } = record
projected[paneKey] = unfenced
}
}
return projected ? { ...session, sleepingAgentSessionsByPaneKey: projected } : session
}
7 changes: 4 additions & 3 deletions src/main/persistence/loading-store/session-host-partitions.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { projectLegacyWorkerSession } from '../legacy-worker-session-projection'
import type { PersistedState } from '../../../shared/persisted-state-types'
import type { WorkspaceSessionState } from '../../../shared/workspace-session-state-types'
import { sanitizeWorkspaceSessionTerminalRetirements } from '../../runtime/mobile-session-terminal-persistence-retirement'
Expand Down Expand Up @@ -55,12 +56,12 @@ export class SessionHostPartitionOperations {
getWorkspaceSession(hostId?: string | null): PersistedState['workspaceSession'] {
const resolved = resolveHostId(hostId)
if (resolved === LOCAL_EXECUTION_HOST_ID) {
return (
return projectLegacyWorkerSession(
this[sessionHostPartitionOperationsContext].runtime.state.workspaceSession ??
getDefaultWorkspaceSession()
getDefaultWorkspaceSession()
Comment on lines +59 to +61

@coderabbitai coderabbitai Bot Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 15 'function removeWorkspaceSessionOwner|removeWorkspaceSessionOwner\(' \
  src/main/persistence
rg -n -C 8 'scheduleSave|workspaceSession =' \
  src/main/persistence/loading-store/session-host-partitions.ts

Repository: stablyai/orca

Length of output: 23109

🤖 get_repo_knowledge executed:

get_repo_knowledge stablyai/orca /tmp/coderabbit-repo-knowledge/stablyai-orca-89dc44e4/learnings /tmp/coderabbit-repo-knowledge/stablyai-orca-89dc44e4/conventions

Length of output: 18363

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- session-host-partitions.ts ---'
cat -n src/main/persistence/loading-store/session-host-partitions.ts | sed -n '1,175p'

printf '%s\n' '--- projection and field references ---'
rg -n -C 12 'projectLegacyWorkerSession|automaticResumeBlockedBy|cloneWorkspaceSessionState|deleteOwnerKeyedSessionFields' src/main/persistence

printf '%s\n' '--- save and persistence boundary references ---'
rg -n -C 10 'workspaceSessionsByHostId|workspaceSession|scheduleSave|serialize|persist|toJSON' src/main/persistence/loading-store src/main/persistence | head -n 500

Repository: stablyai/orca

Length of output: 50369

🏁 Script executed:

#!/bin/bash
set -euo pipefail
cat -n src/main/persistence/loading-store/session-host-partitions.ts | sed -n '1,175p'
printf '\n--- symbols ---\n'
rg -n -C 15 'projectLegacyWorkerSession|automaticResumeBlockedBy|cloneWorkspaceSessionState|deleteOwnerKeyedSessionFields' src/main/persistence
printf '\n--- save boundary ---\n'
rg -n -C 10 'scheduleSave|writeStore|workspaceSessionsByHostId|workspaceSession' src/main/persistence/loading-store src/main/persistence | head -n 500

Repository: stablyai/orca

Length of output: 50369

🏁 Script executed:

#!/bin/bash
set -euo pipefail
printf '%s\n' '--- session owner removal implementation ---'
cat -n src/main/persistence/restoring-sessions/session-owner-removal.ts | sed -n '1,180p'
printf '%s\n' '--- legacy projection definitions and callers ---'
rg -n -C 20 'projectLegacyWorkerSession|automaticResumeBlockedBy' src
printf '%s\n' '--- persistence writer definitions ---'
rg -n -C 20 'scheduleSave|workspaceSessionsByHostId|workspaceSession' src/main/persistence/loading-store src/main/persistence | tail -n 500

Repository: stablyai/orca

Length of output: 50370

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- session partition writes ---'
cat -n src/main/persistence/loading-store/session-host-partitions.ts | sed -n '174,360p'

printf '%s\n' '--- projection-related write helpers and callers ---'
rg -n -C 12 'preserveRuntimeAuthoredWorkspaceSessionFields|withoutRedundantGlobalFields|sanitizeWorkspaceSessionTerminalRetirements|setWorkspaceSession\(' src/main/persistence/loading-store src/main/persistence

printf '%s\n' '--- scheduleSave implementation ---'
cat -n src/main/persistence/loading-store/write-scheduling.ts | sed -n '1,180p'

Repository: stablyai/orca

Length of output: 50370

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- fence write tests around owner removal ---'
cat -n src/main/persistence/loading-store/store-runtime-authored-session-fences.test.ts | sed -n '180,280p'

printf '%s\n' '--- save scheduling and state flush ---'
cat -n src/main/persistence/loading-store/write-scheduling.ts | sed -n '1,120p'
rg -n -C 12 'flushOrThrow|scheduleSave|state\.workspaceSession|workspaceSessionsByHostId' src/main/persistence/loading-store/store.ts src/main/persistence/loading-store/store-runtime-state.ts src/main/persistence/loading-store 2>/dev/null | head -n 300

Repository: stablyai/orca

Length of output: 37189

Keep the compatibility projection out of the persisted session.

getWorkspaceSession adds automaticResumeBlockedBy from the canonical fence map. removeWorkspaceSessionOwnerInPartition passes this projected session to removeWorkspaceSessionOwner, then stores the clone directly in runtime state and schedules a write. This path can persist the compatibility field. Mutate the canonical partition session, or strip automaticResumeBlockedBy before storing the result.

)
}
return (
return projectLegacyWorkerSession(
this[sessionHostPartitionOperationsContext].runtime.state.workspaceSessionsByHostId?.[
resolved
] ?? getDefaultWorkspaceSession()
Expand Down
Loading
Loading