A parallel panel leg inside a manager-hosted Cotal Lang run failed with a RunJournalReplayRaced fault while the run itself was healthy and its two sibling legs settled normally seconds before and after.
Observed 2026-09-14 20:55:58Z on a parallel of three ask steps, each backed by its own spawned seat, 40 s after the second sibling settled:
L4000 handler-fault: run run-<id>: the replay consumer wfj_run-<id>_<takeover> had already delivered 17 records to someone else. Another driver is replaying this run; retry the takeover rather than resuming from a partial prefix.
The durable named in the fault is the run's own takeover consumer: the manager log for the run reads started on endpoint manager (epoch 1, takeover <same id>), and no second driver ever took the run over (one activation, completed in 14 step(s) right after). Operator-side journal reads mint their own takeover ids (newTakeoverId) and the only ones against this run happened five minutes later, so the contender was inside the driver.
What the code allows: replayRunJournal (packages/core/src/run-journal.ts:387) creates a durable named by (space, run, takeoverId), reads, and deletes it in finally. Two replays under the same takeover id therefore share one durable; consumers.add returns the existing one, assertReplayConsumerFresh sees a nonzero delivered count and throws RunJournalReplayRaced, and the first replay's delete tears down the second's fetch. RunScopeAuthority serializes its own replays through a promise chain (implementations/runtime/src/run-scope-authority.ts:49), but other in-driver replays under req.lease.takeoverId do not join that chain (noRecordToResume in run-driver.ts:184, and any caller that hands the lease id to status in run-host.ts:157), so a settle on one parallel branch overlapping a replay on another can collide.
Effect: a completed reviewer verdict was lost as a step failure that is not retried at the ask level. The recovery in resumeRunTakeover (run-journal.ts:775) treats the race as "a lost round, not a lost run" and retries, but the ask path surfaces it as L4000 and the branch fails.
Fix direction:
- Route every in-driver replay under the lease takeover id through the scope authority's serialized reader, or give each replay call its own consumer suffix so concurrent branches never share a durable.
- Treat
RunJournalReplayRaced inside a live driver as retryable at the step, the way the takeover path already does, instead of failing the ask.
- Add a runtime cell: a
parallel of three asks settling within the same second must not fail any branch; register the shared-durable behaviour as the mutant.
Not the #1619 poll-timeout family: this leg was at 149 s of work, not at a deadline.
A parallel panel leg inside a manager-hosted Cotal Lang run failed with a
RunJournalReplayRacedfault while the run itself was healthy and its two sibling legs settled normally seconds before and after.Observed 2026-09-14 20:55:58Z on a
parallelof threeasksteps, each backed by its own spawned seat, 40 s after the second sibling settled:The durable named in the fault is the run's own takeover consumer: the manager log for the run reads
started on endpoint manager (epoch 1, takeover <same id>), and no second driver ever took the run over (one activation,completed in 14 step(s)right after). Operator-side journal reads mint their own takeover ids (newTakeoverId) and the only ones against this run happened five minutes later, so the contender was inside the driver.What the code allows:
replayRunJournal(packages/core/src/run-journal.ts:387) creates a durable named by(space, run, takeoverId), reads, and deletes it infinally. Two replays under the same takeover id therefore share one durable;consumers.addreturns the existing one,assertReplayConsumerFreshsees a nonzero delivered count and throwsRunJournalReplayRaced, and the first replay'sdeletetears down the second's fetch.RunScopeAuthorityserializes its own replays through a promise chain (implementations/runtime/src/run-scope-authority.ts:49), but other in-driver replays underreq.lease.takeoverIddo not join that chain (noRecordToResumein run-driver.ts:184, and any caller that hands the lease id tostatusin run-host.ts:157), so a settle on one parallel branch overlapping a replay on another can collide.Effect: a completed reviewer verdict was lost as a step failure that is not retried at the ask level. The recovery in
resumeRunTakeover(run-journal.ts:775) treats the race as "a lost round, not a lost run" and retries, but the ask path surfaces it as L4000 and the branch fails.Fix direction:
RunJournalReplayRacedinside a live driver as retryable at the step, the way the takeover path already does, instead of failing the ask.parallelof three asks settling within the same second must not fail any branch; register the shared-durable behaviour as the mutant.Not the #1619 poll-timeout family: this leg was at 149 s of work, not at a deadline.