Describe the Bug
With @flue/runtime@1.0.0-beta.9 on Cloudflare, a durable agent submission can remain running indefinitely after its flue:submission-attempt fiber is interrupted. onFiberRecovered() correctly calls requestSubmissionRecovery(), but reconcileSubmissions() skips the submission while its attempt key remains in the process-local activeAttempts set:
if (this.activeAttempts.has(this.submissionAttemptLocalKey(submission))) continue;
The attempt-marker guard immediately below it already lets recoveryRequestedAt override a marker, but the local-active guard does not. If the interrupted runFiber() promise does not settle, its .finally() never clears activeAttempts; each 30-second __flueWakeAgentSubmissions callback then skips the submission again.
We observed one submission recover only after the stale local attempt eventually cleared, while another remained in this wake loop for the full capture. Sanitized diagnostics for the stuck case were one fiber:run:interrupted after about 29 seconds, one fiber:recovery:handled with status: "user", then only schedule:execute / schedule:create every 30 seconds, with no replacement fiber:run:started or terminal submission_settled.
There is a related controller race: attempts share activeControllers[submissionId]. If recovery starts replacement attempt B while old attempt A’s promise remains unsettled, A’s delayed .finally() unconditionally deletes that map entry and can remove B’s controller. Cleanup should only delete when the stored value is A’s captured controller.
Expected Behavior
A durable recovery request should override stale process-local activity state and immediately reconcile the interrupted attempt. A late cleanup from the old attempt should not remove the replacement attempt’s controller.
A minimal fix appears to be:
if (
this.activeAttempts.has(this.submissionAttemptLocalKey(submission)) &&
submission.recoveryRequestedAt === void 0
) continue;
and identity-guarding controller cleanup:
if (this.activeControllers.get(submission.submissionId) === controller) {
this.activeControllers.delete(submission.submissionId);
}
Steps to Reproduce
- Start a Cloudflare durable agent submission and keep its
runFiber() promise pending long enough for the Workers invocation/fiber to be interrupted.
- Let Agents SDK recovery invoke
onFiberRecovered() with the stashed submissionId and attemptId.
- Keep the old promise unsettled so its
.finally() does not remove the process-local attempt key.
- Observe that
requestSubmissionRecovery() persists the request, but immediate reconciliation and subsequent 30-second wake callbacks skip the running submission and never start a replacement fiber.
Environment: @flue/runtime@1.0.0-beta.9, Cloudflare Workers, agents@0.17.x.
Describe the Bug
With
@flue/runtime@1.0.0-beta.9on Cloudflare, a durable agent submission can remainrunningindefinitely after itsflue:submission-attemptfiber is interrupted.onFiberRecovered()correctly callsrequestSubmissionRecovery(), butreconcileSubmissions()skips the submission while its attempt key remains in the process-localactiveAttemptsset:The attempt-marker guard immediately below it already lets
recoveryRequestedAtoverride a marker, but the local-active guard does not. If the interruptedrunFiber()promise does not settle, its.finally()never clearsactiveAttempts; each 30-second__flueWakeAgentSubmissionscallback then skips the submission again.We observed one submission recover only after the stale local attempt eventually cleared, while another remained in this wake loop for the full capture. Sanitized diagnostics for the stuck case were one
fiber:run:interruptedafter about 29 seconds, onefiber:recovery:handledwithstatus: "user", then onlyschedule:execute/schedule:createevery 30 seconds, with no replacementfiber:run:startedor terminalsubmission_settled.There is a related controller race: attempts share
activeControllers[submissionId]. If recovery starts replacement attempt B while old attempt A’s promise remains unsettled, A’s delayed.finally()unconditionally deletes that map entry and can remove B’s controller. Cleanup should only delete when the stored value is A’s captured controller.Expected Behavior
A durable recovery request should override stale process-local activity state and immediately reconcile the interrupted attempt. A late cleanup from the old attempt should not remove the replacement attempt’s controller.
A minimal fix appears to be:
and identity-guarding controller cleanup:
Steps to Reproduce
runFiber()promise pending long enough for the Workers invocation/fiber to be interrupted.onFiberRecovered()with the stashedsubmissionIdandattemptId..finally()does not remove the process-local attempt key.requestSubmissionRecovery()persists the request, but immediate reconciliation and subsequent 30-second wake callbacks skip the running submission and never start a replacement fiber.Environment:
@flue/runtime@1.0.0-beta.9, Cloudflare Workers,agents@0.17.x.