Skip to content

Cloudflare interrupted submission recovery can be blocked by stale activeAttempts entry #457

Description

@Divkix

Describe the Bug

With @flue/runtime@1.0.0-beta.9 on Cloudflare, a durable agent submission can remain running indefinitely after its flue:submission-attempt fiber is interrupted. onFiberRecovered() correctly calls requestSubmissionRecovery(), but reconcileSubmissions() skips the submission while its attempt key remains in the process-local activeAttempts set:

if (this.activeAttempts.has(this.submissionAttemptLocalKey(submission))) continue;

The attempt-marker guard immediately below it already lets recoveryRequestedAt override a marker, but the local-active guard does not. If the interrupted runFiber() promise does not settle, its .finally() never clears activeAttempts; each 30-second __flueWakeAgentSubmissions callback then skips the submission again.

We observed one submission recover only after the stale local attempt eventually cleared, while another remained in this wake loop for the full capture. Sanitized diagnostics for the stuck case were one fiber:run:interrupted after about 29 seconds, one fiber:recovery:handled with status: "user", then only schedule:execute / schedule:create every 30 seconds, with no replacement fiber:run:started or terminal submission_settled.

There is a related controller race: attempts share activeControllers[submissionId]. If recovery starts replacement attempt B while old attempt A’s promise remains unsettled, A’s delayed .finally() unconditionally deletes that map entry and can remove B’s controller. Cleanup should only delete when the stored value is A’s captured controller.

Expected Behavior

A durable recovery request should override stale process-local activity state and immediately reconcile the interrupted attempt. A late cleanup from the old attempt should not remove the replacement attempt’s controller.

A minimal fix appears to be:

if (
  this.activeAttempts.has(this.submissionAttemptLocalKey(submission)) &&
  submission.recoveryRequestedAt === void 0
) continue;

and identity-guarding controller cleanup:

if (this.activeControllers.get(submission.submissionId) === controller) {
  this.activeControllers.delete(submission.submissionId);
}

Steps to Reproduce

  1. Start a Cloudflare durable agent submission and keep its runFiber() promise pending long enough for the Workers invocation/fiber to be interrupted.
  2. Let Agents SDK recovery invoke onFiberRecovered() with the stashed submissionId and attemptId.
  3. Keep the old promise unsettled so its .finally() does not remove the process-local attempt key.
  4. Observe that requestSubmissionRecovery() persists the request, but immediate reconciliation and subsequent 30-second wake callbacks skip the running submission and never start a replacement fiber.

Environment: @flue/runtime@1.0.0-beta.9, Cloudflare Workers, agents@0.17.x.

Metadata

Metadata

Assignees

No one assigned

    Labels

    fix pendingFixed on a development branch; lands with the next release

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions