Let a workflow recover from a temporary failure without discarding completed work

[dingjimmy]

Description

Recovery from an infrastructure failure works well, but there are often situations where a graph workflow consumes remote dependencies that become temporarily unavailable and where recovery can take minutes or hours. For example: a rate-limited API, a service being redeployed, a queue that has backed up.

Today a durable workflow cannot survive that. The moment an executor fails, the whole run ends, and every step that already succeeded is discarded along with it. The only way forward is to start a new run from the beginning, which repeats work that was already done and paid for, including expensive steps such as model calls.

Current behaviour

Given a workflow whose third executor fails because it has a dependency that is temporarily unavailable:

• The run ends in a Failed state within milliseconds of the failure.
• The failing executor ran exactly once. There is no second attempt, and no way to ask for one.
• The results of the first two executors are not reusable. A new run must be triggered, which uneccassaraly repeats the first two.
• Nothing distinguishes "this dependency will be back in ten minutes" (a temporary failure) from "this resource doesn't exist" (a hard failure). Both end the run identically and immediately.

The current documentation states that the extension "automatically checkpoints each step in the graph and recovers from failures", which reads as though it covers this case. From my experience using Durable Workflows it does not, and that distinction is not explicitly stated anywhere.

Desired behaviour

A workflow should be able to treat a failing step as something to (optionally) wait out rather than something fatal.

When an executor fails and the workflow has been told that failure may be temporary:

• The run stays alive instead of ending.
• The failing executor is attempted again later, after a wait that can grow between attempts and can extend to hours.
• Executors that already succeeded are not re-executed. The run continues from the step that failed.
• Waiting consumes no compute and survives a host restart.
• The run gives up after a set number of retries, and the failure reported is the underlying one, not a generic timeout.
• Attempts are visible in the Durable Task Scheduler dashboard, so an operator can see a run is waiting rather than stuck.

The author should also be able to distinguish failures worth waiting for from failures that will never succeed, so an unavailable dependency is waited out while a malformed input fails immediately.

Different steps in the same graph warrant different treatment. A cheap idempotent call can be attempted freely; an expensive model call should not be. Whatever form this takes, it needs to be expressible per executor, not only for the workflow as a whole.

Current workarounds

• Handling the failure inside the executor. The step stays occupied for the whole wait, counts against the host's execution limit, and loses its place entirely if the host restarts. Viable for a fault lasting seconds, not hours.
• Routing the message back to the same executor. Re-attempts happen immediately with no gap, and the number of iterations is capped, so it behaves as a tight loop rather than a wait.
• Pausing for an external signal. A workflow can wait indefinitely for something outside to wake it, but nothing inside the workflow can schedule its own wake-up, so this only helps when a separate component already exists to poll and resume it.
• Writing the orchestration by hand instead of describing a graph. (i.e. use Durable Functions directly) This does produce the desired behavior, but it means giving up the graph model and all the goodness the extension provides.

Proposed API Shape

1. Define retry behaviour when wiring-up the workflow graph

Define fixed retry policy

var retrieveImage = new RetrieveImageExecutor();
var analyseImage = new AnalyseImageWithLLMExecutor();
var submitAnalysis = new SubmitAnalysisExecutor();

var wf = new WorkflowBuilder(retrieveImage)
    .WithName("Example Workflow")
    .WithDescription("Example of a workflow that sets a automatic retry policy for an executor")
    .AddEdge(retrieveImage, analyseImage)
    .AddEdge(analyseImage, submitAnalysis)
    .AddDurableRetry(analyseImage, new RetryPolicy(maxNumberOfAttempts: 3, firstRetryInterval: Timespan.FromSeconds(5)))

Define a custom retry handler

var retrieveImage = new RetrieveImageExecutor();
var analyseImage = new AnalyseImageWithLLMExecutor();
var submitAnalysis = new SubmitAnalysisExecutor();

var wf = new WorkflowBuilder(retrieveImage)
    .WithName("Example Workflow")
    .WithDescription("Example of a workflow that sets a custom retry policy for an executor")
    .AddEdge(retrieveImage, analyseImage)
    .AddEdge(analyseImage, submitAnalysis)
    .AddDurableRetry(analyseImage, (retryContext => 
    {
        if (retryContext.LastFailure.IsCausedBy<ApplicationException>())
        {
            return false;
        }

        return retryContext.LastAttemptNumber < 3;
    }))

2. Provide executors with a way to signal when need to be retried later (and when not too)

Example of executor signaling a temporary failure to runtime

public override async ValueTask HandleAsync(AnalyseImageRequest request, IWorkflowContext context, CancellationToken cancellationToken = default)
{
    // ... call to a LLM service that often becomes temporarally unavailable

    if (response.Status == TemporaralyUnavalable)
    {
        context.SignalFailure(FailureType.Temporary);
    }
}

Example of executor signaling a permanant failure to runtime

public override async ValueTask HandleAsync(AnalyseImageRequest request, IWorkflowContext context, CancellationToken cancellationToken = default)
{
    // ... call to a LLM service

    if (response.Status == InvalidReqeust)
    {
        context.SignalFailure(FailureType.Permanent);
    }
}

[dingjimmy]

I came across Microsoft Agent Framework while researching and sketching-out possible solutions for managing AI enhanced workflows in a product I work on. The preferred option initially was to use Durable Functions, however the Agent Framework programming model in combination with Durable Extensions is a very compelling offering. Notwithstanding this is currently in preview, there are a number of things that make it challenging to adopt right now; no built-in way of retrying temporary failures being the big blocker.

I have looked at the available documentation I can find, the PRs and code and cannot find anything to indicate this is a deliberate design choice. If it is a deliberate design choice, please could the documentation be updated to clarify what type of workflow failures Durable Extensions will attempt to recover from, and which it will not.

If this is something you'll consider adding, the above proposal is just a starting point, a brain-dump, and would love to continue this conversation further.

[kshyju]

James Dingle (@dingjimmy) Thank you for sharing your proposal. We will review both the issue and the proposed solution and provide an update. We welcome community contributions, including issues, pull requests, and other forms of participation.