ObjectDisposedException during host shutdown

[andrey-malkov]

Problem Description

We are running Durable Functions orchestrations on Azure Functions Isolated Worker (.NET 10, Premium plans, Linux). When the Azure Functions platform performs a scale-in (removes a worker instance), in-flight orchestrator functions throw System.ObjectDisposedException: Cannot access a disposed object. Object name: 'IServiceProvider' and are permanently marked as RuntimeStatus = Failed. The Durable Task framework does not retry these orchestrations — they remain in Failed state indefinitely, causing upstream workflows to become stuck.

The cascade

When the host initiates shutdown for scale-in, the DI ServiceProvider is disposed. However, orchestrator functions that have already been dispatched (messages dequeued from the control queue and handed to the worker) attempt to resolve services via DefaultFunctionContext.get_InstanceServices() which calls ServiceProvider.CreateScope(). This throws ObjectDisposedException because the container is already disposed.

The Durable Task extension treats this as an unhandled orchestrator failure and commits RuntimeStatus = Failed to the Instances table. The failure then propagates up the orchestration call chain:

1. wait-for-task (sub-orchestration) — fails first with ObjectDisposedException
2. migrate-site-workflow (parent) — receives SubOrchestrationInstanceFailed from wait-for-task and also transitions to Failed
3. process-one-backlog-item-workflow (top-level) — receives the cascading failure and permanently fails

Inconsistent retry behavior: In some cases, the framework appears to retry the failed orchestration — the queue message is picked up by another instance and the workflow recovers. However, in other cases (the majority we observe), the failure is permanent: RuntimeStatus = Failed is committed to the Instances table and no retry occurs. The business workflow is permanently abandoned with no automatic recovery.

The critical issue: Unlike activity function failures which leave queue messages available for retry, orchestrator failures sometimes commit a terminal Failed status to the instance store. Once committed, the orchestration is permanently dead — no re-dispatch. The non-deterministic nature of retry vs permanent failure makes this particularly dangerous: there is no reliable way to predict which orchestrations will recover and which will be permanently lost.

Observed timeline (real production incident, 2026-06-01 13:45 UTC)

Time (UTC)	Event
13:45:23.834	Host logs: "Application is shutting down..." on instance 0--bb3e733a...
13:45:24.015	29 orchestrations fetch TimerFired messages from control queues
13:45:24.051	Orchestrations begin processing (timer delays 63–64 seconds)
13:45:24.098	ObjectDisposedException ×28 thrown — ServiceProvider already disposed
13:45:24.155	wait-for-task orchestrations set to RuntimeStatus: Failed
13:45:24.165	SubOrchestrationInstanceFailed messages sent to parent orchestrations
13:45:24.197	Instance tables updated: 29 orchestrations permanently Failed
13:45:28.445	Parent process-one-backlog-item-workflow orchestrations also Failed

After shutdown, the instance never appeared again — confirming this was a permanent scale-in, not a restart.

Key observations

1. The ServiceProvider is disposed before in-flight invocations complete. The shutdown signal and ServiceProvider disposal happen within 28–478ms of each other. Orchestrator functions already dispatched cannot resolve dependencies.

2. Orchestrator failures are permanent — no retry. Unlike activity failures (which return the message to the queue), when an orchestrator function fails, the extension commits RuntimeStatus = Failed to the Instances table. The Durable Task framework has no built-in retry for orchestrator failures.

3. One scale-in event kills many orchestrations. In the worst observed incident, a single instance shutdown caused 29 sub-orchestrations and their parent workflows to fail permanently. With many orchestrations sharing a single instance, one scale-in can cascade into dozens of permanently abandoned workflows.

4. SubOrchestrationInstanceFailed propagates fatally. When a child wait-for-task orchestration fails, the parent process-one-backlog-item-workflow receives SubOrchestrationInstanceFailed and also transitions to Failed — extending the blast radius beyond the originally affected instance.

5. Contrast: Timer-based orchestrations survive. Orchestrations that were merely waiting on timers (not actively executing) survive because their timer messages remain on the control queue and are picked up by a new instance. Only orchestrations whose code was actively executing at shutdown time are permanently lost.

Impact (3-day observation window, EU region only)

Metric	Value
Total ObjectDisposedException occurrences	169
Unique worker instances affected (scale-in events)	7
Permanently failed orchestration instances	39
Affected function apps	odmt-eu-b1-files-workers, odmt-eu-b1-mail-workers
Failed orchestration types	wait-for-task, process-one-backlog-item-workflow
Customer impact	Migration tasks stuck permanently — no completion

Extension log confirming permanent failure

739294d1f4be5a8ab7541cc46f088164: Function 'wait-for-task (Orchestrator)' failed with an error.
Reason: Microsoft.Azure.WebJobs.Host.FunctionInvocationException
IsReplay: False. State: Failed. RuntimeStatus: Failed.
HubName: filesworkers. AppName: odmt-eu-b1-files-workers.
SlotName: Production. ExtensionVersion: 3.12.5. SequenceNumber: 16420. TaskEventId: -1

Exception stack trace

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
  Exception while executing function: Functions.process-one-backlog-item-workflow
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException:
  Result: Failure
  Type: System.ObjectDisposedException
  Exception: Cannot access a disposed object.
  Object name: 'IServiceProvider'.
  Stack:
    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper
       .ThrowObjectDisposedException()
    at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
    at Microsoft.Azure.Functions.Worker.DefaultFunctionContext
       .get_InstanceServices()
       in /*/src/DotNetWorker.Core/Context/DefaultFunctionContext.cs:line 46
    at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions
       .<>c__DisplayClass3_0`1.b__1(FunctionContext context)
       in /_/src/DotNetWorker.Core/Hosting/WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
    at Microsoft.Azure.Functions.Worker.FunctionsApplication
       .InvokeFunctionAsync(FunctionContext context)
       in /_/src/DotNetWorker.Core/FunctionsApplication.cs:line 83
    at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler
       .InvokeAsync(InvocationRequest request)
       in /_/src/DotNetWorker.Grpc/Handlers/InvocationHandler.cs:line 91

Expected behavior

When a Functions host undergoes shutdown/scale-in:

1. Pending orchestrator messages should be abandoned (left on queue) so another instance picks them up
2. If an orchestrator function invocation fails due to host disposal, the failure should be treated as retriable (not permanently terminal)
3. The ServiceProvider disposal should be deferred until all in-flight function invocations complete (graceful drain)

---

Product: Durable Azure Functions (Isolated Worker, .NET 10)
Plan: Flex Consumption
Durable Task Extension: Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.16.5
Task Hub: filesworkers, mailboxworkers
OS: Linux
Regions affected: North Europe

---

[andrey-malkov]

Evidence of inconsistent retry (2026-06-02 10:36 UTC, same scale-in event)

The screenshots below shows a single scale-in event on odmt-eu-yellow-files-workers (worker instance 0--ab1d69f7-fa35-4bcd-94d4-9395551731a5) at 10:36:21.987 — "Application is shutting down..." — followed by 3 ObjectDisposedException at 10:36:22.

Two orchestrations were affected:

Instance ID	Outcome	Details
a06cfd95da8857dcac5f2dd09262c314	Recovered	Picked up by a new worker 0--d5681085-dce0-43d4-b858-5593c9f24459 ~11 minutes later (10:47:33). Fetched TimerFired#3, continued processing normally.
9c5f76f1ca4d54539004b8cb020c2b13	Permanently failed	Never appeared again in any logs. No retry, no recovery — the orchestration is permanently dead.

Both orchestrations received the same exception from the same shutdown event, yet one recovered and the other did not. This demonstrates the non-deterministic nature of the failure.

[andrey-malkov]

@sophiatev there was a spike over the weekend, and it's happening again today. For example, workflow da1d5bae-5760-5e83-97b9-4ad990c10bda failed with the same root cause. Could you take a look? Here is the logs attached

Cannot access a disposed object.csv

Here's a detailed description:

Time	Event
10:13:06	process-one-backlog-item-workflow scheduled for backlog item 0050639161601810804634|0001000003
10:14:05	Orchestration started on files-workers, dispatched migrate-drive-files-workflow
10:53:59.155	Worker shutdown begins — IServiceProvider disposed. wait-for-task sub-orchestration fails with System.ObjectDisposedException: Cannot access a disposed object. Object name: 'IServiceProvider'.
10:54:00.417	migrate-drive-files-workflow fails with Object reference not set to an instance of an object. (NRE from framework's inability to invoke orchestrator during shutdown)
10:54:00.942	Exception caught by MigrationBacklogProcessor catch block: BacklogExecutor Graph_Drive: item 0050639161601810804634|0001000003 failed Task 'migrate-drive-files-workflow' failed with an unhandled exception: Object reference not set to an instance of an object.()
10:54:02.590	finally block attempts to update backlog item status — schedules update-backlog-item-status-activity (TaskScheduled#27)
10:54:02.795	update-backlog-item-status-activity fails with ObjectDisposedException: 'IServiceProvider' — worker is still shutting down
10:54:02.795	process-one-backlog-item-workflow terminates with OrchestrationFailureException — unhandled exception propagated from finally block

---

Failure mechanism:

1. Worker process begins shutdown (scale-in or restart), IServiceProvider is disposed.
2. In-flight wait-for-task sub-orchestration fails with ObjectDisposedException.
3. This cascades up: migrate-drive-files-workflow fails with NRE (framework cannot invoke the orchestrator with a disposed DI container).
4. The catch block in MigrationBacklogProcessor.MigrateOneBacklogItemAsync catches the exception — this part works correctly.
5. The finally block executes UpdateStatisticsAsync, which calls update-backlog-item-status-activity.
6. This activity ALSO fails with ObjectDisposedException because the same worker shutdown is still in progress.
7. Since the finally block has no exception handling around the activity call, the TaskFailedException propagates unhandled, terminating process-one-backlog-item-workflow.

---

Impact:

The backlog item status is never updated (neither to Completed nor to Failed) because:

• The catch block logged the error but the finally block was supposed to persist the status
• The finally block's activity call failed due to the same disposed IServiceProvider
• The orchestration terminated fatally without completing its cleanup

This leaves the backlog item permanently stalled in InProgress status with no mechanism to recover it.

There more workflows with the same pattern:
53348025-a95a-57d8-aa19-c0ec0e5d3d96
83d3a6fc-4e28-5049-9465-dea686210c76
e3e3a94a-3168-5722-a749-ee3aa9afe0c9

[andrey-malkov]

Additional finding: Two distinct failure modes

The ObjectDisposedException failures split into two patterns:

Mode 1 — NRE + ObjectDisposedException (workflow da1d5bae-5760-5e83-97b9-4ad990c10bda):

• Sub-orchestration fails with NRE ("Object reference not set to an instance of an object") because the framework cannot invoke the orchestrator during shutdown
• Our catch block executes and logs the error
• finally block attempts update-backlog-item-status-activity — which also fails with ObjectDisposedException
• Unhandled exception from finally terminates the orchestration
• Backlog item never updated → task stalls

Mode 2 — Only ObjectDisposedException, no NRE (workflows below):

• 25b8461f-f57b-58ae-9de4-6f3724a3c92e
• d84cf5b1-fdc8-5978-9d7c-8e87d8889d7b
• 0bdd43e2-c68d-5db5-9d46-77c52b7eb379

In these cases, ObjectDisposedException hits at the framework level — the Durable Functions host cannot invoke the orchestrator function because IServiceProvider is already disposed. Our catch/finally blocks are never reached. The orchestration terminates immediately without any of our error handling code executing.

Implication for fix:

• Wrapping the finally block in try-catch would only help Mode 1 (where the catch block was reached but the finally block threw)
• Mode 2 requires an external recovery mechanism since the application code is never entered — the framework itself fails before our function body executes
• Both modes result in stalled backlog items stuck in InProgress status

Cannot access a disposed object no NRE.csv

[andrey-malkov]

'Object reference not set to an instance of an object' details:

Regarding the NRE stack trace: this is Microsoft.DurableTask.TaskFailedException (both OuterType and InnermostType). The "Object reference not set to an instance of an object." is carried in FailureDetails.ErrorMessage of that exception — it's the serialized failure from the sub-orchestrator migrate-child-resources-workflow, propagated back to the parent orchestrator by the Durable Task framework via CallSubOrchestratorAsync. It is not thrown by our code.

The two user-code frames in the stack are:

• MigrateResourcesWorkflow.cs:307 — await CallMigrateFunctionAsync(...), standard sub-orchestrator invocation
• MigrateResourcesWorkflow.cs:396 — return await func(), the framework-invoked orchestrator body delegate

Neither frame throws anything. They are receiving a TaskFailedException from TaskOrchestrationContextWrapper.CallSubOrchestratorAsync (frame 0 in the stack), which is the Durable Task Worker framework notifying user code that a previously-recorded sub-orchestrator failure is being replayed. The NRE that originally caused migrate-child-resources-workflow to fail on the dying host is what appears in FailureDetails — the same replay consistency issue described earlier.

migrate-child-resources-workflow-NRE.txt