Orchestration is stuck in the Running state

[andrey-malkov]

Description

My workflow is designed to wait for a specific state from an external service. Since this can take days or even weeks, I use the eternal orchestration pattern. If the awaited status is not reached, the orchestration calls ContinueAsNew to start the next iteration.

Out of thousands of executions, the workflow occasionally becomes stuck. This issue started occurring after upgrading the Durable Task extension for .NET isolated functions to version 1.13.0. Upgrading further to 1.14.0 did not resolve the issue — it has happened again.

Using Durable Functions Monitor, I identified a stuck orchestration with ID:

efd34532203e593297b36e38bd89590f

Its last update timestamp is:

2026-02-16T09:41:54 UTC

which is more than 10 hours ago.

From the logs, I can see that the orchestration was running normally for approximately 50 minutes before it stopped making progress.

2026-02-16 08:57:47.9259236 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' started.
2026-02-16 08:58:51.9662074 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' completed.
2026-02-16 08:58:51.9693582 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' started.
2026-02-16 08:59:51.7197587 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' completed.
2026-02-16 08:59:51.7239817 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' started.
2026-02-16 09:00:51.6458984 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' completed.
2026-02-16 09:00:51.6488428 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' started.
2026-02-16 09:01:51.5722086 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' completed.
...
2026-02-16 09:41:54.6424303 | 'wait-for-task' orchestration with ID 'efd34532203e593297b36e38bd89590f' started.

I suspect there may be an issue with the control queue message processing, as I can see errors in the Log Stream

2026-02-16T20:45:04   [Warning]   Error response [ff130abf-c869-44d5-9a42-d6a7d41f42cf] 404 The specified queue does not exist. (00.0s)
Server:Windows-Azure-Queue/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id:b5953f2c-e003-000d-2b85-9f01c4000000
x-ms-client-request-id:ff130abf-c869-44d5-9a42-d6a7d41f42cf
x-ms-version:2025-07-05
x-ms-error-code:QueueNotFound
Date:Mon, 16 Feb 2026 20:45:03 GMT
Content-Length:217
Content-Type:application/xml

I also attempted to identify any stuck messages in the control queue by searching for messages with a high DequeueCount, but I was unable to find any such messages. There are no obvious poison messages or repeatedly dequeued items that would explain the behavior.

Expected behavior

Workflow should not stop progressing

App Details

• Durable Functions extension version 1.14.0:
• Azure Functions runtime version 2.0:
• Programming language used: C# .Net 8

Screenshots

If deployed to Azure

• Timeframe issue observed: 2026-02-16T09:41:54 UTC
• Function App name: odmt-eu-red-files-workers
• Function name(s): wait-for-task
• Azure region: North Europe
• Orchestration instance ID(s): efd34532203e593297b36e38bd89590f
• Azure storage account name: odmteuredfidata

[torosent]

Hi @andrey-malkov ,
Can you confirm in the host.json Durable Task configuration that:

1. extendedSessionsEnabled set to true?
2. What is extendedSessionIdleTimeoutInSeconds?

Can you provide tull Application Insights / DurableFunctionsEvents logs around 09:40:00 - 09:45:00 UTC for instance efd34532203e593297b36e38bd89590?
Look for FunctionAborted, SessionAbortedException, or WorkerProcessExitException entries

[andrey-malkov]

Hi Tomer, thank you for your quick response. Yes, extended sessions are enabled, with extendedSessionIdleTimeoutInSeconds set to 60.

We do not retain logs for an extended period — only exceptions are stored. I have checked and found no entries related to the specified instance ID, nor any FunctionAborted, SessionAbortedException, or WorkerProcessExitException events.

I have now disabled the daily cap for Application Insights. I will wait for the next occurrence of the issue and follow up with additional details once it happens again.

[sophiatev]

Hi @andrey-malkov, it looks like we have a subtle race condition happening in the Azure Storage backend that's the cause of the bug you're seeing. This is due to the fact that we first commit any outbound messages before updating an orchestration's status in storage when completing an orchestration work item. The flow is as follows:

1. An orchestration continues-as-new with a new execution ID "x", and upon completing the orchestration work item, sends a TaskScheduledEvent to start a new Activity.
2. That Activity actually manages to get scheduled and complete, all before the call to complete the orchestration work item is able to update the instance state in storage. So, when we attempt to process the TaskCompletedEvent and find an instance with execution ID "x", we don't find anything, so we delete the message.
3. The orchestration gets stuck permanently as running waiting for a TaskCompletedEvent that will never arrive.

Basically, we have to change the fact that the message gets deleted in step 2. I'll be working on this fix and will update this thread accordingly.

[sophiatev]

@andrey-malkov can you actually provide the instance ID of another stuck orchestration, if you have one. It appears my initial diagnosis of the issue was not complete