Extended Sessions for Isolated (Orchestrations)

[sophiatev]

This PR introduces enabling extended sessions for orchestrations in the .NET isolated framework. The way this is achieved is via an IMemoryCache which maintains the extended sessions state in memory. The main changes in this PR are:

• FunctionsWorkerApplicationBuilderExtensions: Create this cache as a singleton upon worker startup
• FunctionsWorkerApplicationBuilderExtensions: Pass the cache to the dotnet SDK when executing the orchestration request.
• OutOfProcMiddlware: Extracts information about extended sessions passed in the dispatchContext from DT.Core to send to the worker. It also checks for the special edge case of a worker needing an orchestration history when none was attached to the request (OrchestratorResponse.requiresHistory is true), and throws a SessionAbortedException, in which case DT.Core will end the session and retry the work item, this time indicating that a history should be attached with the request.
• RemoteOrchestratorConfiguration and RemoteOrchestratorContext: New fields added to pass additional information to the worker related to extended sessions
• OrchestrationTriggerAttributeBindingProvider: Modified to only include past events in the orchestration request sent to the worker if the host indicates that this is necessary. (Past events are not included if this orchestration request is within an extended session, and not its first execution).
• DurableTaskOptions: Modified to not throw an exception in the case that extended sessions are enabled for .NET isolated (but an exception is still thrown for other OOProc languages).

Other PRs:

• DT.Core
• DotNet SDK
• Protobuf

Open questions

Is there a way to pass information from DurableTaskOptions to the worker upon startup? Specifically, the FunctionsWorkerApplicationBuilderExtensions class. Ideally, we would only make the cache for extended sessions if they are enabled (DurableTaskOptions.ExtendedSessionsEnabled is true), and set the cache ExpirationScanFrequency to be some function of the user-specified DurableTaskOptions.ExtendedSessionIdleTimeoutInSeconds (maybe something along the lines of ExtendedSessionIdleTimeoutInSeconds / 5).

Design Callout

There are several avenues by which the worker can inform the host that it needs an orchestration history in the case that the worker has ended the extended session before the host. The worker will evict an extended session after the user-specified extendedSessionIdleTimeoutInSeconds expires. More specifically, what this means is that if the extended session is not accessed within that timeframe, the worker will evict it from the cache. In the meantime, it has to send the result of the orchestration work item back to the host, the host has to process this and commit it to storage, then wait for new orchestration messages, then send the worker the next work item once new messages arrive. From the host's perspective, the extendedSessionIdleTimeoutInSeconds applies only to the amount of time we wait for new orchestration messages to arrive before ending the extended session. All that to say, there could perhaps be an appreciable number of situations where the worker ends the extended session before the host and will require a history.

The current approach is perhaps the simplest - have the worker inform the host via the requiresHistory flag that it needs a history, in which case the host throws a SessionAbortedException from OutOfProcMiddleware. DT.Core already has all the logic to handle a SessionAbortedException and retry the work item. The cons of this approach are that the SessionAbortedException surfaces to the logs and may alarm customers. There is also an added delay that comes from having DT.Core abort the orchestration work item entirely and try it again later in this approach. All that being said, perhaps the number of situations where this occurs could be reduced by just advising customers to up their extendedSessionIdleTimeoutInSeconds count in the isolated model.

An alternative is to have the worker call StreamInstanceHistory in the case that it has since ended the extended session and needs a history. This will not surface an exception to the customer, and will lead to less of a delay in processing the work item. But it could be much more complicated to implement as I am not sure how to get a client over to the GrpcOrchestrationRunner. to accomplish this, and may require additional edge-case handling for all the network issues that could arise.

Manual testing done thus far

• If extended sessions are not enabled, everything still proceeds as expected (the orchestration history is replayed upon every execution)
• If an extended session expires, a history is sent along with the orchestration request to the worker. The worker replays the orchestration and then saves the state in a new extended session (state is only saved if the orchestration is not completed)
• Upon completion of an orchestration, the extended session state is evicted from the cache
• If the worker ends the extended session before the host does (this should be very rare but can happen if e.g. there is a network delay when sending the orchestration request to the worker, or something along those lines), then it will throw a SessionAbortedException. The work item will then be retried again.
• Other OOProc scenarios still work as expected (they do not have extended sessions enabled so the orchestration is replayed upon every request. If they attempt to enable extended sessions an error occurs upon startup indicating that the extended session feature does not work for these other languages).

Performance testing

Two scenarios were run in in-process with extended sessions enabled/disabled, and in isolated with extended sessions enabled/disabled. Multiple trials were run, and the time it took to complete the orchestration was recorded as well as the number of times the history was loaded in each of these settings (with extendedSessionIdleTimeoutInSeconds set to 30 seconds). The two scenarios tested were

• Fan out to a large number of tasks (30,000) and then fan-in. For both isolated and in-process, without extended sessions this took about 35 minutes and with extended sessions around 25 minutes. In the isolated model there were consistently two history loads with ES enabled, with the second corresponding to a SessionAbortedException being thrown (so the worker ends the extended session before the host, and as expected, throws the exception which leads to a retry of the work item, this time with a history attached). In in-process, there was just one history load. Without ES, there were about 100 history loads in each.
• Run a large number of sequential tasks (1000) with large payloads (10,000 character-length strings). For both isolated and in-process, without extended sessions this took about 30 minutes and with extended sessions enabled around 5 minutes. Without ES there were around 100 history loads, and with ES just 1. Interestingly, in isolated this scenario actually ran somewhat faster both with/without ES as opposed to in-process.

resolves #2905

Pull request checklist

• My changes do not require documentation changes
  • Otherwise: Documentation PR is ready to merge and referenced in pending_docs.md
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• I have added all required tests (Unit tests, E2E tests)
• My changes do not require any extra work to be leveraged by OutOfProc SDKs
  • Otherwise: That work is being tracked here: #issue_or_pr_in_each_sdk
• My changes do not change the version of the WebJobs.Extensions.DurableTask package
  • Otherwise: major or minor version updates are reflected in /src/Worker.Extensions.DurableTask/AssemblyInfo.cs
• My changes do not add EventIds to our EventSource logs
  • Otherwise: Ensure the EventIds are within the supported range in our existing Windows infrastructure. You may validate this with a deployed app's telemetry. You may also extend the range by completing a PR such as this one.
• My changes should be added to v2.x branch.
  • Otherwise: This change applies exclusively to WebJobs.Extensions.DurableTask v3.x. It will be retained only in the dev and main branches and will not be merged into the v2.x branch.

[andystaples]

Some questions about this SessionAbortedException -

1. Is SessionAbortedException retried indefinitely by Core or is there a limit?
2. Is it possible that the retry in Core happens so fast that the cache still hasn't been evicted from WebJobs, meaning that the exception will be thrown multiple times for the same task?
3. Have we observed this exception being thrown and retried in practice?
4. Any other gotchas we should document?

[sophiatev]

1. Very good question, I'm not sure actually. We ultimately abandon the work item in DT.Core here but I'm not sure if there is any limit on how many times it will retry the item.
2. So this exception actually happens when the worker evicts the extended session earlier than the host does, so this is not a concern.
3. Yes, everything looks good to me.
4. Not that I can think of for now, but I think we should definitely test/discuss this more if we ultimately check this design into the main branch.

[bachuv]

Can we add tests for extended sessions in the .NET Isolated testing framework? We could add a test for the use case when this exception gets thrown.

[cgillum]

Were these protobuf changes already merged as part of a different PR?

[sophiatev]

These are actually for a new feature I added to DTS, and they're showing up here just cause I pulled from the main protobuf branch (so sort of, yes)