Extended Sessions for Isolated (Orchestrations)

[sophiatev]

This PR adds a new historyRequired field to the OrchestratorResponse. This will be used by the dotnet SDK to handle the edge case where it has since ended an extended session and needs an orchestration history in order to replay the orchestration up to the execution point, but none was provided in the OrchestratorRequest. The SDK will set the historyRequired field in the response to true, in which case the host will end the extended session and attach a history in the next OrchestratorRequest (specifically, as discussed in this PR, the OutOfProcMiddleware class will check for this flag in the response and throw a SessionAbortedException in the case that it is true).

[cgillum]

Couldn't the SDK simply call StreamInstanceHistory if it receives a work item without history instead of fetching a new work item? This is what we do currently when fetching work items from DTS. In that case, DTS will omit history if it can't be fit in the 4 MB gRPC message size limit, but I imagine you could use the same technique to handle you extended session case.

[sophiatev]

Couldn't the SDK simply call StreamInstanceHistory if it receives a work item without history instead of fetching a new work item? This is what we do currently when fetching work items from DTS. In that case, DTS will omit history if it can't be fit in the 4 MB gRPC message size limit, but I imagine you could use the same technique to handle you extended session case.

Discussed with Chris offline and decided that for now this approach is fine. In order for the worker to call StreamInstanceHistory we would need a way to get a client to the GrpcOrchestrationRunner. I am not sure how to do this, but if it is more easy/doable than we thought, then having the worker call StreamInstanceHistory might indeed be a better approach.

The potential issue with the current approach is that the SessionAbortedException that is thrown is visible to customers and may raise some concerns. The worker will evict an extended session after the user-specified extendedSessionIdleTimeoutInSeconds expires. More specifically, what this means is that if the extended session is not accessed within that timeframe, the worker will evict it from the cache. In the meantime, it has to send the result of the orchestration work item back to the host, the host has to process this and commit it to storage, then wait for new orchestration messages, then send the worker the next work item once new messages arrive. From the host's perspective, the extendedSessionIdleTimeoutInSeconds applies only to the amount of time we wait for new orchestration messages to arrive before evicting the extended session from the cache. All that to say, there could maybe be an appreciable number of situations where the worker ends the extended session before the host and will require a history, and so the SessionAbortedException will be thrown. On top of that, there is also of course an added delay that comes from having DT.Core abort the orchestration work item entirely and try it again later in this approach. But perhaps the number of situations where this occurs could be reduced by just advising customers to up their extendedSessionIdleTimeoutInSeconds count in the isolated model.