Conversation
CleanupOldRunsAsync only ran from the startup recovery pass, only removed Completed/CompletedWithErrors/Failed runs that carried a CompletedUtc, and walked nothing but the Run partition. On a host that is not restarted the three tables grew without bound, and even across restarts Cancelled runs, runs nobody was driving, and Tasks/Results partitions whose Run row was already gone lived forever. The sweep now removes every terminal status (Cancelled included) once past retention, treats a non-terminal run as abandoned when it is not in _activeRuns and nothing about it - its !!run-counter heartbeat, Run row or StartedUtc - has been written within retention, and removes Tasks/Results partitions with no Run row once their newest row is past retention. The orphan scan reads keys and Timestamp only, through a new projected QueryTableAsync overload, so a Results partition's payload is never pulled back just to learn its age. Abandoned runs also lose their durable queue rows. It runs at the end of startup recovery and then every Orchestrator:CleanupIntervalHours (default 4) with Orchestrator:RetentionHours (default 48; it was a fixed 7 days). Both keys are documented in appsettings.example.jsonc and docs/configuration.md and pinned by ConfigurationReferenceTests. OrchestratorRetentionTests covers each rule against the in-memory store; OrchestratorRetentionAzuriteTests proves the projection and the partition deletes against real tables.
…ound it gone A table deleted out from under a live host - through table maintenance, or by a reset that cleared the orchestrator's state - took every write to it down until the next restart, and silently: the batch path's per-entity fallback failed the same way and swallowed each failure, so status writes simply vanished. AzureTableStore now treats 404 TableNotFound (as opposed to a row's ResourceNotFound) as the signal to create the table and run the operation again: single and batch upserts retry, reads and the partition scan start over against the fresh table, and a conditional claim returns false after bringing the table back since its rows cannot exist any more. EnsureTableAsync shares the create path. On the real service a just-deleted table refuses creation with 409 TableBeingDeleted for about a minute (measured 62 s; Azure documents at least 40 s), and during that window it still accepts and then discards writes, so the create polls every 3 s for up to 2 minutes under the caller's cancellation token. A recreate logs a warning naming the table. AzureTableStoreRecreateTests proves each operation kind against a never-created table, and one test gated on CRAFT_TEST_INCLUDE_SLOW=1 proves the deletion window is ridden out against a real account.
JohnDuprey
approved these changes
Aug 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.