feat: orchestration and table optimisations - #32
Merged
Merged
Conversation
Push job-claim filtering down into table queries so the pump only scans claimable rows instead of paging the full backlog on each tick. Add exponential idle poll backoff that resets whenever the pump claims or still holds work, preventing unnecessary storage scans without throttling active queue throughput. Includes Azurite coverage for the server-side filter and new pump backoff regression tests.
Add a shared `TableKeys` utility to detect and sanitize Azure Table-invalid key characters, then apply it when minting orchestrator task IDs before uniqueness checks. This prevents permanent enqueue 400s caused by names like `Owner/Repo` and preserves distinct tasks when lossy sanitization would otherwise collide. Includes focused tests for illegal characters, control chars, safe-pass behavior, repo-name task IDs, collision handling, and queue row-key safety.
Add a shared server-side RunName filter for run-scoped queue operations so cleanup, lease release, and queued-id lookups stop scanning the entire table. This preserves existing client-side checks for correctness while reducing storage load, and adds Azurite coverage to verify cross-run isolation and quoted run names.
JohnDuprey
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and optimizations to the orchestration and storage layers, focusing on job queue efficiency, key safety for Azure Table storage, and server-side filtering for better scalability. The most significant changes include implementing exponential backoff for idle job queue polling, adding server-side filtering to reduce unnecessary data transfer, and introducing a utility for sanitizing table keys to prevent backend errors.
Job Queue Efficiency and Polling Improvements:
JobQueuePumppolling interval when the queue is idle, using a new_idlePollIntervaland a doubling strategy to reduce unnecessary polling and backend load during idle periods. The delay resets when work appears, ensuring responsiveness. (Services/Orchestration/JobQueuePump.cs[1] [2] [3] [4] [5]Azure Table Storage Key Safety:
TableKeysutility class to sanitize partition and row keys for Azure Table storage, replacing illegal characters with underscores to prevent persistent backend errors. This is now used when generating task IDs for storage. (Services/Storage/TableKeys.cs[1]Services/Orchestration/OrchestratorService.cs[2] [3]Server-side Filtering and Storage Optimization:
$filterexpressions inQueryTableAsyncfor Azure Table storage, allowing server-side filtering of claimable jobs. The filter is used in the job claim path to avoid paging the entire backlog to the client, significantly improving efficiency for large queues. (Services/Storage/ICraftTableStore.cs[1]Services/Storage/AzureTableStore.cs[2]Services/Storage/JobQueueStore.cs[3] [4]Testing and Validation:
tests/Craft.Tests/JobQueueAzuriteTests.cstests/Craft.Tests/JobQueueAzuriteTests.csR65-R96)