Fix/memory retention and orchestration priority usage - #36
Merged
Conversation
PowerShell.Create(iss) assigns the runspace rather than creating it lazily, and an assigned runspace is caller-owned: PowerShell.Dispose() does not close it. Every recycled worker therefore left its runspace open, and with ReuseRunspaceThread each open runspace keeps a dedicated pipeline thread alive, rooting the entire session state (every SSFE-injected function of every module) through any GC, however aggressive. Measured in production: ~20 MB retained per recycled worker at RecycleAfterInvocations=1000, i.e. ~20 KB of apparently-leaked heap per invocation, ~2 GB after 95 recycles over 3.5 days, growing until the platform kills the container. Dispose now captures the runspace, disposes the PowerShell object, then disposes the runspace, which closes it and reclaims the pipeline thread. The recycle path, pool shutdown, and the throwaway base worker all funnel through Worker.Dispose(), so one change covers all three.
…forever
Four ways a run could stay in _activeRuns (pinning its whole task graph in memory) for the process lifetime, plus the residue those paths leave behind:
- A lost decrement was permanent. DecrementRemainingAsync exhausting its optimistic retries returned null, the batch writer ignored it, and nothing ever reconciled Remaining against the task rows, so a fully-terminal run deferred finalize on every 60s tick forever ("complete in memory but storage shows N outstanding"). The store can now recount the partition (ReconcileRemainingAsync), the batch writer invokes it whenever a decrement is lost, and CheckRunCompletion recounts after 3 consecutive deferrals of a fully-terminal run and finalizes when the recount says done.
- The re-drive loop was infinite for a task storage keeps rejecting. Each pass reset the deferral counter and never incremented AttemptCount, so the fail-after-3 rule could never fire. Consecutive re-queue failures are now counted (cleared on success) and the task fails terminally after 5; an exhausted deferral cycle now counts as one attempt and the task fails terminally after 3 cycles. Terminal failure flows through the status writer, so the counter decrements and the run can finish without the poison task.
- StartOrResumeRun fell through after finalizing a resumed run and re-created it under the same name, which the finalize''s own post-execution then deleted via CleanupRunAsync, wiping the new run''s rows mid-flight. Finalize now returns; the next scheduler tick starts the fresh outing cleanly.
- The status-timer ContainsKey/TryAdd race leaked the losing Timer undisposed. An active periodic Timer is rooted by the runtime timer queue, so it fired every 60s and pinned the run graph through its closure for the process lifetime. The loser is now disposed.
Run names are also sanitized through TableKeys at dispatch (bridge and StartFromBatch). Names built from user-typed task names ("Alert on Entra ID P1/P2 license over-utilization") put table-illegal characters into PartitionKeys, which 400s every write for the run identically forever: the run could neither start nor be re-driven, and the scheduled task behind it silently never ran.
Finalize now also sweeps the per-task deferral and re-queue tracking entries, which otherwise outlive their run.
Propagate ambient run priority through OperationContext so nested orchestrator starts and post-exec jobs keep the parent run's priority instead of falling back to the default band. This also adds an explicit-priority QueueBridge overload while preserving the legacy P5 default, updates persisted orchestrator defaults to P4 to match live enqueue behavior, and adds coverage for descriptor jobs, closure jobs, reprioritization, and QueueBridge compatibility.
Zacgoose
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.