Conversation
PowerShell.Create(iss) assigns the runspace rather than creating it lazily, and an assigned runspace is caller-owned: PowerShell.Dispose() does not close it. Every recycled worker therefore left its runspace open, and with ReuseRunspaceThread each open runspace keeps a dedicated pipeline thread alive, rooting the entire session state (every SSFE-injected function of every module) through any GC, however aggressive. Measured in production: ~20 MB retained per recycled worker at RecycleAfterInvocations=1000, i.e. ~20 KB of apparently-leaked heap per invocation, ~2 GB after 95 recycles over 3.5 days, growing until the platform kills the container. Dispose now captures the runspace, disposes the PowerShell object, then disposes the runspace, which closes it and reclaims the pipeline thread. The recycle path, pool shutdown, and the throwaway base worker all funnel through Worker.Dispose(), so one change covers all three.
…forever
Four ways a run could stay in _activeRuns (pinning its whole task graph in memory) for the process lifetime, plus the residue those paths leave behind:
- A lost decrement was permanent. DecrementRemainingAsync exhausting its optimistic retries returned null, the batch writer ignored it, and nothing ever reconciled Remaining against the task rows, so a fully-terminal run deferred finalize on every 60s tick forever ("complete in memory but storage shows N outstanding"). The store can now recount the partition (ReconcileRemainingAsync), the batch writer invokes it whenever a decrement is lost, and CheckRunCompletion recounts after 3 consecutive deferrals of a fully-terminal run and finalizes when the recount says done.
- The re-drive loop was infinite for a task storage keeps rejecting. Each pass reset the deferral counter and never incremented AttemptCount, so the fail-after-3 rule could never fire. Consecutive re-queue failures are now counted (cleared on success) and the task fails terminally after 5; an exhausted deferral cycle now counts as one attempt and the task fails terminally after 3 cycles. Terminal failure flows through the status writer, so the counter decrements and the run can finish without the poison task.
- StartOrResumeRun fell through after finalizing a resumed run and re-created it under the same name, which the finalize''s own post-execution then deleted via CleanupRunAsync, wiping the new run''s rows mid-flight. Finalize now returns; the next scheduler tick starts the fresh outing cleanly.
- The status-timer ContainsKey/TryAdd race leaked the losing Timer undisposed. An active periodic Timer is rooted by the runtime timer queue, so it fired every 60s and pinned the run graph through its closure for the process lifetime. The loser is now disposed.
Run names are also sanitized through TableKeys at dispatch (bridge and StartFromBatch). Names built from user-typed task names ("Alert on Entra ID P1/P2 license over-utilization") put table-illegal characters into PartitionKeys, which 400s every write for the run identically forever: the run could neither start nor be re-driven, and the scheduled task behind it silently never ran.
Finalize now also sweeps the per-task deferral and re-queue tracking entries, which otherwise outlive their run.
Propagate ambient run priority through OperationContext so nested orchestrator starts and post-exec jobs keep the parent run's priority instead of falling back to the default band. This also adds an explicit-priority QueueBridge overload while preserving the legacy P5 default, updates persisted orchestrator defaults to P4 to match live enqueue behavior, and adds coverage for descriptor jobs, closure jobs, reprioritization, and QueueBridge compatibility.
Fix/memory retention and orchestration priority usage
AsyncLocal values set per-invocation never reach the runspace's reused thread (ReuseThread), whose ExecutionContext is frozen at pool warmup. PowerShell code reading OperationContext.Current always saw null. Add StampOperationContext() in PowerShellWorker, called before each invocation, which writes OperationContext.Current into $global:CraftOperationContext from the calling thread (which does hold the AsyncLocal). CleanupGlobalVariables removes it post-invocation to prevent stale values across checkouts. Update Start-CraftOrchestrator priority resolution and OperationContext XML docs to reference the global variable. Add a regression test covering stamp, value visibility, and cleanup.
…ration PS-queued child runs (e.g. DomainAnalyser fan-out) lost their lineage because OrchestratorBridge read OperationContext.Current on the pipeline thread, whose frozen ExecutionContext never sees the per-invocation AsyncLocal. Parents finalized and dispatched PostExecution while children were still running. Fixes: - Start-CraftOrchestrator reads RunName from $global:CraftOperationContext and passes it explicitly to QueueOrchestration/QueueOrchestrationFromFile. - Bridge resolves parent via explicit arg first, ambient AsyncLocal as fallback for .NET callers. - Child gate is taken at ENQUEUE time (TryRegisterPendingChildRun) via a counted _pendingChildRuns dict, released in DrainPending's finally block — closing the race where the parent's last task completes before the drain makes the child visible in _activeRuns. - Startup reattachment skips lineage pointing at already-finalized parents, preventing phantom gates on recurring runs. - Docs updated; new OrchestratorBridgeLineageTests added; OrchestratorChildRunGuardTests extended.
The lint gate (dotnet format --verify-no-changes) rejects an alias using placed between plain usings; the formatter orders aliases last.
KelvinTegelaar
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant improvements to how orchestration and queueing handle job priorities and parent-child run relationships. The changes ensure that job priorities are consistently inherited and applied, and that child runs are correctly registered and tracked, preventing race conditions where parent runs might finalize before their children complete. Additionally, the documentation and parameter handling across PowerShell and C# bridges have been clarified and made more robust.
Priority and Parent Run Handling Improvements:
The orchestrator bridge and job manager now explicitly handle and propagate queue priorities and parent run names, ensuring that child orchestrations inherit the correct priority and maintain accurate lineage, even when invoked from PowerShell where ambient context is unavailable. [1] [2] [3] [4] [5] [6]
The default priority for orchestrator runs has been changed from 2 (user-initiated/high) to 4 (background/normal), aligning with actual enqueue behavior and preventing rehydrated runs from being assigned a higher priority than intended. [1] [2]
Queue and Orchestrator Bridge Enhancements:
The queue bridge now supports explicit job priorities, allowing callers to specify them directly. Backward compatibility is maintained for callers that do not specify a priority. [1] [2] [3]
The orchestrator bridge now registers pending child runs at enqueue time (rather than after start), preventing race conditions where a parent could finalize before its children are visible. The pending-child gate is always released after the start attempt, ensuring no gates are leaked. [1] [2] [3] [4]
Documentation and Code Clarity:
Summary of Most Important Changes:
Priority and Parent Run Propagation:
Start-CraftOrchestrator.ps1) now reads and passes priority and parent run name explicitly from the stamped$global:CraftOperationContext, ensuring correct inheritance and lineage for nested orchestrations. [1] [2]OrchestratorBridge.cs) and job manager (JobManager.cs) updated to accept and propagate explicit priority and parent run name parameters, with improved logic for resolving and sanitizing these values. [1] [2] [3] [4] [5]Child Run Registration and Finalization:
Priority Defaults and Levels:
Queue Bridge and Backward Compatibility:
QueueBridge.cs) now supports both default and explicit priorities, maintaining backward compatibility with pre-priority callers. [1] [2] [3]Documentation and Code Comments: