Conversation
The wait for a free HTTP runspace before a request is shed with 503 was a hardcoded TimeSpan.FromSeconds(30) at both checkout sites in PowerShellRunnerService. Surface it as Worker:HttpQueueTimeoutSeconds with a CRAFT_HTTP_QUEUE_TIMEOUT env override, resolved once via CraftHostBuilderExtensions.ResolveHttpQueueTimeout (env > setting > built-in 30s default), mirroring how MinThreads resolves. This is a load-shedding bound, not a capacity knob.
The rate limiter set Retry-After on a throttled request but logged nothing, so every 429 went out invisible to operators. Emit a warning from OnRejected naming the partition the limit fired for (authenticated principal, else client address) plus the method, path and Retry-After. The logger is resolved per rejection (service registration has no built provider yet) and any logging failure is swallowed so it can never turn a throttle into a 500.
…limiter Add an optional per-client concurrency cap on app-only API callers so one automation cannot hold every runspace at once and starve the interactive UI, which is never capped. Config RateLimit:ApiConcurrencyLimit (env override CRAFT_API_CONCURRENCY_LIMIT), 0 = off by default. Because the limiter's lease spans the whole downstream pipeline, a permit covers both the wait for a runspace and execution — so the cap counts in-flight and queued-for-a-worker requests alike. Over-limit is rejected immediately with 429 (QueueLimit 0), reusing the existing Retry-After + log path. Callers are classified by a new, tested CallerClassifier (idp=aad plus a GUID AppId principal). The per-client rate limiter and the concurrency cap are built as chained partitioned limiters, and the middleware now runs whenever either is active.
The image bakes a DOTNET_GCHeapHardLimit sized for the smallest tier, and the CLR consumes it before any managed code runs - so larger tiers were stuck with the small tier's heap cap. Let a SkuProfile carry an optional GCHeapHardLimitMB and apply it on the matched profile at startup through AppContext.SetData + GC.RefreshMemoryLimit (.NET 8), under the same best-effort contract as pool sizing: any refusal logs and keeps the baseline rather than failing startup.
…alse-failing jobs
Under a pegged GC hard limit, an OutOfMemoryException thrown while logging inside
the JobQueuePump and JobManager catch-all blocks escaped ExecuteAsync. With the
host's default BackgroundServiceExceptionBehavior.StopHost that faults the
service and restarts the container mid-run: dispatch stops with work still
queued, and the pending backlog waits for the restart to reclaim its leases.
Guard those two log calls the way BackgroundTaskLimiter already guards its own
("logging is never worth the loop") so an allocation failure in logging can no
longer take the host down.
Separately, jobs still in flight when a shutdown began finished their work and
persisted their data, then threw ObjectDisposedException enqueueing their
terminal status because OrchestratorStatusWriter._signal was already disposed --
so a task that fully succeeded was recorded as Failed. Route every drain-loop
wake through a guarded Signal() that drops the wake once disposed. A lost
coalesced status wake on the way down is harmless; a succeeded task marked Failed
is not.
Introduce opt-in startup telemetry with configurable endpoint, app ID, timeouts, and jittered delays. Register a hosted service that emits one guarded boot report per instance using persisted storage state, and add version stamping plus native catalog injection so reports include meaningful build and surface metadata.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several new configuration options and infrastructure improvements to enhance resource management, telemetry, and rate limiting in the application. The main changes include adding support for startup telemetry reporting, implementing a per-client API concurrency cap to prevent automation clients from monopolizing resources, and enabling dynamic GC heap sizing based on SKU profiles. Additional improvements clarify and extend configuration settings for worker pools and request handling.
Resource Management and Limits
ApiConcurrencyLimit) toRateLimitSettings, with environment variable override and logic to ensure only app-only API clients are limited, preventing automation from starving interactive users. Includes methods to resolve the effective limit and determine if the limiter middleware should run. [1] [2] [3] [4]HttpQueueTimeoutSecondsproperty toWorkerSettingsand a resolution method, allowing configuration (and environment override) of how long HTTP requests wait for a free worker before being rejected. [1] [2]Telemetry
TelemetrySettingsand integrated it intoCraftSettingsto support opt-in startup phone-home telemetry, including configuration for endpoint, app ID, storm guard, and security token. Registered theStartupTelemetryServicefor emission. [1] [2] [3] [4]GC Heap Management
GCHeapHardLimitMB) inSkuProfile, with a newGcHeapLimithelper that applies the limit at runtime using .NET 8 APIs. This allows larger SKUs to increase GC heap size post-startup, improving performance on higher tiers. [1] [2]Build and Versioning
0.0.0-dev) inDirectory.Build.propsto ensure meaningful version stamping for local builds and proper reporting in telemetry.These changes collectively improve operational control, observability, and robustness of the application, especially in multi-tenant and cloud-hosted scenarios.