[luthermonson]

Fixes the two remaining defects behind the 2026-09-01 Windows-node wedge, where one dead containerd shim produced two symptoms and blocked two consecutive upgrade drains.

Incident

A runner env's teardown logged destroying runner environment and then nothing, forever — Runtime.Destroy was blocked on <-exitCh with no select and a context.Background() from every call site; when the shim is dead the exit event never arrives. That hung destroy pinned a MaxConcurrent slot for 1h+ (the wait-goroutine's defer <-s.sem runs after Destroy) while active_jobs had already been decremented — a node that looks idle-ish but can't admit work. Five seconds into the hang, the next env create failed with failed to create shim task: ttrpc: closed, and the create-failure path — unlike the claim-failure branch 15 lines above it — never calls enqueueRetryIfEligible, so the job sat queued for 1h43m (the only recovery is the 30-min reconcile loop, and a drain suppresses every re-dispatch route).

Fixes

Bounded Destroy with an abandon-to-reaper contract (pkg/runtime). Destroy now derives WithTimeout (new [runtime] destroy_timeout, default 5m; zero-value config fails safe to the default). Teardown steps run on their own goroutine; on expiry Destroy logs and returns, abandoning whatever remains — which is exactly what the v0.2.8 dead-container reaper (ReapDeadContainers) and startup CleanOrphans exist to reclaim. The kill-wait itself is separately bounded (30s, waitTaskExit — mirroring the select Runtime.Wait already used 20 lines below) so a missing exit event can't eat the whole budget and starve later steps that would still succeed. A late-finishing teardown still logs runner environment destroyed for the operator trail.

Slot release decoupled from teardown (pkg/scheduler). Both wait-goroutines (local sem, linuxSem) release the concurrency slot immediately after the job is untracked, before Destroy — teardown can no longer delay admission at all. Safe against create/teardown races: every env is keyed by its unique claim.RunnerName (container ID, runner dir, netns/HCN endpoint, dind socket all derive from it). This also unblocks handleCompleted's stranded defer sweepOrphanRunners().

Create failures now retry (pkg/scheduler). In place: 2 extra attempts at 5s/15s for transiently-classified errors (ttrpc: closed → errUnknownRetryable, pinned by test) — the slot is held and the JIT claim minted, so immediate retry is far cheaper than release-and-requeue; short-circuits on drain (checked before and after each sleep) and ctx cancellation. On exhaustion: the same cleanup as before, then enqueueRetryIfEligible onto the existing backoff queue — mirroring the claim branch — in both the local and Linux-dispatch paths.

Windows busy-probe now honors its timeout (pkg/runnerbusy). The hcsshim calls discarded their context.Context, so busyProbeTimeout was not enforced on Windows — the exact wedged-shim-blocks-the-sweep case PR #151 introduced the timeout for. Calls moved to a goroutine with a select; a wedged probe yields Unknown (fail-safe veto).

Deliberately skipped: narrowing network_windows.go's w.mu (shared setup/teardown mutex) — the mutex keeps IPAM allocation and HCN endpoint creation atomic, and per-endpoint locking is a subsystem rewrite; with Destroy bounded, a wedged HNS teardown can delay (not deadlock) the next setup.

Composes with #151: no changes to busy.go veto/nomination; the reap path's destroy gets the same bound via Runtime.Destroy itself.

Testing

• destroy_test.go: an exitCh that never delivers cannot block past the bound; ctx/timer paths covered
• retry_create_test.go: in-place ladder attempts/backoff, drain short-circuit (before and mid-sleep), non-retryable short-circuit, ttrpc: closed classification pinned
• runnerbusy: wedged probe returns within ctx deadline
• go build/go vet clean; go test green across scheduler/runtime/config/runnerbusy (-race not runnable on this box — no cgo; a Linux race run would be welcome belt-and-braces)

[luthermonson]

Adversarial review completed (agent-driven; concurrency, locking, and #151-composition traced end-to-end). The scheduler core came back clean — slot-release single-caller proof, lock balance, ID-keyed teardown safety, retry ctx/drain semantics, and the reap-path interaction with #151 all CONFIRMED correct. The destroy bound itself had holes; all fixed in ee70361:

• Hook outside the bound (MAJOR): OnTaskDestroy ran synchronously before the bounded goroutine — and on Windows it's an HCS Close(), exactly the wedge class this PR defends against. Now runs inside the teardown goroutine.
• Cancel-on-return killed the background teardown (MAJOR): destroySteps shared the caller-bounded ctx, so the moment Destroy returned on timeout, the deferred cancel made every remaining ctx-aware step fail instantly with context.Canceled — "continues in the background" was false. The goroutine now gets a detached ctx (2× DestroyTimeout) so an abandoned teardown can actually finish late.
• Reaper couldn't converge on the abandoned case (MINOR): Task.Delete in destroySteps lacked WithProcessKill; a dead-process/lost-exit-event task failed the delete with FailedPrecondition and the pinned snapshot survived until restart. Now matches teardownContainer.
• TOML footgun (MINOR, verified live): destroy_timeout = 300 decodes as 300 nanoseconds with no error — instant abandon on every teardown. ResolvedDestroyTimeout now floors sub-second values to the default.
• Deadline race (NIT): a teardown finishing at the buzzer could be mis-logged as abandoned; the wait now prefers done when both are ready (destroyWaitCompleted, unit-tested with a 100-iteration anti-coin-flip case).
• The abandonment contract comment now states honestly that leaked HNS endpoints/pool IPs are reclaimed by nothing (restart re-adopts + re-reserves; pool headroom absorbs it) — known residual, accepted over holding the scheduler hostage. An endpoint reaper is possible follow-up work.

Acknowledged, not addressed here: no integration-level test pins slot-release-before-destroy or the new enqueue call sites (the pure-function ladders are pinned; a scheduler-harness test with a blocking Destroy is the right follow-up), and teardownContainer's own <-exitCh retains the pre-existing unbounded receive (startup-only path; worth a follow-up sweep). -race still not runnable on this Windows box — CI on Linux should run it.