macOS: the provisioning-deadline self-heal reclaims the VM but LEAKS the concurrency slot — node goes silently dead #196

Impact

The mac node served zero macOS jobs for ~28 hours while reporting status: ok, active_jobs: 0, /healthz 200, and a healthy uptime. Every macOS job queued in that window sat until GitHub cancelled it at the 24h queue timeout. Nothing in mayfly node status or the tunnel health check reveals it — you have to read the log to see the node is a black hole.

What happened (from /var/log/ephemerd.log, node 192.168.16.168, v0.2.10)

14:08:06 provisioning macOS VM runner for job   job_id=100817345293 repo=php-sdk
14:08:08 booting macOS VM                       id=100817345293
14:08:08 waiting for macOS VM runner to become reachable
14:13:08 ERROR macOS VM stuck in provisioning past deadline; force-stopping VM to reclaim the slot  timeout=5m0s
14:13:23 macOS VM destroyed                     id=100817345293
   ... 28 hours of ONLY "GitHub App token refreshed" + cloudflared lines ...
   ... macOS jobs queued 15:36, 15:42, 16:05, 16:31, 16:34 — no provisioning attempt for ANY of them ...
2026-09-04T14:08:07 webhook event action=completed job_id=100817345293 runner_name=""   <-- GitHub's 24h queue timeout, never ran

The VM boot failure at 14:08 was a one-off transient — after a daemon restart the very same path provisioned in 54s (SSH reachable 10s → runner copied → macOS VM runner ready). The transient is not the bug.

Root cause

The deadline handler destroys the VM and logs "to reclaim the slot", but the scheduler's concurrency semaphore is never released. With the mac pool at max_concurrent = 1, every subsequent macOS job blocks forever on a slot that will never free. Because the job was already untracked, len(s.running) is 0 — so active_jobs reports 0 and the node looks idle and healthy while being incapable of accepting work. No log line is emitted when a job blocks on the semaphore, which is why the log goes completely silent.

This is the same failure shape as #190 (slot release coupled to teardown on the local path), on the macOS dispatch path, and it defeats the self-heal added in #178 — the mechanism intended to make a stuck VM recoverable is exactly what makes it permanent.

Suggested fix

Release the slot on the deadline/force-stop path, the same way #190 restructured the local wait-goroutine: release once the job is untracked, never coupled to teardown completing. Worth auditing every return in the macOS dispatch path for the same leak, and adding a regression test that a provisioning-deadline timeout leaves the pool able to admit the next job.

Two observability gaps worth closing alongside it, since they're what hid this for 28 hours:

• active_jobs counts tracked jobs but not held slots — a node with all slots leaked reports 0. Consider exposing held-slots/capacity in status and /healthz.
• A job blocking on the semaphore logs nothing. A single "waiting for a free slot" line (rate-limited) would have made this instantly visible.

Workaround

Restart ephemerd on the node (launchctl kickstart -k system/dev.ephpm.ephemerd); the startup poll then picks up the stranded queue.