Prepare v0.3.1 release

AGENTS.md

@@ -8,7 +8,7 @@ A provider or onboarding change is incomplete until its wizard and generated con
 
 ## Agent-created temporary files
 
-Do not create ad hoc top-level directories under `.local`. All scratch files, temporary build output, one-off test state, and agent-specific caches created while investigating, implementing, or validating work must live under `.local/tmp/`, grouped beneath a descriptive task or tool directory such as `.local/tmp/codex/<task-name>/`. This includes manually assigned `GOCACHE`, `GOTMPDIR`, `GOMODCACHE`, test state roots, downloaded diagnostic tools, generated fixtures, and transient command output.
+Do not create ad hoc top-level directories under `.local`. All scratch files, temporary build output, one-off test state, and agent-specific caches created while investigating, implementing, or validating work must live under `.local/tmp/`, grouped beneath a descriptive task or tool directory such as `.local/tmp/<agent-name>/<task-name>/`. This includes manually assigned `GOCACHE`, `GOTMPDIR`, `GOMODCACHE`, test state roots, downloaded diagnostic tools, generated fixtures, and transient command output.
 
 Do not place agent-created temporary content in the product-owned `.local/bin`, `.local/cache`, `.local/state`, or `.local/storage` trees unless the product code being exercised creates that exact documented path as part of its normal behavior. Never introduce another top-level `.local/<tool-name>` or `.local/<purpose>` directory for convenience. Prefer the operating system temporary directory when the artifact does not need to remain with the checkout.
 

README.md

@@ -2,6 +2,8 @@
 
 EPAR keeps a warm pool of disposable GitHub Actions runners. Each runner handles one job inside a dedicated [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/) microVM with a private Docker daemon, then is replaced with a clean runner.
 
+**EPAR is for teams that want to use their own compute for CI without running GitHub Actions jobs directly on the host.**
+
 ![Ephemeral Action Runner banner](docs/assets/brand/epar-banner.jpg)
 
 ```mermaid
@@ -14,13 +16,17 @@ flowchart LR
 
 ## Why EPAR
 
-- Keep private-repository CI ready without maintaining a long-lived runner workspace.
-- Recycle the runner, its private daemon, and job state after every job.
-- Run Docker-based Linux jobs from Linux, macOS, or Windows hosts when Docker Sandboxes capability checks pass.
+- **Put spare compute to work** — run long-running E2E, integration, and Docker-heavy CI on machines you already operate.
+- **Add a strong isolation boundary around CI jobs** — each Docker Sandboxes runner runs inside a dedicated microVM rather than directly on the host.
+- **Keep Docker private to the runner** — Docker workloads use the sandbox's private daemon instead of the host Docker socket.
+- **Start clean after every job** — destroy the runner, private daemon, filesystem, and job state, then replace them with a fresh runner.
+- **Stay warm without keeping runner state around** — maintain ready-to-accept runners while preserving the one-runner, one-job lifecycle.
+- **Keep the infrastructure small** — run from Linux, macOS, or Windows hosts without introducing a separate cluster or orchestration platform just to manage CI runners.
 
 ## Quick Start
 
-This quick start uses Docker Sandboxes. Install Docker and the Docker Sandboxes `sbx` CLI, then confirm that `sbx diagnose --output json` reports at least one passing check and no failures. EPAR keeps the detailed helper, proxy, and build guidance in the linked deep guides.
+This quick start uses Docker Sandboxes. Install Docker and the [Docker Sandboxes `sbx` CLI](https://docs.docker.com/ai/sandboxes/).
+Make sure you ran `sbx` once, which will have a wizard to guide on first time setup for Docker Sandboxes (e.g. login), then run `sbx diagnose` to confirm all passes, no failures.
 
 1. Download GitHub's **Source code (zip)** or **Source code (tar.gz)** for the release you want from the [EPAR releases page](https://github.com/solutionforest/ephemeral-action-runner/releases), extract it, and open a terminal in the extracted folder.
 2. Create a GitHub App by following [GitHub App Setup](docs/github-app.md); have the App ID, organization name, and private-key file path ready.
@@ -30,14 +36,14 @@ This quick start uses Docker Sandboxes. Install Docker and the Docker Sandboxes
    ./start
    ```
 
-The first run opens a guided setup wizard for the GitHub App, runner group, Docker Sandboxes host checks, and runner image. Keep the process open while runners should accept work. Press `Ctrl-C` once to stop and wait for cleanup to finish before closing the terminal. See [Usage](docs/usage.md) for configuration, verification, no-Go startup, and cleanup details.
+The first run opens a guided setup wizard for the GitHub App, runner group, Docker Sandboxes host checks, and runner image. Just follow the wizard to setup and start EPAR. Keep the process open while runners should accept work. Press `Ctrl-C` once to stop and wait for cleanup to finish before closing the terminal. See [Usage](docs/usage.md) for configuration, verification, no-Go startup, and cleanup details.
 
 ## Route a workflow to EPAR
 
 Every EPAR runner has GitHub's `self-hosted` label. Add the EPAR Sandboxes label when a workflow should use this environment:
 
 ```yaml
-runs-on: [self-hosted, linux, epar-docker-sandboxes]
+runs-on: [self-hosted, linux]
 ```
 
 Use labels that describe the environment your job needs and keep runner groups limited to the repositories and secrets that require access.

docs/troubleshooting.md

@@ -5,6 +5,7 @@ Start with the symptom that most closely matches the failure. Regardless of prov
 ## Contents
 
 - [Quick diagnostics](#quick-diagnostics)
+- [Startup reports that the pool controller configuration lock is already held](#startup-reports-that-the-pool-controller-configuration-lock-is-already-held)
 - [Windows no-Go startup prints an HTTP/2 named-pipe diagnostic](#windows-no-go-startup-prints-an-http2-named-pipe-diagnostic)
 - [A Docker workload fails with an architecture error](#a-docker-workload-fails-with-an-architecture-error)
 - [Docker Sandboxes is unavailable or its preflight fails](#docker-sandboxes-is-unavailable-or-its-preflight-fails)
@@ -49,6 +50,42 @@ docker run --rm ghcr.io/catthehacker/ubuntu:full-latest df -h /
 
 Container-visible free space is the relevant value for Docker builds. Windows Explorer or Finder free space does not necessarily equal the free space in a Linux VM backing the daemon.
 
+## Startup reports that the pool controller configuration lock is already held
+
+### Symptom
+
+`./start` exits with an error like:
+
+```text
+ephemeral-action-runner: pool controller configuration lock is already held for "/Users/someone/epar/.local/config.yml" (owner config="/Users/someone/epar/.local/config.yml" provider="docker-sandboxes" prefix="example-pool" pid=12345 startedAt=2026-01-02T03:04:05Z)
+```
+
+This can appear after the startup wrapper rebuilds the native controller because its source digest changed. The rebuild prepares a new binary but does not replace an already-running controller, so the new process still stops at the ownership lock.
+
+### Diagnosis and remediation
+
+The lock prevents two mutating controllers from managing the same pool. Treat the reported config path, provider, prefix, PID, and start time as owner-identification evidence rather than deleting a lock file.
+
+On macOS or Linux, inspect the reported PID, replacing `12345` with the value from the error:
+
+```bash
+ps -p 12345 -o pid=,ppid=,lstart=,command=
+```
+
+On Windows PowerShell:
+
+```powershell
+Get-Process -Id 12345 -ErrorAction SilentlyContinue | Format-List Id, StartTime, Path
+```
+
+If the PID is the intended live EPAR controller, do not start another controller for that pool. Return to its terminal and press Ctrl-C once, or stop it through the service manager that launched it, then wait for `Cleanup complete. EPAR can now exit safely.` before running `./start` again. Use this sequence when activating copied source or a rebuilt controller: stop the old controller cleanly, wait for cleanup, then start the new version.
+
+If no process owns the reported PID, rerun `./start`. The operating-system lock is authoritative and is released when its owning process exits; stale descriptive metadata may remain on disk but is replaced on the next successful acquisition. Do not manually delete files under the controller-lock state directory, and do not signal a process solely because its PID matches stale metadata—PID values can be reused.
+
+If the exact EPAR controller is unresponsive, prefer its service manager's graceful stop or a single interrupt and allow time for cleanup. After a confirmed abnormal exit, rerun `./start` so normal lifecycle reconciliation can inspect and recover exact owned resources. Avoid force-killing the process, deleting runner records manually, or using broad Docker, Sandbox, WSL, or GitHub cleanup commands as a first response.
+
+Multiple EPAR controllers may run on one machine, but every concurrently active controller must use both a distinct canonical configuration path and a distinct normalized `pool.namePrefix`. Reusing either the same config path or the same pool prefix intentionally conflicts, including across different project directories or providers. Give each controller its own config and pool identity instead of bypassing the lock.
+
 ## Windows no-Go startup prints an HTTP/2 named-pipe diagnostic
 
 ### Symptom
@@ -182,7 +219,7 @@ EPAR does not query or mutate host-global `sbx` secrets. After creating its exac
 
 ## An idle runner reports GitHub or Sandbox health warnings
 
-A GitHub 429/5xx response or an `sbx` command timeout makes runner health temporarily unknown; it does not prove that the Actions listener stopped. EPAR keeps the exact runner, lets a trust lease expire closed when it cannot refresh it, and retries. Cleanup for an inactive listener requires two consecutive guest probes that successfully execute and explicitly report the process stopped. Review the instance guest transcript when warnings repeat; do not delete the runner merely because one API or Sandbox inspection failed.
+A GitHub 429/5xx response or an `sbx` command timeout makes runner health temporarily unknown; it does not prove that the Actions listener stopped. EPAR preserves uncertain local capacity, but a runner whose host-trust transport or lease cannot be maintained is quarantined and its exact GitHub registration is fenced after immutable name-and-ID verification so it cannot accept new work with an expired lease. Cleanup for an inactive listener requires two consecutive guest probes that successfully execute and explicitly report the process stopped. Review the instance guest transcript when warnings repeat; do not delete the runner merely because one API or Sandbox inspection failed.
 
 `networkBaseline: open` is a sandbox-scoped public-egress compatibility rule with EPAR host-alias deny guardrails. It does not alter the host-global policy. If a required service is blocked, use a narrow `additionalAllow` hostname rule; do not allow `host.docker.internal`, `gateway.docker.internal`, `kubernetes.docker.internal`, or `host.containers.internal` through the Open-policy guardrails.
 

internal/pool/host_trust.go

@@ -470,6 +470,32 @@ func (m *Manager) revokeHostTrustLease(ctx context.Context, instanceName string)
 	return err
 }
 
+func (m *Manager) fenceHostTrustRunnerRegistration(ctx context.Context, instance ProvisionedInstance, cause error) error {
+	// Admission uncertainty must be durable even when the exact remote fence
+	// cannot be completed. The physical instance remains capacity-counting and
+	// reconciliation must not advertise it as Ready again by name alone.
+	fenceCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), hostTrustWriteTimeout)
+	defer cancel()
+	m.quarantineLifecycle(fenceCtx, instance.Name, cause)
+	if m.GitHub == nil || instance.RunnerID == 0 {
+		return fmt.Errorf("exact GitHub runner identity is unavailable; cannot fence registration")
+	}
+	runner, found, err := m.GitHub.RunnerByName(fenceCtx, instance.Name)
+	if err != nil {
+		return fmt.Errorf("verify exact GitHub runner before host-trust fence: %w", err)
+	}
+	if !found {
+		return nil
+	}
+	if runner.ID != instance.RunnerID {
+		return fmt.Errorf("same-name GitHub runner id=%d does not match expected id=%d; refusing registration fence", runner.ID, instance.RunnerID)
+	}
+	if err := m.GitHub.DeleteRunnerIfExists(fenceCtx, runner.ID); err != nil {
+		return fmt.Errorf("delete exact GitHub runner id=%d: %w", instance.RunnerID, err)
+	}
+	return nil
+}
+
 func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[string]ProvisionedInstance, current hosttrust.Snapshot, busyHandoff map[string]bool) int {
 	if m.GitHub == nil {
 		return 0
@@ -487,23 +513,21 @@ func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[stri
 		if _, requiresTransport := m.providerLifecycle().(provider.HostTrustRuntimeActivator); requiresTransport {
 			providerInstance, providerErr := m.providerInstance(ctx, name)
 			if providerErr != nil {
-				revokeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), hostTrustWriteTimeout)
-				revokeErr := m.revokeHostTrustLease(revokeCtx, name)
-				cancel()
 				m.warnf("[%s] host trust transport identity warning; lease not refreshed: %v\n", name, providerErr)
-				if revokeErr != nil {
-					m.warnf("[%s] host trust transport identity fencing warning: %v\n", name, revokeErr)
+				if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, providerErr); fenceErr != nil {
+					m.warnf("[%s] host trust registration fencing warning: %v\n", name, fenceErr)
 				}
+				instance.Phase = LifecycleQuarantined
+				active[name] = instance
 				continue
 			}
 			if err := m.activateProviderHostTrustRuntime(ctx, providerInstance); err != nil {
-				revokeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), hostTrustWriteTimeout)
-				revokeErr := m.revokeHostTrustLease(revokeCtx, name)
-				cancel()
 				m.warnf("[%s] host trust transport refresh warning; lease not refreshed: %v\n", name, err)
-				if revokeErr != nil {
-					m.warnf("[%s] host trust transport refresh fencing warning: %v\n", name, revokeErr)
+				if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+					m.warnf("[%s] host trust registration fencing warning: %v\n", name, fenceErr)
 				}
+				instance.Phase = LifecycleQuarantined
+				active[name] = instance
 				continue
 			}
 		}
@@ -513,6 +537,12 @@ func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[stri
 			// the assignment window even when GitHub status is unavailable.
 			if err := m.issueHostTrustLease(ctx, name, current); err != nil {
 				m.warnf("[%s] old-generation revocation warning: %v\n", name, err)
+				if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+					m.warnf("[%s] host trust registration fencing warning: %v\n", name, fenceErr)
+				}
+				instance.Phase = LifecycleQuarantined
+				active[name] = instance
+				continue
 			}
 		}
 		runner, found, err := m.GitHub.RunnerByName(ctx, name)
@@ -538,6 +568,11 @@ func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[stri
 				// renew it while the job remains busy.
 				if err := m.issueHostTrustLeaseWithLifetime(ctx, name, current, hostTrustHandoffLease); err != nil {
 					m.warnf("[%s] host trust job handoff lease warning: %v\n", name, err)
+					if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+						m.warnf("[%s] host trust registration fencing warning: %v\n", name, fenceErr)
+					}
+					instance.Phase = LifecycleQuarantined
+					active[name] = instance
 					continue
 				}
 				busyHandoff[name] = true
@@ -546,6 +581,11 @@ func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[stri
 			delete(busyHandoff, name)
 			if err := m.issueHostTrustLease(ctx, name, current); err != nil {
 				m.warnf("[%s] host trust lease refresh warning: %v\n", name, err)
+				if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+					m.warnf("[%s] host trust registration fencing warning: %v\n", name, fenceErr)
+				}
+				instance.Phase = LifecycleQuarantined
+				active[name] = instance
 			}
 			continue
 		}
@@ -557,7 +597,7 @@ func (m *Manager) reconcileHostTrustRunners(ctx context.Context, active map[stri
 			continue
 		}
 		reason := fmt.Sprintf("host trust generation changed from %s to %s", instance.HostTrustGeneration, current.Generation)
-		if err := m.retireInstance(context.Background(), instance, reason); err != nil {
+		if err := m.retireInstance(ctx, instance, reason); err != nil {
 			m.warnf("[%s] old-generation retirement warning: %v\n", name, err)
 			continue
 		}
@@ -608,6 +648,11 @@ func (m *Manager) startHostTrustLeaseKeeper(parent context.Context) (func(Provis
 					if instance.HostTrustGeneration != current.Generation {
 						if err := m.issueHostTrustLease(ctx, name, current); err != nil {
 							m.warnf("[%s] host trust initial stale-generation revocation warning: %v\n", name, err)
+							if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+								m.warnf("[%s] host trust initial registration fencing warning: %v\n", name, fenceErr)
+							}
+							instance.Phase = LifecycleQuarantined
+							active[name] = instance
 						}
 						continue
 					}
@@ -617,6 +662,11 @@ func (m *Manager) startHostTrustLeaseKeeper(parent context.Context) (func(Provis
 					}
 					if err := m.issueHostTrustLease(ctx, name, current); err != nil {
 						m.warnf("[%s] host trust initial lease refresh warning: %v\n", name, err)
+						if fenceErr := m.fenceHostTrustRunnerRegistration(ctx, instance, err); fenceErr != nil {
+							m.warnf("[%s] host trust initial registration fencing warning: %v\n", name, fenceErr)
+						}
+						instance.Phase = LifecycleQuarantined
+						active[name] = instance
 					}
 				}
 			}