Harden runner replacement during GitHub outages

[joecklau]

Enforce a strict physical pool cap with lifecycle-aware reconciliation, rollback, quarantine, controller locking, and exponential dependency backoff. Pass registration tokens through stdin, sanitize provider diagnostics, and write private error reports atomically.

[Copilot]

Pull request overview

This PR hardens EPAR’s runner replacement and reconciliation logic so GitHub outages or transient failures don’t trigger runaway provisioning, while also improving secret handling (stdin token passing + pervasive redaction) and making private error reporting safer and atomic.

Changes:

• Enforce a strict physical pool.instances cap with lifecycle-aware reconciliation/quarantine plus transient dependency backoff and controller locking.
• Pass runner registration tokens via stdin (not env), and add sensitive-value redaction for provider execution logs/results/errors.
• Atomically write private error reports with restricted permissions; document and expose replacement retry configuration.

Reviewed changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/guest/ubuntu/configure-runner.sh	Reads registration token from stdin; supports configurable runner dir.
internal/provider/wsl/wsl.go	Adds sensitive-value redaction pipeline for WSL exec.
internal/provider/wsl/wsl_test.go	Tests redaction behavior for WSL exec results/sinks/errors.
internal/provider/tart/tart.go	Adds sensitive-value redaction pipeline + injectable runner for tests.
internal/provider/tart/tart_test.go	Tests redaction behavior for Tart exec results/sinks/errors.
internal/provider/redaction.go	Implements redaction utilities and sink buffering/streaming behavior.
internal/provider/redaction_test.go	Unit tests for redaction, buffering, and error wrapping.
internal/provider/provider.go	Extends ExecOptions with SensitiveValues for downstream providers.
internal/provider/dockerdind/docker_dind.go	Adds sensitive-value redaction pipeline for Docker-DinD exec.
internal/provider/dockerdind/docker_dind_test.go	Tests redaction behavior for Docker-DinD exec results/sinks/errors.
internal/pool/startup_timing.go	Uses shared redaction for timing error sanitization.
internal/pool/startup_timing_test.go	Verifies timing error sanitization redacts secret assignments.
internal/pool/runner_script_test.go	Tests configure-runner stdin token path; improves bash selection in tests.
internal/pool/manager.go	Adds lifecycle phases, strict cap reconciliation, outage backoff, and pool controller locking.
internal/pool/manager_test.go	Extensive tests for new lifecycle/reconciliation/backoff and stdin token handling.
internal/pool/host_trust.go	Marks busy old-generation runners as draining to respect strict cap.
internal/pool/controller_lock.go	Implements pool controller lock keyed by canonical config/provider/prefix.
internal/pool/controller_lock_test.go	Verifies controller lock conflict/independence and lock enforcement.
internal/logging/replace_windows.go	Uses MoveFileExW for atomic replace on Windows (see review comment).
internal/logging/private_file.go	Adds atomic private-file writer enforcing restrictive permissions.
internal/github/client.go	Captures Retry-After on HTTP errors and parses header values.
internal/github/client_test.go	Unit tests for Retry-After parsing.
internal/config/config.go	Adds replacement retry policy fields + validation for pool config.
internal/config/config_test.go	Tests loading/validation of replacement retry settings.
docs/usage.md	Documents strict physical cap and outage backoff behavior.
docs/operations.md	Adds operational guidance for reconciliation/outage recovery and controller locking.
docs/design.md	Documents lifecycle phases, reconciliation model, and transient backoff behavior.
docs/configuration.md	Documents new pool replacement retry settings and strict cap semantics.
configs/wsl.web-e2e.example.yml	Adds replacement retry settings to WSL web e2e example.
configs/wsl.lean.example.yml	Adds replacement retry settings to WSL lean example.
configs/wsl.example.yml	Adds replacement retry settings to WSL example.
configs/tart.web-e2e.example.yml	Adds replacement retry settings to Tart web e2e example.
configs/tart.example.yml	Adds replacement retry settings to Tart example.
configs/docker-dind.web-e2e.example.yml	Adds replacement retry settings to Docker-DinD web e2e example.
configs/docker-dind.example.yml	Adds replacement retry settings to Docker-DinD example.
configs/docker-dind.core.example.yml	Adds replacement retry settings to Docker-DinD core example.
configs/docker-dind.act.example.yml	Adds replacement retry settings to Docker-DinD act example.
cmd/ephemeral-action-runner/start.go	Acquires pool controller lock before starting the supervisor.
cmd/ephemeral-action-runner/main.go	Uses pool controller lock for image operations; writes redacted private error reports atomically.
cmd/ephemeral-action-runner/logging_test.go	Tests redacted + private-permission error report writing.
cmd/ephemeral-action-runner/init.go	Includes replacement retry settings in generated default configs.
cmd/ephemeral-action-runner/init_test.go	Verifies generated configs include replacement retry settings.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.