Add prefect worker start --background and prefect worker stop

[SreeramaYeshwanthGowd]

This PR adds a --background/-b flag to prefect worker start and a new prefect worker stop command, so workers can be launched as detached background processes and stopped later. It mirrors the existing prefect server start --background / prefect server stop behavior, and supports running several background workers at once, each stopped individually.

Behavior

• prefect worker start --background (alias -b) runs all of the normal pre-flight validation (work pool lookup, paused pool and paused queue warnings, worker type resolution, worker class load/install) in the parent process first, then spawns a detached background worker. Because validation runs before backgrounding, a misconfigured worker (push or managed pool, prefect-agent pool, uninstallable worker type) fails in the foreground with the correct error and exit code instead of reporting success and then exiting.
• Each background worker is tracked by name at $PREFECT_HOME/workers/<name>.pid, so multiple workers can run concurrently. When --name is omitted the CLI generates one (the worker otherwise names itself inside the child process, which the launching CLI cannot see) and prints it. Starting a worker whose name is already running is rejected; a stale PID file from a crashed worker is reclaimed automatically.
• The background worker is fully detached: stdout and stderr go to DEVNULL and it runs in its own session (start_new_session on POSIX, CREATE_NEW_PROCESS_GROUP on Windows), so it survives the CLI process exiting rather than dying on a broken pipe once the parent closes the pipe read ends.
• prefect worker stop [NAME] stops a single named worker; --all stops every background worker; with exactly one worker running, no argument stops it; when several are running, it lists them and asks for a name or --all. Stale or corrupt PID files are cleaned up rather than raising, and stopping when none are running exits 0 with a clear message.

Like prefect server start / prefect server stop, these commands emit human-readable output and do not add a --json mode.

Details

• src/prefect/cli/worker.py
  • Adds the --background/-b option to worker start; backgrounding happens only after the existing validation path succeeds. The worker name is resolved up front (generated from the worker type when --name is omitted) and tracked at $PREFECT_HOME/workers/<slug>.pid. A same-named running worker is rejected; a stale or corrupt file is reclaimed.
  • Adds the worker stop command with an optional positional NAME and an --all flag, mirroring the automation delete idiom.
• src/prefect/cli/_worker_utils.py
  • Replaces the single worker.pid constant with WORKER_PID_DIR_NAME = "workers".
  • _run_worker_in_background(...) reconstructs the equivalent foreground python -m prefect worker start ... command (forwarding --pool, --name, --type, repeated --work-queue, --limit, --prefetch-seconds, --run-once, --with-healthcheck, --base-job-template, --install-policy, and --no-create-pool-if-not-found), launches it detached, and records the child PID. The --background/-b flag is intentionally not forwarded so the child runs in the foreground.
  • Reuses the shared PID helpers (_read_pid_file, _is_process_running, _write_pid_file, _cleanup_pid_file) rather than reimplementing them.
• docs/v3/api-ref/cli/worker.mdx regenerated for the new option and command.

# start two background workers
prefect worker start --pool my-pool --name first --background
prefect worker start --pool my-pool --name second --background

# stop one, or all
prefect worker stop first
prefect worker stop --all

Tests in tests/cli/test_worker.py cover: detached spawn (asserting DEVNULL and start_new_session) and PID recording, tracking multiple named workers, generating a name when --name is omitted, reclaiming a stale PID file on start, option forwarding, stopping by name (leaving others running), --all, the name-plus---all conflict, the ambiguous no-argument case, an unknown name, stale and corrupt PID cleanup, and validation failing in the parent (push pool and uninstallable type) before anything is spawned.

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this is a complex change, a maintainer has confirmed the proposed approach on the linked issue.
• If this pull request adds or changes functionality, it includes tests or explains why tests are not needed.
• If this pull request changes user-facing behavior, it updates documentation or explains why documentation is not needed.
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6911e93901

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Detach worker output before returning from background start

When prefect worker start --background is run normally, the short-lived CLI process returns immediately and closes the read ends of these stdout/stderr pipes, while the worker process is still configured to write its startup/log output to them. A normal worker prints status on startup, so the first write can hit a broken pipe and terminate the background worker while leaving worker.pid behind; redirect output to a log file/DEVNULL (and detach the session/process group) instead of using undrained pipes.

Useful? React with 👍 / 👎.

[SreeramaYeshwanthGowd]

Addressed in 4d97d4d. The background worker's stdout/stderr now go to subprocess.DEVNULL and it starts in its own session (start_new_session on POSIX, CREATE_NEW_PROCESS_GROUP on Windows), so it no longer holds the CLI's pipe read ends and cannot die from a broken pipe when the parent process exits. This mirrors the prefect server services --background pattern. Covered by tests/cli/test_worker.py::test_worker_start_background_spawns_detached_process, which asserts DEVNULL for stdout/stderr and start_new_session on non-Windows.

[codspeed-hq]

Merging this PR will not alter performance

✅ 8 untouched benchmarks
⏩ 1 skipped benchmark¹

---

_{Comparing SreeramaYeshwanthGowd:worker-start-background (648ccd3) with main (a917cb7)}

Footnotes

1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports. ↩

[desertaxle]

Thanks for the PR @SreeramaYeshwanthGowd! This implementation looks like it would allow only a single background worker per instance. Could you explore an implementation that allows multiple worker processes to run in the background and be stopped individually?

[SreeramaYeshwanthGowd]

Thanks for the review, @desertaxle! I reworked this to support multiple background workers that can be stopped individually.

prefect worker start --background now tracks each worker by name, writing a per-worker PID file under the Prefect home workers directory, so several can run at once. When --name is omitted the CLI resolves a concrete name and prints it, since the worker otherwise names itself inside the child process where the launching CLI cannot see it. Starting a worker whose name is already running is rejected, and a stale PID file from a crashed worker is reclaimed.

prefect worker stop takes an optional worker name and an --all flag: it stops the named worker, stops every background worker with --all, stops the only one when just one is running, and lists the running workers when the choice is ambiguous. Stale or corrupt PID files are cleaned up.

It reuses the shared server PID helpers, and I added tests for the multiple-worker and individual-stop cases and regenerated the CLI reference docs.