Killed deploy orphans the Python discovery server (serving.py); orphans wedge and later deploys hang forever on connect ETIMEDOUT

[sallyjones6]

Environment

• firebase-tools: 15.9.1 (repo-pinned devDependency, run from node_modules/.bin/firebase)
• Platform: macOS 15 (Apple Silicon), self-hosted GitHub Actions runners
• Project: multi-codebase functions — one nodejs codebase + two Python codebases (firebase-functions Python SDK)
• Command: firebase deploy --only functions --force --project <id> in CI, with a job-level 60-minute timeout

Problem

When a firebase deploy run is killed externally mid-flight (CI job cancellation or timeout), the Python discovery admin server the CLI spawned (firebase_functions/private/serving.py, launched with ADMIN_PORT=<port>) is left running. The orphan keeps its admin port bound but eventually stops accepting connections.

Once one or more such orphans exist on the host, subsequent deploys hang indefinitely during "Loading and analyzing source code for codebase <python-codebase>": every localhost call to the admin port — including /__/quitquitquit — fails with connect ETIMEDOUT 127.0.0.1:<port> (timeout, not refused: a bound-but-not-accepting listener), and the CLI never errors out. In our CI this meant each affected deploy ran until the 60-minute job timeout killed it — which orphaned its discovery server too. Over 5 days ~35 orphaned serving.py processes accumulated on one runner host, at which point every deploy landing on it hung.

Debug log excerpt from a hung deploy (nodejs codebase discovery succeeded in ~1s just before this):

[debug] Running admin server with args: ["python3.12","<venv>/site-packages/firebase_functions/private/serving.py"] and env: {..., "ADMIN_PORT":"8120"} in <project>/firebase/functions/python-ytmusicapi
[info]  * Serving Flask app 'serving'
[debug] Failed to call quitquitquit. This often means the server failed to start request to http://127.0.0.1:8120/__/quitquitquit failed, reason: connect ETIMEDOUT 127.0.0.1:8120
[error]  * Running on http://127.0.0.1:8120

(no further log lines; the process sat there until the CI job timeout ~59 minutes later)

Verification

Killing the orphans (pkill -f "firebase_functions/private/serving.py" as the runner user) immediately restored ~2-minute deploys on the same host — reproduced across two hung runs and one clean rerun.

Expected behavior

1. The CLI should hard-kill the discovery child process (SIGKILL by pid / process group) when /__/quitquitquit is unreachable, and on its own exit paths — an HTTP-only shutdown can never clean up a wedged server.
2. Discovery should fail fast with a clear error when the admin port cannot be reached within a bounded time, instead of hanging the whole deploy indefinitely.

Workaround

A CI step before firebase deploy that pkills stale serving.py processes.

[IzaakGough]

Hi @sallyjones6, thanks for raising this! We will investigate and get back to you as soon as possible.

[IzaakGough]

I managed to reproduce this. This issue seems to be three bugs stacking up:

1. Dead kill switch: in python/index.ts, the code sets a 10s timer to force-kill the stuck process, then immediately cancels that same timer on the next line. So the kill never happens. (Node's version of this code doesn't have the stray cancel — Python's does.)
2. Kill doesn't reach the right process anyway: Python is launched via a shell wrapper (. venv/activate && python serving.py). The CLI only tracks the shell's pid, not Python's. I tested it — killing the shell leaves Python running.
3. No cleanup on external kill: if CI kills firebase deploy itself (your actual trigger), none of this cleanup code runs at all.

Minimal repro of the first point, no Firebase needed:

const { spawn } = require("child_process");
const child = spawn("sleep", ["9999"]); // stand-in for the stuck server

const t = setTimeout(() => child.kill("SIGKILL"), 2000);
clearTimeout(t); // <- the bug

setTimeout(() => console.log("still running:", !child.killed), 5000);

Prints still running: true. Delete the clearTimeout line and it gets killed as expected.

A fix for this needs to address all three of these points. I will follow up with a PR and keep you updated.