keeper deadlocks on a zombie postmaster: get_pgpid() uses kill(pid, 0), which cannot distinguish a zombie from a live process

[tboevil]

Environment

pg_auto_failover 2.2 line, Linux (openEuler / EL). Reproduced under fault injection (blade kill --signal 9 targeting postgres by process name) during node startup.

Symptom

A node hangs forever in its current state (health=0, no FSM progress, no monitor heartbeat) until pgautofailover is restarted manually. postmaster.pid still exists and points at a pid that ps shows as <defunct>, parented by the pg_autoctl: start/stop postgres controller sub-process.

Sequence

1. The Postgres controller sub-process forks the postmaster.
2. Postgres is SIGKILLed mid-startup, before it writes PM_STATUS_READY into postmaster.pid. The postmaster becomes a zombie child of the controller.
3. pg_setup_is_ready() -> get_pgpid() reads the stale pid from postmaster.pid and probes it with kill(pid, 0) (src/bin/common/pgsetup.c, currently line 497). A zombie still has a process-table entry, so this returns 0 and the pid is reported as alive.
4. pg_setup_wait_until_is_ready() therefore keeps retrying every 350ms forever, waiting for a ready status that a dead process will never write.
5. Because the controller is stuck inside that retry loop, it never reaches the waitpid(-1, WNOHANG) in its main loop. The zombie is never reaped, its shared memory segment stays attached, and no new postmaster can be started.

Proposed fix

Replace the bare kill(pid, 0) probe in get_pgpid() with a helper:

• kill(pid, 0) != 0 -> not alive (unchanged fast path)
• waitpid(pid, &st, WNOHANG) == pid -> the pid was our zombie child; it is now reaped, report not alive
• waitpid(...) == 0 -> child is genuinely running -> alive (so slow-startup behaviour is unchanged)
• waitpid(...) == -1, ECHILD -> pid is not our child (keeper or CLI reading postmaster.pid) -> fall back to the pre-existing kill(pid, 0) semantics

[dimitri]

Thanks for the very precise diagnostic and reproduction on this one -- the exact line reference and proposed fix made this quick to confirm and land.

Fixed by #1173 (merged as ef7aa1d), matching your proposed approach: get_pgpid() now tries waitpid(pid, WNOHANG) before falling back to kill(pid, 0). From the postgres controller (the actual parent, since it fork()s + execv()s postgres directly), this reaps a zombie and correctly reports it dead instead of "alive". From every other reader of postmaster.pid (keeper, monitor, CLI), waitpid() on a non-child pid returns ECHILD immediately, falling back to the historical probe -- unchanged there.