Route graceful SIGTERM shutdown through maintenance instead of a signal race

docs/failover-state-machine.rst

@@ -161,29 +161,37 @@ Maintenance
 ^^^^^^^^^^^
 
 The cluster administrator can manually move a secondary into the
-maintenance state to gracefully take it offline. The primary will then
-transition from state primary to wait_primary, during which time the
-secondary will be online to accept writes. When the old primary reaches
-the wait_primary state then the secondary is safe to take offline with
-minimal consequences.
+maintenance state to gracefully take it offline, and a secondary or
+catching-up node reaches this state on its own too when it receives a
+plain ``SIGTERM`` (see :ref:`pg_autoctl_stop`) — ``pg_autoctl`` requests
+maintenance on its own behalf as part of a graceful shutdown. The primary
+will then transition from state primary to wait_primary, during which time
+the secondary will be online to accept writes. When the old primary
+reaches the wait_primary state then the secondary is safe to take offline
+with minimal consequences.
 
 Prepare_maintenance
 ^^^^^^^^^^^^^^^^^^^
 
 The cluster administrator can manually move a primary node into the
-maintenance state to gracefully take it offline. The primary then
-transitions to the prepare_maintenance state to make sure the secondary is
-not missing any writes. In the prepare_maintenance state, the primary shuts
-down.
+maintenance state to gracefully take it offline, and a primary reaches this
+state on its own too when it receives a plain ``SIGTERM`` (see
+:ref:`pg_autoctl_stop`) — ``pg_autoctl`` requests maintenance on its own
+behalf as part of a graceful shutdown. The primary then transitions to the
+prepare_maintenance state to make sure the secondary is not missing any
+writes. In the prepare_maintenance state, the primary shuts down.
 
 Wait_maintenance
 ^^^^^^^^^^^^^^^^
 
-The custer administrator can manually move a secondary into the maintenance
-state to gracefully take it offline. Before reaching the maintenance state
-though, we want to switch the primary node to asynchronous replication, in
-order to avoid writes being blocked. In the state wait_maintenance the
-standby waits until the primary has reached wait_primary.
+The cluster administrator can manually move a secondary into the maintenance
+state to gracefully take it offline, and a secondary reaches this state on
+its own too when it receives a plain ``SIGTERM`` (see :ref:`pg_autoctl_stop`)
+— ``pg_autoctl`` requests maintenance on its own behalf as part of a
+graceful shutdown. Before reaching the maintenance state though, we want to
+switch the primary node to asynchronous replication, in order to avoid
+writes being blocked. In the state wait_maintenance the standby waits until
+the primary has reached wait_primary.
 
 Draining
 ^^^^^^^^

docs/install.rst

@@ -111,7 +111,8 @@ For container and Kubernetes deployments, systemd is not used.  Instead,
 then exec's into the supervisor.  The standard Unix signal contract
 (``SIGTERM`` to stop, ``SIGHUP`` to reload) is preserved because the
 supervisor becomes the direct child process.  See :ref:`pg_autoctl_node`
-for the full reference.
+for the full reference, and :ref:`pg_autoctl_stop` for what a graceful
+``SIGTERM`` actually does before the node stops.
 
 
 Building pg_auto_failover from sources

docs/operations.rst

@@ -204,6 +204,16 @@ following examples we're directly connecting as the ``autoctl`` role.
 The main operations with pg_auto_failover are node maintenance and manual
 failover, also known as a controlled switchover.
 
+.. note::
+
+   A plain ``pg_autoctl stop`` (a graceful ``SIGTERM``, see
+   :ref:`pg_autoctl_stop`) requests maintenance the same way as the manual
+   ``pg_autoctl enable maintenance`` commands shown below, automatically,
+   on the node's own behalf before stopping. This section covers the
+   manual commands for when you want a node to stay registered in
+   maintenance mode independently of a restart, e.g. for a longer OS
+   maintenance window.
+
 Maintenance of a secondary node
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

docs/ref/pg_autoctl_enable_maintenance.rst

@@ -16,6 +16,14 @@ for promotion.
 Typical use of the maintenance state include Operating System or Postgres
 reboot, e.g. when applying security upgrades.
 
+A plain ``pg_autoctl stop`` (a graceful ``SIGTERM``, see
+:ref:`pg_autoctl_stop`) requests maintenance the same way on its own,
+automatically, before stopping — there is no need to run this command
+first just to get a clean handoff before stopping a node. A node that
+entered maintenance that way also leaves it automatically on its next
+start. Use this command directly when you want the node to stay
+registered in maintenance mode for a while, independently of any restart.
+
 ::
 
    usage: pg_autoctl enable maintenance  [ --pgdata --allow-failover ]

docs/ref/pg_autoctl_node_run.rst

@@ -40,7 +40,8 @@ deployments. Given a ``pg_autoctl_node.ini`` file it:
 Because the command uses ``execv()``, the pg_autoctl supervisor becomes
 the direct child process (PID 1 in a container), preserving the standard
 Unix signal contract — ``SIGTERM`` stops the supervisor cleanly,
-``SIGHUP`` reloads configuration.
+``SIGHUP`` reloads configuration. See :ref:`pg_autoctl_stop` for what a
+graceful ``SIGTERM`` actually does before the node stops.
 
 The ``launch = deferred`` pattern
 ----------------------------------

docs/ref/pg_autoctl_stop.rst

@@ -27,12 +27,36 @@ The ``pg_autoctl stop`` commands finds the PID of the running service for
 the given ``--pgdata``, and if the process is still running, sends a
 ``SIGTERM`` signal to the process.
 
-When ``pg_autoclt`` receives a shutdown signal a shutdown sequence is
+When ``pg_autoctl`` receives a shutdown signal a shutdown sequence is
 triggered. Depending on the signal received, an operation that has been
 started (such as a state transition) is either run to completion, stopped as
 the next opportunity, or stopped immediately even when in the middle of the
 transition.
 
+A plain ``SIGTERM`` (the default, no ``--fast`` or ``--immediate`` flag) is
+a **graceful** shutdown: for a primary, secondary, or catching-up node,
+running with the monitor enabled, ``pg_autoctl`` requests :ref:`maintenance
+<pg_autoctl_enable_maintenance>` on its own behalf before stopping, the same
+mechanism used by ``pg_autoctl enable maintenance``. A healthy standby can
+then take over immediately, rather than waiting for the monitor's own
+health-check timeout to notice the node is gone. This can take up to 30
+seconds. If maintenance cannot be requested — the monitor is disabled, the
+monitor is unreachable, or there is no candidate currently available to
+take over — ``pg_autoctl`` falls back to stopping Postgres directly and
+reporting the shutdown to the monitor for up to another 30 seconds, so a
+failover can still be driven by the usual health-check mechanism.
+
+A node that entered maintenance this way (as opposed to an operator running
+``pg_autoctl enable maintenance`` directly) automatically leaves maintenance
+the next time it is started, with no need to run ``pg_autoctl disable
+maintenance`` manually. An operator-initiated maintenance session is left
+untouched by a restart, and still needs an explicit ``pg_autoctl disable
+maintenance`` to end.
+
+The ``--fast`` and ``--immediate`` options skip all of that: they stop the
+node right away without attempting a graceful handoff, which is the closest
+equivalent to simulating a hard crash.
+
 Options
 -------
 

docs/ref/pgaftest.rst

@@ -517,19 +517,26 @@ is actually testing, not whichever one happens to make the test pass.
 
 ``compose stop <service>``
     ``docker compose stop`` — SIGTERM to the container's PID 1
-    (``pg_autoctl``), which propagates an orderly shutdown to its child
-    services (a grace period applies before Docker escalates to SIGKILL).
-    Exercises orderly shutdown: ``pg_autoctl``'s supervisor
-    (``supervisor_stop_subprocesses()``) runs its normal shutdown sequence,
-    so the monitor sees a clean disconnect, then the same failover trigger
-    as any other lost connection.
+    (``pg_autoctl``), which forwards a plain SIGTERM to the node-active
+    (keeper) service only (a grace period applies before Docker escalates to
+    SIGKILL). Exercises graceful shutdown: for a primary, secondary, or
+    catching-up node, the keeper calls ``start_maintenance()`` on its own
+    behalf and drives the ordinary maintenance FSM (``prepare_maintenance``
+    -> ``maintenance`` for a primary, straight to ``maintenance`` or via
+    ``wait_maintenance`` for a secondary/catching-up node), the same
+    transitions an operator-run ``pg_autoctl enable maintenance
+    --allow-failover`` would drive. If maintenance can't be started (the
+    monitor is unreachable, or no candidate is available), Postgres is
+    simply stopped and the process exits, falling back to the monitor's own
+    health-check-driven failover.
 
 ``compose kill <service>``
     ``docker compose kill`` — immediate SIGKILL, no grace period at all.
     Exercises hard-crash recovery: the process gets zero chance to shut
-    down cleanly. Use only when a test specifically needs to rule out a
-    race where the dying node reports a state transition on its way out
-    (see the ``multi_alternate.pgaf`` header comment).
+    down cleanly, not even a signal handler. Use only when a test
+    specifically needs to rule out a race where the dying node reports a
+    state transition on its way out (see the ``multi_alternate.pgaf``
+    header comment).
 
 ``stop postgres <node>`` / ``start postgres <node>``
     ``pg_autoctl manual service pgctl off`` — writes a persistent

src/bin/common/pgctl.c

@@ -1647,6 +1647,23 @@ pg_ctl_postgres(const char *pg_ctl, const char *pgdata, int pgport,
 		args[argsIndex++] = "-h";
 		args[argsIndex++] = (char *) listen_addresses;
 	}
+	else if (env_found_empty("PG_REGRESS_SOCK_DIR"))
+	{
+		/*
+		 * PG_REGRESS_SOCK_DIR="" means unix sockets are unavailable in this
+		 * environment (see pg_setup_get_local_connection_string, which then
+		 * forces client connections to use "host=localhost" instead). If we
+		 * also pass an empty listen_addresses here, postgres has no way to
+		 * create any socket at all -- TCP disabled by "-h ''", unix socket
+		 * disabled by the "-k" added below -- and fails outright with
+		 * "FATAL: no socket created for listening", even though nothing
+		 * external is meant to connect to it in this "do not open the
+		 * service just yet" mode. Fall back to the loopback interface only,
+		 * matching the same PG_REGRESS_SOCK_DIR convention used elsewhere.
+		 */
+		args[argsIndex++] = "-h";
+		args[argsIndex++] = "localhost";
+	}
 	else
 	{
 		args[argsIndex++] = "-h";

src/bin/pg_autoctl/defaults.h

@@ -13,7 +13,7 @@
 #include "git-version.h"
 
 /* to be written in the state file */
-#define PG_AUTOCTL_STATE_VERSION 1
+#define PG_AUTOCTL_STATE_VERSION 2
 
 /* additional version information for printing version on CLI */
 #define PG_AUTOCTL_VERSION GIT_VERSION
@@ -94,6 +94,22 @@
 #define PG_AUTOCTL_MONITOR_SLEEP_TIME 10 /* seconds */
 #define PG_AUTOCTL_MONITOR_RETRY_TIME 1  /* seconds */
 
+/*
+ * A primary's graceful SIGTERM shutdown (see keeper_graceful_shutdown() in
+ * service_keeper.c) may spend up to KEEPER_MAINTENANCE_SHUTDOWN_LOOP_MAX_SECS
+ * attempting a maintenance handoff, and if that does not complete in time,
+ * up to another KEEPER_SHUTDOWN_LOOP_MAX_SECS in the fallback reporting
+ * loop. The supervisor (supervisor.c) must not treat the keeper as "stuck"
+ * and escalate a plain SIGTERM to the rest of the services before that full
+ * combined window has elapsed.
+ */
+#define KEEPER_SHUTDOWN_LOOP_MAX_SECS 30
+#define KEEPER_SHUTDOWN_LOOP_STOPPED_REPORT_INTERVAL_SECS 5
+#define KEEPER_SHUTDOWN_STOPPED_REPORT_MAX_ATTEMPTS 2
+#define KEEPER_MAINTENANCE_SHUTDOWN_LOOP_MAX_SECS 30
+#define KEEPER_GRACEFUL_SHUTDOWN_MAX_SECS \
+	(KEEPER_MAINTENANCE_SHUTDOWN_LOOP_MAX_SECS + KEEPER_SHUTDOWN_LOOP_MAX_SECS)
+
 #define PG_AUTOCTL_LISTEN_NOTIFICATIONS_TIMEOUT 60
 
 #define COORDINATOR_IS_READY_TIMEOUT 300