keeper: send node_active reports to monitor during graceful shutdown (#1146)

* keeper: send node_active reports to monitor during graceful shutdown

On SIGTERM, the node-active service's main loop exits immediately, before
calling keeper_update_pg_state() or service_keeper_node_active().  This
leaves the monitor unaware that node1 is going offline until its next
health-check poll, causing the flaky behaviour seen in ensure::test_004_demoted
where node2 never reached wait_primary within the test timeout.

Add keeper_node_active_shutdown_loop(): after the main loop breaks on
asked_to_stop, report the current Postgres state to the monitor every 1s
for up to 30 seconds (or until Postgres stops or an escalated signal
arrives).  This runs concurrently with the postgres-controller service
calling 'pg_ctl stop -m fast': fast mode sends SIGTERM to the postmaster,
which stops accepting new connections immediately, while the checkpoint
completes asynchronously.  By the time the first shutdown report fires,
Postgres is already refusing connections; the monitor can start failover
right away rather than waiting for a health-check timeout.

SIGINT (asked_to_stop_fast) and SIGQUIT (asked_to_quit) skip the shutdown
loop and exit immediately, as before.

* fix ensure::test_003: re-enable auto-start after stop postgres

test_003_init_secondary stops postgres on node2 then waits for it to
recover as secondary.  The original Python test (node2.stop_postgres())
sends SIGTERM directly to postgres without touching pg_autoctl's service
controller, so the keeper auto-restarts postgres.

The pgaftest port used 'stop postgres node2' which calls
'pg_autoctl manual service pgctl off' — this both stops postgres AND
disables auto-restart.  As a result node2's postgres stayed stopped for
the rest of the spec.

In test_004_demoted, NodeIsHealthy(node2) checks pgIsRunning, which was
false because node2's postgres was never restarted.  The failover
transition at group_state_machine.c:550 requires NodeIsHealthy(activeNode)
to be true, so the failover never triggered and the test timed out at 180s.

Fix: add 'start postgres node2' immediately after 'stop postgres node2'
to call 'pg_autoctl manual service pgctl on', re-enabling the keeper's
auto-restart.  The keeper restarts postgres and node2 converges back to
secondary before test_004 runs.

Author: dimitri

src/bin/pg_autoctl/service_keeper.c

@@ -244,6 +244,61 @@ service_keeper_node_active_init(Keeper *keeper)
 }
 
 
+/*
+ * keeper_node_active_shutdown_loop sends node_active reports to the monitor
+ * every second for up to KEEPER_SHUTDOWN_LOOP_MAX_SECS while PostgreSQL stops.
+ *
+ * This runs after the main node-active loop exits on SIGTERM, concurrently
+ * with the postgres-controller service (a sibling process) calling
+ * "pg_ctl stop -m fast".
+ *
+ * When SIGTERM reaches the postmaster, process_pm_shutdown_request()
+ * (src/backend/postmaster/postmaster.c) sets Shutdown = FastShutdown and
+ * calls UpdatePMState(PM_STOP_BACKENDS).  After that transition,
+ * canAcceptConnections() (same file) returns CAC_SHUTDOWN for every new
+ * connection attempt, so the primary stops accepting writes immediately —
+ * before a single backend has rolled back.  Only the final checkpoint that
+ * follows can be slow.
+ *
+ * By continuing to call node_active with the current pgIsRunning value here,
+ * we ensure the monitor learns the primary is going away within one second of
+ * the shutdown starting, and can begin failover right away rather than waiting
+ * for a health-check timeout.
+ */
+#define KEEPER_SHUTDOWN_LOOP_MAX_SECS 30
+
+static void
+keeper_node_active_shutdown_loop(Keeper *keeper)
+{
+	LocalPostgresServer *postgres = &(keeper->postgres);
+
+	log_info("Graceful shutdown: reporting node state to monitor "
+			 "while PostgreSQL stops (up to %d seconds)",
+			 KEEPER_SHUTDOWN_LOOP_MAX_SECS);
+
+	for (int i = 0; i < KEEPER_SHUTDOWN_LOOP_MAX_SECS; i++)
+	{
+		/* escalated signal: exit without further reporting */
+		if (asked_to_quit || asked_to_stop_fast)
+		{
+			break;
+		}
+
+		(void) keeper_update_pg_state(keeper, LOG_DEBUG);
+		(void) service_keeper_node_active(keeper, false);
+
+		if (!postgres->pgIsRunning)
+		{
+			log_info("PostgreSQL has stopped; "
+					 "final node_active report sent to monitor");
+			break;
+		}
+
+		pg_usleep(1000000L); /* 1 second */
+	}
+}
+
+
 /*
  * keeper_node_active_loop implements the main loop of the keeper, which
  * periodically gets the goal state from the monitor and makes the state
@@ -659,6 +714,16 @@ keeper_node_active_loop(Keeper *keeper, pid_t start_pid)
 		}
 	}
 
+	/*
+	 * Graceful SIGTERM shutdown: keep reporting state to the monitor while
+	 * PostgreSQL finishes its checkpoint and stops.  Skip on SIGINT/SIGQUIT
+	 * which request immediate exit.
+	 */
+	if (asked_to_stop && !asked_to_stop_fast && !asked_to_quit)
+	{
+		(void) keeper_node_active_shutdown_loop(keeper);
+	}
+
 	/* One last check that we do not have any connections open */
 	pgsql_finish(&(keeper->monitor.pgsql));
 	pgsql_finish(&(monitor->notificationClient));

tests/tap/specs/ensure.pgaf

@@ -41,7 +41,13 @@ step test_003_init_secondary {
     wait until node2 state is secondary
         and node1 state is primary
         timeout 90s
+    # Stop postgres while pg_autoctl is running, then re-enable auto-start so
+    # the keeper restarts postgres automatically.  This tests that the keeper
+    # recovers from an unexpected postgres stop.  The Python equivalent is
+    # node2.stop_postgres() which sends SIGTERM to postgres without disabling
+    # the pg_autoctl postgres-controller restart logic.
     stop postgres node2
+    start postgres node2
     wait until node2 state is secondary
         and node1 state is primary
         timeout 90s