pgaftest/docs: regression spec + docs for report/advance, no-autopilot, fsm step

Add fsm_step_report_advance.pgaf: a live spec proving the report/advance
split works end to end using a no-autopilot node, registered in the
default schedule and the node.sch CI schedule.

Document the three new pieces of tooling this branch introduces:
- the no-autopilot cluster node modifier
- the pgaftest "fsm step <node>" DSL command
- the "pg_autoctl manual fsm step [report|advance]" CLI split

in docs/ref/pgaftest.rst (new "Step mode: no-autopilot nodes" section,
node modifiers table, DSL command reference, test catalog entry) and
docs/ref/pg_autoctl_manual.rst (fsm step synopsis).

Author: dimitri

docs/ref/pg_autoctl_manual.rst

@@ -34,6 +34,20 @@ stopped or stuck, and for low-level diagnostic work. Using them while
       step    Make a state transition if instructed by the monitor
     + nodes   Manually manage the keeper's nodes list
 
+``pg_autoctl manual fsm step [report|advance]`` normally does both halves of
+a step in one call: report the node's current state to the monitor, then
+immediately attempt whatever transition the monitor assigns back. Passing
+``report`` or ``advance`` as an argument splits that into its two
+independently-issuable halves — ``report`` reports the current state and
+returns without transitioning; ``advance`` attempts the transition the
+monitor already assigned, without re-reporting first. Neither half re-runs
+the other, so observing the effect of ``advance`` on the monitor's own view
+still needs a following ``report`` (or plain ``step``) call. This is mainly
+useful against a node started with ``pgaftest``'s ``no-autopilot`` cluster
+modifier (see :ref:`pgaftest <pgaftest>`), where the node-active service
+never ticks the FSM on its own — ``step``/``report``/``advance`` become the
+only way to move it forward, one call at a time.
+
     pg_autoctl manual fsm nodes
       get  Get the list of nodes from file (see --disable-monitor)
       set  Set the list of nodes to file (see --disable-monitor)

docs/ref/pgaftest.rst

@@ -440,6 +440,10 @@ Node modifiers:
                                               (``--region``; default: ``default``)
 ``launch deferred``                           Container starts with ``sleep infinity``;
                                               use ``exec node  pg_autoctl node start``
+``no-autopilot``                              Step mode: the node-active service never
+                                              transitions on its own; drive it explicitly
+                                              with the ``fsm step <node>`` DSL command
+                                              (see `Step mode: no-autopilot nodes`_ below)
 ``coordinator`` / ``worker group <N>``        Citus role
 ``no-monitor``                                Standalone node (no monitor)
 ``listen``                                    Bind all interfaces (``--listen 0.0.0.0``)
@@ -478,6 +482,39 @@ See "Deterministic node registration order" in
 and ``cli_node.c``).
 
 
+Step mode: ``no-autopilot`` nodes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A node declared with the ``no-autopilot`` modifier starts with
+``PG_AUTOCTL_STEP_MODE`` set, which changes what its node-active service does
+on each FSM tick: instead of reporting its current state to the monitor and
+immediately attempting whatever transition the monitor assigns back —
+atomically, on every tick, with no way to observe or freeze the moment in
+between — it opens a small Unix-domain-socket server and waits. Nothing
+happens to that node's FSM until the spec explicitly drives it with the
+``fsm step <node>`` command, documented under "Commands inside ``setup``,
+``teardown``, and ``step`` blocks" below.
+
+This exists so a test can hold a node frozen at a specific reported state on
+purpose — e.g. to prove the monitor assigns the right next state before the
+node itself races off to reach it, or to reproduce a specific ordering
+between two nodes that would otherwise be a race under the normal, freely
+ticking FSM. Every other node in the same spec (without the modifier) keeps
+autopiloting normally; ``no-autopilot`` only affects the node(s) it's
+declared on.
+
+``fsm step <node>`` is sugar for the combined
+``pg_autoctl manual fsm step`` command (report the current state, then
+immediately attempt whatever transition the monitor assigns). The underlying
+CLI also exposes the two halves separately as ``manual fsm step report`` and
+``manual fsm step advance`` — see :ref:`pg_autoctl_manual` — for scenarios
+that need to observe the monitor's assigned goal state before deciding
+whether, or when, to actually attempt the transition; the DSL does not yet
+have separate sugar for the split, so use ``exec <node>  pg_autoctl manual
+fsm step report --pgdata /var/lib/postgres/pgaf`` / ``... advance ...``
+directly for that.
+
+
 Commands inside ``setup``, ``teardown``, and ``step`` blocks
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -575,6 +612,12 @@ node ...`` which queries the running node/monitor instead of the file.
    stop postgres  <node>
    start postgres <node>
 
+**FSM step** (``no-autopilot`` nodes only — see `Step mode: no-autopilot nodes`_ above)
+
+.. code-block:: text
+
+   fsm step <node>
+
 **Failover**
 
 .. code-block:: text
@@ -762,6 +805,12 @@ Schedules under ``tests/tap/schedules/*.sch`` group these into CI jobs.
 ``fast_forward``
     Test fast-forward stuck detection and recovery.
 
+``fsm_step_report_advance``
+    Test the ``pg_autoctl manual fsm step report``/``... advance`` split
+    using a ``no-autopilot`` node: report a stale state to the monitor
+    without transitioning, then advance to the transition the monitor
+    already assigned, proving the two halves work independently.
+
 ``guard_data_loss``
     Test ``pgautofailover.guard_data_loss`` /
     ``pg_autoctl perform failover --allow-data-loss``.

tests/tap/schedule

@@ -25,6 +25,7 @@ multi_ifdown
 multi_maintenance
 multi_alternate
 guard_data_loss
+fsm_step_report_advance
 replication_stall_3dc
 fast_forward
 demote_timeout_wait_primary_deadlock

tests/tap/schedules/node.sch

@@ -3,6 +3,7 @@
 # count and GitHub Actions runner queue pressure.
 create_standby_with_pgdata
 launch_deferred_set_metadata
+fsm_step_report_advance
 maintenance_and_drop
 auth
 monitor_disabled

tests/tap/specs/fsm_step_report_advance.pgaf

@@ -0,0 +1,88 @@
+# Regression spec for the new "pg_autoctl manual fsm step report"/"...
+# advance" split (src/bin/pg_autoctl/fsm.c's keeper_fsm_step_report/
+# _advance, service_keeper.c's step-mode socket dispatch, cli_do_fsm.c's
+# cli_do_fsm_step) -- see docs/ref/pgaftest.rst's own "Step mode" section
+# for the full protocol description.
+#
+# The combined "pg_autoctl manual fsm step" (and its pgaftest
+# "fsm step <node>" sugar) reports the current state to the monitor and,
+# in the very same call, immediately attempts whatever transition the
+# monitor just assigned -- atomically, with no way to observe (or hold a
+# node frozen at) the moment in between. "report" and "advance" split
+# that into two independently-issuable halves, so a test (or an operator
+# reproducing a race condition by hand) can freeze a node's own
+# reportedState at will and only decide later, explicitly, whether and
+# when it should attempt the transition the monitor already assigned.
+#
+# This spec proves the split mechanism itself, end to end, on the
+# simplest case that exercises it: node2 registers as node1's standby
+# (reaching reportedState catchingup), node1 is then forcibly removed,
+# leaving node2 alone in the group -- the monitor assigns node2's
+# goalState to single (an existing, pre-existing "alone in group ->
+# single" rule, unrelated to this spec's own point). node2 is running
+# no-autopilot (step mode), so:
+#
+#   - "manual fsm step report" reports node2's unchanged current state
+#     (catchingup) to the monitor and returns -- proven by the monitor
+#     bumping goalState to single while reportedState stays catchingup.
+#   - "manual fsm step advance" then performs the transition the
+#     monitor already assigned (catchingup -> single), without
+#     re-reporting first -- proven by reportedState becoming single.
+cluster {
+    monitor
+    formation {
+        node1
+        node2  no-autopilot
+    }
+}
+
+setup {
+    wait until node2 state = catchingup  timeout 60s
+}
+
+teardown {
+    compose down
+}
+
+step test_001_remove_primary_leaving_node2_alone {
+    sql monitor {
+        SELECT pgautofailover.remove_node(nodehost, nodeport, true)
+        FROM pgautofailover.node WHERE nodename = 'node1';
+    }
+}
+
+step test_002_node2_reports_and_learns_single {
+    exec node2  pg_autoctl manual fsm step report --pgdata /var/lib/postgres/pgaf
+    sql monitor {
+        SELECT reportedstate FROM pgautofailover.node WHERE nodename = 'node2';
+    }
+    expect { catchingup }
+    sql monitor {
+        SELECT goalstate FROM pgautofailover.node WHERE nodename = 'node2';
+    }
+    expect { single }
+}
+
+step test_003_node2_advances_to_single {
+    exec node2  pg_autoctl manual fsm step advance --pgdata /var/lib/postgres/pgaf
+}
+
+step test_004_node2_reports_the_new_single_state {
+    # "advance" (like "report") only performs its own half of the split;
+    # neither one re-reports afterward, so the monitor's own view of
+    # node2 still lags until the next explicit "report" (or combined
+    # "step") call pushes the now-current single state -- exactly the
+    # same "one more call needed" shape the existing gap specs rely on
+    # when chaining repeated combined "fsm step" calls.
+    exec node2  pg_autoctl manual fsm step report --pgdata /var/lib/postgres/pgaf
+    sql monitor {
+        SELECT reportedstate FROM pgautofailover.node WHERE nodename = 'node2';
+    }
+    expect { single }
+}
+
+sequence
+    test_001_remove_primary_leaving_node2_alone
+    test_002_node2_reports_and_learns_single
+    test_003_node2_advances_to_single
+    test_004_node2_reports_the_new_single_state