Add pgautofailover.guard_data_loss GUC and pg_autoctl perform failover --allow-data-loss

docs/architecture-multi-standby.rst

@@ -56,6 +56,12 @@ In more details:
     standby node acknowledges the transactions locally committed, thus
     degrading your Postgres service to read-only.
 
+    The sequence below shows how a commit is confirmed and how a healthy
+    failover completes when all quorum standbys are available:
+
+    .. figure:: ./tikz/seq-normal-failover.svg
+       :align: center
+
  0. It is possible to manually set ``number_sync_standbys`` to zero when
     having registered two standby nodes to the monitor, overriding the
     default behavior.
@@ -73,6 +79,29 @@ In more details:
     data will be lost. How much depends on your backup and recovery
     mechanisms.
 
+The sequence below shows the stuck-election scenario that can arise with any
+``number_sync_standbys >= 1`` setting when the primary and one quorum standby
+fail at the same time, and how ``--allow-data-loss`` unblocks it:
+
+.. figure:: ./tikz/seq-stuck-failover.svg
+   :align: center
+
+.. note::
+
+   **Failover when the primary and a quorum standby fail simultaneously.**
+   If the primary and one quorum standby are lost at the same time, the
+   surviving standby is assigned ``report_lsn`` but will not be promoted
+   automatically.  The monitor cannot know whether the missing standby
+   acknowledged the last synchronous commit, and promoting the survivor
+   could silently discard those transactions.
+
+   Once you have confirmed the missing node is permanently lost and you accept
+   the potential data loss, unblock the election with::
+
+     pg_autoctl perform failover --allow-data-loss
+
+   See :ref:`perform_failover_allow_data_loss` for a full explanation.
+
 .. _architecture_setup:
 
 Replication Settings and Postgres Architectures

docs/conf.py

@@ -47,7 +47,7 @@ def __init__(self, **options):
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ["sphinx.ext.githubpages"]
+extensions = ["sphinx.ext.githubpages", "sphinxcontrib.mermaid"]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]

docs/failover-state-machine.rst

@@ -250,6 +250,17 @@ restarting Postgres without a ``primary_conninfo``. This allows the primary
 node to detect :ref:`network_partitions`, i.e. when the primary can't connect
 to the monitor and there's no standby listed in ``pg_stat_replication``.
 
+If one or more quorum standbys (nodes counted by ``number_sync_standbys``)
+are unreachable and never report their LSN, the monitor will not advance the
+election.  The missing node may have acknowledged the last synchronous commit
+before it disappeared, and promoting a lagging candidate would silently discard
+those transactions.  This protection is controlled by the
+``pgautofailover.guard_data_loss`` GUC (default ``true``).  When the missing
+node cannot be recovered and the operator is willing to accept the potential
+data loss, the election can be unblocked with
+:ref:`pg_autoctl_perform_failover` ``--allow-data-loss``.  See
+:ref:`perform_failover_allow_data_loss` for details.
+
 Fast_forward
 ^^^^^^^^^^^^
 

docs/faq.rst

@@ -84,6 +84,33 @@ default configuration deployed by ``pg_autoctl create ...``. When a custom
 Postgres setup is used, please refer to your actual setup to find Postgres
 logs.
 
+My failover is stuck: standbys are in ``report_lsn`` and nothing moves
+-----------------------------------------------------------------------
+
+This happens when the primary and one or more quorum standbys
+(nodes counted by ``number_sync_standbys``) fail at the same time.  The
+monitor drives all surviving standbys to the ``report_lsn`` state to
+determine the most advanced node, but it refuses to promote any candidate
+while a quorum member is missing.  The missing node may have acknowledged a
+synchronous commit that no surviving standby has yet replicated, and
+promoting would silently discard those transactions.
+
+**First, try to bring the missing node back.**  If it recovers and reports
+its LSN, the election resumes automatically.
+
+**If the missing node is permanently lost and you accept the data-loss
+risk,** unblock the election::
+
+  pg_autoctl perform failover --allow-data-loss
+
+The command promotes the most advanced surviving standby.  Transactions
+acknowledged by the missing node but not yet replicated to any survivor
+will be permanently lost once the new primary starts accepting writes.
+
+See :ref:`perform_failover_allow_data_loss` and the
+``pgautofailover.guard_data_loss`` GUC in :ref:`configuration` for a full
+explanation.
+
 The state of the system is blocked, what should I do?
 -----------------------------------------------------
 

docs/ref/configuration.rst

@@ -101,6 +101,34 @@ database where the extension has been deployed::
   setting    | 10000
   unit       | ms
   short_desc | Wait for at least this much time after startup before initiating a failover.
+  -[ RECORD 10 ]---------------------------------------------------------------------------------------------------
+  name       | pgautofailover.guard_data_loss
+  setting    | true
+  unit       |
+  short_desc | Refuse to proceed with failover when quorum nodes have not yet reported their LSN.
+
+pgautofailover.guard_data_loss
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When ``true`` (the default), ``ProceedGroupStateForMSFailover`` refuses to
+promote any standby if one or more quorum nodes have not reported their LSN
+position during a failover election.  This prevents silent data loss: if the
+missing node acknowledged a synchronous commit that no surviving standby
+replicated, promoting a lagging standby would permanently discard those
+transactions.
+
+Set to ``false`` to allow the election to proceed with only the surviving
+candidates, accepting that committed transactions on the missing node(s) may
+be lost.  The recommended way to use this setting is through
+:ref:`pg_autoctl_perform_failover` with the ``--allow-data-loss`` flag, which
+scopes the change to a single transaction and emits a server log message for
+each guard that is bypassed.
+
+Setting ``guard_data_loss = false`` globally in ``postgresql.conf`` is
+**not** recommended: it would silently suppress the protection for all future
+failovers.  Use ``ALTER DATABASE pg_auto_failover SET
+pgautofailover.guard_data_loss = false;`` only if you want the setting to
+persist across monitor restarts with explicit intent, and document the reason.
 
 You can edit the parameters as usual with PostgreSQL, either in the
 ``postgresql.conf`` file or using ``ALTER DATABASE pg_auto_failover SET parameter =

docs/ref/pg_autoctl_perform_failover.rst

@@ -13,10 +13,11 @@ pg_auto_failover monitor::
 
   usage: pg_autoctl perform failover  [ --pgdata --formation --group ]
 
-  --pgdata      path to data directory
-  --formation   formation to target, defaults to 'default'
-  --group       group to target, defaults to 0
-  --wait        how many seconds to wait, default to 60
+  --pgdata           path to data directory
+  --formation        formation to target, defaults to 'default'
+  --group            group to target, defaults to 0
+  --wait             how many seconds to wait, default to 60
+  --allow-data-loss  proceed even when quorum nodes have not reported their LSN
 
 Description
 -----------
@@ -34,6 +35,43 @@ The failover orchestration is done in the background by the monitor, so even
 if the ``pg_autoctl perform failover`` stops on the timeout, the failover
 orchestration continues at the monitor.
 
+.. _perform_failover_allow_data_loss:
+
+Recovering a failover stuck in ``report_lsn``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In a formation with ``number_sync_standbys >= 1``, a failover that loses the
+primary and one quorum standby at the same time can get permanently stuck.
+When the primary fails, the monitor drives all standby nodes into the
+``report_lsn`` state so it can elect the most advanced one.  If a quorum
+standby is unreachable, it never reports its LSN.  The monitor then refuses to
+promote any remaining candidate — it cannot know whether the missing node
+acknowledged the last synchronous commit and holds WAL that no surviving
+standby has replicated yet.
+
+This is the correct conservative default, controlled by the
+``pgautofailover.guard_data_loss`` GUC (default ``true``).  When you have
+determined that the missing node is permanently lost and you are willing to
+accept the potential data loss, use ``--allow-data-loss`` to unblock the
+election:
+
+.. code-block:: bash
+
+   pg_autoctl perform failover --allow-data-loss
+
+The command opens a single transaction on the monitor, sets
+``pgautofailover.guard_data_loss`` to ``false`` for that transaction only,
+and calls ``perform_failover()``.  The monitor then selects the most advanced
+surviving candidate and drives it toward ``prepare_promotion``.
+
+.. warning::
+
+   ``--allow-data-loss`` means exactly what it says.  If the missing quorum
+   standby had acknowledged a synchronous commit that no surviving standby
+   replicated, those transactions will be permanently lost once the new primary
+   starts accepting writes.  Use this option only when the missing node cannot
+   be recovered and the cluster being stuck is the worse outcome.
+
 Options
 -------
 
@@ -60,6 +98,14 @@ Options
   the timeout has elapsed, whichever comes first. The value 0 (zero)
   disables the timeout and allows the command to wait forever.
 
+--allow-data-loss
+
+  Disable the ``pgautofailover.guard_data_loss`` protection for this
+  failover only.  When set, the monitor will promote the most advanced
+  surviving candidate even if one or more quorum standby nodes have not yet
+  reported their LSN position.  Committed transactions on the missing node
+  may be permanently lost.  See :ref:`perform_failover_allow_data_loss`.
+
 Environment
 -----------
 
@@ -141,3 +187,42 @@ Examples
    node1 |     1 | localhost:5501 | 0/4000F50 |    read-only |           secondary |           secondary
    node2 |     2 | localhost:5502 | 0/4000F50 |   read-write |             primary |             primary
    node3 |     3 | localhost:5503 | 0/4000F50 |    read-only |           secondary |           secondary
+
+Example: unblocking a stuck election with ``--allow-data-loss``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this example ``node1`` (primary) and ``node3`` (a quorum standby) have
+both failed.  ``node2`` reaches ``report_lsn`` and waits indefinitely because
+``node3`` never reported its LSN::
+
+   $ pg_autoctl show state
+    Name |  Node |      Host:Port |       LSN |   Connection |       Current State |      Assigned State
+   ------+-------+----------------+-----------+--------------+---------------------+--------------------
+   node1 |     1 | localhost:5501 |   0/0     |       none   |            draining |            draining
+   node2 |     2 | localhost:5502 | 0/5001F00 |    read-only |          report_lsn |          report_lsn
+   node3 |     3 | localhost:5503 |   0/0     |       none   |           secondary |          report_lsn
+
+``node3`` is assigned ``report_lsn`` but its current state is still
+``secondary`` — it has not reported and likely never will.  The election is
+stuck.  After confirming ``node3`` is permanently lost::
+
+   $ pg_autoctl perform failover --allow-data-loss
+   11:24:05 INFO  Disabling guard_data_loss for this failover (--allow-data-loss)
+   11:24:05 INFO  Listening monitor notifications about state changes in formation "default" and group 0
+       Time |  Name |  Node |      Host:Port |       Current State |      Assigned State
+   ---------+-------+-------+----------------+---------------------+--------------------
+   11:24:06 | node2 |     2 | localhost:5502 |          report_lsn |   prepare_promotion
+   11:24:06 | node2 |     2 | localhost:5502 |   prepare_promotion |   prepare_promotion
+   11:24:06 | node2 |     2 | localhost:5502 |   prepare_promotion |    stop_replication
+   11:24:07 | node2 |     2 | localhost:5502 |    stop_replication |        wait_primary
+   11:24:07 | node2 |     2 | localhost:5502 |        wait_primary |        wait_primary
+
+   $ pg_autoctl show state
+    Name |  Node |      Host:Port |       LSN |   Connection |       Current State |      Assigned State
+   ------+-------+----------------+-----------+--------------+---------------------+--------------------
+   node1 |     1 | localhost:5501 |   0/0     |       none   |            draining |            draining
+   node2 |     2 | localhost:5502 | 0/5001F00 |   read-write |        wait_primary |        wait_primary
+   node3 |     3 | localhost:5503 |   0/0     |       none   |           secondary |          report_lsn
+
+``node2`` is now in ``wait_primary``.  It will become ``primary`` as soon as
+either ``node1`` or ``node3`` reconnects and joins as a standby.

docs/requirements.txt

@@ -2,3 +2,4 @@ Sphinx==8.2.3
 sphinx-rtd-theme==3.0.2
 docutils==0.21.2
 readthedocs-sphinx-search==0.1.0
+sphinxcontrib-mermaid==2.0.3

docs/tikz/Makefile

@@ -1,4 +1,4 @@
-SRC = $(wildcard arch*.tex fsm.tex)
+SRC = $(wildcard arch*.tex fsm.tex seq-*.tex)
 PDF = $(SRC:.tex=.pdf)
 SVG = $(SRC:.tex=.svg)
 PNG = $(SRC:.tex=.png)