diff --git a/docs/architecture-multi-standby.rst b/docs/architecture-multi-standby.rst index c526702d2..2fdc24530 100644 --- a/docs/architecture-multi-standby.rst +++ b/docs/architecture-multi-standby.rst @@ -56,6 +56,12 @@ In more details: standby node acknowledges the transactions locally committed, thus degrading your Postgres service to read-only. + The sequence below shows how a commit is confirmed and how a healthy + failover completes when all quorum standbys are available: + + .. figure:: ./tikz/seq-normal-failover.svg + :align: center + 0. It is possible to manually set ``number_sync_standbys`` to zero when having registered two standby nodes to the monitor, overriding the default behavior. @@ -73,6 +79,29 @@ In more details: data will be lost. How much depends on your backup and recovery mechanisms. +The sequence below shows the stuck-election scenario that can arise with any +``number_sync_standbys >= 1`` setting when the primary and one quorum standby +fail at the same time, and how ``--allow-data-loss`` unblocks it: + +.. figure:: ./tikz/seq-stuck-failover.svg + :align: center + +.. note:: + + **Failover when the primary and a quorum standby fail simultaneously.** + If the primary and one quorum standby are lost at the same time, the + surviving standby is assigned ``report_lsn`` but will not be promoted + automatically. The monitor cannot know whether the missing standby + acknowledged the last synchronous commit, and promoting the survivor + could silently discard those transactions. + + Once you have confirmed the missing node is permanently lost and you accept + the potential data loss, unblock the election with:: + + pg_autoctl perform failover --allow-data-loss + + See :ref:`perform_failover_allow_data_loss` for a full explanation. + .. _architecture_setup: Replication Settings and Postgres Architectures diff --git a/docs/conf.py b/docs/conf.py index 6c56594b3..3db4ede21 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -47,7 +47,7 @@ def __init__(self, **options): # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ["sphinx.ext.githubpages"] +extensions = ["sphinx.ext.githubpages", "sphinxcontrib.mermaid"] # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] diff --git a/docs/failover-state-machine.rst b/docs/failover-state-machine.rst index 026eacec7..35027adef 100644 --- a/docs/failover-state-machine.rst +++ b/docs/failover-state-machine.rst @@ -250,6 +250,17 @@ restarting Postgres without a ``primary_conninfo``. This allows the primary node to detect :ref:`network_partitions`, i.e. when the primary can't connect to the monitor and there's no standby listed in ``pg_stat_replication``. +If one or more quorum standbys (nodes counted by ``number_sync_standbys``) +are unreachable and never report their LSN, the monitor will not advance the +election. The missing node may have acknowledged the last synchronous commit +before it disappeared, and promoting a lagging candidate would silently discard +those transactions. This protection is controlled by the +``pgautofailover.guard_data_loss`` GUC (default ``true``). When the missing +node cannot be recovered and the operator is willing to accept the potential +data loss, the election can be unblocked with +:ref:`pg_autoctl_perform_failover` ``--allow-data-loss``. See +:ref:`perform_failover_allow_data_loss` for details. + Fast_forward ^^^^^^^^^^^^ diff --git a/docs/faq.rst b/docs/faq.rst index 876dad748..16e60550f 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -84,6 +84,33 @@ default configuration deployed by ``pg_autoctl create ...``. When a custom Postgres setup is used, please refer to your actual setup to find Postgres logs. +My failover is stuck: standbys are in ``report_lsn`` and nothing moves +----------------------------------------------------------------------- + +This happens when the primary and one or more quorum standbys +(nodes counted by ``number_sync_standbys``) fail at the same time. The +monitor drives all surviving standbys to the ``report_lsn`` state to +determine the most advanced node, but it refuses to promote any candidate +while a quorum member is missing. The missing node may have acknowledged a +synchronous commit that no surviving standby has yet replicated, and +promoting would silently discard those transactions. + +**First, try to bring the missing node back.** If it recovers and reports +its LSN, the election resumes automatically. + +**If the missing node is permanently lost and you accept the data-loss +risk,** unblock the election:: + + pg_autoctl perform failover --allow-data-loss + +The command promotes the most advanced surviving standby. Transactions +acknowledged by the missing node but not yet replicated to any survivor +will be permanently lost once the new primary starts accepting writes. + +See :ref:`perform_failover_allow_data_loss` and the +``pgautofailover.guard_data_loss`` GUC in :ref:`configuration` for a full +explanation. + The state of the system is blocked, what should I do? ----------------------------------------------------- diff --git a/docs/ref/configuration.rst b/docs/ref/configuration.rst index 52d28a090..b09939bda 100644 --- a/docs/ref/configuration.rst +++ b/docs/ref/configuration.rst @@ -101,6 +101,34 @@ database where the extension has been deployed:: setting | 10000 unit | ms short_desc | Wait for at least this much time after startup before initiating a failover. + -[ RECORD 10 ]--------------------------------------------------------------------------------------------------- + name | pgautofailover.guard_data_loss + setting | true + unit | + short_desc | Refuse to proceed with failover when quorum nodes have not yet reported their LSN. + +pgautofailover.guard_data_loss +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``true`` (the default), ``ProceedGroupStateForMSFailover`` refuses to +promote any standby if one or more quorum nodes have not reported their LSN +position during a failover election. This prevents silent data loss: if the +missing node acknowledged a synchronous commit that no surviving standby +replicated, promoting a lagging standby would permanently discard those +transactions. + +Set to ``false`` to allow the election to proceed with only the surviving +candidates, accepting that committed transactions on the missing node(s) may +be lost. The recommended way to use this setting is through +:ref:`pg_autoctl_perform_failover` with the ``--allow-data-loss`` flag, which +scopes the change to a single transaction and emits a server log message for +each guard that is bypassed. + +Setting ``guard_data_loss = false`` globally in ``postgresql.conf`` is +**not** recommended: it would silently suppress the protection for all future +failovers. Use ``ALTER DATABASE pg_auto_failover SET +pgautofailover.guard_data_loss = false;`` only if you want the setting to +persist across monitor restarts with explicit intent, and document the reason. You can edit the parameters as usual with PostgreSQL, either in the ``postgresql.conf`` file or using ``ALTER DATABASE pg_auto_failover SET parameter = diff --git a/docs/ref/pg_autoctl_perform_failover.rst b/docs/ref/pg_autoctl_perform_failover.rst index cb37c2702..563a5ed6a 100644 --- a/docs/ref/pg_autoctl_perform_failover.rst +++ b/docs/ref/pg_autoctl_perform_failover.rst @@ -13,10 +13,11 @@ pg_auto_failover monitor:: usage: pg_autoctl perform failover [ --pgdata --formation --group ] - --pgdata path to data directory - --formation formation to target, defaults to 'default' - --group group to target, defaults to 0 - --wait how many seconds to wait, default to 60 + --pgdata path to data directory + --formation formation to target, defaults to 'default' + --group group to target, defaults to 0 + --wait how many seconds to wait, default to 60 + --allow-data-loss proceed even when quorum nodes have not reported their LSN Description ----------- @@ -34,6 +35,43 @@ The failover orchestration is done in the background by the monitor, so even if the ``pg_autoctl perform failover`` stops on the timeout, the failover orchestration continues at the monitor. +.. _perform_failover_allow_data_loss: + +Recovering a failover stuck in ``report_lsn`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In a formation with ``number_sync_standbys >= 1``, a failover that loses the +primary and one quorum standby at the same time can get permanently stuck. +When the primary fails, the monitor drives all standby nodes into the +``report_lsn`` state so it can elect the most advanced one. If a quorum +standby is unreachable, it never reports its LSN. The monitor then refuses to +promote any remaining candidate — it cannot know whether the missing node +acknowledged the last synchronous commit and holds WAL that no surviving +standby has replicated yet. + +This is the correct conservative default, controlled by the +``pgautofailover.guard_data_loss`` GUC (default ``true``). When you have +determined that the missing node is permanently lost and you are willing to +accept the potential data loss, use ``--allow-data-loss`` to unblock the +election: + +.. code-block:: bash + + pg_autoctl perform failover --allow-data-loss + +The command opens a single transaction on the monitor, sets +``pgautofailover.guard_data_loss`` to ``false`` for that transaction only, +and calls ``perform_failover()``. The monitor then selects the most advanced +surviving candidate and drives it toward ``prepare_promotion``. + +.. warning:: + + ``--allow-data-loss`` means exactly what it says. If the missing quorum + standby had acknowledged a synchronous commit that no surviving standby + replicated, those transactions will be permanently lost once the new primary + starts accepting writes. Use this option only when the missing node cannot + be recovered and the cluster being stuck is the worse outcome. + Options ------- @@ -60,6 +98,14 @@ Options the timeout has elapsed, whichever comes first. The value 0 (zero) disables the timeout and allows the command to wait forever. +--allow-data-loss + + Disable the ``pgautofailover.guard_data_loss`` protection for this + failover only. When set, the monitor will promote the most advanced + surviving candidate even if one or more quorum standby nodes have not yet + reported their LSN position. Committed transactions on the missing node + may be permanently lost. See :ref:`perform_failover_allow_data_loss`. + Environment ----------- @@ -141,3 +187,42 @@ Examples node1 | 1 | localhost:5501 | 0/4000F50 | read-only | secondary | secondary node2 | 2 | localhost:5502 | 0/4000F50 | read-write | primary | primary node3 | 3 | localhost:5503 | 0/4000F50 | read-only | secondary | secondary + +Example: unblocking a stuck election with ``--allow-data-loss`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In this example ``node1`` (primary) and ``node3`` (a quorum standby) have +both failed. ``node2`` reaches ``report_lsn`` and waits indefinitely because +``node3`` never reported its LSN:: + + $ pg_autoctl show state + Name | Node | Host:Port | LSN | Connection | Current State | Assigned State + ------+-------+----------------+-----------+--------------+---------------------+-------------------- + node1 | 1 | localhost:5501 | 0/0 | none | draining | draining + node2 | 2 | localhost:5502 | 0/5001F00 | read-only | report_lsn | report_lsn + node3 | 3 | localhost:5503 | 0/0 | none | secondary | report_lsn + +``node3`` is assigned ``report_lsn`` but its current state is still +``secondary`` — it has not reported and likely never will. The election is +stuck. After confirming ``node3`` is permanently lost:: + + $ pg_autoctl perform failover --allow-data-loss + 11:24:05 INFO Disabling guard_data_loss for this failover (--allow-data-loss) + 11:24:05 INFO Listening monitor notifications about state changes in formation "default" and group 0 + Time | Name | Node | Host:Port | Current State | Assigned State + ---------+-------+-------+----------------+---------------------+-------------------- + 11:24:06 | node2 | 2 | localhost:5502 | report_lsn | prepare_promotion + 11:24:06 | node2 | 2 | localhost:5502 | prepare_promotion | prepare_promotion + 11:24:06 | node2 | 2 | localhost:5502 | prepare_promotion | stop_replication + 11:24:07 | node2 | 2 | localhost:5502 | stop_replication | wait_primary + 11:24:07 | node2 | 2 | localhost:5502 | wait_primary | wait_primary + + $ pg_autoctl show state + Name | Node | Host:Port | LSN | Connection | Current State | Assigned State + ------+-------+----------------+-----------+--------------+---------------------+-------------------- + node1 | 1 | localhost:5501 | 0/0 | none | draining | draining + node2 | 2 | localhost:5502 | 0/5001F00 | read-write | wait_primary | wait_primary + node3 | 3 | localhost:5503 | 0/0 | none | secondary | report_lsn + +``node2`` is now in ``wait_primary``. It will become ``primary`` as soon as +either ``node1`` or ``node3`` reconnects and joins as a standby. diff --git a/docs/requirements.txt b/docs/requirements.txt index 27103625a..d2bcf11b5 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -2,3 +2,4 @@ Sphinx==8.2.3 sphinx-rtd-theme==3.0.2 docutils==0.21.2 readthedocs-sphinx-search==0.1.0 +sphinxcontrib-mermaid==2.0.3 diff --git a/docs/tikz/Makefile b/docs/tikz/Makefile index 84567c935..016a6e7ef 100644 --- a/docs/tikz/Makefile +++ b/docs/tikz/Makefile @@ -1,4 +1,4 @@ -SRC = $(wildcard arch*.tex fsm.tex) +SRC = $(wildcard arch*.tex fsm.tex seq-*.tex) PDF = $(SRC:.tex=.pdf) SVG = $(SRC:.tex=.svg) PNG = $(SRC:.tex=.png) diff --git a/docs/tikz/seq-normal-failover.svg b/docs/tikz/seq-normal-failover.svg new file mode 100644 index 000000000..63645417d --- /dev/null +++ b/docs/tikz/seq-normal-failover.svg @@ -0,0 +1,776 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-normal-failover.tex b/docs/tikz/seq-normal-failover.tex new file mode 100644 index 000000000..04f766acd --- /dev/null +++ b/docs/tikz/seq-normal-failover.tex @@ -0,0 +1,125 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm): wider spacing avoids mid-arrow label clashes +%% P=0 B=8 C=15 M=23 +\def\px{0} +\def\bx{8} +\def\cx{15} +\def\mx{23} +\def\ybot{14.2} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Bh) at (\bx,0) {Standby B}; +\node[actor, fill=sbox, text=stxt] (Ch) at (\cx,0) {Standby C}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\bx,0.35) -- (\bx,\ybot); +\draw[ll] (\cx,0.35) -- (\cx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-2.2,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\bx,1.6) + node[lbl,midway,above] {WAL stream}; +\draw[msg] (\px,2.3) -- (\cx,2.3) + node[lbl,midway,above] {WAL stream}; + +%% ── client commit ─────────────────────────────────────────────────────── +\node[seclbl] at (-2.2,3.3) {client commit}; + +%% Arrow P→B; label near P to stay clear of C's lifeline +\draw[msg, color=pbox, line width=0.8pt] + (\px,3.6) -- (\bx,3.6); +\node[lbl, above, text=pbox] at (\px+2.4, 3.6) + {wait flush + ack}; +\node[lbl, below, text=pbox, font=\small\itshape] at (\px+2.4, 3.6) + {(ANY 1 of B or C)}; + +\draw[rsp] (\bx,4.5) -- (\px,4.5) + node[lbl,midway,above] {ack}; + +\node[note, fill=green!15, draw=green!50!black] + at (\px+1.8, 5.3) {\small commit confirmed $\checkmark$}; + +%% ── failover ──────────────────────────────────────────────────────────── +\node[seclbl] at (-2.2,6.2) {failover}; + +\node[note, fill=red!20, draw=red!60] + at (\px, 6.5) {\small\bfseries\color{red!80!black} $\times$~primary fails}; + +%% red dashed lifeline after failure +\draw[lldead] (\px,6.85) -- (\px,\ybot); + +%% Monitor → B: label near Monitor end (pos≈0.10) to clear C lifeline +\draw[msg] (\mx,7.5) -- (\bx,7.5); +\node[lbl, above] at (\mx-2.0, 7.5) {assign report\_lsn}; + +%% Monitor → C +\draw[msg] (\mx,8.3) -- (\cx,8.3) + node[lbl,midway,above] {assign report\_lsn}; + +%% B → Monitor: label near B to clear C lifeline +\draw[rsp] (\bx,9.2) -- (\mx,9.2); +\node[lbl, above] at (\bx+2.2, 9.2) {LSN = 0/5001F00}; + +%% C → Monitor +\draw[rsp] (\cx,10.0) -- (\mx,10.0) + node[lbl,midway,above] {LSN = 0/4FFF000}; + +%% Monitor note: B most advanced — placed below M lifeline, above next arrow +\node[note, fill=yellow!30, draw=orange!60] + at (\mx, 11.0) {\small B most advanced $\Rightarrow$ select B}; + +%% Monitor → B: assign prepare_promotion — label near Monitor end +\draw[msg, color=pbox, line width=1pt] + (\mx,12.0) -- (\bx,12.0); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.8, 12.0) + {assign prepare\_promotion}; + +%% outcome bar (midpoint of B–C span = (8+15)/2 = 11.5) +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (11.5, 13.1) {$\checkmark$~~B becomes primary \textemdash\ no data loss}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=red!25, draw=red!60, text=red!80!black] + at (\px, \ybot+0.35) {\small Primary $\times$}; +\node[actor, fill=sbox, text=stxt] at (\bx, \ybot+0.35) {Standby B}; +\node[actor, fill=sbox, text=stxt] at (\cx, \ybot+0.35) {Standby C}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-stuck-failover.svg b/docs/tikz/seq-stuck-failover.svg new file mode 100644 index 000000000..709aeab01 --- /dev/null +++ b/docs/tikz/seq-stuck-failover.svg @@ -0,0 +1,1106 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-stuck-failover.tex b/docs/tikz/seq-stuck-failover.tex new file mode 100644 index 000000000..efe2d136d --- /dev/null +++ b/docs/tikz/seq-stuck-failover.tex @@ -0,0 +1,136 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 B=8 C=15 M=23 +\def\px{0} +\def\bx{8} +\def\cx{15} +\def\mx{23} +\def\ybot{16.2} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] at (\bx,0) {Standby B}; +\node[actor, fill=sbox, text=stxt] at (\cx,0) {Standby C}; +\node[actor, fill=mbox, text=mtxt] at (\mx,0) {Monitor}; + +%% lifelines (all start alive) +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\bx,0.35) -- (\bx,\ybot); +\draw[ll] (\cx,0.35) -- (\cx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-2.2,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\bx,1.6) node[lbl,midway,above] {WAL stream}; +\draw[msg] (\px,2.3) -- (\cx,2.3) node[lbl,midway,above] {WAL stream}; + +%% ── double failure ────────────────────────────────────────────────────── +\node[seclbl] at (-2.2,3.3) {double failure}; + +%% Two separate failure boxes, one per failed node +\node[note, fill=red!20, draw=red!60] + at (\px, 3.65) {\small\bfseries\color{red!80!black} $\times$~primary fails}; +\node[note, fill=red!20, draw=red!60] + at (\bx, 3.65) {\small\bfseries\color{red!80!black} $\times$~Standby B fails}; + +%% Red lifelines after failure +\draw[lldead] (\px,4.05) -- (\px,\ybot); +\draw[lldead] (\bx,4.05) -- (\bx,\ybot); + +%% ── report_lsn election ───────────────────────────────────────────────── +\node[seclbl] at (-2.2,5.0) {report\_lsn election}; + +%% Monitor → C: only surviving standby +\draw[msg] (\mx,5.3) -- (\cx,5.3) + node[lbl,midway,above] {assign report\_lsn}; + +%% C → Monitor +\draw[rsp] (\cx,6.2) -- (\mx,6.2) + node[lbl,midway,above] {LSN = 0/4FFF000}; + +%% ── monitor blocked ───────────────────────────────────────────────────── +\node[seclbl] at (-2.2,7.2) {stuck}; + +%% Block note on Monitor lifeline +\node[note, fill=red!15, draw=red!50, text width=5.5cm] + at (\mx, 8.1) + {\bfseries\color{red!80!black} BLOCKED\\[3pt] + \normalfont quorum member B has not\\ + reported its LSN yet\\[2pt] + promoting could discard\\ + committed transactions}; + +%% wait loop arrow (self-arrow on Monitor) +\draw[gray!50, line width=0.5pt, ->] + (\mx+0.5, 7.2) .. controls (\mx+2.0,7.5) and (\mx+2.0,9.0) .. (\mx+0.5,9.3) + node[right, font=\small\itshape, text=gray!70] {waiting\ldots}; + +%% ── operator action ───────────────────────────────────────────────────── +\node[seclbl] at (-2.2,10.4) {operator action}; + +%% (px+cx)/2 = 7.5 ; (cx+mx)/2 = 19.0 ; (px+mx)/2 = 11.5 +\node[note, fill=blue!10, draw=blue!40, text width=8.0cm] + at (7.5, 10.8) + {\ttfamily\small pg\_autoctl perform failover --allow-data-loss}; + +%% operator arrow → Monitor +\draw[msg, color=blue!60, line width=0.8pt] + (11.5, 11.4) -- (\mx-1.6, 11.4); + +%% Monitor → C: assign prepare_promotion — label near Monitor end +\draw[msg, color=pbox, line width=1pt] + (\mx,12.4) -- (\cx,12.4); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.8,12.4) + {assign prepare\_promotion}; + +%% outcome +\node[note, fill=yellow!20, draw=orange!60, font=\small\bfseries] + at (19.0, 13.5) {C becomes primary}; + +\node[note, fill=red!10, draw=red!40, text width=10cm, font=\small] + at (11.5, 14.7) + {\color{red!80!black}\bfseries Warning:~\normalfont + transactions acknowledged by B but not yet\\ + replicated to C may be permanently lost}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=red!25, draw=red!60, text=red!80!black] + at (\px, \ybot+0.35) {\small Primary $\times$}; +\node[actor, fill=red!25, draw=red!60, text=red!80!black] + at (\bx, \ybot+0.35) {\small Standby B $\times$}; +\node[actor, fill=sbox, text=stxt] at (\cx, \ybot+0.35) {Standby C}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/src/bin/pg_autoctl/cli_perform.c b/src/bin/pg_autoctl/cli_perform.c index f3fd6415e..50c990bbb 100644 --- a/src/bin/pg_autoctl/cli_perform.c +++ b/src/bin/pg_autoctl/cli_perform.c @@ -29,10 +29,14 @@ CommandLine perform_failover_command = make_command("failover", "Perform a failover for given formation and group", " [ --pgdata --formation --group ] ", - " --pgdata path to data directory\n" - " --formation formation to target, defaults to 'default'\n" - " --group group to target, defaults to 0\n" - " --wait how many seconds to wait, default to 60 \n", + " --pgdata path to data directory\n" + " --formation formation to target, defaults to 'default'\n" + " --group group to target, defaults to 0\n" + " --wait how many seconds to wait, default to 60 \n" + " --allow-data-loss Proceed even when quorum nodes have not reported " + "their\n" + " LSN; committed transactions on missing nodes may " + "be lost\n", cli_perform_failover_getopts, cli_perform_failover); @@ -40,10 +44,14 @@ CommandLine perform_switchover_command = make_command("switchover", "Perform a switchover for given formation and group", " [ --pgdata --formation --group ] ", - " --pgdata path to data directory\n" - " --formation formation to target, defaults to 'default'\n" - " --group group to target, defaults to 0\n" - " --wait how many seconds to wait, default to 60 \n", + " --pgdata path to data directory\n" + " --formation formation to target, defaults to 'default'\n" + " --group group to target, defaults to 0\n" + " --wait how many seconds to wait, default to 60 \n" + " --allow-data-loss Proceed even when quorum nodes have not reported " + "their\n" + " LSN; committed transactions on missing nodes may " + "be lost\n", cli_perform_failover_getopts, cli_perform_failover); @@ -87,6 +95,7 @@ cli_perform_failover_getopts(int argc, char **argv) { "formation", required_argument, NULL, 'f' }, { "group", required_argument, NULL, 'g' }, { "wait", required_argument, NULL, 'w' }, + { "allow-data-loss", no_argument, NULL, 'A' }, { "version", no_argument, NULL, 'V' }, { "verbose", no_argument, NULL, 'v' }, { "quiet", no_argument, NULL, 'q' }, @@ -108,7 +117,7 @@ cli_perform_failover_getopts(int argc, char **argv) optind = 0; - while ((c = getopt_long(argc, argv, "D:f:g:n:Vvqh", + while ((c = getopt_long(argc, argv, "D:f:g:n:AVvqh", long_options, &option_index)) != -1) { switch (c) @@ -164,6 +173,13 @@ cli_perform_failover_getopts(int argc, char **argv) break; } + case 'A': + { + options.allowDataLoss = true; + log_trace("--allow-data-loss"); + break; + } + case 'V': { /* keeper_cli_print_version prints version and exits. */ @@ -284,11 +300,31 @@ cli_perform_failover(int argc, char **argv) exit(EXIT_CODE_MONITOR); } - if (!monitor_perform_failover(&monitor, config.formation, config.groupId)) + bool performOk; + + if (keeperOptions.allowDataLoss) { - log_fatal("Failed to perform failover/switchover, " - "see above for details"); - exit(EXIT_CODE_MONITOR); + performOk = monitor_perform_failover_allow_data_loss( + &monitor, config.formation, config.groupId); + + if (!performOk) + { + log_fatal("Failed to perform failover with --allow-data-loss, " + "see above for details"); + exit(EXIT_CODE_MONITOR); + } + } + else + { + performOk = monitor_perform_failover( + &monitor, config.formation, config.groupId); + + if (!performOk) + { + log_fatal("Failed to perform failover/switchover, " + "see above for details"); + exit(EXIT_CODE_MONITOR); + } } /* process state changes notification until we have a new primary */ diff --git a/src/bin/pg_autoctl/keeper_config.h b/src/bin/pg_autoctl/keeper_config.h index 73e6cbd62..0c71a65e0 100644 --- a/src/bin/pg_autoctl/keeper_config.h +++ b/src/bin/pg_autoctl/keeper_config.h @@ -72,6 +72,9 @@ typedef struct KeeperConfig int citus_coordinator_wait_timeout; int citus_coordinator_wait_max_retries; int listen_notifications_timeout; + + /* allow data loss during a perform failover operation */ + bool allowDataLoss; } KeeperConfig; #define PG_AUTOCTL_MONITOR_IS_DISABLED(config) \ diff --git a/src/bin/pg_autoctl/monitor.c b/src/bin/pg_autoctl/monitor.c index a781800c6..7ceec151e 100644 --- a/src/bin/pg_autoctl/monitor.c +++ b/src/bin/pg_autoctl/monitor.c @@ -1534,6 +1534,55 @@ monitor_perform_failover(Monitor *monitor, char *formation, int group) } +/* + * monitor_perform_failover_allow_data_loss runs perform_failover in a + * transaction with guard_data_loss disabled, accepting the risk of data + * loss when quorum nodes have not reported their LSN. + */ +bool +monitor_perform_failover_allow_data_loss(Monitor *monitor, + char *formation, + int group) +{ + PGSQL *pgsql = &monitor->pgsql; + const char *sql = "SELECT pgautofailover.perform_failover($1, $2)"; + int paramCount = 2; + Oid paramTypes[2] = { TEXTOID, INT4OID }; + const char *paramValues[2]; + IntString groupString = intToString(group); + + paramValues[0] = formation; + paramValues[1] = groupString.strValue; + + if (!pgsql_begin(pgsql)) + { + log_error("Failed to open transaction on monitor"); + return false; + } + + if (!pgsql_execute(pgsql, + "SET LOCAL pgautofailover.guard_data_loss TO false")) + { + log_error("Failed to disable guard_data_loss on monitor"); + (void) pgsql_rollback(pgsql); + return false; + } + + if (!pgsql_execute_with_params(pgsql, sql, + paramCount, paramTypes, paramValues, + NULL, NULL)) + { + log_error("Failed to perform failover with --allow-data-loss " + "for formation %s and group %d", + formation, group); + (void) pgsql_rollback(pgsql); + return false; + } + + return pgsql_commit(pgsql); +} + + /* * monitor_perform_promotion calls the pgautofailover.perform_promotion * function on the monitor. diff --git a/src/bin/pg_autoctl/monitor.h b/src/bin/pg_autoctl/monitor.h index 6c6a95477..aea12ae2e 100644 --- a/src/bin/pg_autoctl/monitor.h +++ b/src/bin/pg_autoctl/monitor.h @@ -173,6 +173,9 @@ bool monitor_get_groupId_from_name(Monitor *monitor, int *groupId); bool monitor_perform_failover(Monitor *monitor, char *formation, int group); +bool monitor_perform_failover_allow_data_loss(Monitor *monitor, + char *formation, + int group); bool monitor_perform_promotion(Monitor *monitor, char *formation, char *name); bool monitor_get_current_state(Monitor *monitor, char *formation, int group, diff --git a/src/monitor/Makefile b/src/monitor/Makefile index 39606a2e4..eca708ccf 100644 --- a/src/monitor/Makefile +++ b/src/monitor/Makefile @@ -14,7 +14,7 @@ MODULE_big = $(EXTENSION) OBJS = $(patsubst ${SRC_DIR}%.c,%.o,$(wildcard ${SRC_DIR}*.c)) PG_CPPFLAGS = -std=c99 -Wall -Werror -Wno-unused-parameter -Iinclude -I$(libpq_srcdir) -g SHLIB_LINK = $(libpq) -REGRESS = create_extension monitor workers node_active_protocol dummy_update drop_extension upgrade +REGRESS = create_extension monitor workers node_active_protocol guard_data_loss dummy_update drop_extension upgrade PG_CONFIG ?= pg_config PGXS = $(shell $(PG_CONFIG) --pgxs) diff --git a/src/monitor/expected/guard_data_loss.out b/src/monitor/expected/guard_data_loss.out new file mode 100644 index 000000000..98b07b2f7 --- /dev/null +++ b/src/monitor/expected/guard_data_loss.out @@ -0,0 +1,310 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression tests for pgautofailover.guard_data_loss GUC. +-- +-- Verifies that: +-- 1. With guard_data_loss=true (default), a 3-node failover stuck in the +-- report_lsn phase (one quorum node has not yet reported) does NOT +-- proceed -- perform_failover returns without selecting a candidate. +-- +-- 2. With guard_data_loss=false the same scenario proceeds: the surviving +-- report_lsn node is selected as the failover candidate even though +-- the other quorum node never reported its LSN. +-- +\x on +-- ── formation setup ────────────────────────────────────────────────────────── +SELECT pgautofailover.create_formation('gdl_test', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+------------------------------ +create_formation | (gdl_test,pgsql,postgres,t,1) + +-- Register three nodes (number_sync_standbys=1, so two quorum standbys needed +-- to safely commit). +SELECT * + FROM pgautofailover.register_node('gdl_test', 'p', 5432, + 'postgres', 'p', 1); +-[ RECORD 1 ]---------------+------- +assigned_node_id | 10 +assigned_group_id | 0 +assigned_group_state | single +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | p + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 'p' \gset +SELECT * + FROM pgautofailover.register_node('gdl_test', 's1', 5432, + 'postgres', 's1', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 11 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s1 + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 's1' \gset +SELECT * + FROM pgautofailover.register_node('gdl_test', 's2', 5432, + 'postgres', 's2', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 12 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s2 + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 's2' \gset +-- ── bootstrap ──────────────────────────────────────────────────────────────── +-- +-- Drive the FSM from init through to primary + secondary + secondary. +-- p: single (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'single'); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +-- s1: wait_standby (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +-- p: single → wait_primary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +-- p: wait_primary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +-- s1: wait_standby → catchingup +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +-- s1: catchingup → secondary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +-- s1: secondary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +-- p: wait_primary → primary (now that s1 is secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +-- p: primary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +-- s1: secondary (confirm after primary confirmed) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +-- s2: wait_standby (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +-- s2: catchingup +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +-- s2: secondary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +-- p: primary (refresh to pick up new secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +-- Verify final formation state: p=primary, s1=secondary, s2=secondary. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; +-[ RECORD 1 ]-+--------------- +nodename | p +goalstate | apply_settings +reportedstate | primary +-[ RECORD 2 ]-+--------------- +nodename | s1 +goalstate | secondary +reportedstate | secondary +-[ RECORD 3 ]-+--------------- +nodename | s2 +goalstate | catchingup +reportedstate | secondary + +-- ── simulate stuck failover ────────────────────────────────────────────────── +-- +-- Kill the primary (p) and one quorum standby (s2) simultaneously. +-- We simulate this by: +-- 1. Marking p and s2 as unhealthy (health=BAD, old reporttime). +-- 2. Manually placing s1 in report_lsn/report_lsn (it already called +-- node_active and transitioned itself), while s2 is still in +-- secondary/secondary with goalState=report_lsn assigned (missing). +-- +-- This mirrors what happens in production: the monitor assigns report_lsn +-- to all standbys, s1 reports immediately, but s2 is dead and never will. +SET pgautofailover.startup_grace_period = 1; +-- Mark p as dead (health=BAD, stale report). +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds' + WHERE formationid = 'gdl_test' AND nodename = 'p'; +-- Mark s2 as dead as well (health=BAD, stale report, pgIsRunning=false). +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds', + reportedpgisrunning = false + WHERE formationid = 'gdl_test' AND nodename = 's2'; +-- Demote p: set it to draining/draining so the FSM sees no primary. +UPDATE pgautofailover.node + SET goalstate = 'draining', reportedstate = 'draining' + WHERE formationid = 'gdl_test' AND nodename = 'p'; +-- Put s1 in report_lsn/report_lsn (it already ran node_active and confirmed). +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'report_lsn' + WHERE formationid = 'gdl_test' AND nodename = 's1'; +-- Put s2 in secondary/secondary with goalState=report_lsn assigned but NOT +-- yet confirmed (missingNodesCount will be > 0 for s2). +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'secondary' + WHERE formationid = 'gdl_test' AND nodename = 's2'; +-- Verify the manufactured stuck state. +SELECT nodename, goalstate, reportedstate, health + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; +-[ RECORD 1 ]-+----------- +nodename | p +goalstate | draining +reportedstate | draining +health | 0 +-[ RECORD 2 ]-+----------- +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +health | -1 +-[ RECORD 3 ]-+----------- +nodename | s2 +goalstate | report_lsn +reportedstate | secondary +health | 0 + +-- ── test 1: guard_data_loss = true (default) blocks ───────────────────────── +-- +-- perform_failover should find no primary, locate s1 in report_lsn, drive +-- ProceedGroupState for s1, which calls ProceedGroupStateForMSFailover, which +-- hits the guard and returns false without selecting a candidate. +-- The node states should remain unchanged (no candidate selected). +-- Default is true; make it explicit. +SET pgautofailover.guard_data_loss TO true; +SELECT pgautofailover.perform_failover('gdl_test', 0); +-[ RECORD 1 ]----+- +perform_failover | + +-- States must be unchanged: no candidate was promoted. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; +-[ RECORD 1 ]-+----------- +nodename | p +goalstate | draining +reportedstate | draining +-[ RECORD 2 ]-+----------- +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +-[ RECORD 3 ]-+----------- +nodename | s2 +goalstate | report_lsn +reportedstate | secondary + +-- ── test 2: guard_data_loss = false allows progress ───────────────────────── +-- +-- With guard_data_loss disabled the failover proceeds despite s2 being missing. +-- s1 (the only node in report_lsn) should be selected as the candidate and +-- transition toward prepare_promotion / stop_replication. +SET pgautofailover.guard_data_loss TO false; +SELECT pgautofailover.perform_failover('gdl_test', 0); +-[ RECORD 1 ]----+- +perform_failover | + +-- s1 should now have a goal state of prepare_promotion or stop_replication, +-- indicating the failover candidate was selected. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; +-[ RECORD 1 ]-+------------------ +nodename | p +goalstate | draining +reportedstate | draining +-[ RECORD 2 ]-+------------------ +nodename | s1 +goalstate | prepare_promotion +reportedstate | report_lsn +-[ RECORD 3 ]-+------------------ +nodename | s2 +goalstate | report_lsn +reportedstate | secondary + +RESET pgautofailover.guard_data_loss; +RESET pgautofailover.startup_grace_period; diff --git a/src/monitor/group_state_machine.c b/src/monitor/group_state_machine.c index 145ad6ba6..0ad622dfa 100644 --- a/src/monitor/group_state_machine.c +++ b/src/monitor/group_state_machine.c @@ -18,6 +18,7 @@ #include "formation_metadata.h" #include "group_state_machine.h" +#include "metadata.h" #include "node_metadata.h" #include "notifications.h" #include "replication_state.h" @@ -1404,18 +1405,31 @@ ProceedGroupStateForMSFailover(GroupStateContext *ctx, { char message[BUFSIZE] = { 0 }; + if (GuardDataLoss) + { + LogAndNotifyMessage( + message, BUFSIZE, + "Failover still in progress after %d nodes reported their LSN " + "and we are waiting for %d nodes to report, " + "activeNode is " NODE_FORMAT + " and reported state \"%s\"", + candidateList.candidateCount, + candidateList.missingNodesCount, + NODE_FORMAT_ARGS(activeNode), + ReplicationStateGetName(activeNode->reportedState)); + + return false; + } + LogAndNotifyMessage( message, BUFSIZE, - "Failover still in progress after %d nodes reported their LSN " - "and we are waiting for %d nodes to report, " - "activeNode is " NODE_FORMAT - " and reported state \"%s\"", - candidateList.candidateCount, + "Proceeding with failover despite %d unreported quorum node(s): " + "pgautofailover.guard_data_loss is false. " + "Committed transactions on missing node(s) may be lost. " + "activeNode is " NODE_FORMAT " and reported state \"%s\"", candidateList.missingNodesCount, NODE_FORMAT_ARGS(activeNode), ReplicationStateGetName(activeNode->reportedState)); - - return false; } /* @@ -1438,29 +1452,45 @@ ProceedGroupStateForMSFailover(GroupStateContext *ctx, } /* not enough candidates to promote and then accept writes, pass */ - else if (candidateList.quorumCandidateCount < minCandidates) + if (candidateList.quorumCandidateCount < minCandidates) { char message[BUFSIZE] = { 0 }; + if (GuardDataLoss) + { + LogAndNotifyMessage( + message, BUFSIZE, + "Failover still in progress with %d candidates that participate " + "in the quorum having reported their LSN: %d nodes are required " + "in the quorum to satisfy number_sync_standbys=%d in " + "formation \"%s\", activeNode is " NODE_FORMAT + " and reported state \"%s\"", + candidateList.quorumCandidateCount, + minCandidates, + ctx->formation->number_sync_standbys, + ctx->formation->formationId, + NODE_FORMAT_ARGS(activeNode), + ReplicationStateGetName(activeNode->reportedState)); + + return false; + } + LogAndNotifyMessage( message, BUFSIZE, - "Failover still in progress with %d candidates that participate " - "in the quorum having reported their LSN: %d nodes are required " - "in the quorum to satisfy number_sync_standbys=%d in " - "formation \"%s\", activeNode is " NODE_FORMAT - " and reported state \"%s\"", + "Proceeding with failover with only %d quorum candidate(s) despite " + "number_sync_standbys=%d requiring %d: " + "pgautofailover.guard_data_loss is false. " + "The new primary may start in wait_primary state with fewer " + "sync standbys than required. " + "activeNode is " NODE_FORMAT " and reported state \"%s\"", candidateList.quorumCandidateCount, - minCandidates, ctx->formation->number_sync_standbys, - ctx->formation->formationId, + minCandidates, NODE_FORMAT_ARGS(activeNode), ReplicationStateGetName(activeNode->reportedState)); - - return false; } /* enough candidates to promote and then accept writes, let's do it! */ - else { /* build the list of most advanced standby nodes, not ordered */ List *mostAdvancedNodeList = diff --git a/src/monitor/metadata.c b/src/monitor/metadata.c index 0ddf81d1d..81614b2eb 100644 --- a/src/monitor/metadata.c +++ b/src/monitor/metadata.c @@ -39,6 +39,7 @@ #include "utils/relcache.h" bool EnableVersionChecks = true; /* version checks are enabled */ +bool GuardDataLoss = true; /* guard against data loss during failover */ /* * pgAutoFailoverRelationId returns the OID of a given relation in the diff --git a/src/monitor/metadata.h b/src/monitor/metadata.h index b99049e80..f7325c86e 100644 --- a/src/monitor/metadata.h +++ b/src/monitor/metadata.h @@ -38,6 +38,9 @@ typedef enum AutoFailoverHALocktagClass /* GUC variable for version checks, true by default */ extern bool EnableVersionChecks; +/* GUC variable to guard against data loss during failover, true by default */ +extern bool GuardDataLoss; + /* public function declarations */ extern Oid pgAutoFailoverRelationId(const char *relname); extern Oid pgAutoFailoverSchemaId(void); diff --git a/src/monitor/node_active_protocol.c b/src/monitor/node_active_protocol.c index 83f4453e1..f112e0501 100644 --- a/src/monitor/node_active_protocol.c +++ b/src/monitor/node_active_protocol.c @@ -1313,6 +1313,32 @@ perform_failover(PG_FUNCTION_ARGS) if (primaryNode == NULL) { + /* + * No primary to initiate a failover from. Maybe a failover is already + * in progress and stuck waiting for quorum nodes to report their LSN. + * In that case, drive the state machine for the first report_lsn node: + * if guard_data_loss is false this will proceed despite missing nodes. + */ + AutoFailoverNode *reportLsnNode = NULL; + ListCell *nodeCell = NULL; + + foreach(nodeCell, groupNodeList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(nodeCell); + + if (IsCurrentState(node, REPLICATION_STATE_REPORT_LSN)) + { + reportLsnNode = node; + break; + } + } + + if (reportLsnNode != NULL) + { + (void) ProceedGroupState(reportLsnNode); + PG_RETURN_VOID(); + } + ereport(ERROR, (errmsg("couldn't find the primary node in formation \"%s\", " "group %d", formationId, groupId))); diff --git a/src/monitor/pg_auto_failover.c b/src/monitor/pg_auto_failover.c index 80778fe6f..40b7f1254 100644 --- a/src/monitor/pg_auto_failover.c +++ b/src/monitor/pg_auto_failover.c @@ -181,6 +181,19 @@ StartMonitorNode(void) NULL, &StartupGracePeriodMs, 10 * 1000, 1, INT_MAX, PGC_SUSET, GUC_UNIT_MS, NULL, NULL, NULL); + DefineCustomBoolVariable("pgautofailover.guard_data_loss", + "Refuse to proceed with failover when quorum nodes have not " + "yet reported their LSN, preventing potential data loss from " + "the synchronous replication gap. Set to false to allow " + "failover to proceed despite missing quorum nodes, accepting " + "the risk that committed transactions may be lost.", + NULL, + &GuardDataLoss, + true, + PGC_USERSET, + 0, + NULL, NULL, NULL); + PreviousProcessUtility_hook = ProcessUtility_hook; ProcessUtility_hook = pgautofailover_ProcessUtility; diff --git a/src/monitor/sql/guard_data_loss.sql b/src/monitor/sql/guard_data_loss.sql new file mode 100644 index 000000000..09066b844 --- /dev/null +++ b/src/monitor/sql/guard_data_loss.sql @@ -0,0 +1,222 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression tests for pgautofailover.guard_data_loss GUC. +-- +-- Verifies that: +-- 1. With guard_data_loss=true (default), a 3-node failover stuck in the +-- report_lsn phase (one quorum node has not yet reported) does NOT +-- proceed -- perform_failover returns without selecting a candidate. +-- +-- 2. With guard_data_loss=false the same scenario proceeds: the surviving +-- report_lsn node is selected as the failover candidate even though +-- the other quorum node never reported its LSN. +-- + +\x on + +-- ── formation setup ────────────────────────────────────────────────────────── + +SELECT pgautofailover.create_formation('gdl_test', 'pgsql', 'postgres', true, 1); + +-- Register three nodes (number_sync_standbys=1, so two quorum standbys needed +-- to safely commit). +SELECT * + FROM pgautofailover.register_node('gdl_test', 'p', 5432, + 'postgres', 'p', 1); + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 'p' \gset + +SELECT * + FROM pgautofailover.register_node('gdl_test', 's1', 5432, + 'postgres', 's1', 1); + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 's1' \gset + +SELECT * + FROM pgautofailover.register_node('gdl_test', 's2', 5432, + 'postgres', 's2', 1); + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'gdl_test' AND nodename = 's2' \gset + +-- ── bootstrap ──────────────────────────────────────────────────────────────── +-- +-- Drive the FSM from init through to primary + secondary + secondary. + +-- p: single (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'single'); + +-- s1: wait_standby (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'wait_standby'); + +-- p: single → wait_primary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); + +-- p: wait_primary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +-- s1: wait_standby → catchingup +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'wait_standby'); + +-- s1: catchingup → secondary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +-- s1: secondary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +-- p: wait_primary → primary (now that s1 is secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +-- p: primary (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +-- s1: secondary (confirm after primary confirmed) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +-- s2: wait_standby (confirm) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'wait_standby'); + +-- s2: catchingup +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +-- s2: secondary +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +-- p: primary (refresh to pick up new secondary) +SELECT assigned_group_state + FROM pgautofailover.node_active('gdl_test', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +-- Verify final formation state: p=primary, s1=secondary, s2=secondary. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; + +-- ── simulate stuck failover ────────────────────────────────────────────────── +-- +-- Kill the primary (p) and one quorum standby (s2) simultaneously. +-- We simulate this by: +-- 1. Marking p and s2 as unhealthy (health=BAD, old reporttime). +-- 2. Manually placing s1 in report_lsn/report_lsn (it already called +-- node_active and transitioned itself), while s2 is still in +-- secondary/secondary with goalState=report_lsn assigned (missing). +-- +-- This mirrors what happens in production: the monitor assigns report_lsn +-- to all standbys, s1 reports immediately, but s2 is dead and never will. + +SET pgautofailover.startup_grace_period = 1; + +-- Mark p as dead (health=BAD, stale report). +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds' + WHERE formationid = 'gdl_test' AND nodename = 'p'; + +-- Mark s2 as dead as well (health=BAD, stale report, pgIsRunning=false). +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds', + reportedpgisrunning = false + WHERE formationid = 'gdl_test' AND nodename = 's2'; + +-- Demote p: set it to draining/draining so the FSM sees no primary. +UPDATE pgautofailover.node + SET goalstate = 'draining', reportedstate = 'draining' + WHERE formationid = 'gdl_test' AND nodename = 'p'; + +-- Put s1 in report_lsn/report_lsn (it already ran node_active and confirmed). +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'report_lsn' + WHERE formationid = 'gdl_test' AND nodename = 's1'; + +-- Put s2 in secondary/secondary with goalState=report_lsn assigned but NOT +-- yet confirmed (missingNodesCount will be > 0 for s2). +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'secondary' + WHERE formationid = 'gdl_test' AND nodename = 's2'; + +-- Verify the manufactured stuck state. +SELECT nodename, goalstate, reportedstate, health + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; + +-- ── test 1: guard_data_loss = true (default) blocks ───────────────────────── +-- +-- perform_failover should find no primary, locate s1 in report_lsn, drive +-- ProceedGroupState for s1, which calls ProceedGroupStateForMSFailover, which +-- hits the guard and returns false without selecting a candidate. +-- The node states should remain unchanged (no candidate selected). + +-- Default is true; make it explicit. +SET pgautofailover.guard_data_loss TO true; + +SELECT pgautofailover.perform_failover('gdl_test', 0); + +-- States must be unchanged: no candidate was promoted. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; + +-- ── test 2: guard_data_loss = false allows progress ───────────────────────── +-- +-- With guard_data_loss disabled the failover proceeds despite s2 being missing. +-- s1 (the only node in report_lsn) should be selected as the candidate and +-- transition toward prepare_promotion / stop_replication. + +SET pgautofailover.guard_data_loss TO false; + +SELECT pgautofailover.perform_failover('gdl_test', 0); + +-- s1 should now have a goal state of prepare_promotion or stop_replication, +-- indicating the failover candidate was selected. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'gdl_test' + ORDER BY nodename; + +RESET pgautofailover.guard_data_loss; +RESET pgautofailover.startup_grace_period; diff --git a/tests/tap/schedule b/tests/tap/schedule index a58a8f966..024b4bf81 100644 --- a/tests/tap/schedule +++ b/tests/tap/schedule @@ -22,6 +22,7 @@ multi_async multi_ifdown multi_maintenance multi_alternate +guard_data_loss extension_update installcheck # Upgrade test — requires pgaf:current and pgaf:next images to be pre-built: diff --git a/tests/tap/specs/ensure.pgaf b/tests/tap/specs/ensure.pgaf index 7069b2786..7af5096c4 100644 --- a/tests/tap/specs/ensure.pgaf +++ b/tests/tap/specs/ensure.pgaf @@ -64,7 +64,7 @@ step test_004_demoted { # wait_primary the monitor has committed to node2 as the new primary; when # node1 restarts it will come back as secondary, and node2 will then # transition from wait_primary → primary. - wait until node2 state is wait_primary timeout 90s + wait until node2 state is wait_primary timeout 180s # Bring node1 back: pg_autoctl starts, connects to monitor, gets assigned # 'demoted', executes the demoted transition, then moves to secondary. # The Python test explicitly asserts the 'demoted' state, but 'demoted' is diff --git a/tests/tap/specs/guard_data_loss.pgaf b/tests/tap/specs/guard_data_loss.pgaf new file mode 100644 index 000000000..1030cbce5 --- /dev/null +++ b/tests/tap/specs/guard_data_loss.pgaf @@ -0,0 +1,116 @@ +# Test pgautofailover.guard_data_loss / pg_autoctl perform failover --allow-data-loss +# +# Scenario: 3-node formation (number_sync_standbys=1). +# The primary and one standby fail simultaneously. The surviving standby +# enters REPORT_LSN and waits — it cannot be promoted because the dead +# standby was a quorum member and may have acknowledged the last synchronous +# commit. Without intervention the cluster is stuck forever. +# +# This test verifies: +# 1. After both failures, the surviving standby (node3) stays at report_lsn +# and is NOT promoted automatically (guard_data_loss=true by default). +# 2. Running `pg_autoctl perform failover --allow-data-loss` from the monitor +# unblocks the situation: node3 advances to prepare_promotion and the +# cluster reaches a healthy primary state. +# +# Predecessor: monitor regression test src/monitor/sql/guard_data_loss.sql +# Related issues: #1113, #1060, #1059, #1014 + +cluster { + monitor + ssl off + formation { + node1 + node2 + node3 + } +} + +setup { + wait until primary, secondary timeout 120s + wait until node3 state is secondary timeout 60s + promote node1 + wait until node1 state is primary + and node2 state is secondary + and node3 state is secondary + timeout 120s +} + +teardown { + compose down +} + +# +# Step 1: kill the primary (node1) and one standby (node2) in quick succession. +# +# After node1 dies the monitor assigns report_lsn to both standbys. +# node3 will report its LSN. node2 is killed before it can report — it +# remains as a "missing" quorum member. +# +# With guard_data_loss=true (the default), ProceedGroupStateForMSFailover +# sees missingNodesCount=1 and refuses to promote anyone. node3 stays at +# report_lsn indefinitely. +# + +step test_001_kill_primary_and_one_standby { + compose kill node1 + compose kill node2 + wait until node1 assigned-state = draining timeout 120s + wait until node3 state is report_lsn timeout 120s +} + +# +# Step 2: verify node3 stays stuck — no automatic promotion. +# +# We sleep briefly to let the monitor run a few more health-check cycles. +# If guard_data_loss were incorrectly false, node3 would have advanced by now. +# + +step test_002_verify_stuck { + sleep 10s + assert node3 state is report_lsn +} + +# +# Step 3: unblock with --allow-data-loss. +# +# pg_autoctl perform failover --allow-data-loss opens a transaction on the +# monitor, sets LOCAL pgautofailover.guard_data_loss = false, and calls +# perform_failover(). The monitor selects node3 as the only available +# candidate and drives it toward prepare_promotion. +# + +step test_003_allow_data_loss_failover { + exec monitor pg_autoctl perform failover --allow-data-loss --formation default + wait until node3 state is wait_primary timeout 120s +} + +# +# Step 4: bring node2 back so node3 can become a full primary. +# + +step test_004_bringup_node2 { + compose start node2 + wait until node3 state is primary + and node2 state is secondary + timeout 180s +} + +# +# Step 5: bring node1 back as the second standby. +# + +step test_005_bringup_node1 { + compose start node1 + wait until node1 state is secondary + and node3 state is primary + and node2 state is secondary + timeout 120s +} + +sequence + test_001_kill_primary_and_one_standby + test_002_verify_stuck + test_003_allow_data_loss_failover + test_004_bringup_node2 + test_005_bringup_node1