diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst
index 747f9a894..dc51c50c7 100644
--- a/docs/fault-tolerance.rst
+++ b/docs/fault-tolerance.rst
@@ -46,6 +46,23 @@ and the keeper (``pg_autoctl run``) is still active, but PostgreSQL has failed.
Situations might include *File System is Full* on the WAL disk, some file
system level corruption, missing files, etc.
+A node is only ever considered unhealthy from the combination of these two
+independent signals, not from either alone:
+
+- the monitor's own direct health checks (the periodic libpq connection
+ above), and
+- what the node's own keeper last reported through the node-active
+ protocol -- either that it has stopped reporting at all, or that it is
+ still reporting but says its local PostgreSQL is not running.
+
+A node whose keeper is still actively reporting ``pgIsRunning = true`` is
+never marked unhealthy on the strength of a failed direct health check
+alone -- that combination just means the monitor's own connection is
+having trouble, while the node itself is telling a different story, so the
+monitor keeps trusting the node's own report. Conversely, a keeper that
+reports ``pgIsRunning = false`` marks its node unhealthy immediately,
+whether or not the direct health check is currently failing too.
+
Here's what happens to your PostgreSQL service in case of any single-node
failure is observed:
@@ -68,6 +85,12 @@ failure is observed:
reports success anyway, and the failover sequence continues from the
monitor.
+ .. figure:: ./tikz/seq-primary-unhealthy.svg
+ :alt: Sequence diagram of the monitor failing over to the secondary after the primary's keeper reports Postgres down, and the resulting draining/catch-up/STONITH/promote sequence
+
+ Unhealthy from the keeper's own report (Postgres down, keeper still
+ reporting), then draining, catch-up, STONITH, promote
+
- Secondary node is monitored unhealthy
When the secondary node is unhealthy, the monitor assigns to it the
@@ -87,6 +110,25 @@ failure is observed:
from the nodes. Also, health checks are not performed. It means that no
automated failover may happen, even if needed.
+ pg_auto_failover's design target is to handle any **one** node failure.
+ Losing the monitor, by itself, isn't a problem: the primary and
+ secondary keep working, unattended, for as long as it takes to bring
+ the monitor back. But that node is now down, and while it's down the
+ formation has no automated protection left. If a second node then also
+ fails before the monitor is restored — the primary, say, while the
+ monitor is still out — that's two node failures out of three at the
+ same time, and pg_auto_failover has no automated repair for that: there
+ is no third, healthy node left for the monitor to orchestrate a
+ failover to even once it does come back. Restoring the monitor quickly
+ is what keeps a single further failure from turning into exactly that
+ situation.
+
+ .. figure:: ./tikz/seq-monitor-failed.svg
+ :alt: Sequence diagram showing that primary/secondary roles and application read-write access are unaffected while the monitor is down, only automated failover is unavailable
+
+ No monitor, no automated failover -- but roles and application
+ traffic are entirely unaffected
+
.. _network_partitions:
Network Partitions
diff --git a/docs/tikz/seq-monitor-failed.svg b/docs/tikz/seq-monitor-failed.svg
new file mode 100644
index 000000000..4637bf9cd
--- /dev/null
+++ b/docs/tikz/seq-monitor-failed.svg
@@ -0,0 +1,1430 @@
+
+
diff --git a/docs/tikz/seq-monitor-failed.tex b/docs/tikz/seq-monitor-failed.tex
new file mode 100644
index 000000000..190af8f46
--- /dev/null
+++ b/docs/tikz/seq-monitor-failed.tex
@@ -0,0 +1,127 @@
+\RequirePackage{luatex85}
+\documentclass[border=10pt,12pt]{standalone}
+
+\usepackage{cfr-lm}
+\usepackage{amssymb}
+\usepackage{pgf}
+\usepackage{tikz}
+\usetikzlibrary{arrows.meta,calc,positioning}
+
+\begin{document}
+
+\sffamily\bfseries
+
+%% Actor x-positions (cm)
+%% A=0 P=8 S=16 M=25
+\def\ax{0}
+\def\px{8}
+\def\sx{16}
+\def\mx{25}
+\def\ybot{20.6}
+
+\begin{tikzpicture}[
+ >=Stealth,
+ yscale=-1,
+]
+
+\input{common.tex}
+
+\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm,
+ inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries]
+\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt]
+\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt]
+\tikzstyle{msg}=[->, line width=0.65pt]
+\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt]
+\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt]
+\tikzstyle{lbl}=[font=\small, inner sep=2pt]
+\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt,
+ font=\small, align=center]
+\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west]
+\tikzstyle{appbox}=[rectangle, minimum width=3.6cm, minimum height=0.7cm,
+ inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries]
+
+%% ── actor headers ───────────────────────────────────────────────────────
+\node[appbox, fill=gray!25, text=black] (Ah) at (\ax,0) {Application};
+\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary};
+\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary};
+\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor};
+
+%% lifelines
+\draw[ll] (\ax,0.35) -- (\ax,\ybot);
+\draw[ll] (\px,0.35) -- (\px,\ybot);
+\draw[ll] (\sx,0.35) -- (\sx,\ybot);
+
+%% ── steady state ─────────────────────────────────────────────────────────
+\node[seclbl] at (-3.4,1.3) {steady state};
+
+\draw[msg] (\ax,1.6) -- (\px,1.6)
+ node[lbl,midway,above] {writes};
+\draw[msg] (\px,2.4) -- (\sx,2.4)
+ node[lbl,midway,above] {WAL stream};
+
+\draw[msg] (\px,3.6) -- (\mx,3.6);
+\node[lbl, above] at (\px+4.5,3.6) {node\_active(reported=primary)};
+\draw[rsp] (\mx,4.2) -- (\px,4.2);
+\node[lbl, above] at (\mx-2.2,4.2) {goal=primary};
+
+\draw[msg] (\sx,5.4) -- (\mx,5.4);
+\node[lbl, above] at (\sx+3.3,5.4) {node\_active(reported=secondary)};
+\draw[rsp] (\mx,6.0) -- (\sx,6.0);
+\node[lbl, above] at (\mx-2.2,6.0) {goal=secondary};
+
+%% ── monitor fails ────────────────────────────────────────────────────────
+\node[seclbl] at (-3.4,7.4) {monitor fails (crash, or unreachable)};
+
+\draw[ll] (\mx,0.35) -- (\mx,7.1);
+\node[red!70!black, font=\Large] at (\mx,7.7) {$\times$};
+\draw[lldead] (\mx,8.0) -- (\mx,\ybot);
+
+\node[note, fill=red!12, draw=red!50, text width=15cm, font=\small]
+ at (12.5, 9.2)
+ {no health checks run; no node can poll for or be assigned a goal state.
+ Neither node changes role -- there is nothing here that could trigger a
+ promotion even if the primary were also to fail right now};
+
+%% ── ordinary operation continues, unaffected ────────────────────────────
+\node[seclbl] at (-3.4,10.6) {ordinary operation continues, fully unaffected};
+
+\draw[msg] (\ax,11.5) -- (\px,11.5)
+ node[lbl,midway,above] {writes};
+\draw[msg] (\px,12.3) -- (\sx,12.3)
+ node[lbl,midway,above] {WAL stream};
+
+\draw[msg] (\px,13.6) -- (\mx,13.6);
+\node[red!70!black, font=\Large] at (\mx-0.6,13.6) {$\times$};
+\node[lbl, above, text=red!70!black] at (\px+4.5,13.6)
+ {node\_active(reported=primary) attempted -- no response};
+
+\draw[msg] (\sx,14.8) -- (\mx,14.8);
+\node[red!70!black, font=\Large] at (\mx-0.6,14.8) {$\times$};
+\node[lbl, above, text=red!70!black] at (\sx+3.3,14.8)
+ {node\_active(reported=secondary) attempted -- no response};
+
+\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small]
+ at (12.5, 16.1)
+ {this is exactly ordinary streaming replication -- it was never routed
+ through the monitor to begin with, so it neither needs nor notices it};
+
+\draw[msg] (\ax,17.4) -- (\px,17.4)
+ node[lbl,midway,above] {writes};
+\draw[msg] (\px,18.2) -- (\sx,18.2)
+ node[lbl,midway,above] {WAL stream};
+
+\node[note, fill=green!15, draw=green!50!black,
+ font=\small\bfseries, minimum width=13cm]
+ at (12.5, 19.6)
+ {$\checkmark$~~roles unchanged, application read-write access uninterrupted
+ -- only automated failover is unavailable until the monitor returns};
+
+%% ── actor footers ───────────────────────────────────────────────────────
+\node[appbox, fill=gray!25, text=black] at (\ax, \ybot+0.35) {Application};
+\node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary};
+\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary};
+\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black]
+ at (\mx, \ybot+0.35) {\small Monitor (down)};
+
+\end{tikzpicture}
+\end{document}
diff --git a/docs/tikz/seq-primary-unhealthy.svg b/docs/tikz/seq-primary-unhealthy.svg
new file mode 100644
index 000000000..b463e7aca
--- /dev/null
+++ b/docs/tikz/seq-primary-unhealthy.svg
@@ -0,0 +1,2823 @@
+
+
diff --git a/docs/tikz/seq-primary-unhealthy.tex b/docs/tikz/seq-primary-unhealthy.tex
new file mode 100644
index 000000000..b7e435ba0
--- /dev/null
+++ b/docs/tikz/seq-primary-unhealthy.tex
@@ -0,0 +1,164 @@
+\RequirePackage{luatex85}
+\documentclass[border=10pt,12pt]{standalone}
+
+\usepackage{cfr-lm}
+\usepackage{amssymb}
+\usepackage{pgf}
+\usepackage{tikz}
+\usetikzlibrary{arrows.meta,calc,positioning}
+
+\begin{document}
+
+\sffamily\bfseries
+
+%% Actor x-positions (cm)
+%% P=0 S=9 M=19
+\def\px{0}
+\def\sx{9}
+\def\mx{19}
+\def\ybot{34.4}
+
+\begin{tikzpicture}[
+ >=Stealth,
+ yscale=-1,
+]
+
+\input{common.tex}
+
+\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm,
+ inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries]
+\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt]
+\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt]
+\tikzstyle{msg}=[->, line width=0.65pt]
+\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt]
+\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt]
+\tikzstyle{lbl}=[font=\small, inner sep=2pt]
+\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt,
+ font=\small, align=center]
+\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west]
+
+%% ── actor headers ───────────────────────────────────────────────────────
+\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary};
+\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary};
+\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor};
+
+%% lifelines
+\draw[ll] (\px,0.35) -- (\px,\ybot);
+\draw[ll] (\sx,0.35) -- (\sx,\ybot);
+\draw[ll] (\mx,0.35) -- (\mx,\ybot);
+
+%% ── steady state: two independent signals ──────────────────────────────
+\node[seclbl] at (-3.4,1.3) {steady state: two independent signals per node};
+
+\draw[msg] (\mx,1.6) -- (\px,1.6);
+\node[lbl, above] at (\mx-2.6,1.6) {health check (libpq, like \texttt{pg\_isready})};
+\draw[rsp] (\px,2.2) -- (\mx,2.2)
+ node[lbl,midway,above] {ok};
+
+\draw[msg] (\px,3.4) -- (\mx,3.4);
+\node[lbl, above] at (\px+4.6,3.4) {node\_active(reported=primary, pgIsRunning=true)};
+\draw[rsp] (\mx,4.0) -- (\px,4.0);
+\node[lbl, above] at (\mx-2.2,4.0) {goal=primary};
+
+\draw[msg] (\sx,5.2) -- (\mx,5.2);
+\node[lbl, above] at (\sx+3.8,5.2) {node\_active(reported=secondary, pgIsRunning=true)};
+\draw[rsp] (\mx,5.8) -- (\sx,5.8);
+\node[lbl, above] at (\mx-2.2,5.8) {goal=secondary};
+
+\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small]
+ at (9.5, 7.1)
+ {a node is only ever unhealthy from the combination of what its own
+ keeper reports \emph{and} what the monitor's direct health check finds
+ -- never from either alone};
+
+%% ── primary's Postgres fails locally ────────────────────────────────────
+\node[seclbl] at (-3.4,8.7)
+ {primary's Postgres fails locally (keeper stays up, e.g. disk full)};
+
+\draw[broken] (\mx,9.6) -- (\px,9.6);
+\node[red!70!black, font=\Large] at (4.3,9.6) {$\times$};
+\node[lbl, above, text=red!70!black] at (\mx-2.2,9.6)
+ {health check: connection refused};
+
+\draw[msg] (\px,10.8) -- (\mx,10.8);
+\node[lbl, above] at (\px+4.8,10.8)
+ {node\_active(reported=primary, pgIsRunning=\textbf{false})};
+\draw[rsp] (\mx,11.4) -- (\px,11.4);
+\node[lbl, above] at (\mx-2.2,11.4) {goal=primary (unchanged, still deciding)};
+
+\node[note, fill=red!12, draw=red!50, text width=15cm, font=\small]
+ at (9.5, 12.9)
+ {the keeper's own report already says Postgres is down -- that alone is
+ enough to mark the primary unhealthy, whether or not the direct health
+ check is also currently failing. (The other path to unhealthy -- the
+ keeper goes fully silent \emph{and} the last direct health check was
+ bad -- is the Network Partitions scenario, see \S~Monitor can't
+ connect to Primary)};
+
+\draw[msg] (\sx,16.0) -- (\mx,16.0);
+\node[lbl, above] at (\sx+3.8,16.0) {node\_active(reported=secondary, pgIsRunning=true)};
+\draw[rsp] (\mx,16.6) -- (\sx,16.6);
+\node[lbl, above] at (\mx-2.2,16.6) {goal=secondary};
+
+\node[note, fill=gray!12, draw=gray!55, text width=13cm, font=\small]
+ at (9.5, 17.8)
+ {only proceeds because the secondary is itself healthy -- otherwise the
+ monitor has no safe candidate and stays put};
+
+%% ── failover begins ──────────────────────────────────────────────────────
+\node[seclbl] at (-3.4,19.2) {failover begins, on the primary's own next report};
+
+\draw[msg] (\px,20.1) -- (\mx,20.1);
+\node[lbl, above] at (\px+4.8,20.1)
+ {node\_active(reported=primary, pgIsRunning=false)};
+\draw[rsp, color=pbox, line width=0.8pt] (\mx,20.7) -- (\px,20.7);
+\node[lbl, above, text=pbox] at (\mx-2.2,20.7) {goal=draining};
+
+\draw[msg] (\sx,21.9) -- (\mx,21.9);
+\node[lbl, above] at (\sx+3.8,21.9) {node\_active(reported=secondary)};
+\draw[rsp, color=pbox, line width=0.8pt] (\mx,22.5) -- (\sx,22.5);
+\node[lbl, above, text=pbox] at (\mx-2.6,22.5) {goal=prepare\_promotion};
+
+\node[note, fill=sbox!25, draw=sbox!70!black, text width=13cm, font=\small]
+ at (9.5, 24.1)
+ {secondary catches up on any last WAL still available from the primary,
+ then reports success -- or, after
+ \texttt{PREPARE\_PROMOTION\_CATCHUP\_TIMEOUT}, reports success anyway and
+ the failover continues regardless};
+
+\draw[msg] (\sx,26.2) -- (\mx,26.2);
+\node[lbl, above] at (\sx+3.5,26.2) {node\_active(reported=prepare\_promotion)};
+
+%% ── STONITH ──────────────────────────────────────────────────────────────
+\node[seclbl] at (-3.4,27.6) {monitor stops the primary (STONITH)};
+
+\draw[msg] (\px,28.5) -- (\mx,28.5);
+\node[lbl, above] at (\px+4.6,28.5) {node\_active(reported=draining)};
+\draw[rsp, color=pbox, line width=0.8pt] (\mx,29.1) -- (\px,29.1);
+\node[lbl, above, text=pbox] at (\mx-2.2,29.1) {goal=demoted};
+
+\draw[lldead] (\px,29.4) -- (\px,\ybot);
+
+\node[note, fill=gray!12, draw=gray!55, text width=13cm, font=\small]
+ at (9.5, 30.5)
+ {keeper stops (and prevents restarting) Postgres on the old primary --
+ guards against a split-brain once the secondary is about to accept
+ writes};
+
+%% ── promotion complete ───────────────────────────────────────────────────
+\draw[msg, color=pbox, line width=1pt] (\mx,31.7) -- (\sx,31.7);
+\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,31.7)
+ {goal=wait\_primary};
+
+\node[note, fill=green!15, draw=green!50!black,
+ font=\small\bfseries, minimum width=8cm]
+ at (14, 33.0) {$\checkmark$~~Secondary is the new primary};
+
+%% ── actor footers ───────────────────────────────────────────────────────
+\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black]
+ at (\px, \ybot+0.35) {\small Primary (demoted)};
+\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary};
+\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor};
+
+\end{tikzpicture}
+\end{document}