From 9f5826bd712e22bd4b57260607d4fdf630cc49d3 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sun, 26 Jul 2026 19:10:04 +0200 Subject: [PATCH] docs: add sequence diagrams for the two remaining Unhealthy Nodes scenarios The Unhealthy Nodes section had prose but no diagram for two of its four bullets. Round these out to match the other two (Primary can't connect to Monitor, Monitor can't connect to Secondary), already illustrated in #1160. - Primary node is monitored unhealthy: new diagram with explicit node_active traffic, illustrating the actual dual-signal unhealthy determination (health check + node_active/pgIsRunning) -- plus a new clarifying paragraph on that combination, since neither signal alone is sufficient and the prose didn't previously say so. - Monitor node has failed: new diagram (adds an Application actor) showing primary/secondary roles and read-write access are entirely unaffected while the monitor is down; only automated failover isn't available. Also adds a paragraph on pg_auto_failover's single-failure design scope: losing the monitor alone isn't a problem, but a second node failing in that window is two out of three down at once, which has no automated repair. Split out of the timeline-fork-detection branch, which had bundled these in among unrelated feature commits -- this is pure documentation of existing, pre-existing monitor/keeper behavior, unrelated to that feature. --- docs/fault-tolerance.rst | 42 + docs/tikz/seq-monitor-failed.svg | 1430 ++++++++++++++ docs/tikz/seq-monitor-failed.tex | 127 ++ docs/tikz/seq-primary-unhealthy.svg | 2823 +++++++++++++++++++++++++++ docs/tikz/seq-primary-unhealthy.tex | 164 ++ 5 files changed, 4586 insertions(+) create mode 100644 docs/tikz/seq-monitor-failed.svg create mode 100644 docs/tikz/seq-monitor-failed.tex create mode 100644 docs/tikz/seq-primary-unhealthy.svg create mode 100644 docs/tikz/seq-primary-unhealthy.tex diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index 747f9a894..dc51c50c7 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -46,6 +46,23 @@ and the keeper (``pg_autoctl run``) is still active, but PostgreSQL has failed. Situations might include *File System is Full* on the WAL disk, some file system level corruption, missing files, etc. +A node is only ever considered unhealthy from the combination of these two +independent signals, not from either alone: + +- the monitor's own direct health checks (the periodic libpq connection + above), and +- what the node's own keeper last reported through the node-active + protocol -- either that it has stopped reporting at all, or that it is + still reporting but says its local PostgreSQL is not running. + +A node whose keeper is still actively reporting ``pgIsRunning = true`` is +never marked unhealthy on the strength of a failed direct health check +alone -- that combination just means the monitor's own connection is +having trouble, while the node itself is telling a different story, so the +monitor keeps trusting the node's own report. Conversely, a keeper that +reports ``pgIsRunning = false`` marks its node unhealthy immediately, +whether or not the direct health check is currently failing too. + Here's what happens to your PostgreSQL service in case of any single-node failure is observed: @@ -68,6 +85,12 @@ failure is observed: reports success anyway, and the failover sequence continues from the monitor. + .. figure:: ./tikz/seq-primary-unhealthy.svg + :alt: Sequence diagram of the monitor failing over to the secondary after the primary's keeper reports Postgres down, and the resulting draining/catch-up/STONITH/promote sequence + + Unhealthy from the keeper's own report (Postgres down, keeper still + reporting), then draining, catch-up, STONITH, promote + - Secondary node is monitored unhealthy When the secondary node is unhealthy, the monitor assigns to it the @@ -87,6 +110,25 @@ failure is observed: from the nodes. Also, health checks are not performed. It means that no automated failover may happen, even if needed. + pg_auto_failover's design target is to handle any **one** node failure. + Losing the monitor, by itself, isn't a problem: the primary and + secondary keep working, unattended, for as long as it takes to bring + the monitor back. But that node is now down, and while it's down the + formation has no automated protection left. If a second node then also + fails before the monitor is restored — the primary, say, while the + monitor is still out — that's two node failures out of three at the + same time, and pg_auto_failover has no automated repair for that: there + is no third, healthy node left for the monitor to orchestrate a + failover to even once it does come back. Restoring the monitor quickly + is what keeps a single further failure from turning into exactly that + situation. + + .. figure:: ./tikz/seq-monitor-failed.svg + :alt: Sequence diagram showing that primary/secondary roles and application read-write access are unaffected while the monitor is down, only automated failover is unavailable + + No monitor, no automated failover -- but roles and application + traffic are entirely unaffected + .. _network_partitions: Network Partitions diff --git a/docs/tikz/seq-monitor-failed.svg b/docs/tikz/seq-monitor-failed.svg new file mode 100644 index 000000000..4637bf9cd --- /dev/null +++ b/docs/tikz/seq-monitor-failed.svg @@ -0,0 +1,1430 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-monitor-failed.tex b/docs/tikz/seq-monitor-failed.tex new file mode 100644 index 000000000..190af8f46 --- /dev/null +++ b/docs/tikz/seq-monitor-failed.tex @@ -0,0 +1,127 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% A=0 P=8 S=16 M=25 +\def\ax{0} +\def\px{8} +\def\sx{16} +\def\mx{25} +\def\ybot{20.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] +\tikzstyle{appbox}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[appbox, fill=gray!25, text=black] (Ah) at (\ax,0) {Application}; +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\ax,0.35) -- (\ax,\ybot); +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); + +%% ── steady state ───────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\ax,1.6) -- (\px,1.6) + node[lbl,midway,above] {writes}; +\draw[msg] (\px,2.4) -- (\sx,2.4) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,3.6) -- (\mx,3.6); +\node[lbl, above] at (\px+4.5,3.6) {node\_active(reported=primary)}; +\draw[rsp] (\mx,4.2) -- (\px,4.2); +\node[lbl, above] at (\mx-2.2,4.2) {goal=primary}; + +\draw[msg] (\sx,5.4) -- (\mx,5.4); +\node[lbl, above] at (\sx+3.3,5.4) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,6.0) -- (\sx,6.0); +\node[lbl, above] at (\mx-2.2,6.0) {goal=secondary}; + +%% ── monitor fails ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,7.4) {monitor fails (crash, or unreachable)}; + +\draw[ll] (\mx,0.35) -- (\mx,7.1); +\node[red!70!black, font=\Large] at (\mx,7.7) {$\times$}; +\draw[lldead] (\mx,8.0) -- (\mx,\ybot); + +\node[note, fill=red!12, draw=red!50, text width=15cm, font=\small] + at (12.5, 9.2) + {no health checks run; no node can poll for or be assigned a goal state. + Neither node changes role -- there is nothing here that could trigger a + promotion even if the primary were also to fail right now}; + +%% ── ordinary operation continues, unaffected ──────────────────────────── +\node[seclbl] at (-3.4,10.6) {ordinary operation continues, fully unaffected}; + +\draw[msg] (\ax,11.5) -- (\px,11.5) + node[lbl,midway,above] {writes}; +\draw[msg] (\px,12.3) -- (\sx,12.3) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,13.6) -- (\mx,13.6); +\node[red!70!black, font=\Large] at (\mx-0.6,13.6) {$\times$}; +\node[lbl, above, text=red!70!black] at (\px+4.5,13.6) + {node\_active(reported=primary) attempted -- no response}; + +\draw[msg] (\sx,14.8) -- (\mx,14.8); +\node[red!70!black, font=\Large] at (\mx-0.6,14.8) {$\times$}; +\node[lbl, above, text=red!70!black] at (\sx+3.3,14.8) + {node\_active(reported=secondary) attempted -- no response}; + +\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small] + at (12.5, 16.1) + {this is exactly ordinary streaming replication -- it was never routed + through the monitor to begin with, so it neither needs nor notices it}; + +\draw[msg] (\ax,17.4) -- (\px,17.4) + node[lbl,midway,above] {writes}; +\draw[msg] (\px,18.2) -- (\sx,18.2) + node[lbl,midway,above] {WAL stream}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=13cm] + at (12.5, 19.6) + {$\checkmark$~~roles unchanged, application read-write access uninterrupted + -- only automated failover is unavailable until the monitor returns}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[appbox, fill=gray!25, text=black] at (\ax, \ybot+0.35) {Application}; +\node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary}; +\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary}; +\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black] + at (\mx, \ybot+0.35) {\small Monitor (down)}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-primary-unhealthy.svg b/docs/tikz/seq-primary-unhealthy.svg new file mode 100644 index 000000000..b463e7aca --- /dev/null +++ b/docs/tikz/seq-primary-unhealthy.svg @@ -0,0 +1,2823 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-primary-unhealthy.tex b/docs/tikz/seq-primary-unhealthy.tex new file mode 100644 index 000000000..b7e435ba0 --- /dev/null +++ b/docs/tikz/seq-primary-unhealthy.tex @@ -0,0 +1,164 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{34.4} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, red!40, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state: two independent signals ────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state: two independent signals per node}; + +\draw[msg] (\mx,1.6) -- (\px,1.6); +\node[lbl, above] at (\mx-2.6,1.6) {health check (libpq, like \texttt{pg\_isready})}; +\draw[rsp] (\px,2.2) -- (\mx,2.2) + node[lbl,midway,above] {ok}; + +\draw[msg] (\px,3.4) -- (\mx,3.4); +\node[lbl, above] at (\px+4.6,3.4) {node\_active(reported=primary, pgIsRunning=true)}; +\draw[rsp] (\mx,4.0) -- (\px,4.0); +\node[lbl, above] at (\mx-2.2,4.0) {goal=primary}; + +\draw[msg] (\sx,5.2) -- (\mx,5.2); +\node[lbl, above] at (\sx+3.8,5.2) {node\_active(reported=secondary, pgIsRunning=true)}; +\draw[rsp] (\mx,5.8) -- (\sx,5.8); +\node[lbl, above] at (\mx-2.2,5.8) {goal=secondary}; + +\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small] + at (9.5, 7.1) + {a node is only ever unhealthy from the combination of what its own + keeper reports \emph{and} what the monitor's direct health check finds + -- never from either alone}; + +%% ── primary's Postgres fails locally ──────────────────────────────────── +\node[seclbl] at (-3.4,8.7) + {primary's Postgres fails locally (keeper stays up, e.g. disk full)}; + +\draw[broken] (\mx,9.6) -- (\px,9.6); +\node[red!70!black, font=\Large] at (4.3,9.6) {$\times$}; +\node[lbl, above, text=red!70!black] at (\mx-2.2,9.6) + {health check: connection refused}; + +\draw[msg] (\px,10.8) -- (\mx,10.8); +\node[lbl, above] at (\px+4.8,10.8) + {node\_active(reported=primary, pgIsRunning=\textbf{false})}; +\draw[rsp] (\mx,11.4) -- (\px,11.4); +\node[lbl, above] at (\mx-2.2,11.4) {goal=primary (unchanged, still deciding)}; + +\node[note, fill=red!12, draw=red!50, text width=15cm, font=\small] + at (9.5, 12.9) + {the keeper's own report already says Postgres is down -- that alone is + enough to mark the primary unhealthy, whether or not the direct health + check is also currently failing. (The other path to unhealthy -- the + keeper goes fully silent \emph{and} the last direct health check was + bad -- is the Network Partitions scenario, see \S~Monitor can't + connect to Primary)}; + +\draw[msg] (\sx,16.0) -- (\mx,16.0); +\node[lbl, above] at (\sx+3.8,16.0) {node\_active(reported=secondary, pgIsRunning=true)}; +\draw[rsp] (\mx,16.6) -- (\sx,16.6); +\node[lbl, above] at (\mx-2.2,16.6) {goal=secondary}; + +\node[note, fill=gray!12, draw=gray!55, text width=13cm, font=\small] + at (9.5, 17.8) + {only proceeds because the secondary is itself healthy -- otherwise the + monitor has no safe candidate and stays put}; + +%% ── failover begins ────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,19.2) {failover begins, on the primary's own next report}; + +\draw[msg] (\px,20.1) -- (\mx,20.1); +\node[lbl, above] at (\px+4.8,20.1) + {node\_active(reported=primary, pgIsRunning=false)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,20.7) -- (\px,20.7); +\node[lbl, above, text=pbox] at (\mx-2.2,20.7) {goal=draining}; + +\draw[msg] (\sx,21.9) -- (\mx,21.9); +\node[lbl, above] at (\sx+3.8,21.9) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,22.5) -- (\sx,22.5); +\node[lbl, above, text=pbox] at (\mx-2.6,22.5) {goal=prepare\_promotion}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=13cm, font=\small] + at (9.5, 24.1) + {secondary catches up on any last WAL still available from the primary, + then reports success -- or, after + \texttt{PREPARE\_PROMOTION\_CATCHUP\_TIMEOUT}, reports success anyway and + the failover continues regardless}; + +\draw[msg] (\sx,26.2) -- (\mx,26.2); +\node[lbl, above] at (\sx+3.5,26.2) {node\_active(reported=prepare\_promotion)}; + +%% ── STONITH ────────────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,27.6) {monitor stops the primary (STONITH)}; + +\draw[msg] (\px,28.5) -- (\mx,28.5); +\node[lbl, above] at (\px+4.6,28.5) {node\_active(reported=draining)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,29.1) -- (\px,29.1); +\node[lbl, above, text=pbox] at (\mx-2.2,29.1) {goal=demoted}; + +\draw[lldead] (\px,29.4) -- (\px,\ybot); + +\node[note, fill=gray!12, draw=gray!55, text width=13cm, font=\small] + at (9.5, 30.5) + {keeper stops (and prevents restarting) Postgres on the old primary -- + guards against a split-brain once the secondary is about to accept + writes}; + +%% ── promotion complete ─────────────────────────────────────────────────── +\draw[msg, color=pbox, line width=1pt] (\mx,31.7) -- (\sx,31.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,31.7) + {goal=wait\_primary}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (14, 33.0) {$\checkmark$~~Secondary is the new primary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black] + at (\px, \ybot+0.35) {\small Primary (demoted)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document}