From e2d68561613674676ec265cdfb000654e2ad3347 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 16:57:54 +0200 Subject: [PATCH 1/8] Add 5 tikz sequence diagrams for the monitoring, FSM, and fault-tolerance docs architecture.rst: the node_active() polling and health-check protocols between a node's keeper and the monitor, and when a node counts as unhealthy. failover-state-machine.rst: the "birth of a cluster" walkthrough (two freshly registered nodes converging to a primary/secondary pair) that was previously prose-only. fault-tolerance.rst (previously had no diagrams at all): - primary self-fencing to demote_timeout after losing both the monitor and the secondary (network_partition_timeout) - asymmetric partition: the monitor's 90s safety wait before promoting after assigning the secondary stop_replication - falling back to asynchronous replication when the secondary goes unhealthy, and resynchronizing back to synchronous The two existing sequence diagrams (seq-normal-failover, seq-stuck-failover) stay on architecture-multi-standby.rst, where they illustrate number_sync_standbys and its failure-mode consequences. --- docs/architecture.rst | 5 + docs/failover-state-machine.rst | 5 + docs/fault-tolerance.rst | 14 + docs/tikz/seq-asymmetric-partition.svg | 1012 ++++++++++++++++++++++ docs/tikz/seq-asymmetric-partition.tex | 106 +++ docs/tikz/seq-birth-of-a-cluster.svg | 857 +++++++++++++++++++ docs/tikz/seq-birth-of-a-cluster.tex | 111 +++ docs/tikz/seq-monitoring-protocol.svg | 703 ++++++++++++++++ docs/tikz/seq-monitoring-protocol.tex | 84 ++ docs/tikz/seq-primary-self-fence.svg | 1063 ++++++++++++++++++++++++ docs/tikz/seq-primary-self-fence.tex | 97 +++ docs/tikz/seq-secondary-unhealthy.svg | 942 +++++++++++++++++++++ docs/tikz/seq-secondary-unhealthy.tex | 107 +++ 13 files changed, 5106 insertions(+) create mode 100644 docs/tikz/seq-asymmetric-partition.svg create mode 100644 docs/tikz/seq-asymmetric-partition.tex create mode 100644 docs/tikz/seq-birth-of-a-cluster.svg create mode 100644 docs/tikz/seq-birth-of-a-cluster.tex create mode 100644 docs/tikz/seq-monitoring-protocol.svg create mode 100644 docs/tikz/seq-monitoring-protocol.tex create mode 100644 docs/tikz/seq-primary-self-fence.svg create mode 100644 docs/tikz/seq-primary-self-fence.tex create mode 100644 docs/tikz/seq-secondary-unhealthy.svg create mode 100644 docs/tikz/seq-secondary-unhealthy.tex diff --git a/docs/architecture.rst b/docs/architecture.rst index 5651cc421..6f9c1e411 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -208,6 +208,11 @@ occurred. Monitoring protocol ------------------- +.. figure:: ./tikz/seq-monitoring-protocol.svg + :alt: Sequence diagram of the node_active polling and health check protocols + + The two protocols between a node's keeper and the monitor + The monitor interacts with the data nodes in 2 ways: - Data nodes periodically connect and run `SELECT diff --git a/docs/failover-state-machine.rst b/docs/failover-state-machine.rst index 4d25dac99..acd2f3941 100644 --- a/docs/failover-state-machine.rst +++ b/docs/failover-state-machine.rst @@ -19,6 +19,11 @@ Example of state transitions in a new cluster A good way to get acquainted with the states is by examining the transitions of a cluster from birth to high availability. +.. figure:: ./tikz/seq-birth-of-a-cluster.svg + :alt: Sequence diagram of two nodes and the monitor from registration to primary/secondary + + From two freshly registered nodes to a converged primary/secondary pair + After starting a monitor and running keeper init for the first data node ("node A"), the monitor registers the state of that node as "init" with a goal state of "single." The init state means the monitor knows nothing diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index c960aa023..6a7344ec9 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -114,6 +114,11 @@ PostgreSQL service: The network\_partition\_timeout can be setup in the keeper's configuration and defaults to 20s. + .. figure:: ./tikz/seq-primary-self-fence.svg + :alt: Sequence diagram of a primary self-fencing to demote_timeout after losing contact with both the monitor and the secondary + + The primary self-fences rather than risk a split brain + - Monitor can't connect to Primary Once all the retries have been done and the timeouts are elapsed, then @@ -140,6 +145,11 @@ PostgreSQL service: assigned the SECONDARY state, and the replication will be switched back to synchronous. + .. figure:: ./tikz/seq-secondary-unhealthy.svg + :alt: Sequence diagram of the fallback to asynchronous replication and back + + Falling back to asynchronous replication and resynchronizing + Failure handling and network partition detection ------------------------------------------------ @@ -170,6 +180,10 @@ expected to shut down after at least 30 and at most 60 seconds. To factor in worst-case scenarios, the monitor waits for 90 seconds before promoting the secondary to become the new primary. +.. figure:: ./tikz/seq-asymmetric-partition.svg + :alt: Sequence diagram of an asymmetric partition where the primary can still reach the secondary but not the monitor + + Asymmetric partition: the monitor's 90s safety wait before promoting See also -------- diff --git a/docs/tikz/seq-asymmetric-partition.svg b/docs/tikz/seq-asymmetric-partition.svg new file mode 100644 index 000000000..7fc947885 --- /dev/null +++ b/docs/tikz/seq-asymmetric-partition.svg @@ -0,0 +1,1012 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-asymmetric-partition.tex b/docs/tikz/seq-asymmetric-partition.tex new file mode 100644 index 000000000..56a99f1c0 --- /dev/null +++ b/docs/tikz/seq-asymmetric-partition.tex @@ -0,0 +1,106 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{14.4} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── asymmetric partition ──────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {asymmetric partition}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream (still fine)}; + +\node[note, fill=red!12, draw=red!50, text width=8.5cm] + at (\px, 2.5) + {\color{red!80!black} Primary $\leftrightarrow$ Monitor unreachable\\ + Monitor $\leftrightarrow$ Secondary still fine}; + +%% ── monitor initiates failover ───────────────────────────────────────── +\node[seclbl] at (-3.4,4.2) {monitor initiates failover}; + +\draw[msg, color=pbox, line width=0.8pt] (\mx,4.5) -- (\sx,4.5); +\node[lbl, above, text=pbox] at (\mx-2.4, 4.5) {goal: stop\_replication}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=8cm] + at (\sx, 5.4) {\small disconnects from Primary}; + +%% ── primary loses its last connection ────────────────────────────────── +\node[seclbl] at (-3.4,7.0) {primary loses its last connection}; + +\node[note, fill=orange!15, draw=orange!70!black, text width=12.5cm] + at (4.5, 7.9) + {\small with the secondary gone too, the primary self-fences the same + way as a full partition (previous diagram): expected to stop within + 30--60s}; + +\draw[lldead] (\px,8.7) -- (\px,\ybot); + +%% ── 90s safety wait ───────────────────────────────────────────────────── +\node[seclbl] at (-3.4,10.0) {90s safety wait}; + +\node[note, fill=blue!8, draw=blue!40, text width=13.5cm] + at (\mx-1, 10.9) + {\small worst case is 60s, so the monitor waits a full 90s before + promoting -- a margin against edge cases}; + +\draw[gray!50, line width=0.5pt, ->] + (\mx+2.6, 9.6) .. controls (\mx+4.3,9.9) and (\mx+4.3,11.9) .. (\mx+2.6,12.2) + node[right, font=\small\itshape, text=gray!70] {waiting\ldots}; + +%% ── promotion ──────────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,13.0) {promotion}; + +\draw[msg, color=pbox, line width=1pt] (\mx,13.3) -- (\sx,13.3); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.6,13.3) + {goal: wait\_primary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=orange!25, draw=orange!70!black, text=orange!30!black] + at (\px, \ybot+0.35) {\small Primary (self-fenced)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-birth-of-a-cluster.svg b/docs/tikz/seq-birth-of-a-cluster.svg new file mode 100644 index 000000000..eb8db137f --- /dev/null +++ b/docs/tikz/seq-birth-of-a-cluster.svg @@ -0,0 +1,857 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-birth-of-a-cluster.tex b/docs/tikz/seq-birth-of-a-cluster.tex new file mode 100644 index 000000000..cc812ad30 --- /dev/null +++ b/docs/tikz/seq-birth-of-a-cluster.tex @@ -0,0 +1,111 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% A=0 B=9 M=19 +\def\ax{0} +\def\bx{9} +\def\mx{19} +\def\ybot{16.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{actorn}=[actor, fill=gray!15, text=stxt, draw=gray!50] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actorn] (Ah) at (\ax,0) {Node A}; +\node[actorn] (Bh) at (\bx,0) {Node B}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\ax,0.35) -- (\ax,\ybot); +\draw[ll] (\bx,0.35) -- (\bx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── node A registers ─────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {node A registers}; + +\draw[msg] (\ax,1.6) -- (\mx,1.6) + node[lbl,midway,above] {keeper init}; + +\draw[rsp] (\mx,2.5) -- (\ax,2.5); +\node[lbl, above] at (\mx-3.0, 2.5) {init $\rightarrow$ single (goal: single)}; + +%% ── node B joins ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,3.6) {node B joins}; + +\draw[msg] (\bx,3.9) -- (\mx,3.9) + node[lbl,midway,above] {keeper init}; + +\draw[rsp] (\mx,4.8) -- (\ax,4.8) + node[lbl,midway,above] {goal: wait\_primary}; +\draw[rsp] (\mx,5.6) -- (\bx,5.6); +\node[lbl, above] at (\mx-2.6, 5.6) {init $\rightarrow$ wait\_standby}; + +%% ── A prepares, B catches up ─────────────────────────────────────────── +\node[seclbl] at (-3.4,6.7) {A prepares, B catches up}; + +\node[note, fill=pbox!12, draw=pbox!60, text width=6.2cm] + at (\ax, 7.6) {\small allows B in \texttt{pg\_hba.conf}}; + +\draw[msg] (\ax,8.4) -- (\mx,8.4) + node[lbl,midway,above] {report: wait\_primary}; + +\draw[rsp] (\mx,9.0) -- (\bx,9.0) + node[lbl,midway,above] {goal: catchingup}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=7.5cm] + at (\bx, 9.9) {\small \texttt{pg\_basebackup}, hot standby}; + +\draw[msg] (\bx,10.9) -- (\mx,10.9) + node[lbl,midway,above] {report: catchingup, connected}; + +%% ── convergence ───────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,12.0) {convergence}; + +\draw[rsp] (\mx,12.3) -- (\bx,12.3) + node[lbl,midway,above] {goal: secondary}; +\draw[rsp] (\mx,13.1) -- (\ax,13.1) + node[lbl,midway,above] {goal: primary}; + +\draw[msg, color=pbox, line width=0.8pt] (\ax,14.0) -- (\bx,14.0) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\bx,14.9) -- (\mx,14.9); +\node[lbl, above] at (\bx+3.6, 14.9) {sync\_state=sync, lag=0}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (9.5, 15.9) {$\checkmark$~~A primary, B secondary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\ax, \ybot+0.35) {Primary A}; +\node[actor, fill=sbox, text=stxt] at (\bx, \ybot+0.35) {Secondary B}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-monitoring-protocol.svg b/docs/tikz/seq-monitoring-protocol.svg new file mode 100644 index 000000000..98a5208e5 --- /dev/null +++ b/docs/tikz/seq-monitoring-protocol.svg @@ -0,0 +1,703 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-monitoring-protocol.tex b/docs/tikz/seq-monitoring-protocol.tex new file mode 100644 index 000000000..e8155f64e --- /dev/null +++ b/docs/tikz/seq-monitoring-protocol.tex @@ -0,0 +1,84 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% N=0 M=12 +\def\nx{0} +\def\mx{12} +\def\ybot{9.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=sbox, text=stxt] (Nh) at (\nx,0) {Node}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\nx,0.35) -- (\nx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── node_active polling (node-initiated) ────────────────────────────────── +\node[seclbl] at (-4.4,1.3) {state reporting}; + +\draw[msg] (\nx,1.6) -- (\mx,1.6); +\node[lbl, above] at (\nx+3.4, 1.6) {node\_active(reported\_state, lsn, \ldots)}; + +\node[note, fill=mbox!25, draw=mbox!70!black, text width=6.5cm] + at (\mx, 2.6) + {\small store reported state\\ compute goal state}; + +\draw[rsp] (\mx,3.6) -- (\nx,3.6) + node[lbl,midway,above] {goal\_state}; + +\draw[gray!50, line width=0.5pt, ->] + (\nx-0.5, 4.0) .. controls (\nx-2.2,4.3) and (\nx-2.2,5.6) .. (\nx-0.5,5.9) + node[left, font=\small\itshape, text=gray!70] {repeat\ldots}; + +%% ── health check (monitor-initiated) ────────────────────────────────────── +\node[seclbl] at (-4.4,6.6) {health check}; + +\draw[msg, color=mbox!70!black] (\mx,6.9) -- (\nx,6.9); +\node[lbl, above, text=mbox!70!black] at (\mx-3.4, 6.9) + {connect (\texttt{pg\_isready}-equivalent)}; + +\draw[rsp] (\nx,7.8) -- (\mx,7.8) + node[lbl,midway,above] {ready}; + +%% ── unhealthy note ────────────────────────────────────────────────────── +\node[note, fill=red!10, draw=red!40, text width=12.5cm, font=\small] + at (6, \ybot-0.3) + {\color{red!80!black}\bfseries unhealthy \normalfont only when \emph{both} + signals are missing: the monitor cannot connect, \emph{and} no + node\_active report has arrived for a while}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=sbox, text=stxt] at (\nx, \ybot+1.1) {Node}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+1.1) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg new file mode 100644 index 000000000..4379e2efa --- /dev/null +++ b/docs/tikz/seq-primary-self-fence.svg @@ -0,0 +1,1063 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex new file mode 100644 index 000000000..529d48cff --- /dev/null +++ b/docs/tikz/seq-primary-self-fence.tex @@ -0,0 +1,97 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{12.9} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; + +%% ── network partition ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,2.6) {network partition}; + +\node[note, fill=red!12, draw=red!50, text width=10.5cm] + at (9.5, 3.0) + {\color{red!80!black} Primary $\leftrightarrow$ Monitor unreachable}; + +%% ── local decision on the primary ────────────────────────────────────── +\node[seclbl] at (-3.4,4.3) {primary checks: secondary still connected?}; + +\node[note, fill=blue!8, draw=blue!40, text width=13.5cm] + at (9.5, 5.2) + {\small \textbf{yes} $\Rightarrow$ keep serving reads/writes, nothing else to do\\ + \textbf{no} $\Rightarrow$ wait \texttt{network\_partition\_timeout} (20s by default)}; + +%% ── self-fence ────────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,7.0) {self-fence (no secondary, timeout elapsed)}; + +\node[note, fill=orange!15, draw=orange!70!black, text width=8cm] + at (\px, 7.9) + {\small stop Postgres\\ local state $\rightarrow$ \texttt{demote\_timeout}}; + +\draw[lldead] (\px,8.6) -- (\px,\ybot); + +\node[note, fill=orange!10, draw=orange!60, text width=13cm, font=\small] + at (9.5, 9.6) + {avoids a split brain: with no secondary and no monitor reachable, the + primary cannot be sure it isn't isolated, so it refuses to keep + accepting writes}; + +\node[note, fill=gray!10, draw=gray!50, text width=13cm, font=\small] + at (9.5, 11.1) + {reported to the monitor as \texttt{demote\_timeout} once connectivity is + restored}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=orange!25, draw=orange!70!black, text=orange!30!black] + at (\px, \ybot+0.35) {\small Primary (self-fenced)}; +\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-secondary-unhealthy.svg b/docs/tikz/seq-secondary-unhealthy.svg new file mode 100644 index 000000000..c4c6c1c87 --- /dev/null +++ b/docs/tikz/seq-secondary-unhealthy.svg @@ -0,0 +1,942 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-secondary-unhealthy.tex b/docs/tikz/seq-secondary-unhealthy.tex new file mode 100644 index 000000000..32fc7e541 --- /dev/null +++ b/docs/tikz/seq-secondary-unhealthy.tex @@ -0,0 +1,107 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{15.0} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state: synchronous ────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state (synchronous)}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; +\draw[rsp] (\sx,2.4) -- (\px,2.4) + node[lbl,midway,above] {sync ack}; + +%% ── secondary goes unhealthy ──────────────────────────────────────────── +\node[seclbl] at (-3.4,3.5) {secondary goes unhealthy}; + +\node[note, fill=red!12, draw=red!50, text width=9.5cm] + at (\sx, 3.9) + {\color{red!80!black} health checks / node\_active reports stop + arriving}; + +%% ── fallback to async ─────────────────────────────────────────────────── +\node[seclbl] at (-3.4,5.3) {fallback to async}; + +\draw[rsp] (\mx,5.6) -- (\px,5.6); +\node[lbl, above] at (\mx-2.6, 5.6) {goal: wait\_primary}; + +\draw[rsp] (\mx,6.4) -- (\sx,6.4) + node[lbl,midway,above] {goal: catchingup}; + +\node[note, fill=async, draw=gray!50, text width=13.5cm] + at (9.5, 7.4) + {\small synchronous replication disabled -- writes on the primary no + longer wait for the secondary; secondary can't be promoted while it + isn't in state \texttt{secondary}}; + +\draw[msg, color=pbox, line width=0.8pt] (\px,8.6) -- (\sx,8.6) + node[lbl,midway,above] {WAL stream (async)}; + +%% ── secondary catches up ──────────────────────────────────────────────── +\node[seclbl] at (-3.4,10.0) {secondary catches up}; + +\draw[msg] (\sx,10.3) -- (\mx,10.3); +\node[lbl, above] at (\sx+3.2, 10.3) {report: WAL delta acceptable}; + +%% ── back to synchronous ───────────────────────────────────────────────── +\node[seclbl] at (-3.4,11.7) {back to synchronous}; + +\draw[rsp] (\mx,12.0) -- (\sx,12.0) + node[lbl,midway,above] {goal: secondary}; +\draw[rsp] (\mx,12.8) -- (\px,12.8) + node[lbl,midway,above] {goal: primary}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (4.5, 13.9) {$\checkmark$~~synchronous replication restored}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary}; +\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} From 6215f156e2e212e19a8cbffd13942b5c878a9eb5 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:03:43 +0200 Subject: [PATCH 2/8] Update project history note: no longer corporately maintained Remove the "Citus Data team at Microsoft generously maintains" framing, which is no longer accurate. State plainly that the project is now maintained on volunteers' free time with no financial backing, and link to oss.theartofpostgresql.com for professional support or sponsoring. --- docs/index.rst | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index b1e315db7..5f7f8440e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,14 +20,16 @@ __ https://github.com/hapostgres/pg_auto_failover .. note:: - The development of pg_auto_failover has been driven by Citus Data, since - then a team at Microsoft. The Citus Data team at Microsoft generously - maintains the pg_auto_failover Open Source Software so that its users may - continue using it in production. + The development of pg_auto_failover was originally driven by Citus Data, + and then a team at Microsoft. The project is now orphaned in terms of + financial support: maintenance and development happen on volunteers' + free time. For enhancements, improvements, and new features, consider contributing - to the project. Pull Requests are reviewed as part of the offered - maintenance. + to the project. Pull Requests are reviewed as time allows. + + For professional support or to sponsor ongoing maintenance, see + `oss.theartofpostgresql.com `_. .. note:: From 047e639564c9820861f0243f2f46c7a053fa8336 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:07:12 +0200 Subject: [PATCH 3/8] Move monitoring protocol diagram to the end of its section --- docs/architecture.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/architecture.rst b/docs/architecture.rst index 6f9c1e411..0ee01f345 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -208,11 +208,6 @@ occurred. Monitoring protocol ------------------- -.. figure:: ./tikz/seq-monitoring-protocol.svg - :alt: Sequence diagram of the node_active polling and health check protocols - - The two protocols between a node's keeper and the monitor - The monitor interacts with the data nodes in 2 ways: - Data nodes periodically connect and run `SELECT @@ -240,6 +235,11 @@ cannot connect *and* it hasn't reported its state through `node_active` for a while. This allows, for example, PostgreSQL to be restarted without causing a health check failure. +.. figure:: ./tikz/seq-monitoring-protocol.svg + :alt: Sequence diagram of the node_active polling and health check protocols + + The two protocols between a node's keeper and the monitor + Synchronous vs. asynchronous replication ---------------------------------------- From 1b6725bfd9fa6125a707f55e78350578a26c85ba Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:19:29 +0200 Subject: [PATCH 4/8] Make the primary self-fence diagram show the actual failures, not just state Add two explicit broken-connection arrows in the "network partition" section: the attempted node_active() call that gets no response, and the WAL stream to the secondary going dark. Previously this was only a text note asserting both were unreachable; now the diagram shows why the primary concludes it might be isolated. --- docs/tikz/seq-primary-self-fence.svg | 1541 +++++++++++++++----------- docs/tikz/seq-primary-self-fence.tex | 39 +- 2 files changed, 909 insertions(+), 671 deletions(-) diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg index 4379e2efa..311474596 100644 --- a/docs/tikz/seq-primary-self-fence.svg +++ b/docs/tikz/seq-primary-self-fence.svg @@ -1,1063 +1,1292 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - + + - - + + - + - + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + - - + + - + - - + + - - - - - - + + + + + + + + + + + - - - + - + - + + - - + + - - + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + + + - - - - + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - + - - - - - - - + + + + - - + + + + - - - - - + + - - - - - - - - - - + + + - + + + + + + + + + + + - + + - - + + + + + + + - + + - + - - - + + - - - - - - - + + + + + + - - - - - - - - - - - - - + - - - - - - - + - - - - + - - + + + - - + + + - - + - + + + + + + + + + + + + + + + + + + + + + - + - - - + + - - - + - + - - - - - - - - - - - - - - - - - - - - - + - + + + + + + + + + - - + + + + - + + + + + + + + + + - + + + - + + + - - - - - - - - + + - - - - - - - - - - - + + - - - - + + + + + + + - - - - - - - - + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + - - - - - - - + + + + + + + + - - - - - - - - + + + + + + + + - + - - - - + + + + - + - - - - - - - + + + + + + + - - + + - - - + + + - - - - - + + + + + - + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - + + - - - - - - + + + + + + - + - - - - - + + + + + - + - - - - - + + + + + - - - - + + + + - - + + - - - - - - - + + + + + + + - - + + - - - + + + - - + + - - - - + + + + - - - + + + - + - - - - - - - - - - + + + + + + + + + + - - - + + + - + - - - - + + + + - - + + - - - - - - + + + + + + - + - + - - - - + + + + - - + + - - - - - + + + + + - - - - - - - - - + + + + + + + + + - - + + - - + + - - - - - - - + + + + + + + - - + + - + - - - + + + - - - - - - - - - + + + + + + + + + - - - - - - + + + + + + - + - - - + + + - + - - - - + + + + - - + + - - - + + + - - - - - - + + + + + + - + - - + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - + + + + - - - - - - - - - - - + + + + + + + + + + + - + - - + + - - - - - + + + + + - - - + + + - + - - - - - + + + + + - - + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex index 529d48cff..726eb8b10 100644 --- a/docs/tikz/seq-primary-self-fence.tex +++ b/docs/tikz/seq-primary-self-fence.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{12.9} +\def\ybot{14.6} \begin{tikzpicture}[ >=Stealth, @@ -31,6 +31,7 @@ \tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] \tikzstyle{msg}=[->, line width=0.65pt] \tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] \tikzstyle{lbl}=[font=\small, inner sep=2pt] \tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, font=\small, align=center] @@ -52,38 +53,46 @@ \draw[msg] (\px,1.6) -- (\sx,1.6) node[lbl,midway,above] {WAL stream}; -%% ── network partition ────────────────────────────────────────────────── -\node[seclbl] at (-3.4,2.6) {network partition}; +%% ── network partition: both connections lost ────────────────────────── +\node[seclbl] at (-3.4,2.8) {network partition: both connections lost}; -\node[note, fill=red!12, draw=red!50, text width=10.5cm] - at (9.5, 3.0) - {\color{red!80!black} Primary $\leftrightarrow$ Monitor unreachable}; +\draw[broken] (\px,3.7) -- (13,3.7); +\node[red!70!black, font=\Large] at (13.4,3.7) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,3.7) + {node\_active(\ldots) attempted -- no response}; + +\draw[broken] (\px,4.9) -- (6,4.9); +\node[red!70!black, font=\Large] at (6.4,4.9) {$\times$}; +\node[lbl, above, text=red!70!black] at (3,4.9) + {WAL stream -- connection lost}; %% ── local decision on the primary ────────────────────────────────────── -\node[seclbl] at (-3.4,4.3) {primary checks: secondary still connected?}; +\node[seclbl] at (-3.4,6.3) {primary's local decision}; \node[note, fill=blue!8, draw=blue!40, text width=13.5cm] - at (9.5, 5.2) - {\small \textbf{yes} $\Rightarrow$ keep serving reads/writes, nothing else to do\\ - \textbf{no} $\Rightarrow$ wait \texttt{network\_partition\_timeout} (20s by default)}; + at (9.5, 7.3) + {\small no secondary, no monitor $\Rightarrow$ wait + \texttt{network\_partition\_timeout} (20s by default), then self-fence\\ + \normalfont\itshape\small (if the secondary were still reachable, the + primary would just keep serving instead)}; %% ── self-fence ────────────────────────────────────────────────────────── -\node[seclbl] at (-3.4,7.0) {self-fence (no secondary, timeout elapsed)}; +\node[seclbl] at (-3.4,9.3) {self-fence (timeout elapsed)}; \node[note, fill=orange!15, draw=orange!70!black, text width=8cm] - at (\px, 7.9) + at (\px, 10.2) {\small stop Postgres\\ local state $\rightarrow$ \texttt{demote\_timeout}}; -\draw[lldead] (\px,8.6) -- (\px,\ybot); +\draw[lldead] (\px,10.9) -- (\px,\ybot); \node[note, fill=orange!10, draw=orange!60, text width=13cm, font=\small] - at (9.5, 9.6) + at (9.5, 12.1) {avoids a split brain: with no secondary and no monitor reachable, the primary cannot be sure it isn't isolated, so it refuses to keep accepting writes}; \node[note, fill=gray!10, draw=gray!50, text width=13cm, font=\small] - at (9.5, 11.1) + at (9.5, 13.6) {reported to the monitor as \texttt{demote\_timeout} once connectivity is restored}; From ce5833dca6d9499d57243cb79bf91ec6b232f78b Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:24:55 +0200 Subject: [PATCH 5/8] Add diagram for "Monitor can't connect to Primary" Distinct from the primary's own self-fence diagram: this one is from the monitor's side, which doesn't know why the primary stopped responding, only that it did. Shows the health-check retries failing, the secondary being promoted through prepare_promotion (with the 30s WAL-delta timeout noted), and the old primary's goal being fenced to demoted regardless of whether it's actually reachable -- cross-referencing the self-fence diagram for what happens if it's still alive. --- docs/fault-tolerance.rst | 5 + docs/tikz/seq-monitor-cant-reach-primary.svg | 1582 ++++++++++++++++++ docs/tikz/seq-monitor-cant-reach-primary.tex | 113 ++ 3 files changed, 1700 insertions(+) create mode 100644 docs/tikz/seq-monitor-cant-reach-primary.svg create mode 100644 docs/tikz/seq-monitor-cant-reach-primary.tex diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index 6a7344ec9..747f9a894 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -131,6 +131,11 @@ PostgreSQL service: delta to resorb (30s by default) then the secondary can be promoted with uncertainty about the data durability in the group. + .. figure:: ./tikz/seq-monitor-cant-reach-primary.svg + :alt: Sequence diagram of the monitor failing over to the secondary after losing contact with the primary + + The monitor promotes the secondary and fences the old primary + - Monitor can't connect to Secondary As soon as the secondary is considered unhealthy then the monitor diff --git a/docs/tikz/seq-monitor-cant-reach-primary.svg b/docs/tikz/seq-monitor-cant-reach-primary.svg new file mode 100644 index 000000000..8dd0ee81b --- /dev/null +++ b/docs/tikz/seq-monitor-cant-reach-primary.svg @@ -0,0 +1,1582 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-monitor-cant-reach-primary.tex b/docs/tikz/seq-monitor-cant-reach-primary.tex new file mode 100644 index 000000000..ce6a1ff85 --- /dev/null +++ b/docs/tikz/seq-monitor-cant-reach-primary.tex @@ -0,0 +1,113 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{16.4} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; + +%% ── health checks fail ───────────────────────────────────────────────── +\node[seclbl] at (-3.4,2.8) {monitor's health checks to primary fail}; + +\draw[broken] (\mx,3.7) -- (6,3.7); +\node[red!70!black, font=\Large] at (5.6,3.7) {$\times$}; +\node[lbl, above, text=red!70!black] at (13,3.7) + {health check: no response (retried)}; + +\node[note, fill=red!10, draw=red!50, text width=13cm, font=\small] + at (9.5, 5.0) + {after \texttt{health\_check\_max\_retries} (2 by default), the primary is + marked unhealthy -- this says nothing about whether it is actually + down, only that the monitor can no longer reach it}; + +%% ── failover begins ───────────────────────────────────────────────────── +\node[seclbl] at (-3.4,6.6) {failover begins: promote the secondary}; + +\draw[msg, color=pbox, line width=0.8pt] (\mx,7.4) -- (\sx,7.4); +\node[lbl, above, text=pbox] at (\mx-2.6,7.4) {goal: prepare\_promotion}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=11cm] + at (\sx, 8.4) + {\small catches up on any last WAL still available from the primary, + then reports success}; + +\node[note, fill=blue!8, draw=blue!40, text width=13.5cm, font=\small] + at (9.5, 9.7) + {if the WAL delta doesn't resorb within 30s (default), the secondary is + promoted anyway -- with uncertainty about data durability}; + +%% ── old primary fenced ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,11.1) {old primary fenced (STONITH)}; + +\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] + at (9.5, 12.0) + {goal recorded as \texttt{draining} $\rightarrow$ \texttt{demoted}, + whether or not the primary is actually reachable right now}; + +\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] + at (9.5, 13.4) + {if the primary is in fact still alive, it demotes itself as soon as it + can reach the monitor again -- or it may have already self-fenced + independently (see the previous diagram)}; + +%% ── promotion complete ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,14.8) {promotion complete}; + +\draw[msg, color=pbox, line width=1pt] (\mx,15.1) -- (\sx,15.1); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,15.1) + {goal: wait\_primary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black] + at (\px, \ybot+0.35) {\small Primary (fenced)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} From 697b8a83b8c68db2d64ba70c4a3c105ca5faec39 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:40:58 +0200 Subject: [PATCH 6/8] Show real node_active traffic and reported/goal states in every diagram Every "goal: X" arrow was an abstraction hiding how the protocol actually works: the monitor never pushes a goal state, a node only ever learns its goal as the return value of its own node_active(reported=...) call. Replace every one of those abstracted pushes with the real exchange -- and where a goal change is decided while processing a *different* node's call (e.g. B converging triggers A's promotion), show it landing on that node's own next poll, not as an immediate cross-node push. Also make explicit which node can actually reach the monitor at each point: - seq-monitor-cant-reach-primary: primary's node_active attempts fail alongside the health-check failures already shown; secondary keeps polling and getting responses throughout, visibly unaffected, until the monitor uses its next poll to hand it the failover goal. - seq-primary-self-fence: same contrast -- secondary's node_active keeps succeeding while primary's own attempt goes unanswered. - seq-asymmetric-partition: only the primary-monitor link is shown broken; secondary's polling continues normally, which is how it receives stop_replication. - seq-secondary-unhealthy: secondary's failed attempt is shown explicitly, followed by primary's own next poll revealing wait_primary, and secondary's eventual recovery shown as real reported/goal exchanges rather than a bare "report:"/"goal:" shorthand. - seq-birth-of-a-cluster: register_node() kept distinct from node_active() polling; every FSM transition now shows the actual reported/goal pair, including the two places (A's promotion, B's catchingup assignment) where the goal is decided while processing the other node's call and only delivered on that node's own next poll. seq-monitoring-protocol.svg (architecture.rst) is unchanged -- it's already the generic node_active/goal_state protocol reference with no concrete scenario to trace state through. --- docs/tikz/seq-asymmetric-partition.svg | 1688 +++++++---- docs/tikz/seq-asymmetric-partition.tex | 67 +- docs/tikz/seq-birth-of-a-cluster.svg | 1480 ++++++---- docs/tikz/seq-birth-of-a-cluster.tex | 90 +- docs/tikz/seq-monitor-cant-reach-primary.svg | 2635 ++++++++++++------ docs/tikz/seq-monitor-cant-reach-primary.tex | 111 +- docs/tikz/seq-primary-self-fence.svg | 1744 +++++++----- docs/tikz/seq-primary-self-fence.tex | 52 +- docs/tikz/seq-secondary-unhealthy.svg | 1687 +++++++---- docs/tikz/seq-secondary-unhealthy.tex | 87 +- 10 files changed, 6205 insertions(+), 3436 deletions(-) diff --git a/docs/tikz/seq-asymmetric-partition.svg b/docs/tikz/seq-asymmetric-partition.svg index 7fc947885..e497612ca 100644 --- a/docs/tikz/seq-asymmetric-partition.svg +++ b/docs/tikz/seq-asymmetric-partition.svg @@ -1,5 +1,5 @@ - + @@ -51,57 +51,84 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -132,88 +159,85 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - + + - + @@ -235,778 +259,1216 @@ - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - + + + + - - - - + + + + - - - - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + - - - - + + + + + + - - + - - - - - + + + + + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + - - - - - - - - - - + + + - + + + + + + + + + + + + + + + + + + + - - - - + + + + - - - + + + - - - - - - - + + + + + + + - - + + - - - - + + + + - - + + - - + + - - - + + + - + - - - - + + + + - - + + - - - - - - - - - - - + + + + + + + + + + + - - - + + + - - - - + + + + - + - + - + - - + + - + - - - - + + + + - - + + - - - - - - - + + + + + + + - - + + - - - - - - - + + + + + + + - - - - - - - - - + + + + + + + + + - - - + + + - - - - - + + + + + - - + + - - - - + + + + - - - - - - + + + + + + - - + + - + - - + + - + - + - - + + - + - - - - - + + + + + - + - + - - - + + + - + - + - + - - - + + + - - - - + + + + - - + + - - + + - - + + - - + + - - - + + + - - - - - - + + + + + + - + - + - - - - + + + + - + - - - - + + + + - - + + - + - + - - - + + + - - + + - + - - - - - - - - + + + + + + + + - + - + - - + + - - - - + + + + - - - - - - - + + + + + + + - - - - + + + + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + - - - - - + + + + + - - - - + + + + - - + + - + - - - - - + + + + + - - + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - + + + - - - - - + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-asymmetric-partition.tex b/docs/tikz/seq-asymmetric-partition.tex index 56a99f1c0..1ae030f16 100644 --- a/docs/tikz/seq-asymmetric-partition.tex +++ b/docs/tikz/seq-asymmetric-partition.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{14.4} +\def\ybot{22.2} \begin{tikzpicture}[ >=Stealth, @@ -31,6 +31,7 @@ \tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] \tikzstyle{msg}=[->, line width=0.65pt] \tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] \tikzstyle{lbl}=[font=\small, inner sep=2pt] \tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, font=\small, align=center] @@ -46,55 +47,77 @@ \draw[ll] (\sx,0.35) -- (\sx,\ybot); \draw[ll] (\mx,0.35) -- (\mx,\ybot); -%% ── asymmetric partition ──────────────────────────────────────────────── -\node[seclbl] at (-3.4,1.3) {asymmetric partition}; +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; \draw[msg] (\px,1.6) -- (\sx,1.6) - node[lbl,midway,above] {WAL stream (still fine)}; + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; + +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; + +%% ── asymmetric partition ──────────────────────────────────────────────── +\node[seclbl] at (-3.4,6.3) {asymmetric partition: only the Primary-Monitor link breaks}; + +\draw[broken] (\px,7.2) -- (13,7.2); +\node[red!70!black, font=\Large] at (13.4,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,7.2) + {node\_active(reported=primary) attempted -- no response}; -\node[note, fill=red!12, draw=red!50, text width=8.5cm] - at (\px, 2.5) - {\color{red!80!black} Primary $\leftrightarrow$ Monitor unreachable\\ - Monitor $\leftrightarrow$ Secondary still fine}; +\node[note, fill=gray!10, draw=gray!45, text width=13.5cm, font=\small] + at (9.5, 8.4) + {the primary keeps replicating normally to the secondary throughout -- + only its link to the monitor is down}; %% ── monitor initiates failover ───────────────────────────────────────── -\node[seclbl] at (-3.4,4.2) {monitor initiates failover}; +\node[seclbl] at (-3.4,9.8) {secondary is still reachable, monitor initiates failover}; -\draw[msg, color=pbox, line width=0.8pt] (\mx,4.5) -- (\sx,4.5); -\node[lbl, above, text=pbox] at (\mx-2.4, 4.5) {goal: stop\_replication}; +\draw[msg] (\sx,10.7) -- (\mx,10.7); +\node[lbl, above] at (\sx+3.3,10.7) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,11.3) -- (\sx,11.3); +\node[lbl, above, text=pbox] at (\mx-2.6, 11.3) {goal=stop\_replication}; \node[note, fill=sbox!25, draw=sbox!70!black, text width=8cm] - at (\sx, 5.4) {\small disconnects from Primary}; + at (\sx, 12.3) {\small disconnects from Primary}; %% ── primary loses its last connection ────────────────────────────────── -\node[seclbl] at (-3.4,7.0) {primary loses its last connection}; +\node[seclbl] at (-3.4,13.7) {primary loses its last connection}; \node[note, fill=orange!15, draw=orange!70!black, text width=12.5cm] - at (4.5, 7.9) + at (4.5, 14.6) {\small with the secondary gone too, the primary self-fences the same way as a full partition (previous diagram): expected to stop within 30--60s}; -\draw[lldead] (\px,8.7) -- (\px,\ybot); +\draw[lldead] (\px,15.4) -- (\px,\ybot); %% ── 90s safety wait ───────────────────────────────────────────────────── -\node[seclbl] at (-3.4,10.0) {90s safety wait}; +\node[seclbl] at (-3.4,16.9) {90s safety wait}; \node[note, fill=blue!8, draw=blue!40, text width=13.5cm] - at (\mx-1, 10.9) + at (\mx-1, 17.8) {\small worst case is 60s, so the monitor waits a full 90s before promoting -- a margin against edge cases}; \draw[gray!50, line width=0.5pt, ->] - (\mx+2.6, 9.6) .. controls (\mx+4.3,9.9) and (\mx+4.3,11.9) .. (\mx+2.6,12.2) + (\mx+2.6, 16.5) .. controls (\mx+4.3,16.8) and (\mx+4.3,18.8) .. (\mx+2.6,19.1) node[right, font=\small\itshape, text=gray!70] {waiting\ldots}; %% ── promotion ──────────────────────────────────────────────────────────── -\node[seclbl] at (-3.4,13.0) {promotion}; +\node[seclbl] at (-3.4,20.2) {promotion}; -\draw[msg, color=pbox, line width=1pt] (\mx,13.3) -- (\sx,13.3); -\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.6,13.3) - {goal: wait\_primary}; +\draw[msg] (\sx,21.1) -- (\mx,21.1); +\node[lbl, above] at (\sx+3.7,21.1) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,21.7) -- (\sx,21.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,21.7) + {goal=wait\_primary}; %% ── actor footers ─────────────────────────────────────────────────────── \node[actor, fill=orange!25, draw=orange!70!black, text=orange!30!black] diff --git a/docs/tikz/seq-birth-of-a-cluster.svg b/docs/tikz/seq-birth-of-a-cluster.svg index eb8db137f..09f53e78e 100644 --- a/docs/tikz/seq-birth-of-a-cluster.svg +++ b/docs/tikz/seq-birth-of-a-cluster.svg @@ -1,857 +1,1303 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - - - - + + - + - - + + - - + + - + - + - - + + - - + + - + - + - - - - - - + + + + + + - + - - - + + + - - + + - - + + - + - - - - - - - - - + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + - - - + + + + - - + - - - - + - - - - - - + + + - + + + + + + + + + + + - - - - - - + + + + + + - - - - - - + + + + + + + + + + - - - - - - - + - - + + + - - + + + + - + - - - - - - - - - - + + + - - - - + + + + + + + + - - - + + + + - - - - - + + + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + - - - - - - + + + + + - - - - - + + + + - - + + + + - - - - - - + + - + - + + - - - - - - - - - - + + + + - + + + + + + + - + - + + + + + + + + + - - - - + + - - - - + - + + + + + - - - - - - - + + + + - - + + - + + - - - - + + - - + + + + + + + + + + + + + - + - - + + + + + + - - - - + + + + + - - - - - - - + + + + - - - - - + + + + + - + + + + + + - - - + + + + + - + + + + - - - - - + + - - - - - + + + + - - - + + + + + - - - - - - - - + + - - - - - - - - - - - + + + + - - - - - - - - - - - - + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + - - - + + + - - - - - - + - + + + - - - - - + + + + + + - + - - - - + + - - - - - - - - - - - - + + + + + + + + + + + - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + - - - - - - - + + + + + + + + + + + + + - - + - - - - - - - + + + + + + - + + + + + - - - - + + + + - - + + + - - + + - + + + + + + - - + + + + - - - - - - + + - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - + + + - - + + + + + + + - + - + - + - + - - - - + + + + - - + + - + - + - - - - - - - + + + + + + + - - + + - + - - - - - + + + + + - - + + - + - + - - - - - - - + + + + + + + - - + + - + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-birth-of-a-cluster.tex b/docs/tikz/seq-birth-of-a-cluster.tex index cc812ad30..475b44201 100644 --- a/docs/tikz/seq-birth-of-a-cluster.tex +++ b/docs/tikz/seq-birth-of-a-cluster.tex @@ -16,7 +16,7 @@ \def\ax{0} \def\bx{9} \def\mx{19} -\def\ybot{16.6} +\def\ybot{26.6} \begin{tikzpicture}[ >=Stealth, @@ -46,61 +46,83 @@ \draw[ll] (\bx,0.35) -- (\bx,\ybot); \draw[ll] (\mx,0.35) -- (\mx,\ybot); -%% ── node A registers ─────────────────────────────────────────────────── -\node[seclbl] at (-3.4,1.3) {node A registers}; +%% ── node A registers, converges alone ────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {node A registers, converges alone}; \draw[msg] (\ax,1.6) -- (\mx,1.6) - node[lbl,midway,above] {keeper init}; + node[lbl,midway,above] {register\_node(\ldots)}; +\draw[rsp] (\mx,2.2) -- (\ax,2.2); +\node[lbl, above] at (\mx-2.6,2.2) {state=init, goal=single}; -\draw[rsp] (\mx,2.5) -- (\ax,2.5); -\node[lbl, above] at (\mx-3.0, 2.5) {init $\rightarrow$ single (goal: single)}; +\draw[msg] (\ax,3.3) -- (\mx,3.3); +\node[lbl, above] at (\ax+4.2,3.3) {node\_active(reported=single)}; +\draw[rsp] (\mx,3.9) -- (\ax,3.9); +\node[lbl, above] at (\mx-2.2,3.9) {goal=single}; %% ── node B joins ──────────────────────────────────────────────────────── -\node[seclbl] at (-3.4,3.6) {node B joins}; +\node[seclbl] at (-3.4,5.1) {node B joins}; -\draw[msg] (\bx,3.9) -- (\mx,3.9) - node[lbl,midway,above] {keeper init}; +\draw[msg] (\bx,5.4) -- (\mx,5.4) + node[lbl,midway,above] {register\_node(\ldots)}; +\draw[rsp] (\mx,6.0) -- (\bx,6.0); +\node[lbl, above] at (\mx-2.9,6.0) {state=init, goal=wait\_standby}; -\draw[rsp] (\mx,4.8) -- (\ax,4.8) - node[lbl,midway,above] {goal: wait\_primary}; -\draw[rsp] (\mx,5.6) -- (\bx,5.6); -\node[lbl, above] at (\mx-2.6, 5.6) {init $\rightarrow$ wait\_standby}; +%% ── A's next poll reveals the new goal ────────────────────────────────── +\node[seclbl] at (-3.4,7.4) {A's next poll reveals the new goal}; -%% ── A prepares, B catches up ─────────────────────────────────────────── -\node[seclbl] at (-3.4,6.7) {A prepares, B catches up}; +\draw[msg] (\ax,8.3) -- (\mx,8.3); +\node[lbl, above] at (\ax+4.2,8.3) {node\_active(reported=single)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,8.9) -- (\ax,8.9); +\node[lbl, above, text=pbox] at (\mx-2.4,8.9) {goal=wait\_primary}; \node[note, fill=pbox!12, draw=pbox!60, text width=6.2cm] - at (\ax, 7.6) {\small allows B in \texttt{pg\_hba.conf}}; + at (\ax, 10.0) {\small allows B in \texttt{pg\_hba.conf}}; -\draw[msg] (\ax,8.4) -- (\mx,8.4) - node[lbl,midway,above] {report: wait\_primary}; +%% ── A converges to wait_primary ───────────────────────────────────────── +\node[seclbl] at (-3.4,11.4) {A converges to wait\_primary}; -\draw[rsp] (\mx,9.0) -- (\bx,9.0) - node[lbl,midway,above] {goal: catchingup}; +\draw[msg] (\ax,12.3) -- (\mx,12.3); +\node[lbl, above] at (\ax+4.6,12.3) {node\_active(reported=wait\_primary)}; +\draw[rsp] (\mx,12.9) -- (\ax,12.9); +\node[lbl, above] at (\mx-2.2,12.9) {goal=wait\_primary}; + +%% ── B's next poll reveals catchingup ──────────────────────────────────── +\node[seclbl] at (-3.4,14.3) {B's next poll reveals catchingup}; + +\draw[msg] (\bx,15.2) -- (\mx,15.2); +\node[lbl, above] at (\bx+3.7,15.2) {node\_active(reported=wait\_standby)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,15.8) -- (\bx,15.8); +\node[lbl, above, text=pbox] at (\mx-2.4,15.8) {goal=catchingup}; \node[note, fill=sbox!25, draw=sbox!70!black, text width=7.5cm] - at (\bx, 9.9) {\small \texttt{pg\_basebackup}, hot standby}; + at (\bx, 16.9) {\small \texttt{pg\_basebackup}, hot standby}; -\draw[msg] (\bx,10.9) -- (\mx,10.9) - node[lbl,midway,above] {report: catchingup, connected}; +%% ── B converges; A promoted in the same pass ──────────────────────────── +\node[seclbl] at (-3.4,18.3) {B converges; monitor promotes A in the same pass}; -%% ── convergence ───────────────────────────────────────────────────────── -\node[seclbl] at (-3.4,12.0) {convergence}; +\draw[msg] (\bx,19.2) -- (\mx,19.2); +\node[lbl, above] at (\bx+3.8,19.2) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,19.8) -- (\bx,19.8); +\node[lbl, above, text=pbox] at (\mx-2.2,19.8) {goal=secondary}; -\draw[rsp] (\mx,12.3) -- (\bx,12.3) - node[lbl,midway,above] {goal: secondary}; -\draw[rsp] (\mx,13.1) -- (\ax,13.1) - node[lbl,midway,above] {goal: primary}; +%% ── A's next poll reveals primary ─────────────────────────────────────── +\node[seclbl] at (-3.4,21.2) {A's next poll reveals primary}; -\draw[msg, color=pbox, line width=0.8pt] (\ax,14.0) -- (\bx,14.0) - node[lbl,midway,above] {WAL stream}; +\draw[msg] (\ax,22.1) -- (\mx,22.1); +\node[lbl, above] at (\ax+4.6,22.1) {node\_active(reported=wait\_primary)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,22.7) -- (\ax,22.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,22.7) + {goal=primary}; -\draw[msg] (\bx,14.9) -- (\mx,14.9); -\node[lbl, above] at (\bx+3.6, 14.9) {sync\_state=sync, lag=0}; +%% ── convergence ───────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,24.1) {convergence}; + +\draw[msg, color=pbox, line width=0.8pt] (\ax,25.0) -- (\bx,25.0) + node[lbl,midway,above] {WAL stream}; \node[note, fill=green!15, draw=green!50!black, font=\small\bfseries, minimum width=8cm] - at (9.5, 15.9) {$\checkmark$~~A primary, B secondary}; + at (9.5, 25.9) {$\checkmark$~~A primary, B secondary}; %% ── actor footers ─────────────────────────────────────────────────────── \node[actor, fill=pbox, text=ptxt] at (\ax, \ybot+0.35) {Primary A}; diff --git a/docs/tikz/seq-monitor-cant-reach-primary.svg b/docs/tikz/seq-monitor-cant-reach-primary.svg index 8dd0ee81b..0aa16d301 100644 --- a/docs/tikz/seq-monitor-cant-reach-primary.svg +++ b/docs/tikz/seq-monitor-cant-reach-primary.svg @@ -1,5 +1,5 @@ - + @@ -69,77 +69,95 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + @@ -168,78 +186,105 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -285,16 +330,25 @@ - + - + - + - + + + + + + + + + + @@ -308,1275 +362,2090 @@ + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + - - + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + - - - + + + + + + - - - - - - + + + - - - - - - + + + - - + - + + - - - - + + - - + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + - - - - - - + + + + - - - - - - + + + - - + + + + + + + + + - - - - + - - - - + + - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + - - - - - - - - + + + + + + + + + + + + + - - - + - - - - - - - - - - - - - + - + - + - + + + + + + + + + - + + + - - - - - - - - - + - - - + + + - + + - - - - + + - - + + - - + + - - + + - - + + - - + + + + + + + + + - - - - - - - - - + - + + + + + + + + + - - - - + + - - + + + + + + - - + + + + - - - - - - - + - - + + + - - - + + - - - - - - - + - - + - - + + + + - - - - - - - - + + + + + + - - + + - - - + + - - - - + + + - - - - + + - - - + - - - - - - + - + - - - + + - - + + + - - - - - - + + - - - - - + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + - - - - - - - - + + - + + + + + + + + + + - - - - - - + + + + + - + - - - - - - + + + + + + - - - + + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + + - + + + + - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - + - - + + - - + + + + + - - - + + + + + + - - - - + + + + - + + - - + + - - - - - + + + + + + - - - - - - + + - - - - + - - - - + + + + + + + + + + - - - + + + - + + + + + + + - - - - + + - - + - + + + + - - - - + + + + - - - + - + + + - - - + - - - - - - - + + + + + + + - - - + + + + - - - + - + - - + - - - - - + + + + - - + + + + + + + + + + - - - - - + - - - - + + - - + + + - - - - - - + + + + - - + + + + + + + + + - + - - - - - - - - - - + - - - + + + + + + + + + - - - - + + + + - - - - + - - + + + + + + + + + + + + + + + + - - + - + - - - - - - - + + + + + + + + + + + - - - - + - + - + + - + + + - - - - + + + + - - - - - - - - - - + + - + + + - - + - - - + + + + + + + + + - - - - + + + - - - - - - - - - + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + - + + + + - - - - + + - - + + + + + + - - - - - - + + + + + + + + + - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + - - - - + + + + + + + - - - - + + - - + - - - - - - - - + + + - + - - - - - - - + + + + - + + - - - - - - - + + - + + + - + - - - + + - - - + + + + + - + - - - - + + - - + + + + + + + + - - + + + + + + + + + + - - - - - - - - + + + + + - - - - - - - - - + + + - - - - - + - - + + + + - + + - - - + + + + + + - - - + - + + + + + - - - - + + + + + - - + + + + - - + + + + + + + - - + + + - - - - + + + + + - - - - - + + - - - - - - + + + + - - + + - - - - - - - + + + + - - - - - - + + + + + - - + + + + + + - - + + - - + + + - - + + + - - + + + + - - - + + + + - - - - - + + + + + + + - - - + + + + + + - - - - - - + - + + + - - - - - + + + + - + + + + - + + - + + + - - + + + + + - - + + + - + + + + + + - - - - + - - - + - - - - - + - - - - - - - - - - - + - - - - - + + - - - - - - - - + + - - - - + - - - + + + + - + + + + + + + - - - - - - - + + + + + + + + + + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + + - + + + + + - - - - - + + + + - - - - + + - - - + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + - - + + - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + - - - - - + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-monitor-cant-reach-primary.tex b/docs/tikz/seq-monitor-cant-reach-primary.tex index ce6a1ff85..f07f000c8 100644 --- a/docs/tikz/seq-monitor-cant-reach-primary.tex +++ b/docs/tikz/seq-monitor-cant-reach-primary.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{16.4} +\def\ybot{28.8} \begin{tikzpicture}[ >=Stealth, @@ -46,66 +46,103 @@ \draw[ll] (\sx,0.35) -- (\sx,\ybot); \draw[ll] (\mx,0.35) -- (\mx,\ybot); -%% ── steady state ──────────────────────────────────────────────────────── +%% ── steady state: both nodes reachable ───────────────────────────────── \node[seclbl] at (-3.4,1.3) {steady state}; \draw[msg] (\px,1.6) -- (\sx,1.6) node[lbl,midway,above] {WAL stream}; -%% ── health checks fail ───────────────────────────────────────────────── -\node[seclbl] at (-3.4,2.8) {monitor's health checks to primary fail}; +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; -\draw[broken] (\mx,3.7) -- (6,3.7); -\node[red!70!black, font=\Large] at (5.6,3.7) {$\times$}; -\node[lbl, above, text=red!70!black] at (13,3.7) +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; + +%% ── primary goes silent ───────────────────────────────────────────────── +\node[seclbl] at (-3.4,6.3) + {primary goes silent: health checks and its own node\_active both fail}; + +\draw[broken] (\mx,7.2) -- (6,7.2); +\node[red!70!black, font=\Large] at (5.6,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (13,7.2) {health check: no response (retried)}; +\draw[broken] (\px,8.3) -- (13,8.3); +\node[red!70!black, font=\Large] at (13.4,8.3) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,8.3) + {node\_active(reported=primary) attempted -- no response}; + \node[note, fill=red!10, draw=red!50, text width=13cm, font=\small] - at (9.5, 5.0) + at (9.5, 9.6) {after \texttt{health\_check\_max\_retries} (2 by default), the primary is - marked unhealthy -- this says nothing about whether it is actually - down, only that the monitor can no longer reach it}; + marked unhealthy -- meanwhile it cannot even poll for a goal state, so + it has no way to learn anything changed}; + +%% ── secondary is still reachable ──────────────────────────────────────── +\node[seclbl] at (-3.4,11.0) {secondary is unaffected, still polling normally}; + +\draw[msg] (\sx,11.9) -- (\mx,11.9); +\node[lbl, above] at (\sx+3.3,11.9) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,12.5) -- (\sx,12.5); +\node[lbl, above] at (\mx-2.2,12.5) {goal=secondary}; %% ── failover begins ───────────────────────────────────────────────────── -\node[seclbl] at (-3.4,6.6) {failover begins: promote the secondary}; +\node[seclbl] at (-3.4,13.9) {failover begins: promote the secondary}; -\draw[msg, color=pbox, line width=0.8pt] (\mx,7.4) -- (\sx,7.4); -\node[lbl, above, text=pbox] at (\mx-2.6,7.4) {goal: prepare\_promotion}; +\draw[msg] (\sx,14.8) -- (\mx,14.8); +\node[lbl, above] at (\sx+3.3,14.8) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,15.4) -- (\sx,15.4); +\node[lbl, above, text=pbox] at (\mx-2.6,15.4) {goal=prepare\_promotion}; \node[note, fill=sbox!25, draw=sbox!70!black, text width=11cm] - at (\sx, 8.4) - {\small catches up on any last WAL still available from the primary, - then reports success}; - -\node[note, fill=blue!8, draw=blue!40, text width=13.5cm, font=\small] - at (9.5, 9.7) - {if the WAL delta doesn't resorb within 30s (default), the secondary is - promoted anyway -- with uncertainty about data durability}; + at (\sx, 16.4) + {\small catches up on any last WAL still available from the primary}; + +\draw[msg] (\sx,17.4) -- (\mx,17.4); +\node[lbl, above] at (\sx+3.5,17.4) {node\_active(reported=prepare\_promotion)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,18.0) -- (\sx,18.0); +\node[lbl, above, text=pbox] at (\mx-2.6,18.0) {goal=stop\_replication}; + +\node[note, fill=blue!8, draw=blue!40, text width=15cm, font=\small] + at (9.5, 19.2) + {if the WAL delta hadn't resorbed within 30s (default), the secondary + would have been promoted anyway -- with uncertainty about data + durability. Remaining hops (\texttt{stop\_replication} $\rightarrow$ + \texttt{wait\_primary}) are the same as any promotion; see the + Failover State Machine page}; %% ── old primary fenced ────────────────────────────────────────────────── -\node[seclbl] at (-3.4,11.1) {old primary fenced (STONITH)}; +\node[seclbl] at (-3.4,21.4) {old primary fenced (STONITH) -- monitor-side only}; -\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] - at (9.5, 12.0) - {goal recorded as \texttt{draining} $\rightarrow$ \texttt{demoted}, - whether or not the primary is actually reachable right now}; - -\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] - at (9.5, 13.4) - {if the primary is in fact still alive, it demotes itself as soon as it - can reach the monitor again -- or it may have already self-fenced - independently (see the previous diagram)}; +\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small] + at (9.5, 22.8) + {records goal \texttt{draining} $\rightarrow$ \texttt{demoted} for the + primary in its own table -- no delivery attempted, since the primary + hasn't polled since this section started. If it's in fact still alive, + it learns this (and demotes itself) the next time it can reach the + monitor -- or it may have already self-fenced independently (see the + previous diagram)}; %% ── promotion complete ────────────────────────────────────────────────── -\node[seclbl] at (-3.4,14.8) {promotion complete}; +\node[seclbl] at (-3.4,25.3) {promotion complete}; + +\draw[msg] (\sx,26.2) -- (\mx,26.2); +\node[lbl, above] at (\sx+3.7,26.2) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,26.8) -- (\sx,26.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,26.8) + {goal=wait\_primary}; -\draw[msg, color=pbox, line width=1pt] (\mx,15.1) -- (\sx,15.1); -\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,15.1) - {goal: wait\_primary}; +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (14, 27.9) {$\checkmark$~~Secondary is the new primary}; %% ── actor footers ─────────────────────────────────────────────────────── \node[actor, fill=gray!30, draw=gray!60, text=gray!30!black] - at (\px, \ybot+0.35) {\small Primary (fenced)}; + at (\px, \ybot+0.35) {\small Primary (fenced, unaware)}; \node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; \node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg index 311474596..328e5668e 100644 --- a/docs/tikz/seq-primary-self-fence.svg +++ b/docs/tikz/seq-primary-self-fence.svg @@ -1,1292 +1,1554 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + - - + + + + + + + + + + + + - - - - - - + + - - - + + + - + - + - - + + - - + + - - - - - - - - + + + + + + + + - + - - - + + + + - - - - - - - - - - - + + - - - - + + + + + + + + + + + + - - + + - - + + + + + + + + + + + + + - - - - - - - - - - - + - + + + + + + - + + + + - + + + - - - - - - - - - + + + + + + + + + - + - - + + - - - - + + + + - - - - + + + + - + - + - + - - + + - - - - - - + + + + + + - + - - - - - - - - - - + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - + + + + - - - - + + + + - - + + - - - + + + - - - - - - - - + + + + + + + + - + - - + + - - - - - - - + + + + + + + - - + + - + - - + + - - - - - - + + + + + + - + - + - + - - - + + + - - - + + + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + - + - - + + - + - + - + - - - - - - - - - + + + + + + + + + - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - - + + + - - - + + + - - + + - + - - - - - - + + + + + + - - - + + + - + - - - - + + + + - + - + - + - - - - - - - - + + + + + + + + - - - + + + - - - - - - - + + + + + + + - - - - - + + + + + - - - - + + + + - - + + - - + + - - - - - - - + + + + + + + - - - - - + + + + + - - - + + + - - - - - - - - - - + + + + + + + + + + - - - - - - - - + + + + + + + + - - - - - - - - + + + + + + + + - + - - - - + + + + - + - - - - - - - + + + + + + + - - + + - - - + + + - - - - - + + + + + - + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - + + - - - - - - + + + + + + - + - - - - - + + + + + - + - - - - - + + + + + - - - - + + + + - - + + - - - - - - - + + + + + + + - - + + - - - + + + - - + + - - - - + + + + - - - + + + - + - - - - - - - - - - + + + + + + + + + + - - - + + + - + - - - - + + + + - - + + - - - - - - + + + + + + - + - + - - - - + + + + - - + + - - - - - + + + + + - - - - - - - - - + + + + + + + + + - - + + - - + + - - - - - - - + + + + + + + - - + + - + - - - + + + - - - - - - - - - + + + + + + + + + - - - - - - + + + + + + - + - - - + + + - + - - - - + + + + - - + + - - - + + + - - - - - - + + + + + + - + - - + + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - + + + + - - - - - - - - - - - + + + + + + + + + + + - + - - + + - - - - - + + + + + - - - + + + - + - - - - - + + + + + - - + + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex index 726eb8b10..c06919af5 100644 --- a/docs/tikz/seq-primary-self-fence.tex +++ b/docs/tikz/seq-primary-self-fence.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{14.6} +\def\ybot{21.0} \begin{tikzpicture}[ >=Stealth, @@ -53,46 +53,64 @@ \draw[msg] (\px,1.6) -- (\sx,1.6) node[lbl,midway,above] {WAL stream}; -%% ── network partition: both connections lost ────────────────────────── -\node[seclbl] at (-3.4,2.8) {network partition: both connections lost}; +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; -\draw[broken] (\px,3.7) -- (13,3.7); -\node[red!70!black, font=\Large] at (13.4,3.7) {$\times$}; -\node[lbl, above, text=red!70!black] at (6.5,3.7) - {node\_active(\ldots) attempted -- no response}; +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; -\draw[broken] (\px,4.9) -- (6,4.9); -\node[red!70!black, font=\Large] at (6.4,4.9) {$\times$}; -\node[lbl, above, text=red!70!black] at (3,4.9) +%% ── network partition: primary isolated ──────────────────────────────── +\node[seclbl] at (-3.4,6.3) {network partition: primary isolated}; + +\draw[broken] (\px,7.2) -- (13,7.2); +\node[red!70!black, font=\Large] at (13.4,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,7.2) + {node\_active(reported=primary) attempted -- no response}; + +\draw[broken] (\px,8.4) -- (6,8.4); +\node[red!70!black, font=\Large] at (6.4,8.4) {$\times$}; +\node[lbl, above, text=red!70!black] at (3,8.4) {WAL stream -- connection lost}; +%% ── secondary is unaffected ───────────────────────────────────────────── +\node[seclbl] at (-3.4,9.8) {secondary is unaffected, still polling normally}; + +\draw[msg] (\sx,10.7) -- (\mx,10.7); +\node[lbl, above] at (\sx+3.3,10.7) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,11.3) -- (\sx,11.3); +\node[lbl, above] at (\mx-2.2,11.3) {goal=secondary}; + %% ── local decision on the primary ────────────────────────────────────── -\node[seclbl] at (-3.4,6.3) {primary's local decision}; +\node[seclbl] at (-3.4,12.7) {primary's local decision}; \node[note, fill=blue!8, draw=blue!40, text width=13.5cm] - at (9.5, 7.3) + at (9.5, 13.7) {\small no secondary, no monitor $\Rightarrow$ wait \texttt{network\_partition\_timeout} (20s by default), then self-fence\\ \normalfont\itshape\small (if the secondary were still reachable, the primary would just keep serving instead)}; %% ── self-fence ────────────────────────────────────────────────────────── -\node[seclbl] at (-3.4,9.3) {self-fence (timeout elapsed)}; +\node[seclbl] at (-3.4,15.7) {self-fence (timeout elapsed)}; \node[note, fill=orange!15, draw=orange!70!black, text width=8cm] - at (\px, 10.2) + at (\px, 16.6) {\small stop Postgres\\ local state $\rightarrow$ \texttt{demote\_timeout}}; -\draw[lldead] (\px,10.9) -- (\px,\ybot); +\draw[lldead] (\px,17.3) -- (\px,\ybot); \node[note, fill=orange!10, draw=orange!60, text width=13cm, font=\small] - at (9.5, 12.1) + at (9.5, 18.5) {avoids a split brain: with no secondary and no monitor reachable, the primary cannot be sure it isn't isolated, so it refuses to keep accepting writes}; \node[note, fill=gray!10, draw=gray!50, text width=13cm, font=\small] - at (9.5, 13.6) + at (9.5, 20.0) {reported to the monitor as \texttt{demote\_timeout} once connectivity is restored}; diff --git a/docs/tikz/seq-secondary-unhealthy.svg b/docs/tikz/seq-secondary-unhealthy.svg index c4c6c1c87..48613c87d 100644 --- a/docs/tikz/seq-secondary-unhealthy.svg +++ b/docs/tikz/seq-secondary-unhealthy.svg @@ -1,5 +1,5 @@ - + @@ -102,6 +102,12 @@ + + + + + + @@ -144,799 +150,1392 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - + + + + - + - + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + - + - - + + - - + + - - + + - + + + + + + + - + + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + - - + - - - - - - + + + + + + + + + + + + - - - - - - + + + + + - - - + + + + + + + + + + + + + + + + - - - - - - - + + + + + + + - - + + - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - + + + + + + + + - - + + - - - - - + + + + + - - + - - - - - + + + + - + + + - - - - - + + + + + - - - - + + + + - - + + + + + + - - - - - - - + + - - - - - - - - - - + - - - - - - - - - - - - + + + - - - - - - - - - - - + + + + + + + - - - - - - - - + + - + - - - - - - + + - - + + + + + + + + - - - + + + + + + + + + + - + + - - - - + + + + + - - + + + + + + + + + - - + + + + + - - - - - - + + - + + + - - - + + - - + + + + + + + + + + + + + - + - - - + + + + + + - - - - - - - + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - + + + + + + + + + + + - - + + + + + + + + - - - - - + - + + + + + + - + + - + + + - - - - - - - + - - - - - + + + + - - + + - - - - - + + - - + + + + + + - - - - - + - - - - - - - - - + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + - - - - - - + + + + + + - - - - - - - + + + + + + + + + + + + + + + + - - - - - - - + + - - + + + + + + + + + - - - - - - - + + + + + + + - - + + - - + + - - - + + - + + + + + + + + + + + + + - - - + - + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + + + + + + - - - - - - - - - - + + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + - - - - - - - - - - - + + - - + + + + + + + + + + + + + + + - - - - - + + - - - - - - - + + + + + + + + + + + + + - - + - - - - - - - + + + + + + - + + + + + - - - - + + + + - - + + + + + + + + + + + + + + + + + + + + + + + - + - + - - - - - - - - - - - + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + - - - - - + + + + + - - - + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-secondary-unhealthy.tex b/docs/tikz/seq-secondary-unhealthy.tex index 32fc7e541..20befec66 100644 --- a/docs/tikz/seq-secondary-unhealthy.tex +++ b/docs/tikz/seq-secondary-unhealthy.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{15.0} +\def\ybot{28.4} \begin{tikzpicture}[ >=Stealth, @@ -28,9 +28,9 @@ \tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] \tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] -\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] \tikzstyle{msg}=[->, line width=0.65pt] \tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] \tikzstyle{lbl}=[font=\small, inner sep=2pt] \tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, font=\small, align=center] @@ -54,49 +54,80 @@ \draw[rsp] (\sx,2.4) -- (\px,2.4) node[lbl,midway,above] {sync ack}; +\draw[msg] (\px,3.6) -- (\mx,3.6); +\node[lbl, above] at (\px+4.5,3.6) {node\_active(reported=primary)}; +\draw[rsp] (\mx,4.2) -- (\px,4.2); +\node[lbl, above] at (\mx-2.2,4.2) {goal=primary}; + +\draw[msg] (\sx,5.4) -- (\mx,5.4); +\node[lbl, above] at (\sx+3.3,5.4) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,6.0) -- (\sx,6.0); +\node[lbl, above] at (\mx-2.2,6.0) {goal=secondary}; + %% ── secondary goes unhealthy ──────────────────────────────────────────── -\node[seclbl] at (-3.4,3.5) {secondary goes unhealthy}; +\node[seclbl] at (-3.4,7.4) {secondary goes unhealthy}; + +\draw[broken] (\sx,8.3) -- (13,8.3); +\node[red!70!black, font=\Large] at (13.4,8.3) {$\times$}; +\node[lbl, above, text=red!70!black] at (\sx+3.4,8.3) + {node\_active(reported=secondary) attempted -- no response}; -\node[note, fill=red!12, draw=red!50, text width=9.5cm] - at (\sx, 3.9) - {\color{red!80!black} health checks / node\_active reports stop - arriving}; +\node[note, fill=red!12, draw=red!50, text width=13cm, font=\small] + at (9.5, 9.5) + {after \texttt{health\_check\_max\_retries} (2 by default), the secondary + is marked unhealthy}; %% ── fallback to async ─────────────────────────────────────────────────── -\node[seclbl] at (-3.4,5.3) {fallback to async}; +\node[seclbl] at (-3.4,11.0) {fallback to async}; -\draw[rsp] (\mx,5.6) -- (\px,5.6); -\node[lbl, above] at (\mx-2.6, 5.6) {goal: wait\_primary}; +\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] + at (9.5, 11.9) + {records goal=\texttt{catchingup} for the secondary -- no delivery + attempted, it isn't reachable right now}; -\draw[rsp] (\mx,6.4) -- (\sx,6.4) - node[lbl,midway,above] {goal: catchingup}; +\draw[msg] (\px,13.3) -- (\mx,13.3); +\node[lbl, above] at (\px+4.5,13.3) {node\_active(reported=primary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,13.9) -- (\px,13.9); +\node[lbl, above, text=pbox] at (\mx-2.2,13.9) {goal=wait\_primary}; -\node[note, fill=async, draw=gray!50, text width=13.5cm] - at (9.5, 7.4) - {\small synchronous replication disabled -- writes on the primary no - longer wait for the secondary; secondary can't be promoted while it - isn't in state \texttt{secondary}}; +\node[note, fill=async, draw=gray!50, text width=13.5cm, font=\small] + at (9.5, 15.3) + {synchronous replication disabled -- writes on the primary no longer + wait for the secondary; it can't be promoted while it isn't in state + \texttt{secondary}}; -\draw[msg, color=pbox, line width=0.8pt] (\px,8.6) -- (\sx,8.6) +\draw[msg, color=pbox, line width=0.8pt] (\px,17.0) -- (\sx,17.0) node[lbl,midway,above] {WAL stream (async)}; -%% ── secondary catches up ──────────────────────────────────────────────── -\node[seclbl] at (-3.4,10.0) {secondary catches up}; +%% ── secondary recovers, catches up ────────────────────────────────────── +\node[seclbl] at (-3.4,18.4) {secondary recovers, catches up}; + +\draw[msg] (\sx,19.3) -- (\mx,19.3); +\node[lbl, above] at (\sx+3.3,19.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,19.9) -- (\sx,19.9); +\node[lbl, above] at (\mx-2.4,19.9) {goal=catchingup}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=11cm] + at (\sx, 21.1) + {\small reconnects and catches up on replication lag}; -\draw[msg] (\sx,10.3) -- (\mx,10.3); -\node[lbl, above] at (\sx+3.2, 10.3) {report: WAL delta acceptable}; +\draw[msg] (\sx,22.3) -- (\mx,22.3); +\node[lbl, above] at (\sx+3.5,22.3) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,22.9) -- (\sx,22.9); +\node[lbl, above, text=pbox] at (\mx-2.2,22.9) {goal=secondary}; %% ── back to synchronous ───────────────────────────────────────────────── -\node[seclbl] at (-3.4,11.7) {back to synchronous}; +\node[seclbl] at (-3.4,24.3) {back to synchronous}; -\draw[rsp] (\mx,12.0) -- (\sx,12.0) - node[lbl,midway,above] {goal: secondary}; -\draw[rsp] (\mx,12.8) -- (\px,12.8) - node[lbl,midway,above] {goal: primary}; +\draw[msg] (\px,25.2) -- (\mx,25.2); +\node[lbl, above] at (\px+4.6,25.2) {node\_active(reported=wait\_primary)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,25.8) -- (\px,25.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,25.8) + {goal=primary}; \node[note, fill=green!15, draw=green!50!black, font=\small\bfseries, minimum width=8cm] - at (4.5, 13.9) {$\checkmark$~~synchronous replication restored}; + at (4.5, 27.0) {$\checkmark$~~synchronous replication restored}; %% ── actor footers ─────────────────────────────────────────────────────── \node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary}; From 9321a9d9a58f995a05ad725a5e576b1b058caf4e Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:52:30 +0200 Subject: [PATCH 7/8] Extend self-fence diagram: parallel monitor failover, then primary reconnects Answers what happens to the secondary once the primary demotes itself: in this full-isolation scenario the monitor's own health checks to the primary are failing too, on a shorter timer (~10s vs. the primary's 20s), so the monitor's own failover promotes the secondary before the primary even finishes self-fencing -- cross-referencing the "Monitor can't connect to Primary" diagram for the promotion mechanics rather than repeating them. Then shows the old primary reconnecting: it reports demote_timeout and gets back goal=demoted directly, because that goal was already recorded during the monitor's STONITH action -- a real demote_timeout -> demoted edge, so no guard logic is even needed for this path. Ties the self-fence and monitor-failover scenarios together into one coherent, consistent outcome instead of leaving them as two disconnected stories. --- docs/tikz/seq-primary-self-fence.svg | 3129 ++++++++++++++++++-------- docs/tikz/seq-primary-self-fence.tex | 70 +- 2 files changed, 2237 insertions(+), 962 deletions(-) diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg index 328e5668e..b26781465 100644 --- a/docs/tikz/seq-primary-self-fence.svg +++ b/docs/tikz/seq-primary-self-fence.svg @@ -1,1554 +1,2775 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + - + - + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + - - + + - + - - + + - - - - - - + + + + + + - - + + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - + + + + + + - - - - + + + + - - - + + + - - + + - - - - - - + + + + + + - - - - + + + + - - + + - - + + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - - - - - - - + + + + + + + + + + + + - - - + + + - - + + - - - - - - - - - - - - + + + + + + + + + + + + - - + + - - - + + + - + - + - - + + - - + + - - - - - - - - + + + + + + + + - + - - - - + + + + - - + + - - - - - - - - - - + + + + + + + + + + - + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - + + + + + + - - - - + + + + - - - + + + - - - - - - - - - + + + + + + + + + - + - - + + - - - - + + + + - - - - + + + + - + - + - + - - + + - - - - - - + + + + + + - + - - - - - - - - - - + + + + + + + + + + - - - - + + + + - - - - - - - + + + + + + + - - + + - - + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - + - - - - - - + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - + + + + - - - - + + + + - - + + - - - + + + - - - - - - - - + + + + + + + + - + - - + + - - - - - - - + + + + + + + - - + + - + - - + + - - - - - - + + + + + + - + - + - + - - - + + + - - - + + + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + - + - - + + - + - + - + - - - - - - - - - + + + + + + + + + - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - - + + + - - - + + + - - + + - + - - - - - - + + + + + + - - - + + + - + - - - - + + + + - + - + - + - - - - - - - - + + + + + + + + - - - + + + - - - - - - - + + + + + + + - - - - - + + + + + - - - - + + + + - - + + - - + + - - - - - - - + + + + + + + - - - - - + + + + + - - - + + + - - - - - - - - - - + + + + + + + + + + - - - - - - - - + + + + + + + + - - - - - - - - + + + + + + + + - + - - - - + + + + - + - - - - - - - + + + + + + + - - + + - - - + + + - - - - - + + + + + - + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - + + - - - - - - + + + + + + - + - - - - - + + + + + - + - - - - - + + + + + - - - - + + + + - - + + - - - - - - - + + + + + + + - - + + - - - + + + - - + + - - - - + + + + - - - + + + - + - - - - - - - - - - + + + + + + + + + + - - - + + + - + - - - - + + + + - - + + - - - - - - + + + + + + - + - + - - - - + + + + - - + + - - - - - + + + + + - - - - - - - - - + + + + + + + + + - - + + - - + + - - - - - - - + + + + + + + - - + + - + - - - + + + - - - - - - - - - + + + + + + + + + - - - - - - + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + + + + - + + + - - - - + + + + + + + + + + + + + - - + - - - + - - - - - - + - + - - + + + + + + + + - - - - - - - - - - - - - - + + + - - - - + + + - - - - - - - - - - - + + + + - + - - + - - - - - + - - - + + - - - - - - - + + - - - + + + - - - - - - - - - - - - - - + + - - - - - - - - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex index c06919af5..6f9585982 100644 --- a/docs/tikz/seq-primary-self-fence.tex +++ b/docs/tikz/seq-primary-self-fence.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{21.0} +\def\ybot{35.5} \begin{tikzpicture}[ >=Stealth, @@ -109,15 +109,69 @@ primary cannot be sure it isn't isolated, so it refuses to keep accepting writes}; -\node[note, fill=gray!10, draw=gray!50, text width=13cm, font=\small] - at (9.5, 20.0) - {reported to the monitor as \texttt{demote\_timeout} once connectivity is - restored}; +%% ── meanwhile: monitor's own health checks to primary also fail ──────── +\node[seclbl] at (-3.4,20.0) + {meanwhile: monitor's own health checks to primary also fail}; + +\draw[broken] (\mx,20.9) -- (6,20.9); +\node[red!70!black, font=\Large] at (5.6,20.9) {$\times$}; +\node[lbl, above, text=red!70!black] at (13,20.9) + {health check: no response (retried)}; + +\node[note, fill=gray!10, draw=gray!45, text width=14cm, font=\small] + at (9.5, 22.1) + {\texttt{health\_check\_max\_retries} (2) $\times$ + \texttt{health\_check\_period} (5s) $\approx$ 10s -- shorter than the + primary's own 20s timer, so the monitor's failover starts first}; + +%% ── monitor fails over: secondary is promoted ─────────────────────────── +\node[seclbl] at (-3.4,23.5) {monitor fails over: secondary is promoted}; + +\draw[msg] (\sx,24.4) -- (\mx,24.4); +\node[lbl, above] at (\sx+3.3,24.4) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,25.0) -- (\sx,25.0); +\node[lbl, above, text=pbox] at (\mx-2.6, 25.0) {goal=prepare\_promotion}; + +\node[note, fill=blue!8, draw=blue!40, text width=14cm, font=\small] + at (9.5, 26.2) + {same promotion sequence as the ``Monitor can't connect to Primary'' + diagram: \texttt{prepare\_promotion} $\rightarrow$ + \texttt{stop\_replication} $\rightarrow$ \texttt{wait\_primary}}; + +\draw[msg] (\sx,27.4) -- (\mx,27.4); +\node[lbl, above] at (\sx+3.7,27.4) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,28.0) -- (\sx,28.0); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,28.0) + {goal=wait\_primary}; + +\node[note, fill=gray!12, draw=gray!55, text width=14cm, font=\small] + at (9.5, 29.2) + {records the old primary's goal: \texttt{draining} $\rightarrow$ + \texttt{demoted} -- no delivery attempted, it still isn't reachable}; + +%% ── primary reconnects ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,30.6) {primary reconnects: finds an already-resolved state}; + +\draw[msg] (\px,31.5) -- (\mx,31.5); +\node[lbl, above] at (\px+4.8,31.5) {node\_active(reported=demote\_timeout)}; +\draw[rsp] (\mx,32.1) -- (\px,32.1); +\node[lbl, above] at (\mx-2.2,32.1) {goal=demoted}; + +\node[note, fill=gray!10, draw=gray!50, text width=14cm, font=\small] + at (9.5, 33.3) + {the goal was already \texttt{demoted} from the STONITH action above -- + a real \texttt{demote\_timeout} $\rightarrow$ \texttt{demoted} edge, so + the primary complies immediately. No extra guard logic is even needed + for this case}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (14, 34.5) {$\checkmark$~~cluster converged: old primary stays fenced}; %% ── actor footers ─────────────────────────────────────────────────────── -\node[actor, fill=orange!25, draw=orange!70!black, text=orange!30!black] - at (\px, \ybot+0.35) {\small Primary (self-fenced)}; -\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary}; +\node[actor, fill=gray!25, draw=gray!55, text=gray!30!black] + at (\px, \ybot+0.35) {\small Primary (demoted)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; \node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; \end{tikzpicture} From 9449c3e290b23155919e9909e7d12c91686a60f6 Mon Sep 17 00:00:00 2001 From: Dimitri Fontaine Date: Sat, 25 Jul 2026 17:59:28 +0200 Subject: [PATCH 8/8] Fix promotion-sequence gap and add full reintegration to self-fence diagram The compressed promotion sequence jumped straight from goal=prepare_promotion to reported=stop_replication, skipping the candidate's own confirmation of reaching prepare_promotion (and the goal=stop_replication response to it). Added that missing exchange back, matching the sibling diagram, and replaced the vague "same sequence as..." note with a short explanation of what stop_replication actually gates on. Also extends the story past "old primary stays fenced": once the new primary has confirmed wait_primary, demoted -> catchingup is a valid FSM edge (group_state_machine.c:909), so the diagram now shows the old primary reconnecting further, rewinding/rebasing, and rejoining as a secondary -- full role swap, not just a fenced dead end. The primary's lifeline switches back from "dead" to normal once it successfully reconnects. --- docs/tikz/seq-primary-self-fence.svg | 3965 +++++++++++++++----------- docs/tikz/seq-primary-self-fence.tex | 75 +- 2 files changed, 2333 insertions(+), 1707 deletions(-) diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg index b26781465..df3e65f5b 100644 --- a/docs/tikz/seq-primary-self-fence.svg +++ b/docs/tikz/seq-primary-self-fence.svg @@ -1,5 +1,5 @@ - + @@ -44,12 +44,6 @@ - - - - - - @@ -137,6 +131,9 @@ + + + @@ -246,40 +243,34 @@ - + - + - + - + - + - + - + - + - + - - - - - - - + @@ -354,10 +345,19 @@ - + - + + + + + + + + + + @@ -450,2326 +450,2921 @@ + + + + + + - + - - - - - + + + + + - - + + - + - - - - - - - + + + + + + + - - + + - + - - - - - - + + + + + + - + - - - + + + - - - - - - + + + + + + - - - - - + + + + + - - + + - + - - + + - - - - - - + + + + + + - - + + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - + + + + + + - - - - + + + + - - - + + + - - + + - - - - - - + + + + + + - - - - + + + + - - + + - - + + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - - - - - - - + + + + + + + + + + + + - - - + + + - - + + - - - - - - - - - - - - + + + + + + + + + + + + - - + + - - - + + + - + - + - - + + - - + + - - - - - - - - + + + + + + + + - + - - - - + + + + - - + + - - - - - - - - - - + + + + + + + + + + - + - - + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - - - - - - + + + + + + - - - - + + + + - - - + + + - - - - - - - - - + + + + + + + + + - + - - + + - - - - + + + + - - - - + + + + - + - + - + - - + + - - - - - - + + + + + + - + - - - - - - - - - - + + + + + + + + + + - - - - + + + + - - - - - - - + + + + + + + - - + + - - + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - + - - - - - - + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - + + + + - - - - + + + + - - + + - - - + + + - - - - - - - - + + + + + + + + - + - - + + - - - - - - - + + + + + + + - - + + - + - - + + - - - - - - + + + + + + - + - + - + - - - + + + - - - + + + - + - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + - + - - + + - + - + - + - - - - - - - - - + + + + + + + + + - - - - + + + + - - - - - - - - - - + + + + + + + + + + - - - + + + - - - + + + - - + + - + - - - - - - + + + + + + - - - + + + - + - - - - + + + + - + - + - + - - - - - - - - + + + + + + + + - - - + + + - - - - - - - + + + + + + + - - - - - + + + + + - - - - + + + + - - + + - - + + - - - - - - - + + + + + + + - - - - - + + + + + - - - + + + - - - - - - - - - - + + + + + + + + + + - - - - - - - - + + + + + + + + - - - - - - - - + + + + + + + + - + - - - - + + + + - + - - - - - - - + + + + + + + - - + + - - - + + + - - - - - + + + + + - + - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - + + + - - - - - - + + + + + + - + - - - - - + + + + + - + - - - - - + + + + + - - - - + + + + - - + + - - - - - - - + + + + + + + - - + + - - - + + + - - + + - - - - + + + + - - - + + + - + - - - - - - - - - - + + + + + + + + + + - - - + + + - + - - - - + + + + - - + + - - - - - - + + + + + + - + - + - - - - + + + + - - + + - - - - - + + + + + - - - - - - - - - + + + + + + + + + - - + + - - + + - - - - - - - + + + + + + + - - + + - + - - - + + + - - - - - - - - - + + + + + + + + + - - - - - - + + + + + + - - - - - - - - - - + + + + + + + + + + - - - - - - + + + + + + - - - + + + - + - - + + - - - - - - + + + + + + - - - - - - + + + + + + - - + + - + - - - - + + + + - - + + - - - - + + + + - - - - + + + + - + - + - - - - - - + + + + + + - - - - - - + + + + + + - - + + - - - - + + + + - - - - + + + + - - - - - - - - - + + + + + + + + + - + - - - - - - - - + + + + + + + + - - - + + + - - - - - - - - - - - - - + + + + + + + + + + + + + - + - + - + - + - - - - - - - - + + + + + + + + - - - + + + - - - + + + - - - - + + + + - + - + - + - - + + - + - - + + - + - + - - - + + + - - - - + + + + - - - - + + + + - - - + + + - + - - - - + + + + - - - - + + + + - + - - + + - - + + - + - - - - - - + + + + + + - - + + - - - + + + - - - - - - + + + + + + - - - + + + - - - - - - - - + + + + + + + + - - - + + + - - - + + + - - - - + + + + - - - - - - + + + + + + - + - - - - - + + + + + - - - - - + + + + + - - - - - - - + + + + + + + - - + + - - + + - + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + - - - - - - - - - - - - - + + - + - - - - - - - - - - - - + + + + + - - - + - - - - - - - - - + + + + + + - - - - - + + + + - - - - - + + + + + + + + + + - - - - - - - - - + + - - - - - + + + + - + + + + + - - - - - - - - + + + + - - - - - - - - + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + - - - - - - - + - + + + + + + + + - - - - - + + + + + + + + + + + - - - - - - - + + - - - - - - - - - + + + + + + + + + + + + + - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + - - - - - + + + + + - - - - + + + + - - + + - + - - - - + + + + - - - + + + - - - + + + - - - + + + - + - - - - + + + + - - - - + + + + - - - - - + + + + + - - - - - - - - + + + + + + + + - + - - - - - - - + + + + + + + - + - - + + - - - - - - - - + + + + + + + + - - - - - - - - - - + + + + + + + + + + - - + + - - - - - + + + + + - - - - - + + + + + - - - - - - - - - + + + + + + + + + - + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - - - - - - - - - - - - - - + + + + - - - - + - - - - + + - + + + + + + + - - + + + + + + + - - - - - - - + + + + - - - - - - - + + + - - - - + + + + + + + - - - + + + + + + - - - - - - - + + - - - - - - + + + - - + - - - + - + + + + - + + + + + + + + + + + + + + - - - - + - - - - - - - - - - - - - - + + + + + + + - + + + + + - - - - - - - + + - - - - - + + + - - + - - - + + + + - + + - - - - + + + + - - + + + + + - - - - + + + + + + + + + + + - - - - - + - - - - - - - - - - - + + - + + + + + - - + + + - - - - - + + - - - + + + + + - - + + - - - - - + + + + - - + + + + + + - - - - + + - - - - - - + - - + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + - - - - + - - + + - - - - - - - + - - - - - - - - - - + + + - - - + + + - + - - - - + + + + - - + + - - - + + + - - + + + + + + + + - - - - - - + + + + + + + + - - - - - - - + + - - - + + + + + + + + + + + + - - - - - - - - - - + + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + - - + + - + - - - - - - + + + + + + - + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex index 6f9585982..808af1a9b 100644 --- a/docs/tikz/seq-primary-self-fence.tex +++ b/docs/tikz/seq-primary-self-fence.tex @@ -16,7 +16,7 @@ \def\px{0} \def\sx{9} \def\mx{19} -\def\ybot{35.5} +\def\ybot{44.9} \begin{tikzpicture}[ >=Stealth, @@ -101,7 +101,8 @@ at (\px, 16.6) {\small stop Postgres\\ local state $\rightarrow$ \texttt{demote\_timeout}}; -\draw[lldead] (\px,17.3) -- (\px,\ybot); +\draw[lldead] (\px,17.3) -- (\px,33.3); +\draw[ll] (\px,33.3) -- (\px,\ybot); \node[note, fill=orange!10, draw=orange!60, text width=13cm, font=\small] at (9.5, 18.5) @@ -132,46 +133,76 @@ \draw[rsp, color=pbox, line width=0.8pt] (\mx,25.0) -- (\sx,25.0); \node[lbl, above, text=pbox] at (\mx-2.6, 25.0) {goal=prepare\_promotion}; +\draw[msg] (\sx,26.2) -- (\mx,26.2); +\node[lbl, above] at (\sx+3.9,26.2) {node\_active(reported=prepare\_promotion)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,26.8) -- (\sx,26.8); +\node[lbl, above, text=pbox] at (\mx-2.4, 26.8) {goal=stop\_replication}; + \node[note, fill=blue!8, draw=blue!40, text width=14cm, font=\small] - at (9.5, 26.2) - {same promotion sequence as the ``Monitor can't connect to Primary'' - diagram: \texttt{prepare\_promotion} $\rightarrow$ - \texttt{stop\_replication} $\rightarrow$ \texttt{wait\_primary}}; - -\draw[msg] (\sx,27.4) -- (\mx,27.4); -\node[lbl, above] at (\sx+3.7,27.4) {node\_active(reported=stop\_replication)}; -\draw[rsp, color=pbox, line width=1pt] (\mx,28.0) -- (\sx,28.0); -\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,28.0) + at (9.5, 28.0) + {\texttt{stop\_replication} is a safety gate: the candidate waits until + it's confident the old primary has actually stopped, before promotion + completes}; + +\draw[msg] (\sx,29.2) -- (\mx,29.2); +\node[lbl, above] at (\sx+3.7,29.2) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,29.8) -- (\sx,29.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,29.8) {goal=wait\_primary}; \node[note, fill=gray!12, draw=gray!55, text width=14cm, font=\small] - at (9.5, 29.2) + at (9.5, 31.0) {records the old primary's goal: \texttt{draining} $\rightarrow$ \texttt{demoted} -- no delivery attempted, it still isn't reachable}; %% ── primary reconnects ────────────────────────────────────────────────── -\node[seclbl] at (-3.4,30.6) {primary reconnects: finds an already-resolved state}; +\node[seclbl] at (-3.4,32.4) {primary reconnects: finds an already-resolved state}; -\draw[msg] (\px,31.5) -- (\mx,31.5); -\node[lbl, above] at (\px+4.8,31.5) {node\_active(reported=demote\_timeout)}; -\draw[rsp] (\mx,32.1) -- (\px,32.1); -\node[lbl, above] at (\mx-2.2,32.1) {goal=demoted}; +\draw[msg] (\px,33.3) -- (\mx,33.3); +\node[lbl, above] at (\px+4.8,33.3) {node\_active(reported=demote\_timeout)}; +\draw[rsp] (\mx,33.9) -- (\px,33.9); +\node[lbl, above] at (\mx-2.2,33.9) {goal=demoted}; \node[note, fill=gray!10, draw=gray!50, text width=14cm, font=\small] - at (9.5, 33.3) + at (9.5, 35.1) {the goal was already \texttt{demoted} from the STONITH action above -- a real \texttt{demote\_timeout} $\rightarrow$ \texttt{demoted} edge, so the primary complies immediately. No extra guard logic is even needed for this case}; +%% ── later: the demoted primary rejoins as a secondary ─────────────────── +\node[seclbl] at (-3.4,36.5) {later: the demoted primary rejoins as a secondary}; + +\node[note, fill=gray!10, draw=gray!45, text width=14cm, font=\small] + at (9.5, 37.6) + {by now the new primary has confirmed reaching \texttt{wait\_primary} on + its own next poll (not shown), so \texttt{demoted} $\rightarrow$ + \texttt{catchingup} is a valid FSM edge + (\texttt{group\_state\_machine.c:909})}; + +\draw[msg] (\px,39.0) -- (\mx,39.0); +\node[lbl, above] at (\px+4.4,39.0) {node\_active(reported=demoted)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,39.6) -- (\px,39.6); +\node[lbl, above, text=pbox] at (\mx-2.2,39.6) {goal=catchingup}; + +\node[note, fill=gray!12, draw=gray!55, text width=10cm] + at (\px, 40.7) + {\small \texttt{pg\_rewind} or \texttt{pg\_basebackup}, reconnects as a + hot standby to the new primary}; + +\draw[msg] (\px,42.1) -- (\mx,42.1); +\node[lbl, above] at (\px+4.4,42.1) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,42.7) -- (\px,42.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,42.7) + {goal=secondary}; + \node[note, fill=green!15, draw=green!50!black, font=\small\bfseries, minimum width=8cm] - at (14, 34.5) {$\checkmark$~~cluster converged: old primary stays fenced}; + at (9.5, 43.9) {$\checkmark$~~roles swapped, cluster healthy again}; %% ── actor footers ─────────────────────────────────────────────────────── -\node[actor, fill=gray!25, draw=gray!55, text=gray!30!black] - at (\px, \ybot+0.35) {\small Primary (demoted)}; -\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=sbox, text=stxt] at (\px, \ybot+0.35) {Secondary}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {Primary}; \node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; \end{tikzpicture}