diff --git a/docs/architecture.rst b/docs/architecture.rst index 5651cc421..0ee01f345 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -235,6 +235,11 @@ cannot connect *and* it hasn't reported its state through `node_active` for a while. This allows, for example, PostgreSQL to be restarted without causing a health check failure. +.. figure:: ./tikz/seq-monitoring-protocol.svg + :alt: Sequence diagram of the node_active polling and health check protocols + + The two protocols between a node's keeper and the monitor + Synchronous vs. asynchronous replication ---------------------------------------- diff --git a/docs/failover-state-machine.rst b/docs/failover-state-machine.rst index 4d25dac99..acd2f3941 100644 --- a/docs/failover-state-machine.rst +++ b/docs/failover-state-machine.rst @@ -19,6 +19,11 @@ Example of state transitions in a new cluster A good way to get acquainted with the states is by examining the transitions of a cluster from birth to high availability. +.. figure:: ./tikz/seq-birth-of-a-cluster.svg + :alt: Sequence diagram of two nodes and the monitor from registration to primary/secondary + + From two freshly registered nodes to a converged primary/secondary pair + After starting a monitor and running keeper init for the first data node ("node A"), the monitor registers the state of that node as "init" with a goal state of "single." The init state means the monitor knows nothing diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index c960aa023..747f9a894 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -114,6 +114,11 @@ PostgreSQL service: The network\_partition\_timeout can be setup in the keeper's configuration and defaults to 20s. + .. figure:: ./tikz/seq-primary-self-fence.svg + :alt: Sequence diagram of a primary self-fencing to demote_timeout after losing contact with both the monitor and the secondary + + The primary self-fences rather than risk a split brain + - Monitor can't connect to Primary Once all the retries have been done and the timeouts are elapsed, then @@ -126,6 +131,11 @@ PostgreSQL service: delta to resorb (30s by default) then the secondary can be promoted with uncertainty about the data durability in the group. + .. figure:: ./tikz/seq-monitor-cant-reach-primary.svg + :alt: Sequence diagram of the monitor failing over to the secondary after losing contact with the primary + + The monitor promotes the secondary and fences the old primary + - Monitor can't connect to Secondary As soon as the secondary is considered unhealthy then the monitor @@ -140,6 +150,11 @@ PostgreSQL service: assigned the SECONDARY state, and the replication will be switched back to synchronous. + .. figure:: ./tikz/seq-secondary-unhealthy.svg + :alt: Sequence diagram of the fallback to asynchronous replication and back + + Falling back to asynchronous replication and resynchronizing + Failure handling and network partition detection ------------------------------------------------ @@ -170,6 +185,10 @@ expected to shut down after at least 30 and at most 60 seconds. To factor in worst-case scenarios, the monitor waits for 90 seconds before promoting the secondary to become the new primary. +.. figure:: ./tikz/seq-asymmetric-partition.svg + :alt: Sequence diagram of an asymmetric partition where the primary can still reach the secondary but not the monitor + + Asymmetric partition: the monitor's 90s safety wait before promoting See also -------- diff --git a/docs/index.rst b/docs/index.rst index b1e315db7..5f7f8440e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,14 +20,16 @@ __ https://github.com/hapostgres/pg_auto_failover .. note:: - The development of pg_auto_failover has been driven by Citus Data, since - then a team at Microsoft. The Citus Data team at Microsoft generously - maintains the pg_auto_failover Open Source Software so that its users may - continue using it in production. + The development of pg_auto_failover was originally driven by Citus Data, + and then a team at Microsoft. The project is now orphaned in terms of + financial support: maintenance and development happen on volunteers' + free time. For enhancements, improvements, and new features, consider contributing - to the project. Pull Requests are reviewed as part of the offered - maintenance. + to the project. Pull Requests are reviewed as time allows. + + For professional support or to sponsor ongoing maintenance, see + `oss.theartofpostgresql.com `_. .. note:: diff --git a/docs/tikz/seq-asymmetric-partition.svg b/docs/tikz/seq-asymmetric-partition.svg new file mode 100644 index 000000000..e497612ca --- /dev/null +++ b/docs/tikz/seq-asymmetric-partition.svg @@ -0,0 +1,1474 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-asymmetric-partition.tex b/docs/tikz/seq-asymmetric-partition.tex new file mode 100644 index 000000000..1ae030f16 --- /dev/null +++ b/docs/tikz/seq-asymmetric-partition.tex @@ -0,0 +1,129 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{22.2} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; + +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; + +%% ── asymmetric partition ──────────────────────────────────────────────── +\node[seclbl] at (-3.4,6.3) {asymmetric partition: only the Primary-Monitor link breaks}; + +\draw[broken] (\px,7.2) -- (13,7.2); +\node[red!70!black, font=\Large] at (13.4,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,7.2) + {node\_active(reported=primary) attempted -- no response}; + +\node[note, fill=gray!10, draw=gray!45, text width=13.5cm, font=\small] + at (9.5, 8.4) + {the primary keeps replicating normally to the secondary throughout -- + only its link to the monitor is down}; + +%% ── monitor initiates failover ───────────────────────────────────────── +\node[seclbl] at (-3.4,9.8) {secondary is still reachable, monitor initiates failover}; + +\draw[msg] (\sx,10.7) -- (\mx,10.7); +\node[lbl, above] at (\sx+3.3,10.7) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,11.3) -- (\sx,11.3); +\node[lbl, above, text=pbox] at (\mx-2.6, 11.3) {goal=stop\_replication}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=8cm] + at (\sx, 12.3) {\small disconnects from Primary}; + +%% ── primary loses its last connection ────────────────────────────────── +\node[seclbl] at (-3.4,13.7) {primary loses its last connection}; + +\node[note, fill=orange!15, draw=orange!70!black, text width=12.5cm] + at (4.5, 14.6) + {\small with the secondary gone too, the primary self-fences the same + way as a full partition (previous diagram): expected to stop within + 30--60s}; + +\draw[lldead] (\px,15.4) -- (\px,\ybot); + +%% ── 90s safety wait ───────────────────────────────────────────────────── +\node[seclbl] at (-3.4,16.9) {90s safety wait}; + +\node[note, fill=blue!8, draw=blue!40, text width=13.5cm] + at (\mx-1, 17.8) + {\small worst case is 60s, so the monitor waits a full 90s before + promoting -- a margin against edge cases}; + +\draw[gray!50, line width=0.5pt, ->] + (\mx+2.6, 16.5) .. controls (\mx+4.3,16.8) and (\mx+4.3,18.8) .. (\mx+2.6,19.1) + node[right, font=\small\itshape, text=gray!70] {waiting\ldots}; + +%% ── promotion ──────────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,20.2) {promotion}; + +\draw[msg] (\sx,21.1) -- (\mx,21.1); +\node[lbl, above] at (\sx+3.7,21.1) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,21.7) -- (\sx,21.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,21.7) + {goal=wait\_primary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=orange!25, draw=orange!70!black, text=orange!30!black] + at (\px, \ybot+0.35) {\small Primary (self-fenced)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-birth-of-a-cluster.svg b/docs/tikz/seq-birth-of-a-cluster.svg new file mode 100644 index 000000000..09f53e78e --- /dev/null +++ b/docs/tikz/seq-birth-of-a-cluster.svg @@ -0,0 +1,1303 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-birth-of-a-cluster.tex b/docs/tikz/seq-birth-of-a-cluster.tex new file mode 100644 index 000000000..475b44201 --- /dev/null +++ b/docs/tikz/seq-birth-of-a-cluster.tex @@ -0,0 +1,133 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% A=0 B=9 M=19 +\def\ax{0} +\def\bx{9} +\def\mx{19} +\def\ybot{26.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{actorn}=[actor, fill=gray!15, text=stxt, draw=gray!50] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actorn] (Ah) at (\ax,0) {Node A}; +\node[actorn] (Bh) at (\bx,0) {Node B}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\ax,0.35) -- (\ax,\ybot); +\draw[ll] (\bx,0.35) -- (\bx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── node A registers, converges alone ────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {node A registers, converges alone}; + +\draw[msg] (\ax,1.6) -- (\mx,1.6) + node[lbl,midway,above] {register\_node(\ldots)}; +\draw[rsp] (\mx,2.2) -- (\ax,2.2); +\node[lbl, above] at (\mx-2.6,2.2) {state=init, goal=single}; + +\draw[msg] (\ax,3.3) -- (\mx,3.3); +\node[lbl, above] at (\ax+4.2,3.3) {node\_active(reported=single)}; +\draw[rsp] (\mx,3.9) -- (\ax,3.9); +\node[lbl, above] at (\mx-2.2,3.9) {goal=single}; + +%% ── node B joins ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,5.1) {node B joins}; + +\draw[msg] (\bx,5.4) -- (\mx,5.4) + node[lbl,midway,above] {register\_node(\ldots)}; +\draw[rsp] (\mx,6.0) -- (\bx,6.0); +\node[lbl, above] at (\mx-2.9,6.0) {state=init, goal=wait\_standby}; + +%% ── A's next poll reveals the new goal ────────────────────────────────── +\node[seclbl] at (-3.4,7.4) {A's next poll reveals the new goal}; + +\draw[msg] (\ax,8.3) -- (\mx,8.3); +\node[lbl, above] at (\ax+4.2,8.3) {node\_active(reported=single)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,8.9) -- (\ax,8.9); +\node[lbl, above, text=pbox] at (\mx-2.4,8.9) {goal=wait\_primary}; + +\node[note, fill=pbox!12, draw=pbox!60, text width=6.2cm] + at (\ax, 10.0) {\small allows B in \texttt{pg\_hba.conf}}; + +%% ── A converges to wait_primary ───────────────────────────────────────── +\node[seclbl] at (-3.4,11.4) {A converges to wait\_primary}; + +\draw[msg] (\ax,12.3) -- (\mx,12.3); +\node[lbl, above] at (\ax+4.6,12.3) {node\_active(reported=wait\_primary)}; +\draw[rsp] (\mx,12.9) -- (\ax,12.9); +\node[lbl, above] at (\mx-2.2,12.9) {goal=wait\_primary}; + +%% ── B's next poll reveals catchingup ──────────────────────────────────── +\node[seclbl] at (-3.4,14.3) {B's next poll reveals catchingup}; + +\draw[msg] (\bx,15.2) -- (\mx,15.2); +\node[lbl, above] at (\bx+3.7,15.2) {node\_active(reported=wait\_standby)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,15.8) -- (\bx,15.8); +\node[lbl, above, text=pbox] at (\mx-2.4,15.8) {goal=catchingup}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=7.5cm] + at (\bx, 16.9) {\small \texttt{pg\_basebackup}, hot standby}; + +%% ── B converges; A promoted in the same pass ──────────────────────────── +\node[seclbl] at (-3.4,18.3) {B converges; monitor promotes A in the same pass}; + +\draw[msg] (\bx,19.2) -- (\mx,19.2); +\node[lbl, above] at (\bx+3.8,19.2) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,19.8) -- (\bx,19.8); +\node[lbl, above, text=pbox] at (\mx-2.2,19.8) {goal=secondary}; + +%% ── A's next poll reveals primary ─────────────────────────────────────── +\node[seclbl] at (-3.4,21.2) {A's next poll reveals primary}; + +\draw[msg] (\ax,22.1) -- (\mx,22.1); +\node[lbl, above] at (\ax+4.6,22.1) {node\_active(reported=wait\_primary)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,22.7) -- (\ax,22.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,22.7) + {goal=primary}; + +%% ── convergence ───────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,24.1) {convergence}; + +\draw[msg, color=pbox, line width=0.8pt] (\ax,25.0) -- (\bx,25.0) + node[lbl,midway,above] {WAL stream}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (9.5, 25.9) {$\checkmark$~~A primary, B secondary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\ax, \ybot+0.35) {Primary A}; +\node[actor, fill=sbox, text=stxt] at (\bx, \ybot+0.35) {Secondary B}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-monitor-cant-reach-primary.svg b/docs/tikz/seq-monitor-cant-reach-primary.svg new file mode 100644 index 000000000..0aa16d301 --- /dev/null +++ b/docs/tikz/seq-monitor-cant-reach-primary.svg @@ -0,0 +1,2451 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-monitor-cant-reach-primary.tex b/docs/tikz/seq-monitor-cant-reach-primary.tex new file mode 100644 index 000000000..f07f000c8 --- /dev/null +++ b/docs/tikz/seq-monitor-cant-reach-primary.tex @@ -0,0 +1,150 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{28.8} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state: both nodes reachable ───────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; + +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; + +%% ── primary goes silent ───────────────────────────────────────────────── +\node[seclbl] at (-3.4,6.3) + {primary goes silent: health checks and its own node\_active both fail}; + +\draw[broken] (\mx,7.2) -- (6,7.2); +\node[red!70!black, font=\Large] at (5.6,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (13,7.2) + {health check: no response (retried)}; + +\draw[broken] (\px,8.3) -- (13,8.3); +\node[red!70!black, font=\Large] at (13.4,8.3) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,8.3) + {node\_active(reported=primary) attempted -- no response}; + +\node[note, fill=red!10, draw=red!50, text width=13cm, font=\small] + at (9.5, 9.6) + {after \texttt{health\_check\_max\_retries} (2 by default), the primary is + marked unhealthy -- meanwhile it cannot even poll for a goal state, so + it has no way to learn anything changed}; + +%% ── secondary is still reachable ──────────────────────────────────────── +\node[seclbl] at (-3.4,11.0) {secondary is unaffected, still polling normally}; + +\draw[msg] (\sx,11.9) -- (\mx,11.9); +\node[lbl, above] at (\sx+3.3,11.9) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,12.5) -- (\sx,12.5); +\node[lbl, above] at (\mx-2.2,12.5) {goal=secondary}; + +%% ── failover begins ───────────────────────────────────────────────────── +\node[seclbl] at (-3.4,13.9) {failover begins: promote the secondary}; + +\draw[msg] (\sx,14.8) -- (\mx,14.8); +\node[lbl, above] at (\sx+3.3,14.8) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,15.4) -- (\sx,15.4); +\node[lbl, above, text=pbox] at (\mx-2.6,15.4) {goal=prepare\_promotion}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=11cm] + at (\sx, 16.4) + {\small catches up on any last WAL still available from the primary}; + +\draw[msg] (\sx,17.4) -- (\mx,17.4); +\node[lbl, above] at (\sx+3.5,17.4) {node\_active(reported=prepare\_promotion)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,18.0) -- (\sx,18.0); +\node[lbl, above, text=pbox] at (\mx-2.6,18.0) {goal=stop\_replication}; + +\node[note, fill=blue!8, draw=blue!40, text width=15cm, font=\small] + at (9.5, 19.2) + {if the WAL delta hadn't resorbed within 30s (default), the secondary + would have been promoted anyway -- with uncertainty about data + durability. Remaining hops (\texttt{stop\_replication} $\rightarrow$ + \texttt{wait\_primary}) are the same as any promotion; see the + Failover State Machine page}; + +%% ── old primary fenced ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,21.4) {old primary fenced (STONITH) -- monitor-side only}; + +\node[note, fill=gray!12, draw=gray!55, text width=15cm, font=\small] + at (9.5, 22.8) + {records goal \texttt{draining} $\rightarrow$ \texttt{demoted} for the + primary in its own table -- no delivery attempted, since the primary + hasn't polled since this section started. If it's in fact still alive, + it learns this (and demotes itself) the next time it can reach the + monitor -- or it may have already self-fenced independently (see the + previous diagram)}; + +%% ── promotion complete ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,25.3) {promotion complete}; + +\draw[msg] (\sx,26.2) -- (\mx,26.2); +\node[lbl, above] at (\sx+3.7,26.2) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,26.8) -- (\sx,26.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,26.8) + {goal=wait\_primary}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (14, 27.9) {$\checkmark$~~Secondary is the new primary}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=gray!30, draw=gray!60, text=gray!30!black] + at (\px, \ybot+0.35) {\small Primary (fenced, unaware)}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {New Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-monitoring-protocol.svg b/docs/tikz/seq-monitoring-protocol.svg new file mode 100644 index 000000000..98a5208e5 --- /dev/null +++ b/docs/tikz/seq-monitoring-protocol.svg @@ -0,0 +1,703 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-monitoring-protocol.tex b/docs/tikz/seq-monitoring-protocol.tex new file mode 100644 index 000000000..e8155f64e --- /dev/null +++ b/docs/tikz/seq-monitoring-protocol.tex @@ -0,0 +1,84 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% N=0 M=12 +\def\nx{0} +\def\mx{12} +\def\ybot{9.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.2cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=sbox, text=stxt] (Nh) at (\nx,0) {Node}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\nx,0.35) -- (\nx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── node_active polling (node-initiated) ────────────────────────────────── +\node[seclbl] at (-4.4,1.3) {state reporting}; + +\draw[msg] (\nx,1.6) -- (\mx,1.6); +\node[lbl, above] at (\nx+3.4, 1.6) {node\_active(reported\_state, lsn, \ldots)}; + +\node[note, fill=mbox!25, draw=mbox!70!black, text width=6.5cm] + at (\mx, 2.6) + {\small store reported state\\ compute goal state}; + +\draw[rsp] (\mx,3.6) -- (\nx,3.6) + node[lbl,midway,above] {goal\_state}; + +\draw[gray!50, line width=0.5pt, ->] + (\nx-0.5, 4.0) .. controls (\nx-2.2,4.3) and (\nx-2.2,5.6) .. (\nx-0.5,5.9) + node[left, font=\small\itshape, text=gray!70] {repeat\ldots}; + +%% ── health check (monitor-initiated) ────────────────────────────────────── +\node[seclbl] at (-4.4,6.6) {health check}; + +\draw[msg, color=mbox!70!black] (\mx,6.9) -- (\nx,6.9); +\node[lbl, above, text=mbox!70!black] at (\mx-3.4, 6.9) + {connect (\texttt{pg\_isready}-equivalent)}; + +\draw[rsp] (\nx,7.8) -- (\mx,7.8) + node[lbl,midway,above] {ready}; + +%% ── unhealthy note ────────────────────────────────────────────────────── +\node[note, fill=red!10, draw=red!40, text width=12.5cm, font=\small] + at (6, \ybot-0.3) + {\color{red!80!black}\bfseries unhealthy \normalfont only when \emph{both} + signals are missing: the monitor cannot connect, \emph{and} no + node\_active report has arrived for a while}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=sbox, text=stxt] at (\nx, \ybot+1.1) {Node}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+1.1) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-primary-self-fence.svg b/docs/tikz/seq-primary-self-fence.svg new file mode 100644 index 000000000..df3e65f5b --- /dev/null +++ b/docs/tikz/seq-primary-self-fence.svg @@ -0,0 +1,3370 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-primary-self-fence.tex b/docs/tikz/seq-primary-self-fence.tex new file mode 100644 index 000000000..808af1a9b --- /dev/null +++ b/docs/tikz/seq-primary-self-fence.tex @@ -0,0 +1,209 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{44.9} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{lldead}=[densely dashed, orange!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ──────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; + +\draw[msg] (\px,2.7) -- (\mx,2.7); +\node[lbl, above] at (\px+4.5,2.7) {node\_active(reported=primary)}; +\draw[rsp] (\mx,3.3) -- (\px,3.3); +\node[lbl, above] at (\mx-2.2,3.3) {goal=primary}; + +\draw[msg] (\sx,4.3) -- (\mx,4.3); +\node[lbl, above] at (\sx+3.3,4.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,4.9) -- (\sx,4.9); +\node[lbl, above] at (\mx-2.2,4.9) {goal=secondary}; + +%% ── network partition: primary isolated ──────────────────────────────── +\node[seclbl] at (-3.4,6.3) {network partition: primary isolated}; + +\draw[broken] (\px,7.2) -- (13,7.2); +\node[red!70!black, font=\Large] at (13.4,7.2) {$\times$}; +\node[lbl, above, text=red!70!black] at (6.5,7.2) + {node\_active(reported=primary) attempted -- no response}; + +\draw[broken] (\px,8.4) -- (6,8.4); +\node[red!70!black, font=\Large] at (6.4,8.4) {$\times$}; +\node[lbl, above, text=red!70!black] at (3,8.4) + {WAL stream -- connection lost}; + +%% ── secondary is unaffected ───────────────────────────────────────────── +\node[seclbl] at (-3.4,9.8) {secondary is unaffected, still polling normally}; + +\draw[msg] (\sx,10.7) -- (\mx,10.7); +\node[lbl, above] at (\sx+3.3,10.7) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,11.3) -- (\sx,11.3); +\node[lbl, above] at (\mx-2.2,11.3) {goal=secondary}; + +%% ── local decision on the primary ────────────────────────────────────── +\node[seclbl] at (-3.4,12.7) {primary's local decision}; + +\node[note, fill=blue!8, draw=blue!40, text width=13.5cm] + at (9.5, 13.7) + {\small no secondary, no monitor $\Rightarrow$ wait + \texttt{network\_partition\_timeout} (20s by default), then self-fence\\ + \normalfont\itshape\small (if the secondary were still reachable, the + primary would just keep serving instead)}; + +%% ── self-fence ────────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,15.7) {self-fence (timeout elapsed)}; + +\node[note, fill=orange!15, draw=orange!70!black, text width=8cm] + at (\px, 16.6) + {\small stop Postgres\\ local state $\rightarrow$ \texttt{demote\_timeout}}; + +\draw[lldead] (\px,17.3) -- (\px,33.3); +\draw[ll] (\px,33.3) -- (\px,\ybot); + +\node[note, fill=orange!10, draw=orange!60, text width=13cm, font=\small] + at (9.5, 18.5) + {avoids a split brain: with no secondary and no monitor reachable, the + primary cannot be sure it isn't isolated, so it refuses to keep + accepting writes}; + +%% ── meanwhile: monitor's own health checks to primary also fail ──────── +\node[seclbl] at (-3.4,20.0) + {meanwhile: monitor's own health checks to primary also fail}; + +\draw[broken] (\mx,20.9) -- (6,20.9); +\node[red!70!black, font=\Large] at (5.6,20.9) {$\times$}; +\node[lbl, above, text=red!70!black] at (13,20.9) + {health check: no response (retried)}; + +\node[note, fill=gray!10, draw=gray!45, text width=14cm, font=\small] + at (9.5, 22.1) + {\texttt{health\_check\_max\_retries} (2) $\times$ + \texttt{health\_check\_period} (5s) $\approx$ 10s -- shorter than the + primary's own 20s timer, so the monitor's failover starts first}; + +%% ── monitor fails over: secondary is promoted ─────────────────────────── +\node[seclbl] at (-3.4,23.5) {monitor fails over: secondary is promoted}; + +\draw[msg] (\sx,24.4) -- (\mx,24.4); +\node[lbl, above] at (\sx+3.3,24.4) {node\_active(reported=secondary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,25.0) -- (\sx,25.0); +\node[lbl, above, text=pbox] at (\mx-2.6, 25.0) {goal=prepare\_promotion}; + +\draw[msg] (\sx,26.2) -- (\mx,26.2); +\node[lbl, above] at (\sx+3.9,26.2) {node\_active(reported=prepare\_promotion)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,26.8) -- (\sx,26.8); +\node[lbl, above, text=pbox] at (\mx-2.4, 26.8) {goal=stop\_replication}; + +\node[note, fill=blue!8, draw=blue!40, text width=14cm, font=\small] + at (9.5, 28.0) + {\texttt{stop\_replication} is a safety gate: the candidate waits until + it's confident the old primary has actually stopped, before promotion + completes}; + +\draw[msg] (\sx,29.2) -- (\mx,29.2); +\node[lbl, above] at (\sx+3.7,29.2) {node\_active(reported=stop\_replication)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,29.8) -- (\sx,29.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.4,29.8) + {goal=wait\_primary}; + +\node[note, fill=gray!12, draw=gray!55, text width=14cm, font=\small] + at (9.5, 31.0) + {records the old primary's goal: \texttt{draining} $\rightarrow$ + \texttt{demoted} -- no delivery attempted, it still isn't reachable}; + +%% ── primary reconnects ────────────────────────────────────────────────── +\node[seclbl] at (-3.4,32.4) {primary reconnects: finds an already-resolved state}; + +\draw[msg] (\px,33.3) -- (\mx,33.3); +\node[lbl, above] at (\px+4.8,33.3) {node\_active(reported=demote\_timeout)}; +\draw[rsp] (\mx,33.9) -- (\px,33.9); +\node[lbl, above] at (\mx-2.2,33.9) {goal=demoted}; + +\node[note, fill=gray!10, draw=gray!50, text width=14cm, font=\small] + at (9.5, 35.1) + {the goal was already \texttt{demoted} from the STONITH action above -- + a real \texttt{demote\_timeout} $\rightarrow$ \texttt{demoted} edge, so + the primary complies immediately. No extra guard logic is even needed + for this case}; + +%% ── later: the demoted primary rejoins as a secondary ─────────────────── +\node[seclbl] at (-3.4,36.5) {later: the demoted primary rejoins as a secondary}; + +\node[note, fill=gray!10, draw=gray!45, text width=14cm, font=\small] + at (9.5, 37.6) + {by now the new primary has confirmed reaching \texttt{wait\_primary} on + its own next poll (not shown), so \texttt{demoted} $\rightarrow$ + \texttt{catchingup} is a valid FSM edge + (\texttt{group\_state\_machine.c:909})}; + +\draw[msg] (\px,39.0) -- (\mx,39.0); +\node[lbl, above] at (\px+4.4,39.0) {node\_active(reported=demoted)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,39.6) -- (\px,39.6); +\node[lbl, above, text=pbox] at (\mx-2.2,39.6) {goal=catchingup}; + +\node[note, fill=gray!12, draw=gray!55, text width=10cm] + at (\px, 40.7) + {\small \texttt{pg\_rewind} or \texttt{pg\_basebackup}, reconnects as a + hot standby to the new primary}; + +\draw[msg] (\px,42.1) -- (\mx,42.1); +\node[lbl, above] at (\px+4.4,42.1) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,42.7) -- (\px,42.7); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,42.7) + {goal=secondary}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (9.5, 43.9) {$\checkmark$~~roles swapped, cluster healthy again}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=sbox, text=stxt] at (\px, \ybot+0.35) {Secondary}; +\node[actor, fill=pbox, text=ptxt] at (\sx, \ybot+0.35) {Primary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/docs/tikz/seq-secondary-unhealthy.svg b/docs/tikz/seq-secondary-unhealthy.svg new file mode 100644 index 000000000..48613c87d --- /dev/null +++ b/docs/tikz/seq-secondary-unhealthy.svg @@ -0,0 +1,1541 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-secondary-unhealthy.tex b/docs/tikz/seq-secondary-unhealthy.tex new file mode 100644 index 000000000..20befec66 --- /dev/null +++ b/docs/tikz/seq-secondary-unhealthy.tex @@ -0,0 +1,138 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{28.4} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Secondary}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state: synchronous ────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state (synchronous)}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; +\draw[rsp] (\sx,2.4) -- (\px,2.4) + node[lbl,midway,above] {sync ack}; + +\draw[msg] (\px,3.6) -- (\mx,3.6); +\node[lbl, above] at (\px+4.5,3.6) {node\_active(reported=primary)}; +\draw[rsp] (\mx,4.2) -- (\px,4.2); +\node[lbl, above] at (\mx-2.2,4.2) {goal=primary}; + +\draw[msg] (\sx,5.4) -- (\mx,5.4); +\node[lbl, above] at (\sx+3.3,5.4) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,6.0) -- (\sx,6.0); +\node[lbl, above] at (\mx-2.2,6.0) {goal=secondary}; + +%% ── secondary goes unhealthy ──────────────────────────────────────────── +\node[seclbl] at (-3.4,7.4) {secondary goes unhealthy}; + +\draw[broken] (\sx,8.3) -- (13,8.3); +\node[red!70!black, font=\Large] at (13.4,8.3) {$\times$}; +\node[lbl, above, text=red!70!black] at (\sx+3.4,8.3) + {node\_active(reported=secondary) attempted -- no response}; + +\node[note, fill=red!12, draw=red!50, text width=13cm, font=\small] + at (9.5, 9.5) + {after \texttt{health\_check\_max\_retries} (2 by default), the secondary + is marked unhealthy}; + +%% ── fallback to async ─────────────────────────────────────────────────── +\node[seclbl] at (-3.4,11.0) {fallback to async}; + +\node[note, fill=gray!12, draw=gray!55, text width=13.5cm, font=\small] + at (9.5, 11.9) + {records goal=\texttt{catchingup} for the secondary -- no delivery + attempted, it isn't reachable right now}; + +\draw[msg] (\px,13.3) -- (\mx,13.3); +\node[lbl, above] at (\px+4.5,13.3) {node\_active(reported=primary)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,13.9) -- (\px,13.9); +\node[lbl, above, text=pbox] at (\mx-2.2,13.9) {goal=wait\_primary}; + +\node[note, fill=async, draw=gray!50, text width=13.5cm, font=\small] + at (9.5, 15.3) + {synchronous replication disabled -- writes on the primary no longer + wait for the secondary; it can't be promoted while it isn't in state + \texttt{secondary}}; + +\draw[msg, color=pbox, line width=0.8pt] (\px,17.0) -- (\sx,17.0) + node[lbl,midway,above] {WAL stream (async)}; + +%% ── secondary recovers, catches up ────────────────────────────────────── +\node[seclbl] at (-3.4,18.4) {secondary recovers, catches up}; + +\draw[msg] (\sx,19.3) -- (\mx,19.3); +\node[lbl, above] at (\sx+3.3,19.3) {node\_active(reported=secondary)}; +\draw[rsp] (\mx,19.9) -- (\sx,19.9); +\node[lbl, above] at (\mx-2.4,19.9) {goal=catchingup}; + +\node[note, fill=sbox!25, draw=sbox!70!black, text width=11cm] + at (\sx, 21.1) + {\small reconnects and catches up on replication lag}; + +\draw[msg] (\sx,22.3) -- (\mx,22.3); +\node[lbl, above] at (\sx+3.5,22.3) {node\_active(reported=catchingup)}; +\draw[rsp, color=pbox, line width=0.8pt] (\mx,22.9) -- (\sx,22.9); +\node[lbl, above, text=pbox] at (\mx-2.2,22.9) {goal=secondary}; + +%% ── back to synchronous ───────────────────────────────────────────────── +\node[seclbl] at (-3.4,24.3) {back to synchronous}; + +\draw[msg] (\px,25.2) -- (\mx,25.2); +\node[lbl, above] at (\px+4.6,25.2) {node\_active(reported=wait\_primary)}; +\draw[rsp, color=pbox, line width=1pt] (\mx,25.8) -- (\px,25.8); +\node[lbl, above, text=pbox, font=\small\bfseries] at (\mx-2.2,25.8) + {goal=primary}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=8cm] + at (4.5, 27.0) {$\checkmark$~~synchronous replication restored}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary}; +\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Secondary}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document}