diff --git a/docs/architecture.rst b/docs/architecture.rst index 0ee01f345..0df322890 100644 --- a/docs/architecture.rst +++ b/docs/architecture.rst @@ -304,3 +304,21 @@ goal state from the monitor. If the failed node was a primary and was demoted, it will learn this from the monitor. Once the node reports, it is allowed to come back as a standby by running ``pg_rewind``. If it is too far behind, the node performs a new ``pg_basebackup``. + +Whether ``pg_rewind`` is even the right call isn't decided from a bare +timeline-number comparison. The node walks the upstream's real timeline +history to distinguish "still catching up" from "genuinely diverged onto a +dead branch" — the latter can happen when a standby was written to, or +promoted, outside of ``pg_autoctl``'s control. Every node also periodically +publishes its own known timeline lineage to the monitor +(``pgautofailover.node_timeline_history``), so that a failover election can +reason about forks centrally and exclude a diverged candidate rather than +block on it. The monitor uses this same lineage data proactively too: a +node currently acting as a healthy secondary is checked against the +group's reference lineage on every report, and gets pushed into +``catchingup`` — where the rewind actually happens — as soon as a genuine +divergence is found, rather than waiting for the node to go through some +other, unrelated transition first. See :ref:`timeline_forks` for the full +scenario, and :ref:`pg_autoctl_show_timeline` / +:ref:`pg_autoctl_accept_timeline` for the commands that surface and, when +needed, resolve this. diff --git a/docs/failover-state-machine.rst b/docs/failover-state-machine.rst index a0bd2d604..37806b851 100644 --- a/docs/failover-state-machine.rst +++ b/docs/failover-state-machine.rst @@ -155,6 +155,19 @@ The standby node keeper runs pg_basebackup, connecting to the primary's hostname and port. The keeper then edits recovery.conf and starts PostgreSQL in hot standby node. +Before doing so — and also when a healthy secondary is reconnecting to a +(possibly new) primary rather than bootstrapping from scratch — the keeper +compares its own timeline history against the upstream's. A node that is +simply behind rewinds cleanly onto the upstream once it catches up; a node +whose local WAL has genuinely diverged onto a dead branch (for example, one +that was promoted directly at the Postgres level outside of +``pg_autoctl``'s control) can never resolve that divergence through +ordinary streaming replication, and Postgres itself will refuse the +reconnect. In that case the keeper runs ``pg_rewind`` to discard the +diverged WAL and rejoin the real lineage, falling back to a fresh +``pg_basebackup`` if ``pg_rewind`` itself cannot connect. See +:ref:`timeline_forks` for the full scenario. + Secondary ^^^^^^^^^ @@ -162,6 +175,21 @@ A node with this state is acting as a hot standby for the primary, and is up to date with the WAL log there. In particular, it is within 16MB or 1 WAL segment of the primary. +Streaming replication alone can look perfectly healthy even when the +secondary is on a genuinely forked branch of history — the WAL receiver +just keeps reporting progress against whatever local timeline the standby +happens to be on. To catch this without waiting for an incidental +transition, the monitor applies the same ``FilterNodesByTimelineAncestry()`` +check used during an election (see ``Report_LSN`` below) to every node +currently in the secondary state, on each of its regular reports. As soon +as a secondary's reported timeline is found not to be an ancestor of the +group's reference lineage, the monitor assigns it the ``catchingup`` goal +state right away — typically within about a second, on that very report — +rather than waiting for a health-check cycle or an operator-forced resync +to reveal the problem. See :ref:`timeline_forks` for the full scenario and +:ref:`pg_autoctl_accept_timeline` for pinning the reference lineage when +auto-detection can't tell which branch is real. + Maintenance ^^^^^^^^^^^ @@ -274,6 +302,18 @@ data loss, the election can be unblocked with :ref:`pg_autoctl_perform_failover` ``--allow-data-loss``. See :ref:`perform_failover_allow_data_loss` for details. +Once quorum standbys have reported their LSN, candidates are further +filtered by timeline ancestry (``FilterNodesByTimelineAncestry()``): a node +whose reported timeline is not the group's reference lineage — the +:ref:`pg_autoctl_accept_timeline`-pinned timeline if one has been set, +otherwise the highest reported timeline that nothing else in the group +disagrees with — is excluded from candidacy rather than counted as missing. +This is what lets the election proceed among the remaining, non-diverged +candidates instead of blocking on a node that can never actually win. Use +:ref:`pg_autoctl_show_timeline` to see the group's known timeline history +and each node's status against it. See :ref:`timeline_forks` for the +failure scenario this guards against. + Fast_forward ^^^^^^^^^^^^ diff --git a/docs/faq.rst b/docs/faq.rst index 16e60550f..4e1dcd3d5 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -1,3 +1,5 @@ +.. _faq: + Frequently Asked Questions ========================== @@ -111,6 +113,43 @@ See :ref:`perform_failover_allow_data_loss` and the ``pgautofailover.guard_data_loss`` GUC in :ref:`configuration` for a full explanation. +My standby won't rejoin — Postgres logs a timeline mismatch, what should I do? +------------------------------------------------------------------------------ + +This happens when a standby's local WAL has genuinely diverged from the +rest of the group — for example, it was promoted or written to directly at +the Postgres level, outside of ``pg_autoctl``'s control, during an incident. +Postgres itself refuses the reconnect and logs something like ``requested +timeline N is not a child of this server's history``. + +**This is usually automatic.** ``pg_autoctl`` walks the upstream's real +timeline history to tell a standby that's simply behind apart from one +that's genuinely diverged, and the monitor checks every currently-healthy +secondary's ancestry on each of its regular reports — not just at the next +incidental transition. As soon as a genuine divergence is found, the node +is pushed to ``catchingup`` and rewound with ``pg_rewind`` (or a fresh +``pg_basebackup`` if ``pg_rewind`` can't connect), typically within about a +second, with no health-check cycle or maintenance toggle to wait for. See +:ref:`timeline_forks` for the full scenario. + +**If it doesn't resolve on its own,** check what the monitor knows:: + + pg_autoctl show timeline + +A node flagged ``FORK: diverges from the reference timeline, pg_rewind +required`` needs an operator decision. This most often means auto-detection +couldn't tell which branch was real — for instance in a two-node formation, +where there's no sibling node to disagree with the diverged one. Confirm +which timeline is actually ground truth from other evidence (which node was +manually promoted, ``pg_controldata`` output, etc.), then pin it:: + + pg_autoctl accept timeline --tli --reason "..." + +The diverged node then rewinds and rejoins automatically. See +:ref:`pg_autoctl_show_timeline`, :ref:`pg_autoctl_accept_timeline`, and +:ref:`resolving_timeline_fork` for the full command reference and +walkthrough. + The state of the system is blocked, what should I do? ----------------------------------------------------- diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst index dc51c50c7..ac9da4f87 100644 --- a/docs/fault-tolerance.rst +++ b/docs/fault-tolerance.rst @@ -197,6 +197,64 @@ PostgreSQL service: Falling back to asynchronous replication and resynchronizing +.. _timeline_forks: + +Timeline Forks +-------------- + +A different kind of failure doesn't come from a node being unreachable, but +from a node whose local WAL has genuinely diverged from the rest of the +group. This happens when a standby is written to, or promoted, outside of +``pg_autoctl``'s control — a manual intervention during an incident, a +monitoring bug elsewhere, a previous split-brain — and generates local WAL +that no other node in the group has, on a branch of history the primary +never took. + +Ordinary streaming replication can never resolve this: it's not lag, it's +divergence. Postgres itself refuses the reconnect (``requested timeline N +is not a child of this server's history``). + +pg_auto_failover detects and resolves this automatically in the common +case: before trusting a bare timeline-number comparison, a standby +reconnecting to a (possibly new) primary walks the primary's real +timeline history to tell "still catching up" apart from "diverged onto a +dead branch," and runs ``pg_rewind`` in either direction as needed (falling +back to a fresh ``pg_basebackup`` if ``pg_rewind`` itself can't connect). +See the ``Catchingup`` section of :ref:`failover_state_machine` for where +this check runs. + +The monitor doesn't wait for that reconnect to notice, either. It applies +the same ancestry check to every node currently reported as a healthy +secondary, and as soon as one is found not to be an ancestor of the group's +reference lineage, it is pushed to ``catchingup`` right away — typically +within about a second, on that node's very next report — rather than +waiting for an incidental health-check cycle or an operator-forced resync +to reveal the problem (see the ``Report_LSN`` section of +:ref:`failover_state_machine` for where the election applies this same +ancestry filter, and this section's own diagram below for the monitor-side +push). + +The reference lineage itself is either pinned explicitly, or auto-detected +as the branch containing the highest reported timeline. Auto-detection only +excludes a candidate when a genuinely *competing* branch is reported by +someone else — two nodes each diverging from the same point onto two +different timelines — in which case the loser is caught and rewound with no +operator action at all. It doesn't help when there's no sibling to disagree +with the diverged node: in a two-node formation, or whenever every +surviving node happens to already be on the same diverged branch, the fork +reads as clean and an operator decision is needed. Use +:ref:`pg_autoctl_show_timeline` to see the group's known timeline history +and each node's status against it, and :ref:`pg_autoctl_accept_timeline` to +pin the correct lineage explicitly — once pinned, the same immediate, +automatic push applies. See :ref:`resolving_timeline_fork` for the full +walkthrough. + +.. figure:: ./tikz/seq-timeline-fork.svg + :alt: Sequence diagram of a standby forking out-of-band and being detected and rewound back onto the real lineage + + A standby forks out-of-band; once the mismatch is visible to the + monitor, it is pushed to catchingup and rewound within about a second + Failure handling and network partition detection ------------------------------------------------ diff --git a/docs/operations.rst b/docs/operations.rst index f619e63db..928f3ef62 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -379,6 +379,93 @@ unplanned failover are all handled by the monitor, rather than the client side command line, at the client level the two command ``pg_autoctl perform failover`` and ``pg_autoctl perform switchover`` are synonyms, or aliases. +.. _resolving_timeline_fork: + +Resolving a detected timeline fork +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A standby whose local WAL has genuinely diverged from the rest of the group +— for example, one that was promoted or written to directly at the +Postgres level, outside of ``pg_autoctl``'s control, during an incident — +cannot resolve that divergence through ordinary streaming replication. See +:ref:`timeline_forks` for the full scenario and how pg_auto_failover +detects and, in most cases, automatically resolves this. + +Start by checking what the monitor currently knows:: + + $ pg_autoctl show timeline + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/3000130 | ok, on accepted lineage + node2 | 2 | 2 | 0/3016330 | ok, on accepted lineage + +The first table is every timeline the group has ever seen; the second is +each node's current position and whether it's on the group's reference +lineage. Here both nodes read as clean — this is the known two-node +limitation: auto-detection only has something to compare against when a +sibling node disagrees, and with only ``node1`` (never advanced past +timeline 1) and ``node2`` (the fork) in the group, ``node2``'s fork looks +like a normal, legitimate promotion. + +Having confirmed from other evidence which timeline is really ground truth +— which node was manually promoted, ``pg_controldata`` output, application +logs — pin it explicitly:: + + $ pg_autoctl accept timeline --tli 1 --formation default \ + --reason "node2 self-promoted out of band during a network partition" + Timeline 1 accepted as ground truth for formation "default" group 0. The election will now only consider nodes on that lineage; other nodes need pg_rewind before rejoining. + +``node2`` is now unambiguously flagged:: + + $ pg_autoctl show timeline + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/30599B8 | ok, on accepted lineage + node2 | 2 | 2 | 0/3016330 | FORK: diverges from the reference timeline, pg_rewind required + + One or more nodes have diverged from the reference timeline (see FORK above). + See `pg_autoctl accept timeline --help` to resolve. + +No further command is needed to make ``node2`` rewind. The pin doesn't just +change how the flagged node is evaluated at the next election — the monitor +re-checks every currently-``secondary`` node's ancestry against the +freshly-pinned lineage right away, so ``node2`` is pushed to ``catchingup`` +within about a second of the ``accept timeline`` command above, with no +health-check cycle or maintenance toggle needed to trigger it. + +``node2`` goes ``secondary`` → ``catchingup`` → ``secondary``, running +``pg_rewind`` onto timeline 1 along the way (or, if ``pg_rewind`` itself +can't connect, a fresh ``pg_basebackup`` — both pre-existing recovery +paths) — usually so quickly that polling ``pg_autoctl show state`` a second +or two apart never even catches the intermediate ``catchingup`` state. The +fork clears on its own, with no need to run an "accept" or "resolve" +command a second time, and no need to cycle the node through maintenance to +force anything:: + + $ pg_autoctl show timeline + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/70000F8 | ok, on accepted lineage + node2 | 2 | 1 | 0/70000F8 | ok, on accepted lineage + +See :ref:`pg_autoctl_show_timeline` and :ref:`pg_autoctl_accept_timeline` +for the full command reference. + Current state, last events -------------------------- @@ -458,6 +545,10 @@ possible to try ``pg_autoctl create`` again. pg_auto_failover will review its pr progress and repeat idempotent operations (``create database``, ``create extension`` etc), gracefully handling errors. +If a standby won't rejoin and Postgres logs a timeline mismatch, see +:ref:`resolving_timeline_fork` and the related FAQ entry on the same +topic in :ref:`faq`. + .. _container-and-kubernetes-deployments: Container and Kubernetes Deployments diff --git a/docs/ref/manual.rst b/docs/ref/manual.rst index f7be581fa..340a54234 100644 --- a/docs/ref/manual.rst +++ b/docs/ref/manual.rst @@ -20,6 +20,7 @@ have their own manual page. pg_autoctl_get pg_autoctl_set pg_autoctl_perform + pg_autoctl_accept pg_autoctl_node pg_autoctl_inspect pg_autoctl_manual diff --git a/docs/ref/pg_autoctl.rst b/docs/ref/pg_autoctl.rst index c1e94d22b..86b143f48 100644 --- a/docs/ref/pg_autoctl.rst +++ b/docs/ref/pg_autoctl.rst @@ -20,6 +20,7 @@ pg_autoctl provides the following commands:: + get Get a pg_auto_failover node, or formation setting + set Set a pg_auto_failover node, or formation setting + perform Perform an action orchestrated by the monitor + + accept Accept an operator decision orchestrated by the monitor activate Activate a Citus worker from the Citus coordinator run Run the pg_autoctl service (monitor or keeper) stop signal the pg_autoctl service for it to stop @@ -51,6 +52,7 @@ pg_autoctl provides the following commands:: state Prints monitor's state of nodes in a given formation and group settings Print replication settings for a formation from the monitor standby-names Prints synchronous_standby_names for a given group + timeline Show the timeline history known to the monitor for a group, and each node's position against it file List pg_autoctl internal files (config, state, pid) systemd Print systemd service file for this node @@ -97,6 +99,9 @@ pg_autoctl provides the following commands:: switchover Perform a switchover for given formation and group promotion Perform a failover that promotes a target node + pg_autoctl accept + timeline Accept a timeline as the ground truth after a detected fork + Description ----------- diff --git a/docs/ref/pg_autoctl_accept.rst b/docs/ref/pg_autoctl_accept.rst new file mode 100644 index 000000000..c3b338363 --- /dev/null +++ b/docs/ref/pg_autoctl_accept.rst @@ -0,0 +1,11 @@ +.. _pg_autoctl_accept: + +pg_autoctl accept +================= + +pg_autoctl accept - Accept an operator decision orchestrated by the monitor + +.. toctree:: + :maxdepth: 1 + + pg_autoctl_accept_timeline diff --git a/docs/ref/pg_autoctl_accept_timeline.rst b/docs/ref/pg_autoctl_accept_timeline.rst new file mode 100644 index 000000000..ceae99b8e --- /dev/null +++ b/docs/ref/pg_autoctl_accept_timeline.rst @@ -0,0 +1,154 @@ +.. _pg_autoctl_accept_timeline: + +pg_autoctl accept timeline +========================== + +pg_autoctl accept timeline - Accept a timeline as the ground truth after a detected fork + +Synopsis +-------- + +This command pins a timeline as the group's ground-truth lineage on the +pg_auto_failover monitor:: + + usage: pg_autoctl accept timeline [ --pgdata --formation --group ] --tli + + --pgdata path to data directory + --formation formation to target, defaults to 'default' + --group group to target, defaults to 0 + --tli timeline to accept, as shown by `pg_autoctl show timeline` + --reason free-text note explaining the decision + +Description +----------- + +pg_auto_failover normally detects and resolves timeline forks on its own: a +standby whose local WAL diverges from the group's reference lineage is +excluded from candidacy and pushed to ``catchingup`` — where it is rewound +onto the reference lineage with ``pg_rewind`` — as soon as the mismatch is +visible to the monitor, without waiting for a health-check cycle or a +maintenance toggle (see :ref:`timeline_forks` for the full scenario, and the +``Report_LSN`` and ``Catchingup`` sections of :ref:`failover_state_machine` +for how the election and the automatic recovery path use this). + +Auto-detection compares each node's known timeline history against every +other node's, and only reaches a confident answer when there is a sibling to +compare against. In a two-node formation, or when every surviving node +happens to already be on the same diverged branch, there is nothing left to +disagree with — the fork stays invisible to the automatic check even though +one of the branches is not real ground truth. ``pg_autoctl accept timeline`` +is the operator override for that case: it pins which timeline is ground +truth, and the election's ancestry filter (``FilterNodesByTimelineAncestry``) +uses the pinned value instead of trying to auto-detect it. Nodes not on the +accepted lineage need a ``pg_rewind`` (done automatically once they +reconnect) before they can rejoin. + +Use :ref:`pg_autoctl_show_timeline` first to see the group's known timeline +history and each node's current status against it. + +The pin is automatically marked resolved once a primary is promoted on the +accepted lineage — there is no ``pg_autoctl resolve timeline`` command to +run afterwards. + +Options +------- + +--pgdata + + Location of the Postgres node being managed locally. Defaults to the + environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor + from anywhere, rather than the monitor URI used by a local Postgres node + managed with ``pg_autoctl``. + +--formation + + Formation to target for the operation. Defaults to ``default``. + +--group + + Postgres group to target for the operation. Defaults to ``0``, only Citus + formations may have more than one group. + +--tli + + The timeline id to accept as ground truth, as shown in the ``TLI`` column + of ``pg_autoctl show timeline``. Mandatory. The monitor refuses to pin a + timeline that no node in the group has ever reported. + +--reason + + A free-text note recording why this timeline was accepted. Stored + alongside the pin as a permanent audit record; purely informational. + +Environment +----------- + +PGDATA + + Postgres directory location. Can be used instead of the ``--pgdata`` + option. + +PG_AUTOCTL_MONITOR + + Postgres URI to connect to the monitor node, can be used instead of the + ``--monitor`` option. + +PGHOST, PGPORT, PGDATABASE, PGUSER, PGCONNECT_TIMEOUT, ... + + See the `Postgres docs about Environment Variables`__ for details. + + __ https://www.postgresql.org/docs/current/libpq-envars.html + +XDG_CONFIG_HOME + + The pg_autoctl command stores its configuration files in the standard + place XDG_CONFIG_HOME. See the `XDG Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +XDG_DATA_HOME + + The pg_autoctl command stores its internal states files in the standard + place XDG_DATA_HOME, which defaults to ``~/.local/share``. See the `XDG + Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +Examples +-------- + +In this two-node example, ``node2`` was network-partitioned, promoted +directly at the Postgres level, and given a few local-only writes — a +genuine fork onto timeline 2. Unpinned, ``pg_autoctl show timeline`` reads +both nodes as clean, because there's no sibling node to disagree with +``node2``'s fork (see :ref:`pg_autoctl_show_timeline` for the full example +this continues from):: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/3000130 | ok, on accepted lineage + node2 | 2 | 2 | 0/3016330 | ok, on accepted lineage + +That's exactly the case ``pg_autoctl accept timeline`` is for: pinning +which timeline is really ground truth, confirmed from other evidence (here, +knowing which node was manually promoted). This succeeds and immediately +affects the next election:: + + $ pg_autoctl accept timeline --tli 1 --formation default \ + --reason "node2 self-promoted out of band during a network partition; tli 2 confirmed false via pg_controldata" + 07:50:35 569 INFO Targetting group 0 in formation "default" + Timeline 1 accepted as ground truth for formation "default" group 0. The election will now only consider nodes on that lineage; other nodes need pg_rewind before rejoining. + +From this point on, ``node2`` is excluded from candidacy until it rewinds +onto timeline 1. It doesn't have to reconnect or go through maintenance for +that to happen: the monitor re-checks every currently-``secondary`` node's +ancestry against the newly-pinned lineage right away, so ``node2`` is pushed +to ``catchingup`` — and rewound with ``pg_rewind`` — within about a second of +this command, with no further operator action (see the recovery example in +:ref:`pg_autoctl_show_timeline`). diff --git a/docs/ref/pg_autoctl_show.rst b/docs/ref/pg_autoctl_show.rst index 7ff5fef27..896b94342 100644 --- a/docs/ref/pg_autoctl_show.rst +++ b/docs/ref/pg_autoctl_show.rst @@ -13,5 +13,6 @@ pg_autoctl show - Show pg_auto_failover information pg_autoctl_show_state pg_autoctl_show_settings pg_autoctl_show_standby_names + pg_autoctl_show_timeline pg_autoctl_show_file pg_autoctl_show_systemd diff --git a/docs/ref/pg_autoctl_show_timeline.rst b/docs/ref/pg_autoctl_show_timeline.rst new file mode 100644 index 000000000..06583c9ba --- /dev/null +++ b/docs/ref/pg_autoctl_show_timeline.rst @@ -0,0 +1,270 @@ +.. _pg_autoctl_show_timeline: + +pg_autoctl show timeline +======================== + +pg_autoctl show timeline - Show the timeline history known to the monitor for a group, and each node's position against it + +Synopsis +-------- + +This command prints the group's known timeline history and each node's +current position against it, as computed by the monitor:: + + usage: pg_autoctl show timeline [ --pgdata ] --formation --group + + --pgdata path to data directory + --monitor pg_auto_failover Monitor Postgres URL + --formation formation to query, defaults to 'default' + --group group to query formation, defaults to 0 + +Description +----------- + +Every node periodically publishes its own known timeline history (each +timeline it has ever been on, its parent timeline, and the LSN at which it +switched) to the monitor. ``pg_autoctl show timeline`` prints two tables +built from that data: + +- A **timeline history** table: every ``(tli, parent tli, switchpoint LSN)`` + triple known to the group, one row per timeline, in ascending order. +- A **per-node status** table: each node's currently reported timeline and + LSN, and whether that timeline is on the group's reference lineage — the + accepted timeline if one has been pinned with + :ref:`pg_autoctl_accept_timeline`, otherwise the highest reported timeline + that nothing else in the group disagrees with. A node not on the reference + lineage is flagged ``FORK: diverges from the reference timeline, pg_rewind + required``. + +See :ref:`timeline_forks` for the failure scenario this detects, and the +``Report_LSN`` section of :ref:`failover_state_machine` for how the +election uses this same ancestry information to exclude a diverged +candidate. + +.. important:: + + Auto-detection only has something to compare against when at least two + nodes disagree. In a two-node formation — or whenever every surviving + node happens to already be on the same diverged branch — a fork can read + as clean, because it's the only lineage anyone is reporting. Use + :ref:`pg_autoctl_accept_timeline` to pin the correct lineage explicitly + when you know, from other evidence, that auto-detection got it wrong. + +Options +------- + +--pgdata + + Location of the Postgres node being managed locally. Defaults to the + environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor + from anywhere, rather than the monitor URI used by a local Postgres node + managed with ``pg_autoctl``. + +--monitor + + Postgres URI used to connect to the monitor. Must use the ``autoctl_node`` + username and target the ``pg_auto_failover`` database name. It is possible + to show the Postgres URI from the monitor node using the command + :ref:`pg_autoctl_show_uri`. + + Defaults to the value of the environment variable ``PG_AUTOCTL_MONITOR``. + +--formation + + Show the timeline history for the given formation. Defaults to the + ``default`` formation. + +--group + + Show the timeline history for the given group in the given formation. + Defaults to group ``0``. + +Environment +----------- + +PGDATA + + Postgres directory location. Can be used instead of the ``--pgdata`` + option. + +PG_AUTOCTL_MONITOR + + Postgres URI to connect to the monitor node, can be used instead of the + ``--monitor`` option. + +XDG_CONFIG_HOME + + The pg_autoctl command stores its configuration files in the standard + place XDG_CONFIG_HOME. See the `XDG Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +XDG_DATA_HOME + + The pg_autoctl command stores its internal states files in the standard + place XDG_DATA_HOME, which defaults to ``~/.local/share``. See the `XDG + Base Directory Specification`__. + + __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +Examples +-------- + +In this two-node example, ``node2`` was network-partitioned, promoted +directly at the Postgres level (bypassing ``pg_autoctl``), and given a few +local-only writes — a genuine fork onto timeline 2. Unpinned, ``node2``'s +fork reads as clean: it's the only lineage being reported, and nothing +disagrees with it (the two-node limitation described above):: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/3000130 | ok, on accepted lineage + node2 | 2 | 2 | 0/3016330 | ok, on accepted lineage + +After confirming, from other evidence (here, knowing which node was +manually promoted), that timeline 1 is the real lineage, pin it with +:ref:`pg_autoctl_accept_timeline`. ``node2`` is now unambiguously flagged:: + + $ pg_autoctl accept timeline --tli 1 --formation default \ + --reason "node2 self-promoted out of band during a network partition" + Timeline 1 accepted as ground truth for formation "default" group 0. The election will now only consider nodes on that lineage; other nodes need pg_rewind before rejoining. + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/30599B8 | ok, on accepted lineage + node2 | 2 | 2 | 0/3016330 | FORK: diverges from the reference timeline, pg_rewind required + + One or more nodes have diverged from the reference timeline (see FORK above). + See `pg_autoctl accept timeline --help` to resolve. + +No resync needs to be forced: the monitor re-checks every currently-secondary +node's ancestry against the freshly-pinned lineage right away, so ``node2`` +is pushed to ``catchingup`` — and rewound onto timeline 1 with ``pg_rewind`` +— within about a second of the ``accept timeline`` command above. It rejoins +on the accepted lineage, and the fork clears on its own — no maintenance +cycle, no further operator action, and no need to run an "accept" or +"resolve" command a second time:: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/3000130 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/70000F8 | ok, on accepted lineage + node2 | 2 | 1 | 0/70000F8 | ok, on accepted lineage + +See :ref:`resolving_timeline_fork` for the full walkthrough, including how +to drive the resync. + +More nodes doesn't fix the blind spot by itself +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It's tempting to assume that a third node closes the gap above — surely, +with a sibling around to disagree, auto-detection catches the fork? A +normal three-node formation's ``show timeline`` looks like this:: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/5000060 | ok, on accepted lineage + node2 | 2 | 1 | 0/5000060 | ok, on accepted lineage + node3 | 3 | 1 | 0/5000060 | ok, on accepted lineage + +The application keeps writing to the primary the whole time below — a +fork developing on one standby elsewhere in the formation is not a reason +for the main system to pause traffic, and an example where node1/node2 +sit idle while only node3 does anything would be misleading about what +this actually looks like in production. Fork ``node3`` the same way as +before (network-partition it, promote it directly at the Postgres level, +give it a couple of local-only writes) while ``node1`` keeps taking +ordinary application writes and replicating them to ``node2`` throughout. +Unpinned, this **still reads as clean**:: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/5016BC0 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/5042910 | ok, on accepted lineage + node2 | 2 | 1 | 0/5042910 | ok, on accepted lineage + node3 | 3 | 2 | 0/502D758 | ok, on accepted lineage + +Notice ``node1``/``node2`` are well ahead of ``node3`` in LSN, on their own +unbroken timeline 1 — that's the application's own ordinary traffic having +kept flowing the entire time, completely unrelated to node3's fork. The +auto-detection heuristic (no operator pin) is "the reference lineage is +whichever branch contains the highest reported timeline" — and it only +*excludes* a candidate when a genuinely **competing** branch is reported by +someone else: two nodes each diverging from the same point onto two +different timelines. ``node1`` and ``node2`` aren't competing with +``node3`` here, they're simply on a different, ongoing timeline, and +timeline 1 really is ``node3``'s own recorded parent. Structurally that's +indistinguishable from ``node3`` having been legitimately promoted past two +ordinary, honestly lagging standbys. A third node only helps when it +*also* reports a divergent history from the same switchpoint; a sibling +that just keeps working on its own lineage doesn't contest anything. + +:ref:`pg_autoctl_accept_timeline` is still the way out, exactly as in the +two-node case. The application's writes to node1 don't stop for this +either:: + + $ pg_autoctl accept timeline --tli 1 --formation default --reason "node3 self-promoted out of band during a network partition" + Timeline 1 accepted as ground truth for formation "default" group 0. The election will now only consider nodes on that lineage; other nodes need pg_rewind before rejoining. + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/5016BC0 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/509CE00 | ok, on accepted lineage + node2 | 2 | 1 | 0/509CE00 | ok, on accepted lineage + node3 | 3 | 2 | 0/502D758 | FORK: diverges from the reference timeline, pg_rewind required + + One or more nodes have diverged from the reference timeline (see FORK above). + See `pg_autoctl accept timeline --help` to resolve. + +Same as before, no resync needs to be forced — the pin alone is enough, and +node3 is caught and rewound within about a second, no maintenance cycle +involved. Every row the application wrote to node1 in the meantime — 200 +rows, none of them lost or delayed by node3's fork — is there once node3 +rejoins:: + + $ pg_autoctl show timeline --formation default + TLI | Parent TLI | Switchpoint LSN + ---------+------------+---------------- + 1 | 0 | 0/0 + 2 | 1 | 0/5016BC0 + + Name | NodeId | TLI | LSN | Status + ---------------------+--------+------+-------------+----------------------------------------- + node1 | 1 | 1 | 0/7000060 | ok, on accepted lineage + node2 | 2 | 1 | 0/7000060 | ok, on accepted lineage + node3 | 3 | 1 | 0/7000060 | ok, on accepted lineage + +See ``tests/tap/specs/timeline_fork_3node_auto_detect.pgaf`` for the +automated version of this exact scenario. diff --git a/docs/tikz/seq-timeline-fork.svg b/docs/tikz/seq-timeline-fork.svg new file mode 100644 index 000000000..f2cf8c283 --- /dev/null +++ b/docs/tikz/seq-timeline-fork.svg @@ -0,0 +1,2561 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/tikz/seq-timeline-fork.tex b/docs/tikz/seq-timeline-fork.tex new file mode 100644 index 000000000..1ea667d71 --- /dev/null +++ b/docs/tikz/seq-timeline-fork.tex @@ -0,0 +1,151 @@ +\RequirePackage{luatex85} +\documentclass[border=10pt,12pt]{standalone} + +\usepackage{cfr-lm} +\usepackage{amssymb} +\usepackage{pgf} +\usepackage{tikz} +\usetikzlibrary{arrows.meta,calc,positioning} + +\begin{document} + +\sffamily\bfseries + +%% Actor x-positions (cm) +%% P=0 S=9 M=19 +\def\px{0} +\def\sx{9} +\def\mx{19} +\def\ybot{33.6} + +\begin{tikzpicture}[ + >=Stealth, + yscale=-1, +] + +\input{common.tex} + +\tikzstyle{actor}=[rectangle, minimum width=3.6cm, minimum height=0.7cm, + inner sep=4pt, rounded corners=2pt, align=center, font=\bfseries] +\tikzstyle{ll}=[densely dashed, gray!60, line width=0.5pt] +\tikzstyle{msg}=[->, line width=0.65pt] +\tikzstyle{rsp}=[->, densely dashed, line width=0.65pt] +\tikzstyle{broken}=[red!70, densely dotted, line width=0.8pt] +\tikzstyle{lbl}=[font=\small, inner sep=2pt] +\tikzstyle{note}=[rectangle, rounded corners=2pt, inner sep=5pt, + font=\small, align=center] +\tikzstyle{seclbl}=[font=\small\itshape, text=gray!70!black, anchor=west] +\tikzstyle{selfloop}=[->, line width=0.65pt] + +%% ── actor headers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] (Ph) at (\px,0) {Primary}; +\node[actor, fill=sbox, text=stxt] (Sh) at (\sx,0) {Standby}; +\node[actor, fill=mbox, text=mtxt] (Mh) at (\mx,0) {Monitor}; + +%% lifelines +\draw[ll] (\px,0.35) -- (\px,\ybot); +\draw[ll] (\sx,0.35) -- (\sx,\ybot); +\draw[ll] (\mx,0.35) -- (\mx,\ybot); + +%% ── steady state ───────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,1.3) {steady state, both on timeline 1}; + +\draw[msg] (\px,1.6) -- (\sx,1.6) + node[lbl,midway,above] {WAL stream}; +\draw[msg] (\sx,2.6) -- (\mx,2.6); +\node[lbl, above] at (\sx+3.3,2.6) {node\_active(reported=secondary, tli=1)}; +\draw[rsp] (\mx,3.2) -- (\sx,3.2); +\node[lbl, above] at (\mx-2.2,3.2) {goal=secondary}; + +%% ── network partition ──────────────────────────────────────────────────── +\node[seclbl] at (-3.4,4.6) {network partition isolates the standby}; + +\draw[broken] (\sx,5.5) -- (\px,5.5); +\node[red!70!black, font=\Large] at (5.3,5.5) {$\times$}; +\draw[broken] (\sx,6.1) -- (\mx,6.1); +\node[red!70!black, font=\Large] at (14.3,6.1) {$\times$}; +\node[lbl, above, text=red!70!black] at (\sx,6.8) + {WAL stream and monitor connection both cut}; + +%% ── out-of-band promotion ──────────────────────────────────────────────── +\node[seclbl] at (-3.4,8.2) {promoted directly at the Postgres level}; + +\draw[selfloop] (\sx,9.1) .. controls (\sx+2.6,9.1) and (\sx+2.6,10.3) .. (\sx,10.3); +\node[lbl, right] at (\sx+2.7,9.7) {\texttt{pg\_ctl promote}}; + +\node[note, fill=red!12, draw=red!50, text width=13.5cm, font=\small] + at (9.5, 11.6) + {bypasses \texttt{pg\_autoctl} entirely -- the keeper's own \texttt{current\_role} + assumption (secondary, should be in recovery) is about to stop matching + Postgres reality (promoted, writable)}; + +\draw[selfloop] (\sx,13.0) .. controls (\sx+2.6,13.0) and (\sx+2.6,14.0) .. (\sx,14.0); +\node[lbl, right] at (\sx+2.7,13.5) {local writes + \texttt{CHECKPOINT}}; + +\node[note, fill=gray!12, draw=gray!55, text width=11cm, font=\small] + at (9.5, 15.2) + {standby is now on a genuine local timeline 2 -- WAL no other node has, + and no other node can ever stream}; + +%% ── reconnects, looks routine ──────────────────────────────────────────── +\node[seclbl] at (-3.4,16.6) {network restored -- reconnects, still ``secondary''}; + +\draw[msg] (\sx,17.5) -- (\mx,17.5); +\node[lbl, above] at (\sx+3.3,17.5) {node\_active(reported=secondary, tli=2)}; +\draw[rsp] (\mx,18.1) -- (\sx,18.1); +\node[lbl, above] at (\mx-2.2,18.1) {goal=secondary}; + +\node[note, fill=async, draw=gray!50, text width=15cm, font=\small] + at (9.5, 19.6) + {no sibling standby to disagree with, so auto-detection reads this as a + legitimate lineage rather than a fork. The divergence is durably + published to \texttt{pgautofailover.node\_timeline\_history} but stays + invisible until an operator pins the correct timeline}; + +%% ── operator pins ground truth ─────────────────────────────────────────── +\node[seclbl] at (-3.4,21.4) {operator pins ground truth}; + +\draw[selfloop] (\mx,21.6) .. controls (\mx+2.6,21.6) and (\mx+2.6,22.6) .. (\mx,22.6); +\node[lbl, right] at (\mx+2.7,22.1) {\texttt{pg\_autoctl accept timeline --tli 1}}; + +\draw[msg] (\mx,23.5) -- (\sx,23.5); +\node[lbl, above] at (\mx-2.2,23.5) {goal=catchingup}; + +\node[note, fill=gray!12, draw=gray!55, text width=13cm, font=\small] + at (9.5, 24.7) + {the monitor re-checks every currently-secondary node's ancestry against + the freshly-pinned lineage right away -- pushed to catchingup within + about a second, no maintenance toggle and no incidental health-check + cycle needed}; + +%% ── ancestry check ─────────────────────────────────────────────────────── +\node[seclbl] at (-3.4,26.1) {ancestry check}; + +\draw[selfloop] (\sx,27.0) .. controls (\sx+2.6,27.0) and (\sx+2.6,28.0) .. (\sx,28.0); +\node[lbl, right] at (\sx+2.7,27.5) {walk primary's real \texttt{TIMELINE\_HISTORY}}; + +\node[note, fill=red!12, draw=red!50, text width=13cm, font=\small] + at (9.5, 29.2) + {tli 2 is not an ancestor of tli 1 -- genuine divergence, not lag. + Pre-fix: Postgres refuses the reconnect + (\emph{requested timeline 2 is not a child of this server's history}) + and the standby retries the same doomed reconnect forever}; + +%% ── pg_rewind, rejoins ─────────────────────────────────────────────────── +\node[seclbl] at (-3.4,30.6) {\texttt{pg\_rewind}, rejoins cleanly}; + +\draw[selfloop] (\sx,31.5) .. controls (\sx+2.6,31.5) and (\sx+2.6,32.3) .. (\sx,32.3); +\node[lbl, right] at (\sx+2.7,31.9) {\texttt{pg\_rewind} onto tli 1 (or fresh + \texttt{pg\_basebackup} if unreachable)}; + +\node[note, fill=green!15, draw=green!50!black, + font=\small\bfseries, minimum width=9cm] + at (4.5, 33.3) {$\checkmark$~~rejoins as secondary on tli 1, divergent WAL discarded}; + +%% ── actor footers ─────────────────────────────────────────────────────── +\node[actor, fill=pbox, text=ptxt] at (\px, \ybot+0.35) {Primary}; +\node[actor, fill=sbox, text=stxt] at (\sx, \ybot+0.35) {Standby}; +\node[actor, fill=mbox, text=mtxt] at (\mx, \ybot+0.35) {Monitor}; + +\end{tikzpicture} +\end{document} diff --git a/src/bin/common/pgctl.c b/src/bin/common/pgctl.c index 7a1193f55..a8e5cad28 100644 --- a/src/bin/common/pgctl.c +++ b/src/bin/common/pgctl.c @@ -1452,12 +1452,26 @@ pg_rewind(const char *pgdata, setenv("PGPASSWORD", replicationSource->password, 1); } + /* + * pg_rewind needs a database to connect to, but not any particular one -- + * it only runs a handful of catalog queries and a replication-mode + * connection. Using the formation's own configured database name (rather + * than a hardcoded "postgres") matters because pg_autoctl's own HBA rules + * (pghba_ensure_host_rules_exist) are written for exactly that database + * name plus the "replication" pseudo-database -- never literally + * "postgres" unless that also happens to be the configured name. A + * formation created with --dbname other than "postgres" would otherwise + * have pg_rewind's connection rejected by every peer's pg_hba.conf, + * every time, unconditionally. + */ if (!prepare_primary_conninfo(primaryConnInfo, MAXCONNINFO, primaryNode->host, primaryNode->port, replicationSource->userName, - "postgres", /* pg_rewind needs a database */ + IS_EMPTY_STRING_BUFFER(replicationSource->dbname) + ? "postgres" + : replicationSource->dbname, NULL, /* no password here */ replicationSource->applicationName, replicationSource->sslOptions, @@ -2190,7 +2204,8 @@ pg_write_recovery_conf(const char *pgdata, ReplicationSource *replicationSource) }; GUC *recoverySettings = - IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN) + (!replicationSource->pauseAtRecoveryTarget || + IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN)) ? recoverySettingsStandby : recoverySettingsTargetLSN; @@ -2259,7 +2274,8 @@ pg_write_standby_signal(const char *pgdata, }; GUC *recoverySettings = - IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN) + (!replicationSource->pauseAtRecoveryTarget || + IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN)) ? recoverySettingsStandby : recoverySettingsTargetLSN; @@ -2418,8 +2434,17 @@ prepare_recovery_settings(const char *pgdata, replicationSource->targetTimeline); } - /* We use the targetLSN only when doing a WAL fast_forward operation */ - if (!IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN)) + /* + * Only write recovery_target_lsn when the caller explicitly wants + * Postgres's own target-reached logic (see pauseAtRecoveryTarget's own + * comment). Otherwise targetLSN, if set, is purely an application-level + * polling threshold the caller checks for itself (pg_last_wal_replay_lsn + * via pgsql_has_reached_target_lsn); leaving it out of the recovery + * configuration means Postgres just streams normally, with no target of + * its own to get stuck waiting on. + */ + if (replicationSource->pauseAtRecoveryTarget && + !IS_EMPTY_STRING_BUFFER(replicationSource->targetLSN)) { sformat(targetLSN, PG_LSN_MAXLENGTH, "'%s'", replicationSource->targetLSN); diff --git a/src/bin/common/pgsql.h b/src/bin/common/pgsql.h index b8f46a240..f27c6f6e0 100644 --- a/src/bin/common/pgsql.h +++ b/src/bin/common/pgsql.h @@ -252,6 +252,21 @@ typedef struct ReplicationSource char targetLSN[PG_LSN_MAXLENGTH]; char targetAction[NAMEDATALEN]; char targetTimeline[NAMEDATALEN]; + + /* + * When true, targetLSN is written into Postgres's own + * recovery_target_lsn GUC, making Postgres itself refuse to consider + * the target reached until it has replayed a genuine WAL record at or + * past that exact LSN. That requires targetLSN to be a real record + * boundary (or the special value "immediate"); an arbitrary snapshot of + * another node's current replay position -- as used to fast-forward a + * lagging standby -- is not guaranteed to be one, and Postgres will then + * wait for a record that can never arrive. Callers that only want + * targetLSN as their own application-level "catch up to at least this + * much" polling threshold (see pgsql_has_reached_target_lsn), deciding + * for themselves when to promote, should leave this false. + */ + bool pauseAtRecoveryTarget; SSLOptions sslOptions; IdentifySystem system; } ReplicationSource; diff --git a/src/bin/pg_autoctl/cli_accept.c b/src/bin/pg_autoctl/cli_accept.c new file mode 100644 index 000000000..7c4fcb506 --- /dev/null +++ b/src/bin/pg_autoctl/cli_accept.c @@ -0,0 +1,273 @@ +/* + * src/bin/pg_autoctl/cli_accept.c + * Implementation of the pg_autoctl accept CLI, used to resolve a + * detected timeline fork by pinning which lineage is ground truth. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + * + */ + +#include "cli_common.h" +#include "commandline.h" +#include "defaults.h" +#include "env_utils.h" +#include "keeper_config.h" +#include "keeper.h" +#include "monitor.h" +#include "string_utils.h" + +static int cli_accept_timeline_getopts(int argc, char **argv); +static void cli_accept_timeline(int argc, char **argv); + +static int acceptTimelineTLI = -1; +static char acceptTimelineDecidedBy[BUFSIZE] = { 0 }; + +CommandLine accept_timeline_command = + make_command("timeline", + "Accept a timeline as the ground truth after a detected fork", + " [ --pgdata --formation --group ] --tli ", + " --pgdata path to data directory\n" + " --formation formation to target, defaults to 'default'\n" + " --group group to target, defaults to 0\n" + " --tli timeline to accept, as shown by " + "`pg_autoctl show timeline`\n" + " --reason free-text note explaining the decision\n", + cli_accept_timeline_getopts, + cli_accept_timeline); + +CommandLine *accept_subcommands[] = { + &accept_timeline_command, + NULL, +}; + +CommandLine accept_commands = + make_command_set("accept", + "Accept an operator decision orchestrated by the monitor", + NULL, NULL, NULL, accept_subcommands); + + +/* + * cli_accept_timeline_getopts parses the command line options for the + * command `pg_autoctl accept timeline`. + */ +static int +cli_accept_timeline_getopts(int argc, char **argv) +{ + KeeperConfig options = { 0 }; + int c, option_index = 0, errors = 0; + int verboseCount = 0; + + static struct option long_options[] = { + { "pgdata", required_argument, NULL, 'D' }, + { "monitor", required_argument, NULL, 'm' }, + { "formation", required_argument, NULL, 'f' }, + { "group", required_argument, NULL, 'g' }, + { "tli", required_argument, NULL, 't' }, + { "reason", required_argument, NULL, 'r' }, + { "version", no_argument, NULL, 'V' }, + { "verbose", no_argument, NULL, 'v' }, + { "quiet", no_argument, NULL, 'q' }, + { "help", no_argument, NULL, 'h' }, + { NULL, 0, NULL, 0 } + }; + + options.groupId = -1; + options.network_partition_timeout = -1; + options.prepare_promotion_catchup = -1; + options.prepare_promotion_walreceiver = -1; + options.postgresql_restart_failure_timeout = -1; + options.postgresql_restart_failure_max_retries = -1; + + optind = 0; + + while ((c = getopt_long(argc, argv, "D:f:g:t:r:Vvqh", + long_options, &option_index)) != -1) + { + switch (c) + { + case 'D': + { + strlcpy(options.pgSetup.pgdata, optarg, MAXPGPATH); + log_trace("--pgdata %s", options.pgSetup.pgdata); + break; + } + + case 'm': + { + if (!validate_connection_string(optarg)) + { + log_fatal("Failed to parse --monitor connection string, " + "see above for details."); + exit(EXIT_CODE_BAD_ARGS); + } + strlcpy(options.monitor_pguri, optarg, MAXCONNINFO); + log_trace("--monitor %s", options.monitor_pguri); + break; + } + + case 'f': + { + strlcpy(options.formation, optarg, NAMEDATALEN); + log_trace("--formation %s", options.formation); + break; + } + + case 'g': + { + if (!stringToInt(optarg, &options.groupId)) + { + log_fatal("--group argument is not a valid group ID: \"%s\"", + optarg); + exit(EXIT_CODE_BAD_ARGS); + } + log_trace("--group %d", options.groupId); + break; + } + + case 't': + { + if (!stringToInt(optarg, &acceptTimelineTLI) || + acceptTimelineTLI <= 0) + { + log_fatal("--tli argument is not a valid timeline id: \"%s\"", + optarg); + exit(EXIT_CODE_BAD_ARGS); + } + log_trace("--tli %d", acceptTimelineTLI); + break; + } + + case 'r': + { + strlcpy(acceptTimelineDecidedBy, optarg, BUFSIZE); + log_trace("--reason %s", acceptTimelineDecidedBy); + break; + } + + case 'V': + { + keeper_cli_print_version(argc, argv); + break; + } + + case 'v': + { + ++verboseCount; + switch (verboseCount) + { + case 1: + { + log_set_level(LOG_INFO); + break; + } + + case 2: + { + log_set_level(LOG_DEBUG); + break; + } + + default: + { + log_set_level(LOG_TRACE); + break; + } + } + break; + } + + case 'q': + { + log_set_level(LOG_ERROR); + break; + } + + case 'h': + { + commandline_help(stderr); + exit(EXIT_CODE_QUIT); + break; + } + + default: + { + errors++; + } + } + } + + if (acceptTimelineTLI <= 0) + { + log_fatal("Option --tli is mandatory"); + errors++; + } + + if (errors > 0) + { + commandline_help(stderr); + exit(EXIT_CODE_BAD_ARGS); + } + + if (cli_use_monitor_option(&options)) + { + if (!IS_EMPTY_STRING_BUFFER(options.pgSetup.pgdata)) + { + log_warn("Given --monitor URI, the --pgdata option is ignored"); + log_info("Connecting to monitor at \"%s\"", options.monitor_pguri); + } + + bzero((void *) options.pgSetup.pgdata, sizeof(options.pgSetup.pgdata)); + } + else + { + cli_common_get_set_pgdata_or_exit(&(options.pgSetup)); + + if (!keeper_config_set_pathnames_from_pgdata(&(options.pathnames), + options.pgSetup.pgdata)) + { + /* errors have already been logged */ + exit(EXIT_CODE_BAD_ARGS); + } + } + + if (!cli_common_ensure_formation(&options)) + { + /* errors have already been logged */ + exit(EXIT_CODE_BAD_ARGS); + } + + keeperOptions = options; + + return optind; +} + + +/* + * cli_accept_timeline calls the SQL function pgautofailover.accept_timeline() + * on the monitor. + */ +static void +cli_accept_timeline(int argc, char **argv) +{ + KeeperConfig config = keeperOptions; + Monitor monitor = { 0 }; + + (void) cli_monitor_init_from_option_or_config(&monitor, &config); + (void) cli_set_groupId(&monitor, &config); + + if (!monitor_accept_timeline(&monitor, config.formation, config.groupId, + acceptTimelineTLI, acceptTimelineDecidedBy)) + { + log_fatal("Failed to accept timeline %d for formation \"%s\" group %d, " + "see above for details", + acceptTimelineTLI, config.formation, config.groupId); + exit(EXIT_CODE_MONITOR); + } + + fformat(stdout, + "Timeline %d accepted as ground truth for formation \"%s\" " + "group %d. The election will now only consider nodes on that " + "lineage; other nodes need pg_rewind before rejoining.\n", + acceptTimelineTLI, config.formation, config.groupId); +} diff --git a/src/bin/pg_autoctl/cli_common.h b/src/bin/pg_autoctl/cli_common.h index 66c62fa40..b18d8f7b1 100644 --- a/src/bin/pg_autoctl/cli_common.h +++ b/src/bin/pg_autoctl/cli_common.h @@ -118,6 +118,12 @@ extern CommandLine perform_switchover_command; extern CommandLine *perform_subcommands[]; extern CommandLine perform_commands; +/* cli_accept.c */ +extern CommandLine accept_timeline_command; + +extern CommandLine *accept_subcommands[]; +extern CommandLine accept_commands; + /* cli_service.c */ extern CommandLine service_run_command; extern CommandLine service_stop_command; @@ -131,6 +137,7 @@ extern CommandLine show_state_command; extern CommandLine show_settings_command; extern CommandLine show_file_command; extern CommandLine show_standby_names_command; +extern CommandLine show_timeline_command; /* cli_watch.c */ extern CommandLine watch_command; diff --git a/src/bin/pg_autoctl/cli_root.c b/src/bin/pg_autoctl/cli_root.c index 34fd705cb..211c64aab 100644 --- a/src/bin/pg_autoctl/cli_root.c +++ b/src/bin/pg_autoctl/cli_root.c @@ -46,6 +46,7 @@ CommandLine *show_subcommands_with_debug[] = { &show_state_command, &show_settings_command, &show_standby_names_command, + &show_timeline_command, &show_file_command, &systemd_cat_service_file_command, NULL @@ -62,6 +63,7 @@ CommandLine *show_subcommands[] = { &show_state_command, &show_settings_command, &show_standby_names_command, + &show_timeline_command, &show_file_command, &systemd_cat_service_file_command, NULL @@ -99,6 +101,7 @@ CommandLine *root_subcommands[] = { &get_commands, &set_commands, &perform_commands, + &accept_commands, &activate_node_command, &inspect_commands, &manual_commands, diff --git a/src/bin/pg_autoctl/cli_show.c b/src/bin/pg_autoctl/cli_show.c index f8e5a85c6..3821cb07c 100644 --- a/src/bin/pg_autoctl/cli_show.c +++ b/src/bin/pg_autoctl/cli_show.c @@ -46,6 +46,9 @@ static void cli_show_events(int argc, char **argv); static int cli_show_standby_names_getopts(int argc, char **argv); static void cli_show_standby_names(int argc, char **argv); +static int cli_show_timeline_getopts(int argc, char **argv); +static void cli_show_timeline(int argc, char **argv); + static int cli_show_file_getopts(int argc, char **argv); static void cli_show_file(int argc, char **argv); static bool fprint_file_contents(const char *filename); @@ -125,6 +128,18 @@ CommandLine show_standby_names_command = cli_show_standby_names_getopts, cli_show_standby_names); +CommandLine show_timeline_command = + make_command("timeline", + "Show the timeline history known to the monitor for a group, " + "and each node's position against it", + " [ --pgdata ] --formation --group", + " --pgdata path to data directory \n" + " --monitor pg_auto_failover Monitor Postgres URL\n" + " --formation formation to query, defaults to 'default'\n" + " --group group to query formation, defaults to 0\n", + cli_show_timeline_getopts, + cli_show_timeline); + CommandLine show_file_command = make_command("file", "List pg_autoctl internal files (config, state, pid)", @@ -908,6 +923,204 @@ cli_show_standby_names(int argc, char **argv) } +/* + * cli_show_timeline_getopts parses the command line options for the + * command `pg_autoctl show timeline`. + */ +static int +cli_show_timeline_getopts(int argc, char **argv) +{ + KeeperConfig options = { 0 }; + int c, option_index = 0, errors = 0; + int verboseCount = 0; + + static struct option long_options[] = { + { "pgdata", required_argument, NULL, 'D' }, + { "monitor", required_argument, NULL, 'm' }, + { "formation", required_argument, NULL, 'f' }, + { "group", required_argument, NULL, 'g' }, + { "version", no_argument, NULL, 'V' }, + { "verbose", no_argument, NULL, 'v' }, + { "quiet", no_argument, NULL, 'q' }, + { "help", no_argument, NULL, 'h' }, + { NULL, 0, NULL, 0 } + }; + + /* set default values for our options, when we have some */ + options.groupId = -1; + options.network_partition_timeout = -1; + options.prepare_promotion_catchup = -1; + options.prepare_promotion_walreceiver = -1; + options.postgresql_restart_failure_timeout = -1; + options.postgresql_restart_failure_max_retries = -1; + + optind = 0; + + while ((c = getopt_long(argc, argv, "D:f:g:Vvqh", + long_options, &option_index)) != -1) + { + switch (c) + { + case 'D': + { + strlcpy(options.pgSetup.pgdata, optarg, MAXPGPATH); + log_trace("--pgdata %s", options.pgSetup.pgdata); + break; + } + + case 'm': + { + if (!validate_connection_string(optarg)) + { + log_fatal("Failed to parse --monitor connection string, " + "see above for details."); + exit(EXIT_CODE_BAD_ARGS); + } + strlcpy(options.monitor_pguri, optarg, MAXCONNINFO); + log_trace("--monitor %s", options.monitor_pguri); + break; + } + + case 'f': + { + strlcpy(options.formation, optarg, NAMEDATALEN); + log_trace("--formation %s", options.formation); + break; + } + + case 'g': + { + if (!stringToInt(optarg, &options.groupId)) + { + log_fatal("--group argument is not a valid group ID: \"%s\"", + optarg); + exit(EXIT_CODE_BAD_ARGS); + } + log_trace("--group %d", options.groupId); + break; + } + + case 'V': + { + /* keeper_cli_print_version prints version and exits. */ + keeper_cli_print_version(argc, argv); + break; + } + + case 'v': + { + ++verboseCount; + switch (verboseCount) + { + case 1: + { + log_set_level(LOG_INFO); + break; + } + + case 2: + { + log_set_level(LOG_DEBUG); + break; + } + + default: + { + log_set_level(LOG_TRACE); + break; + } + } + break; + } + + case 'q': + { + log_set_level(LOG_ERROR); + break; + } + + case 'h': + { + commandline_help(stderr); + exit(EXIT_CODE_QUIT); + break; + } + + default: + { + /* getopt_long already wrote an error message */ + errors++; + } + } + } + + if (errors > 0) + { + commandline_help(stderr); + exit(EXIT_CODE_BAD_ARGS); + } + + /* when we have a monitor URI we don't need PGDATA */ + if (cli_use_monitor_option(&options)) + { + if (!IS_EMPTY_STRING_BUFFER(options.pgSetup.pgdata)) + { + log_warn("Given --monitor URI, the --pgdata option is ignored"); + log_info("Connecting to monitor at \"%s\"", options.monitor_pguri); + } + } + else + { + cli_common_get_set_pgdata_or_exit(&(options.pgSetup)); + } + + /* when --pgdata is given, still initialise our pathnames */ + if (!IS_EMPTY_STRING_BUFFER(options.pgSetup.pgdata)) + { + if (!keeper_config_set_pathnames_from_pgdata(&(options.pathnames), + options.pgSetup.pgdata)) + { + /* errors have already been logged */ + exit(EXIT_CODE_BAD_CONFIG); + } + } + + /* ensure --formation, or get it from the configuration file */ + if (!cli_common_ensure_formation(&options)) + { + /* errors have already been logged */ + exit(EXIT_CODE_BAD_ARGS); + } + + keeperOptions = options; + + return optind; +} + + +/* + * cli_show_timeline prints the group's known timeline history and each + * node's current position against it. + */ +static void +cli_show_timeline(int argc, char **argv) +{ + KeeperConfig config = keeperOptions; + Monitor monitor = { 0 }; + + (void) cli_monitor_init_from_option_or_config(&monitor, &config); + + (void) cli_set_groupId(&monitor, &config); + + if (!monitor_print_timeline(&monitor, config.formation, config.groupId)) + { + log_fatal("Failed to get the group's timeline history " + "from the monitor, see above for details"); + exit(EXIT_CODE_MONITOR); + } +} + + /* * keeper_show_uri_getopts parses the command line options for the * command `pg_autoctl show uri`. diff --git a/src/bin/pg_autoctl/fsm_mermaid.c b/src/bin/pg_autoctl/fsm_mermaid.c index fd8f55d14..20ecec902 100644 --- a/src/bin/pg_autoctl/fsm_mermaid.c +++ b/src/bin/pg_autoctl/fsm_mermaid.c @@ -138,6 +138,7 @@ FsmMermaidColorForState(NodeState state) } } + /* * FsmMermaidId returns the identifier to use for a state in Mermaid source. * NodeStateToString(ANY_STATE) returns "#any state#", which is not a valid diff --git a/src/bin/pg_autoctl/fsm_transition.c b/src/bin/pg_autoctl/fsm_transition.c index 23380471d..b0579adfa 100644 --- a/src/bin/pg_autoctl/fsm_transition.c +++ b/src/bin/pg_autoctl/fsm_transition.c @@ -36,12 +36,15 @@ #include "keeper_pg_init.h" #include "log.h" #include "monitor.h" +#include "parson.h" #include "pghba.h" #include "primary_standby.h" #include "state.h" +#include "timeline_history.h" static bool fsm_init_standby_from_upstream(Keeper *keeper); +static void keeper_defensive_publish_timeline_history(Keeper *keeper); /* @@ -229,6 +232,8 @@ fsm_init_primary(Keeper *keeper) "postgres failed, see above for details"); return false; } + + keeper_defensive_publish_timeline_history(keeper); } /* @@ -1047,6 +1052,42 @@ fsm_rewind_or_init(Keeper *keeper) } +/* + * keeper_defensive_publish_timeline_history does a synchronous, defensive + * publish of what we know about our own timeline history right now, + * immediately before code that might rewind us: rewinding to reach a common + * history with a new upstream (see #683) erases the local evidence of a + * fork (pg_control's timeline_id goes back down), and the periodic per-tick + * publish (service_keeper.c) runs on the same cadence as FSM transition + * processing, so a fork that both appears and gets rewound within one tick + * can otherwise go completely unpublished, taking away the operator's only + * record that it ever happened. + */ +static void +keeper_defensive_publish_timeline_history(Keeper *keeper) +{ + PostgresSetup *pgSetup = &(keeper->postgres.postgresSetup); + uint32_t currentTLI = pgSetup->control.timeline_id; + IdentifySystem system = { 0 }; + + if (currentTLI > 0 && + keeper_fetch_local_timeline_history(pgSetup, currentTLI, &system)) + { + char *historyJSON = timeline_history_to_json(&system); + + if (historyJSON != NULL) + { + (void) monitor_report_timeline_history( + &(keeper->monitor), + keeper->state.current_node_id, + historyJSON); + + json_free_serialized_string(historyJSON); + } + } +} + + /* * fsm_prepare_for_secondary is used when going from CATCHINGUP to SECONDARY, * to create missing replication slots. We want to maintain a replication slot @@ -1105,6 +1146,15 @@ fsm_prepare_for_secondary(Keeper *keeper) postgres->pgIsRunning = true; + /* + * Defensively publish what we know about our own timeline history right + * now, before the ancestry check below might rewind us and erase the + * local evidence (see #683 and keeper_defensive_publish_timeline_history + * for why: the periodic per-tick publish alone can otherwise miss a + * fork that both appears and gets rewound within a single tick). + */ + keeper_defensive_publish_timeline_history(keeper); + /* check that we're on the same timeline as the new primary */ if (!standby_check_timeline_with_upstream(postgres)) { @@ -1235,6 +1285,14 @@ fsm_promote_standby(Keeper *keeper) return false; } + /* + * Publish our fresh timeline right away rather than waiting for the next + * periodic tick: other nodes (and an operator running `pg_autoctl show + * timeline`) should be able to see this promotion's timeline as soon as + * it's genuinely confirmed, not several seconds later. + */ + keeper_defensive_publish_timeline_history(keeper); + if (!standby_cleanup_as_primary(postgres)) { log_error("Failed to cleanup replication settings, " @@ -1300,6 +1358,18 @@ fsm_report_lsn(Keeper *keeper) return false; } + /* + * One more, synchronous, defensive publish of what we know about our + * own timeline history right now, immediately before the risky part: + * restarting standalone (no primary_conninfo) can hit a hard Postgres + * FATAL if we've diverged (see #683) and never get to report anything + * again. The periodic per-tick publish (service_keeper.c) already + * covers the common case; this call guarantees freshness even on a + * node's very first tick after a restart, before that periodic path + * has had a chance to run. + */ + keeper_defensive_publish_timeline_history(keeper); + log_info("Restarting standby node to disconnect replication " "from failed primary node, to prepare failover"); @@ -1431,21 +1501,24 @@ fsm_fast_forward(Keeper *keeper) /* - * fsm_cleanup_as_primary cleans-up the replication setting. It's called after - * a fast-forward operation. + * fsm_cleanup_as_primary is called after a fast-forward operation, at the + * fast_forward -> prepare_promotion transition. Postgres is still running as + * a standby at this point, with standby.signal in place and replication + * caught up to at least our target LSN -- the actual promotion (pg_ctl + * promote) only happens later, from stop_replication. + * + * It must NOT remove standby.signal or otherwise clean up the replication + * setup here: Postgres's own promotion completion logic removes + * standby.signal itself as part of processing the promote request, and + * FATAL-crashes ("could not remove file "standby.signal": No such file or + * directory") if it's already gone by then. The correct place for that + * cleanup is after the real promotion, which fsm_promote_standby already + * does via its own standby_cleanup_as_primary call once standby_promote has + * genuinely completed. */ bool fsm_cleanup_as_primary(Keeper *keeper) { - LocalPostgresServer *postgres = &(keeper->postgres); - - if (!standby_cleanup_as_primary(postgres)) - { - log_error("Failed to cleanup replication settings and restart Postgres " - "to continue as a primary, see above for details"); - return false; - } - return true; } @@ -1507,12 +1580,17 @@ fsm_follow_new_primary(Keeper *keeper) } /* - * Finally, check that we're on the same timeline as the new primary when - * assigned secondary as a goal state. This transition function is also - * used when going from secondary to catchingup, as the primary might have - * changed also in that situation. + * Finally, check that we're on the same timeline as the new primary. + * This transition function is also used when going from secondary to + * catchingup (assigned_role == CATCHINGUP_STATE), as the primary might + * have changed also in that situation -- and the underlying + * standby_follow_new_primary() call above just reconfigured and + * restarted Postgres to point at it either way, so the same divergence + * risk (#683) applies regardless of which of the two goal states we're + * headed to. */ - if (keeper->state.assigned_role == SECONDARY_STATE) + if (keeper->state.assigned_role == SECONDARY_STATE || + keeper->state.assigned_role == CATCHINGUP_STATE) { PGSQL *pgsql = &(postgres->sqlClient); @@ -1538,6 +1616,13 @@ fsm_follow_new_primary(Keeper *keeper) postgres->pgIsRunning = true; + /* + * Defensively publish our own timeline history before the ancestry + * check below might rewind us -- see #683 and + * keeper_defensive_publish_timeline_history. + */ + keeper_defensive_publish_timeline_history(keeper); + return standby_check_timeline_with_upstream(postgres); } diff --git a/src/bin/pg_autoctl/fsm_transition_citus.c b/src/bin/pg_autoctl/fsm_transition_citus.c index 26307b785..002f5cce8 100644 --- a/src/bin/pg_autoctl/fsm_transition_citus.c +++ b/src/bin/pg_autoctl/fsm_transition_citus.c @@ -546,30 +546,32 @@ fsm_citus_worker_promote_standby_to_single(Keeper *keeper) /* - * fsm_citus_cleanup_and_resume_as_primary cleans-up the replication setting - * and start the local node as primary. It's called after a fast-forward - * operation. + * fsm_citus_cleanup_and_resume_as_primary prepares the coordinator-side + * master_update_node() call after a fast-forward operation. It's the Citus + * counterpart of fsm_cleanup_as_primary(), called at the same + * FAST_FORWARD_STATE -> PREP_PROMOTION_STATE transition, i.e. BEFORE the + * real pg_ctl promote (which only happens later, from standby_promote() via + * fsm_promote_standby at PREP_PROMOTION_STATE -> STOP_REPLICATION_STATE / + * WAIT_PRIMARY_STATE). + * + * This must NOT touch the on-disk standby setup: Postgres's own + * promotion-completion logic removes standby.signal itself while + * processing the later pg_ctl promote, and FATAL-crashes if it's already + * gone ("could not remove file \"standby.signal\": No such file or + * directory") -- reproduced live against tests/tap/specs/ + * debug_citus_worker_fast_forward.pgaf. The crash-recovery restart that + * follows finds no standby.signal (already removed) and silently completes + * ordinary crash recovery of what looks like a primary, without ever + * genuinely promoting. This function used to call standby_cleanup_as_primary() + * here (and, before that, also restart Postgres) -- see the docstring on + * standby_cleanup_as_primary() in primary_standby.c for the full + * explanation of why that on-disk cleanup belongs solely in the + * post-promotion path (fsm_promote_standby), never here. */ bool fsm_citus_cleanup_and_resume_as_primary(Keeper *keeper) { - LocalPostgresServer *postgres = &(keeper->postgres); - - if (!standby_cleanup_as_primary(postgres)) - { - log_error("Failed to cleanup replication settings and restart Postgres " - "to continue as a primary, see above for details"); - return false; - } - - if (!keeper_restart_postgres(keeper)) - { - log_error("Failed to restart Postgres after changing its " - "primary conninfo, see above for details"); - return false; - } - - /* now prepare and commit the call to master_update_node() */ + /* prepare and commit the call to master_update_node() */ return fsm_citus_worker_prepare_standby_for_promotion(keeper); } diff --git a/src/bin/pg_autoctl/monitor.c b/src/bin/pg_autoctl/monitor.c index eb9a266c0..136319fb1 100644 --- a/src/bin/pg_autoctl/monitor.c +++ b/src/bin/pg_autoctl/monitor.c @@ -1195,6 +1195,222 @@ monitor_report_postgres_version(Monitor *monitor, int64_t nodeId, } +/* + * monitor_report_timeline_history reports this node's own timeline history + * (as encoded by timeline_history_to_json()) to the monitor. Called from + * the keeper's main loop whenever the local timeline has advanced since the + * last successful report (see service_keeper.c), and once more, + * synchronously, right before fsm_report_lsn() restarts Postgres. + */ +bool +monitor_report_timeline_history(Monitor *monitor, int64_t nodeId, + const char *historyJSON) +{ + PGSQL *pgsql = &monitor->pgsql; + const char *sql = + "SELECT pgautofailover.report_timeline_history($1, $2::jsonb)"; + + int paramCount = 2; + Oid paramTypes[2] = { INT8OID, TEXTOID }; + const char *paramValues[2]; + + IntString nodeIdString = intToString(nodeId); + + paramValues[0] = nodeIdString.strValue; + paramValues[1] = historyJSON; + + if (!pgsql_execute_with_params(pgsql, sql, paramCount, paramTypes, + paramValues, NULL, NULL)) + { + log_error("Failed to report timeline history for node %" PRId64, + nodeId); + + return false; + } + + return true; +} + + +/* + * monitor_accept_timeline pins the accepted timeline for a (formation, + * group) after an operator has resolved a detected fork, by calling + * pgautofailover.accept_timeline() on the monitor. + */ +bool +monitor_accept_timeline(Monitor *monitor, char *formation, int group, + int tli, char *decidedBy) +{ + PGSQL *pgsql = &monitor->pgsql; + const char *sql = + "SELECT pgautofailover.accept_timeline($1, $2, $3, $4)"; + + int paramCount = 4; + Oid paramTypes[4] = { TEXTOID, INT4OID, INT4OID, TEXTOID }; + const char *paramValues[4]; + + IntString groupString = intToString(group); + IntString tliString = intToString(tli); + + paramValues[0] = formation; + paramValues[1] = groupString.strValue; + paramValues[2] = tliString.strValue; + paramValues[3] = (decidedBy == NULL || IS_EMPTY_STRING_BUFFER(decidedBy)) + ? NULL + : decidedBy; + + if (!pgsql_execute_with_params(pgsql, sql, paramCount, paramTypes, + paramValues, NULL, NULL)) + { + log_error("Failed to accept timeline %d for formation %s and group %d", + tli, formation, group); + + return false; + } + + return true; +} + + +/* + * printTimelineHistoryResult is a ParsePostgresResultCB that prints one row + * per (tli, parenttli, switchpoint_lsn) known to the group. + */ +static void +printTimelineHistoryResult(void *ctx, PGresult *result) +{ + fformat(stdout, "%8s | %10s | %15s\n", "TLI", "Parent TLI", "Switchpoint LSN"); + fformat(stdout, "%8s-+-%10s-+-%15s\n", + "--------", "----------", "---------------"); + + for (int rowNumber = 0; rowNumber < PQntuples(result); rowNumber++) + { + char *tli = PQgetvalue(result, rowNumber, 0); + char *parentTli = PQgetvalue(result, rowNumber, 1); + char *switchpoint = PQgetvalue(result, rowNumber, 2); + + fformat(stdout, "%8s | %10s | %15s\n", tli, parentTli, switchpoint); + } + + fformat(stdout, "\n"); +} + + +/* + * printTimelineStatusResult is a ParsePostgresResultCB that prints one row + * per node's current timeline/LSN position and whether it's on the group's + * reference lineage. + */ +static void +printTimelineStatusResult(void *ctx, PGresult *result) +{ + bool *sawFork = (bool *) ctx; + + fformat(stdout, "%20s | %6s | %4s | %11s | %s\n", + "Name", "NodeId", "TLI", "LSN", "Status"); + fformat(stdout, "%20s-+-%6s-+-%4s-+-%11s-+-%s\n", + "--------------------", "------", "----", "-----------", + "----------------------------------------"); + + for (int rowNumber = 0; rowNumber < PQntuples(result); rowNumber++) + { + char *nodeName = PQgetvalue(result, rowNumber, 1); + char *tli = PQgetvalue(result, rowNumber, 2); + char *lsn = PQgetvalue(result, rowNumber, 3); + char *onLineage = PQgetvalue(result, rowNumber, 5); + bool isOnLineage = strcmp(onLineage, "t") == 0; + + if (!isOnLineage && sawFork != NULL) + { + *sawFork = true; + } + + fformat(stdout, "%20s | %6s | %4s | %11s | %s\n", + nodeName, + PQgetvalue(result, rowNumber, 0), + tli, + lsn, + isOnLineage + ? "ok, on accepted lineage" + : "FORK: diverges from the reference timeline, " + "pg_rewind required"); + } + + fformat(stdout, "\n"); +} + + +/* + * monitor_print_timeline prints the group's known timeline history and each + * node's current position against it, for `pg_autoctl show timeline`. + */ +bool +monitor_print_timeline(Monitor *monitor, char *formation, int group) +{ + PGSQL *pgsql = &monitor->pgsql; + bool sawFork = false; + + { + const char *sql = + "SELECT h.tli, h.parenttli, h.switchpoint_lsn " + " FROM pgautofailover.node_timeline_history h " + " JOIN pgautofailover.node n ON n.nodeid = h.nodeid " + " WHERE n.formationid = $1 AND n.groupid = $2 " + " GROUP BY h.tli, h.parenttli, h.switchpoint_lsn " + " ORDER BY h.tli"; + + int paramCount = 2; + Oid paramTypes[2] = { TEXTOID, INT4OID }; + const char *paramValues[2]; + IntString groupString = intToString(group); + + paramValues[0] = formation; + paramValues[1] = groupString.strValue; + + if (!pgsql_execute_with_params(pgsql, sql, paramCount, paramTypes, + paramValues, NULL, + &printTimelineHistoryResult)) + { + log_error("Failed to retrieve the group's timeline history"); + return false; + } + } + + { + const char *sql = + "SELECT node_id, node_name, tli, lsn, reference_tli, " + " on_accepted_lineage " + " FROM pgautofailover.node_timeline_status($1, $2)"; + + int paramCount = 2; + Oid paramTypes[2] = { TEXTOID, INT4OID }; + const char *paramValues[2]; + IntString groupString = intToString(group); + + paramValues[0] = formation; + paramValues[1] = groupString.strValue; + + if (!pgsql_execute_with_params(pgsql, sql, paramCount, paramTypes, + paramValues, &sawFork, + &printTimelineStatusResult)) + { + log_error("Failed to retrieve per-node timeline status"); + return false; + } + } + + if (sawFork) + { + fformat(stdout, + "One or more nodes have diverged from the reference " + "timeline (see FORK above).\n" + "See `pg_autoctl accept timeline --help` to resolve.\n"); + } + + return true; +} + + /* * monitor_get_node_region retrieves the region label of a node from the * monitor. diff --git a/src/bin/pg_autoctl/monitor.h b/src/bin/pg_autoctl/monitor.h index 8704f354b..260f1e517 100644 --- a/src/bin/pg_autoctl/monitor.h +++ b/src/bin/pg_autoctl/monitor.h @@ -165,6 +165,11 @@ bool monitor_get_node_region(Monitor *monitor, char *region, size_t size); bool monitor_report_postgres_version(Monitor *monitor, int64_t nodeId, PostgresVersionInfo *pgVersion); +bool monitor_report_timeline_history(Monitor *monitor, int64_t nodeId, + const char *historyJSON); +bool monitor_accept_timeline(Monitor *monitor, char *formation, int group, + int tli, char *decidedBy); +bool monitor_print_timeline(Monitor *monitor, char *formation, int group); bool monitor_get_formation_number_sync_standbys(Monitor *monitor, char *formation, int *numberSyncStandbys); bool monitor_set_formation_number_sync_standbys(Monitor *monitor, char *formation, diff --git a/src/bin/pg_autoctl/primary_standby.c b/src/bin/pg_autoctl/primary_standby.c index e08f77871..52cff187b 100644 --- a/src/bin/pg_autoctl/primary_standby.c +++ b/src/bin/pg_autoctl/primary_standby.c @@ -24,6 +24,7 @@ #include "primary_standby.h" #include "signals.h" #include "state.h" +#include "timeline_history.h" static bool local_postgres_wait_until_ready(LocalPostgresServer *postgres); @@ -1182,6 +1183,14 @@ postgres_maybe_do_crash_recovery(LocalPostgresServer *postgres) "pause", sizeof(crashRecoveryReplicationSource.targetAction)); + /* + * "immediate" is always a well-defined, reachable target (the + * earliest consistent state), unlike an arbitrary peer LSN + * snapshot -- so it's safe to let Postgres enforce reaching it via + * its own recovery_target_lsn/action GUCs. + */ + crashRecoveryReplicationSource.pauseAtRecoveryTarget = true; + strlcpy(crashRecoveryReplicationSource.targetTimeline, "current", sizeof(crashRecoveryReplicationSource.targetTimeline)); @@ -1313,6 +1322,92 @@ postgres_maybe_do_crash_recovery(LocalPostgresServer *postgres) } +/* + * standby_promotion_advanced_timeline verifies that reaching a not-in-recovery + * state genuinely came from a real Postgres promotion -- which always + * allocates a new, strictly higher timeline and writes a pg_wal/.history + * file recording the switchpoint -- rather than from ordinary crash recovery + * of a data directory whose standby.signal had already been removed. + * + * That second case is not hypothetical: a premature standby.signal removal + * (or a restart racing one) can make Postgres boot as an ordinary, + * non-standby server and complete plain crash recovery on its current + * timeline -- reaching a not-in-recovery state without ever going through + * the real promotion code path at all, and ending up stuck on its old + * timeline, indistinguishable by number from a demoted former primary. See + * standby_cleanup_as_primary's own comment for the specific sequence this + * guards against. + * + * priorTLI is the timeline this node was on before this promotion attempt; + * pass 0 when checking idempotently (no promotion was attempted in this + * call, e.g. the "already not in recovery" early return below) to rely + * solely on the on-disk history-file evidence. + */ +static bool +standby_promotion_advanced_timeline(LocalPostgresServer *postgres, + uint32_t priorTLI) +{ + PGSQL *pgsql = &(postgres->sqlClient); + PostgresSetup *pgSetup = &(postgres->postgresSetup); + + if (!pgsql_get_postgres_metadata(pgsql, + &(pgSetup->is_in_recovery), + postgres->pgsrSyncState, + postgres->currentLSN, + &(pgSetup->control))) + { + log_error("Failed to fetch Postgres metadata to verify promotion"); + return false; + } + + uint32_t currentTLI = pgSetup->control.timeline_id; + + if (currentTLI <= 1) + { + log_error("Promotion could not be confirmed: still on timeline %d, " + "which is never reached by a genuine promotion", + currentTLI); + return false; + } + + if (priorTLI > 0 && currentTLI <= priorTLI) + { + log_error("Promotion did not advance the timeline: still on " + "timeline %d, same as before promoting", + currentTLI); + return false; + } + + IdentifySystem system = { 0 }; + + if (!keeper_fetch_local_timeline_history(pgSetup, currentTLI, &system)) + { + log_error("Failed to read local timeline history to verify promotion"); + return false; + } + + /* + * A genuine promotion to currentTLI always writes a history file + * recording its parent timeline's switchpoint, so the parsed history has + * at least one ancestor entry plus the current tip (count > 1). A node + * that never really left its prior timeline only ever has the bare tip + * (count == 1, see keeper_fetch_local_timeline_history). + */ + if (system.timelines.count <= 1) + { + log_error("Promotion to timeline %d could not be confirmed: no " + "local history file records how this timeline started; " + "refusing to trust an unverified promotion", + currentTLI); + return false; + } + + log_info("Confirmed promotion to timeline %d", currentTLI); + + return true; +} + + /* * standby_promote promotes a standby postgres server to primary. */ @@ -1338,8 +1433,19 @@ standby_promote(LocalPostgresServer *postgres) /* * Ensure idempotency: if in the last run we managed to promote, but - * failed to checkpoint, we still need to checkpoint. + * failed to checkpoint, we still need to checkpoint. But first make + * sure this really is a prior promotion and not the "reached + * not-in-recovery without ever really promoting" failure mode + * documented on standby_promotion_advanced_timeline. */ + if (!standby_promotion_advanced_timeline(postgres, 0)) + { + log_error("Refusing to proceed: postgres is not in recovery " + "mode, but this does not look like a genuine prior " + "promotion, see above for details"); + return false; + } + if (!pgsql_checkpoint(pgsql)) { log_error("Failed to checkpoint after promotion"); @@ -1349,9 +1455,77 @@ standby_promote(LocalPostgresServer *postgres) return true; } + uint32_t priorTLI = pgSetup->control.timeline_id; + + /* + * When we were prepared by fast_forward (standby_fetch_missing_wal), + * targetLSN is our own application-level "caught up enough to promote" + * threshold (see ReplicationSource.pauseAtRecoveryTarget) rather than a + * Postgres recovery target -- this node just streams normally. + * Fast_forward itself already confirmed replay had reached that point + * once -- but every Postgres restart since then (and there can be + * several, e.g. the prepare_promotion / stop_replication transitions + * each reconfigure and restart Postgres for unrelated reasons) starts + * recovery over from the last checkpoint, which is not guaranteed to be + * at or past the target either. Re-verify it here, on whichever + * instance is actually about to receive the promote request, rather + * than trusting a check some earlier, possibly since-restarted instance + * made. + */ + if (!IS_EMPTY_STRING_BUFFER(postgres->replicationSource.targetLSN)) + { + char currentLSN[PG_LSN_MAXLENGTH] = { 0 }; + bool hasReachedLSN = false; + + do { + if (asked_to_stop || asked_to_stop_fast) + { + log_trace("standby_promote: signaled"); + pgsql_finish(pgsql); + return false; + } + + if (!pgsql_has_reached_target_lsn(pgsql, + postgres->replicationSource.targetLSN, + currentLSN, + &hasReachedLSN)) + { + /* errors have already been logged */ + return false; + } + + if (!hasReachedLSN) + { + log_info("Waiting for recovery to reach LSN %s before " + "promoting (currently at %s)", + postgres->replicationSource.targetLSN, + currentLSN); + pg_usleep(AWAIT_PROMOTION_SLEEP_TIME_MS * 1000); + } + } while (!hasReachedLSN); + } + /* disconnect from PostgreSQL now */ pgsql_finish(pgsql); + /* + * The Postgres controller service (service_postgres_ctl.c, a separate + * process from this one) polls every 100ms and would otherwise + * interpret Postgres's own transient unavailability while ending + * recovery and switching timeline as "Postgres died", forcing a + * restart of its own right in the middle of the promotion -- see the + * same reasoning in standby_fetch_missing_wal. Tell it to stand down + * for the duration of this wait and hand control back once we're done + * either way. + */ + if (!keeper_set_postgres_state_unknown(&(postgres->expectedPgStatus.state), + postgres->expectedPgStatus.pgStatusPath)) + { + log_error("Failed to signal the Postgres controller service to " + "stand down during promotion"); + return false; + } + log_info("Promoting postgres"); if (!pg_ctl_promote(pgSetup->pg_ctl, pgSetup->pgdata)) @@ -1380,6 +1554,22 @@ standby_promote(LocalPostgresServer *postgres) } } while (inRecovery); + /* hand control back to the Postgres controller service */ + if (!keeper_set_postgres_state_running(&(postgres->expectedPgStatus.state), + postgres->expectedPgStatus.pgStatusPath)) + { + log_error("Failed to signal the Postgres controller service to " + "resume supervising Postgres after promotion"); + return false; + } + + if (!standby_promotion_advanced_timeline(postgres, priorTLI)) + { + log_error("Failed to promote standby: reached a not-in-recovery " + "state without genuinely promoting, see above for details"); + return false; + } + /* * It's necessary to do a checkpoint before allowing the old primary to * rewind, since there can be a race condition in which pg_rewind detects @@ -1590,6 +1780,37 @@ standby_fetch_missing_wal(LocalPostgresServer *postgres) upstreamNode->port); } + /* + * This node's recovery configuration is deliberately left WITHOUT a + * recovery_target_lsn (see ReplicationSource.pauseAtRecoveryTarget): + * targetLSN here is a snapshot of another standby's current replay + * position, not necessarily a genuine WAL record boundary, and asking + * Postgres to enforce it as a hard recovery target can leave it waiting + * forever for a record that will never arrive. Instead this node just + * streams normally, and we decide for ourselves, by polling + * pg_last_wal_replay_lsn(), once replay has caught up to at least + * targetLSN -- which is all fast-forward ever needed. The eventual + * pg_ctl promote (issued later, by standby_promote) then completes + * recovery at wherever replay genuinely stands, which is always a valid + * position. + * + * The Postgres controller service (service_postgres_ctl.c, a separate + * process from this one) polls every 100ms and, as long as the expected + * status we just set via standby_restart_with_current_replication_source + * above is RUNNING, would otherwise interpret any transient + * unavailability during that wait as "Postgres died" and force a + * restart of its own. Tell it to stand down for the duration -- same + * mechanism postgres_maybe_do_crash_recovery uses -- and hand control + * back once we're done either way. + */ + if (!keeper_set_postgres_state_unknown(&(postgres->expectedPgStatus.state), + postgres->expectedPgStatus.pgStatusPath)) + { + log_error("Failed to signal the Postgres controller service to " + "stand down during fast-forward"); + return false; + } + /* * Now loop until replay has reached our targetLSN. */ @@ -1631,6 +1852,15 @@ standby_fetch_missing_wal(LocalPostgresServer *postgres) return false; } + /* hand control back to the Postgres controller service */ + if (!keeper_set_postgres_state_running(&(postgres->expectedPgStatus.state), + postgres->expectedPgStatus.pgStatusPath)) + { + log_error("Failed to signal the Postgres controller service to " + "resume supervising Postgres after fast-forward"); + return false; + } + log_info("Fast-forward is done, now at LSN %s", postgres->currentLSN); /* @@ -1719,9 +1949,37 @@ standby_restart_with_current_replication_source(LocalPostgresServer *postgres) /* - * standby_cleanup_as_primary removes the setup for a standby server and - * restarts as a primary. It's typically called after standby_fetch_missing_wal - * so we expect Postgres to be running as a standby and be "paused". + * standby_cleanup_as_primary removes the standby setup (standby.signal, + * primary_conninfo, ...) from the data directory. + * + * This is called from two different moments in the FSM, with two different + * expectations of the currently-running Postgres process: + * + * 1. fast_forward -> prepare_promotion (fsm_citus_cleanup_and_resume_as_primary + * for Citus nodes only -- fsm_cleanup_as_primary, its non-Citus + * counterpart, deliberately does NOT call this function; see its own + * comment): Postgres is still running as an ordinary streaming standby, + * caught up to at least the fast-forward target LSN (see + * standby_fetch_missing_wal) but NOT promoted yet -- the actual pg_ctl + * promote only happens later, from standby_promote(). Restarting + * Postgres here, after standby.signal has been removed but before a + * real promotion took place, makes it boot as an ordinary (non-standby) + * server and complete plain crash recovery on its CURRENT timeline -- + * reaching a not-in-recovery state without ever genuinely promoting, + * indistinguishable by timeline number from a demoted former primary. + * standby_promote()'s own standby_promotion_advanced_timeline check + * catches that case defensively, but this function itself only removes + * the on-disk standby setup and must NOT restart Postgres, leaving the + * running process untouched and ready for the FSM's later, explicit + * pg_ctl promote. + * + * 2. prepare_promotion -> stop_replication (fsm_promote_standby): called + * right after standby_promote() has genuinely promoted this node (new, + * strictly higher timeline, confirmed via standby_promotion_advanced_timeline). + * Here too we only need to remove the on-disk standby setup so that a + * later restart of this now-primary node doesn't accidentally pick stale + * replication configuration back up; the already-promoted, running + * process needs no restart of its own. */ bool standby_cleanup_as_primary(LocalPostgresServer *postgres) @@ -1745,6 +2003,91 @@ standby_cleanup_as_primary(LocalPostgresServer *postgres) } +/* + * standby_verify_timeline_ancestry checks whether the local timeline is a + * genuine ancestor of the upstream's current timeline, using the upstream's + * own TIMELINE_HISTORY data (already fetched into + * postgres->replicationSource.system.timelines by the pgctl_identify_system() + * call that standby_check_timeline_with_upstream() makes before calling us). + * + * Unlike a bare timeline-number comparison, this can tell "genuine ancestor, + * just behind" apart from "diverged, but the timeline number still happens to + * be lower": it walks the upstream's linear timeline history (oldest ancestor + * first, current tip last) looking for the entry whose tli matches our local + * timeline, and confirms our local checkpoint LSN lies strictly before the + * point where that timeline was superseded (entry.end). If our checkpoint is + * past that point, we replayed WAL that the upstream's lineage never + * produced: a genuine fork, not just lag. + * + * Returns true when ancestry is confirmed (or trivially, when the timelines + * already match). Returns false both when a genuine divergence is detected + * and when the local timeline doesn't appear in the upstream's history at + * all (unrelated history); callers should treat both as "needs pg_rewind", + * not "just keep retrying". + */ +static bool +standby_verify_timeline_ancestry(LocalPostgresServer *postgres) +{ + ReplicationSource *replicationSource = &(postgres->replicationSource); + NodeAddress *primaryNode = &(replicationSource->primaryNode); + TimeLineHistory *timelines = &(replicationSource->system.timelines); + + uint32_t localTLI = postgres->postgresSetup.control.timeline_id; + uint64_t localLSN = 0; + + if (!parseLSN(postgres->postgresSetup.control.latestCheckpointLSN, + &localLSN)) + { + log_error("Failed to parse local latest checkpoint LSN \"%s\"", + postgres->postgresSetup.control.latestCheckpointLSN); + return false; + } + + for (int i = 0; i < timelines->count; i++) + { + TimeLineHistoryEntry *entry = &(timelines->history[i]); + + if (entry->tli != localTLI) + { + continue; + } + + /* entry->end is InvalidXLogRecPtr (0) only for the current tip */ + bool isAncestor = + XLogRecPtrIsInvalid(entry->end) || localLSN < entry->end; + + if (!isAncestor) + { + log_error("Local timeline %d diverges from upstream " NODE_FORMAT + ": local checkpoint is at %X/%X, but timeline %d was " + "superseded at %X/%X -- pg_rewind is required", + localTLI, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + (uint32) (localLSN >> 32), + (uint32) localLSN, + localTLI, + (uint32) (entry->end >> 32), + (uint32) entry->end); + } + + return isAncestor; + } + + log_error("Local timeline %d does not appear in the timeline history " + "reported by upstream " NODE_FORMAT, + localTLI, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port); + + return false; +} + + /* * standby_check_timeline_with_upstream returns true when the current timeline * on the local node (a standby) is the same as the timeline fetched on the @@ -1787,40 +2130,218 @@ standby_check_timeline_with_upstream(LocalPostgresServer *postgres) } /* - * We only allow this transition when the standby node as caught-up with - * the upstream timeline. As streaming replication is supposed to be a - * clean history replay (no PITR shenanigans), it is never expected that - * the local timeline would be greater than the timeline found on the - * upstream node. + * Streaming replication is supposed to be a clean history replay (no + * PITR shenanigans), so it is not normally expected that our local + * timeline would be ahead of the upstream's. When it happens (#683: this + * node was promoted, or otherwise advanced, outside of pg_autoctl's + * control, and is now being pointed at an upstream that never saw that + * promotion), there is no common-ancestor question to settle first the + * way there is when we're behind: we hold local WAL the upstream does + * not have, on a timeline it will never grow into, so no amount of + * waiting resolves it and pg_rewind is unconditionally required to + * reach a common history. */ if (upstreamTimeline < localTimeline) { - log_error("Current timeline on upstream node " NODE_FORMAT - " is %d, and current timeline on this standby node is %d", - primaryNode->nodeId, - primaryNode->name, - primaryNode->host, - primaryNode->port, - upstreamTimeline, - localTimeline); + log_warn("Local timeline %d is ahead of upstream " NODE_FORMAT + "'s timeline %d; rewinding to reach a common history", + localTimeline, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + upstreamTimeline); - return false; + if (!primary_rewind_to_standby(postgres)) + { + log_error("Failed to rewind after detecting that our local " + "timeline is ahead of the upstream, see above for " + "details"); + return false; + } + + /* re-fetch local metadata: the rewind changed our timeline/LSN */ + if (!pgsql_get_postgres_metadata(&(postgres->sqlClient), + &(postgres->postgresSetup.is_in_recovery), + postgres->pgsrSyncState, + postgres->currentLSN, + &(postgres->postgresSetup.control))) + { + log_error("Failed to update the local Postgres metadata " + "after rewind"); + return false; + } + + localTimeline = postgres->postgresSetup.control.timeline_id; + + log_info("Reached timeline %d after rewind, upstream node " NODE_FORMAT + " is at timeline %d", + localTimeline, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + upstreamTimeline); + + return upstreamTimeline == localTimeline; } else if (upstreamTimeline > localTimeline) { - log_warn("Current timeline on upstream node " NODE_FORMAT - " is %d, and current timeline on this standby node is still %d", + /* + * Being behind is normal while still catching up: most of the time + * our local timeline is a genuine ancestor of the upstream's, and we + * just haven't replayed our way to its tip yet. But it can also mean + * we've diverged onto a dead branch (see #683) -- in which case no + * amount of retrying will ever let us reach it through ordinary + * streaming replication. Tell the two apart using the upstream's own + * timeline history. + */ + if (standby_verify_timeline_ancestry(postgres)) + { + log_warn("Current timeline on upstream node " NODE_FORMAT + " is %d, and current timeline on this standby node is " + "still %d", + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + upstreamTimeline, + localTimeline); + + return false; + } + + log_warn("Local timeline %d has diverged from upstream " NODE_FORMAT + "; rewinding to reach a common history", + localTimeline, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port); + + if (!primary_rewind_to_standby(postgres)) + { + log_error("Failed to rewind after detecting a timeline " + "divergence, see above for details"); + return false; + } + + /* re-fetch local metadata: the rewind changed our timeline/LSN */ + if (!pgsql_get_postgres_metadata(&(postgres->sqlClient), + &(postgres->postgresSetup.is_in_recovery), + postgres->pgsrSyncState, + postgres->currentLSN, + &(postgres->postgresSetup.control))) + { + log_error("Failed to update the local Postgres metadata " + "after rewind"); + return false; + } + + localTimeline = postgres->postgresSetup.control.timeline_id; + + log_info("Reached timeline %d after rewind, upstream node " NODE_FORMAT + " is at timeline %d", + localTimeline, primaryNode->nodeId, primaryNode->name, primaryNode->host, primaryNode->port, - upstreamTimeline, - localTimeline); + upstreamTimeline); - return false; + return upstreamTimeline == localTimeline; } else if (upstreamTimeline == localTimeline) { + /* + * Matching timeline numbers alone is not proof of a shared history: + * two nodes can each keep believing they're on timeline N when one + * of them never genuinely left it through a real promotion (see + * standby_promotion_advanced_timeline and the fast_forward race it + * guards against) while independently generating its own local WAL + * on that same numbered timeline. pg_rewind itself trusts a bare + * timeline-number match without checking further -- a known, + * documented limitation shared with Patroni's own rewind-decision + * code -- so this check doesn't rely on it alone either. + * + * What a genuine, unbroken standby can never do is get ahead of its + * own upstream in LSN terms while remaining on the exact same + * timeline: ordinary streaming replication only ever replays WAL + * the upstream produced. If our local position is past what the + * upstream reports as its own current position, we hold WAL bytes + * it never produced -- the same genuine divergence #683 handles for + * differing timeline numbers, just not expressed as one here. + */ + uint64_t upstreamLSN = 0; + uint64_t localLSN = 0; + + if (!parseLSN(replicationSource->system.xlogpos, &upstreamLSN)) + { + log_error("Failed to parse upstream node " NODE_FORMAT + "'s reported LSN \"%s\"", + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + replicationSource->system.xlogpos); + return false; + } + + if (!parseLSN(postgres->currentLSN, &localLSN)) + { + log_error("Failed to parse local current LSN \"%s\"", + postgres->currentLSN); + return false; + } + + if (localLSN > upstreamLSN) + { + log_warn("Local timeline %d matches upstream " NODE_FORMAT + "'s timeline, but local LSN %s is past upstream's " + "reported LSN %s; this is not reachable through " + "ordinary streaming replication, rewinding to reach a " + "common history", + localTimeline, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + postgres->currentLSN, + replicationSource->system.xlogpos); + + if (!primary_rewind_to_standby(postgres)) + { + log_error("Failed to rewind after detecting a same-timeline " + "LSN divergence, see above for details"); + return false; + } + + /* re-fetch local metadata: the rewind changed our timeline/LSN */ + if (!pgsql_get_postgres_metadata(&(postgres->sqlClient), + &(postgres->postgresSetup.is_in_recovery), + postgres->pgsrSyncState, + postgres->currentLSN, + &(postgres->postgresSetup.control))) + { + log_error("Failed to update the local Postgres metadata " + "after rewind"); + return false; + } + + localTimeline = postgres->postgresSetup.control.timeline_id; + + log_info("Reached timeline %d after rewind, upstream node " + NODE_FORMAT " is at timeline %d", + localTimeline, + primaryNode->nodeId, + primaryNode->name, + primaryNode->host, + primaryNode->port, + upstreamTimeline); + + return upstreamTimeline == localTimeline; + } + log_info("Reached timeline %d, same as upstream node " NODE_FORMAT, localTimeline, primaryNode->nodeId, diff --git a/src/bin/pg_autoctl/service_keeper.c b/src/bin/pg_autoctl/service_keeper.c index 9150447f4..c82209c12 100644 --- a/src/bin/pg_autoctl/service_keeper.c +++ b/src/bin/pg_autoctl/service_keeper.c @@ -23,6 +23,7 @@ #include "keeper_pg_init.h" #include "log.h" #include "monitor.h" +#include "parson.h" #include "pgctl.h" #include "pidfile.h" #include "primary_standby.h" @@ -32,6 +33,7 @@ #include "state.h" #include "string_utils.h" #include "supervisor.h" +#include "timeline_history.h" #include "runprogram.h" @@ -58,6 +60,7 @@ KeeperNodesArrayRefreshFunction *KeeperRefreshHooks = static bool service_keeper_node_active(Keeper *keeper, bool doInit); +static bool keeper_maybe_report_timeline_history(Keeper *keeper); static void check_for_network_partitions(Keeper *keeper); static bool is_network_healthy(Keeper *keeper); static bool in_network_partition(KeeperStateData *keeperState, uint64_t now, @@ -826,6 +829,22 @@ keeper_node_active_loop(Keeper *keeper, pid_t start_pid) } couldContactMonitor = couldContactMonitorThisRound; + + /* + * Check for a new local timeline and publish it to the monitor, + * unconditionally of the current FSM state: an uneventful + * secondary quietly follows timeline switches through ordinary + * streaming replication, with no fsm_* transition function ever + * running, so this can't be hooked off specific transitions -- + * it has to poll every tick like this. Cheap: only touches the + * filesystem when postgresSetup.control.timeline_id (already + * refreshed above by keeper_update_pg_state) has advanced past + * what we last published. + */ + if (couldContactMonitorThisRound) + { + (void) keeper_maybe_report_timeline_history(keeper); + } } if (keeperState->assigned_role != keeperState->current_role) @@ -1148,6 +1167,73 @@ service_keeper_node_active(Keeper *keeper, bool doInit) } +/* + * keeper_maybe_report_timeline_history checks the local node's current + * timeline (already refreshed this tick by keeper_update_pg_state(), from + * pg_control -- no extra syscall here) against the highest timeline this + * process has already published, and reports the local timeline history to + * the monitor when it has advanced. + * + * lastPublishedTLI is deliberately a plain in-memory watermark, not + * persisted state: on every fresh process start it's zero again, so the + * first tick after any pg_autoctl restart always re-publishes -- cheap + * (ON CONFLICT DO NOTHING makes the insert a no-op when nothing changed), + * and it means we don't need to worry about the watermark itself getting + * out of sync with what the monitor actually has on file. + */ +static bool +keeper_maybe_report_timeline_history(Keeper *keeper) +{ + static uint32_t lastPublishedTLI = 0; + static IdentifySystem system = { 0 }; + + LocalPostgresServer *postgres = &(keeper->postgres); + uint32_t currentTLI = postgres->postgresSetup.control.timeline_id; + + if (currentTLI == 0 || currentTLI <= lastPublishedTLI) + { + return true; + } + + if (!keeper_fetch_local_timeline_history(&(postgres->postgresSetup), + currentTLI, + &system)) + { + log_warn("Failed to read the local timeline history for timeline %d", + currentTLI); + return false; + } + + char *historyJSON = timeline_history_to_json(&system); + + if (historyJSON == NULL) + { + log_warn("Failed to encode the local timeline history as JSON"); + return false; + } + + bool reported = + monitor_report_timeline_history(&(keeper->monitor), + keeper->state.current_node_id, + historyJSON); + + json_free_serialized_string(historyJSON); + + if (reported) + { + lastPublishedTLI = currentTLI; + } + else + { + log_warn("Failed to report the local timeline history " + "for timeline %d", + currentTLI); + } + + return reported; +} + + /* * check_for_network_partitions checks whether we're likely to be in a network * partition. That will cause the assigned_role to become demoted. diff --git a/src/bin/pg_autoctl/timeline_history.c b/src/bin/pg_autoctl/timeline_history.c new file mode 100644 index 000000000..bcc2b4e43 --- /dev/null +++ b/src/bin/pg_autoctl/timeline_history.c @@ -0,0 +1,130 @@ +/* + * src/bin/pg_autoctl/timeline_history.c + * Reads the local node's own timeline history (from pg_wal/.history, not + * over a replication connection) and encodes it for publishing to the + * monitor. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + * + */ + +#include + +#include "postgres_fe.h" + +#include "file_utils.h" +#include "log.h" +#include "parson.h" +#include "timeline_history.h" + + +/* + * keeper_fetch_local_timeline_history reads this node's own + * pg_wal/%08X.history file for the given current timeline, without any + * replication connection: this is what makes it possible to call on every + * keeper tick, unconditionally of FSM state, cheaply. + * + * A node still on the very first timeline it has ever known has no history + * file at all (nothing has forked yet): that's expected, not an error, and + * results in a single-entry history (the current tip only). + * + * On return, system->timeline and system->timelines are populated exactly + * as they would be by parsing a TIMELINE_HISTORY replication command result + * (see parseTimeLineHistory), oldest known ancestor first, current tip last. + */ +bool +keeper_fetch_local_timeline_history(PostgresSetup *pgSetup, + uint32_t currentTLI, + IdentifySystem *system) +{ + char historyFileName[MAXPGPATH] = { 0 }; + char historyFilePath[MAXPGPATH] = { 0 }; + char *content = NULL; + long fileSize = 0L; + bool ok = false; + + system->timeline = currentTLI; + + sformat(historyFileName, sizeof(historyFileName), + "%08X.history", currentTLI); + + join_path_components(historyFilePath, pgSetup->pgdata, "pg_wal"); + join_path_components(historyFilePath, historyFilePath, historyFileName); + + if (currentTLI <= 1 || !read_file_if_exists(historyFilePath, + &content, + &fileSize)) + { + /* + * No history file: either we're still on timeline 1 (nothing has + * ever forked), or the file isn't there for some other reason. In + * both cases we still know our own current tip, we just don't know + * of any ancestor beyond it. + */ + content = strdup(""); + + if (content == NULL) + { + log_error(ALLOCATION_FAILED_ERROR); + return false; + } + } + + ok = parseTimeLineHistory(historyFilePath, content, system); + + free(content); + + return ok; +} + + +/* + * timeline_history_to_json encodes a node's timeline history (as fetched by + * keeper_fetch_local_timeline_history) as a JSON array of + * {tli, parenttli, switchpoint} objects, oldest first, suitable for + * pgautofailover.report_timeline_history()'s history jsonb argument. + * + * parenttli is 0 for the oldest entry (its parent, if any, predates + * anything we know about locally). switchpoint is the LSN at which that + * entry's timeline began (entry->begin), which is exactly the LSN at which + * the parent timeline was superseded. + * + * Returns a malloc'd string the caller must free with + * json_free_serialized_string(), or NULL on error. + */ +char * +timeline_history_to_json(IdentifySystem *system) +{ + JSON_Value *jsArray = json_value_init_array(); + JSON_Array *array = json_value_get_array(jsArray); + + for (int i = 0; i < system->timelines.count; i++) + { + TimeLineHistoryEntry *entry = &(system->timelines.history[i]); + + uint32_t parentTLI = + (i == 0) ? 0 : system->timelines.history[i - 1].tli; + + char switchpoint[PG_LSN_MAXLENGTH] = { 0 }; + + sformat(switchpoint, sizeof(switchpoint), "%X/%X", + (uint32_t) (entry->begin >> 32), + (uint32_t) entry->begin); + + JSON_Value *jsEntry = json_value_init_object(); + JSON_Object *jsObj = json_value_get_object(jsEntry); + + json_object_set_number(jsObj, "tli", entry->tli); + json_object_set_number(jsObj, "parenttli", parentTLI); + json_object_set_string(jsObj, "switchpoint", switchpoint); + + json_array_append_value(array, jsEntry); + } + + char *serialized = json_serialize_to_string(jsArray); + + json_value_free(jsArray); + + return serialized; +} diff --git a/src/bin/pg_autoctl/timeline_history.h b/src/bin/pg_autoctl/timeline_history.h new file mode 100644 index 000000000..fcb1d7c59 --- /dev/null +++ b/src/bin/pg_autoctl/timeline_history.h @@ -0,0 +1,27 @@ +/* + * src/bin/pg_autoctl/timeline_history.h + * Reads the local node's own timeline history (from pg_wal/.history, not + * over a replication connection) and encodes it for publishing to the + * monitor. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + * + */ + +#ifndef TIMELINE_HISTORY_H +#define TIMELINE_HISTORY_H + +#include +#include + +#include "pgsetup.h" +#include "pgsql.h" + +bool keeper_fetch_local_timeline_history(PostgresSetup *pgSetup, + uint32_t currentTLI, + IdentifySystem *system); + +char * timeline_history_to_json(IdentifySystem *system); + +#endif /* TIMELINE_HISTORY_H */ diff --git a/src/bin/pgaftest/Makefile b/src/bin/pgaftest/Makefile index f5da13c0e..20729aadc 100644 --- a/src/bin/pgaftest/Makefile +++ b/src/bin/pgaftest/Makefile @@ -34,7 +34,7 @@ SHARED_SRCS = cli_common.c config.c coordinator.c fsm.c fsm_transition.c \ nodespec.c nodestate_utils.c pghba.c primary_standby.c \ service_keeper.c service_keeper_init.c service_monitor.c \ service_monitor_init.c service_postgres.c service_postgres_ctl.c \ - state.c supervisor.c systemd_config.c + state.c supervisor.c systemd_config.c timeline_history.c SHARED_OBJS = $(patsubst %.c,shared-%.o,$(SHARED_SRCS)) diff --git a/src/bin/pgaftest/test_runner.c b/src/bin/pgaftest/test_runner.c index 3f54d81df..bb294ed01 100644 --- a/src/bin/pgaftest/test_runner.c +++ b/src/bin/pgaftest/test_runner.c @@ -4655,7 +4655,7 @@ runner_run(TestSpec *spec, const char *workDir, bool noCleanup) /* setup{} */ if (spec->setup) { - char err[512] = ""; + char err[8192] = ""; log_info("Running setup block"); if (!runner_exec_step(&r, spec->setup, err, sizeof(err), 0)) { @@ -4671,7 +4671,7 @@ runner_run(TestSpec *spec, const char *workDir, bool noCleanup) /* teardown{} — always run, even on setup failure */ if (spec->teardown) { - char tdErr[512] = ""; + char tdErr[8192] = ""; log_info("Running teardown block"); runner_exec_step(&r, spec->teardown, tdErr, sizeof(tdErr), 0); } @@ -4693,7 +4693,7 @@ runner_run(TestSpec *spec, const char *workDir, bool noCleanup) } log_info("STEP %d: %s", i + 1, name); - char err[512] = ""; + char err[8192] = ""; struct timespec t0, t1; clock_gettime(CLOCK_MONOTONIC, &t0); @@ -4735,7 +4735,7 @@ runner_run(TestSpec *spec, const char *workDir, bool noCleanup) /* teardown{} — always runs */ if (spec->teardown) { - char err[512] = ""; + char err[8192] = ""; log_info("Running teardown block"); runner_exec_step(&r, spec->teardown, err, sizeof(err), 0); } @@ -4870,7 +4870,7 @@ runner_setup(TestSpec *spec, const char *workDir, bool withTmux) /* run setup{} block synchronously when not using tmux */ if (spec->setup) { - char err[512] = ""; + char err[8192] = ""; log_info("Running setup block"); if (!runner_exec_step(&r, spec->setup, err, sizeof(err), 0)) { @@ -5051,7 +5051,7 @@ runner_step(TestSpec *spec, const char *workDir, const char *stepName) return false; } - char err[512] = ""; + char err[8192] = ""; bool ok = runner_exec_step(&r, step, err, sizeof(err), 0); /* Update interactive state file so `show step` markers stay accurate */ @@ -5102,7 +5102,7 @@ runner_run_setup_only(TestSpec *spec, const char *workDir) return true; } - char err[512] = ""; + char err[8192] = ""; log_info("Running setup block"); if (!runner_exec_step(&r, spec->setup, err, sizeof(err), 0)) { @@ -5124,7 +5124,7 @@ runner_down(TestSpec *spec, const char *workDir) /* teardown{} */ if (spec->teardown) { - char err[512] = ""; + char err[8192] = ""; runner_exec_step(&r, spec->teardown, err, sizeof(err), 0); } diff --git a/src/monitor/Makefile b/src/monitor/Makefile index ea5337fc9..43f1f85c8 100644 --- a/src/monitor/Makefile +++ b/src/monitor/Makefile @@ -14,7 +14,7 @@ MODULE_big = $(EXTENSION) OBJS = $(patsubst ${SRC_DIR}%.c,%.o,$(wildcard ${SRC_DIR}*.c)) PG_CPPFLAGS = -Wall -Werror -Wno-unused-parameter -Iinclude -I$(libpq_srcdir) -g SHLIB_LINK = $(libpq) -REGRESS = create_extension monitor workers node_active_protocol guard_data_loss fast_forward drop_node stale_primary_report lock_and_fetch_migration dummy_update drop_extension upgrade +REGRESS = create_extension monitor workers node_active_protocol guard_data_loss fast_forward drop_node stale_primary_report lock_and_fetch_migration timeline_fork_detection dummy_update drop_extension upgrade ISOLATION = concurrent_remove_node concurrent_remove_standby_and_primary_report concurrent_remove_standby_and_standby_report concurrent_second_primary_death_report concurrent_health_check_and_report concurrent_candidate_priority_and_quorum PG_CONFIG ?= pg_config diff --git a/src/monitor/expected/pg19/expected/timeline_fork_detection.out b/src/monitor/expected/pg19/expected/timeline_fork_detection.out new file mode 100644 index 000000000..23aac6f12 --- /dev/null +++ b/src/monitor/expected/pg19/expected/timeline_fork_detection.out @@ -0,0 +1,542 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression tests for timeline-fork detection: node_timeline_history, +-- report_timeline_history(), accept_timeline(), resolve_accepted_timeline(), +-- node_timeline_status(), and the election's ancestry filter +-- (FilterNodesByTimelineAncestry(), called from ProceedGroupStateForMSFailover). +\x on +-- ── Part A: table + function unit tests ───────────────────────────────────── +-- +-- Three nodes are enough to exercise report_timeline_history(), +-- node_timeline_status(), accept_timeline(), and resolve_accepted_timeline() +-- directly, without driving the full FSM: reportedtli/reportedlsn are set by +-- hand, the same way other regress tests here manufacture node state +-- (see guard_data_loss.sql). +SELECT pgautofailover.create_formation('tlf_unit', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+------------------------------ +create_formation | (tlf_unit,pgsql,postgres,t,1) + +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-p', 5432, + 'postgres', 'p', 1); +-[ RECORD 1 ]---------------+------- +assigned_node_id | 24 +assigned_group_id | 0 +assigned_group_state | single +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | p + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 'p' \gset +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s1', 5432, + 'postgres', 's1', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 25 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s1 + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's1' \gset +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s2', 5432, + 'postgres', 's2', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 26 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s2 + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's2' \gset +-- p stayed on the original timeline (tli=1). s1 was promoted onto tli=2 at +-- some point in the past. s2 was promoted onto tli=3 -- a sibling branch of +-- s1's tli=2, both children of tli=1 (a genuine fork: two nodes each think +-- they are the continuation). +SELECT pgautofailover.report_timeline_history(:np, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +-- report_timeline_history() must be idempotent: re-reporting the exact same +-- facts (or a subset of them) must not create duplicate or conflicting rows. +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT nodeid, tli, parenttli, switchpoint_lsn + FROM pgautofailover.node_timeline_history + ORDER BY nodeid, tli; +-[ RECORD 1 ]---+----------- +nodeid | 24 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/00000000 +-[ RECORD 2 ]---+----------- +nodeid | 25 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/00000000 +-[ RECORD 3 ]---+----------- +nodeid | 25 +tli | 2 +parenttli | 1 +switchpoint_lsn | 0/00006000 +-[ RECORD 4 ]---+----------- +nodeid | 26 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/00000000 +-[ RECORD 5 ]---+----------- +nodeid | 26 +tli | 3 +parenttli | 1 +switchpoint_lsn | 0/00006000 + +UPDATE pgautofailover.node SET reportedtli = 1, reportedlsn = '0/5500' + WHERE nodeid = :np; +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns1; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns2; +-- No operator pin yet: reference_tli defaults to the highest reportedtli in +-- the group (3, from s2). s1 (tli=2) is a sibling of s2's branch, not an +-- ancestor of it -- on_accepted_lineage must be false. p (tli=1) and s2 +-- (tli=3, the reference itself) must both be true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 3 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 3 +on_accepted_lineage | f +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 3 +on_accepted_lineage | t + +-- accept_timeline() must refuse to pin a timeline nobody in the group has +-- ever reported. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 99, 'operator typo'); +ERROR: timeline 99 has never been reported by any node in formation "tlf_unit" group 0 +CONTEXT: PL/pgSQL function pgautofailover.accept_timeline(text,integer,integer,text) line 15 at RAISE +-- Pin tli=2 (s1's branch) as ground truth instead of the auto-detected tli=3. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 2, 'operator override'); +-[ RECORD 1 ]---+-- +accept_timeline | t + +-- The pin flips the reference: now s1 (tli=2) is on_accepted_lineage, and s2 +-- (tli=3, the sibling the operator rejected) is not. p (tli=1) is still an +-- ancestor of tli=2, so it stays true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 2 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 2 +on_accepted_lineage | t +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 2 +on_accepted_lineage | f + +SELECT resolve_accepted_timeline + FROM pgautofailover.resolve_accepted_timeline('tlf_unit', 0); +-[ RECORD 1 ]-------------+-- +resolve_accepted_timeline | t + +SELECT formationid, groupid, accepted_tli, decided_by, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_unit' AND groupid = 0; +-[ RECORD 1 ]+------------------ +formationid | tlf_unit +groupid | 0 +accepted_tli | 2 +decided_by | operator override +resolved | t + +-- Once resolved, the pin no longer applies: reference_tli reverts to the +-- auto-detected max (3), same as before the pin was ever set. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 3 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 3 +on_accepted_lineage | f +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 3 +on_accepted_lineage | t + +-- ── Part B: election-level test ────────────────────────────────────────────── +-- +-- Drive a real 4-node group (primary + 3 standbys) through the FSM, then +-- kill the primary and simulate two standbys that have each, at some earlier +-- point, been promoted onto their own diverging branch -- a genuine fork. +-- The operator pins one branch as ground truth *before* calling +-- perform_failover(); the election must pick the candidate on the pinned +-- lineage and exclude the other, even though the excluded one has a more +-- advanced (higher) timeline number. +SELECT pgautofailover.create_formation('tlf_election', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+---------------------------------- +create_formation | (tlf_election,pgsql,postgres,t,1) + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-p', 5432, + 'postgres', 'p', 1); +-[ RECORD 1 ]---------------+------- +assigned_node_id | 27 +assigned_group_id | 0 +assigned_group_state | single +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | p + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 'p' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s1', 5432, + 'postgres', 's1', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 28 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s1 + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's1' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s2', 5432, + 'postgres', 's2', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 29 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s2 + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's2' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s3', 5432, + 'postgres', 's3', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 30 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s3 + +SELECT nodeid AS ns3 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's3' \gset +-- ── bootstrap: p primary, s1/s2/s3 secondary ──────────────────────────────── +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single'); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +-- Verify final formation state: p=primary, s1/s2/s3=secondary. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+--------------- +nodename | p +goalstate | apply_settings +reportedstate | primary +-[ RECORD 2 ]-+--------------- +nodename | s1 +goalstate | secondary +reportedstate | secondary +-[ RECORD 3 ]-+--------------- +nodename | s2 +goalstate | catchingup +reportedstate | secondary +-[ RECORD 4 ]-+--------------- +nodename | s3 +goalstate | wait_standby +reportedstate | secondary + +-- ── publish two diverging branches ────────────────────────────────────────── +-- +-- s2 was, at some earlier point, promoted onto tli=2. s3 was promoted onto +-- tli=3, a sibling branch, both children of tli=1. s1 stayed on the original +-- tli=1. +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns3, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns2; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns3; +-- ── operator pins s2's branch (tli=2) as ground truth ─────────────────────── +-- +-- Left to auto-detection this group would pick tli=3 (the highest reported +-- tli, s3's branch). The operator instead pins tli=2: the election must +-- honor that and pick s2, excluding s3 even though s3's tli number is +-- higher. +SELECT pgautofailover.accept_timeline('tlf_election', 0, 2, 'operator override'); +-[ RECORD 1 ]---+-- +accept_timeline | t + +-- ── simulate primary death, standbys already at report_lsn ───────────────── +SET pgautofailover.startup_grace_period = 1; +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds' + WHERE formationid = 'tlf_election' AND nodename = 'p'; +UPDATE pgautofailover.node + SET goalstate = 'draining', reportedstate = 'draining' + WHERE formationid = 'tlf_election' AND nodename = 'p'; +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'report_lsn' + WHERE formationid = 'tlf_election' AND nodename IN ('s1', 's2', 's3'); +SELECT nodename, goalstate, reportedstate, reportedtli, reportedlsn + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+----------- +nodename | p +goalstate | draining +reportedstate | draining +reportedtli | 1 +reportedlsn | 0/00005000 +-[ RECORD 2 ]-+----------- +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 1 +reportedlsn | 0/00005000 +-[ RECORD 3 ]-+----------- +nodename | s2 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 2 +reportedlsn | 0/00006500 +-[ RECORD 4 ]-+----------- +nodename | s3 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 3 +reportedlsn | 0/00007500 + +-- The dead primary itself always counts as a "missing" quorum node (it +-- can never report_lsn), which trips guard_data_loss regardless of the +-- ancestry filter under test here; disable it, same as guard_data_loss.sql +-- does for its own "proceed despite a missing report" scenario. +SET pgautofailover.guard_data_loss TO false; +SELECT pgautofailover.perform_failover('tlf_election', 0); +-[ RECORD 1 ]----+- +perform_failover | + +-- s2 (the pinned lineage, tli=2) must be the promoted candidate. s3 (tli=3, +-- the higher-numbered sibling branch) must be excluded and left untouched +-- in report_lsn. s1 (tli=1, an ancestor of the pinned tli=2) remains a +-- comparable standby but was less advanced than s2, so it is not selected +-- either. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+------------------ +nodename | p +goalstate | draining +reportedstate | draining +-[ RECORD 2 ]-+------------------ +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +-[ RECORD 3 ]-+------------------ +nodename | s2 +goalstate | prepare_promotion +reportedstate | report_lsn +-[ RECORD 4 ]-+------------------ +nodename | s3 +goalstate | report_lsn +reportedstate | report_lsn + +-- PromoteSelectedNode() must auto-resolve the operator's pin once a primary +-- has been promoted on the accepted lineage. +SELECT formationid, groupid, accepted_tli, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_election' AND groupid = 0; +-[ RECORD 1 ]+------------- +formationid | tlf_election +groupid | 0 +accepted_tli | 2 +resolved | t + +RESET pgautofailover.guard_data_loss; +RESET pgautofailover.startup_grace_period; diff --git a/src/monitor/expected/timeline_fork_detection.out b/src/monitor/expected/timeline_fork_detection.out new file mode 100644 index 000000000..0cb535d8a --- /dev/null +++ b/src/monitor/expected/timeline_fork_detection.out @@ -0,0 +1,542 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression tests for timeline-fork detection: node_timeline_history, +-- report_timeline_history(), accept_timeline(), resolve_accepted_timeline(), +-- node_timeline_status(), and the election's ancestry filter +-- (FilterNodesByTimelineAncestry(), called from ProceedGroupStateForMSFailover). +\x on +-- ── Part A: table + function unit tests ───────────────────────────────────── +-- +-- Three nodes are enough to exercise report_timeline_history(), +-- node_timeline_status(), accept_timeline(), and resolve_accepted_timeline() +-- directly, without driving the full FSM: reportedtli/reportedlsn are set by +-- hand, the same way other regress tests here manufacture node state +-- (see guard_data_loss.sql). +SELECT pgautofailover.create_formation('tlf_unit', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+------------------------------ +create_formation | (tlf_unit,pgsql,postgres,t,1) + +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-p', 5432, + 'postgres', 'p', 1); +-[ RECORD 1 ]---------------+------- +assigned_node_id | 24 +assigned_group_id | 0 +assigned_group_state | single +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | p + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 'p' \gset +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s1', 5432, + 'postgres', 's1', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 25 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s1 + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's1' \gset +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s2', 5432, + 'postgres', 's2', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 26 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s2 + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's2' \gset +-- p stayed on the original timeline (tli=1). s1 was promoted onto tli=2 at +-- some point in the past. s2 was promoted onto tli=3 -- a sibling branch of +-- s1's tli=2, both children of tli=1 (a genuine fork: two nodes each think +-- they are the continuation). +SELECT pgautofailover.report_timeline_history(:np, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +-- report_timeline_history() must be idempotent: re-reporting the exact same +-- facts (or a subset of them) must not create duplicate or conflicting rows. +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT nodeid, tli, parenttli, switchpoint_lsn + FROM pgautofailover.node_timeline_history + ORDER BY nodeid, tli; +-[ RECORD 1 ]---+------- +nodeid | 24 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/0 +-[ RECORD 2 ]---+------- +nodeid | 25 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/0 +-[ RECORD 3 ]---+------- +nodeid | 25 +tli | 2 +parenttli | 1 +switchpoint_lsn | 0/6000 +-[ RECORD 4 ]---+------- +nodeid | 26 +tli | 1 +parenttli | 0 +switchpoint_lsn | 0/0 +-[ RECORD 5 ]---+------- +nodeid | 26 +tli | 3 +parenttli | 1 +switchpoint_lsn | 0/6000 + +UPDATE pgautofailover.node SET reportedtli = 1, reportedlsn = '0/5500' + WHERE nodeid = :np; +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns1; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns2; +-- No operator pin yet: reference_tli defaults to the highest reportedtli in +-- the group (3, from s2). s1 (tli=2) is a sibling of s2's branch, not an +-- ancestor of it -- on_accepted_lineage must be false. p (tli=1) and s2 +-- (tli=3, the reference itself) must both be true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 3 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 3 +on_accepted_lineage | f +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 3 +on_accepted_lineage | t + +-- accept_timeline() must refuse to pin a timeline nobody in the group has +-- ever reported. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 99, 'operator typo'); +ERROR: timeline 99 has never been reported by any node in formation "tlf_unit" group 0 +CONTEXT: PL/pgSQL function pgautofailover.accept_timeline(text,integer,integer,text) line 15 at RAISE +-- Pin tli=2 (s1's branch) as ground truth instead of the auto-detected tli=3. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 2, 'operator override'); +-[ RECORD 1 ]---+-- +accept_timeline | t + +-- The pin flips the reference: now s1 (tli=2) is on_accepted_lineage, and s2 +-- (tli=3, the sibling the operator rejected) is not. p (tli=1) is still an +-- ancestor of tli=2, so it stays true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 2 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 2 +on_accepted_lineage | t +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 2 +on_accepted_lineage | f + +SELECT resolve_accepted_timeline + FROM pgautofailover.resolve_accepted_timeline('tlf_unit', 0); +-[ RECORD 1 ]-------------+-- +resolve_accepted_timeline | t + +SELECT formationid, groupid, accepted_tli, decided_by, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_unit' AND groupid = 0; +-[ RECORD 1 ]+------------------ +formationid | tlf_unit +groupid | 0 +accepted_tli | 2 +decided_by | operator override +resolved | t + +-- Once resolved, the pin no longer applies: reference_tli reverts to the +-- auto-detected max (3), same as before the pin was ever set. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; +-[ RECORD 1 ]-------+--- +node_name | p +tli | 1 +reference_tli | 3 +on_accepted_lineage | t +-[ RECORD 2 ]-------+--- +node_name | s1 +tli | 2 +reference_tli | 3 +on_accepted_lineage | f +-[ RECORD 3 ]-------+--- +node_name | s2 +tli | 3 +reference_tli | 3 +on_accepted_lineage | t + +-- ── Part B: election-level test ────────────────────────────────────────────── +-- +-- Drive a real 4-node group (primary + 3 standbys) through the FSM, then +-- kill the primary and simulate two standbys that have each, at some earlier +-- point, been promoted onto their own diverging branch -- a genuine fork. +-- The operator pins one branch as ground truth *before* calling +-- perform_failover(); the election must pick the candidate on the pinned +-- lineage and exclude the other, even though the excluded one has a more +-- advanced (higher) timeline number. +SELECT pgautofailover.create_formation('tlf_election', 'pgsql', 'postgres', true, 1); +-[ RECORD 1 ]----+---------------------------------- +create_formation | (tlf_election,pgsql,postgres,t,1) + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-p', 5432, + 'postgres', 'p', 1); +-[ RECORD 1 ]---------------+------- +assigned_node_id | 27 +assigned_group_id | 0 +assigned_group_state | single +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | p + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 'p' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s1', 5432, + 'postgres', 's1', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 28 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s1 + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's1' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s2', 5432, + 'postgres', 's2', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 29 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s2 + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's2' \gset +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s3', 5432, + 'postgres', 's3', 1); +-[ RECORD 1 ]---------------+------------- +assigned_node_id | 30 +assigned_group_id | 0 +assigned_group_state | wait_standby +assigned_candidate_priority | 100 +assigned_replication_quorum | t +assigned_node_name | s3 + +SELECT nodeid AS ns3 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's3' \gset +-- ── bootstrap: p primary, s1/s2/s3 secondary ──────────────────────────────── +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single'); +-[ RECORD 1 ]--------+------- +assigned_group_state | single + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+-------- +assigned_group_state | primary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+---------- +assigned_group_state | secondary + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+----------- +assigned_group_state | catchingup + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'wait_standby'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+------------- +assigned_group_state | wait_standby + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); +-[ RECORD 1 ]--------+--------------- +assigned_group_state | apply_settings + +-- Verify final formation state: p=primary, s1/s2/s3=secondary. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+--------------- +nodename | p +goalstate | apply_settings +reportedstate | primary +-[ RECORD 2 ]-+--------------- +nodename | s1 +goalstate | secondary +reportedstate | secondary +-[ RECORD 3 ]-+--------------- +nodename | s2 +goalstate | catchingup +reportedstate | secondary +-[ RECORD 4 ]-+--------------- +nodename | s3 +goalstate | wait_standby +reportedstate | secondary + +-- ── publish two diverging branches ────────────────────────────────────────── +-- +-- s2 was, at some earlier point, promoted onto tli=2. s3 was promoted onto +-- tli=3, a sibling branch, both children of tli=1. s1 stayed on the original +-- tli=1. +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +SELECT pgautofailover.report_timeline_history(:ns3, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); +-[ RECORD 1 ]-----------+- +report_timeline_history | + +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns2; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns3; +-- ── operator pins s2's branch (tli=2) as ground truth ─────────────────────── +-- +-- Left to auto-detection this group would pick tli=3 (the highest reported +-- tli, s3's branch). The operator instead pins tli=2: the election must +-- honor that and pick s2, excluding s3 even though s3's tli number is +-- higher. +SELECT pgautofailover.accept_timeline('tlf_election', 0, 2, 'operator override'); +-[ RECORD 1 ]---+-- +accept_timeline | t + +-- ── simulate primary death, standbys already at report_lsn ───────────────── +SET pgautofailover.startup_grace_period = 1; +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds' + WHERE formationid = 'tlf_election' AND nodename = 'p'; +UPDATE pgautofailover.node + SET goalstate = 'draining', reportedstate = 'draining' + WHERE formationid = 'tlf_election' AND nodename = 'p'; +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'report_lsn' + WHERE formationid = 'tlf_election' AND nodename IN ('s1', 's2', 's3'); +SELECT nodename, goalstate, reportedstate, reportedtli, reportedlsn + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+----------- +nodename | p +goalstate | draining +reportedstate | draining +reportedtli | 1 +reportedlsn | 0/5000 +-[ RECORD 2 ]-+----------- +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 1 +reportedlsn | 0/5000 +-[ RECORD 3 ]-+----------- +nodename | s2 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 2 +reportedlsn | 0/6500 +-[ RECORD 4 ]-+----------- +nodename | s3 +goalstate | report_lsn +reportedstate | report_lsn +reportedtli | 3 +reportedlsn | 0/7500 + +-- The dead primary itself always counts as a "missing" quorum node (it +-- can never report_lsn), which trips guard_data_loss regardless of the +-- ancestry filter under test here; disable it, same as guard_data_loss.sql +-- does for its own "proceed despite a missing report" scenario. +SET pgautofailover.guard_data_loss TO false; +SELECT pgautofailover.perform_failover('tlf_election', 0); +-[ RECORD 1 ]----+- +perform_failover | + +-- s2 (the pinned lineage, tli=2) must be the promoted candidate. s3 (tli=3, +-- the higher-numbered sibling branch) must be excluded and left untouched +-- in report_lsn. s1 (tli=1, an ancestor of the pinned tli=2) remains a +-- comparable standby but was less advanced than s2, so it is not selected +-- either. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; +-[ RECORD 1 ]-+------------------ +nodename | p +goalstate | draining +reportedstate | draining +-[ RECORD 2 ]-+------------------ +nodename | s1 +goalstate | report_lsn +reportedstate | report_lsn +-[ RECORD 3 ]-+------------------ +nodename | s2 +goalstate | prepare_promotion +reportedstate | report_lsn +-[ RECORD 4 ]-+------------------ +nodename | s3 +goalstate | report_lsn +reportedstate | report_lsn + +-- PromoteSelectedNode() must auto-resolve the operator's pin once a primary +-- has been promoted on the accepted lineage. +SELECT formationid, groupid, accepted_tli, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_election' AND groupid = 0; +-[ RECORD 1 ]+------------- +formationid | tlf_election +groupid | 0 +accepted_tli | 2 +resolved | t + +RESET pgautofailover.guard_data_loss; +RESET pgautofailover.startup_grace_period; diff --git a/src/monitor/group_state_machine.c b/src/monitor/group_state_machine.c index 0ff353f77..8927c916f 100644 --- a/src/monitor/group_state_machine.c +++ b/src/monitor/group_state_machine.c @@ -22,6 +22,7 @@ #include "node_metadata.h" #include "notifications.h" #include "replication_state.h" +#include "timeline_history.h" #include "version_compat.h" #include "access/htup_details.h" @@ -324,6 +325,67 @@ ProceedGroupStateFromContext(GroupStateContext *ctx) ReplicationStateGetName(activeNode->goalState)))); } + /* + * Detect a genuine timeline fork on a healthy secondary as soon as its + * newly reported timeline is visible, rather than waiting for an + * incidental health-check cycle or an explicit maintenance toggle to + * eventually drive it through catchingup (see #683 and the timeline + * fork detection design). Reuses the exact same ancestry filter the + * report_lsn election path already applies (below, and in + * ProceedGroupStateForMSFailover) -- this only changes when a + * diverged secondary gets pushed to catchingup, not how that's + * decided or how the resync itself recovers it. + * + * Scoped to activeNode currently being SECONDARY_STATE: that's the + * same state the "unhealthy secondary" transition just below already + * uses for the identical CATCHINGUP goal assignment, so this is + * additive to an existing, tested transition rather than a new kind + * of one. A node with reportedTLI == 0 hasn't reported a timeline yet + * (e.g. still in wait_standby) and has nothing to check. + */ + if (IsCurrentState(activeNode, REPLICATION_STATE_SECONDARY) && + activeNode->reportedTLI > 0) + { + int referenceTli = 0; + List *comparableNodeList = + FilterNodesByTimelineAncestry(ctx->groupNodeList, formationId, + groupId, &referenceTli); + + bool activeNodeIsComparable = false; + ListCell *cell = NULL; + + foreach(cell, comparableNodeList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(cell); + + if (node->nodeId == activeNode->nodeId) + { + activeNodeIsComparable = true; + break; + } + } + + if (referenceTli > 0 && !activeNodeIsComparable) + { + char message[BUFSIZE] = { 0 }; + + LogAndNotifyMessage( + message, BUFSIZE, + "Setting goal state of " NODE_FORMAT + " to catchingup: its reported timeline %d does not appear " + "to be an ancestor of the group's reference timeline %d -- " + "forcing a resync so the ancestry check can run and, if " + "needed, rewind it onto the correct lineage.", + NODE_FORMAT_ARGS(activeNode), + activeNode->reportedTLI, + referenceTli); + + AssignGoalState(activeNode, REPLICATION_STATE_CATCHINGUP, message); + + return true; + } + } + /* * Replication stall detection (issue #997 — 3-DC split-brain). * @@ -1561,10 +1623,24 @@ ProceedGroupStateForMSFailover(GroupStateContext *ctx, * or get unhealthy: then the next call to node_active() might build a * different candidateNodesGroupList in which every node has reported their * LSN position, allowing progress to be made. + * + * Before any of that: filter out nodes whose reported timeline has + * genuinely diverged from the group's reference lineage (see #683). + * They are excluded, not deprioritized -- a diverged node can never + * become comparable no matter how long we wait for it, so leaving it + * in would risk either comparing incomparable LSNs, or blocking the + * whole election on a node that will never resolve on its own. */ + int referenceTli = 0; + List *comparableNodesGroupList = + FilterNodesByTimelineAncestry(nodesGroupList, + ctx->formationId, + ctx->groupId, + &referenceTli); + candidateList.numberSyncStandbys = ctx->formation->number_sync_standbys; - BuildCandidateList(ctx, nodesGroupList, &candidateList); + BuildCandidateList(ctx, comparableNodesGroupList, &candidateList); /* * Time to select a candidate? @@ -1670,7 +1746,7 @@ ProceedGroupStateForMSFailover(GroupStateContext *ctx, { /* build the list of most advanced standby nodes, not ordered */ List *mostAdvancedNodeList = - ListMostAdvancedStandbyNodes(nodesGroupList); + ListMostAdvancedStandbyNodes(comparableNodesGroupList); /* select a node to failover to */ @@ -2139,6 +2215,14 @@ PromoteSelectedNode(AutoFailoverNode *selectedNode, (errmsg("BUG: selectedNode is NULL in PromoteSelectedNode"))); } + /* + * A candidate was selected from a pool already filtered to the + * accepted (or auto-detected) lineage: whatever operator-pinned fork + * resolution was in effect has done its job. Mark it resolved so a + * future, unrelated fork doesn't inherit a stale pin. + */ + ResolveAcceptedTimeline(selectedNode->formationId, selectedNode->groupId); + /* * Ok so we now may start the failover process, we have selected a * candidate after all nodes reported their LSN. We still have two diff --git a/src/monitor/pgautofailover--2.2--2.3.sql b/src/monitor/pgautofailover--2.2--2.3.sql index 16e77a460..7b858f5a1 100644 --- a/src/monitor/pgautofailover--2.2--2.3.sql +++ b/src/monitor/pgautofailover--2.2--2.3.sql @@ -148,3 +148,312 @@ grant execute on function pgautofailover.replication_state,text, int,bool,text,text) to autoctl_node; + + +-- +-- Add set_node_region(), making the region column (added above) mutable +-- after registration (#1156). +-- + +CREATE FUNCTION pgautofailover.set_node_region + ( + IN formation_id text, + IN node_name text, + IN region text + ) +RETURNS bool LANGUAGE C STRICT SECURITY DEFINER +AS 'MODULE_PATHNAME', $$set_node_region$$; + +comment on function pgautofailover.set_node_region(text, text, text) + is 'sets the region label for a node, identifying its data-centre or availability zone'; + +grant execute on function + pgautofailover.set_node_region(text, text, text) + to autoctl_node; + + +-- +-- Add Postgres/Citus version tracking columns and report_postgres_version() +-- (#1157). +-- + +ALTER TABLE pgautofailover.node + ADD COLUMN IF NOT EXISTS pg_versionnum int, + ADD COLUMN IF NOT EXISTS pg_version text, + ADD COLUMN IF NOT EXISTS pg_versionstring text, + ADD COLUMN IF NOT EXISTS citus_version text; + +-- Deliberately NOT STRICT: version_num/version/versionstring/citus_version +-- are all allowed to be NULL. citus_version legitimately is, whenever +-- Citus isn't installed on that node. This is a plain self-report, called +-- once per Postgres restart by the keeper -- not part of the FSM, and not +-- routed through node_active() since none of this can change without a +-- restart. +CREATE FUNCTION pgautofailover.report_postgres_version + ( + IN node_id bigint, + IN pg_versionnum int default null, + IN pg_version text default null, + IN pg_versionstring text default null, + IN citus_version text default null + ) +RETURNS void LANGUAGE C SECURITY DEFINER +AS 'MODULE_PATHNAME', $$report_postgres_version$$; + +comment on function pgautofailover.report_postgres_version(bigint,int,text,text,text) + is 'reports a node''s Postgres server version and, when installed, Citus extension version'; + +grant execute on function + pgautofailover.report_postgres_version(bigint,int,text,text,text) + to autoctl_node; + + +-- +-- Add timeline-fork detection: node_timeline_history / accepted_timeline +-- tables, and the report_timeline_history() / accept_timeline() / +-- resolve_accepted_timeline() / node_timeline_status() functions. +-- + +-- Every Postgres timeline has exactly one parent and one fork LSN, so the +-- union of every node's known history forms a single tree. Append-only: a +-- (tli, parenttli, switchpoint_lsn) triple never changes once recorded, and +-- is an objective fact about that timeline, not about which node reported +-- it -- nodeid is provenance (who told us), not part of what the row means. +CREATE TABLE pgautofailover.node_timeline_history + ( + nodeid bigint not null + references pgautofailover.node(nodeid) on delete cascade, + tli int not null, + parenttli int not null, + switchpoint_lsn pg_lsn not null, + + PRIMARY KEY (nodeid, tli) + ); + +-- An operator's explicit pin of which timeline is ground truth after a +-- detected fork. Normally empty: absent any unresolved row, the election +-- runs its ordinary auto-detection unchanged. +CREATE TABLE pgautofailover.accepted_timeline + ( + formationid text not null, + groupid int not null, + accepted_tli int not null, + decided_by text, + decided_at timestamptz not null default now(), + resolved_at timestamptz, + + PRIMARY KEY (formationid, groupid, decided_at) + ); + +-- matches the blanket grant used by a fresh install (pgautofailover.sql) +GRANT SELECT ON pgautofailover.node_timeline_history TO autoctl_node; +GRANT SELECT ON pgautofailover.accepted_timeline TO autoctl_node; + +-- history is a JSON array of {"tli": int, "parenttli": int, +-- "switchpoint": text} objects, oldest first, as produced by the keeper's +-- timeline_history_to_json(). Called both periodically (whenever the local +-- timeline has advanced) and synchronously right before a report_lsn +-- restart, so this needs to be cheap and idempotent: an unconditional +-- ON CONFLICT DO NOTHING insert of already-known facts is a no-op. +CREATE FUNCTION pgautofailover.report_timeline_history + ( + IN node_id bigint, + IN history jsonb + ) +RETURNS void LANGUAGE plpgsql SECURITY DEFINER +AS $$ +BEGIN + INSERT INTO pgautofailover.node_timeline_history + (nodeid, tli, parenttli, switchpoint_lsn) + SELECT node_id, + (entry->>'tli')::int, + (entry->>'parenttli')::int, + (entry->>'switchpoint')::pg_lsn + FROM jsonb_array_elements(history) AS entry + ON CONFLICT (nodeid, tli) DO NOTHING; +END; +$$; + +comment on function pgautofailover.report_timeline_history(bigint,jsonb) + is 'reports a node''s own known timeline history (tli, parent tli, switchpoint LSN)'; + +grant execute on function + pgautofailover.report_timeline_history(bigint,jsonb) + to autoctl_node; + +-- Explicit operator resolution of a detected timeline fork: pins which +-- lineage is ground truth for a (formation, group), so the election's +-- ancestry filter uses it instead of auto-detecting (see +-- FilterNodesByTimelineAncestry() in timeline_history.c). Refuses to pin a +-- timeline nobody in the group has ever reported. +CREATE FUNCTION pgautofailover.accept_timeline + ( + IN formation_id text, + IN group_id int, + IN target_tli int, + IN decided_by text default null + ) +RETURNS bool LANGUAGE plpgsql SECURITY DEFINER +AS $$ +DECLARE + known bool; +BEGIN + SELECT EXISTS ( + SELECT 1 + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.formationid = formation_id + AND n.groupid = group_id + AND h.tli = target_tli + ) INTO known; + + IF NOT known THEN + RAISE EXCEPTION + 'timeline % has never been reported by any node in formation "%" group %', + target_tli, formation_id, group_id; + END IF; + + INSERT INTO pgautofailover.accepted_timeline + (formationid, groupid, accepted_tli, decided_by) + VALUES (formation_id, group_id, target_tli, decided_by); + + RETURN true; +END; +$$; + +comment on function pgautofailover.accept_timeline(text,int,int,text) + is 'pins the accepted timeline for a (formation, group) after an operator resolves a detected fork'; + +grant execute on function + pgautofailover.accept_timeline(text,int,int,text) + to autoctl_node; + +-- Marks the most recent unresolved accepted_timeline pin for a +-- (formation, group) as resolved, once a primary has been promoted on the +-- accepted lineage. Kept as a permanent audit record rather than deleted. +CREATE FUNCTION pgautofailover.resolve_accepted_timeline + ( + IN formation_id text, + IN group_id int + ) +RETURNS bool LANGUAGE SQL SECURITY DEFINER +AS $$ + UPDATE pgautofailover.accepted_timeline + SET resolved_at = now() + WHERE formationid = formation_id + AND groupid = group_id + AND resolved_at IS NULL; + + SELECT true; +$$; + +comment on function pgautofailover.resolve_accepted_timeline(text,int) + is 'marks the current accepted timeline pin resolved, once a primary has been promoted on it'; + +grant execute on function + pgautofailover.resolve_accepted_timeline(text,int) + to autoctl_node; + +-- Per-node status used by `pg_autoctl show timeline`: whether each node's +-- reported timeline is on the group's reference lineage (an operator's pin +-- in accepted_timeline if any, otherwise the branch containing the highest +-- reported tli in the group), the same rule +-- FilterNodesByTimelineAncestry() applies during an election, expressed +-- here as SQL for read-only reporting. +CREATE FUNCTION pgautofailover.node_timeline_status + ( + IN formation_id text, + IN group_id int, + OUT node_id bigint, + OUT node_name text, + OUT tli int, + OUT lsn pg_lsn, + OUT reference_tli int, + OUT on_accepted_lineage bool + ) +RETURNS SETOF record LANGUAGE SQL STRICT +AS $$ + WITH reference AS ( + SELECT COALESCE( + (SELECT accepted_tli + FROM pgautofailover.accepted_timeline + WHERE formationid = formation_id AND groupid = group_id + AND resolved_at IS NULL + ORDER BY decided_at DESC LIMIT 1), + (SELECT max(reportedtli) + FROM pgautofailover.node + WHERE formationid = formation_id AND groupid = group_id) + ) AS tli + ), + ancestry AS ( + WITH RECURSIVE chain AS ( + SELECT h.tli, h.parenttli + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.formationid = formation_id AND n.groupid = group_id + AND h.tli = (SELECT tli FROM reference) + UNION ALL + SELECT h.tli, h.parenttli + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + JOIN chain c ON h.tli = c.parenttli + WHERE n.formationid = formation_id AND n.groupid = group_id + ) + SELECT tli FROM chain + UNION + SELECT tli FROM reference + ) + SELECT n.nodeid, + n.nodename, + n.reportedtli, + n.reportedlsn, + (SELECT tli FROM reference), + (n.reportedtli IN (SELECT tli FROM ancestry)) + FROM pgautofailover.node n + WHERE n.formationid = formation_id AND n.groupid = group_id + ORDER BY n.nodeid; +$$; + +comment on function pgautofailover.node_timeline_status(text,int) + is 'per-node timeline ancestry status against the group''s reference lineage, for pg_autoctl show timeline'; + +grant execute on function + pgautofailover.node_timeline_status(text,int) + to autoctl_node; + + +-- +-- Fix fast_forward self-reference infinite loop and add a +-- guard_data_loss-aware health filter to get_most_advanced_standby() +-- (#1143, fixes #1060). +-- + +DROP FUNCTION IF EXISTS pgautofailover.get_most_advanced_standby(text, int); + +CREATE FUNCTION pgautofailover.get_most_advanced_standby + ( + IN formationid text default 'default', + IN groupid int default 0, + IN caller_node_id bigint default 0, + OUT node_id bigint, + OUT node_name text, + OUT node_host text, + OUT node_port int, + OUT node_lsn pg_lsn, + OUT node_is_primary bool + ) +RETURNS SETOF record LANGUAGE SQL STRICT +AS $$ + select nodeid, nodename, nodehost, nodeport, reportedlsn, false + from pgautofailover.node + where formationid = $1 + and groupid = $2 + and nodeid != $3 + and reportedstate = 'report_lsn' + and (current_setting('pgautofailover.guard_data_loss')::bool or health > 0) + order by reportedlsn desc, health desc + limit 1; +$$; + +grant execute on function pgautofailover.get_most_advanced_standby(text,int,bigint) + to autoctl_node; diff --git a/src/monitor/pgautofailover.sql b/src/monitor/pgautofailover.sql index 83ae1acb4..b7494bf65 100644 --- a/src/monitor/pgautofailover.sql +++ b/src/monitor/pgautofailover.sql @@ -198,6 +198,37 @@ CREATE TABLE pgautofailover.event PRIMARY KEY (eventid) ); +-- Every Postgres timeline has exactly one parent and one fork LSN, so the +-- union of every node's known history forms a single tree. Append-only: a +-- (tli, parenttli, switchpoint_lsn) triple never changes once recorded, and +-- is an objective fact about that timeline, not about which node reported +-- it -- nodeid is provenance (who told us), not part of what the row means. +CREATE TABLE pgautofailover.node_timeline_history + ( + nodeid bigint not null + references pgautofailover.node(nodeid) on delete cascade, + tli int not null, + parenttli int not null, + switchpoint_lsn pg_lsn not null, + + PRIMARY KEY (nodeid, tli) + ); + +-- An operator's explicit pin of which timeline is ground truth after a +-- detected fork. Normally empty: absent any unresolved row, the election +-- runs its ordinary auto-detection unchanged. +CREATE TABLE pgautofailover.accepted_timeline + ( + formationid text not null, + groupid int not null, + accepted_tli int not null, + decided_by text, + decided_at timestamptz not null default now(), + resolved_at timestamptz, + + PRIMARY KEY (formationid, groupid, decided_at) + ); + GRANT SELECT ON ALL TABLES IN SCHEMA pgautofailover TO autoctl_node; CREATE FUNCTION pgautofailover.set_node_system_identifier @@ -854,6 +885,178 @@ grant execute on function pgautofailover.report_postgres_version(bigint,int,text,text,text) to autoctl_node; +-- history is a JSON array of {"tli": int, "parenttli": int, +-- "switchpoint": text} objects, oldest first, as produced by the keeper's +-- timeline_history_to_json(). Called both periodically (whenever the local +-- timeline has advanced) and synchronously right before a report_lsn +-- restart, so this needs to be cheap and idempotent: an unconditional +-- ON CONFLICT DO NOTHING insert of already-known facts is a no-op. +CREATE FUNCTION pgautofailover.report_timeline_history + ( + IN node_id bigint, + IN history jsonb + ) +RETURNS void LANGUAGE plpgsql SECURITY DEFINER +AS $$ +BEGIN + INSERT INTO pgautofailover.node_timeline_history + (nodeid, tli, parenttli, switchpoint_lsn) + SELECT node_id, + (entry->>'tli')::int, + (entry->>'parenttli')::int, + (entry->>'switchpoint')::pg_lsn + FROM jsonb_array_elements(history) AS entry + ON CONFLICT (nodeid, tli) DO NOTHING; +END; +$$; + +comment on function pgautofailover.report_timeline_history(bigint,jsonb) + is 'reports a node''s own known timeline history (tli, parent tli, switchpoint LSN)'; + +grant execute on function + pgautofailover.report_timeline_history(bigint,jsonb) + to autoctl_node; + +-- Explicit operator resolution of a detected timeline fork: pins which +-- lineage is ground truth for a (formation, group), so the election's +-- ancestry filter uses it instead of auto-detecting (see +-- FilterNodesByTimelineAncestry() in timeline_history.c). Refuses to pin a +-- timeline nobody in the group has ever reported. +CREATE FUNCTION pgautofailover.accept_timeline + ( + IN formation_id text, + IN group_id int, + IN target_tli int, + IN decided_by text default null + ) +RETURNS bool LANGUAGE plpgsql SECURITY DEFINER +AS $$ +DECLARE + known bool; +BEGIN + SELECT EXISTS ( + SELECT 1 + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.formationid = formation_id + AND n.groupid = group_id + AND h.tli = target_tli + ) INTO known; + + IF NOT known THEN + RAISE EXCEPTION + 'timeline % has never been reported by any node in formation "%" group %', + target_tli, formation_id, group_id; + END IF; + + INSERT INTO pgautofailover.accepted_timeline + (formationid, groupid, accepted_tli, decided_by) + VALUES (formation_id, group_id, target_tli, decided_by); + + RETURN true; +END; +$$; + +comment on function pgautofailover.accept_timeline(text,int,int,text) + is 'pins the accepted timeline for a (formation, group) after an operator resolves a detected fork'; + +grant execute on function + pgautofailover.accept_timeline(text,int,int,text) + to autoctl_node; + +-- Marks the most recent unresolved accepted_timeline pin for a +-- (formation, group) as resolved, once a primary has been promoted on the +-- accepted lineage. Kept as a permanent audit record rather than deleted. +CREATE FUNCTION pgautofailover.resolve_accepted_timeline + ( + IN formation_id text, + IN group_id int + ) +RETURNS bool LANGUAGE SQL SECURITY DEFINER +AS $$ + UPDATE pgautofailover.accepted_timeline + SET resolved_at = now() + WHERE formationid = formation_id + AND groupid = group_id + AND resolved_at IS NULL; + + SELECT true; +$$; + +comment on function pgautofailover.resolve_accepted_timeline(text,int) + is 'marks the current accepted timeline pin resolved, once a primary has been promoted on it'; + +grant execute on function + pgautofailover.resolve_accepted_timeline(text,int) + to autoctl_node; + +-- Per-node status used by `pg_autoctl show timeline`: whether each node's +-- reported timeline is on the group's reference lineage (an operator's pin +-- in accepted_timeline if any, otherwise the branch containing the highest +-- reported tli in the group), the same rule +-- FilterNodesByTimelineAncestry() applies during an election, expressed +-- here as SQL for read-only reporting. +CREATE FUNCTION pgautofailover.node_timeline_status + ( + IN formation_id text, + IN group_id int, + OUT node_id bigint, + OUT node_name text, + OUT tli int, + OUT lsn pg_lsn, + OUT reference_tli int, + OUT on_accepted_lineage bool + ) +RETURNS SETOF record LANGUAGE SQL STRICT +AS $$ + WITH reference AS ( + SELECT COALESCE( + (SELECT accepted_tli + FROM pgautofailover.accepted_timeline + WHERE formationid = formation_id AND groupid = group_id + AND resolved_at IS NULL + ORDER BY decided_at DESC LIMIT 1), + (SELECT max(reportedtli) + FROM pgautofailover.node + WHERE formationid = formation_id AND groupid = group_id) + ) AS tli + ), + ancestry AS ( + WITH RECURSIVE chain AS ( + SELECT h.tli, h.parenttli + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.formationid = formation_id AND n.groupid = group_id + AND h.tli = (SELECT tli FROM reference) + UNION ALL + SELECT h.tli, h.parenttli + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + JOIN chain c ON h.tli = c.parenttli + WHERE n.formationid = formation_id AND n.groupid = group_id + ) + SELECT tli FROM chain + UNION + SELECT tli FROM reference + ) + SELECT n.nodeid, + n.nodename, + n.reportedtli, + n.reportedlsn, + (SELECT tli FROM reference), + (n.reportedtli IN (SELECT tli FROM ancestry)) + FROM pgautofailover.node n + WHERE n.formationid = formation_id AND n.groupid = group_id + ORDER BY n.nodeid; +$$; + +comment on function pgautofailover.node_timeline_status(text,int) + is 'per-node timeline ancestry status against the group''s reference lineage, for pg_autoctl show timeline'; + +grant execute on function + pgautofailover.node_timeline_status(text,int) + to autoctl_node; + create function pgautofailover.synchronous_standby_names ( diff --git a/src/monitor/regress_schedule b/src/monitor/regress_schedule index 1f9952ffe..d09d29af6 100644 --- a/src/monitor/regress_schedule +++ b/src/monitor/regress_schedule @@ -39,6 +39,7 @@ test: fast_forward test: drop_node test: stale_primary_report test: lock_and_fetch_migration +test: timeline_fork_detection test: dummy_update test: drop_extension test: upgrade diff --git a/src/monitor/sql/timeline_fork_detection.sql b/src/monitor/sql/timeline_fork_detection.sql new file mode 100644 index 000000000..67b987789 --- /dev/null +++ b/src/monitor/sql/timeline_fork_detection.sql @@ -0,0 +1,319 @@ +-- Copyright (c) Microsoft Corporation. All rights reserved. +-- Licensed under the PostgreSQL License. +-- +-- Regression tests for timeline-fork detection: node_timeline_history, +-- report_timeline_history(), accept_timeline(), resolve_accepted_timeline(), +-- node_timeline_status(), and the election's ancestry filter +-- (FilterNodesByTimelineAncestry(), called from ProceedGroupStateForMSFailover). + +\x on + +-- ── Part A: table + function unit tests ───────────────────────────────────── +-- +-- Three nodes are enough to exercise report_timeline_history(), +-- node_timeline_status(), accept_timeline(), and resolve_accepted_timeline() +-- directly, without driving the full FSM: reportedtli/reportedlsn are set by +-- hand, the same way other regress tests here manufacture node state +-- (see guard_data_loss.sql). + +SELECT pgautofailover.create_formation('tlf_unit', 'pgsql', 'postgres', true, 1); + +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-p', 5432, + 'postgres', 'p', 1); + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 'p' \gset + +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s1', 5432, + 'postgres', 's1', 1); + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's1' \gset + +SELECT * + FROM pgautofailover.register_node('tlf_unit', 'tlfu-s2', 5432, + 'postgres', 's2', 1); + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_unit' AND nodename = 's2' \gset + +-- p stayed on the original timeline (tli=1). s1 was promoted onto tli=2 at +-- some point in the past. s2 was promoted onto tli=3 -- a sibling branch of +-- s1's tli=2, both children of tli=1 (a genuine fork: two nodes each think +-- they are the continuation). + +SELECT pgautofailover.report_timeline_history(:np, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); + +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); + +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); + +-- report_timeline_history() must be idempotent: re-reporting the exact same +-- facts (or a subset of them) must not create duplicate or conflicting rows. +SELECT pgautofailover.report_timeline_history(:ns1, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}]'::jsonb); + +SELECT nodeid, tli, parenttli, switchpoint_lsn + FROM pgautofailover.node_timeline_history + ORDER BY nodeid, tli; + +UPDATE pgautofailover.node SET reportedtli = 1, reportedlsn = '0/5500' + WHERE nodeid = :np; +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns1; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns2; + +-- No operator pin yet: reference_tli defaults to the highest reportedtli in +-- the group (3, from s2). s1 (tli=2) is a sibling of s2's branch, not an +-- ancestor of it -- on_accepted_lineage must be false. p (tli=1) and s2 +-- (tli=3, the reference itself) must both be true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; + +-- accept_timeline() must refuse to pin a timeline nobody in the group has +-- ever reported. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 99, 'operator typo'); + +-- Pin tli=2 (s1's branch) as ground truth instead of the auto-detected tli=3. +SELECT pgautofailover.accept_timeline('tlf_unit', 0, 2, 'operator override'); + +-- The pin flips the reference: now s1 (tli=2) is on_accepted_lineage, and s2 +-- (tli=3, the sibling the operator rejected) is not. p (tli=1) is still an +-- ancestor of tli=2, so it stays true. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; + +SELECT resolve_accepted_timeline + FROM pgautofailover.resolve_accepted_timeline('tlf_unit', 0); + +SELECT formationid, groupid, accepted_tli, decided_by, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_unit' AND groupid = 0; + +-- Once resolved, the pin no longer applies: reference_tli reverts to the +-- auto-detected max (3), same as before the pin was ever set. +SELECT node_name, tli, reference_tli, on_accepted_lineage + FROM pgautofailover.node_timeline_status('tlf_unit', 0) + ORDER BY node_name; + + +-- ── Part B: election-level test ────────────────────────────────────────────── +-- +-- Drive a real 4-node group (primary + 3 standbys) through the FSM, then +-- kill the primary and simulate two standbys that have each, at some earlier +-- point, been promoted onto their own diverging branch -- a genuine fork. +-- The operator pins one branch as ground truth *before* calling +-- perform_failover(); the election must pick the candidate on the pinned +-- lineage and exclude the other, even though the excluded one has a more +-- advanced (higher) timeline number. + +SELECT pgautofailover.create_formation('tlf_election', 'pgsql', 'postgres', true, 1); + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-p', 5432, + 'postgres', 'p', 1); + +SELECT nodeid AS np FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 'p' \gset + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s1', 5432, + 'postgres', 's1', 1); + +SELECT nodeid AS ns1 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's1' \gset + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s2', 5432, + 'postgres', 's2', 1); + +SELECT nodeid AS ns2 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's2' \gset + +SELECT * + FROM pgautofailover.register_node('tlf_election', 'tlfe-s3', 5432, + 'postgres', 's3', 1); + +SELECT nodeid AS ns3 FROM pgautofailover.node + WHERE formationid = 'tlf_election' AND nodename = 's3' \gset + +-- ── bootstrap: p primary, s1/s2/s3 secondary ──────────────────────────────── + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'single', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'wait_primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns1, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns2, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'wait_standby'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'catchingup', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :ns3, 0, + current_group_role => 'secondary', + current_lsn => '0/5000'); + +SELECT assigned_group_state + FROM pgautofailover.node_active('tlf_election', :np, 0, + current_group_role => 'primary', + current_lsn => '0/5000'); + +-- Verify final formation state: p=primary, s1/s2/s3=secondary. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; + +-- ── publish two diverging branches ────────────────────────────────────────── +-- +-- s2 was, at some earlier point, promoted onto tli=2. s3 was promoted onto +-- tli=3, a sibling branch, both children of tli=1. s1 stayed on the original +-- tli=1. + +SELECT pgautofailover.report_timeline_history(:ns2, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":2,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); + +SELECT pgautofailover.report_timeline_history(:ns3, + '[{"tli":1,"parenttli":0,"switchpoint":"0/0"}, + {"tli":3,"parenttli":1,"switchpoint":"0/6000"}]'::jsonb); + +UPDATE pgautofailover.node SET reportedtli = 2, reportedlsn = '0/6500' + WHERE nodeid = :ns2; +UPDATE pgautofailover.node SET reportedtli = 3, reportedlsn = '0/7500' + WHERE nodeid = :ns3; + +-- ── operator pins s2's branch (tli=2) as ground truth ─────────────────────── +-- +-- Left to auto-detection this group would pick tli=3 (the highest reported +-- tli, s3's branch). The operator instead pins tli=2: the election must +-- honor that and pick s2, excluding s3 even though s3's tli number is +-- higher. + +SELECT pgautofailover.accept_timeline('tlf_election', 0, 2, 'operator override'); + +-- ── simulate primary death, standbys already at report_lsn ───────────────── + +SET pgautofailover.startup_grace_period = 1; + +UPDATE pgautofailover.node + SET health = 0, + healthchecktime = now(), + reporttime = now() - interval '60 seconds' + WHERE formationid = 'tlf_election' AND nodename = 'p'; + +UPDATE pgautofailover.node + SET goalstate = 'draining', reportedstate = 'draining' + WHERE formationid = 'tlf_election' AND nodename = 'p'; + +UPDATE pgautofailover.node + SET goalstate = 'report_lsn', reportedstate = 'report_lsn' + WHERE formationid = 'tlf_election' AND nodename IN ('s1', 's2', 's3'); + +SELECT nodename, goalstate, reportedstate, reportedtli, reportedlsn + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; + +-- The dead primary itself always counts as a "missing" quorum node (it +-- can never report_lsn), which trips guard_data_loss regardless of the +-- ancestry filter under test here; disable it, same as guard_data_loss.sql +-- does for its own "proceed despite a missing report" scenario. +SET pgautofailover.guard_data_loss TO false; + +SELECT pgautofailover.perform_failover('tlf_election', 0); + +-- s2 (the pinned lineage, tli=2) must be the promoted candidate. s3 (tli=3, +-- the higher-numbered sibling branch) must be excluded and left untouched +-- in report_lsn. s1 (tli=1, an ancestor of the pinned tli=2) remains a +-- comparable standby but was less advanced than s2, so it is not selected +-- either. +SELECT nodename, goalstate, reportedstate + FROM pgautofailover.node + WHERE formationid = 'tlf_election' + ORDER BY nodename; + +-- PromoteSelectedNode() must auto-resolve the operator's pin once a primary +-- has been promoted on the accepted lineage. +SELECT formationid, groupid, accepted_tli, resolved_at IS NOT NULL AS resolved + FROM pgautofailover.accepted_timeline + WHERE formationid = 'tlf_election' AND groupid = 0; + +RESET pgautofailover.guard_data_loss; +RESET pgautofailover.startup_grace_period; diff --git a/src/monitor/timeline_history.c b/src/monitor/timeline_history.c new file mode 100644 index 000000000..4f7af0e08 --- /dev/null +++ b/src/monitor/timeline_history.c @@ -0,0 +1,377 @@ +/*------------------------------------------------------------------------- + * + * src/monitor/timeline_history.c + * + * Ancestry-aware comparisons over pgautofailover.node_timeline_history. + * + * Every Postgres timeline has exactly one parent and one fork LSN, so the + * union of every node's known history forms a single tree (in the normal, + * non-split-brain case, a single chain). This lets the election tell + * "genuinely comparable, just behind" apart from "on a forked, incomparable + * timeline" without any node needing to contact any other node -- there is + * no elected primary at report_lsn time to check against, which is exactly + * why this has to live here, centrally, on the monitor. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + * + *------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "fmgr.h" +#include "funcapi.h" +#include "miscadmin.h" +#include "nodes/pg_list.h" +#include "utils/builtins.h" + +#include "access/xact.h" +#include "access/xlogdefs.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "utils/pg_lsn.h" +#include "utils/timestamp.h" + +#include "node_metadata.h" +#include "notifications.h" +#include "timeline_history.h" + + +/* + * FetchGroupTimelineHistory returns the union of every node's reported + * timeline history for the given (formation, group), as a list of + * TimelineHistoryEdge, deduplicated: honest nodes always agree on the facts + * (a timeline's parent and switchpoint are physical properties of that + * timeline, not opinions), so a plain DISTINCT collapses every node's + * repeated report of the same edge into one. + */ +List * +FetchGroupTimelineHistory(char *formationId, int groupId) +{ + List *history = NIL; + MemoryContext callerContext = CurrentMemoryContext; + + Oid argTypes[] = { TEXTOID, INT4OID }; + Datum argValues[] = { + CStringGetTextDatum(formationId), + Int32GetDatum(groupId) + }; + const int argCount = sizeof(argValues) / sizeof(argValues[0]); + + const char *selectQuery = + "SELECT DISTINCT h.tli, h.parenttli, h.switchpoint_lsn " + " FROM pgautofailover.node_timeline_history h " + " JOIN pgautofailover.node n ON n.nodeid = h.nodeid " + " WHERE n.formationid = $1 AND n.groupid = $2"; + + SPI_connect(); + + int spiStatus = SPI_execute_with_args(selectQuery, argCount, argTypes, + argValues, NULL, false, 0); + + if (spiStatus != SPI_OK_SELECT) + { + elog(ERROR, "could not select from pgautofailover.node_timeline_history"); + } + + MemoryContext spiContext = MemoryContextSwitchTo(callerContext); + + for (uint64 rowNumber = 0; rowNumber < SPI_processed; rowNumber++) + { + HeapTuple heapTuple = SPI_tuptable->vals[rowNumber]; + TupleDesc tupdesc = SPI_tuptable->tupdesc; + bool isNull = false; + + TimelineHistoryEdge *edge = + (TimelineHistoryEdge *) palloc0(sizeof(TimelineHistoryEdge)); + + edge->tli = DatumGetInt32( + SPI_getbinval(heapTuple, tupdesc, 1, &isNull)); + edge->parentTli = DatumGetInt32( + SPI_getbinval(heapTuple, tupdesc, 2, &isNull)); + edge->switchpointLSN = DatumGetLSN( + SPI_getbinval(heapTuple, tupdesc, 3, &isNull)); + + history = lappend(history, edge); + } + + MemoryContextSwitchTo(spiContext); + + SPI_finish(); + + return history; +} + + +/* + * FindTimelineHistoryEdge returns the edge describing the given tli, or + * NULL when it doesn't appear in history at all (unknown ancestry). + */ +static TimelineHistoryEdge * +FindTimelineHistoryEdge(List *history, int tli) +{ + ListCell *cell = NULL; + + foreach(cell, history) + { + TimelineHistoryEdge *edge = (TimelineHistoryEdge *) lfirst(cell); + + if (edge->tli == tli) + { + return edge; + } + } + + return NULL; +} + + +/* + * TimelineIsAncestor returns true when candidateTli is referenceTli itself, + * or a genuine ancestor of it: walking referenceTli's parent chain (via + * history) reaches candidateTli. When it returns true and outSwitchpoint is + * non-NULL, *outSwitchpoint is set to the LSN at which candidateTli was + * superseded (InvalidXLogRecPtr when candidateTli == referenceTli, i.e. + * there's no switchpoint to speak of). + * + * Postgres timelines are created by promotion, always numbered higher than + * their parent, so walking strictly decreasing tli values can never cycle; + * the list_length(history) bound below is a defensive backstop against + * malformed data, not something normal operation can reach. + */ +bool +TimelineIsAncestor(List *history, int candidateTli, int referenceTli, + XLogRecPtr *outSwitchpoint) +{ + if (candidateTli == referenceTli) + { + if (outSwitchpoint != NULL) + { + *outSwitchpoint = InvalidXLogRecPtr; + } + + return true; + } + + int currentTli = referenceTli; + int maxSteps = list_length(history) + 1; + + for (int steps = 0; steps < maxSteps; steps++) + { + TimelineHistoryEdge *edge = FindTimelineHistoryEdge(history, currentTli); + + if (edge == NULL) + { + /* we don't know currentTli's ancestry beyond this point */ + return false; + } + + if (edge->parentTli == candidateTli) + { + if (outSwitchpoint != NULL) + { + *outSwitchpoint = edge->switchpointLSN; + } + + return true; + } + + if (edge->parentTli <= 0 || edge->parentTli >= currentTli) + { + /* reached the root, or malformed/cyclic data: stop */ + return false; + } + + currentTli = edge->parentTli; + } + + return false; +} + + +/* + * GetAcceptedTimeline returns the currently pinned tli for the given + * (formation, group), from the most recent unresolved row in + * pgautofailover.accepted_timeline, or 0 when there is none (the normal + * case, always, absent an active fork). + */ +int +GetAcceptedTimeline(char *formationId, int groupId) +{ + int acceptedTli = 0; + MemoryContext callerContext = CurrentMemoryContext; + + Oid argTypes[] = { TEXTOID, INT4OID }; + Datum argValues[] = { + CStringGetTextDatum(formationId), + Int32GetDatum(groupId) + }; + const int argCount = sizeof(argValues) / sizeof(argValues[0]); + + const char *selectQuery = + "SELECT accepted_tli " + " FROM pgautofailover.accepted_timeline " + " WHERE formationid = $1 AND groupid = $2 AND resolved_at IS NULL " + " ORDER BY decided_at DESC " + " LIMIT 1"; + + SPI_connect(); + + int spiStatus = SPI_execute_with_args(selectQuery, argCount, argTypes, + argValues, NULL, false, 0); + + if (spiStatus != SPI_OK_SELECT) + { + elog(ERROR, "could not select from pgautofailover.accepted_timeline"); + } + + MemoryContext spiContext = MemoryContextSwitchTo(callerContext); + + if (SPI_processed > 0) + { + bool isNull = false; + HeapTuple heapTuple = SPI_tuptable->vals[0]; + + acceptedTli = DatumGetInt32( + SPI_getbinval(heapTuple, SPI_tuptable->tupdesc, 1, &isNull)); + + if (isNull) + { + acceptedTli = 0; + } + } + + MemoryContextSwitchTo(spiContext); + + SPI_finish(); + + return acceptedTli; +} + + +/* + * ResolveAcceptedTimeline marks the current unresolved accepted_timeline + * pin, if any, resolved for the given (formation, group). Called once a + * primary has been selected for promotion on the (now filtered, so + * necessarily on the accepted or auto-detected lineage) candidate pool: a + * no-op when there was no pin to begin with, which is the normal case. + */ +void +ResolveAcceptedTimeline(char *formationId, int groupId) +{ + Oid argTypes[] = { TEXTOID, INT4OID }; + Datum argValues[] = { + CStringGetTextDatum(formationId), + Int32GetDatum(groupId) + }; + const int argCount = sizeof(argValues) / sizeof(argValues[0]); + + const char *updateQuery = + "UPDATE pgautofailover.accepted_timeline " + " SET resolved_at = now() " + " WHERE formationid = $1 AND groupid = $2 AND resolved_at IS NULL"; + + SPI_connect(); + + int spiStatus = SPI_execute_with_args(updateQuery, argCount, argTypes, + argValues, NULL, false, 0); + + if (spiStatus != SPI_OK_UPDATE) + { + elog(ERROR, "could not update pgautofailover.accepted_timeline"); + } + + SPI_finish(); +} + + +/* + * FilterNodesByTimelineAncestry returns the subset of nodeList whose + * reportedTLI is genuinely comparable to the group's reference lineage -- + * either pinned explicitly via pgautofailover.accepted_timeline, or, in the + * normal case (no pin), auto-detected as the branch containing the highest + * reportedTLI among nodeList itself (the most recently promoted-to + * timeline is the most likely real continuation). + * + * Nodes outside that lineage are excluded, not deprioritized: they simply + * don't appear in the returned list, exactly as if they hadn't reported at + * all. This is deliberate -- a node that has genuinely diverged can never + * become comparable no matter how long the election waits for it, so + * counting it as "missing" would block the group forever; excluding it + * lets the remaining, genuinely comparable candidates proceed normally + * (see #683). If excluding a diverged node happens to leave zero + * candidates, the caller's existing "not enough candidates yet" handling + * already does the right, graceful thing. + * + * *outReferenceTli is set to the reference tli used (0 when nodeList had no + * reported tli to work from at all, in which case the input is returned + * unfiltered -- there's nothing to filter against yet). + */ +List * +FilterNodesByTimelineAncestry(List *nodeList, char *formationId, int groupId, + int *outReferenceTli) +{ + ListCell *cell = NULL; + int referenceTli = GetAcceptedTimeline(formationId, groupId); + + if (referenceTli == 0) + { + foreach(cell, nodeList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(cell); + + if (node->reportedTLI > referenceTli) + { + referenceTli = node->reportedTLI; + } + } + } + + if (outReferenceTli != NULL) + { + *outReferenceTli = referenceTli; + } + + if (referenceTli == 0) + { + /* nobody has reported a timeline yet: nothing to filter against */ + return nodeList; + } + + List *history = FetchGroupTimelineHistory(formationId, groupId); + List *filteredList = NIL; + + foreach(cell, nodeList) + { + AutoFailoverNode *node = (AutoFailoverNode *) lfirst(cell); + + if (node->reportedTLI <= 0) + { + /* hasn't reported a timeline yet, don't exclude on that basis */ + filteredList = lappend(filteredList, node); + continue; + } + + if (TimelineIsAncestor(history, node->reportedTLI, referenceTli, NULL)) + { + filteredList = lappend(filteredList, node); + continue; + } + + char message[BUFSIZE] = { 0 }; + + LogAndNotifyMessage( + message, BUFSIZE, + "Excluding " NODE_FORMAT + " from the election: its reported timeline %d does not appear " + "to be an ancestor of the group's reference timeline %d -- " + "it may have diverged and could need pg_rewind; see " + "`pg_autoctl show timeline` and `pg_autoctl accept timeline`", + NODE_FORMAT_ARGS(node), + node->reportedTLI, + referenceTli); + } + + return filteredList; +} diff --git a/src/monitor/timeline_history.h b/src/monitor/timeline_history.h new file mode 100644 index 000000000..f0259aed2 --- /dev/null +++ b/src/monitor/timeline_history.h @@ -0,0 +1,37 @@ +/* + * src/monitor/timeline_history.h + * Ancestry-aware comparisons over pgautofailover.node_timeline_history, + * used by the election to tell "genuinely comparable, just behind" apart + * from "on a forked, incomparable timeline" -- see group_state_machine.c. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + * + */ + +#pragma once + +#include "access/xlogdefs.h" +#include "nodes/pg_list.h" + +#include "node_metadata.h" + +typedef struct TimelineHistoryEdge +{ + int tli; + int parentTli; + XLogRecPtr switchpointLSN; +} TimelineHistoryEdge; + +extern List * FetchGroupTimelineHistory(char *formationId, int groupId); + +extern bool TimelineIsAncestor(List *history, int candidateTli, + int referenceTli, XLogRecPtr *outSwitchpoint); + +extern int GetAcceptedTimeline(char *formationId, int groupId); + +extern void ResolveAcceptedTimeline(char *formationId, int groupId); + +extern List * FilterNodesByTimelineAncestry(List *nodeList, + char *formationId, int groupId, + int *outReferenceTli); diff --git a/tests/pgautofailover_utils.py b/tests/pgautofailover_utils.py index 9e235279e..b1a11c185 100644 --- a/tests/pgautofailover_utils.py +++ b/tests/pgautofailover_utils.py @@ -1496,7 +1496,26 @@ def destroy( Cleans up processes and files created for this data node. """ - self.stop_pg_autoctl() + try: + # SIGINT (unlike the stop_pg_autoctl() default of SIGTERM) skips + # pg_autoctl's graceful-shutdown maintenance handoff: there's no + # standby to protect when we're about to tear down the whole + # cluster anyway, and that handoff can legitimately take up to + # KEEPER_MAINTENANCE_SHUTDOWN_LOOP_MAX_SECS + + # KEEPER_SHUTDOWN_LOOP_MAX_SECS (60s total, see defaults.h) -- + # right up against this module's own COMMAND_TIMEOUT (60s), with + # no margin for overhead. A node that hits that ceiling used to + # raise TimeoutExpired straight out of this call (uncaught, since + # it ran before the try/except below), aborting the rest of + # Cluster.destroy()'s loop and leaking every remaining node plus + # the monitor as orphaned processes for the calling test to hang + # on. + self.stop_pg_autoctl(sig=signal.SIGINT) + except Exception as e: + if ignore_failure: + print(str(e)) + else: + raise flags = ["--destroy"] if force: @@ -2016,11 +2035,20 @@ def destroy(self): Cleans up processes and files created for this monitor node. """ if self.pg_autoctl: - out, err, ret = self.pg_autoctl.stop() - - if ret != 0: - print() - print("Monitor logs:\n%s\n%s\n" % (out, err)) + try: + # see Datanode.destroy()'s use of SIGINT for the rationale; + # the monitor doesn't run the keeper FSM so it isn't subject + # to the same graceful-maintenance delay, but using the same + # signal here keeps teardown intent consistent, and catching + # the call keeps a slow/failed stop from aborting the rest of + # Cluster.destroy() (in particular self.vlan.destroy()). + out, err, ret = self.pg_autoctl.stop(signal.SIGINT) + + if ret != 0: + print() + print("Monitor logs:\n%s\n%s\n" % (out, err)) + except Exception as e: + print(str(e)) try: destroy = PGAutoCtl(self) diff --git a/tests/tap/schedule b/tests/tap/schedule index 23a7f4ec4..f7943bc96 100644 --- a/tests/tap/schedule +++ b/tests/tap/schedule @@ -7,10 +7,12 @@ basic_operation basic_operation_listen_flag maintenance_and_drop create_standby_with_pgdata +launch_deferred_set_metadata ensure monitor_disabled replace_monitor config_get_set +postgres_version_tracking skip_pg_hba #debian_clusters auth @@ -25,7 +27,10 @@ multi_alternate guard_data_loss replication_stall_3dc fast_forward +demote_timeout_wait_primary_deadlock +timeline_fork_report_lsn_deadlock extension_update +tablespaces installcheck # Upgrade test — requires pgaf:current and pgaf:next images to be pre-built: # make -C tests/upgrade pgaf-current pgaf-next diff --git a/tests/tap/schedules/multi-misc.sch b/tests/tap/schedules/multi-misc.sch index 1e4097e0b..a2439f018 100644 --- a/tests/tap/schedules/multi-misc.sch +++ b/tests/tap/schedules/multi-misc.sch @@ -1,6 +1,14 @@ -# Multi-node misc: standbys, maintenance, ensure, network partition, drop node (~9 min) +# Multi-node misc: standbys, maintenance, ensure, network partition, drop node, +# quorum/candidate-selection edge cases (~9 min) +# +# fast_forward is deliberately NOT included here: making it actually +# exercise the fast_forward FSM state surfaced an unbounded hang in +# standby_fetch_missing_wal() (no timeout / liveness check on the WAL +# source) that's pre-existing and unrelated to this branch. Tracked for a +# separate branch/PR. multi_standbys multi_maintenance ensure multi_ifdown drop_node_destroy +guard_data_loss diff --git a/tests/tap/schedules/node.sch b/tests/tap/schedules/node.sch index a000daff7..699585d79 100644 --- a/tests/tap/schedules/node.sch +++ b/tests/tap/schedules/node.sch @@ -10,3 +10,7 @@ replace_monitor extension_update debian_clusters tablespaces +replication_stall_3dc +demote_timeout_wait_primary_deadlock +timeline_fork_report_lsn_deadlock +timeline_fork_3node_auto_detect diff --git a/tests/tap/specs/debug_citus_worker_fast_forward.pgaf b/tests/tap/specs/debug_citus_worker_fast_forward.pgaf new file mode 100644 index 000000000..cc8462808 --- /dev/null +++ b/tests/tap/specs/debug_citus_worker_fast_forward.pgaf @@ -0,0 +1,93 @@ +# Diagnostic spec to reproduce the fast_forward + restart corruption bug on +# the Citus worker path, mirroring tests/tap/specs/multi_ifdown.pgaf's +# non-Citus repro of the same class of bug. +# +# Root cause under investigation: fsm_citus_cleanup_and_resume_as_primary() +# (src/bin/pg_autoctl/fsm_transition_citus.c) is the Citus-node counterpart +# of fsm_cleanup_as_primary() (src/bin/pg_autoctl/fsm_transition.c), both +# firing at the FAST_FORWARD_STATE -> PREP_PROMOTION_STATE transition, i.e. +# BEFORE the real pg_ctl promote (which only happens later, from +# standby_promote() at PREP_PROMOTION_STATE -> STOP_REPLICATION_STATE via +# fsm_promote_standby). +# +# The non-Citus path was already fixed: standby_cleanup_as_primary() no +# longer restarts Postgres (see its docstring in primary_standby.c), because +# restarting right after standby.signal is removed -- but before a genuine +# promotion -- makes Postgres boot as an ordinary (non-standby) server via +# plain crash recovery, reaching a not-in-recovery state on its OLD timeline +# without ever genuinely promoting. +# +# fsm_citus_cleanup_and_resume_as_primary() calls standby_cleanup_as_primary() +# and then unconditionally does its OWN explicit keeper_restart_postgres() +# call right after -- the exact same unsafe restart, just not living inside +# the shared function anymore. This spec sets up a 3-node Citus worker group +# (worker1a primary, worker1b candidate-priority 0 kept always connected as +# the fast_forward source, worker1c candidate-priority 90 disconnected then +# reconnected while behind) so that failing worker1a over to worker1c forces +# worker1c through fast_forward (catching up from worker1b) before promotion. + +cluster { + monitor + formation { + coordinator1a coordinator + worker1a worker group 1 + worker1b worker group 1 + worker1c worker group 1 + } +} + +setup { + wait until primary, secondary in group 0 timeout 90s + wait until primary, secondary in group 1 timeout 90s + promote worker1a +} + +teardown { + compose down +} + +step test_001_init { + wait until worker1a state is primary + and worker1b state is secondary + and worker1c state is secondary + timeout 90s + exec worker1a sh -c 'for i in $(seq 1 30); do n=$(psql -U docker -d demo -tAc "SELECT count(*) FROM pg_dist_node WHERE metadatasynced = false AND isactive = true"); [ "$n" = "0" ] && exit 0; sleep 2; done; exit 1' + sql coordinator1a { ALTER DATABASE demo SET citus.shard_count TO 4; } + sql coordinator1a { CREATE TABLE t1 (a int); } + sql coordinator1a { SELECT create_distributed_table('t1', 'a'); } + sql coordinator1a { INSERT INTO t1 VALUES (1), (2); } +} + +step test_002_set_candidate_priorities { + exec worker1a pg_autoctl set node candidate-priority 90 + exec worker1b pg_autoctl set node candidate-priority 0 + exec worker1c pg_autoctl set node candidate-priority 90 + wait until worker1a state = primary + and worker1b state = secondary + and worker1c state = secondary + timeout 60s +} + +step test_003_ifdown_worker1c { + network disconnect worker1c +} + +step test_004_insert_rows { + sql coordinator1a { INSERT INTO t1 SELECT x + 10 FROM generate_series(1, 1000) AS gs(x); } +} + +step test_005_failover { + network disconnect worker1a + network connect worker1c + wait until worker1b state is secondary + and worker1c state is primary + timeout 300s +} + +step test_006_verify_timeline_and_data { + exec worker1c bash -c "psql -d demo -tAc 'select pg_is_in_recovery()'" + expect { f } + exec worker1c bash -c "psql -d demo -tAc 'select timeline_id from pg_control_checkpoint()'" + sql coordinator1a { SELECT count(*) FROM t1; } + expect { 1002 } +} diff --git a/tests/tap/specs/timeline_fork_3node_auto_detect.pgaf b/tests/tap/specs/timeline_fork_3node_auto_detect.pgaf new file mode 100644 index 000000000..065d654cc --- /dev/null +++ b/tests/tap/specs/timeline_fork_3node_auto_detect.pgaf @@ -0,0 +1,177 @@ +# Companion to timeline_fork_report_lsn_deadlock.pgaf's 2-node scenario. +# +# The natural assumption is that a 3rd node fixes the auto-detection blind +# spot documented in that spec's header ("no sibling to disagree with the +# forked node"), since now there IS a sibling. That assumption is WRONG, and +# demonstrating why is this spec's actual point. +# +# FilterNodesByTimelineAncestry()'s auto-detection heuristic (no operator +# pin) is "the reference lineage is whichever branch contains the highest +# reported tli". It only excludes a candidate when a genuinely COMPETING +# branch is reported by someone else -- two nodes both diverging from the +# same point onto two DIFFERENT tlis. A sibling that simply stays behind, +# never advancing past the original timeline, isn't competing with +# anything: node3 forking onto tli=2 while node1/node2 stay on tli=1 is, +# structurally, indistinguishable from node3 having legitimately been +# promoted past two normal, honestly-lagging standbys -- tli=1 really is +# node3's own recorded parent, so it passes the ancestry check as tli=2's +# ancestor. More nodes alone doesn't help; only a genuinely competing +# report would. `pg_autoctl accept timeline` is still required here, same +# as the 2-node case -- this spec exercises that explicitly instead of +# leaving it as a documented-but-unexercised limitation. +# +# Once pinned, recovery is now automatic and immediate: the monitor's +# ProceedGroupState() detects the ancestry mismatch on node3's very next +# report and pushes it to catchingup by itself -- no enable/disable +# maintenance step, no incidental health-check cycle to wait on. This +# spec exercises that directly (test_005), not the older maintenance- +# forced path. +# +# Same fork-manufacturing technique as timeline_fork_report_lsn_deadlock.pgaf +# (network disconnect, pg_ctl promote, local writes), applied to node3 +# instead of node2, with node1 (primary) and node2 (secondary) continuing +# to stream normally throughout. + +cluster { + monitor + ssl off + formation { + node1 + node2 + node3 + } +} + +setup { + wait until primary, secondary timeout 120s + wait until node2 state is secondary + and node3 state is secondary + timeout 60s + promote node1 + # The application keeps writing to the primary throughout this whole + # spec -- a forked standby elsewhere in the formation has no reason to + # pause traffic on the main system, and a doc example that freezes + # node1/node2 while only node3 does anything is misleading about what + # this actually looks like in production. + sql node1 { CREATE TABLE app_data(a int); } +} + +teardown { + compose down +} + +step test_001_fork_node3_out_of_band { + wait until node1 state is primary + and node2 state is secondary + and node3 state is secondary + timeout 60s + # ordinary application traffic, already flowing before node3 forks + sql node1 { INSERT INTO app_data SELECT generate_series(1, 50); } + network disconnect node3 + exec node3 pg_ctl promote --pgdata /var/lib/postgres/pgaf + sleep 3s + # more application traffic while node3 is isolated and forking -- the + # main system doesn't know or care that node3 is off doing its own + # thing + sql node1 { INSERT INTO app_data SELECT generate_series(51, 100); } + # node3's synchronous_standby_names is still the 3-node-formation value + # (set on every node's local config, not only the primary's, so it's + # ready to serve if promoted for real) -- ANY 1 (..., ...). Now that + # it's isolated and standalone, no write could ever satisfy that + # quorum, so ordinary writes would just hang forever. A real admin (or + # buggy tooling) forcing writes on an isolated, out-of-band-promoted + # node in this situation would hit the exact same wall and route + # around it the same way -- this is not a workaround specific to the + # test. + sql node3 { SET synchronous_commit TO local; CREATE TABLE divergent_data(a int); } + sql node3 { SET synchronous_commit TO local; INSERT INTO divergent_data VALUES (1), (2), (3); } + sql node3 { CHECKPOINT; } +} + +step test_002_reconnect_and_confirm_published { + network connect node3 + wait until node3 state is secondary timeout 60s + # give node3's keeper at least one periodic tick to publish its new + # (diverged) timeline history before checking + sleep 5s + # application traffic keeps flowing after node3 reconnects too -- + # nothing about this scenario ever pauses the main system + sql node1 { INSERT INTO app_data SELECT generate_series(101, 150); } + sql monitor { + SELECT EXISTS ( + SELECT 1 + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.nodename = 'node3' AND h.tli = 2 AND h.parenttli = 1 + ); + } + expect { t } +} + +step test_003_show_timeline_still_reads_clean { + # node3 (tli=2) is not competing with node1/node2 (tli=1) -- it's simply + # ahead of them, and tli=1 genuinely is tli=2's recorded parent. The + # auto-detected reference becomes tli=2 (the highest reported), under + # which node1/node2 correctly show as ancestors and node3 shows as the + # (wrongly trusted) tip. Nothing here reads as a fork despite one + # genuinely existing -- three nodes, same blind spot as two. + exec node1 pg_autoctl show timeline + expect { ok, on accepted lineage } +} + +step test_004_accept_timeline_flags_the_fork { + # still just ordinary application traffic on the main system, same as + # every other step in this spec + sql node1 { INSERT INTO app_data SELECT generate_series(151, 200); } + exec node1 pg_autoctl accept timeline --tli 1 --formation default --reason "node3 self-promoted out of band during a network partition" + expect { accepted as ground truth } + + exec node1 pg_autoctl show timeline + expect { FORK } +} + +step test_005_automatic_catchingup_without_maintenance { + # No enable/disable maintenance anywhere in this step -- that used to + # be required to force node3 through a real transition. Now the + # monitor's own ProceedGroupState() detects the ancestry mismatch on + # node3's own next report after the operator's pin and pushes it to + # catchingup by itself, without needing any operator action or + # incidental health-check cycle to drive it there. This is the part + # that's new; asserted on its own, deterministically and quickly, + # rather than folded into the full recovery check below. + wait until node3 assigned-state = catchingup timeout 15s +} + +step test_006_maintenance_cycle_recovers { + # The rewind itself -- once node3 is in catchingup, actually reaching + # node1 and running pg_rewind -- depends on node1's pg_hba.conf + # hostname resolution being current, which is unrelated to and + # unaffected by test_005's immediate detection above; it's the same + # dependency the pre-existing #683 recovery path has always had. + # Cycling maintenance here (as the 2-node companion spec also does) + # is the same proven, reliable way to drive that recovery through to + # completion for this test's purposes. + exec node3 pg_autoctl enable maintenance + wait until node3 state is maintenance timeout 60s + exec node3 pg_autoctl disable maintenance + wait until node3 state is secondary timeout 60s + + sql monitor { + SELECT reportedtli FROM pgautofailover.node WHERE nodename = 'node3'; + } + expect { 1 } + + sql node3 { SELECT to_regclass('public.divergent_data'); } + expect { } + + # the main system's own data was never at risk and never paused -- + # confirm all 200 application rows made it to node3 once it rejoined + # on the correct lineage + sql node3 { SELECT count(*) FROM app_data; } + expect { 200 } +} + +step test_007_show_timeline_clean_after_recovery { + exec node1 pg_autoctl show timeline + expect { ok, on accepted lineage } +} diff --git a/tests/tap/specs/timeline_fork_report_lsn_deadlock.pgaf b/tests/tap/specs/timeline_fork_report_lsn_deadlock.pgaf new file mode 100644 index 000000000..3ab424a94 --- /dev/null +++ b/tests/tap/specs/timeline_fork_report_lsn_deadlock.pgaf @@ -0,0 +1,200 @@ +# Reproduces and validates the fix for +# https://github.com/hapostgres/pg_auto_failover/issues/683 ("Unrecoverable +# state after failed switchover"): a standby that has generated local WAL +# past the point its timeline actually diverged can never resolve via +# ordinary streaming replication -- Postgres itself refuses ("requested +# timeline N is not a child of this server's history" / "highest timeline N +# of the primary is behind recovery timeline M"), and pre-fix, pg_autoctl had +# no code path that ever called pg_rewind to get out of it: the standby just +# retried the same doomed reconnect forever. +# +# node2 is cut off from the network, then promoted directly at the Postgres +# level (bypassing pg_autoctl entirely) and given a couple of local-only +# writes -- this +# manufactures a genuine, local timeline fork on node2 without needing to +# race the real production timing (an isolated primary keeps streaming to +# one specific reachable standby while the rest of the cluster promotes +# someone else, which pgaftest's node-granular network disconnect can't +# selectively reproduce). +# +# When node2 is handed back to pg_autoctl, its current_role assumption +# ("secondary", should be in recovery) no longer matches Postgres reality +# (promoted, writable). Node2 keeps reporting "secondary" throughout and the +# fork stays invisible to auto-detection -- no sibling standby to disagree +# with it, see the known limitation below -- so nothing forces a real +# transition on its own *until* the reference lineage is pinned. Once +# test_003 below pins tli=1, the monitor's own ancestry check on every +# currently-secondary node (ProceedGroupStateFromContext) picks up the +# mismatch on node2's very next report and pushes it to catchingup within +# about a second, with no maintenance toggle or health-check cycle needed to +# trigger it -- test_004 asserts this directly. That's the moment the #683 +# fix actually runs; test_005 re-drives the same recovery through the older, +# maintenance-cycle path as a belt-and-suspenders check that it still works +# when triggered that way too. +# +# Known, separate limitation, not exercised here: in a 2-node formation with +# no other standby to compare against, an explicit `perform failover` while +# node2 is still forked will promote it anyway -- the ancestry filter only +# excludes a candidate when a competing branch is reported by someone else, +# and with only node1 (never advanced past its original timeline) and node2 +# (the fork) in the group, node2's fork looks like a normal, legitimate +# promotion. See node_timeline_history.c's FilterNodesByTimelineAncestry(). + +cluster { + monitor + ssl off + formation { + node1 + node2 + } +} + +setup { + wait until primary, secondary timeout 120s +} + +teardown { + compose down +} + +step test_001_fork_node2_out_of_band { + wait until node1 state is primary + and node2 state is secondary + timeout 60s + # node2's own keeper stays running throughout: "stop postgres" (pgctl + # off) stops Postgres outright, which would defeat the raw promote + # below. The keeper can't do anything about a promotion it never + # issued anyway -- it just keeps reporting what it observes, which is + # the whole point being tested here. + network disconnect node2 + exec node2 pg_ctl promote --pgdata /var/lib/postgres/pgaf + sleep 3s + sql node2 { CREATE TABLE divergent_data(a int); } + sql node2 { INSERT INTO divergent_data VALUES (1), (2), (3); } + sql node2 { CHECKPOINT; } +} + +step test_002_hand_node2_back_to_pg_autoctl { + network connect node2 + wait until node2 state is secondary timeout 60s + # Give node2's keeper at least one periodic tick (service_keeper.c's + # per-tick timeline-history publish) to run before checking -- reportedstate + # never actually changes here (it was "secondary" throughout), so "wait + # until ... secondary" above resolves immediately and doesn't itself wait + # for that tick. + sleep 5s + # node2 is back. Both sides may or may not have already noticed anything + # is wrong by the time this step's assertions run -- the health checker + # runs on its own schedule and, once it marks node2 unhealthy, drives it + # through the very same catchingup cycle test_004 forces explicitly + # below, so whether that has already happened here is a race this spec + # doesn't need to pin down. What's NOT racy is node_timeline_history: it + # is append-only, so node2's diverged branch being in there at all is + # permanent proof the fork was correctly detected and published, + # regardless of what's happened to node2's live state since. + sql monitor { + SELECT EXISTS ( + SELECT 1 + FROM pgautofailover.node_timeline_history h + JOIN pgautofailover.node n ON n.nodeid = h.nodeid + WHERE n.nodename = 'node2' AND h.tli = 2 AND h.parenttli = 1 + ); + } + expect { t } +} + +step test_003_show_and_accept_timeline_cli { + # `pg_autoctl show timeline` and `pg_autoctl accept timeline` (Part 3/4) + # are otherwise only covered at the monitor-SQL level + # (timeline_fork_detection.sql) -- exercise the actual CLI against a + # live cluster here. Pinning tli=1 below is what test_004 immediately + # reacts to: standby_check_timeline_with_upstream() (the #683 fix) is a + # purely local, standby-side check that runs once node2 reaches + # catchingup, but the monitor-side push into catchingup in the first + # place -- test_004's whole point -- is exactly what accepting a pin + # here triggers. + # Unpinned, node2's fork (tli=2) IS the reference by default (highest + # reported tli, and nobody else is arguing) -- this is the "known, + # separate limitation" from the header comment, and it reads as clean. + exec node1 pg_autoctl show timeline + expect { ok, on accepted lineage } + + exec-fails node1 pg_autoctl accept timeline --tli 99 + exec node1 pg_autoctl accept timeline --tli 1 --reason "node2 self-promoted out of band" + expect { accepted as ground truth } + + # With tli=1 pinned as ground truth instead, node2 is now unambiguously + # flagged as diverged. + exec node1 pg_autoctl show timeline + expect { FORK } +} + +step test_004_automatic_catchingup_without_maintenance { + # No enable/disable maintenance anywhere in this step -- that used to be + # required to force node2 through a real transition. Now the monitor's + # own ProceedGroupState() detects the ancestry mismatch on node2's very + # next report after test_003's pin and pushes it to catchingup by + # itself, without needing any operator action or incidental + # health-check cycle to drive it there. + # + # node2 is this formation's only standby, so excluding it also means + # node1 can no longer keep synchronous replication -- node1 transitions + # through wait_primary at the same time, settling back to primary once + # node2's automatic recovery (pg_rewind, or a fresh pg_basebackup if + # pg_rewind itself can't connect) completes. Waiting for both to settle + # here, rather than just asserting the initial catchingup assignment, + # is what makes test_005 below safe to run immediately after: without + # it, test_005's `enable maintenance` can race node1's own wait_primary + # dance and get rejected by the monitor's start-maintenance precondition + # (seen intermittently in CI before this step existed). + wait until node2 assigned-state = catchingup timeout 15s + wait until node1 state is primary + and node2 state is secondary + timeout 60s +} + +step test_005_maintenance_cycle_recovers { + # By this point node2 has already recovered automatically (test_004). + # This re-drives the same recovery through the older, operator-forced + # maintenance-cycle path -- the same transition an ordinary "briefly + # unreachable, please resync" health-check cycle would also drive -- + # as a belt-and-suspenders check that it still works when triggered + # that way, not because it's still the only way to recover here. + exec node2 pg_autoctl enable maintenance + wait until node2 state is maintenance timeout 60s + exec node2 pg_autoctl disable maintenance + wait until node2 state is secondary timeout 60s + + # node2 must have actually resynced to node1's real lineage, not just + # gotten stuck reporting a stale state. + sql monitor { + SELECT reportedtli FROM pgautofailover.node WHERE nodename = 'node1'; + } + expect { 1 } + sql monitor { + SELECT reportedtli FROM pgautofailover.node WHERE nodename = 'node2'; + } + expect { 1 } + + # The out-of-band writes did not exist on any real lineage -- they must + # be gone after recovery, whichever recovery path was taken. + sql node2 { SELECT to_regclass('public.divergent_data'); } + expect { } + + # node1 was never disturbed: still primary, still holds the one + # legitimate row of data throughout. + sql monitor { + SELECT reportedstate FROM pgautofailover.node WHERE nodename = 'node1'; + } + expect { primary } +} + +step test_006_writes_still_flow_after_recovery { + # The cluster is fully healthy again: a real write on the primary + # replicates to the now-recovered node2, proving this isn't just a + # state-label reshuffle. + sql node1 { CREATE TABLE post_recovery(a int); INSERT INTO post_recovery VALUES (1); } + wait until node2 state is secondary timeout 30s + sql node2 { SELECT * FROM post_recovery; } + expect { 1 } +}