hapostgres · dimitri · Jul 27, 2026 · Jul 25, 2026 · Jul 25, 2026 · Jul 25, 2026
diff --git a/docs/architecture.rst b/docs/architecture.rst
@@ -304,3 +304,21 @@ goal state from the monitor. If the failed node was a primary and was demoted,
 it will learn this from the monitor. Once the node reports, it is allowed to
 come back as a standby by running ``pg_rewind``. If it is too far behind, the
 node performs a new ``pg_basebackup``.
+
+Whether ``pg_rewind`` is even the right call isn't decided from a bare
+timeline-number comparison. The node walks the upstream's real timeline
+history to distinguish "still catching up" from "genuinely diverged onto a
+dead branch" — the latter can happen when a standby was written to, or
+promoted, outside of ``pg_autoctl``'s control. Every node also periodically
+publishes its own known timeline lineage to the monitor
+(``pgautofailover.node_timeline_history``), so that a failover election can
+reason about forks centrally and exclude a diverged candidate rather than
+block on it. The monitor uses this same lineage data proactively too: a
+node currently acting as a healthy secondary is checked against the
+group's reference lineage on every report, and gets pushed into
+``catchingup`` — where the rewind actually happens — as soon as a genuine
+divergence is found, rather than waiting for the node to go through some
+other, unrelated transition first. See :ref:`timeline_forks` for the full
+scenario, and :ref:`pg_autoctl_show_timeline` /
+:ref:`pg_autoctl_accept_timeline` for the commands that surface and, when
+needed, resolve this.
diff --git a/docs/failover-state-machine.rst b/docs/failover-state-machine.rst
@@ -155,13 +155,41 @@ The standby node keeper runs pg_basebackup, connecting to the primary's
 hostname and port. The keeper then edits recovery.conf and starts
 PostgreSQL in hot standby node.
 
+Before doing so — and also when a healthy secondary is reconnecting to a
+(possibly new) primary rather than bootstrapping from scratch — the keeper
+compares its own timeline history against the upstream's. A node that is
+simply behind rewinds cleanly onto the upstream once it catches up; a node
+whose local WAL has genuinely diverged onto a dead branch (for example, one
+that was promoted directly at the Postgres level outside of
+``pg_autoctl``'s control) can never resolve that divergence through
+ordinary streaming replication, and Postgres itself will refuse the
+reconnect. In that case the keeper runs ``pg_rewind`` to discard the
+diverged WAL and rejoin the real lineage, falling back to a fresh
+``pg_basebackup`` if ``pg_rewind`` itself cannot connect. See
+:ref:`timeline_forks` for the full scenario.
+
 Secondary
 ^^^^^^^^^
 
 A node with this state is acting as a hot standby for the primary, and
 is up to date with the WAL log there. In particular, it is within 16MB
 or 1 WAL segment of the primary.
 
+Streaming replication alone can look perfectly healthy even when the
+secondary is on a genuinely forked branch of history — the WAL receiver
+just keeps reporting progress against whatever local timeline the standby
+happens to be on. To catch this without waiting for an incidental
+transition, the monitor applies the same ``FilterNodesByTimelineAncestry()``
+check used during an election (see ``Report_LSN`` below) to every node
+currently in the secondary state, on each of its regular reports. As soon
+as a secondary's reported timeline is found not to be an ancestor of the
+group's reference lineage, the monitor assigns it the ``catchingup`` goal
+state right away — typically within about a second, on that very report —
+rather than waiting for a health-check cycle or an operator-forced resync
+to reveal the problem. See :ref:`timeline_forks` for the full scenario and
+:ref:`pg_autoctl_accept_timeline` for pinning the reference lineage when
+auto-detection can't tell which branch is real.
+
 Maintenance
 ^^^^^^^^^^^
 
@@ -274,6 +302,18 @@ data loss, the election can be unblocked with
 :ref:`pg_autoctl_perform_failover` ``--allow-data-loss``.  See
 :ref:`perform_failover_allow_data_loss` for details.
 
+Once quorum standbys have reported their LSN, candidates are further
+filtered by timeline ancestry (``FilterNodesByTimelineAncestry()``): a node
+whose reported timeline is not the group's reference lineage — the
+:ref:`pg_autoctl_accept_timeline`-pinned timeline if one has been set,
+otherwise the highest reported timeline that nothing else in the group
+disagrees with — is excluded from candidacy rather than counted as missing.
+This is what lets the election proceed among the remaining, non-diverged
+candidates instead of blocking on a node that can never actually win. Use
+:ref:`pg_autoctl_show_timeline` to see the group's known timeline history
+and each node's status against it. See :ref:`timeline_forks` for the
+failure scenario this guards against.
+
 Fast_forward
 ^^^^^^^^^^^^
 

diff --git a/docs/faq.rst b/docs/faq.rst
@@ -1,3 +1,5 @@
+.. _faq:
+
 Frequently Asked Questions
 ==========================
 
@@ -111,6 +113,43 @@ See :ref:`perform_failover_allow_data_loss` and the
 ``pgautofailover.guard_data_loss`` GUC in :ref:`configuration` for a full
 explanation.
 
+My standby won't rejoin — Postgres logs a timeline mismatch, what should I do?
+------------------------------------------------------------------------------
+
+This happens when a standby's local WAL has genuinely diverged from the
+rest of the group — for example, it was promoted or written to directly at
+the Postgres level, outside of ``pg_autoctl``'s control, during an incident.
+Postgres itself refuses the reconnect and logs something like ``requested
+timeline N is not a child of this server's history``.
+
+**This is usually automatic.** ``pg_autoctl`` walks the upstream's real
+timeline history to tell a standby that's simply behind apart from one
+that's genuinely diverged, and the monitor checks every currently-healthy
+secondary's ancestry on each of its regular reports — not just at the next
+incidental transition. As soon as a genuine divergence is found, the node
+is pushed to ``catchingup`` and rewound with ``pg_rewind`` (or a fresh
+``pg_basebackup`` if ``pg_rewind`` can't connect), typically within about a
+second, with no health-check cycle or maintenance toggle to wait for. See
+:ref:`timeline_forks` for the full scenario.
+
+**If it doesn't resolve on its own,** check what the monitor knows::
+
+  pg_autoctl show timeline
+
+A node flagged ``FORK: diverges from the reference timeline, pg_rewind
+required`` needs an operator decision. This most often means auto-detection
+couldn't tell which branch was real — for instance in a two-node formation,
+where there's no sibling node to disagree with the diverged one. Confirm
+which timeline is actually ground truth from other evidence (which node was
+manually promoted, ``pg_controldata`` output, etc.), then pin it::
+
+  pg_autoctl accept timeline --tli <N> --reason "..."
+
+The diverged node then rewinds and rejoins automatically. See
+:ref:`pg_autoctl_show_timeline`, :ref:`pg_autoctl_accept_timeline`, and
+:ref:`resolving_timeline_fork` for the full command reference and
+walkthrough.
+
 The state of the system is blocked, what should I do?
 -----------------------------------------------------
 

diff --git a/docs/fault-tolerance.rst b/docs/fault-tolerance.rst
@@ -197,6 +197,64 @@ PostgreSQL service:
 
        Falling back to asynchronous replication and resynchronizing
 
+.. _timeline_forks:
+
+Timeline Forks
+--------------
+
+A different kind of failure doesn't come from a node being unreachable, but
+from a node whose local WAL has genuinely diverged from the rest of the
+group. This happens when a standby is written to, or promoted, outside of
+``pg_autoctl``'s control — a manual intervention during an incident, a
+monitoring bug elsewhere, a previous split-brain — and generates local WAL
+that no other node in the group has, on a branch of history the primary
+never took.
+
+Ordinary streaming replication can never resolve this: it's not lag, it's
+divergence. Postgres itself refuses the reconnect (``requested timeline N
+is not a child of this server's history``).
+
+pg_auto_failover detects and resolves this automatically in the common
+case: before trusting a bare timeline-number comparison, a standby
+reconnecting to a (possibly new) primary walks the primary's real
+timeline history to tell "still catching up" apart from "diverged onto a
+dead branch," and runs ``pg_rewind`` in either direction as needed (falling
+back to a fresh ``pg_basebackup`` if ``pg_rewind`` itself can't connect).
+See the ``Catchingup`` section of :ref:`failover_state_machine` for where
+this check runs.
+
+The monitor doesn't wait for that reconnect to notice, either. It applies
+the same ancestry check to every node currently reported as a healthy
+secondary, and as soon as one is found not to be an ancestor of the group's
+reference lineage, it is pushed to ``catchingup`` right away — typically
+within about a second, on that node's very next report — rather than
+waiting for an incidental health-check cycle or an operator-forced resync
+to reveal the problem (see the ``Report_LSN`` section of
+:ref:`failover_state_machine` for where the election applies this same
+ancestry filter, and this section's own diagram below for the monitor-side
+push).
+
+The reference lineage itself is either pinned explicitly, or auto-detected
+as the branch containing the highest reported timeline. Auto-detection only
+excludes a candidate when a genuinely *competing* branch is reported by
+someone else — two nodes each diverging from the same point onto two
+different timelines — in which case the loser is caught and rewound with no
+operator action at all. It doesn't help when there's no sibling to disagree
+with the diverged node: in a two-node formation, or whenever every
+surviving node happens to already be on the same diverged branch, the fork
+reads as clean and an operator decision is needed. Use
+:ref:`pg_autoctl_show_timeline` to see the group's known timeline history
+and each node's status against it, and :ref:`pg_autoctl_accept_timeline` to
+pin the correct lineage explicitly — once pinned, the same immediate,
+automatic push applies. See :ref:`resolving_timeline_fork` for the full
+walkthrough.
+
+.. figure:: ./tikz/seq-timeline-fork.svg
+   :alt: Sequence diagram of a standby forking out-of-band and being detected and rewound back onto the real lineage
+
+   A standby forks out-of-band; once the mismatch is visible to the
+   monitor, it is pushed to catchingup and rewound within about a second
+
 Failure handling and network partition detection
 ------------------------------------------------
 

diff --git a/docs/operations.rst b/docs/operations.rst
@@ -379,6 +379,93 @@ unplanned failover are all handled by the monitor, rather than the client
 side command line, at the client level the two command ``pg_autoctl perform
 failover`` and ``pg_autoctl perform switchover`` are synonyms, or aliases.
 
+.. _resolving_timeline_fork:
+
+Resolving a detected timeline fork
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A standby whose local WAL has genuinely diverged from the rest of the group
+— for example, one that was promoted or written to directly at the
+Postgres level, outside of ``pg_autoctl``'s control, during an incident —
+cannot resolve that divergence through ordinary streaming replication. See
+:ref:`timeline_forks` for the full scenario and how pg_auto_failover
+detects and, in most cases, automatically resolves this.
+
+Start by checking what the monitor currently knows::
+
+  $ pg_autoctl show timeline
+       TLI | Parent TLI | Switchpoint LSN
+  ---------+------------+----------------
+         1 |          0 |             0/0
+         2 |          1 |       0/3000130
+
+                  Name | NodeId |  TLI |         LSN | Status
+  ---------------------+--------+------+-------------+-----------------------------------------
+                 node1 |      1 |    1 |   0/3000130 | ok, on accepted lineage
+                 node2 |      2 |    2 |   0/3016330 | ok, on accepted lineage
+
+The first table is every timeline the group has ever seen; the second is
+each node's current position and whether it's on the group's reference
+lineage. Here both nodes read as clean — this is the known two-node
+limitation: auto-detection only has something to compare against when a
+sibling node disagrees, and with only ``node1`` (never advanced past
+timeline 1) and ``node2`` (the fork) in the group, ``node2``'s fork looks
+like a normal, legitimate promotion.
+
+Having confirmed from other evidence which timeline is really ground truth
+— which node was manually promoted, ``pg_controldata`` output, application
+logs — pin it explicitly::
+
+  $ pg_autoctl accept timeline --tli 1 --formation default \
+      --reason "node2 self-promoted out of band during a network partition"
+  Timeline 1 accepted as ground truth for formation "default" group 0. The election will now only consider nodes on that lineage; other nodes need pg_rewind before rejoining.
+
+``node2`` is now unambiguously flagged::
+
+  $ pg_autoctl show timeline
+       TLI | Parent TLI | Switchpoint LSN
+  ---------+------------+----------------
+         1 |          0 |             0/0
+         2 |          1 |       0/3000130
+
+                  Name | NodeId |  TLI |         LSN | Status
+  ---------------------+--------+------+-------------+-----------------------------------------
+                 node1 |      1 |    1 |   0/30599B8 | ok, on accepted lineage
+                 node2 |      2 |    2 |   0/3016330 | FORK: diverges from the reference timeline, pg_rewind required
+
+  One or more nodes have diverged from the reference timeline (see FORK above).
+  See `pg_autoctl accept timeline --help` to resolve.
+
+No further command is needed to make ``node2`` rewind. The pin doesn't just
+change how the flagged node is evaluated at the next election — the monitor
+re-checks every currently-``secondary`` node's ancestry against the
+freshly-pinned lineage right away, so ``node2`` is pushed to ``catchingup``
+within about a second of the ``accept timeline`` command above, with no
+health-check cycle or maintenance toggle needed to trigger it.
+
+``node2`` goes ``secondary`` → ``catchingup`` → ``secondary``, running
+``pg_rewind`` onto timeline 1 along the way (or, if ``pg_rewind`` itself
+can't connect, a fresh ``pg_basebackup`` — both pre-existing recovery
+paths) — usually so quickly that polling ``pg_autoctl show state`` a second
+or two apart never even catches the intermediate ``catchingup`` state. The
+fork clears on its own, with no need to run an "accept" or "resolve"
+command a second time, and no need to cycle the node through maintenance to
+force anything::
+
+  $ pg_autoctl show timeline
+       TLI | Parent TLI | Switchpoint LSN
+  ---------+------------+----------------
+         1 |          0 |             0/0
+         2 |          1 |       0/3000130
+
+                  Name | NodeId |  TLI |         LSN | Status
+  ---------------------+--------+------+-------------+-----------------------------------------
+                 node1 |      1 |    1 |   0/70000F8 | ok, on accepted lineage
+                 node2 |      2 |    1 |   0/70000F8 | ok, on accepted lineage
+
+See :ref:`pg_autoctl_show_timeline` and :ref:`pg_autoctl_accept_timeline`
+for the full command reference.
+
 Current state, last events
 --------------------------
 
@@ -458,6 +545,10 @@ possible to try ``pg_autoctl create`` again. pg_auto_failover will review its pr
 progress and repeat idempotent operations (``create database``, ``create
 extension`` etc), gracefully handling errors.
 
+If a standby won't rejoin and Postgres logs a timeline mismatch, see
+:ref:`resolving_timeline_fork` and the related FAQ entry on the same
+topic in :ref:`faq`.
+
 .. _container-and-kubernetes-deployments:
 
 Container and Kubernetes Deployments

diff --git a/docs/ref/manual.rst b/docs/ref/manual.rst
@@ -20,6 +20,7 @@ have their own manual page.
    pg_autoctl_get
    pg_autoctl_set
    pg_autoctl_perform
+   pg_autoctl_accept
    pg_autoctl_node
    pg_autoctl_inspect
    pg_autoctl_manual

diff --git a/docs/ref/pg_autoctl.rst b/docs/ref/pg_autoctl.rst
@@ -20,6 +20,7 @@ pg_autoctl provides the following commands::
   + get       Get a pg_auto_failover node, or formation setting
   + set       Set a pg_auto_failover node, or formation setting
   + perform   Perform an action orchestrated by the monitor
+  + accept    Accept an operator decision orchestrated by the monitor
     activate  Activate a Citus worker from the Citus coordinator
     run       Run the pg_autoctl service (monitor or keeper)
     stop      signal the pg_autoctl service for it to stop
@@ -51,6 +52,7 @@ pg_autoctl provides the following commands::
     state          Prints monitor's state of nodes in a given formation and group
     settings       Print replication settings for a formation from the monitor
     standby-names  Prints synchronous_standby_names for a given group
+    timeline       Show the timeline history known to the monitor for a group, and each node's position against it
     file           List pg_autoctl internal files (config, state, pid)
     systemd        Print systemd service file for this node
 
@@ -97,6 +99,9 @@ pg_autoctl provides the following commands::
     switchover  Perform a switchover for given formation and group
     promotion   Perform a failover that promotes a target node
 
+  pg_autoctl accept
+    timeline  Accept a timeline as the ground truth after a detected fork
+
 Description
 -----------
 

diff --git a/docs/ref/pg_autoctl_accept.rst b/docs/ref/pg_autoctl_accept.rst
@@ -0,0 +1,11 @@
+.. _pg_autoctl_accept:
+
+pg_autoctl accept
+=================
+
+pg_autoctl accept - Accept an operator decision orchestrated by the monitor
+
+.. toctree::
+   :maxdepth: 1
+
+   pg_autoctl_accept_timeline