Can the Cluster Recover in This Scenario?

[spkinger]

I am running a Citus cluster with the following versions:

pg_autoctl version: 2.0
Citus version: 11.2.0
Postgres version: 15.1
The cluster has 5 primary worker nodes and 5 secondary worker nodes. I am wondering if the cluster can recover in the following scenario:

Primary worker node A-1 goes offline, and its corresponding secondary worker node A-2 is promoted to the primary worker node.
After 10 minutes, A-1 comes back online and is prepared to serve as a secondary node. A-2 begins syncing data to A-1.
Before the synchronization completes, A-2 fails and goes offline.
In the absence of A-2, can the cluster return to a normal operational state?

Thank you for your insights.

[dimitri]

The answer depends on whether A-1 was a quorum member when A-2 failed.

In step 3 of your scenario, A-2 is the primary and A-1 is re-joining as a standby. While A-1 is still in catching_up it is not yet a quorum member, so A-2 can continue accepting synchronous writes without waiting for A-1. If A-2 then fails before A-1 has caught up:

• Because number_sync_standbys = 1 ensures A-2 never committed anything A-1 did not replicate once A-1 became a quorum standby, the data that A-1 is missing is exactly the data written after A-1 rejoined but before it reached secondary state — no synchronous commit guarantee covers that window.
• If only A-1 survives it will be assigned report_lsn. If no other quorum node is missing it will be promoted automatically, accepting that uncommitted-to-quorum data may be lost.
• If the election stalls (e.g. another quorum node in the formation is also unreachable), PR Add pgautofailover.guard_data_loss GUC and pg_autoctl perform failover --allow-data-loss #1142 adds the escape:

pg_autoctl perform failover --allow-data-loss

For a 5-worker-group Citus cluster, each worker group runs its own independent FSM, so this applies per group.