Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/ref/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,11 @@ database where the extension has been deployed::
setting | true
unit |
short_desc | Refuse to proceed with failover when quorum nodes have not yet reported their LSN.
-[ RECORD 11 ]---------------------------------------------------------------------------------------------------
name | pgautofailover.replication_stall_timeout
setting | 10000
unit | ms
short_desc | Assign wait_primary when the primary has had no standby connected in pg_stat_replication for this long.

pgautofailover.guard_data_loss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand All @@ -130,6 +135,26 @@ failovers. Use ``ALTER DATABASE pg_auto_failover SET
pgautofailover.guard_data_loss = false;`` only if you want the setting to
persist across monitor restarts with explicit intent, and document the reason.

pgautofailover.replication_stall_timeout
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This setting addresses a 3-datacentre split-brain scenario: the primary and
its standby lose connectivity to each other, but both remain reachable from
the monitor. In that situation the primary can neither detect the standby
is gone (it's still reachable, just not to the standby) nor get demoted by
the monitor (the monitor still sees it as healthy), so
``synchronous_standby_names`` stays set and every ``COMMIT`` on the primary
hangs indefinitely waiting for a standby acknowledgement that will never
come.

When a ``PRIMARY`` node reports an empty ``pg_stat_replication`` sync state
for longer than ``pgautofailover.replication_stall_timeout`` (default
``10s``), the monitor assigns ``wait_primary``. This clears
``synchronous_standby_names`` and unblocks writes, without initiating a
failover — the standby simply rejoins as ``secondary`` once the link is
restored. See :ref:`pg_autoctl_show_state` and :ref:`pg_autoctl_watch` for
how to observe a node in ``wait_primary``.

You can edit the parameters as usual with PostgreSQL, either in the
``postgresql.conf`` file or using ``ALTER DATABASE pg_auto_failover SET parameter =
value;`` commands, then issuing a reload.
Expand Down
2 changes: 2 additions & 0 deletions docs/ref/pg_autoctl.rst
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ pg_autoctl provides the following commands::
pg_autoctl get node
replication-quorum get replication-quorum property from the monitor
candidate-priority get candidate property from the monitor
region get region property from the monitor

pg_autoctl get formation
settings get replication settings for a formation from the monitor
Expand All @@ -86,6 +87,7 @@ pg_autoctl provides the following commands::
metadata set metadata on the monitor
replication-quorum set replication-quorum property on the monitor
candidate-priority set candidate property on the monitor
region set region property on the monitor

pg_autoctl set formation
number-sync-standbys set number-sync-standbys for a formation on the monitor
Expand Down
10 changes: 10 additions & 0 deletions docs/ref/pg_autoctl_create_postgres.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ registered too, and is known to be healthy).
--candidate-priority priority of the node to be promoted to become primary
--replication-quorum true if node participates in write quorum
--maximum-backup-rate maximum transfer rate of data transferred from the server during initial sync
--region data-centre or availability-zone label for this node

Description
-----------
Expand Down Expand Up @@ -268,6 +269,15 @@ The following options are available to ``pg_autoctl create postgres``:
initial sync. This is used by ``pg_basebackup``.
Defaults to ``100M``.

--region

Sets a free-form label identifying the data-centre or availability zone
this node runs in, at node registration on the monitor. Defaults to
``default``. Purely informational at the moment: it is displayed as its
own column by ``pg_autoctl watch`` in the verbose and higher policies, to
help make sense of the topology of a multi-region deployment. It does not
currently affect any failover or quorum decision.

--run

Immediately run the ``pg_autoctl`` service after having created this node.
Expand Down
1 change: 1 addition & 0 deletions docs/ref/pg_autoctl_get.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ pg_autoctl get - Get a pg_auto_failover node, or formation setting
pg_autoctl_get_formation_number_sync_standbys
pg_autoctl_get_node_replication_quorum
pg_autoctl_get_node_candidate_priority
pg_autoctl_get_node_region
60 changes: 60 additions & 0 deletions docs/ref/pg_autoctl_get_node_region.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
.. _pg_autoctl_get_node_region:

pg_autoctl get node region
===========================

pg_autoctl get region - get region property from the monitor

Synopsis
--------

This command prints the ``pg_autoctl`` region label for a given node::

usage: pg_autoctl get node region [ --pgdata ] [ --json ] [ --formation ] [ --name ]

--pgdata path to data directory
--formation pg_auto_failover formation
--name pg_auto_failover node name
--json output data in the JSON format

Description
-----------

See also :ref:`pg_autoctl_set_node_region` to change the value, and
:ref:`pg_autoctl_show_settings` for the full list of replication settings.

Options
-------

--pgdata

Location of the Postgres node being managed locally. Defaults to the
environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor
from anywhere, rather than the monitor URI used by a local Postgres node
managed with ``pg_autoctl``.

--json

Output JSON formatted data.

--formation

Show replication settings for given formation. Defaults to ``default``.

--name

Show replication settings for given node, selected by name.

Examples
--------

::

$ pg_autoctl get node region --name node1
dc1

$ pg_autoctl get node region --name node1 --json
{
"name": "node1",
"region": "dc1"
}
11 changes: 10 additions & 1 deletion docs/ref/pg_autoctl_node.rst
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,15 @@ Settings in this section are applied live without restarting the node.
Whether this node participates in the synchronous replication quorum.
Boolean, defaults to ``true``. **Mutable**.

``region``

Free-form label identifying the data-centre or availability zone this
node runs in. Defaults to ``default``. **Mutable**: changes are applied
via ``pg_autoctl set node region`` (see :ref:`pg_autoctl_set_node_region`).
Purely informational: shown as its own column by ``pg_autoctl watch`` in
the verbose and higher policies, and does not currently affect any
failover or quorum decision.

``[options]``
^^^^^^^^^^^^^

Expand Down Expand Up @@ -248,7 +257,7 @@ The supervisor that ``pg_autoctl node run`` exec's into watches the ini file
for changes. When it detects a write it re-reads the file and converges any
**mutable** fields without restarting the node or interrupting replication:

``candidate_priority`` and ``replication_quorum``
``candidate_priority``, ``replication_quorum``, and ``region``
Applied by calling ``pg_autoctl set node`` against the running node.

``ssl``, ``ca_file``, ``cert_file``, ``key_file``
Expand Down
1 change: 1 addition & 0 deletions docs/ref/pg_autoctl_set.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ pg_autoctl set - Set a pg_auto_failover node, or formation setting
pg_autoctl_set_formation_number_sync_standbys
pg_autoctl_set_node_replication_quorum
pg_autoctl_set_node_candidate_priority
pg_autoctl_set_node_region
98 changes: 98 additions & 0 deletions docs/ref/pg_autoctl_set_node_region.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
.. _pg_autoctl_set_node_region:

pg_autoctl set node region
===========================

pg_autoctl set region - set region property on the monitor

Synopsis
--------

This command sets the ``pg_autoctl`` region label for a given node::

usage: pg_autoctl set node region [ --pgdata ] [ --json ] [ --formation ] [ --name ] <region>

--pgdata path to data directory
--formation pg_auto_failover formation
--name pg_auto_failover node name
--json output data in the JSON format

Description
-----------

Sets the data-centre or availability-zone label for a node that is already
registered on the monitor. Unlike :ref:`pg_autoctl_set_node_candidate_priority`
and :ref:`pg_autoctl_set_node_replication_quorum`, this never triggers a
failover or a replication settings change on the primary: region is purely
informational, displayed by :ref:`pg_autoctl_watch` in the verbose and
higher policies. The command takes effect immediately, with no wait.

This is also the mechanism ``pg_autoctl node run`` uses to apply a live edit
of the ``region`` property in a ``pg_autoctl_node.ini`` file — see
:ref:`pg_autoctl_node` for the full ``[settings]`` reference.

See also :ref:`pg_autoctl_show_settings` for the full list of replication
settings.

Options
-------

--pgdata

Location of the Postgres node being managed locally. Defaults to the
environment variable ``PGDATA``. Use ``--monitor`` to connect to a monitor
from anywhere, rather than the monitor URI used by a local Postgres node
managed with ``pg_autoctl``.

--json

Output JSON formatted data.

--formation

Show replication settings for given formation. Defaults to ``default``.

--name

Show replication settings for given node, selected by name.

Environment
-----------

PGDATA

Postgres directory location. Can be used instead of the ``--pgdata``
option.

PG_AUTOCTL_MONITOR

Postgres URI to connect to the monitor node, can be used instead of the
``--monitor`` option.

XDG_CONFIG_HOME

The pg_autoctl command stores its configuration files in the standard
place XDG_CONFIG_HOME. See the `XDG Base Directory Specification`__.

__ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

XDG_DATA_HOME

The pg_autoctl command stores its internal states files in the standard
place XDG_DATA_HOME, which defaults to ``~/.local/share``. See the `XDG
Base Directory Specification`__.

__ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

Examples
--------

::

$ pg_autoctl set node region --name node1 dc2
dc2

$ pg_autoctl set node region --name node1 dc2 --json
{
"region": "dc2"
}
8 changes: 8 additions & 0 deletions docs/ref/pg_autoctl_watch.rst
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,14 @@ your terminal window size changes:
the Node column contains the groupId and the nodeId, separated by a
colon, such as ``0:1`` for the first coordinator node.

- Region

The data-centre or availability zone label set for the node at
creation time, either with ``--region`` or the ``[settings] region``
property of a ``pg_autoctl_node.ini`` file. Only shown in wider
terminal windows, alongside more detailed columns. Defaults to
``default`` when not set.

- Last Report, or Report

Time interval between now and the last known time when a node has
Expand Down
Loading