diff --git a/docs/citus-quickstart.rst b/docs/citus-quickstart.rst index e0e2868b7..f4b20e20a 100644 --- a/docs/citus-quickstart.rst +++ b/docs/citus-quickstart.rst @@ -50,46 +50,55 @@ or run the docker build command directly: Our first Citus Cluster ----------------------- -To create a cluster we use the following docker compose definition: +Each node in the cluster is described by a ``pg_autoctl_node.ini`` file +bind-mounted into its container. There are three files: -.. literalinclude:: citus/docker-compose-scale.yml - :language: yaml - :emphasize-lines: 5,15,27 - :linenos: +.. literalinclude:: citus/ini/monitor.ini + :language: ini + :caption: citus/ini/monitor.ini -To run the full Citus cluster with HA from this definition, we can use the -following command: +.. literalinclude:: citus/ini/coordinator.ini + :language: ini + :caption: citus/ini/coordinator.ini -:: +.. literalinclude:: citus/ini/worker.ini + :language: ini + :caption: citus/ini/worker.ini - $ docker compose up --scale coord=2 --scale worker=6 +The ``worker.ini`` has no ``[formation] group`` entry. When that field is +absent the monitor assigns each worker to a group automatically — the first +worker to register in a group becomes primary, the second becomes secondary. +This is the right setup for the scaled deploy below, where we ask Docker +Compose to start six worker containers and let the monitor pair them. + +The docker compose definition for the scalable cluster is: -The command above starts the services up. The command also specifies a -``--scale`` option that is different for each service. We need: +.. literalinclude:: citus/docker-compose-scale.yml + :language: yaml + :linenos: - - one monitor node, and the default scale for a service is 1, +Every service runs ``pg_autoctl node run`` — creating the node on first +start, resuming on subsequent starts. All ``PG_AUTOCTL_*`` environment +variables are gone; everything lives in the ini files. - - one primary Citus coordinator node and one secondary Cituscoordinator - node, which is to say two coordinator nodes, +To run the full Citus cluster with HA from this definition: - - and three Citus worker nodes, each worker with both a primary Postgres - node and a secondary Postgres node, so that's a scale of 6 here. +:: -The default policy for the pg_auto_failover monitor is to assign a primary -and a secondary per auto failover :ref:`group`. In our case, every node -being provisioned with the same command, we benefit from that default policy:: + $ docker compose up --scale coord=2 --scale worker=6 - $ pg_autoctl create worker --ssl-self-signed --auth trust --pg-hba-lan --run +The ``--scale`` options tell Docker Compose how many containers to start for +each service: -When provisioning a production cluster, it is often required to have a -better control over which node participates in which group, then using the -``--group N`` option in the ``pg_autoctl create worker`` command line. + - one monitor node (default scale is 1), + - two coordinator containers — one primary, one secondary, + - six worker containers — the monitor pairs them into three groups of two, + assigning a primary and secondary in each group. -Within a given group, the first node that registers is a primary, and the -other nodes are secondary nodes. The monitor takes care of that in a way -that we don't have to. In a High Availability setup, every node should be -ready to be promoted primary at any time, so knowing which node in a group -is assigned primary first is not very interesting. +Within a given group the first node that registers becomes primary; the +monitor handles the assignment so we don't have to track it. In a High +Availability setup every node must be ready for promotion at any time, so +the initial primary assignment within a group is not significant. While the cluster is being provisionned by docker compose, you can run the following command and have a dynamic dashboard to follow what's happening. @@ -177,26 +186,36 @@ more complex docker compose file than in the previous section. pg_auto_failover architecture with a Citus formation -This time we create a cluster using the following docker compose definition: +This time we need per-group worker ini files so that each worker pair +lands in the right Citus shard group: + +.. literalinclude:: citus/ini/worker1.ini + :language: ini + :caption: citus/ini/worker1.ini (worker1a and worker1b) + +Worker 2 and worker 3 are identical except for ``group = 2`` and +``group = 3`` respectively. When ``group`` is set, ``pg_autoctl node run`` +passes ``--group N`` to ``pg_autoctl create worker``, pinning the pair to +that shard group. + +The docker compose definition is: .. literalinclude:: citus/docker-compose.yml :language: yaml - :emphasize-lines: 3,15,40,44,48,52,56,60,64,68 :linenos: -This definition is a little more involved than the previous one. We take -benefit from `YAML anchors and aliases`__ to define a *template* for our -coordinator nodes and worker nodes, and then apply that template to the -actual nodes. +We use `YAML anchors and aliases`__ to define templates for the coordinator +and each worker group, then apply them to the named services. Each service +sets its own ``hostname:`` — ``pg_autoctl node run`` uses the container +hostname as the node name when ``name`` is not set in the ini file. __ https://yaml101.com/anchors-and-aliases/ -Also this time we provision an application service (named "app") that sits -in the background and allow us to later connect to our current primary -coordinator. See :download:`Dockerfile.app ` for the -complete definition of this service. +Also this time we provision an application service (``app``) that sits in +the background and allows us to connect to the current primary coordinator. +See :download:`Dockerfile.app ` for its definition. -We start this cluster with a simplified command line this time: +We start this cluster with: :: @@ -474,13 +493,15 @@ makes it simple to introduce faults and see how the pg_auto_failover High Availability reacts to those faults. One obvious missing element to better test the system is the lack of -persistent volumes in our docker compose based test rig. It is possible to +persistent volumes in our docker compose based test rig. It is possible to create external volumes and use them for each node in the docker compose -definition. This allows restarting nodes over the same data set. +definition, allowing nodes to restart over the same data set. -See the command :ref:`pg_autoctl_do_tmux_compose_session` for more details -about how to run a docker compose test environment with docker compose, -including external volumes for each node. +For production Kubernetes deployments, the same ini files work as +ConfigMaps: bind-mount them alongside a persistent volume claim for +``/tmp/pgaf`` and use ``pg_autoctl node run`` as the container command. +See :ref:`pg_autoctl_node` for the full property reference and +:ref:`container-and-kubernetes-deployments` for production patterns. Now is a good time to go read `Citus Documentation`__ too, so that you know how to use this cluster you just created! diff --git a/docs/citus/docker-compose-scale.yml b/docs/citus/docker-compose-scale.yml index 08648c236..7184c0e67 100644 --- a/docs/citus/docker-compose-scale.yml +++ b/docs/citus/docker-compose-scale.yml @@ -1,36 +1,34 @@ -version: "3.9" # optional since v1.27.0 - services: monitor: image: pg_auto_failover:citus - environment: - PGDATA: /tmp/pgaf - command: | - pg_autoctl create monitor --ssl-self-signed --auth trust --run + volumes: + - /tmp/pgaf + - ./ini/monitor.ini:/etc/pgaf/node.ini:ro expose: - 5432 + command: pg_autoctl node run coord: image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/coordinator.ini:/etc/pgaf/node.ini:ro environment: - PGDATA: /tmp/pgaf PGUSER: citus PGDATABASE: citus - PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover" expose: - 5432 - command: | - pg_autoctl create coordinator --ssl-self-signed --auth trust --pg-hba-lan --run + command: pg_autoctl node run worker: image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/worker.ini:/etc/pgaf/node.ini:ro environment: - PGDATA: /tmp/pgaf PGUSER: citus PGDATABASE: citus - PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover" expose: - - 5432 - command: | - pg_autoctl create worker --ssl-self-signed --auth trust --pg-hba-lan --run + - 5432 + command: pg_autoctl node run diff --git a/docs/citus/docker-compose.yml b/docs/citus/docker-compose.yml index e56216d62..02aaed8f5 100644 --- a/docs/citus/docker-compose.yml +++ b/docs/citus/docker-compose.yml @@ -1,28 +1,14 @@ x-coord: &coordinator image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/coordinator.ini:/etc/pgaf/node.ini:ro environment: - PGDATA: /tmp/pgaf PGUSER: citus PGDATABASE: citus - PG_AUTOCTL_HBA_LAN: true - PG_AUTOCTL_AUTH_METHOD: "trust" - PG_AUTOCTL_SSL_SELF_SIGNED: true - PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover" - expose: - - 5432 - -x-worker: &worker - image: pg_auto_failover:citus - environment: - PGDATA: /tmp/pgaf - PGUSER: citus - PGDATABASE: citus - PG_AUTOCTL_HBA_LAN: true - PG_AUTOCTL_AUTH_METHOD: "trust" - PG_AUTOCTL_SSL_SELF_SIGNED: true - PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover" expose: - 5432 + command: pg_autoctl node run services: app: @@ -40,58 +26,71 @@ services: monitor: image: pg_auto_failover:citus - environment: - PGDATA: /tmp/pgaf - PG_AUTOCTL_SSL_SELF_SIGNED: true + volumes: + - /tmp/pgaf + - ./ini/monitor.ini:/etc/pgaf/node.ini:ro expose: - 5432 - command: | - pg_autoctl create monitor --auth trust --run + command: pg_autoctl node run coord0a: <<: *coordinator hostname: coord0a - command: | - pg_autoctl create coordinator --name coord0a --run coord0b: <<: *coordinator hostname: coord0b - command: | - pg_autoctl create coordinator --name coord0b --run worker1a: - <<: *worker + <<: &worker1 + image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/worker1.ini:/etc/pgaf/node.ini:ro + environment: + PGUSER: citus + PGDATABASE: citus + expose: + - 5432 + command: pg_autoctl node run hostname: worker1a - command: | - pg_autoctl create worker --group 1 --name worker1a --run worker1b: - <<: *worker + <<: *worker1 hostname: worker1b - command: | - pg_autoctl create worker --group 1 --name worker1b --run worker2a: - <<: *worker + <<: &worker2 + image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/worker2.ini:/etc/pgaf/node.ini:ro + environment: + PGUSER: citus + PGDATABASE: citus + expose: + - 5432 + command: pg_autoctl node run hostname: worker2a - command: | - pg_autoctl create worker --group 2 --name worker2a --run worker2b: - <<: *worker + <<: *worker2 hostname: worker2b - command: | - pg_autoctl create worker --group 2 --name worker2b --run worker3a: - <<: *worker + <<: &worker3 + image: pg_auto_failover:citus + volumes: + - /tmp/pgaf + - ./ini/worker3.ini:/etc/pgaf/node.ini:ro + environment: + PGUSER: citus + PGDATABASE: citus + expose: + - 5432 + command: pg_autoctl node run hostname: worker3a - command: | - pg_autoctl create worker --group 3 --name worker3a --run worker3b: - <<: *worker + <<: *worker3 hostname: worker3b - command: | - pg_autoctl create worker --group 3 --name worker3b --run diff --git a/docs/citus/ini/coordinator.ini b/docs/citus/ini/coordinator.ini new file mode 100644 index 000000000..3f84f7d86 --- /dev/null +++ b/docs/citus/ini/coordinator.ini @@ -0,0 +1,14 @@ +[node] +kind = coordinator +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/docs/citus/ini/monitor.ini b/docs/citus/ini/monitor.ini new file mode 100644 index 000000000..2aa88a73e --- /dev/null +++ b/docs/citus/ini/monitor.ini @@ -0,0 +1,10 @@ +[node] +kind = monitor +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[options] +ssl = self-signed +auth = trust diff --git a/docs/citus/ini/worker.ini b/docs/citus/ini/worker.ini new file mode 100644 index 000000000..677b3a6c9 --- /dev/null +++ b/docs/citus/ini/worker.ini @@ -0,0 +1,14 @@ +[node] +kind = worker +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/docs/citus/ini/worker1.ini b/docs/citus/ini/worker1.ini new file mode 100644 index 000000000..ad34e11eb --- /dev/null +++ b/docs/citus/ini/worker1.ini @@ -0,0 +1,17 @@ +[node] +kind = worker +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[formation] +group = 1 + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/docs/citus/ini/worker2.ini b/docs/citus/ini/worker2.ini new file mode 100644 index 000000000..798fe7884 --- /dev/null +++ b/docs/citus/ini/worker2.ini @@ -0,0 +1,17 @@ +[node] +kind = worker +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[formation] +group = 2 + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/docs/citus/ini/worker3.ini b/docs/citus/ini/worker3.ini new file mode 100644 index 000000000..6cecbdeb8 --- /dev/null +++ b/docs/citus/ini/worker3.ini @@ -0,0 +1,17 @@ +[node] +kind = worker +port = 5432 + +[postgresql] +pgdata = /tmp/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[formation] +group = 3 + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/docs/how-to.rst b/docs/how-to.rst index f99f61f09..e9720a7a0 100644 --- a/docs/how-to.rst +++ b/docs/how-to.rst @@ -59,6 +59,10 @@ you can manually run the following command on every node:: $ pg_autoctl run +For container and Kubernetes deployments, :ref:`pg_autoctl_node_run` +combines the create and run steps into a single command driven by a +``pg_autoctl_node.ini`` file — see :ref:`pg_autoctl_node` for details. + It is also possible (and recommended) to integrate the pg_auto_failover service in your usual service management facility. When using **systemd** the following commands can be used to produce the unit file configuration diff --git a/docs/install.rst b/docs/install.rst index 372550c16..0a69ac790 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -106,6 +106,13 @@ systemd itself, as it might be that a failover has been done during a reboot, for instance, and that once the reboot complete we want the local Postgres to re-join as a secondary node where it used to be a primary node. +For container and Kubernetes deployments, systemd is not used. Instead, +:ref:`pg_autoctl_node_run` acts as PID 1: it creates the node if absent, +then exec's into the supervisor. The standard Unix signal contract +(``SIGTERM`` to stop, ``SIGHUP`` to reload) is preserved because the +supervisor becomes the direct child process. See :ref:`pg_autoctl_node` +for the full reference. + Building pg_auto_failover from sources -------------------------------------- diff --git a/docs/operations.rst b/docs/operations.rst index 6149100d9..60a6566a0 100644 --- a/docs/operations.rst +++ b/docs/operations.rst @@ -19,6 +19,13 @@ pg_auto_failover monitor. The ``pg_autoctl create`` command honors the running. If Postgres is detected, the new node is registered in SINGLE mode, bypassing the monitor's role assignment policy. +For container and Kubernetes environments, :ref:`pg_autoctl_node_run` +provides a declarative alternative. The complete node description lives in +a single ``pg_autoctl_node.ini`` file that is bind-mounted into the +container; ``pg_autoctl node run`` creates the node if absent and starts the +supervisor in one step, making it a natural ``CMD`` / ``command:`` entry-point +for every node type. See `Container and Kubernetes Deployments`_ below. + Postgres configuration management --------------------------------- @@ -440,3 +447,54 @@ time -- for instance because a firewall rule hasn't yet activated -- it's possible to try ``pg_autoctl create`` again. pg_auto_failover will review its previous progress and repeat idempotent operations (``create database``, ``create extension`` etc), gracefully handling errors. + +.. _container-and-kubernetes-deployments: + +Container and Kubernetes Deployments +------------------------------------- + +:ref:`pg_autoctl_node` provides a declarative, file-driven entry-point +designed for container and Kubernetes environments. Instead of assembling +long ``pg_autoctl create`` flag sequences per node, the complete node +description lives in a single ``pg_autoctl_node.ini`` file:: + + [node] + kind = postgres + hostname = node1.internal + port = 5432 + + [postgresql] + pgdata = /var/lib/postgresql/data + + [monitor] + pguri = postgres://autoctl_node@monitor:5432/pg_auto_failover + + [settings] + candidate_priority = 50 + replication_quorum = true + + [options] + ssl = self-signed + +A single command then creates the node if absent and starts the supervisor:: + + pg_autoctl node run /etc/pgaf/node.ini + +This makes ``pg_autoctl node run`` a natural ``CMD`` (Docker) or +``command:`` (Kubernetes) entry-point for every node type. The same image +works for monitor, primary, standby, coordinator, and worker nodes — per-node +differences live entirely in the bind-mounted ini file. + +**Live reconfiguration** — the supervisor watches the ini file. Editing +``candidate_priority``, ``replication_quorum``, ``ssl`` settings, or +``monitor.pguri`` and saving the file is sufficient to converge the running +node; no restart is required. + +**Ordered startup** — add ``[launch] mode = deferred`` to any node that +should wait for an external signal before initialising. Call +``pg_autoctl node start `` from a sidecar or init container to release +it. This replaces external orchestration for the common case where data +nodes must wait until the monitor is ready. + +For the full property reference and mutability table see +:ref:`pg_autoctl_node`. diff --git a/docs/ref/configuration.rst b/docs/ref/configuration.rst index 6157c87a7..52d28a090 100644 --- a/docs/ref/configuration.rst +++ b/docs/ref/configuration.rst @@ -231,3 +231,50 @@ The pg_auto_failover keeper tries to restart PostgreSQL (default 3) or up to ``timeout.postgresql_restart_failure_timeout`` (defaults 20s) since it detected that PostgreSQL is not running, whichever comes first. + +.. _nodespec_configuration: + +Declarative Node Configuration (pg_autoctl_node.ini) +----------------------------------------------------- + +When using :ref:`pg_autoctl_node`, the node is described entirely in a +``pg_autoctl_node.ini`` file. This is an alternative to passing flags on +the ``pg_autoctl create`` command line, and it is the recommended approach +for container and Kubernetes deployments. + +The ini file maps every ``pg_autoctl create`` flag to a named key in one of +the following sections: + +``[node]`` + ``kind``, ``name``, ``hostname``, ``port`` + +``[postgresql]`` + ``pgdata`` + +``[monitor]`` + ``pguri``, ``no_monitor``, ``node_id`` + +``[formation]`` + ``name``, ``group`` + +``[settings]`` *(mutable — changes applied live without restart)* + ``candidate_priority``, ``replication_quorum`` + +``[options]`` + ``ssl`` *(mutable — applied via* ``pg_autoctl enable ssl`` *)*, + ``auth`` *(create-time only)*, + ``pg_hba_lan`` *(create-time only)* + +``[ssl]`` *(mutable — changes applied via* ``pg_autoctl enable ssl`` *)* + ``ca_file``, ``cert_file``, ``key_file`` + +The pg_autoctl keeper configuration file (``pg_autoctl.cfg`` inside +``PGDATA``) is still the authoritative runtime configuration. The +``pg_autoctl_node.ini`` file is read at startup and on every file-change +event; its mutable fields are converged into the running node state, while +immutable fields are only applied at node creation time. + +Use ``pg_autoctl node show --pgdata `` to generate a +``pg_autoctl_node.ini`` from an existing node's current configuration. + +See :ref:`pg_autoctl_node` for the full property reference. diff --git a/docs/ref/manual.rst b/docs/ref/manual.rst index 93638bea5..d71e00d7b 100644 --- a/docs/ref/manual.rst +++ b/docs/ref/manual.rst @@ -20,6 +20,7 @@ have their own manual page. pg_autoctl_get pg_autoctl_set pg_autoctl_perform + pg_autoctl_node pg_autoctl_inspect pg_autoctl_manual pg_autoctl_do diff --git a/docs/ref/pg_autoctl_create_coordinator.rst b/docs/ref/pg_autoctl_create_coordinator.rst index 40cf8f914..e5da597e6 100644 --- a/docs/ref/pg_autoctl_create_coordinator.rst +++ b/docs/ref/pg_autoctl_create_coordinator.rst @@ -90,3 +90,12 @@ options. This section now lists the options that are specific to are part of this cluster. See :ref:`citus_secondaries` for more information. + +See Also +-------- + +:ref:`pg_autoctl_node_run` provides a declarative alternative to this +command: set ``kind = coordinator`` in a ``pg_autoctl_node.ini`` file and +run ``pg_autoctl node run`` — it creates the coordinator if absent and starts +the supervisor in one step. See :ref:`pg_autoctl_node` for the full +reference. diff --git a/docs/ref/pg_autoctl_create_monitor.rst b/docs/ref/pg_autoctl_create_monitor.rst index f65c5d6e9..c3a5a73e0 100644 --- a/docs/ref/pg_autoctl_create_monitor.rst +++ b/docs/ref/pg_autoctl_create_monitor.rst @@ -201,3 +201,11 @@ XDG_DATA_HOME __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +See Also +-------- + +:ref:`pg_autoctl_node_run` provides a declarative alternative to this +command: set ``kind = monitor`` in a ``pg_autoctl_node.ini`` file and run +``pg_autoctl node run`` — it creates the monitor if absent and starts the +supervisor in one step. See :ref:`pg_autoctl_node` for the full reference. diff --git a/docs/ref/pg_autoctl_create_postgres.rst b/docs/ref/pg_autoctl_create_postgres.rst index e800d1d0f..b2d8b2089 100644 --- a/docs/ref/pg_autoctl_create_postgres.rst +++ b/docs/ref/pg_autoctl_create_postgres.rst @@ -370,3 +370,11 @@ XDG_DATA_HOME Base Directory Specification`__. __ https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html + +See Also +-------- + +:ref:`pg_autoctl_node_run` provides a declarative alternative to this +command: describe the node once in a ``pg_autoctl_node.ini`` file and run +``pg_autoctl node run`` — it creates the node if absent and starts the +supervisor in one step. See :ref:`pg_autoctl_node` for the full reference. diff --git a/docs/ref/pg_autoctl_create_worker.rst b/docs/ref/pg_autoctl_create_worker.rst index daf301e73..6537e56ef 100644 --- a/docs/ref/pg_autoctl_create_worker.rst +++ b/docs/ref/pg_autoctl_create_worker.rst @@ -112,3 +112,11 @@ options. This section now lists the options that are specific to are part of this cluster. See :ref:`citus_secondaries` for more information. + +See Also +-------- + +:ref:`pg_autoctl_node_run` provides a declarative alternative to this +command: set ``kind = worker`` in a ``pg_autoctl_node.ini`` file and run +``pg_autoctl node run`` — it creates the worker if absent and starts the +supervisor in one step. See :ref:`pg_autoctl_node` for the full reference. diff --git a/docs/ref/pg_autoctl_node.rst b/docs/ref/pg_autoctl_node.rst new file mode 100644 index 000000000..e84a32b44 --- /dev/null +++ b/docs/ref/pg_autoctl_node.rst @@ -0,0 +1,293 @@ +.. _pg_autoctl_node: + +pg_autoctl node +=============== + +pg_autoctl node - Declarative node lifecycle from a single configuration file + +Synopsis +-------- + +``pg_autoctl node`` manages the full lifecycle of a pg_auto_failover node — +creation, startup, and live reconfiguration — driven by a single +``pg_autoctl_node.ini`` file rather than long sequences of command-line flags:: + + pg_autoctl node + run Create (if needed) and run a node described by a pg_autoctl_node.ini file + apply Apply mutable settings from a pg_autoctl_node.ini to a running node + start Start a node waiting in launch=deferred mode (idempotent) + show Dump current node configuration as a pg_autoctl_node.ini file + check Validate a pg_autoctl_node.ini file without creating anything + +.. toctree:: + :maxdepth: 1 + + pg_autoctl_node_run + pg_autoctl_node_apply + pg_autoctl_node_start + pg_autoctl_node_show + pg_autoctl_node_check + +Description +----------- + +``pg_autoctl node run `` is the recommended entry-point for container +and Kubernetes deployments. The complete node description lives in one ini +file that can be version-controlled, templated, and bind-mounted into a +container. The same image and the same entry-point work for every node type +(monitor, primary, standby, Citus coordinator, Citus worker); per-node +differences live entirely in the mounted ini file. + +The ``pg_autoctl_node.ini`` File +-------------------------------- + +The file uses ``.ini`` sections. A typical data node:: + + [node] + kind = postgres + name = node1 + hostname = node1.internal + port = 5432 + + [postgresql] + pgdata = /var/lib/postgresql/data + + [monitor] + pguri = postgres://autoctl_node@monitor:5432/pg_auto_failover + + [formation] + name = default + + [settings] + candidate_priority = 50 + replication_quorum = true + + [options] + ssl = self-signed + auth = trust + pg_hba_lan = true + +A monitor node omits ``[monitor]`` entirely:: + + [node] + kind = monitor + hostname = monitor.internal + port = 5432 + + [postgresql] + pgdata = /var/lib/postgresql/monitor + + [formation default] + kind = pgsql + +Section and Property Reference +------------------------------- + +The following sections and properties are supported in +``pg_autoctl_node.ini``. Properties marked *mutable* can be changed on a +running node by editing the file; the supervisor converges the change without +restarting Postgres. Immutable properties take effect only the next time the +node is created or started from scratch. + +``[node]`` +^^^^^^^^^^ + +``kind`` + + Node role. One of ``postgres``, ``monitor``, ``coordinator``, or + ``worker``. Required; immutable. + +``name`` + + Human-readable node name shown in ``pg_autoctl show state``. Defaults to + the value of ``hostname`` when not set. Immutable. + +``hostname`` + + Address that other nodes (including the monitor) use to reach this node. + Defaults to the auto-detected FQDN of the host. Immutable. + +``port`` + + Postgres port number. Defaults to ``5432``. Immutable. + +``[postgresql]`` +^^^^^^^^^^^^^^^^ + +``pgdata`` + + Path to the Postgres data directory. Required; immutable. + +``[monitor]`` +^^^^^^^^^^^^^ + +``pguri`` + + Connection string to the pg_auto_failover monitor. Omit for monitor + nodes. **Mutable**: changing this value re-registers the node to the new + monitor without restarting Postgres (``pg_autoctl disable monitor --force`` + followed by ``pg_autoctl enable monitor ``). + +``no_monitor`` + + Set to ``true`` to run in :ref:`pg_autoctl_disable_monitor` (standalone) + mode. Immutable. + +``node_id`` + + Node identifier to use when ``no_monitor = true``. Immutable. + +``[formation]`` +^^^^^^^^^^^^^^^ + +``name`` + + Formation name. Defaults to ``default``. Immutable. + +``group`` + + Citus group identifier. ``0`` means coordinator. Defaults to ``0``. + Immutable. + +``[settings]`` +^^^^^^^^^^^^^^ + +Settings in this section are applied live without restarting the node. + +``candidate_priority`` + + Failover weight, an integer from 0 to 100. A value of ``0`` means this + node is never promoted to primary. Defaults to ``50``. **Mutable**. + +``replication_quorum`` + + Whether this node participates in the synchronous replication quorum. + Boolean, defaults to ``true``. **Mutable**. + +``[options]`` +^^^^^^^^^^^^^ + +``ssl`` + + SSL mode. One of ``self-signed``, ``verify-ca``, ``verify-full``, or + ``off``. Defaults to ``self-signed``. **Mutable**: changes are applied via + ``pg_autoctl enable ssl``; Postgres reloads SSL without a full restart. + +``auth`` + + Authentication method written into ``pg_hba.conf`` at create time. One of + ``trust``, ``md5``, ``scram``, or ``cert``. Defaults to ``trust``. + Immutable (create-time only). + +``pg_hba_lan`` + + When ``true``, add LAN-range entries to ``pg_hba.conf`` at create time. + Defaults to ``true``. Immutable (create-time only). + +``[ssl]`` +^^^^^^^^^ + +Certificate paths for ``verify-ca`` and ``verify-full`` SSL modes. All +three are **mutable**: editing them and saving the file reconfigures Postgres +SSL live via ``pg_autoctl enable ssl``. + +``ca_file`` + + Path to the CA certificate file (``ssl_ca_file`` in ``postgresql.conf``). + +``cert_file`` + + Path to the server certificate file (``ssl_cert_file``). + +``key_file`` + + Path to the server private key file (``ssl_key_file``). + +``[launch]`` +^^^^^^^^^^^^ + +``mode`` + + When set to ``deferred``, the node starts a polling loop and waits instead + of creating or starting Postgres immediately. Call ``pg_autoctl node + start`` to release it. Defaults to ``immediate``. See + :ref:`pg_autoctl_node_start`. + +``[formation ]`` +^^^^^^^^^^^^^^^^^^^^^^ + +Repeat this section for each non-default formation to create. Applies to +monitor nodes only. + +``kind`` + + Formation kind. One of ``pgsql`` or ``citus``. Defaults to ``pgsql``. + Immutable. + +Password sections +^^^^^^^^^^^^^^^^^ + +These sections hold credentials that are kept out of the main ini file where +possible. + +``[pg_auto_failover]`` + ``autoctl_node_password`` — password for the ``pgautofailover_monitor`` + role (monitor nodes) or the ``autoctl_node`` role (data nodes). + +``[replication]`` + ``replication_password`` — password for the ``pgautofailover_replicator`` + role. + +``[citus]`` + ``role = secondary`` and ``cluster_name`` for Citus secondary clusters. + +Live Reconfiguration +-------------------- + +The supervisor that ``pg_autoctl node run`` exec's into watches the ini file +for changes. When it detects a write it re-reads the file and converges any +**mutable** fields without restarting the node or interrupting replication: + +``candidate_priority`` and ``replication_quorum`` + Applied by calling ``pg_autoctl set node`` against the running node. + +``ssl``, ``ca_file``, ``cert_file``, ``key_file`` + Applied by calling ``pg_autoctl enable ssl`` with the appropriate flags. + Postgres reloads its SSL configuration without a full restart. + +``monitor.pguri`` + Applied by calling ``pg_autoctl disable monitor --force`` (which removes + the node from the old monitor) followed by + ``pg_autoctl enable monitor `` (which registers the node to the + new monitor). Postgres keeps running throughout; only the monitoring + relationship changes. + +Changing an **immutable** field (``kind``, ``pgdata``, ``hostname``, ``port``, +``auth``, ``pg_hba_lan``) while the node is running is logged as a warning; +the value takes effect the next time the node is started. + +The ``launch = deferred`` Pattern +---------------------------------- + +:: + + [launch] + mode = deferred + +A node configured with ``mode = deferred`` starts a polling loop and waits. +A sidecar container or init script then calls:: + + pg_autoctl node start /etc/pgaf/node.ini + +which rewrites the ini file with ``mode = immediate``. The waiting node +detects the change within the poll interval and proceeds to create or run. +This enables ordered startup without an external orchestrator: the monitor +container can be given ``mode = immediate`` while all data nodes start with +``mode = deferred``, and each data node is released with ``node start`` only +after the monitor is confirmed ready. + +See Also +-------- + +:ref:`pg_autoctl_node_run`, :ref:`pg_autoctl_create_postgres`, +:ref:`pg_autoctl_run`, :ref:`pg_autoctl_set` diff --git a/docs/ref/pg_autoctl_node_apply.rst b/docs/ref/pg_autoctl_node_apply.rst new file mode 100644 index 000000000..ea3cf2986 --- /dev/null +++ b/docs/ref/pg_autoctl_node_apply.rst @@ -0,0 +1,55 @@ +.. _pg_autoctl_node_apply: + +pg_autoctl node apply +===================== + +pg_autoctl node apply - Apply mutable settings from a pg_autoctl_node.ini to a running node + +Synopsis +-------- + +:: + + pg_autoctl node apply [] + + path to the pg_autoctl_node.ini file + (default: /etc/pgaf/node.ini) + +Description +----------- + +``pg_autoctl node apply`` reads a ``pg_autoctl_node.ini`` file, compares its +mutable fields against the node's current configuration, and converges any +differences without restarting the node or interrupting replication. + +This is the same convergence logic the supervisor runs automatically when it +detects a file change. Running ``pg_autoctl node apply`` manually is useful +when the supervisor is not running, or when you want to apply a change +immediately rather than waiting for the next watch interval. + +Mutable fields applied by this command: + +``candidate_priority`` + Applied via :ref:`pg_autoctl_set_node_candidate_priority`. + +``replication_quorum`` + Applied via :ref:`pg_autoctl_set_node_replication_quorum`. + +``ssl``, ``ca_file``, ``cert_file``, ``key_file`` + Applied via ``pg_autoctl enable ssl``. Postgres reloads its SSL + configuration without a full restart. + +``monitor.pguri`` + Applied via ``pg_autoctl disable monitor --force`` followed by + ``pg_autoctl enable monitor ``. The node is re-registered to + the new monitor without stopping Postgres. + +Immutable fields (``kind``, ``pgdata``, ``hostname``, ``port``, +``auth``, ``pg_hba_lan``) are logged as warnings when they differ; they take +effect only the next time the node is created or started from scratch. + +See Also +-------- + +:ref:`pg_autoctl_node`, :ref:`pg_autoctl_node_run`, +:ref:`pg_autoctl_set`, :ref:`pg_autoctl_enable` diff --git a/docs/ref/pg_autoctl_node_check.rst b/docs/ref/pg_autoctl_node_check.rst new file mode 100644 index 000000000..deed6b7cf --- /dev/null +++ b/docs/ref/pg_autoctl_node_check.rst @@ -0,0 +1,40 @@ +.. _pg_autoctl_node_check: + +pg_autoctl node check +===================== + +pg_autoctl node check - Validate a pg_autoctl_node.ini file without creating anything + +Synopsis +-------- + +:: + + pg_autoctl node check [] + + path to the pg_autoctl_node.ini file + (default: /etc/pgaf/node.ini) + +Description +----------- + +``pg_autoctl node check`` parses a ``pg_autoctl_node.ini`` file, resolves +defaults, and prints the fully-resolved configuration to stdout. It exits +with a non-zero status if the file contains any syntax errors or invalid +field values. + +No Postgres instance is touched and no pg_autoctl state is modified. This +command is safe to run at any time. + +Use ``pg_autoctl node check`` to: + +- Validate a newly written ini file before deploying it to a container. +- Verify that defaults are resolved as expected (for example, that + ``hostname`` is auto-detected correctly). +- Catch errors in CI before reaching the node creation step. + +See Also +-------- + +:ref:`pg_autoctl_node`, :ref:`pg_autoctl_node_run`, +:ref:`pg_autoctl_node_show` diff --git a/docs/ref/pg_autoctl_node_run.rst b/docs/ref/pg_autoctl_node_run.rst new file mode 100644 index 000000000..51140694c --- /dev/null +++ b/docs/ref/pg_autoctl_node_run.rst @@ -0,0 +1,68 @@ +.. _pg_autoctl_node_run: + +pg_autoctl node run +=================== + +pg_autoctl node run - Create (if needed) and run a node from a pg_autoctl_node.ini file + +Synopsis +-------- + +:: + + pg_autoctl node run [] + + path to the pg_autoctl_node.ini file + (default: /etc/pgaf/node.ini) + +Description +----------- + +``pg_autoctl node run`` is the single entry-point for container-based +deployments. Given a ``pg_autoctl_node.ini`` file it: + +1. Reads and validates the ini file. +2. If ``[launch] mode = deferred``, polls the file until the section is + removed or changed to ``mode = immediate`` (see ``pg_autoctl node start``). +3. Checks whether the node already exists (looks for ``pg_autoctl.cfg`` + inside ``pgdata``). + + - **First start** — builds the ``pg_autoctl create [flags] --run`` + argument list from the ini file and exec's into it, which creates Postgres + and starts the supervisor in one step. + - **Subsequent starts** — applies any mutable setting changes found in the + ini file, then exec's into ``pg_autoctl run --pgdata ``. + +4. Sets the ``PG_AUTOCTL_NODESPEC`` environment variable to the ini file + path before exec'ing, so the supervisor can watch the file for live + changes to ``[settings]``. + +Because the command uses ``execv()``, the pg_autoctl supervisor becomes +the direct child process (PID 1 in a container), preserving the standard +Unix signal contract — ``SIGTERM`` stops the supervisor cleanly, +``SIGHUP`` reloads configuration. + +The ``launch = deferred`` pattern +---------------------------------- + +The ``[launch]`` section enables ordered startup without an external +orchestrator:: + + [launch] + mode = deferred + +A node with ``mode = deferred`` starts the polling loop and waits. A +second container, sidecar, or init script calls:: + + pg_autoctl node start /etc/pgaf/node.ini + +which rewrites the file with ``mode = immediate``. The waiting node detects +the change and proceeds. This is useful when you need to ensure the monitor +is fully up before any data node attempts registration, or when bringing up +Citus workers in a specific order. + +See Also +-------- + +:ref:`pg_autoctl_node`, :ref:`pg_autoctl_create_postgres`, +:ref:`pg_autoctl_run` diff --git a/docs/ref/pg_autoctl_node_show.rst b/docs/ref/pg_autoctl_node_show.rst new file mode 100644 index 000000000..0a2b6e509 --- /dev/null +++ b/docs/ref/pg_autoctl_node_show.rst @@ -0,0 +1,40 @@ +.. _pg_autoctl_node_show: + +pg_autoctl node show +==================== + +pg_autoctl node show - Dump current node configuration as a pg_autoctl_node.ini file + +Synopsis +-------- + +:: + + pg_autoctl node show [--pgdata ] + + --pgdata path to data directory (default: $PGDATA) + +Description +----------- + +``pg_autoctl node show`` reads the running node's ``pg_autoctl.cfg`` +configuration file and writes its contents to stdout in +``pg_autoctl_node.ini`` format. + +This is useful for: + +- Capturing the current configuration of a node that was created with + ``pg_autoctl create postgres`` flags, so it can be managed declaratively + going forward. +- Comparing the effective configuration of a running node against a desired + ``pg_autoctl_node.ini`` file. +- Generating a baseline ini file to store in version control. + +The output is a valid ``pg_autoctl_node.ini`` file that can be passed +directly to ``pg_autoctl node run`` or ``pg_autoctl node apply``. + +See Also +-------- + +:ref:`pg_autoctl_node`, :ref:`pg_autoctl_node_run`, +:ref:`pg_autoctl_node_check` diff --git a/docs/ref/pg_autoctl_node_start.rst b/docs/ref/pg_autoctl_node_start.rst new file mode 100644 index 000000000..c70718658 --- /dev/null +++ b/docs/ref/pg_autoctl_node_start.rst @@ -0,0 +1,49 @@ +.. _pg_autoctl_node_start: + +pg_autoctl node start +===================== + +pg_autoctl node start - Release a node waiting in launch=deferred mode + +Synopsis +-------- + +:: + + pg_autoctl node start [] + + path to the pg_autoctl_node.ini file + (default: /etc/pgaf/node.ini) + +Description +----------- + +``pg_autoctl node start`` releases a node that is waiting in +``[launch] mode = deferred``. It rewrites the ini file with +``mode = immediate``; the waiting node detects the change within the poll +interval and proceeds to create or run. + +This command is idempotent: calling it on a node that is already running +(or has ``mode = immediate``) is a no-op. + +The ``launch = deferred`` Pattern +---------------------------------- + +A node configured with ``[launch] mode = deferred`` starts a polling loop +and waits instead of immediately creating or starting Postgres. This +enables ordered startup without an external orchestrator:: + + # In the ini file for each data node: + [launch] + mode = deferred + +The monitor can be given ``mode = immediate`` (the default), while data nodes +start with ``mode = deferred``. Once the monitor is confirmed ready, release +each data node:: + + pg_autoctl node start /etc/pgaf/node.ini + +See Also +-------- + +:ref:`pg_autoctl_node`, :ref:`pg_autoctl_node_run` diff --git a/docs/ref/pg_autoctl_run.rst b/docs/ref/pg_autoctl_run.rst index 08d8650cd..f57f02e79 100644 --- a/docs/ref/pg_autoctl_run.rst +++ b/docs/ref/pg_autoctl_run.rst @@ -87,3 +87,11 @@ Options --pgport Postgres port to use, defaults to 5432. + +See Also +-------- + +:ref:`pg_autoctl_node_run` combines node creation and the service run in a +single command driven by a ``pg_autoctl_node.ini`` file — it is the +recommended entry-point for new deployments, especially in container and +Kubernetes environments. diff --git a/docs/tutorial.rst b/docs/tutorial.rst index 881058957..f9917c3b5 100644 --- a/docs/tutorial.rst +++ b/docs/tutorial.rst @@ -61,31 +61,45 @@ provide with High Availability of both the Postgres service and the data. See the :ref:`multi_node_architecture` chapter of our docs to understand more about this. -To create a cluster we use the following docker compose definition: +Each node in the cluster is described by a small ``pg_autoctl_node.ini`` +file that is bind-mounted into its container. There are two files: + +.. literalinclude:: tutorial/ini/monitor.ini + :language: ini + :caption: tutorial/ini/monitor.ini + +.. literalinclude:: tutorial/ini/postgres.ini + :language: ini + :caption: tutorial/ini/postgres.ini + +The monitor file sets ``kind = monitor``; it has no ``[monitor]`` section +because the monitor is not itself a client of a monitor. The postgres file +is shared by every data node — the node's ``name`` and ``hostname`` are +omitted, so they default to the container hostname set by Docker Compose +(``node1``, ``node2``, ``node3``). + +The docker compose definition that assembles these pieces is: .. literalinclude:: tutorial/docker-compose.yml :language: yaml :linenos: -To run the full Citus cluster with HA from this definition, we can use the -following command: - -:: - - $ docker compose up app monitor node1 node2 +Every service runs the same entry-point — ``pg_autoctl node run`` — with a +different ini file bind-mounted at ``/etc/pgaf/node.ini``. On first start +the command creates the Postgres cluster and registers the node; on +subsequent starts it picks up the existing cluster and runs the supervisor. +Either way the container never needs updating to change node configuration: +edit the ini file and restart. -The command above starts the services up. The first service is the monitor -and is created with the command ``pg_autoctl create monitor``. The options -for this command are exposed in the environment, and could have been -specified on the command line too: +To start the two-node cluster run: :: - $ pg_autoctl create postgres --ssl-self-signed --auth trust --pg-hba-lan --run + $ docker compose up app monitor node1 node2 -While the Postgres nodes are being provisionned by docker compose, you can -run the following command and have a dynamic dashboard to follow what's -happening. The following command is like ``top`` for pg_auto_failover:: +While the nodes are being provisioned you can run the following command and +have a dynamic dashboard to follow what's happening. The following command +is like ``top`` for pg_auto_failover:: $ docker compose exec monitor pg_autoctl watch @@ -235,18 +249,75 @@ We can see the resulting replication settings with the following command: Editing the replication settings while in production ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -It's then possible to change the production architecture obtained with -playing with the :ref:`architecture_setup` commands. Specifically, try the -following command to change the candidate_priority of the node3 to zero, in -order for it to never be a candidate for failover: +Because node3 uses the same ``postgres.ini`` as node1 and node2, its +``candidate_priority`` starts at ``50``. To make node3 a read-only standby +that is never promoted there are two ways to proceed. -:: +**Direct command** — takes effect immediately on the running cluster:: $ docker compose exec node3 pg_autoctl set candidate-priority 0 --name node3 -To see the replication settings for all the nodes, the following command can -be useful, and is described in more details in the :ref:`architecture_setup` -section. +This reaches into the running supervisor and updates the setting on the +monitor in one step. It is the fastest option when you need an immediate +change on a live cluster. + +**Declarative ini file** — the persistent, version-controlled option. +Because node3 shares ``postgres.ini`` with the other two nodes, you first +create a dedicated ini file for it: + +.. code-block:: ini + + # tutorial/ini/node3.ini + [node] + kind = postgres + port = 5432 + + [postgresql] + pgdata = /var/lib/postgres/pgaf + + [monitor] + pguri = postgresql://autoctl_node@monitor/pg_auto_failover + + [settings] + candidate_priority = 0 + replication_quorum = true + + [options] + ssl = self-signed + auth = trust + pg_hba_lan = true + +Then update the ``node3`` service in ``docker-compose.yml`` to mount this +file instead of the shared one:: + + node3: + <<: *node + hostname: node3 + volumes: + - /var/lib/postgres + - ./ini/node3.ini:/etc/pgaf/node.ini:ro + +Changing the ``volumes:`` list requires recreating the container — Docker +cannot swap a bind mount into a running container. Run:: + + $ docker compose up -d node3 + +Docker Compose stops node3, recreates it with the new mount, and starts it +again. ``pg_autoctl node run`` detects that the Postgres cluster already +exists inside the volume, reads ``node3.ini``, and calls ``pg_autoctl node +apply`` before exec'ing into the supervisor. The apply step calls +``pg_autoctl set node candidate-priority 0``, registering the change on the +monitor. node3 rejoins the formation as a secondary within a few seconds. + +.. note:: + + Once node3 has its own ini file mounted, any further edits to + ``tutorial/ini/node3.ini`` on the host are picked up live by the running + supervisor — Docker bind mounts reflect host-side writes immediately, and + the supervisor's file watcher applies mutable fields (``candidate_priority``, + ``replication_quorum``, ``ssl``) without restarting the container. + +Verify with: :: @@ -298,12 +369,13 @@ To dispose of the entire tutorial environment, just use the following command: Next steps ---------- -As mentioned in the first section of this tutorial, the way we use -docker compose here is not meant to be production ready. It's useful to -understand and play with a distributed system such as Postgres multiple -nodes system and failovers. +The docker compose setup in this tutorial is a good playground for +understanding pg_auto_failover's behaviour. For production container and +Kubernetes deployments, the same ``pg_autoctl node run`` entry-point scales +directly: store your ini files in a ConfigMap or alongside your Compose +file, bind-mount them, and live reconfiguration handles the rest. -See the command :ref:`pg_autoctl_do_tmux_compose_session` for more details -about how to run a docker compose test environment with docker compose, -including external volumes for each node. +See :ref:`pg_autoctl_node` for the full property reference and mutability +table, and the :ref:`container-and-kubernetes-deployments` section of the +Operations guide for production patterns. diff --git a/docs/tutorial/docker-compose.yml b/docs/tutorial/docker-compose.yml index 3a60594d9..7128120f3 100644 --- a/docs/tutorial/docker-compose.yml +++ b/docs/tutorial/docker-compose.yml @@ -6,16 +6,13 @@ x-node: &node - pg_auto_failover:tutorial volumes: - /var/lib/postgres + - ./ini/postgres.ini:/etc/pgaf/node.ini:ro environment: - PGDATA: /var/lib/postgres/pgaf PGUSER: tutorial PGDATABASE: tutorial - PG_AUTOCTL_HBA_LAN: true - PG_AUTOCTL_AUTH_METHOD: "trust" - PG_AUTOCTL_SSL_SELF_SIGNED: true - PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover" expose: - 5432 + command: pg_autoctl node run services: @@ -36,28 +33,19 @@ services: image: pg_auto_failover:tutorial volumes: - /var/lib/postgres - environment: - PGDATA: /var/lib/postgres/pgaf - PG_AUTOCTL_SSL_SELF_SIGNED: true + - ./ini/monitor.ini:/etc/pgaf/node.ini:ro expose: - 5432 - command: | - pg_autoctl create monitor --auth trust --run + command: pg_autoctl node run node1: <<: *node hostname: node1 - command: | - pg_autoctl create postgres --name node1 --run node2: <<: *node hostname: node2 - command: | - pg_autoctl create postgres --name node2 --run node3: <<: *node hostname: node3 - command: | - pg_autoctl create postgres --name node3 --run diff --git a/docs/tutorial/ini/monitor.ini b/docs/tutorial/ini/monitor.ini new file mode 100644 index 000000000..ca835df78 --- /dev/null +++ b/docs/tutorial/ini/monitor.ini @@ -0,0 +1,10 @@ +[node] +kind = monitor +port = 5432 + +[postgresql] +pgdata = /var/lib/postgres/pgaf + +[options] +ssl = self-signed +auth = trust diff --git a/docs/tutorial/ini/postgres.ini b/docs/tutorial/ini/postgres.ini new file mode 100644 index 000000000..8c54b5ec7 --- /dev/null +++ b/docs/tutorial/ini/postgres.ini @@ -0,0 +1,18 @@ +[node] +kind = postgres +port = 5432 + +[postgresql] +pgdata = /var/lib/postgres/pgaf + +[monitor] +pguri = postgresql://autoctl_node@monitor/pg_auto_failover + +[settings] +candidate_priority = 50 +replication_quorum = true + +[options] +ssl = self-signed +auth = trust +pg_hba_lan = true diff --git a/src/bin/pg_autoctl/cli_node.c b/src/bin/pg_autoctl/cli_node.c new file mode 100644 index 000000000..639480743 --- /dev/null +++ b/src/bin/pg_autoctl/cli_node.c @@ -0,0 +1,534 @@ +/* + * src/bin/pg_autoctl/cli_node.c + * pg_autoctl node — declarative node lifecycle driven by a pg_autoctl_node.ini file. + * + * pg_autoctl node run Create (if needed) then run the supervisor. + * pg_autoctl node apply Converge mutable fields on a running node. + * pg_autoctl node show Dump current config as pg_autoctl_node.ini. + * pg_autoctl node check Validate the file without creating anything. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + */ + +#include +#include +#include +#include +#include + +#include "cli_common.h" +#include "cli_node.h" +#include "cli_root.h" +#include "commandline.h" +#include "defaults.h" +#include "env_utils.h" +#include "file_utils.h" +#include "keeper_config.h" +#include "log.h" +#include "nodespec.h" +#include "pgsetup.h" +#include "runprogram.h" +#include "string_utils.h" + + +static int cli_node_run_getopts(int argc, char **argv); +static void cli_node_run(int argc, char **argv); +static int cli_node_apply_getopts(int argc, char **argv); +static void cli_node_apply(int argc, char **argv); +static int cli_node_start_getopts(int argc, char **argv); +static void cli_node_start(int argc, char **argv); +static int cli_node_show_getopts(int argc, char **argv); +static void cli_node_show(int argc, char **argv); +static int cli_node_check_getopts(int argc, char **argv); +static void cli_node_check(int argc, char **argv); + + +static CommandLine node_run_command = + make_command( + "run", + "Create (if needed) and run a node described by a pg_autoctl_node.ini file", + "", + " path to the pg_autoctl_node.ini file\n" + " (default: " PG_AUTOCTL_NODESPEC_PATH ")\n", + cli_node_run_getopts, + cli_node_run); + +static CommandLine node_apply_command = + make_command( + "apply", + "Apply mutable settings from a pg_autoctl_node.ini to a running node", + "", + " path to the pg_autoctl_node.ini file\n", + cli_node_apply_getopts, + cli_node_apply); + +static CommandLine node_start_command = + make_command( + "start", + "Start a node waiting in launch=deferred mode (idempotent)", + "[]", + " path to the pg_autoctl_node.ini file\n" + " (default: " PG_AUTOCTL_NODESPEC_PATH ")\n", + cli_node_start_getopts, + cli_node_start); + +static CommandLine node_show_command = + make_command( + "show", + "Dump current node configuration as a pg_autoctl_node.ini file", + "[--pgdata ]", + " --pgdata location of the Postgres data directory\n", + cli_node_show_getopts, + cli_node_show); + +static CommandLine node_check_command = + make_command( + "check", + "Validate a pg_autoctl_node.ini file without creating anything", + "", + " path to the pg_autoctl_node.ini file\n", + cli_node_check_getopts, + cli_node_check); + +static CommandLine *node_subcommands[] = { + &node_run_command, + &node_apply_command, + &node_start_command, + &node_show_command, + &node_check_command, + NULL +}; + +CommandLine node_commands = + make_command_set( + "node", + "Declarative node lifecycle — create, run, and reconfigure from a .ini file", + NULL, NULL, NULL, node_subcommands); + + +/* ----------------------------------------------------------------------- + * Shared state + * ----------------------------------------------------------------------- */ + +static char nodeSpecPath[MAXPGPATH] = { 0 }; + + +/* ----------------------------------------------------------------------- + * pg_autoctl node run + * ----------------------------------------------------------------------- */ +static int +cli_node_run_getopts(int argc, char **argv) +{ + /* argv[0] is the subcommand name ("run"/"apply"/"check"); the optional + * file path is the first remaining positional argument at argv[1]. */ + if (argc > 1 && argv[1][0] != '-') + { + strlcpy(nodeSpecPath, argv[1], sizeof(nodeSpecPath)); + } + else + { + strlcpy(nodeSpecPath, PG_AUTOCTL_NODESPEC_PATH, sizeof(nodeSpecPath)); + } + + return 0; +} + + +/* + * cli_node_run reads the pg_autoctl_node.ini file, runs `pg_autoctl create + * --run` if PGDATA does not yet exist, or `pg_autoctl run` if it does. + * + * We exec() rather than call the internal functions directly so that: + * 1. The supervisor's fork+exec live-upgrade pattern is preserved. + * 2. The already-running supervisor (PID 1) can restart this process after + * a binary upgrade without special-casing the "node run" path. + */ +static void +cli_node_run(int argc, char **argv) +{ + NodeSpec spec = { 0 }; + char *args[40]; + int nargs; + + if (!nodespec_read(nodeSpecPath, &spec)) + { + exit(EXIT_CODE_BAD_CONFIG); + } + + /* + * [launch] mode = deferred: spin here re-reading nodeSpecPath until + * pg_autoctl node start rewrites it with mode = immediate (or removes + * the [launch] section). The file is the rendezvous point — no external + * sentinel needed. + */ + if (spec.launchDeferred) + { + log_info("Node configured with launch = deferred in \"%s\"; " + "waiting for pg_autoctl node start", nodeSpecPath); + + for (;;) + { + pg_usleep(500 * 1000); /* 0.5 s poll */ + + NodeSpec polled = { 0 }; + if (nodespec_read(nodeSpecPath, &polled) && !polled.launchDeferred) + { + spec = polled; + break; + } + } + + log_info("launch = immediate; proceeding with node initialization"); + } + + /* + * If PGDATA already has a pg_autoctl.cfg, the node was created before. + * In that case run `pg_autoctl run` (no --run, no create flags). + * Otherwise run `pg_autoctl create ... --run`. + */ + char cfgPath[MAXPGPATH]; + bool cfgExists = false; + + if (!IS_EMPTY_STRING_BUFFER(spec.pgdata)) + { + sformat(cfgPath, sizeof(cfgPath), + "%s/pg_autoctl.cfg", spec.pgdata); + cfgExists = file_exists(cfgPath); + } + + if (cfgExists) + { + /* + * Node was already created. Apply any mutable changes that might + * have been made to the spec file since the last run, then hand off + * to the normal run path. + */ + NodeSpec prev = { 0 }; + + /* best-effort: ignore errors if we can't re-read the spec */ + (void) nodespec_read(nodeSpecPath, &prev); + (void) nodespec_apply(&spec, &prev); + + args[0] = (char *) pg_autoctl_program; + args[1] = "run"; + args[2] = "--pgdata"; + args[3] = spec.pgdata; + args[4] = NULL; + nargs = 4; + } + else + { + nargs = nodespec_create_argv(&spec, pg_autoctl_program, + args, 32); + if (nargs < 0) + { + exit(EXIT_CODE_INTERNAL_ERROR); + } + } + + /* Tell the supervisor which spec file to watch for live changes */ + setenv("PG_AUTOCTL_NODESPEC", nodeSpecPath, 1); + + /* + * PG_AUTOCTL_TEST_DELAY: stagger node registration so that node IDs + * are assigned in name order (node1 → id 1, node2 → id 2, …). + * Extract the trailing integer from the node name and sleep 2×N seconds. + */ + if (env_exists("PG_AUTOCTL_TEST_DELAY") && spec.name[0] != '\0') + { + const char *p = spec.name + strlen(spec.name); + while (p > spec.name && isdigit((unsigned char) p[-1])) + { + p--; + } + if (*p != '\0') + { + int n = atoi(p) /* IGNORE-BANNED */; + int secs = 2 * n; + log_info("PG_AUTOCTL_TEST_DELAY: sleeping %ds before " + "registration (node %s, index %d)", + secs, spec.name, n); + sleep(secs); + } + } + + /* Log the command we're about to exec, masking password arguments. */ + { + PQExpBuffer cmd = createPQExpBuffer(); + static const char *pwFlags[] = { + "--monitor-password", + "--replication-password", + "--autoctl-node-password", + NULL + }; + for (int i = 0; i < nargs; i++) + { + if (i > 0) + { + appendPQExpBufferChar(cmd, ' '); + } + + /* Check if the previous arg was a password flag. */ + bool maskThis = false; + if (i > 0) + { + for (int k = 0; pwFlags[k]; k++) + { + if (strcmp(args[i - 1], pwFlags[k]) == 0) + { + maskThis = true; + break; + } + } + } + appendPQExpBufferStr(cmd, maskThis ? "****" : args[i]); + } + log_info("pg_autoctl node run: %s", cmd->data); + destroyPQExpBuffer(cmd); + } + + execv(args[0], args); + + /* If we get here execv failed */ + log_fatal("execv(\"%s\"): %m", args[0]); + exit(EXIT_CODE_INTERNAL_ERROR); +} + + +/* ----------------------------------------------------------------------- + * pg_autoctl node apply + * ----------------------------------------------------------------------- */ +static int +cli_node_apply_getopts(int argc, char **argv) +{ + if (argc > 0 && argv[0][0] != '-') + { + strlcpy(nodeSpecPath, argv[0], sizeof(nodeSpecPath)); + } + else + { + log_error("pg_autoctl node apply requires a file argument"); + exit(EXIT_CODE_BAD_ARGS); + } + return 0; +} + + +static void +cli_node_apply(int argc, char **argv) +{ + NodeSpec new_spec = { 0 }; + NodeSpec cur_spec = { 0 }; + + if (!nodespec_read(nodeSpecPath, &new_spec)) + { + exit(EXIT_CODE_BAD_CONFIG); + } + + /* Read the current spec from the same path as a baseline */ + (void) nodespec_read(nodeSpecPath, &cur_spec); + + if (!nodespec_apply(&new_spec, &cur_spec)) + { + log_error("Failed to apply node spec from \"%s\"", nodeSpecPath); + exit(EXIT_CODE_INTERNAL_ERROR); + } +} + + +/* ----------------------------------------------------------------------- + * pg_autoctl node start [] + * + * Clears launch = deferred in the spec file so a waiting pg_autoctl node run + * proceeds. Idempotent: if the node is already immediate, exits 0 quietly. + * ----------------------------------------------------------------------- */ +static int +cli_node_start_getopts(int argc, char **argv) +{ + if (argc > 1 && argv[1][0] != '-') + { + strlcpy(nodeSpecPath, argv[1], sizeof(nodeSpecPath)); + } + else + { + strlcpy(nodeSpecPath, PG_AUTOCTL_NODESPEC_PATH, sizeof(nodeSpecPath)); + } + + return 0; +} + + +static void +cli_node_start(int argc, char **argv) +{ + NodeSpec spec = { 0 }; + + if (!nodespec_read(nodeSpecPath, &spec)) + { + exit(EXIT_CODE_BAD_CONFIG); + } + + if (!spec.launchDeferred) + { + log_info("Node \"%s\" launch is already immediate; nothing to do", + nodeSpecPath); + exit(0); + } + + spec.launchDeferred = false; + + if (!nodespec_write_to_path(&spec, nodeSpecPath)) + { + log_error("Failed to update \"%s\"", nodeSpecPath); + exit(EXIT_CODE_INTERNAL_ERROR); + } + + log_info("Cleared launch = deferred in \"%s\"; node will now start", + nodeSpecPath); +} + + +/* ----------------------------------------------------------------------- + * pg_autoctl node show [--pgdata ] + * ----------------------------------------------------------------------- */ +static int +cli_node_show_getopts(int argc, char **argv) +{ + /* honour --pgdata from the global keeperOptions */ + return cli_getopt_pgdata(argc, argv); +} + + +static void +cli_node_show(int argc, char **argv) +{ + /* + * Build a NodeSpec from the running keeper/monitor config and emit it. + * We read the existing pg_autoctl.cfg rather than the node spec file so + * that `pg_autoctl node show` reflects the live configuration. + */ + KeeperConfig config = keeperOptions; + + bool missingPgdataIsOk = true; + bool pgIsNotRunningIsOk = true; + bool monitorDisabledIsOk = true; + + if (!keeper_config_read_file(&config, + missingPgdataIsOk, + pgIsNotRunningIsOk, + monitorDisabledIsOk)) + { + log_error("Failed to read pg_autoctl configuration"); + exit(EXIT_CODE_BAD_CONFIG); + } + + NodeSpec spec = { 0 }; + + spec.kind = config.pgSetup.pgKind; + strlcpy(spec.pgdata, config.pgSetup.pgdata, sizeof(spec.pgdata)); + strlcpy(spec.hostname, config.hostname, sizeof(spec.hostname)); + spec.port = config.pgSetup.pgport; + strlcpy(spec.monitor_pguri, config.monitor_pguri, + sizeof(spec.monitor_pguri)); + strlcpy(spec.formation, config.formation, sizeof(spec.formation)); + spec.group = config.groupId; + + /* Mutable settings defaults — we'd need a monitor round-trip for live values */ + spec.candidate_priority = 50; + spec.replication_quorum = true; + + /* Create-time defaults */ + strlcpy(spec.ssl, "self-signed", sizeof(spec.ssl)); + strlcpy(spec.auth, "trust", sizeof(spec.auth)); + spec.pg_hba_lan = true; + + (void) nodespec_write(&spec, stdout); +} + + +/* ----------------------------------------------------------------------- + * pg_autoctl node check + * ----------------------------------------------------------------------- */ +static int +cli_node_check_getopts(int argc, char **argv) +{ + if (argc > 0 && argv[0][0] != '-') + { + strlcpy(nodeSpecPath, argv[0], sizeof(nodeSpecPath)); + } + else + { + log_error("pg_autoctl node check requires a file argument"); + exit(EXIT_CODE_BAD_ARGS); + } + return 0; +} + + +static void +cli_node_check(int argc, char **argv) +{ + NodeSpec spec = { 0 }; + + if (!nodespec_read(nodeSpecPath, &spec)) + { + log_error("Invalid node spec file \"%s\"", nodeSpecPath); + exit(EXIT_CODE_BAD_CONFIG); + } + + const char *kindStr; + switch (spec.kind) + { + case NODE_KIND_UNKNOWN: + { + kindStr = "monitor"; + break; + } + + case NODE_KIND_STANDALONE: + { + kindStr = "postgres"; + break; + } + + case NODE_KIND_CITUS_COORDINATOR: + { + kindStr = "coordinator"; + break; + } + + case NODE_KIND_CITUS_WORKER: + { + kindStr = "worker"; + break; + } + + default: + { + kindStr = "unknown"; + break; + } + } + + fformat(stdout, "Node spec \"%s\" is valid.\n", nodeSpecPath); + fformat(stdout, " kind : %s\n", kindStr); + fformat(stdout, " pgdata : %s\n", spec.pgdata); + fformat(stdout, " hostname : %s\n", spec.hostname); + fformat(stdout, " port : %d\n", spec.port); + + if (spec.kind != NODE_KIND_UNKNOWN) + { + fformat(stdout, " monitor_pguri : %s\n", spec.monitor_pguri); + fformat(stdout, " formation : %s\n", spec.formation); + fformat(stdout, " group : %d\n", spec.group); + } + + fformat(stdout, " candidate_priority : %d\n", spec.candidate_priority); + fformat(stdout, " replication_quorum : %s\n", + spec.replication_quorum ? "true" : "false"); + fformat(stdout, " ssl : %s\n", spec.ssl); + fformat(stdout, " auth : %s\n", spec.auth); + fformat(stdout, " pg_hba_lan : %s\n", + spec.pg_hba_lan ? "true" : "false"); +} diff --git a/src/bin/pg_autoctl/cli_node.h b/src/bin/pg_autoctl/cli_node.h new file mode 100644 index 000000000..c236eeada --- /dev/null +++ b/src/bin/pg_autoctl/cli_node.h @@ -0,0 +1,16 @@ +/* + * src/bin/pg_autoctl/cli_node.h + * pg_autoctl node — declarative node lifecycle from a .ini file. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + */ + +#ifndef CLI_NODE_H +#define CLI_NODE_H + +#include "commandline.h" + +extern CommandLine node_commands; + +#endif /* CLI_NODE_H */ diff --git a/src/bin/pg_autoctl/cli_root.c b/src/bin/pg_autoctl/cli_root.c index dd5029687..3c9221109 100644 --- a/src/bin/pg_autoctl/cli_root.c +++ b/src/bin/pg_autoctl/cli_root.c @@ -12,6 +12,7 @@ #include "cli_do_root.h" #include "cli_inspect.h" #include "cli_manual.h" +#include "cli_node.h" #include "cli_root.h" #include "commandline.h" @@ -102,6 +103,7 @@ CommandLine *root_subcommands[] = { &inspect_commands, &manual_commands, &internal_commands, + &node_commands, &do_commands, &service_run_command, diff --git a/src/bin/pg_autoctl/nodespec.c b/src/bin/pg_autoctl/nodespec.c new file mode 100644 index 000000000..5f61e6d91 --- /dev/null +++ b/src/bin/pg_autoctl/nodespec.c @@ -0,0 +1,1063 @@ +/* + * src/bin/pg_autoctl/nodespec.c + * Parse, write, and converge a pg_autoctl_node.ini file. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + */ + +#include +#include +#include +#include + +#ifdef __linux__ +#include +#endif + +#include "nodespec.h" +#include "ini_file.h" +#include "ini.h" +#include "log.h" +#include "pgsetup.h" +#include "string_utils.h" +#include "file_utils.h" /* sformat */ +#include "env_utils.h" +#include "cli_root.h" /* pg_autoctl_program */ +#include "runprogram.h" /* Program, run_program, free_program */ + + +/* + * nodespec_read parses a pg_autoctl_node.ini file. + * + * File format: + * + * [node] + * kind = postgres # postgres | monitor | coordinator | worker + * hostname = node1 + * port = 5432 + * + * [postgresql] + * pgdata = /var/lib/postgres/pgaf + * + * [monitor] + * pguri = postgresql://autoctl_node@monitor/pg_auto_failover + * + * [formation] + * name = default + * group = 0 + * + * [settings] + * candidate_priority = 50 + * replication_quorum = true + * + * [options] + * ssl = self-signed + * auth = trust + * pg_hba_lan = true + */ +bool +nodespec_read(const char *path, NodeSpec *spec) +{ + char kindStr[NAMEDATALEN] = { 0 }; + char replicationQuorumStr[8] = { 0 }; + char pgHbaLanStr[8] = { 0 }; + char launchModeStr[16] = { 0 }; + char noMonitorStr[8] = { 0 }; + char citusRoleStr[NAMEDATALEN] = { 0 }; + int port = 5432; + int group = 0; + int candidatePriority = 50; + + /* + * Provide string buffers for booleans and the kind enum — we parse them + * after read_ini_file() returns so we can give decent error messages. + */ + IniOption opts[] = { + /* [node] */ + make_strbuf_option_default("node", "kind", NULL, true, + sizeof(kindStr), kindStr, + "postgres"), + make_strbuf_option_default("node", "name", NULL, false, + sizeof(spec->name), spec->name, + ""), + make_strbuf_option_default("node", "hostname", NULL, false, + sizeof(spec->hostname), spec->hostname, + ""), + make_int_option_default("node", "port", NULL, false, + &port, 5432), + + /* [postgresql] */ + make_strbuf_option("postgresql", "pgdata", NULL, true, + sizeof(spec->pgdata), spec->pgdata), + + /* [monitor] */ + make_strbuf_option_default("monitor", "pguri", NULL, false, + sizeof(spec->monitor_pguri), + spec->monitor_pguri, ""), + make_strbuf_option_default("monitor", "no_monitor", NULL, false, + sizeof(noMonitorStr), noMonitorStr, "false"), + make_int_option_default("monitor", "node_id", NULL, false, + &spec->nodeId, 0), + + /* [formation] */ + make_strbuf_option_default("formation", "name", NULL, false, + sizeof(spec->formation), spec->formation, + "default"), + make_int_option_default("formation", "group", NULL, false, + &group, 0), + + /* [settings] — mutable */ + make_int_option_default("settings", "candidate_priority", NULL, false, + &candidatePriority, 50), + make_strbuf_option_default("settings", "replication_quorum", NULL, false, + sizeof(replicationQuorumStr), + replicationQuorumStr, "true"), + + /* [options] — ssl is mutable (applied via `pg_autoctl enable ssl`); + * auth and pg_hba_lan are create-time only */ + make_strbuf_option_default("options", "ssl", NULL, false, + sizeof(spec->ssl), spec->ssl, + "self-signed"), + make_strbuf_option_default("options", "auth", NULL, false, + sizeof(spec->auth), spec->auth, + "trust"), + make_strbuf_option_default("options", "pg_hba_lan", NULL, false, + sizeof(pgHbaLanStr), pgHbaLanStr, + "true"), + + /* [ssl] — certificate paths for verify-ca / verify-full mode */ + make_strbuf_option_default("ssl", "ca_file", NULL, false, + sizeof(spec->ssl_ca_file), + spec->ssl_ca_file, ""), + make_strbuf_option_default("ssl", "cert_file", NULL, false, + sizeof(spec->ssl_cert_file), + spec->ssl_cert_file, ""), + make_strbuf_option_default("ssl", "key_file", NULL, false, + sizeof(spec->ssl_key_file), + spec->ssl_key_file, ""), + + /* [launch] — optional section; mode=deferred delays node init */ + make_strbuf_option_default("launch", "mode", NULL, false, + sizeof(launchModeStr), launchModeStr, + "immediate"), + + /* [pg_auto_failover] — monitor: password for autoctl_node role */ + make_strbuf_option_default("pg_auto_failover", "autoctl_node_password", + NULL, false, + sizeof(spec->autoctl_node_password), + spec->autoctl_node_password, ""), + + /* [replication] — postgres: password for pgautofailover_replicator */ + make_strbuf_option_default("replication", "password", NULL, false, + sizeof(spec->replication_password), + spec->replication_password, ""), + + /* [pg_auto_failover] — postgres: password for pgautofailover_monitor */ + make_strbuf_option_default("pg_auto_failover", "monitor_password", + NULL, false, + sizeof(spec->monitor_password), + spec->monitor_password, ""), + + /* [citus] — optional; present only for Citus secondary/read-replica nodes */ + make_strbuf_option_default("citus", "role", NULL, false, + sizeof(citusRoleStr), citusRoleStr, ""), + make_strbuf_option_default("citus", "cluster_name", NULL, false, + sizeof(spec->citusClusterName), + spec->citusClusterName, ""), + + INI_OPTION_LAST + }; + + if (!read_ini_file(path, opts)) + { + log_error("Failed to parse node spec file \"%s\"", path); + return false; + } + + /* resolve kind string → enum */ + if (strcmp(kindStr, "monitor") == 0) + { + spec->kind = NODE_KIND_UNKNOWN; /* handled specially: no formation */ + } + else if (strcmp(kindStr, "postgres") == 0) + { + spec->kind = NODE_KIND_STANDALONE; + } + else if (strcmp(kindStr, "coordinator") == 0) + { + spec->kind = NODE_KIND_CITUS_COORDINATOR; + } + else if (strcmp(kindStr, "worker") == 0) + { + spec->kind = NODE_KIND_CITUS_WORKER; + } + else + { + log_error("Unknown node kind \"%s\" in \"%s\"; " + "expected: monitor, postgres, coordinator, worker", + kindStr, path); + return false; + } + + spec->port = port; + spec->group = group; + spec->candidate_priority = candidatePriority; + + /* parse boolean strings */ + spec->replication_quorum = + (strcmp(replicationQuorumStr, "true") == 0 || + strcmp(replicationQuorumStr, "yes") == 0 || + strcmp(replicationQuorumStr, "1") == 0); + + spec->pg_hba_lan = + (strcmp(pgHbaLanStr, "true") == 0 || + strcmp(pgHbaLanStr, "yes") == 0 || + strcmp(pgHbaLanStr, "1") == 0); + + spec->launchDeferred = (strcmp(launchModeStr, "deferred") == 0); + spec->noMonitor = + (strcmp(noMonitorStr, "true") == 0 || + strcmp(noMonitorStr, "yes") == 0 || + strcmp(noMonitorStr, "1") == 0); + + spec->citusSecondary = (strcmp(citusRoleStr, "secondary") == 0); + + /* + * Second pass: enumerate [formation ] sections. + * The standard IniOption machinery can't handle variable-count sections, + * so we open the file again with ini_load directly. + */ + { + char *fileContents = NULL; + long fileSize = 0L; + + if (read_file(path, &fileContents, &fileSize)) + { + ini_t *raw = ini_load(fileContents, NULL); + free(fileContents); + + if (raw) + { + int nsec = ini_section_count(raw); + spec->formationCount = 0; + + for (int si = 0; si < nsec; si++) + { + const char *sname = ini_section_name(raw, si); + if (!sname) + { + continue; + } + if (strncmp(sname, "formation ", 10) != 0) + { + continue; + } + + const char *fname = sname + 10; /* skip "formation " */ + if (fname[0] == '\0') + { + continue; + } + + if (spec->formationCount >= NODESPEC_MAX_FORMATIONS) + { + log_warn("nodespec: too many [formation ] sections " + "in \"%s\"; max %d", path, NODESPEC_MAX_FORMATIONS); + break; + } + + int fi = spec->formationCount++; + strlcpy(spec->formationNames[fi], fname, + sizeof(spec->formationNames[fi])); + + /* optional: kind = ha (default) */ + int ki = ini_find_property(raw, si, "kind", 0); + if (ki != INI_NOT_FOUND) + { + const char *kv = ini_property_value(raw, si, ki); + if (kv && kv[0]) + { + strlcpy(spec->formationKinds[fi], kv, + sizeof(spec->formationKinds[fi])); + } + else + { + strlcpy(spec->formationKinds[fi], "pgsql", + sizeof(spec->formationKinds[fi])); + } + } + else + { + strlcpy(spec->formationKinds[fi], "pgsql", + sizeof(spec->formationKinds[fi])); + } + } + ini_destroy(raw); + } + } + } + + /* validate: non-monitor nodes need a monitor URI unless no_monitor=true */ + if (spec->kind != NODE_KIND_UNKNOWN && + IS_EMPTY_STRING_BUFFER(spec->monitor_pguri) && + !spec->noMonitor) + { + log_error("Node kind \"%s\" requires [monitor] pguri in \"%s\"", + kindStr, path); + return false; + } + + return true; +} + + +/* + * nodespec_write serialises a NodeSpec as a pg_autoctl_node.ini file. + */ +bool +nodespec_write(const NodeSpec *spec, FILE *out) +{ + const char *kindStr; + + switch (spec->kind) + { + case NODE_KIND_UNKNOWN: + { + kindStr = "monitor"; + break; + } + + case NODE_KIND_STANDALONE: + { + kindStr = "postgres"; + break; + } + + case NODE_KIND_CITUS_COORDINATOR: + { + kindStr = "coordinator"; + break; + } + + case NODE_KIND_CITUS_WORKER: + { + kindStr = "worker"; + break; + } + + default: + { + kindStr = "postgres"; + break; + } + } + + fformat(out, + "[node]\n" + "kind = %s\n", + kindStr); + + if (!IS_EMPTY_STRING_BUFFER(spec->name)) + { + fformat(out, "name = %s\n", spec->name); + } + + fformat(out, + "hostname = %s\n" + "port = %d\n" + "\n" + "[postgresql]\n" + "pgdata = %s\n" + "\n", + spec->hostname, + spec->port, + spec->pgdata); + + if (spec->kind != NODE_KIND_UNKNOWN) + { + if (spec->noMonitor) + { + fformat(out, + "[monitor]\n" + "no_monitor = true\n" + "\n"); + } + else + { + fformat(out, + "[monitor]\n" + "pguri = %s\n" + "\n", + spec->monitor_pguri); + } + + fformat(out, + "[formation]\n" + "name = %s\n" + "group = %d\n" + "\n", + spec->formation, + spec->group); + } + + fformat(out, + "[settings]\n" + "candidate_priority = %d\n" + "replication_quorum = %s\n" + "\n" + "[options]\n" + "ssl = %s\n" + "auth = %s\n" + "pg_hba_lan = %s\n", + spec->candidate_priority, + spec->replication_quorum ? "true" : "false", + spec->ssl, + spec->auth, + spec->pg_hba_lan ? "true" : "false"); + + /* only emit [launch] when deferred — omitting the section means immediate */ + if (spec->launchDeferred) + { + fformat(out, "\n[launch]\nmode = deferred\n"); + } + + /* [formation ] sections — monitor kind only */ + for (int fi = 0; fi < spec->formationCount; fi++) + { + fformat(out, "\n[formation %s]\n", spec->formationNames[fi]); + if (spec->formationKinds[fi][0] && + strcmp(spec->formationKinds[fi], "pgsql") != 0) + { + fformat(out, "kind = %s\n", spec->formationKinds[fi]); + } + } + + return true; +} + + +/* + * nodespec_create_argv builds an argv[] array for: + * + * pg_autoctl create --pgdata

--hostname --pgport + * --monitor [--formation ] [--group ] + * [--ssl-self-signed | --no-ssl] + * [--auth ] [--pg-hba-lan] --run + * + * Returns the number of entries written (terminating NULL not counted). + * args[] must have room for at least 40 pointers. + * + * String values are pointers into *spec — the caller must keep spec alive. + */ +int +nodespec_create_argv(const NodeSpec *spec, + const char *pg_autoctl_path, + char **args, int args_size) +{ + int i = 0; + +#define PUSH(v) do { \ + if (i >= args_size - 1) { \ + log_error("nodespec_create_argv: args[] overflow"); \ + return -1; \ + } \ + args[i++] = (char *) (v); \ +} while (0) + + PUSH(pg_autoctl_path); + PUSH("create"); + + switch (spec->kind) + { + case NODE_KIND_UNKNOWN: + { + PUSH("monitor"); + break; + } + + case NODE_KIND_STANDALONE: + { + PUSH("postgres"); + break; + } + + case NODE_KIND_CITUS_COORDINATOR: + { + PUSH("coordinator"); + break; + } + + case NODE_KIND_CITUS_WORKER: + { + PUSH("worker"); + break; + } + + default: + { + PUSH("postgres"); + break; + } + } + + PUSH("--pgdata"); + PUSH(spec->pgdata); + + if (!IS_EMPTY_STRING_BUFFER(spec->name)) + { + PUSH("--name"); + PUSH(spec->name); + } + + if (!IS_EMPTY_STRING_BUFFER(spec->hostname)) + { + PUSH("--hostname"); + PUSH(spec->hostname); + } + + if (spec->port != 5432) + { + /* static buffer — safe because spec is long-lived */ + static char portbuf[16]; + sformat(portbuf, sizeof(portbuf), "%d", spec->port); + PUSH("--pgport"); + PUSH(portbuf); + } + + if (spec->kind != NODE_KIND_UNKNOWN) + { + if (spec->noMonitor) + { + PUSH("--disable-monitor"); + if (spec->nodeId > 0) + { + static char nodeIdBuf[16]; + sformat(nodeIdBuf, sizeof(nodeIdBuf), "%d", spec->nodeId); + PUSH("--node-id"); + PUSH(nodeIdBuf); + } + } + else + { + PUSH("--monitor"); + PUSH(spec->monitor_pguri); + } + + if (!IS_EMPTY_STRING_BUFFER(spec->formation) && + strcmp(spec->formation, "default") != 0) + { + PUSH("--formation"); + PUSH(spec->formation); + } + + if (spec->kind == NODE_KIND_CITUS_WORKER && spec->group > 0) + { + static char groupbuf[16]; + sformat(groupbuf, sizeof(groupbuf), "%d", spec->group); + PUSH("--group"); + PUSH(groupbuf); + } + } + + /* SSL */ + if (strcmp(spec->ssl, "self-signed") == 0) + { + PUSH("--ssl-self-signed"); + } + else if (strcmp(spec->ssl, "off") == 0) + { + PUSH("--no-ssl"); + } + else if (!IS_EMPTY_STRING_BUFFER(spec->ssl_ca_file)) + { + /* verify-ca / verify-full: pass the cert paths explicitly */ + PUSH("--ssl-ca-file"); + PUSH(spec->ssl_ca_file); + PUSH("--server-cert"); + PUSH(spec->ssl_cert_file); + PUSH("--server-key"); + PUSH(spec->ssl_key_file); + PUSH("--ssl-mode"); + PUSH(spec->ssl); + } + + /* auth */ + if (!IS_EMPTY_STRING_BUFFER(spec->auth)) + { + PUSH("--auth"); + PUSH(spec->auth); + } + + if (spec->pg_hba_lan && spec->kind != NODE_KIND_UNKNOWN) + { + PUSH("--pg-hba-lan"); + } + + /* passwords */ + if (spec->kind == NODE_KIND_UNKNOWN && + !IS_EMPTY_STRING_BUFFER(spec->autoctl_node_password)) + { + PUSH("--autoctl-node-password"); + PUSH(spec->autoctl_node_password); + } + + /* non-default formations to create during monitor init */ + if (spec->kind == NODE_KIND_UNKNOWN) + { + for (int fi = 0; fi < spec->formationCount; fi++) + { + PUSH("--formation"); + PUSH(spec->formationNames[fi]); + } + } + + if (spec->kind != NODE_KIND_UNKNOWN) + { + if (!IS_EMPTY_STRING_BUFFER(spec->monitor_password)) + { + PUSH("--monitor-password"); + PUSH(spec->monitor_password); + } + if (!IS_EMPTY_STRING_BUFFER(spec->replication_password)) + { + PUSH("--replication-password"); + PUSH(spec->replication_password); + } + + /* candidate priority and replication quorum (non-default values) */ + if (spec->candidate_priority != 50) + { + static char pribuf[16]; + sformat(pribuf, sizeof(pribuf), "%d", spec->candidate_priority); + PUSH("--candidate-priority"); + PUSH(pribuf); + } + if (!spec->replication_quorum) + { + PUSH("--replication-quorum"); + PUSH("false"); + } + + /* Citus secondary/read-replica cluster settings */ + if (spec->citusSecondary) + { + PUSH("--citus-secondary"); + } + if (!IS_EMPTY_STRING_BUFFER(spec->citusClusterName)) + { + PUSH("--citus-cluster"); + PUSH(spec->citusClusterName); + } + } + + PUSH("--run"); + + args[i] = NULL; + +#undef PUSH + return i; +} + + +/* + * nodespec_write_to_path writes the spec to the given filesystem path, + * replacing the file in-place. Used by pg_autoctl node start to clear the + * [launch] deferred flag so the waiting pg_autoctl node run can proceed. + */ +bool +nodespec_write_to_path(const NodeSpec *spec, const char *path) +{ + FILE *f = fopen(path, "w"); /* IGNORE-BANNED */ + if (!f) + { + log_error("Cannot open \"%s\" for writing: %m", path); + return false; + } + bool ok = nodespec_write(spec, f); + fclose(f); + return ok; +} + + +/* + * nodespec_apply compares new_spec against old_spec and applies any changes + * to the mutable fields by calling into the keeper / monitor APIs. + * + * Mutable fields: + * - candidate_priority → pg_autoctl set node candidate-priority + * - replication_quorum → pg_autoctl set node replication-quorum + * - ssl / ssl_*_file → pg_autoctl enable ssl + * - monitor_pguri → pg_autoctl disable monitor --force + * pg_autoctl enable monitor + * + * Immutable fields (kind, pgdata, hostname, port, auth, + * pg_hba_lan) require a node restart to take effect. + * + * The [launch] mode field is handled separately by pg_autoctl node start. + * Applying a spec with mode=deferred to an already-started node is a + * non-fatal warning (ignored). + */ +bool +nodespec_apply(const NodeSpec *new_spec, const NodeSpec *old_spec) +{ + bool changed = false; + + if (new_spec->candidate_priority != old_spec->candidate_priority) + { + char prio[16]; + sformat(prio, sizeof(prio), "%d", new_spec->candidate_priority); + + Program prog = run_program(pg_autoctl_program, + "set", "node", "candidate-priority", + "--pgdata", new_spec->pgdata, + prio, NULL); + + if (prog.returnCode != 0) + { + log_warn("nodespec_apply: set candidate-priority %s failed (rc=%d)", + prio, prog.returnCode); + if (prog.stdOut) + { + log_warn("%s", prog.stdOut); + } + } + else + { + log_info("nodespec: applied candidate_priority = %d", + new_spec->candidate_priority); + changed = true; + } + free_program(&prog); + } + + if (new_spec->replication_quorum != old_spec->replication_quorum) + { + const char *quorum = new_spec->replication_quorum ? "true" : "false"; + + Program prog = run_program(pg_autoctl_program, + "set", "node", "replication-quorum", + "--pgdata", new_spec->pgdata, + quorum, NULL); + + if (prog.returnCode != 0) + { + log_warn("nodespec_apply: set replication-quorum %s failed (rc=%d)", + quorum, prog.returnCode); + if (prog.stdOut) + { + log_warn("%s", prog.stdOut); + } + } + else + { + log_info("nodespec: applied replication_quorum = %s", quorum); + changed = true; + } + free_program(&prog); + } + + /* + * Monitor URI changed: disable the current monitor (removing this node + * from it) then re-register to the new one. The --force flag allows + * disable to proceed even if the old monitor is unreachable. + */ + if (strcmp(new_spec->monitor_pguri, old_spec->monitor_pguri) != 0 && + !IS_EMPTY_STRING_BUFFER(new_spec->monitor_pguri)) + { + Program disable_prog = run_program(pg_autoctl_program, + "disable", "monitor", + "--force", + "--pgdata", new_spec->pgdata, + NULL); + + if (disable_prog.returnCode != 0) + { + log_warn("nodespec_apply: disable monitor failed (rc=%d)", + disable_prog.returnCode); + if (disable_prog.stdOut) + { + log_warn("%s", disable_prog.stdOut); + } + free_program(&disable_prog); + } + else + { + free_program(&disable_prog); + + Program enable_prog = run_program(pg_autoctl_program, + "enable", "monitor", + "--pgdata", new_spec->pgdata, + new_spec->monitor_pguri, + NULL); + + if (enable_prog.returnCode != 0) + { + log_warn("nodespec_apply: enable monitor \"%s\" failed (rc=%d)", + new_spec->monitor_pguri, enable_prog.returnCode); + if (enable_prog.stdOut) + { + log_warn("%s", enable_prog.stdOut); + } + } + else + { + log_info("nodespec: applied monitor_pguri = %s", + new_spec->monitor_pguri); + changed = true; + } + free_program(&enable_prog); + } + } + + /* + * SSL mode or certificate paths changed: call `pg_autoctl enable ssl` + * with the appropriate flags so Postgres is reconfigured live. + * + * ssl=off maps to --no-ssl; ssl=self-signed maps to --ssl-self-signed; + * anything else is a CA-verified mode and requires the cert paths. + */ + { + bool ssl_changed = + (strcmp(new_spec->ssl, old_spec->ssl) != 0) || + (strcmp(new_spec->ssl_ca_file, old_spec->ssl_ca_file) != 0) || + (strcmp(new_spec->ssl_cert_file, old_spec->ssl_cert_file) != 0) || + (strcmp(new_spec->ssl_key_file, old_spec->ssl_key_file) != 0); + + if (ssl_changed) + { + Program prog; + + if (strcmp(new_spec->ssl, "off") == 0) + { + prog = run_program(pg_autoctl_program, + "enable", "ssl", + "--pgdata", new_spec->pgdata, + "--no-ssl", NULL); + } + else if (strcmp(new_spec->ssl, "self-signed") == 0 || + IS_EMPTY_STRING_BUFFER(new_spec->ssl_ca_file)) + { + prog = run_program(pg_autoctl_program, + "enable", "ssl", + "--pgdata", new_spec->pgdata, + "--ssl-self-signed", NULL); + } + else + { + prog = run_program(pg_autoctl_program, + "enable", "ssl", + "--pgdata", new_spec->pgdata, + "--ssl-mode", new_spec->ssl, + "--ssl-ca-file", new_spec->ssl_ca_file, + "--server-cert", new_spec->ssl_cert_file, + "--server-key", new_spec->ssl_key_file, + NULL); + } + + if (prog.returnCode != 0) + { + log_warn("nodespec_apply: enable ssl failed (rc=%d)", + prog.returnCode); + if (prog.stdOut) + { + log_warn("%s", prog.stdOut); + } + } + else + { + log_info("nodespec: applied ssl = %s", new_spec->ssl); + changed = true; + } + free_program(&prog); + } + } + + /* immediate → deferred on an already-started node: non-fatal, ignored */ + if (!old_spec->launchDeferred && new_spec->launchDeferred) + { + log_warn("nodespec_apply: ignoring attempt to set launch=deferred " + "on a node that is already running"); + } + + if (!changed) + { + log_debug("nodespec_apply: no mutable fields changed"); + } + + return true; +} + + +/* ----------------------------------------------------------------------- + * Watcher + * ----------------------------------------------------------------------- */ + +/* + * nodespec_watcher_init sets up file watching for *path. + * On Linux we try inotify first; everywhere else (and on failure) we fall + * back to mtime polling every NODESPEC_WATCH_INTERVAL_SECS. + */ +bool +nodespec_watcher_init(NodeSpecWatcher *w, const char *path) +{ + struct stat st; + + memset(w, 0, sizeof(*w)); + strlcpy(w->path, path, sizeof(w->path)); + + w->last_checked = time(NULL); + +#ifdef __linux__ + w->inotify_fd = inotify_init1(IN_NONBLOCK | IN_CLOEXEC); + + if (w->inotify_fd >= 0) + { + w->watch_fd = inotify_add_watch(w->inotify_fd, path, + IN_CLOSE_WRITE | IN_MOVED_TO); + if (w->watch_fd < 0) + { + log_debug("nodespec_watcher_init: inotify_add_watch(\"%s\"): %m; " + "falling back to mtime poll", path); + close(w->inotify_fd); + w->inotify_fd = -1; + } + else + { + log_info("nodespec: watching \"%s\" via inotify", path); + } + } + else + { + log_debug("nodespec_watcher_init: inotify_init1: %m; " + "falling back to mtime poll"); + w->inotify_fd = -1; + w->watch_fd = -1; + } +#endif + + /* record initial mtime so we don't fire on the very first check */ + if (stat(path, &st) == 0) + { + w->last_mtime = st.st_mtime; + } + else + { + w->last_mtime = 0; + } + + w->active = true; + return true; +} + + +/* + * nodespec_watcher_check is called from the supervisor's 100 ms tick. + * + * Returns true if the file changed and nodespec_apply() was attempted. + */ +bool +nodespec_watcher_check(NodeSpecWatcher *w, const NodeSpec *current) +{ + bool file_changed = false; + + if (!w->active) + { + return false; + } + +#ifdef __linux__ + if (w->inotify_fd >= 0) + { + /* + * Drain all pending inotify events. We don't care about the event + * details — any event means the file was written. + */ + char buf[sizeof(struct inotify_event) + NAME_MAX + 1]; + ssize_t n; + + while ((n = read(w->inotify_fd, buf, sizeof(buf))) > 0) + { + file_changed = true; + } + + if (n < 0 && errno != EAGAIN && errno != EWOULDBLOCK) + { + log_warn("nodespec watcher: inotify read error: %m; " + "switching to mtime poll"); + close(w->inotify_fd); + w->inotify_fd = -1; + w->watch_fd = -1; + } + } + else +#endif + { + /* mtime poll — only stat every NODESPEC_WATCH_INTERVAL_SECS */ + time_t now = time(NULL); + + if (now - w->last_checked < NODESPEC_WATCH_INTERVAL_SECS) + { + return false; + } + + w->last_checked = now; + + struct stat st; + if (stat(w->path, &st) == 0 && st.st_mtime != w->last_mtime) + { + w->last_mtime = st.st_mtime; + file_changed = true; + } + } + + if (!file_changed) + { + return false; + } + + /* File changed — re-parse and apply */ + log_info("nodespec: \"%s\" changed, re-reading and converging", w->path); + + NodeSpec new_spec = { 0 }; + if (!nodespec_read(w->path, &new_spec)) + { + log_warn("nodespec: failed to parse updated file \"%s\"; " + "keeping current configuration", w->path); + return false; + } + + /* Warn about immutable field changes rather than silently ignoring them */ + if (new_spec.kind != current->kind) + { + log_warn("nodespec: 'kind' changed in \"%s\" but cannot be applied " + "to a running node — restart required", w->path); + } + + if (strcmp(new_spec.pgdata, current->pgdata) != 0) + { + log_warn("nodespec: 'pgdata' changed in \"%s\" but cannot be applied " + "to a running node — restart required", w->path); + } + + (void) nodespec_apply(&new_spec, current); + return true; +} + + +/* + * nodespec_watcher_close releases inotify resources. + */ +void +nodespec_watcher_close(NodeSpecWatcher *w) +{ +#ifdef __linux__ + if (w->inotify_fd >= 0) + { + close(w->inotify_fd); + w->inotify_fd = -1; + w->watch_fd = -1; + } +#endif + w->active = false; +} diff --git a/src/bin/pg_autoctl/nodespec.h b/src/bin/pg_autoctl/nodespec.h new file mode 100644 index 000000000..128ef15fb --- /dev/null +++ b/src/bin/pg_autoctl/nodespec.h @@ -0,0 +1,162 @@ +/* + * src/bin/pg_autoctl/nodespec.h + * Declarative node description file (.ini) — read, write, and converge. + * + * A NodeSpec is the in-memory representation of a pg_autoctl_node.ini file. + * It covers all parameters needed to create and run one pg_auto_failover node. + * + * Lifecycle: + * 1. nodespec_read() — parse the file into a NodeSpec + * 2. nodespec_create() — run `pg_autoctl create ` if PGDATA absent + * 3. nodespec_run() — hand off to the normal run path + * 4. nodespec_apply() — converge mutable fields on an already-running node + * + * The supervisor calls nodespec_check_and_apply() on a timer so that editing + * the file and saving it is sufficient to reconfigure a running node. + * + * Copyright (c) Microsoft Corporation. All rights reserved. + * Licensed under the PostgreSQL License. + */ + +#ifndef NODESPEC_H +#define NODESPEC_H + +#include +#include +#include + +#include "postgres_fe.h" /* MAXPGPATH, NAMEDATALEN, etc. */ +#include "pgsql.h" /* MAXCONNINFO */ +#include "pgsetup.h" /* PgInstanceKind, NODE_KIND_* */ + +/* + * Fixed location inside every container. The compose generator writes one + * ini file per node and bind-mounts it here. Having a fixed path means the + * Docker image needs only a single CMD: + * + * CMD ["pg_autoctl", "node", "run", PG_AUTOCTL_NODESPEC_PATH] + */ +#define PG_AUTOCTL_NODESPEC_PATH "/etc/pgaf/node.ini" + +/* + * How often the supervisor polls the spec file for changes (seconds). + * inotify/kqueue are used when available; this is the fallback interval. + */ +#define NODESPEC_WATCH_INTERVAL_SECS 10 + +/* ----------------------------------------------------------------------- + * NodeSpec — one-to-one with the [sections] of pg_autoctl_node.ini + * ----------------------------------------------------------------------- */ + +typedef struct NodeSpec +{ + /* [node] */ + PgInstanceKind kind; /* postgres | coordinator | worker | monitor */ + char name[_POSIX_HOST_NAME_MAX]; /* --name; defaults to hostname when empty */ + char hostname[_POSIX_HOST_NAME_MAX]; + int port; /* Postgres port, default 5432 */ + + /* [postgresql] */ + char pgdata[MAXPGPATH]; + + /* [monitor] — empty for kind == monitor + * monitor_pguri is mutable: changing it triggers disable monitor --force + * followed by enable monitor (re-registers without restarting). */ + char monitor_pguri[MAXCONNINFO]; + bool noMonitor; /* [monitor] no_monitor=true: standalone mode */ + int nodeId; /* [monitor] node_id: required with --disable-monitor */ + + /* [formation] */ + char formation[NAMEDATALEN]; /* default "default" */ + int group; /* Citus group; 0 = coordinator */ + + /* [settings] — mutable; applied on SIGHUP / file change */ + int candidate_priority; /* 0-100, default 50 */ + bool replication_quorum; /* sync quorum participant, default true */ + + /* [options] — immutable; used only at pg_autoctl create time */ + char ssl[32]; /* self-signed | verify-ca | verify-full | off */ + char auth[32]; /* trust | md5 | scram | cert */ + bool pg_hba_lan; /* add --pg-hba-lan flag */ + + /* [ssl] — certificate paths for verify-ca / verify-full mode */ + char ssl_ca_file[MAXPGPATH]; + char ssl_cert_file[MAXPGPATH]; + char ssl_key_file[MAXPGPATH]; + bool launchDeferred; /* [launch] mode=deferred: wait for node start */ + + /* [formation ] — monitor kind: non-default formations to create */ +#define NODESPEC_MAX_FORMATIONS 16 + int formationCount; + char formationNames[NODESPEC_MAX_FORMATIONS][NAMEDATALEN]; + char formationKinds[NODESPEC_MAX_FORMATIONS][NAMEDATALEN]; /* "pgsql" default */ + + /* [pg_auto_failover] — monitor kind: password for autoctl_node role */ + char autoctl_node_password[MAXCONNINFO]; + + /* [replication] — postgres kind: password for pgautofailover_replicator */ + char replication_password[MAXCONNINFO]; + + /* [pg_auto_failover] — postgres kind: password for pgautofailover_monitor */ + char monitor_password[MAXCONNINFO]; + + /* [citus] — Citus secondary/read-replica cluster settings */ + bool citusSecondary; /* role = secondary */ + char citusClusterName[NAMEDATALEN]; /* cluster_name = */ +} NodeSpec; + +/* ----------------------------------------------------------------------- + * File watcher state — embedded in Supervisor + * ----------------------------------------------------------------------- */ + +typedef struct NodeSpecWatcher +{ + bool active; /* true once nodespec_watcher_init() succeeds */ + char path[MAXPGPATH]; /* path of the watched file */ + time_t last_mtime; /* mtime at last check */ + time_t last_checked; /* wall-clock time of last poll */ + +#ifdef __linux__ + int inotify_fd; /* inotify instance fd, -1 if unavailable */ + int watch_fd; /* inotify watch descriptor */ +#endif +} NodeSpecWatcher; + +/* ----------------------------------------------------------------------- + * API + * ----------------------------------------------------------------------- */ + +/* Parse a pg_autoctl_node.ini file into *spec. Returns false on error. */ +bool nodespec_read(const char *path, NodeSpec *spec); + +/* Write *spec as a pg_autoctl_node.ini file (for pg_autoctl node show). */ +bool nodespec_write(const NodeSpec *spec, FILE *out); + +/* Build the argv[] for `pg_autoctl create [flags]`. + * Caller provides args[] with room for at least 40 char* entries. + * Returns the number of entries filled (not counting the trailing NULL). */ +int nodespec_create_argv(const NodeSpec *spec, + const char *pg_autoctl_path, + char **args, int args_size); + +/* Apply mutable fields (candidate_priority, replication_quorum) to a + * running node by calling into the keeper/monitor APIs directly. + * Logs what changed. Returns false only on hard error. */ +bool nodespec_apply(const NodeSpec *new_spec, const NodeSpec *old_spec); +bool nodespec_write_to_path(const NodeSpec *spec, const char *path); + +/* ----------------------------------------------------------------------- + * Watcher — called from supervisor loop + * ----------------------------------------------------------------------- */ + +/* Initialise watcher state. Sets up inotify on Linux if available. */ +bool nodespec_watcher_init(NodeSpecWatcher *w, const char *path); + +/* Called from the supervisor's 100 ms tick. + * Returns true if the file changed and nodespec_apply() was called. */ +bool nodespec_watcher_check(NodeSpecWatcher *w, const NodeSpec *current); + +/* Release inotify fds (called at supervisor shutdown). */ +void nodespec_watcher_close(NodeSpecWatcher *w); + +#endif /* NODESPEC_H */ diff --git a/src/bin/pg_autoctl/supervisor.c b/src/bin/pg_autoctl/supervisor.c index 9e20853d5..c05c19d04 100644 --- a/src/bin/pg_autoctl/supervisor.c +++ b/src/bin/pg_autoctl/supervisor.c @@ -20,6 +20,7 @@ #include "cli_root.h" #include "defaults.h" +#include "nodespec.h" #include "env_utils.h" #include "fsm.h" #include "keeper.h" @@ -76,6 +77,32 @@ supervisor_start(Service services[], int serviceCount, const char *pidfile) /* copy the pidfile over to our supervisor structure */ strlcpy(supervisor.pidfile, pidfile, MAXPGPATH); + /* + * If we were started by `pg_autoctl node run`, the node spec path is + * passed via the PG_AUTOCTL_NODESPEC env var. Set up the file watcher + * so the supervisor can converge mutable settings when the file changes. + */ + { + char specPath[MAXPGPATH] = { 0 }; + + if (env_exists("PG_AUTOCTL_NODESPEC") && + get_env_copy("PG_AUTOCTL_NODESPEC", specPath, sizeof(specPath)) && + !IS_EMPTY_STRING_BUFFER(specPath)) + { + if (nodespec_read(specPath, &supervisor.watchedSpec) && + nodespec_watcher_init(&supervisor.watcher, specPath)) + { + log_info("Supervisor: watching node spec \"%s\"", specPath); + } + else + { + log_warn("Supervisor: failed to initialise node spec watcher " + "for \"%s\"; changes to the file will be ignored", + specPath); + } + } + } + /* * Create our PID file, or quit now if another pg_autoctl instance is * runnning. @@ -219,6 +246,14 @@ supervisor_loop(Supervisor *supervisor) { /* avoid busy looping on waitpid(WNOHANG) */ pg_usleep(100 * 1000); /* 100 ms */ + + /* + * Check if the node spec file has changed and apply mutable + * settings if so. Uses inotify on Linux, mtime poll elsewhere. + * No-op when watcher.active is false (normal run path). + */ + (void) nodespec_watcher_check(&supervisor->watcher, + &supervisor->watchedSpec); } /* ignore errors */ diff --git a/src/bin/pg_autoctl/supervisor.h b/src/bin/pg_autoctl/supervisor.h index 903b587ae..a043f1a11 100644 --- a/src/bin/pg_autoctl/supervisor.h +++ b/src/bin/pg_autoctl/supervisor.h @@ -12,13 +12,15 @@ #include #include +#include "nodespec.h" + /* * pg_autoctl runs sub-processes as "services", and we need to use the same * service names in several places: * * - the main pidfile, * - the per-service name for the pidfile is derived from this, - * - the pg_autoctl do service getpid|restart commands + * - the pg_autoctl manual service getpid|restart commands */ #define SERVICE_NAME_POSTGRES "postgres" #define SERVICE_NAME_KEEPER "node-active" @@ -122,6 +124,16 @@ typedef struct Supervisor bool shutdownSequenceInProgress; int shutdownSignal; int stoppingLoopCounter; + + /* + * Optional node spec watcher. When pg_autoctl is started via + * `pg_autoctl node run `, the supervisor watches the ini file for + * changes and converges mutable settings automatically. + * + * watcher.active is false when not in use (normal create/run path). + */ + NodeSpecWatcher watcher; + NodeSpec watchedSpec; /* last-applied spec — baseline for diff */ } Supervisor;