feat: pg_autoctl node — declarative node lifecycle from a single ini file (#1131)

* feat: pg_autoctl node — declarative node lifecycle from pg_autoctl_node.ini

Adds a new pg_autoctl node sub-command tree and a NodeSpec file format
designed as the recommended entry-point for container and Kubernetes
deployments.

## pg_autoctl_node.ini sections

  [node]         kind (postgres|monitor|coordinator|worker), name, hostname, port
  [postgresql]   pgdata
  [monitor]      pguri (empty for monitor nodes), no_monitor, node_id
  [formation]    name, group
  [settings]     candidate_priority, replication_quorum   ← mutable, applied live
  [options]      ssl, auth, pg_hba_lan                    ← create-time only
  [ssl]          ssl_ca_file, ssl_cert_file, ssl_key_file
  [launch]       mode=deferred: wait for pg_autoctl node start
  [formation N]  monitor only: additional named formations

## pg_autoctl node sub-commands

  run   <file>   Read ini, create node if absent, exec() into supervisor.
                 Sets PG_AUTOCTL_NODESPEC so the supervisor watches for
                 live [settings] changes via inotify (Linux) or mtime poll.
  apply <file>   Converge mutable settings on an already-running node.
  start [<file>] Clear launch=deferred so a waiting node run proceeds.
  show           Dump live config as pg_autoctl_node.ini on stdout.
  check <file>   Parse-only validate; print resolved fields.

## Supervisor file watcher (nodespec_watcher)

The supervisor initialises a NodeSpecWatcher when PG_AUTOCTL_NODESPEC is
set. Every tick it checks for file changes:
  - Linux:   drain inotify IN_CLOSE_WRITE / IN_MOVED_TO events
  - Others:  stat() every NODESPEC_WATCH_INTERVAL_SECS (10 s)
On change, re-parse [settings] and call nodespec_apply() to converge
mutable fields without restarting the node.

## Files

  src/bin/pg_autoctl/cli_node.c / cli_node.h
  src/bin/pg_autoctl/nodespec.c / nodespec.h
  src/bin/pg_autoctl/supervisor.c / supervisor.h  (watcher integration)
  docs/ref/pg_autoctl_node.rst
  docs/ref/pg_autoctl_node_run.rst

* docs,feat: refine pg_autoctl node docs and make ssl mutable

- pg_autoctl_node.rst: replace prose property list with a structured
  table (section/property/type/mutable/default/description), each
  property on its own row; fix 'disabled-monitor' label → actual ref

- nodespec_apply: add ssl apply block — when [options].ssl or any [ssl]
  cert path changes, call 'pg_autoctl enable ssl' with the appropriate
  flags (--ssl-self-signed, --no-ssl, or --ssl-mode + cert paths);
  update nodespec_apply comment to list ssl as mutable

- docs/index.rst: remove 'Container and Kubernetes' as a top-level
  toctree caption; the manual page is still reachable via Manual Pages

- docs/operations.rst: add 'Container and Kubernetes Deployments'
  section describing pg_autoctl node run, live reconfiguration, and
  the launch=deferred pattern; cross-refs to pg_autoctl_node for details

- docs/ref/configuration.rst: add 'Declarative Node Configuration'
  section covering pg_autoctl_node.ini sections and their relationship
  to pg_autoctl.cfg; cross-ref to pg_autoctl_node

* nodespec: make monitor_pguri mutable via disable/enable monitor

When the [monitor] pguri changes in pg_autoctl_node.ini, nodespec_apply
now re-registers the node to the new monitor without stopping Postgres:

  pg_autoctl disable monitor --force --pgdata <dir>
  pg_autoctl enable monitor <new_uri> --pgdata <dir>

The disable step removes the node from the old monitor (--force allows
this even if the old monitor is temporarily unreachable). The enable step
registers the node to the new monitor and signals the running supervisor
to start using the new monitor_pguri for node_active calls.

Document the change in the property table and Live Reconfiguration section
of pg_autoctl_node.rst.

* docs: complete pg_autoctl node sub-command pages, remove Container section

Follow the pg_autoctl_create pattern: each pg_autoctl node sub-command
now has its own manual page, all listed in pg_autoctl_node.rst's toctree:

  pg_autoctl node run    (existing)
  pg_autoctl node apply  (new)
  pg_autoctl node start  (new)
  pg_autoctl node show   (new)
  pg_autoctl node check  (new)

Remove the redundant 'Container and Kubernetes Deployments' section from
operations.rst. The full documentation for this feature lives in the
pg_autoctl_node manual page (ref/pg_autoctl_node.rst) and its sub-pages.

* docs: replace list-table with definition-list style in pg_autoctl_node

The list-table rendered poorly in the Sphinx HTML theme. Replace it with
the same definition-list style used throughout the other manual pages
(e.g. pg_autoctl_create_postgres Options section): each property is a
bare term followed by indented description paragraphs, grouped under
ini-section sub-headings.

* docs: restore Container/K8s section in operations.rst, note in Provisioning

Re-add the 'Container and Kubernetes Deployments' section at the end of
operations.rst with a named anchor so it can be referenced from elsewhere.

Add a short note in the Provisioning section pointing to pg_autoctl node run
as the declarative alternative for container and Kubernetes deployments,
with a cross-reference down to the new section.

* docs: add pg_autoctl node cross-references throughout the manual

Add references to pg_autoctl node / pg_autoctl node run in all the places
identified by the documentation audit:

Reference pages (new See Also section):
- pg_autoctl_create_postgres.rst
- pg_autoctl_create_monitor.rst
- pg_autoctl_create_coordinator.rst
- pg_autoctl_create_worker.rst
- pg_autoctl_run.rst

Narrative docs (short note pointing to the declarative alternative):
- how-to.rst — after the pg_autoctl run step in Quick Start
- tutorial.rst — after the docker-compose.yml literalinclude
- citus-quickstart.rst — after the docker-compose-scale.yml literalinclude
- install.rst — after the systemd unit section

* docs: rewrite tutorial to use pg_autoctl node run

Replace the imperative pg_autoctl create postgres / pg_autoctl create
monitor commands with the declarative pg_autoctl node run approach:

- Add tutorial/ini/monitor.ini and tutorial/ini/postgres.ini — two small
  ini files that describe the monitor and every data node respectively.
  Data nodes share one ini file; hostname and name default to the
  container hostname set by Docker Compose.

- Rewrite tutorial/docker-compose.yml: all PG_AUTOCTL_* environment
  variables are gone; each service bind-mounts its ini file at
  /etc/pgaf/node.ini and runs 'pg_autoctl node run'. The x-node anchor
  is now clean — no env vars, single command.

- Update tutorial.rst to introduce the ini files before the compose file,
  explain that pg_autoctl node run handles both create and run, and show
  that live reconfiguration (candidate_priority change for node3) is done
  by editing the ini file rather than calling pg_autoctl set.

- Replace the stale pg_autoctl_do_tmux_compose_session reference in Next
  steps with a pointer to the pg_autoctl_node reference and the Container
  and Kubernetes Deployments section.

* docs: clarify candidate_priority change — direct command and ini file paths

Show both approaches side by side:

- Direct command (pg_autoctl set candidate-priority): immediate, no restart
- Declarative ini file: explain that changing docker-compose.yml volumes
  requires 'docker compose up -d node3' to recreate the container, that
  pg_autoctl node run applies the ini diff on startup before exec'ing into
  the supervisor, and that once the dedicated ini file is mounted any
  subsequent edits to it are picked up live by the running supervisor.

* docs: rewrite Citus tutorial to use pg_autoctl node run

Replace all pg_autoctl create coordinator/worker/monitor commands with
the declarative pg_autoctl node run approach:

New ini files:
  citus/ini/monitor.ini     — kind = monitor
  citus/ini/coordinator.ini — kind = coordinator, shared by coord0a/coord0b
  citus/ini/worker.ini      — kind = worker, no group (monitor auto-assigns)
  citus/ini/worker1.ini     — kind = worker, group = 1
  citus/ini/worker2.ini     — kind = worker, group = 2
  citus/ini/worker3.ini     — kind = worker, group = 3

Rewritten compose files:
  docker-compose-scale.yml  — three services (monitor/coord/worker), each
    bind-mounts its ini file; all PG_AUTOCTL_* env vars removed
  docker-compose.yml        — named services with per-group worker ini files;
    YAML anchors kept for coord and per-group worker templates

Updated citus-quickstart.rst:
  - Show all ini files with literalinclude captions before each compose file
  - Explain that worker.ini without [formation] group triggers monitor
    auto-assignment (right for --scale), while workerN.ini with group = N
    pins the pair to a specific shard group (right for named services)
  - Explain that pg_autoctl node run uses the container hostname as the
    node name when name is not set in the ini file
  - Replace stale pg_autoctl_do_tmux_compose_session reference in Next
    steps with pointer to pg_autoctl_node and container deployments guide

* style: apply citus_indent to nodespec.c

* style: fix nodespec.c PUSH macro indentation per citus_indent

Author: dimitri

docs/citus-quickstart.rst

@@ -50,46 +50,55 @@ or run the docker build command directly:
 Our first Citus Cluster
 -----------------------
 
-To create a cluster we use the following docker compose definition:
+Each node in the cluster is described by a ``pg_autoctl_node.ini`` file
+bind-mounted into its container.  There are three files:
 
-.. literalinclude:: citus/docker-compose-scale.yml
-   :language: yaml
-   :emphasize-lines: 5,15,27
-   :linenos:
+.. literalinclude:: citus/ini/monitor.ini
+   :language: ini
+   :caption: citus/ini/monitor.ini
 
-To run the full Citus cluster with HA from this definition, we can use the
-following command:
+.. literalinclude:: citus/ini/coordinator.ini
+   :language: ini
+   :caption: citus/ini/coordinator.ini
 
-::
+.. literalinclude:: citus/ini/worker.ini
+   :language: ini
+   :caption: citus/ini/worker.ini
 
-   $ docker compose up --scale coord=2 --scale worker=6
+The ``worker.ini`` has no ``[formation] group`` entry.  When that field is
+absent the monitor assigns each worker to a group automatically — the first
+worker to register in a group becomes primary, the second becomes secondary.
+This is the right setup for the scaled deploy below, where we ask Docker
+Compose to start six worker containers and let the monitor pair them.
+
+The docker compose definition for the scalable cluster is:
 
-The command above starts the services up. The command also specifies a
-``--scale`` option that is different for each service. We need:
+.. literalinclude:: citus/docker-compose-scale.yml
+   :language: yaml
+   :linenos:
 
- - one monitor node, and the default scale for a service is 1,
+Every service runs ``pg_autoctl node run`` — creating the node on first
+start, resuming on subsequent starts.  All ``PG_AUTOCTL_*`` environment
+variables are gone; everything lives in the ini files.
 
- - one primary Citus coordinator node and one secondary Cituscoordinator
-   node, which is to say two coordinator nodes,
+To run the full Citus cluster with HA from this definition:
 
- - and three Citus worker nodes, each worker with both a primary Postgres
-   node and a secondary Postgres node, so that's a scale of 6 here.
+::
 
-The default policy for the pg_auto_failover monitor is to assign a primary
-and a secondary per auto failover :ref:`group`. In our case, every node
-being provisioned with the same command, we benefit from that default policy::
+   $ docker compose up --scale coord=2 --scale worker=6
 
-  $ pg_autoctl create worker --ssl-self-signed --auth trust --pg-hba-lan --run
+The ``--scale`` options tell Docker Compose how many containers to start for
+each service:
 
-When provisioning a production cluster, it is often required to have a
-better control over which node participates in which group, then using the
-``--group N`` option in the ``pg_autoctl create worker`` command line.
+ - one monitor node (default scale is 1),
+ - two coordinator containers — one primary, one secondary,
+ - six worker containers — the monitor pairs them into three groups of two,
+   assigning a primary and secondary in each group.
 
-Within a given group, the first node that registers is a primary, and the
-other nodes are secondary nodes. The monitor takes care of that in a way
-that we don't have to. In a High Availability setup, every node should be
-ready to be promoted primary at any time, so knowing which node in a group
-is assigned primary first is not very interesting.
+Within a given group the first node that registers becomes primary; the
+monitor handles the assignment so we don't have to track it.  In a High
+Availability setup every node must be ready for promotion at any time, so
+the initial primary assignment within a group is not significant.
 
 While the cluster is being provisionned by docker compose, you can run the
 following command and have a dynamic dashboard to follow what's happening.
@@ -177,26 +186,36 @@ more complex docker compose file than in the previous section.
 
    pg_auto_failover architecture with a Citus formation
 
-This time we create a cluster using the following docker compose definition:
+This time we need per-group worker ini files so that each worker pair
+lands in the right Citus shard group:
+
+.. literalinclude:: citus/ini/worker1.ini
+   :language: ini
+   :caption: citus/ini/worker1.ini  (worker1a and worker1b)
+
+Worker 2 and worker 3 are identical except for ``group = 2`` and
+``group = 3`` respectively.  When ``group`` is set, ``pg_autoctl node run``
+passes ``--group N`` to ``pg_autoctl create worker``, pinning the pair to
+that shard group.
+
+The docker compose definition is:
 
 .. literalinclude:: citus/docker-compose.yml
    :language: yaml
-   :emphasize-lines: 3,15,40,44,48,52,56,60,64,68
    :linenos:
 
-This definition is a little more involved than the previous one. We take
-benefit from `YAML anchors and aliases`__ to define a *template* for our
-coordinator nodes and worker nodes, and then apply that template to the
-actual nodes.
+We use `YAML anchors and aliases`__ to define templates for the coordinator
+and each worker group, then apply them to the named services.  Each service
+sets its own ``hostname:`` — ``pg_autoctl node run`` uses the container
+hostname as the node name when ``name`` is not set in the ini file.
 
 __ https://yaml101.com/anchors-and-aliases/
 
-Also this time we provision an application service (named "app") that sits
-in the background and allow us to later connect to our current primary
-coordinator. See :download:`Dockerfile.app <citus/Dockerfile.app>` for the
-complete definition of this service.
+Also this time we provision an application service (``app``) that sits in
+the background and allows us to connect to the current primary coordinator.
+See :download:`Dockerfile.app <citus/Dockerfile.app>` for its definition.
 
-We start this cluster with a simplified command line this time:
+We start this cluster with:
 
 ::
 
@@ -474,13 +493,15 @@ makes it simple to introduce faults and see how the pg_auto_failover High
 Availability reacts to those faults.
 
 One obvious missing element to better test the system is the lack of
-persistent volumes in our docker compose based test rig. It is possible to
+persistent volumes in our docker compose based test rig.  It is possible to
 create external volumes and use them for each node in the docker compose
-definition. This allows restarting nodes over the same data set.
+definition, allowing nodes to restart over the same data set.
 
-See the command :ref:`pg_autoctl_do_tmux_compose_session` for more details
-about how to run a docker compose test environment with docker compose,
-including external volumes for each node.
+For production Kubernetes deployments, the same ini files work as
+ConfigMaps: bind-mount them alongside a persistent volume claim for
+``/tmp/pgaf`` and use ``pg_autoctl node run`` as the container command.
+See :ref:`pg_autoctl_node` for the full property reference and
+:ref:`container-and-kubernetes-deployments` for production patterns.
 
 Now is a good time to go read `Citus Documentation`__ too, so that you know
 how to use this cluster you just created!

docs/citus/docker-compose-scale.yml

@@ -1,36 +1,34 @@
-version: "3.9"  # optional since v1.27.0
-
 services:
 
   monitor:
     image: pg_auto_failover:citus
-    environment:
-      PGDATA: /tmp/pgaf
-    command: |
-      pg_autoctl create monitor --ssl-self-signed --auth trust --run
+    volumes:
+      - /tmp/pgaf
+      - ./ini/monitor.ini:/etc/pgaf/node.ini:ro
     expose:
       - 5432
+    command: pg_autoctl node run
 
   coord:
     image: pg_auto_failover:citus
+    volumes:
+      - /tmp/pgaf
+      - ./ini/coordinator.ini:/etc/pgaf/node.ini:ro
     environment:
-      PGDATA: /tmp/pgaf
       PGUSER: citus
       PGDATABASE: citus
-      PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover"
     expose:
       - 5432
-    command: |
-      pg_autoctl create coordinator --ssl-self-signed --auth trust --pg-hba-lan --run
+    command: pg_autoctl node run
 
   worker:
     image: pg_auto_failover:citus
+    volumes:
+      - /tmp/pgaf
+      - ./ini/worker.ini:/etc/pgaf/node.ini:ro
     environment:
-      PGDATA: /tmp/pgaf
       PGUSER: citus
       PGDATABASE: citus
-      PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover"
     expose:
-     - 5432
-    command: |
-      pg_autoctl create worker --ssl-self-signed --auth trust --pg-hba-lan --run
+      - 5432
+    command: pg_autoctl node run

docs/citus/docker-compose.yml

@@ -1,28 +1,14 @@
 x-coord: &coordinator
   image: pg_auto_failover:citus
+  volumes:
+    - /tmp/pgaf
+    - ./ini/coordinator.ini:/etc/pgaf/node.ini:ro
   environment:
-    PGDATA: /tmp/pgaf
     PGUSER: citus
     PGDATABASE: citus
-    PG_AUTOCTL_HBA_LAN: true
-    PG_AUTOCTL_AUTH_METHOD: "trust"
-    PG_AUTOCTL_SSL_SELF_SIGNED: true
-    PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover"
-  expose:
-    - 5432
-
-x-worker: &worker
-  image: pg_auto_failover:citus
-  environment:
-    PGDATA: /tmp/pgaf
-    PGUSER: citus
-    PGDATABASE: citus
-    PG_AUTOCTL_HBA_LAN: true
-    PG_AUTOCTL_AUTH_METHOD: "trust"
-    PG_AUTOCTL_SSL_SELF_SIGNED: true
-    PG_AUTOCTL_MONITOR: "postgresql://autoctl_node@monitor/pg_auto_failover"
   expose:
     - 5432
+  command: pg_autoctl node run
 
 services:
   app:
@@ -40,58 +26,71 @@ services:
 
   monitor:
     image: pg_auto_failover:citus
-    environment:
-      PGDATA: /tmp/pgaf
-      PG_AUTOCTL_SSL_SELF_SIGNED: true
+    volumes:
+      - /tmp/pgaf
+      - ./ini/monitor.ini:/etc/pgaf/node.ini:ro
     expose:
       - 5432
-    command: |
-      pg_autoctl create monitor --auth trust --run
+    command: pg_autoctl node run
 
   coord0a:
     <<: *coordinator
     hostname: coord0a
-    command: |
-      pg_autoctl create coordinator --name coord0a --run
 
   coord0b:
     <<: *coordinator
     hostname: coord0b
-    command: |
-      pg_autoctl create coordinator --name coord0b --run
 
   worker1a:
-    <<: *worker
+    <<: &worker1
+      image: pg_auto_failover:citus
+      volumes:
+        - /tmp/pgaf
+        - ./ini/worker1.ini:/etc/pgaf/node.ini:ro
+      environment:
+        PGUSER: citus
+        PGDATABASE: citus
+      expose:
+        - 5432
+      command: pg_autoctl node run
     hostname: worker1a
-    command: |
-      pg_autoctl create worker --group 1 --name worker1a --run
 
   worker1b:
-    <<: *worker
+    <<: *worker1
     hostname: worker1b
-    command: |
-      pg_autoctl create worker --group 1 --name worker1b --run
 
   worker2a:
-    <<: *worker
+    <<: &worker2
+      image: pg_auto_failover:citus
+      volumes:
+        - /tmp/pgaf
+        - ./ini/worker2.ini:/etc/pgaf/node.ini:ro
+      environment:
+        PGUSER: citus
+        PGDATABASE: citus
+      expose:
+        - 5432
+      command: pg_autoctl node run
     hostname: worker2a
-    command: |
-      pg_autoctl create worker --group 2 --name worker2a --run
 
   worker2b:
-    <<: *worker
+    <<: *worker2
     hostname: worker2b
-    command: |
-      pg_autoctl create worker --group 2 --name worker2b --run
 
   worker3a:
-    <<: *worker
+    <<: &worker3
+      image: pg_auto_failover:citus
+      volumes:
+        - /tmp/pgaf
+        - ./ini/worker3.ini:/etc/pgaf/node.ini:ro
+      environment:
+        PGUSER: citus
+        PGDATABASE: citus
+      expose:
+        - 5432
+      command: pg_autoctl node run
     hostname: worker3a
-    command: |
-      pg_autoctl create worker --group 3 --name worker3a --run
 
   worker3b:
-    <<: *worker
+    <<: *worker3
     hostname: worker3b
-    command: |
-      pg_autoctl create worker --group 3 --name worker3b --run

docs/citus/ini/coordinator.ini

@@ -0,0 +1,14 @@
+[node]
+kind = coordinator
+port = 5432
+
+[postgresql]
+pgdata = /tmp/pgaf
+
+[monitor]
+pguri = postgresql://autoctl_node@monitor/pg_auto_failover
+
+[options]
+ssl        = self-signed
+auth       = trust
+pg_hba_lan = true

docs/citus/ini/monitor.ini

@@ -0,0 +1,10 @@
+[node]
+kind = monitor
+port = 5432
+
+[postgresql]
+pgdata = /tmp/pgaf
+
+[options]
+ssl  = self-signed
+auth = trust

docs/citus/ini/worker.ini

@@ -0,0 +1,14 @@
+[node]
+kind = worker
+port = 5432
+
+[postgresql]
+pgdata = /tmp/pgaf
+
+[monitor]
+pguri = postgresql://autoctl_node@monitor/pg_auto_failover
+
+[options]
+ssl        = self-signed
+auth       = trust
+pg_hba_lan = true

docs/citus/ini/worker1.ini

@@ -0,0 +1,17 @@
+[node]
+kind = worker
+port = 5432
+
+[postgresql]
+pgdata = /tmp/pgaf
+
+[monitor]
+pguri = postgresql://autoctl_node@monitor/pg_auto_failover
+
+[formation]
+group = 1
+
+[options]
+ssl        = self-signed
+auth       = trust
+pg_hba_lan = true