test tooling: pgaftest binary, .pgaf spec suite, CI modernisation - #1132
Closed
dimitri wants to merge 69 commits into
Closed
test tooling: pgaftest binary, .pgaf spec suite, CI modernisation#1132dimitri wants to merge 69 commits into
dimitri wants to merge 69 commits into
Conversation
…de.ini
Adds a new pg_autoctl node sub-command tree and a NodeSpec file format
designed as the recommended entry-point for container and Kubernetes
deployments.
## pg_autoctl_node.ini sections
[node] kind (postgres|monitor|coordinator|worker), name, hostname, port
[postgresql] pgdata
[monitor] pguri (empty for monitor nodes), no_monitor, node_id
[formation] name, group
[settings] candidate_priority, replication_quorum ← mutable, applied live
[options] ssl, auth, pg_hba_lan ← create-time only
[ssl] ssl_ca_file, ssl_cert_file, ssl_key_file
[launch] mode=deferred: wait for pg_autoctl node start
[formation N] monitor only: additional named formations
## pg_autoctl node sub-commands
run <file> Read ini, create node if absent, exec() into supervisor.
Sets PG_AUTOCTL_NODESPEC so the supervisor watches for
live [settings] changes via inotify (Linux) or mtime poll.
apply <file> Converge mutable settings on an already-running node.
start [<file>] Clear launch=deferred so a waiting node run proceeds.
show Dump live config as pg_autoctl_node.ini on stdout.
check <file> Parse-only validate; print resolved fields.
## Supervisor file watcher (nodespec_watcher)
The supervisor initialises a NodeSpecWatcher when PG_AUTOCTL_NODESPEC is
set. Every tick it checks for file changes:
- Linux: drain inotify IN_CLOSE_WRITE / IN_MOVED_TO events
- Others: stat() every NODESPEC_WATCH_INTERVAL_SECS (10 s)
On change, re-parse [settings] and call nodespec_apply() to converge
mutable fields without restarting the node.
## Files
src/bin/pg_autoctl/cli_node.c / cli_node.h
src/bin/pg_autoctl/nodespec.c / nodespec.h
src/bin/pg_autoctl/supervisor.c / supervisor.h (watcher integration)
docs/ref/pg_autoctl_node.rst
docs/ref/pg_autoctl_node_run.rst
- pg_autoctl_node.rst: replace prose property list with a structured table (section/property/type/mutable/default/description), each property on its own row; fix 'disabled-monitor' label → actual ref - nodespec_apply: add ssl apply block — when [options].ssl or any [ssl] cert path changes, call 'pg_autoctl enable ssl' with the appropriate flags (--ssl-self-signed, --no-ssl, or --ssl-mode + cert paths); update nodespec_apply comment to list ssl as mutable - docs/index.rst: remove 'Container and Kubernetes' as a top-level toctree caption; the manual page is still reachable via Manual Pages - docs/operations.rst: add 'Container and Kubernetes Deployments' section describing pg_autoctl node run, live reconfiguration, and the launch=deferred pattern; cross-refs to pg_autoctl_node for details - docs/ref/configuration.rst: add 'Declarative Node Configuration' section covering pg_autoctl_node.ini sections and their relationship to pg_autoctl.cfg; cross-ref to pg_autoctl_node
When the [monitor] pguri changes in pg_autoctl_node.ini, nodespec_apply now re-registers the node to the new monitor without stopping Postgres: pg_autoctl disable monitor --force --pgdata <dir> pg_autoctl enable monitor <new_uri> --pgdata <dir> The disable step removes the node from the old monitor (--force allows this even if the old monitor is temporarily unreachable). The enable step registers the node to the new monitor and signals the running supervisor to start using the new monitor_pguri for node_active calls. Document the change in the property table and Live Reconfiguration section of pg_autoctl_node.rst.
…ction Follow the pg_autoctl_create pattern: each pg_autoctl node sub-command now has its own manual page, all listed in pg_autoctl_node.rst's toctree: pg_autoctl node run (existing) pg_autoctl node apply (new) pg_autoctl node start (new) pg_autoctl node show (new) pg_autoctl node check (new) Remove the redundant 'Container and Kubernetes Deployments' section from operations.rst. The full documentation for this feature lives in the pg_autoctl_node manual page (ref/pg_autoctl_node.rst) and its sub-pages.
The list-table rendered poorly in the Sphinx HTML theme. Replace it with the same definition-list style used throughout the other manual pages (e.g. pg_autoctl_create_postgres Options section): each property is a bare term followed by indented description paragraphs, grouped under ini-section sub-headings.
…ioning Re-add the 'Container and Kubernetes Deployments' section at the end of operations.rst with a named anchor so it can be referenced from elsewhere. Add a short note in the Provisioning section pointing to pg_autoctl node run as the declarative alternative for container and Kubernetes deployments, with a cross-reference down to the new section.
Add references to pg_autoctl node / pg_autoctl node run in all the places identified by the documentation audit: Reference pages (new See Also section): - pg_autoctl_create_postgres.rst - pg_autoctl_create_monitor.rst - pg_autoctl_create_coordinator.rst - pg_autoctl_create_worker.rst - pg_autoctl_run.rst Narrative docs (short note pointing to the declarative alternative): - how-to.rst — after the pg_autoctl run step in Quick Start - tutorial.rst — after the docker-compose.yml literalinclude - citus-quickstart.rst — after the docker-compose-scale.yml literalinclude - install.rst — after the systemd unit section
Replace the imperative pg_autoctl create postgres / pg_autoctl create monitor commands with the declarative pg_autoctl node run approach: - Add tutorial/ini/monitor.ini and tutorial/ini/postgres.ini — two small ini files that describe the monitor and every data node respectively. Data nodes share one ini file; hostname and name default to the container hostname set by Docker Compose. - Rewrite tutorial/docker-compose.yml: all PG_AUTOCTL_* environment variables are gone; each service bind-mounts its ini file at /etc/pgaf/node.ini and runs 'pg_autoctl node run'. The x-node anchor is now clean — no env vars, single command. - Update tutorial.rst to introduce the ini files before the compose file, explain that pg_autoctl node run handles both create and run, and show that live reconfiguration (candidate_priority change for node3) is done by editing the ini file rather than calling pg_autoctl set. - Replace the stale pg_autoctl_do_tmux_compose_session reference in Next steps with a pointer to the pg_autoctl_node reference and the Container and Kubernetes Deployments section.
… paths Show both approaches side by side: - Direct command (pg_autoctl set candidate-priority): immediate, no restart - Declarative ini file: explain that changing docker-compose.yml volumes requires 'docker compose up -d node3' to recreate the container, that pg_autoctl node run applies the ini diff on startup before exec'ing into the supervisor, and that once the dedicated ini file is mounted any subsequent edits to it are picked up live by the running supervisor.
Replace all pg_autoctl create coordinator/worker/monitor commands with
the declarative pg_autoctl node run approach:
New ini files:
citus/ini/monitor.ini — kind = monitor
citus/ini/coordinator.ini — kind = coordinator, shared by coord0a/coord0b
citus/ini/worker.ini — kind = worker, no group (monitor auto-assigns)
citus/ini/worker1.ini — kind = worker, group = 1
citus/ini/worker2.ini — kind = worker, group = 2
citus/ini/worker3.ini — kind = worker, group = 3
Rewritten compose files:
docker-compose-scale.yml — three services (monitor/coord/worker), each
bind-mounts its ini file; all PG_AUTOCTL_* env vars removed
docker-compose.yml — named services with per-group worker ini files;
YAML anchors kept for coord and per-group worker templates
Updated citus-quickstart.rst:
- Show all ini files with literalinclude captions before each compose file
- Explain that worker.ini without [formation] group triggers monitor
auto-assignment (right for --scale), while workerN.ini with group = N
pins the pair to a specific shard group (right for named services)
- Explain that pg_autoctl node run uses the container hostname as the
node name when name is not set in the ini file
- Replace stale pg_autoctl_do_tmux_compose_session reference in Next
steps with pointer to pg_autoctl_node and container deployments guide
…pec suite Combines what would have been two separate PRs: - src/bin/common/: move ~20 utility sources (pgsql, pgsetup, ini_file, …) out of pg_autoctl/ into a shared static library linked by both binaries - src/bin/pgaftest/: new test binary — reads .pgaf specs, runs headless CI (TAP output) or interactive setup with docker compose - tests/tap/specs/: 35 .pgaf spec files covering the full test matrix - tests/tap/schedule: ordered run list for CI - tests/upgrade/: dual-binary upgrade test image and shim script - .github/workflows/run-pgaftest.yml: CI workflow — builds pgaf:run image and pgaftest binary once, fans out one job per spec in parallel - Dockerfile: add pgaftest build/install target, bison touch guard - style: all new C files reformatted with citus/stylechecker:no-py Docker image
Split the single build job into: - build-images: matrix over PG14/15/16/17, one pgaf:run-pgNN artifact each - build-pgaftest: single job (PG17), binary is version-agnostic at runtime The test matrix now crosses every spec × every PG version (19 specs × 4 versions = 76 parallel jobs), matching the run-tests.yml coverage scope. installcheck likewise runs on all four versions. This positions pgaftest as the replacement for run-tests.yml once all scenarios are ported — run-tests.yml will be removed at that point.
Three panes, laid out evenly: top — docker compose logs -f (all services) middle — pg_autoctl watch (on the monitor) bottom — bash inside the first non-deferred data node Previously the session was created detached (-d) and the user had to manually run 'tmux attach'. Now runner_setup attaches immediately after creating the session so the terminal lands inside tmux. The shell pane target is the first non-deferred data node in the spec's formation block; falls back to the monitor when no data node is found.
…c --tmux) The commandline library calls getopt, which on POSIX/macOS stops scanning at the first non-option argument (the spec file path). Any flags given after the spec file — the natural invocation form for setup/run -- are silently ignored, so --tmux and --work-dir have no effect. Fix: after cli_setup extracts argv[0] as the spec path, reset getopt state (optreset=1 on BSD, optind=1 everywhere) and call pgaftest_getopts again on the remaining argv. argv[0] serves as a dummy program name so getopt starts scanning from index 1, picking up --tmux, --work-dir, etc. Also applies to the tmux session creation: - session is now attached immediately (tmux attach-session after new-session -d) - third pane execs bash inside the first non-deferred data node
setup --tmux timing
The setup{} block now runs inside the tmux bottom pane instead of
before tmux is created. The user immediately gets the three-pane
session; the bottom pane shows setup progress and transitions to an
interactive bash shell when done.
- add runner_run_setup_only() — runs setup{} against an existing stack
- add pgaftest _setup_ sub-command — internal entry-point for the pane
- runner_setup() launches tmux first (after compose up + monitor ready)
then the bottom pane calls _setup_ and exec's bash on success
- set pg_autoctl_program to argv[0] in main() so runner_setup can
reference the binary path in the tmux pane command
pgaftest down --work-dir
When no spec file is found, cli_down now derives the project name from
the workDir basename and runs:
docker compose -p <name> -f <workDir>/docker-compose.yml down ...
instead of a bare docker compose down that silently targets nothing.
docs
- docs/ref/pgaftest.rst: full reference (sub-commands, DSL, env vars, TAP)
- docs/operations.rst: new Testing section
- docs/tutorial.rst: pgaftest alternative section with interactive_tutorial.pgaf
- docs/citus-quickstart.rst: pgaftest alternative with citus_tutorial.pgaf
- docs/index.rst: add ref/pgaftest to toctree
- docs/tutorial/interactive_tutorial.pgaf: two-node HA interactive spec
- docs/tutorial/citus_tutorial.pgaf: eight-node Citus interactive spec
Dockerfile: replace deprecated apt-key with gpg --dearmor
Three stages (base/build, run, testrun) were using:
curl ... | apt-key add -
which triggers a deprecation warning in recent apt versions.
Replaced with the modern approach:
gpg --dearmor -o /usr/share/keyrings/pgdg-archive-keyring.gpg
and added signed-by= to the sources list entry.
C: add missing prototype declarations
- pgsql_alter_role_password() declared in common/pgsql.h was not
visible when pg_autoctl/pgsql.h was included first; added to
pg_autoctl/pgsql.h so the declaration is found regardless of
include path ordering.
- keeper_cli_pgsetup_hba_lan() in cli_do_misc.c was missing from
cli_do_root.h; added alongside the other pgsetup helpers.
dimitri
added
enhancement
The 16 .c/.h files that were moved to src/bin/common/ were still present in src/bin/pg_autoctl/ as copies. pg_autoctl was compiling them locally via the SRC = $(wildcard *.c) glob AND linking libpgaf_common.a, causing duplicate symbol resolution (local .o silently winning over the archive). All 32 files (16 .c + 16 .h pairs) are removed; pg_autoctl now uses only the common/ versions via libpgaf_common.a and its include path. The 6 files that had minor divergences were analysed: - ini_file.c / ipaddr.c / pgsetup.c / signals.c: only brace-style differences from citus_indent; common/ version is correct. - pgctl.c: common/ is a superset (adds --checkpoint=fast for pg_basebackup, fixes intToString dangling-pointer use). - pgsql.c: common/ is a superset (adds pgsql_alter_role_password). Both .c and .h pairs for pgctl and pgsql were checked; common/ contains all declarations.
Replace strncpy/strncat with strlcpy/strlcat in hand-written pgaftest source files (test_runner.c, cli_root.c, cli_indent.c, test_spec_parse.c), and add IGNORE-BANNED to unavoidable memcpy calls. Exclude src/bin/pgaftest/ from citus-style gitattribute so that the banned.h.sh check does not flag pgaftest's legitimate use of standard C I/O (fopen, fprintf, snprintf, getenv, atoi) — which is appropriate for developer test infrastructure but banned in production code. The Flex/Bison generated files (test_spec_scan.c, test_spec_parse.c) are covered by the same exclusion.
v13.0.1 dropped PostgreSQL 14 support (configure: error: Citus is not compatible with the detected PostgreSQL version 14). v12.1.5 is what main uses and supports PG14 through PG17.
Replace banned C stdlib calls with the project's safe wrappers: - snprintf() → sformat() (wraps pg_vsnprintf; from file_utils.h) - fprintf() → fformat() (wraps pg_vfprintf; from file_utils.h) - vsnprintf() → pg_vsnprintf() - printf() → fformat(stdout, ...) - atoi() → stringToInt() (from string_utils.h) - strncat() → strlcat() (already done in earlier commit) - strncpy() → strlcpy() (already done in earlier commit) For calls with no safe-wrapper alternative (fopen, getenv, sscanf, memcpy) that already follow the documented safe-use pattern, add /* IGNORE-BANNED */ on the calling line — matching the approach used in pg_autoctl/nodespec.c and common/env_utils.c. The Bison/Flex generated files (test_spec_parse.c, test_spec_scan.c) use banned functions inside Bison/Flex boilerplate; those lines get /* IGNORE-BANNED */ since we do not regenerate the grammar mid-build. Add #include "file_utils.h" and #include "string_utils.h" to the files that now call sformat/fformat/stringToInt. Remove the src/bin/pgaftest/** -citus-style .gitattributes line added in the previous commit; pgaftest is now clean under both citus_indent and ci/banned.h.sh.
When auth=cert is configured, pg_autoctl writes HBA rules for standby
replication connections with the plain 'cert' method. Cert auth
requires the client certificate CN to match the database username, but
both monitor and replication connections use the same client certificate
(CN=autoctl_node) while the replication role is pgautofailover_replicator.
Without an ident map the replication connection always fails:
FATAL: certificate authentication failed for user "pgautofailover_replicator"
Fix keeper_update_group_hba() to:
1. Use 'cert map=pgautofailover' for the replication HBA rule, matching
the same pattern already used in cli_do_misc.c for LAN CIDR rules.
2. Call pghba_ensure_ident_map_entry() to write
pgautofailover autoctl_node pgautofailover_replicator
to pg_ident.conf, so the cert CN maps to the replication role.
Verified with ssl_cert.pgaf: all 5 tests pass, formation completes in
~5 seconds.
When --auth md5 is used, upstream_has_replication_slot() needs to supply the replication password to connect to the upstream node. Previously it built the connection string without a password, causing authentication failures when the upstream requires md5. After 5 consecutive failures wait_until_primary_has_created_our_replication_slot() gives up, the init process exits, the supervisor restarts it up to 5 times in 300s, and the container dies -- all while the test waits for the formation to reach primary/secondary, timing out after 300s. Fix: set PGPASSWORD from upstream->password before calling pgsql_init(), following the same save/restore pattern used in pg_basebackup().
Replace the bare-runner pattern (extract binary + apt-get install libpq5)
with a self-contained pgaf:pgaftest Docker image that already bundles
libpq5, docker-ce-cli, and docker-compose-plugin.
build-pgaftest now builds --target pgaftest and uploads the image as an
artifact. All test/installcheck/upgrade jobs load that image and run
pgaftest via:
docker run --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp/pgaftest:/tmp/pgaftest \
-v $(pwd):/work:ro \
pgaf:pgaftest \
pgaftest run ...
The /tmp/pgaftest bind-mount lets compose stacks started by pgaftest
(inside the container) access the generated INI and compose files via
the same host paths the Docker daemon sees — the standard DooD pattern
on Linux runners.
This eliminates the fragile dependency on Microsoft's apt repositories
that are pre-installed on GitHub's ubuntu-latest image and have been
causing sporadic apt-get update failures (NOSPLIT / auth errors) that
have nothing to do with our code.
pgaf:pgaftest runs as USER docker (UID 1000). On GitHub's ubuntu-latest runners /var/run/docker.sock is srw-rw---- root:docker where the docker GID is ~999, not 1000. The container's docker group GID doesn't match the host GID so all docker compose calls fail immediately with permission denied, causing every test job to exit in <30 seconds. Fix: run the container as root and add the host docker socket GID via --group-add so both the socket access and writes to /tmp/pgaftest (bind-mounted from the runner) work without restriction.
GitHub Actions ubuntu-latest now uses rootless Docker. Container root (uid 0) maps to the rootless daemon uid on the host, which is different from the runner uid that created /tmp/pgaftest (mode 755). Writes from inside the pgaf:pgaftest container therefore fail with EACCES when pgaftest tries to create its per-spec work directories. Making the bind-mount root world-writable (777) allows any mapped uid to create sub-directories inside it, restoring write access regardless of the uid remapping in effect.
assert stays primary while { stop postgres monitor } fails because
the stays-while logic checks the state via the monitor (LISTEN or SQL)
after each body command. When the body stops the monitor's own postgres,
both the LISTEN connection and the SQL fallback become unavailable, so
the runner reports 'could not get state of node3 after command' and the
assertion fails incorrectly.
The monitor cannot assign state changes while its postgres is down, so
there is nothing to guard against during the outage. Replace the
stays-while block with a plain stop + sleep + wait, matching the intent
of the original Python test: verify that a brief monitor outage does not
trigger a spurious failover.
…thcheck multi_ifdown test_014: compose stop node3/node1 caused pg_autoctl perform failover to block ~68s waiting for dead primaries, leaving only 22s of a 90s timeout. Switch to network disconnect (matches the Python test's ifdown()), run the failover in background so the wait starts immediately, and increase the timeout to 300s. Update test_015 to use network connect instead of compose start. citus_skip_pg_hba: coord0a started simultaneously with the monitor but auth=skip leaves the monitor's pg_hba.conf with only localhost rules, so coord0a's registration attempts from another container all fail and its healthcheck never passes. Make coord0a launch deferred; move the monitor HBA setup into the setup block (before starting coord0a); rename test_000 to reflect its new sole job of adding coord0a's own HBA rules. compose_gen.c: a deferred node in the first slot (now coord0a) was incorrectly chosen as the healthcheck anchor. Its pg_autoctl status always returns non-zero during sleep infinity, blocking all subsequent nodes via service_healthy. Skip deferred nodes when selecting firstNode so the healthcheck lands on the first node that actually starts postgres.
The pg_rewind after this test requires a clean shutdown of node3 (primary) and node1 so their WAL is preserved for the rewind-against-node2 step. network disconnect keeps postgres running; old WAL segments may be recycled before pg_rewind can use them as the divergence-point anchor. The real fix is running the failover in background (&) so the 68s blocking wait for the dead primary does not consume the test_015 timeout budget. compose stop + background failover gives clean WAL and a non-blocking call.
…ip_pg_hba multi_ifdown test_014: replace background perform-failover + wait-for-report_lsn with blocking exec and wait for stable end state. The report_lsn election completes in under a second in local Docker; polling at 500ms granularity cannot catch the intermediate state. Also drop the now-unnecessary test_015_start_node3 compose start step (all nodes already online). citus_skip_pg_hba test_005: remove the wait-for-wait_primary intermediate check. perform failover --group 1 already blocks until the new primary is established, so wait_primary is already gone by the time polling starts. Root cause in both cases: the monitor FSM transitions from report_lsn (or wait_primary) to the next state in under 1s, while the TAP runner polls via docker compose exec at ~500ms intervals, making the intermediate state reliably invisible under load.
ssl (all PG versions): On slow Azure CI runners the first data node's ssl_cert initialization (postgres init + SSL cert config + monitor TLS handshake) consistently exceeds the 360s Docker healthcheck window (start_period=120s + retries=120 × interval=2s = 360s). For SSL clusters (verify-ca / verify-full), double start_period to 300s and increase retries to 150, giving a max window of 600s. multi_async test_011 (PG17): The perform_promotion SQL in test_010 is non-blocking. On fast CI runners the LSN election completes in under 1s, so by the time test_011 polls for 'node4 assigned-state = report_lsn' (180s timeout), node4 is already secondary and the transient state is long gone. Fix: disconnect node4 before the promotion so the election runs without it — same code path (candidate missing during LSN election), same observable outcome, no race window.
The ssl_cert test was permanently failing in CI (Azure runners) because
node1's 600s healthcheck window had to cover both the monitor's slow
SSL initialisation (cert copy + pg_autoctl SSL config + pghba setup)
AND node1's own postgres init. On fast local machines the sum fits
comfortably, but on Azure it was exceeding 600s.
Fix: give the monitor its own healthcheck (same pg_autoctl status
probe; 300s start_period for SSL clusters, 60s otherwise) and make
node1 depend on service_healthy for the monitor. The timing is now
serial:
monitor fully initialised (≤600s)
→ node1 starts and initialises (≤600s)
→ node2 starts (depends_on node1: service_healthy)
Each stage has its full window, so slow CI runners no longer hit the
combined deadline.
Without this, pgaftest's setup timer starts as soon as the monitor is
connectable, but node1 may not have started yet (it now waits for the
monitor healthcheck to pass first). For ssl_cert clusters the 300s
setup timeout was being consumed by node1+node2 initialisation from
scratch rather than just a status check.
With this change the start ordering is fully serialised:
monitor healthy → node1 starts → node1 healthy → pgaftest starts setup
→ node2 starts
The setup {} block's wait timeouts only need to cover the remaining
state transitions after the cluster skeleton is up.
…dump container logs on failure
Two fixes:
1. For verify-ca / verify-full clusters with cert auth, embed explicit
SSL connection parameters in the monitor pguri written to each node's
ini file:
pguri = postgresql://autoctl_node@monitor/pg_auto_failover
?sslmode=verify-ca
&sslrootcert=/etc/pgaf/ssl/ca.crt
&sslcert=/var/lib/postgres/.postgresql/postgresql.crt
&sslkey=/var/lib/postgres/.postgresql/postgresql.key
The implicit form (sslmode=prefer with libpq auto-discovery of
~/.postgresql/) works on macOS Docker Desktop but fails in the CI
Linux DinD environment: node1 retried the monitor connection for 607s
and was declared unhealthy. Making the params explicit removes the
dependency on $HOME and libpq search paths.
2. On docker compose up failure, dump all container logs before
returning false. Every previous ssl_cert CI failure produced just
'node1 is unhealthy' with no indication of what pg_autoctl was
actually printing inside node1 for 600 seconds. This makes the next
failure self-diagnosing.
… compose-up failure Debian Bullseye went EOL June 2026; apt repositories are no longer reliably served, causing the base-stage apt-get install to fail with exit 100 in the upgrade test's pgaf-next build. Switch all three independent stages (base, run, pgaftest) to debian:bookworm-slim and update the PGDG + Docker apt source codenames accordingly. Also remove python3-nose and mg which were dropped from Bookworm. test_runner: add container log dump in runner_run() when docker compose up fails. The existing dump in runner_compose_up() is not reached by the schedule runner (which calls compose up inline in runner_run); this gap meant ssl_cert failures showed no pg_autoctl output from node1.
pyroute2 was only used by the old Python test suite (tests/network.py). pgaftest implements network disconnect/connect via 'docker network disconnect' so Python is not needed. On Bookworm, pip3 can no longer install into the system Python without --break-system-packages anyway.
python3-nose was removed from Debian Bookworm apt repos. Bookworm also enforces PEP 668 which rejects bare pip3 installs. Use --break-system-packages to restore nosetests (needed by run-tests.yml's NOSETESTS Makefile variable) and pyroute2 (imported by all old Python tests via pgautofailover_utils -> tests/network.py). Drop the <0.7.0 upper bound on pyroute2; the constraint was added as a compatibility guard that no longer applies to the old test suite.
python3-nose is not available in Bookworm, and even when installed via pip, nose crashes on Python 3.11 because collections.Callable was removed in Python 3.10 (nose/suite.py still uses it). Switch the Makefile test runner from nosetests to pytest and install pytest via pip3 in the Dockerfile alongside nose (kept for nose.tools imports like eq_ and raises which do not trigger the broken suite path). The TEST_ARGUMENT format switches from nose's --where/--tests style to pytest's path-based format (tests/ for all, tests/foo.py for named).
…ytest pytest tests/ would descend into tests/tablespaces/ which uses a different import layout (no __init__.py, imports as tests.tablespaces.*) and fails with ModuleNotFoundError. Tablespaces tests have their own make target; exclude them from the default pytest run by globbing only top-level .py files.
…tness Four independent fixes addressing the failures in the old Python test suite (run-tests.yml) and one pgaftest improvement: 1. Makefile.citus: fix TEST_ARGUMENT for pytest The switch from nosetests to pytest (commit 1efe48b) updated the main Makefile but missed Makefile.citus. The old nosetests-style '--where=tests --tests=...' arguments are unrecognized by pytest, causing all four citus test runs to fail with exit code 4. Use the same pattern as multi/single/monitor/ssl: TEST_ARGUMENT = $(TESTS_CITUS:%=tests/%.py) 2. tests/network.py: add LC_ALL=C LANG=C to network-namespace subprocesses pyroute2 >= 0.5.17 (installed in the Bookworm test image via 'pip3 install pyroute2>=0.5.17') strictly decodes NSPopen output as UTF-8. When pg_autoctl logs a PostgreSQL error whose message contains a raw non-UTF8 byte (e.g. the byte value embedded in 'invalid byte sequence for encoding'), the decode raises UnicodeDecodeError, failing test_multi_async.py::test_004_set_async on all four PG versions. Set LC_ALL=C and LANG=C in the subprocess env so PostgreSQL and pg_autoctl produce ASCII-only error messages inside network namespaces. 3. tests/tablespaces/conftest.py: add project root to sys.path The tablespaces tests import as 'from tests.tablespaces.xxx import ...' which requires the project root on sys.path. nosetests satisfied this implicitly (tests/__init__.py makes it a package discovered from cwd); pytest does not. A conftest.py in tests/tablespaces/ is the standard pytest hook for per-directory path setup. 4. citus_skip_pg_hba.pgaf: add pgsetup wait before every hba-lan call pg_autoctl node start on a deferred-launch node returns immediately while the container's PID 1 is still polling for launchDeferred=false, then execv()ing into pg_autoctl create. Calling hba-lan immediately races with that execv and the subsequent supervisor + initdb startup. pg_autoctl inspect pgsetup wait blocks until postgres is accepting connections, ensuring pg_hba.conf exists before hba-lan tries to edit it. Applied to coord0a (test_000), worker1a (test_002d), and worker1b (test_003). 5. pgaftest: mount spec directory at /etc/pgaf/specs in all containers Add specDir (dirname of the .pgaf spec file) as a read-only bind-mount at /etc/pgaf/specs in every service container. This allows specs to reference static JSON or other helper files shipped alongside them in git, without resorting to inline shell printf hackery. Use this to clean up monitor_disabled.pgaf: the pg_autoctl manual fsm nodes set JSON payloads are now checked-in as monitor_disabled_nodes12.json / monitor_disabled_nodes123.json and referenced as /etc/pgaf/specs/monitor_disabled_nodes12.json. Avoids command-line length issues and makes the intent readable. Also fix run_cmd_capture() in test_runner.c to drain remaining output after the 4096-byte buffer fills, preventing early pipe close and the SIGPIPE -> SIGKILL (exit 137) it would send to the docker exec'd process.
…ce autocommit - pgaftest CI: pass PGAFTEST_HOST_WORK_DIR to docker run so compose_gen translates the container-internal specDir (/work/tests/tap/specs) to the HOST-side path that the docker daemon can resolve for the /etc/pgaf/specs bind mount in node containers. Fixes test_005_fsm_nodes_set (exit 256). - multi_standbys test_005: add 'wait until node1 state is primary' after each 'set formation number-sync-standbys N' so synchronous_standby_names is checked after the apply_settings → primary FSM cycle completes. - multi_async test_004_set_async: add 'wait until node1 state is primary' after 'set formation number-sync-standbys 0' before setting per-node replication-quorum. Without the wait, the primary enters apply_settings and the monitor rejects the third node's replication-quorum change. - pgautofailover_utils run_sql_query: avoid 'with conn:' transaction context when autocommit=True; 'with conn:' wraps an implicit BEGIN that blocks DDL like CREATE TABLESPACE. Use explicit commit/rollback instead.
…F8 output monitor.c had two classes of dangling pointer bugs that caused garbage bytes to appear as SQL parameter values, triggering Monitor ERROR messages with non-UTF8 bytes and test UnicodeDecodeErrors: 1. IntString declared inside an if-block scope: paramValues[N] pointed into myGroupIdString.strValue, but that stack variable was destroyed when the if-block exited — before pgsql_execute_with_params was called. Fixed in monitor_get_nodes and monitor_print_nodes_as_json by moving the IntString declaration to function scope. 2. intToString(x).strValue inline temporaries: the returned IntString struct is destroyed at the semicolon, leaving paramValues[N] pointing to freed stack. Fixed across all 12 affected functions (monitor_node_active, monitor_register_node, monitor_set_node_candidate_priority, monitor_set_formation_number_sync_standbys, monitor_remove_by_hostname, monitor_perform_failover, monitor_create_formation, monitor_update_node_metadata, monitor_set_node_system_identifier, monitor_set_group_system_identifier, monitor_start_maintenance, monitor_stop_maintenance, monitor_find_node_by_nodeid) by naming the IntString variables at function scope. Clang -Wdangling-assignment now reports zero warnings for monitor.c. network.py: replace universal_newlines=True with encoding='utf-8', errors='replace' in both NSPopen calls so that any residual non-UTF8 bytes in pg_autoctl output are replaced rather than raising UnicodeDecodeError.
Split multi-1/multi-2 into three focused schedules: - multi-alternate: multi_alternate alone (~6m) - multi-async: multi_async alone (~6m) - multi-misc: multi_standbys + multi_maintenance + ensure + multi_ifdown (~8m) This brings the slow-schedule ceiling from 10.6m down to ~8m, with quick and node (~7m each) now the practical floor. Rebalance citus schedules: - citus-1: citus_cluster_name + citus_force_failover + citus_multi_standbys (~4m) - citus-2: basic_citus_operation + nonha_citus_operation + citus_skip_pg_hba (~6m) Add style_checker job to run-pgaftest.yml (was only in run-tests.yml), and gate both build-images and build-pgaftest on it passing. Also gate build_images in run-tests.yml on style_checker.
The azure command stubs were already guarded by #if 0 after the initial removal. Delete them outright — dead code hidden behind preprocessor guards is worse than no code at all.
…count - Delete Makefile.azure and its include/COPY references — the three make targets (azcluster, az, azdrop) called pg_autoctl do azure which was deleted; the file was pure dead weight - Drop pipenv from Dockerfile base apt-get block; nothing uses it - Fix job-count formula comment: 3×4 + 5×1 = 17 (not 16) - Remove tombstone comment from cli_demo.c file header - Drop stale log_info pointing users to deleted pg_autoctl do demo run
dimitri
changed the title
Contributor
Author
Extracted PRs — all mergedThis WIP has been fully extracted into the following PRs, all of which are now merged:
Closing in favour of the above. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Replaces the Python/pytest test suite with a purpose-built test binary (
pgaftest) and a declarative spec format (.pgaf). Ports all existing tests to the new format, wires up a full CI matrix for PG 14–17, and removes the unmaintained Azure integration.New:
pgaftestbinary (src/bin/pgaftest/)A self-contained test runner that reads
.pgafspec files and drives Docker Compose clusters:pgaftest run <spec.pgaf>— headless CI mode: compose up →setup{}→ steps in order →teardown{}→ TAP output + exit codepgaftest run --schedule <file>— run all specs listed in a schedule filepgaftest setup <spec.pgaf> [--tmux]— interactive mode: brings the cluster up and ready, then drops into a shell (or tmux session with log/watch panes)pgaftest step <name>,pgaftest down— run individual steps or tear down against a live stackThe
.pgafformat covers: cluster topology (cluster{}withmonitor, per-node options),setup{}/teardown{}blocks, namedstep{}blocks withexec,sql,expect,wait until,network disconnect/connect,compose stop/start,sleep, andpromote.Shared library (
src/bin/common/)~20 utility files moved out of
src/bin/pg_autoctl/into a shared static library (libpgaf_common.a). Bothpg_autoctlandpgaftestlink against it, eliminating source duplication.Test suite (
tests/tap/specs/,tests/tap/schedules/)29 spec files, 301 steps total, covering the full test matrix:
Plus:
installcheck(PG 14–17) andupgrade(PG 16).CI (
.github/workflows/run-pgaftest.yml)New workflow: style check gates all jobs; builds one
pgaf:runimage per PG version plus a singlepgaf:pgaftestimage (built once from PG 17); test jobs download only the image they need. 3×4 + 5×1 = 17 test jobs + 4 installcheck + 1 upgrade..github/workflows/run-tests.yml(existing Python suite): style check now also gatesbuild_imagesand all downstream jobs.New
pg_autoctlsub-commandspg_autoctl inspect— read-only diagnostics (FSM state, monitor queries, networking) safe to run on live nodespg_autoctl node— declarative node lifecycle viapg_autoctl_node.inifiles:run,apply,start,show,checkpg_autoctl manual— low-level FSM and service operations for manual recoveryBug fixes in
pg_autoctlmonitor.c— fixed 17 instances of danglingIntStringpointer:intToString(x).strValuereturns a pointer into a temporary that is destroyed at the end of the expression. The garbage bytes appeared in libpq parameter arrays and propagated into PostgreSQL error messages. Fixed by hoisting eachIntStringto the enclosing function scope.Removed
azure.c,azure_config.c,cli_do_azure.c,cli_do_tmux_azure.c,azure.h,azure_config.h,Makefile.azure,docs/azure-tutorial.rstand all references. The unmaintainedpg_autoctl do azuresub-tree is gone entirely.Docs
docs/ref/pgaftest.rst— fullpgaftestreferencedocs/ref/pg_autoctl_inspect*.rst,pg_autoctl_node*.rst,pg_autoctl_manual*.rst— new sub-command reference pagesdocs/tutorial/interactive_tutorial.pgaf,citus_tutorial.pgaf— spec files used by the tutorialsdocs/operations.rst— Testing section addeddocs/azure-tutorial.rst— deleted