pgedge 2.0.1 — workload-list broke replication and backups

[jacobecox]

Review diff: git diff 1930167..HEAD -- pgedge/versions/2.0.1/
(1930167 is the verbatim cp -R of 2.0.0; everything real is in the two commits after it.)

A patch to the merged 2.0.0. Both defects were found by other templates' rounds, not by testing pgedge — neither would have surfaced from pgedge alone.

1. workload-list cut the release off from itself

2.0.0 rendered the user's list verbatim on both workloads. That list governs intra-release traffic, and pgEdge nodes replicate to each other with Spock across locations while pgcat connects to every node — so a list naming only clients breaks the mesh. Confirmed live in etcd-multi-location (quorum lost cluster-wide) and clickhouse (Raft denied, Code: 279 on every cross-shard query), both reported every replica ready=true throughout, and both reached by following the template's own README.

A catalog audit found 74 templates expose workload-list and only 6 include their own workloads. pgedge was the one already merged.

The failure mode here is nastier than in the other two. It is not a cluster that goes down — it is nodes that keep accepting writes which silently never propagate. So the test proves replication, not liveness: all-to-all row-level replication with each node holding all 3 rows, all 6 Spock subscriptions replicating, all 6 slots active=true lag=0.

2. The backup cron was a third own-workload I initially missed

The first fix added pgedge and pgcat and stopped there. The cron connects directly to replica-0.{pgedge}.{location}.{gvc}.cpln.local:5432, so from the moment a release moved to workload-list, every scheduled backup failed:

pg_dumpall: error: connection to server at "replica-0.test-pge-pgedge…", port 5432 failed:
server closed the connection unexpectedly

Ten consecutive runs, against zero such errors before the change. Now added when backup.enabled, and left out when it is not, so the list never names a workload that does not exist.

Verified as a real backup, not a job that ran. The failing runs uploaded a 20-byte gzip decompressing to zero bytes, so "completed" and "backed something up" are different claims:

object	when	gz	decompressed	rows	contains post-change row
…T14-20-10Z	before change	48475 B	109070 B	2000	no
…T14-25-07Z	after	48538 B	109307 B	2000	yes
…T14-30-06Z	after	48524 B	109307 B	2000	yes

A row written after the firewall change appears only in the post-change dumps — so those are provably fresh reads through the newly-allowed path, and the pre-change dump missing it is the control proving the check discriminates.

3. The backup cron's partial localOptions contradicted the chart

2.0.0 sent location + suspend only. The API backfills a partial block with platform defaults rather than the workload's defaultOptions — measured on cockroach, where two fields became seven, turning capacityAI on in the one location the cron runs and cutting timeoutSeconds to 5, both against the chart's explicit settings. Every field is now repeated; the stored spec matches the render in all three locations, and three healthy backups confirm a complete block did not break the schedule.

Testing

Two rounds, 0 FAIL. Enforcement control with both arms: listed client → pgcat returns 1 byte and rows=10 over real psql; unlisted client → 0 bytes and server closed the connection unexpectedly; a dead port returns 0 in every case, so the probe can produce a negative. Log greps were proven to find this workload's own lines before any zero was trusted (36/36 lines correctly labelled).

Drift gate: both no-op upgrades reported all 8 resources Unchanged with stored specs diffing to 0 lines. The stored list is fully qualified and alphabetically re-sorted versus the render, causing no churn — the fourth template to confirm that.

Open question, not introduced here

Turning backup.enabled off leaves the stored identity holding its aws: block (cloudAccountLink, policyRefs) even though the chart renders no aws: key at all. That is the documented catalog-wide platform merge behaviour (CLAUDE.md, 2026-08-21) and affects every backup-capable template — but it is a least-privilege leak invisible from the user's values file, so it may deserve its own issue.

Briefing updated in the same PR.

🤖 Generated with Claude Code