postgres-multi-location 2.0.0 — deploy into an existing GVC - #513
Merged
Conversation
This chart no longer creates a GVC. Every resource lands in
`.Values.global.cpln.gvc`, `createsGvc: false`, and `templates/gvc.yaml` is
deleted. Third in the grafana stack's forced conversion order —
grafana-multi-location cannot convert until this publishes.
The three-layer defence, all three proven in the four conversions merged today:
1. A render-time `fail` on `hasKey .Values.global "gvc"`. 1.x carried the
location list at `global.gvc.locations`, not a top-level `gvc`, so the guard
matches that path. Verified against the real 1.1.0 values.yaml via -f and
against --set global.gvc.name=..., --set global.gvc.locations[0].name=... and
--set-json 'global.gvc={}'. The upgrade this prevents deletes the GVC and
every workload, volumeset and identity in it in ~6s while printing
`upgraded successfully`.
2. `defaultOptions.minScale/maxScale: 0` on all three long-running tiers, with
`localOptions` supplying the real per-location counts. This matters more here
than in etcd: a Patroni member in an undeclared location gets the FULL
etcd3.hosts list, so it joins the DCS and can WIN the leader election, while
the HAProxy backend list is rendered from the same values and does not
contain it — every client loses the primary with all replicas `ready: true`.
3. A boot-time GVC read via a new `{release}-postgres-gvc-policy`, scoped
`targetKind: gvc` + `targetLinks` to the one install GVC. Check A (values
only, unconditional), Check B (etcd DCS quorum arithmetically impossible) and
Check C (a declared location the GVC lacks); B and C hard-fail on a FRESH
PGDATA and warn on an initialised one. The same read also warns when
`primaryLocation` or `backup.location` names a location the GVC lacks — the
backup cron has no script of its own to warn from.
All nine rows measured by running the rendered script in the real image
against a stub GVC API. The bound was tested against a blackhole (192.0.2.1,
connect hangs), not NXDOMAIN: 31s including container start, vs a 40s worst
case and a ~110s liveness budget. curl's --max-time is per attempt, so the
retry is done in shell under `timeout 12`.
Also in this version:
- Vendored subchart bumped etcd-multi-location 1.0.2 -> 2.0.0 (the converted
child). `global.gvc.locations` -> `global.locations`, which is the same rename
this chart makes, so one edit serves both charts.
- `workload-list` self-inclusion, via a single `pg-ml.ownWorkloadLinks` helper
used at every call site (cockroach 2.0.0's pattern). The list governs
Patroni-to-Patroni replication, HAProxy's health checks, PgBouncer's pooling
and the nightly dump; a list naming only clients cut the cluster off from
itself in all four templates tested this batch.
- R7 backfills declared to match PR #512's measured set: inboundBlockedCIDR,
outboundAllowHostname, outboundBlockedCIDR, inboundAllowWorkload and
secret-volume recoveryPolicy.
- Both startup scripts derive the GVC from ${CPLN_GVC} rather than from Helm, so
a hostname cannot drift from where the workload runs.
- README: Migrating from 1.x, converted Prerequisites/Architecture/Notes, and
the logical-restore psql host corrected to the fully-qualified internal name
(a bare short name does not resolve for a `standard` workload).
- briefings/postgres-multi-location.md updated against the shipped values.yaml.
The >=2 location minimum stays: the DCS is a stretched etcd cluster with one
member per location and cannot run in one. postgres-highly-available is the
single-location template.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review diff:
git diff 73bec9a..HEAD -- postgres-multi-location/versions/2.0.0/(
73bec9ais the verbatimcp -Rof 1.1.0; everything real is in the commit after it.)Chain-critical:
grafana-multi-locationvendors this chart and cannot convert until it merges and publishes.Why a major version
The chart no longer creates a GVC. The upgrade across that boundary is destructive: a chart that stops declaring a GVC makes
helm upgradeprune it, taking every workload, volumeset and identity inside.The gate holds. 1.1.0 was installed into its own chart-created GVC; the upgrade onto 2.0.0 with 1.1.0's real values file was refused at render, and the GVC, 3 workloads, 2 volumesets, 2 identities and 2 policies all still existed afterwards. Confirmed with both
-fand--values. Notecpln helm upgradeexited 0 on that refusal — state was verified directly rather than from the exit code.Also bumps the vendored subchart to
etcd-multi-location 2.0.0, and renamesglobal.gvc.locations→global.locationsso one list still serves both charts.The three-layer defence, each with both controls
minScale/maxScale: 0+localOptionsreplicas: 0,deactivated because maxScale is set to 0, while both declared locations reached 1/1200for its own GVC,403for two others — and403for its own once the scoped policy was deleted, back to 200 in 3 s when restoredexitCode: 1on a fresh volume, WARNING-and-keeps-serving on an initialised oneThe middle row is the one worth noting: deleting the policy and watching the same call flip to 403 proves the grant is what makes it work, rather than the call succeeding for some unrelated reason.
The headline promise, proven properly
A genuinely hung leader — SIGSTOP on Patroni's worker and the postmaster, REST returning
000— was replaced by automatic promotion in a surviving location in 43 s, with writes resuming through both proxy and pooler at +10 s and the old leader rejoining atstreaming lag 0with no split brain.An earlier
kill -9restarted the container in ~12 s, under the 45 s TTL, so no failover occurred. That is recorded rather than discarded: it is exactly the "crash that looks like a failover test but is not one" trap, and only the hung-leader method actually exercises the promotion path.This is also why the 3-location default stays: two locations cannot deliver automatic promotion, since the DCS loses quorum on either loss. Single-location is impossible here by construction —
postgres-highly-availableis the single-location template.workload-list, with an independent controlUnlisted client times out on all three tiers; listed client exchanges SSLRequest bytes with all three and writes through the proxy; replication continues. Then switching to
same-orgrestored the unlisted client's access — an independent control that the block really was the firewall and not something else.Gates
Unchanged, stored specs byte-identical.render-vs-stored: 0 unexplained differences on two chart shapes (15 and 16 resources). R7 backfills were declared to match PR cockroach/pgedge/clickhouse/etcd-ml — make rendered match stored #512's measured set rather than copying a sibling's superset.backup.location) and wal-g (6 base backups exactly 900 s apart, 29 WAL segments, standbys loggingNot the primary).Known, pre-existing, deliberately not fixed
backup-listlists all six andbackup-fetchproduced a valid 31 MB PGDATA with a matching system identifier. Steps 3 and 5 cannot: the volumeset name is{release}-postgres-vswith no values knob, and the workload hard-renders that URI at two call sites. No replacement was invented — an invented restore is a worse defect than the broken one.primaryLocationchange.etcd-multi-location, because the subchart renders first. The refusal and remedy are correct; Helm offers no way to reorder.Testing
17 PASS · 0 FAIL · 1 PARTIAL (the pre-existing restore) · 0 BLOCKED.
test-gvc-2was never reshaped — both undeclared-location arms were driven from the values side — and its spec diffs identical against the recorded before-state.Briefing updated in the same PR.
🤖 Generated with Claude Code