From fb89059c021f655be8c1d92c0424d5691fd24910 Mon Sep 17 00:00:00 2001 From: Jacob Cox Date: Sun, 30 Aug 2026 17:16:31 -0600 Subject: [PATCH 1/2] docs truth: a shipped version claimed a change it never made Found by the docs writer treating the CHART as source of truth rather than the README -- which is exactly what that rule is for. **postgres-multi-location 1.1.0's headline change never shipped.** Its README says "1.1.0 widens the tolerance" and describes moving worst-case failover from 45 to 60 seconds. The chart ships `ttl: 45` / `loop_wait: 10` / `retry_timeout: 15` -- and so do 1.0.3 and 2.0.0, byte for byte. The README's own `edit-config` remediation command sets 45/10/15, contradicting the paragraph three lines above it, and the published CHANGELOG entry repeats the claim. So a user reading either doc believes their cluster tolerates a 60-second etcd outage when it gives up at 15. Both are corrected: the README now states what is actually shipped and explains widening as a deliberate trade left to the operator, and the CHANGELOG entry carries a dated correction rather than being silently rewritten -- it was published, people may have read it. Also fixes a stale key name in `grafana-multi-location` 2.0.0's values.yaml, which still told users a location "must be one of `global.gvc.locations`" -- a key the conversion removed. Comment-only; `helm template` output verified byte-identical, so both are in-place fixes rather than new versions. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 2 +- .../versions/2.0.0/values.yaml | 2 +- .../versions/2.0.0/README.md | 20 ++++++++++--------- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 56afc1b4..0a6de1d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ High-level, user-facing catalog changes by month: new templates and notable vers - **postgres-highly-available 2.7.0** — Patroni's cluster metrics now flow into the built-in metrics stack, so `patroni_primary`, replication streaming state, WAL position, timeline and failsafe status show up in Grafana next to the standard workload metrics. Nothing to configure and no exporter to run. These are cluster-state metrics — who is primary, is replication healthy, did we fail over — not database performance; connection counts and query stats would need a separate exporter this template does not ship - **postgres-highly-available 2.6.0** — three defects that only show up when you try to restore, plus one that breaks connection pooling. Backups: the base backup was skipped for the first **six hours** of every install (WAL was archiving the whole time, so everything looked healthy while there was nothing to restore against), and the backup sidecar was being killed for running out of memory before it could finish — on Azure and GCS that meant **no base backup at all, silently**. Connection pooling: any database password containing a double quote broke *every* PgBouncer login, because the credential was written into PgBouncer's user list unescaped. Failover: the consensus timeouts were an invalid combination that Patroni silently rewrote, so a cluster configured to tolerate a 30-second etcd outage actually gave up after 14 and demoted a healthy primary. The cluster now also keeps serving through an etcd outage instead of failing over, when every member is still reachable. **Upgrading does not change a running cluster's timeouts** — they were written to etcd when it was created; the README gives the one `patronictl` command to move an existing cluster - **timescaledb-highly-available 1.2.0** — the same failover, credential-escaping and backup-memory fixes as `postgres-highly-available` 2.6.0 above, and the same note applies: upgrading does not retune a cluster that already exists -- **postgres-multi-location 1.1.0** — failover timings brought in line with the other two Patroni templates. This one was already correct; the change is a wider margin for the cross-region case, so existing clusters need no action +- **postgres-multi-location 1.1.0** — ~~failover timings brought in line with the other two Patroni templates~~ **Correction (2026-08-30): this change never shipped.** 1.0.x, 1.1.0 and 2.0.0 all ship the same `ttl: 45` / `loop_wait: 10` / `retry_timeout: 15`, which is a valid combination Patroni honours as written. No cluster is misconfigured and no action is needed; the README now explains how to widen the tolerance yourself if your etcd quorum spans regions - **clickhouse 2.8.0** — the Azure Storage account key was still a plain template value, so it landed in your Helm release and in a stored config file. It is now a secret you create before installing, like the GCS and Hetzner credentials already were - **clickhouse 2.7.0** — the GCS and Hetzner object-storage credentials were template values; both are now secrets you create before installing - **debezium-server 1.2.0** — every credential the connector uses — the source database, the offset and schema-history stores, all six sink types and the schema registry — was a plain value that ended up in your Helm release. They are now keys in one secret you create before installing; which keys you need depends on your source and sink, and the README lists them per combination. Five settings that were previously inferred from a credential being non-empty now have explicit switches diff --git a/grafana-multi-location/versions/2.0.0/values.yaml b/grafana-multi-location/versions/2.0.0/values.yaml index 588636f0..1843ef05 100644 --- a/grafana-multi-location/versions/2.0.0/values.yaml +++ b/grafana-multi-location/versions/2.0.0/values.yaml @@ -63,7 +63,7 @@ alerting: highAvailability: enabled: false # Dedicated-evaluator mode ONLY — where the one evaluator runs. Must be one of - # global.gvc.locations. IGNORED when highAvailability.enabled is true. + # global.locations. IGNORED when highAvailability.enabled is true. location: aws-us-east-1 # Dedicated-evaluator mode ONLY — in HA mode the UI tier's `resources` apply. resources: diff --git a/postgres-multi-location/versions/2.0.0/README.md b/postgres-multi-location/versions/2.0.0/README.md index 85fa1226..00d7d9a1 100644 --- a/postgres-multi-location/versions/2.0.0/README.md +++ b/postgres-multi-location/versions/2.0.0/README.md @@ -483,15 +483,17 @@ a replica, the primary demotes as it would have before. three move together: raising `ttl` on its own leaves `retry_timeout` clamped, and raising `retry_timeout` on its own does nothing at all. -**1.1.0 widens the tolerance.** Versions through 1.0.x shipped `ttl: 45`, `loop_wait: 10`, -`retry_timeout: 15`. That is a valid combination and Patroni honoured it as written, so existing clusters -are not misconfigured. The change is one of margin: 15 seconds is thin for a cluster whose etcd quorum -spans regions, where a blip between locations can plausibly outlast it and demote a healthy primary. The -new values widen that margin, at the cost of taking worst-case failover from 45 to 60 seconds. - -**Upgrading does not change a cluster that already exists** — its configuration was written to etcd when it -was created, and 45 / 10 / 15 keeps working. Run the `edit-config` command above only if you want the wider -tolerance. +**This template ships `ttl: 45`, `loop_wait: 10`, `retry_timeout: 15`, and always has** — 1.0.x, 1.1.0 and +2.0.0 are identical. That is a valid combination and Patroni honours it as written. + +15 seconds is arguably thin for a cluster whose etcd quorum spans regions, where a blip between locations +can outlast it and demote a healthy primary. Widening it is a deliberate trade — a wider margin costs a +longer worst-case failover — so it is left to you rather than changed underneath a running cluster. The +`edit-config` command above sets the shipped values; raise `ttl` and `retry_timeout` together if you want +more tolerance, keeping Patroni's constraint `loop_wait + 2 * retry_timeout <= ttl`. + +**Changing it does not affect a cluster that already exists** unless you run that command — its +configuration was written to etcd when it was created. ## Operating the cluster From 7fd5f7ca62915a132e5460cff9eaa04a6d740969 Mon Sep 17 00:00:00 2001 From: Jacob Cox Date: Mon, 31 Aug 2026 08:07:59 -0600 Subject: [PATCH 2/2] cpln-advisor: the restore WAS executed, and needs psql >= 18 The 2.0.0 README still says the restore has not been executed against this template. The test round executed it end to end -- 90 KB dump, 8 tables, database dropped, restored, application working on the restored data -- so the caveat now understates what is known and the docs page says the opposite. It also needs a requirement the round found the hard way: the dump comes from postgres:18, whose pg_dumpall emits \restrict directives that older clients reject while psql EXITS 0. It succeeded only because those lines were not load-bearing. The command now carries -v ON_ERROR_STOP=1. README-only; helm template output verified byte-identical. Co-Authored-By: Claude Opus 5 --- cpln-advisor/versions/2.0.0/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/cpln-advisor/versions/2.0.0/README.md b/cpln-advisor/versions/2.0.0/README.md index e281ce61..5ad198c1 100644 --- a/cpln-advisor/versions/2.0.0/README.md +++ b/cpln-advisor/versions/2.0.0/README.md @@ -268,9 +268,11 @@ postgres: Each run writes **one gzipped `pg_dumpall` plain-SQL file**, `postgres-.sql.gz`, under `//` (read out of the pinned backup image's `/usr/local/bin/backup.sh`). `pg_dumpall` is a whole-cluster script including `CREATE ROLE` and `CREATE DATABASE`, so it restores into an empty server; it is not a merge into a running one. -The shape of a restore is: download the object with your own cloud tooling (the Postgres container has no `aws` or `gsutil`), reach the database with `cpln port-forward {release}-postgres 5432:5432 --gvc `, and pipe it in with `gunzip -c postgres-….sql.gz | psql -h 127.0.0.1 -p 5432 -U -d postgres`. Stop the API, worker and scheduler first, or they will be writing while you restore. +The shape of a restore is: download the object with your own cloud tooling (the Postgres container has no `aws` or `gsutil`), reach the database with `cpln port-forward {release}-postgres 5432:5432 --gvc `, and pipe it in with `gunzip -c postgres-….sql.gz | psql -v ON_ERROR_STOP=1 -h 127.0.0.1 -p 5432 -U -d postgres`. Stop the API, worker and scheduler first, or they will be writing while you restore. -> **This restore has not been executed against this template.** The artifact format above was read from the backup image; the steps follow from what `pg_dumpall` produces and what the chart configures, but nothing here verifies them end to end. Rehearse it against a scratch release before you need it. +> **This restore was executed end to end against this template** (2026-08-30): a real backup was taken and verified (10,377 B gzipped → 90,072 B, 8 tables with actual scan rows), the database dropped, the dump restored, and the application confirmed working on the restored data. +> +> **Use a `psql` of version 18 or newer, and keep `-v ON_ERROR_STOP=1`.** The dump comes from `postgres:18`, whose `pg_dumpall` emits `\restrict` / `\unrestrict` directives older clients do not understand. Without `ON_ERROR_STOP` an older `psql` prints `invalid command \restrict` for each one and still **exits 0** — measured, and the restore happened to succeed anyway only because those lines were not load-bearing. A silently-successful restore is the dangerous case. ### Scaling past one replica