diff --git a/briefings/etcd-multi-location.md b/briefings/etcd-multi-location.md index d654a340..1b4f64a9 100644 --- a/briefings/etcd-multi-location.md +++ b/briefings/etcd-multi-location.md @@ -2,9 +2,10 @@ ## What it is -- A stretched **etcd** cluster — one member per location, one raft quorum (a majority of members must agree before a write is accepted) — deployed into a GVC the chart creates. +- A stretched **etcd** cluster — one member per location, one raft quorum (a majority of members must agree before a write is accepted) — deployed into an **existing** GVC. - etcd is **Apache-2.0** (permissive open source: free to run, modify and redistribute, nothing to buy or register). Nothing is enterprise-gated; clustering *is* the product. - The single-location `etcd` template is untouched and remains the right choice for one location. This chart **requires ≥2 locations** and says so in the validation message. +- **2.0.0 is the GVC-removal conversion.** 1.x created its own GVC; 2.0.0 does not, and refuses to render if the 1.x `global.gvc` key is still present. It is the leaf of the grafana stack — `postgres-multi-location` → `redis-multi-location` → `grafana-multi-location` all wait on it. ## Common use cases @@ -16,13 +17,14 @@ | Resource | Purpose | |---|---| -| `gvc` (suppressed only as a subchart) | Multi-location GVC pinned to `global.gvc.locations` via `staticPlacement` | | `workload` (stateful) `{release}-etcd` | The members; exactly 1 replica per location, `replicaDirect: true` | | `volumeset` `{release}-etcd-vs` | `/var/lib/etcd` — raft WAL and snapshots, 10 GiB, `ext4`, final snapshot on uninstall | -| `identity` + `policy` | `reveal` on exactly the startup secret, nothing else | +| `identity` + `policy` `{release}-etcd-policy` | `reveal` on exactly the startup secret, nothing else | +| `policy` `{release}-etcd-gvc-policy` | **New in 2.0.0.** `view` on exactly the one install GVC, for the boot-time location check. Never `target: all` | | `secret` (opaque, plain) `{release}-etcd-startup` | `start.sh`, which computes member name, peer URLs and the full cluster list at boot | -- Peers find each other by per-replica DNS `replica-0.{workload}.{location}.{gvc}.cpln.local:2380` — no operator, no Kubernetes API, nothing platform-specific beyond `replicaDirect`. +- **No `kind: gvc`.** `createsGvc: false`. Every resource lands in `.Values.global.cpln.gvc`. +- Peers find each other by per-replica DNS `replica-0.{workload}.{location}.${CPLN_GVC}.cpln.local:2380` — the GVC comes from the runtime built-in now, not from Helm, so it cannot drift from where the workload actually runs. - Internal only. No public access, no `loadBalancer.direct`; etcd here has no TLS and no auth. - Ports 2379 (client) / 2380 (peer). **No probes, deliberately** — a readiness probe failing during a raft election would withdraw the member from discovery exactly when its peers need it. @@ -30,36 +32,57 @@ | Knob | Default | Meaning | |---|---|---| -| `global.gvc.name` | `etcd-multi-location-gvc` | The GVC all resources land in — **not** `global.cpln.gvc`, which this chart ignores | -| `global.gvc.locations[]` | 3 AWS locations | The member map; **≥2 required**, `replicas` must be `1` | -| `image` | `controlplanecorporation/etcd:0.1` | etcd **3.6.5** inside (verified `etcd --version` in the image) | +| `global.locations[]` | `aws-us-east-1`, `aws-us-west-2` | The member map. **≥2 required**, `replicas` must be `1`, no duplicates. Renamed from `global.gvc.locations` in 2.0.0; `global.gvc` now hard-fails at render | +| `image` | `controlplanecorporation/etcd:0.1` | etcd **3.6.5**, Alpine, runs as root, ships `bash`, `curl 8.12`, `timeout` | | `resources.cpu` / `.memory` | `500m` / `512Mi` | Per member; limits only, so no `cpu:minCpu` ratio concern | | `tuning.heartbeatIntervalMs` / `.electionTimeoutMs` | `250` / `5000` | Raft timers, tuned for the 63–140 ms cross-region RTTs measured in the spike | -| `tuning.autoCompactionMode` / `.autoCompactionRetention` | `periodic` / `1h` | Knobs since **1.0.2**; were hard-coded literals in 1.0.0–1.0.1. Cannot be set to "off" | -| `tuning.quotaBackendBytes` | `0` | Backend ceiling in bytes; `0` means etcd's own 2 GiB. Omitted from the render when `0`, so a default install is byte-identical to 1.0.1 | +| `tuning.autoCompactionMode` / `.autoCompactionRetention` | `periodic` / `1h` | Cannot be set to "off" | +| `tuning.quotaBackendBytes` | `0` | Backend ceiling in bytes; `0` means etcd's own 2 GiB, and is omitted from the render | | `volumeset.capacity` | `10` GiB | Platform minimum | -| `internalAccess.type` / `.workloads` | `same-gvc` / `[]` | Who may reach `:2379` | +| `internalAccess.type` / `.workloads` | `same-gvc` / `[]` | Who may reach `:2379`. The list also governs **raft peer traffic between the members**, so the chart appends its own workload — list only clients. Omitting it destroyed the cluster while Helm reported success (2026-08-27). | | `recovery.forceNewClusterInLocation` | `""` | Emergency single-member rebuild after permanent quorum loss | -- `global.gvc` sits under `global` **on purpose**: it is the only channel by which `postgres-multi-location` propagates one GVC + location list to this chart as a subchart. `global.gvc.locations[].replicas` is read only by the parent — etcd always runs one member per location — so the `replicas != 1` guard fires **only in standalone mode**, discriminated by Helm's `.Chart.IsRoot`. The same built-in gates `gvc.yaml`, so a standalone install always creates its GVC while a parent release renders exactly one. Verified on `cpln helm` specifically, because a Helm that does not support `IsRoot` returns nil, which is falsy — the chart would then silently render NO GVC rather than failing loudly. (Helm does not document which version introduced `IsRoot`; it is a real method on the Chart type but is absent from the Built-in Objects page, so treat it as low-discoverability rather than well-known.) -- Auto-compaction defaults to `periodic` / `1h`. Confirmed accepted by the image: `"auto-compaction-mode":"periodic","auto-compaction-retention":"1h0m0s"`. **1.0.2 promoted it from a literal to a knob** alongside the raft timers, and added `quotaBackendBytes`; the `etcd` template got the same three knobs in 1.4.2, with identical names, because it had **no compaction flags at all** (see that briefing). +- **The default is 2 locations, deliberately, and it is not a recommendation.** 3 is the recommended production shape (the README table says so). With the GVC gone, no hardcoded list can be right for a user's GVC, so the default is the *smallest cluster the chart allows* rather than a three-location guess that quietly triples spend. It also keeps `helm template --set global.cpln.gvc=…` rendering clean, which a one-location default could not (the ≥2 guard). +- `global.locations` stays under `global` **on purpose**: it is the only channel by which `postgres-multi-location` propagates one location list to this chart as a subchart, so the two lists can never be edited apart. `replicas` in that shared list is read only by the parent — etcd always runs one member per location — so the `replicas != 1` guard fires **only in standalone mode**, discriminated by Helm's `.Chart.IsRoot`. Verified both ways: a fake parent passing `replicas: 3` renders, the same values as root fails. + +## Migrating from 1.x — the whole point of the major bump + +- **Never `helm upgrade` a 1.x release onto 2.0.0.** The upgrade drops `kind: gvc` from the manifest and Helm prunes what a chart no longer declares — measured elsewhere at **6 seconds to delete a GVC and everything in it, while printing `upgraded successfully`**. +- The chart therefore **fails at render** on `hasKey .Values.global "gvc"`. Proven against the real 1.0.2 `values.yaml` via `-f`, and against `--set global.gvc.name=…`, `--set global.gvc.locations[0].name=…` and `--set-json 'global.gvc={}'`. +- **One hole is not closable at render**: an upgrade run with *no* values at all sees only 2.0.0's defaults, and the guard cannot fire. That is why the migration prose exists in both the README and here. +- Migration is: new release into an existing GVC → `etcdctl get "" --prefix -w json` out of the old cluster → write into the new → uninstall the old release (which takes the GVC it created). Do **not** point the new release at the GVC the old release created: that GVC is still owned by the old release. + +## The three-layer defence against a GVC/values mismatch + +The failure this version exists to kill: **a GVC with fewer locations than `global.locations` bootstraps on a bare majority and is permanently one location from total quorum loss, while reporting healthy.** `--initial-cluster` is the *declared* set and a `new` bootstrap only needs a majority of it, so 3 declared / 2 present forms a fine-looking cluster with zero fault tolerance. + +| Layer | Mechanism | Closes | +|---|---|---| +| `defaultOptions.minScale/maxScale: 0` | `localOptions` supplies the real count (1) per configured location; an undeclared GVC location gets `desiredScale: None` | GVC has locations the values do not list — **by construction** | +| **Check A** (values only, unconditional) | `CPLN_LOCATION` must be in the configured list, else `exit 1` naming the member that could never appear in `--initial-cluster` | Last line of defence if the platform places a replica anyway | +| **Checks B and C** (boot-time GVC read) | `curl $CPLN_ENDPOINT/org/$CPLN_ORG/gvc/$CPLN_GVC` → `spec.staticPlacement.locationLinks`. **B**: quorum arithmetically impossible (`2*PRESENT <= TOTAL`). **C**: any declared location missing from the GVC | Values list a location the GVC lacks — which the platform does not validate at all | + +- **B and C are asymmetric on purpose: hard-fail on a FRESH data directory, WARNING on an initialised one.** A location removed from the GVC is indistinguishable from a location that is down, so hard-failing an initialised member would turn "lost one member" into "lost the cluster". It also must never block the `--force-new-cluster` recovery path, which by definition runs on a non-fresh data directory. +- **B is checked before C** because when locations are missing *and* that costs the cluster its majority, both are true and "quorum can never be reached" is the more specific and more actionable message. It also turns an unbounded wait for peers that can never exist into an immediate, named failure. +- **A failed GVC read is a WARNING, never a failure.** A control-plane hiccup must not be the reason etcd refuses to start. Without the `gvc` policy the call is a 403 and the check skips itself. +- **`recovery.forceNewClusterInLocation` has a gap render cannot close**, and it is read for the first time during an outage: the render-time check compares it against the *values* list, so a location that is in the values but not in the GVC passes, no member runs there, and the flag is silently never applied. Every **surviving** member therefore logs a named WARNING when the recovery location is absent from the GVC. ## Troubleshooting / considerations -- **`{release}-etcd` is a cross-chart invariant.** `postgres-multi-location` builds its Patroni `etcd3.host` list from `{{ .Release.Name }}-etcd`, exactly as `postgres-highly-available` does with `etcd` today. Renaming `etcd-ml.name` silently breaks the parent's DCS wiring; the helper carries that comment. +- **`{release}-etcd` is a cross-chart invariant.** `postgres-multi-location` builds its Patroni `etcd3.host` list from `{{ .Release.Name }}-etcd`. Renaming `etcd-ml.name` silently breaks the parent's DCS wiring; the helper carries that comment. +- **Testing 2.0.0 needs a multi-location GVC.** `test-gvc` and `test-gvc-3` are single-location, so a default install there hard-fails Check B (correctly). **`test-gvc-2` has three locations** (`aws-us-east-1`, `aws-us-east-2`, `aws-us-west-2`) and is the slot for this template. - **Quorum arithmetic decides everything.** One member per location means you survive `floor((N-1)/2)` location losses: 2 locations survive **0**, 3 survive 1, 5 survive 2. Even counts buy nothing over the odd count below. "Why didn't it fail over on 2 locations" has an arithmetic answer, not a bug. -- **`IS LEADER: true` is not proof of leadership.** An isolated survivor reports itself leader for up to ~50 s while unable to commit anything. Diagnose with a linearizable read (`etcdctl get`, no `--consistency=s`), never with `endpoint status`. +- **`IS LEADER: true` is not proof of leadership.** An isolated survivor reports itself leader for ~6 s with this chart's 5 s election timeout while unable to commit anything. Diagnose with a linearizable read (`etcdctl get`), never with `endpoint status`. - **Writes time out under quorum loss; they do not fail fast.** Clients without a short `--command-timeout` pile up connections. Serializable reads keep succeeding on stale data, which can fool a naive health check. -- **Never use `localOptions[].suspend` on this workload — or advise a user to.** Suspending a location and resuming it permanently withdraws that location's endpoints from the other locations' service discovery (proven twice, two different images); every status surface reads healthy while inbound traffic is dead, and only deleting and recreating the workload fixes it. The chart exposes no suspend knob for that reason. Genuine crashes and replica reschedules recover automatically in ~15–23 s, so cross-region designs are fine. -- **`kill -9` from inside the container does nothing when etcd is PID 1** — unhandled signals to PID 1 from inside its own namespace are discarded. Any crash test done that way measured nothing. +- **Never use `localOptions[].suspend` on this workload — or advise a user to.** Suspending a location and resuming it permanently withdraws that location's endpoints from the other locations' service discovery; every status surface reads healthy while inbound traffic is dead, and only deleting and recreating the workload fixes it. Genuine crashes and replica reschedules recover automatically in ~15–23 s. +- **`kill -9` from inside the container does nothing when etcd is PID 1.** Any crash test done that way measured nothing. - **Allow ~2 minutes of cross-region convergence** after a cold deploy before believing a peer is unreachable, and expect a 45–92 s lag between a replica-stop API call and the replica actually stopping. -- **`--set global.gvc.locations[0].replicas=3` does NOT do what it looks like.** Helm replaces lists wholesale on `--set`, so that command yields a one-element list and trips the ≥2-locations guard instead of the replicas guard. Test list-shaped values with a `-f` values file. -- **This chart is `createsGvc: true`.** Pointing it at a GVC that already exists makes Helm adopt it, and `helm uninstall` will then **delete** that GVC and everything in it (this destroyed `test-gvc` on 2026-08-07 despite a `resource-policy: keep` annotation). So `global.gvc.name` must name a GVC that does NOT already exist; testing uses a `test-`prefixed name the release creates and `helm uninstall` removes. -- **Removing auto-compaction is a slow-fuse outage, and it is not conditional on write traffic.** Revisions accumulate with **time alone** — Patroni renews its DCS leader lease every ~10 s, measured at ~151k revisions and ~19 MB/day on an idle install, reaching the 2 GiB default quota (`quota-backend-bytes: 2147483648`, confirmed in the image) in ~110 days. etcd then raises a cluster-wide `NOSPACE` alarm and every member goes read-only. This is why 1.0.2's validation refuses a retention of `0`, an unrecognised mode and a negative quota (which etcd reads as "no quota at all"): each renders a chart that fails weeks later, never at install. -- **A quota-full cluster misdiagnoses as a Postgres problem.** Patroni replicas that cannot renew their DCS lease exit *cleanly*, so they restart-loop with `exitCode: 0` / `reason: Completed` and a climbing restart count — which reads as healthy. Check `etcdctl alarm list` and `etcdctl endpoint status --cluster` before touching Postgres. Enabling compaction on an already-alarmed cluster does **not** rescue it: the alarm must be disarmed by an operator, and the existing backend file never shrinks on its own. Both READMEs carry the inspection commands; the recovery procedure is deliberately left to the operator. -- **Changing `global.gvc.locations` reprovisions**, restarting every member with a new `--initial-cluster`. etcd's graceful `member add`/`member remove` path is a follow-up, not what this chart does. +- **`--set global.locations[0].replicas=3` does NOT do what it looks like.** Helm replaces lists wholesale on `--set`, so that yields a one-element list and trips the ≥2-locations guard instead of the replicas guard. Test list-shaped values with `-f` or `--set-json`. +- **Removing auto-compaction is a slow-fuse outage, and it is not conditional on write traffic.** Revisions accumulate with **time alone** — Patroni renews its DCS leader lease every ~10 s, measured at ~151k revisions and ~19 MB/day on an idle install, reaching the 2 GiB default quota in ~110 days. etcd then raises a cluster-wide `NOSPACE` alarm and every member goes read-only. Hence the render-time refusal of a retention of `0`, an unrecognised mode and a negative quota. +- **A quota-full cluster misdiagnoses as a Postgres problem.** Patroni replicas that cannot renew their DCS lease exit *cleanly*, so they restart-loop with `exitCode: 0` / `reason: Completed` and a climbing restart count. Check `etcdctl alarm list` and `etcdctl endpoint status --cluster` before touching Postgres. Enabling compaction on an already-alarmed cluster does not rescue it. +- **Changing `global.locations` reprovisions**, restarting every member with a new `--initial-cluster`. etcd's graceful `member add`/`member remove` path is a follow-up, not what this chart does. -## Measured behaviour (test run 2026-08-11, 3 locations) +## Measured behaviour (test run 2026-08-11, 3 locations, 1.0.x) | Event | Result | |---|---| | Idle stability | **10.8 h, zero spontaneous elections** (raft term unchanged) | @@ -68,17 +91,24 @@ | Write latency (leader in eu-central) | p50 **95.7 / 185.7 / 236.4 ms** from eu / east / west — one cross-region RTT is the floor | | Auto-compaction | fired at the 1 h mark, freed 569 KB, `dbSize` flat afterwards under load | | Firewall propagation | 147 s to deny, 124 s to re-allow | -| **`helm upgrade`** | **all three locations restart together — 66 s of lost quorum** | - -- **The upgrade window is the one thing to plan around.** Root cause (confirmed 2026-08-11 on the live workload): **the API does not retain `rolloutOptions.maxUnavailableReplicas`** — it is absent from the stored spec, while `maxSurgeReplicas`, `minReadySeconds` and `scalingPolicy` survive. Nothing constrains the rollout, so from 1.0.1 the chart deliberately does not send the field at all (see the zero-drift section below). This is a platform behaviour affecting every multi-location workload, not an etcd bug. -- **Do NOT claim more members per location fixes it.** That was an early inference made while assuming `maxUnavailableReplicas` worked; with the field dropped there is no evidence it helps, and it is untested. -- For `postgres-multi-location`, `failsafe_mode: true` is what keeps the Patroni primary alive through that window — it holds only while the primary can reach ALL members via REST. -- `IS LEADER: true` persists ~**6 s** after isolation with this chart's 5 s election timeout (an earlier 50 s figure came from the single-location chart's 50 s timeout). +| **`helm upgrade`** | **all three locations restart together — ~105 s rollout, ~20-30 s of lost quorum. A NO-OP upgrade costs the same, so there is no safe one.** | -## Zero-drift rendering (1.0.1 for etcd; folded into 1.0.0 for postgres) -A default install originally showed configuration drift against its own manifest the moment it was created. Both causes were ours, and the fix is the same principle everywhere: **render what the API actually stores.** -- `staticPlacement.locationLinks` must be rendered **alphabetically sorted** — the API stores them that way, so values order never matched. -- **Stop sending `rolloutOptions.maxUnavailableReplicas`.** The API does not retain it. Sending it produced permanent drift AND implied a limited rolling restart that is not in force. -- **Declare what the API backfills**: `rolloutOptions.terminationGracePeriodSeconds: 90`, and probe `initialDelaySeconds` (60 liveness / 10 readiness on Patroni). - -Why it matters beyond tidiness: a template that drifts from creation teaches its users that drift is normal, which is exactly how a real, unintended change later goes unnoticed. Check with a no-op `helm upgrade` — every resource must report `Unchanged`. +## Guard timing (measured 2026-08-27, in the shipped image, Docker) +| Case | Result | +|---|---| +| GVC read succeeds | ~0 s of added boot time | +| API returns HTTP 403 (no `gvc` policy) | **7 s**, then WARNING and start | +| API blackholed (`192.0.2.1`) | **28 s**, then WARNING and start. Worst case is 40 s (3 × `timeout 12`, + 2 × `sleep 2`) | +| curl's own `--retry 3 --retry-delay 2 --max-time 10` against the same blackhole | **46 s** — `--max-time` is per attempt, not a retry cap, which is why the retry is done in shell | +| NXDOMAIN control | **6 s** — a fast negative control would have proved nothing about the bound | + +- **The upgrade window is the one thing to plan around.** Root cause (confirmed on the live workload): **the API does not retain `rolloutOptions.maxUnavailableReplicas`** — it is absent from the stored spec while `maxSurgeReplicas`, `minReadySeconds` and `scalingPolicy` survive. Nothing constrains the rollout, so the chart deliberately does not send the field. This is platform behaviour affecting every multi-location workload, not an etcd bug. +- **Do NOT claim more members per location fixes it** — untested, and there is no evidence it helps. +- For `postgres-multi-location`, `failsafe_mode: true` is what keeps the Patroni primary alive through that window. + +## Zero-drift rendering +A default install originally showed configuration drift against its own manifest the moment it was created. **Render what the API actually stores.** +- **Stop sending `rolloutOptions.maxUnavailableReplicas`.** The API does not retain it; sending it produced permanent drift AND implied a limited rolling restart that is not in force. +- **Declare what the API backfills**: `rolloutOptions.terminationGracePeriodSeconds: 90`. +- The `staticPlacement.locationLinks` sorting fix from 1.0.1 is moot in 2.0.0 — the chart no longer renders a GVC. +- Check with a no-op `helm upgrade` — every resource must report `Unchanged`. diff --git a/etcd-multi-location/versions/2.0.0/Chart.yaml b/etcd-multi-location/versions/2.0.0/Chart.yaml new file mode 100644 index 00000000..4a3c42d2 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/Chart.yaml @@ -0,0 +1,17 @@ +apiVersion: v2 +name: etcd-multi-location +description: A stretched etcd cluster with one member per location for cross-region coordination +type: application +version: 2.0.0 +appVersion: "3.6.5" + +annotations: + created: "2026-08-10" + lastModified: "2026-08-27" + category: "key-value store" + createsGvc: false + +dependencies: + - name: cpln-common + version: 1.0.0 + repository: "oci://ghcr.io/controlplane-com/templates" diff --git a/etcd-multi-location/versions/2.0.0/README.md b/etcd-multi-location/versions/2.0.0/README.md new file mode 100644 index 00000000..b67fedba --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/README.md @@ -0,0 +1,224 @@ +# etcd Multi-Location + +etcd is an Apache-2.0 strongly-consistent key-value store — the coordination layer behind leader election, distributed locking and service configuration. This template deploys a **single stretched etcd cluster with exactly one member per Control Plane location**, sharing one raft quorum across regions, with raft timers retuned for cross-region round trips and auto-compaction enabled. It is also the consensus store consumed by `postgres-multi-location`. + +For a single-location cluster, use the **`etcd`** template instead — this one requires at least two locations. + +## Architecture + +- **etcd workload** — a `stateful` workload (`{release}-etcd`) with `replicaDirect` enabled and exactly one replica in each location listed in `global.locations`. Client API on 2379, raft peer traffic on 2380. +- **Volumeset** — 10 GiB `ext4` per member at `/var/lib/etcd` (raft WAL and snapshots), with a final snapshot on uninstall and 7-day retention. +- **Identity** and two **policies** — `reveal` on exactly the startup secret, and `view` on exactly the one GVC this release installs into (used by the boot-time location check below). +- **Secret** (opaque, plain) — the startup script, which computes each member's name, peer URL and full cluster list at container start from `CPLN_LOCATION` and `CPLN_GVC`. + +Members find each other by per-replica mesh DNS (`replica-0.{workload}.{location}.{gvc}.cpln.local:2380`). There is no operator, no discovery service and no join step: every member is given the full cluster list up front and they elect a leader among themselves. + +**This chart does not create a GVC.** It deploys into the GVC you install into, and `global.locations` must name locations that GVC already has. + +## Quorum arithmetic + +etcd commits a write only when a **majority** of members accept it. With one member per location, the location *is* the failure domain: + +| Locations | Members | Majority needed | Location losses survived | What that means | +|---|---|---|---|---| +| 2 | 2 | 2 | **0** | Losing either stops writes. The survivor holds current data; recovery is manual. | +| 3 | 3 | 2 | **1** | Automatic failover. The recommended shape. | +| 4 | 4 | 3 | **1** | No better than 3, and costs more. | +| 5 | 5 | 3 | **2** | Survives losing two locations. | + +With N locations you survive `floor((N-1)/2)` losses, so an even count never buys anything over the odd count below it. Two locations is still permitted — it is a deliberate warm-standby topology — but it survives nothing automatically. + +## Prerequisites + +- **A GVC whose locations include every entry in `global.locations`.** This chart deploys into an existing GVC and does not create one. Check with `cpln gvc get {gvc} -o yaml` and read `spec.staticPlacement.locationLinks`. + +Nothing else: no cloud account, no bucket, no pre-created secret, no custom domain. + +### Why the location lists must match + +The list in your values is the **declared** cluster roster — it becomes etcd's `--initial-cluster`. The platform does not validate it against the GVC, so a location the GVC lacks is accepted, stored, and simply never runs. That is dangerous rather than merely wrong, because a `new` bootstrap only needs a *majority of the declared set*: + +| `global.locations` | GVC locations | Without a guard | +|---|---|---| +| 3 | the same 3 | Correct. Survives one location loss. | +| 3 | 2 of those 3 | Cluster **forms and reports healthy** with **zero** fault tolerance — losing either surviving location loses quorum permanently. Nothing says so. | +| 3 | 1 of those 3 | Never forms. Members wait forever for an election that cannot complete. | +| 3 | those 3 **plus** another | The extra location runs **nothing** — `defaultOptions.minScale/maxScale` are `0`. | + +The members check this themselves at boot, by reading their own GVC: + +- On a **fresh** data directory, any declared location the GVC lacks is a **hard failure**: the member prints how many of the declared members can actually start, and exits rather than bootstrapping a bare-majority cluster. +- On an **already-initialised** member it is a **warning only**. A location removed from the GVC is indistinguishable from a location that is down, and refusing to start would turn "lost one member" into "lost the cluster". This check can never take a live cluster down. +- If the GVC read itself fails (a control-plane hiccup, or a missing policy), the check is **skipped with a warning** — it never blocks a start on its own unavailability. + +Look for `[etcd]` lines in `cpln logs` when a member will not come up. + +## Configuration + +### Locations + +```yaml + +global: + # One etcd member per location. Minimum 2; 3 survives losing one location, + # 5 survives losing two. See the quorum table in the README — 3 is the + # recommended production shape. This default is the smallest cluster the + # chart allows, because no location list can be right for every GVC: set it + # to YOUR GVC's locations before installing. + locations: + - name: aws-us-east-1 + - name: aws-us-west-2 +``` + +Every name here must already exist in the GVC you install into. The block lives under `global` so a parent chart — `postgres-multi-location` consumes this one as a subchart — sets the location list once and Helm propagates it here; the two lists can then never be edited apart. `replicas` is fixed at `1`; anything else fails the render in a standalone install. + +### Image and resources + +```yaml +image: controlplanecorporation/etcd:0.1 + +resources: + cpu: 500m + memory: 512Mi +``` + +### Raft timers and storage growth + +```yaml +# Raft timers, tuned for cross-region round trips (measured 63-140 ms). +# Raise both if your locations are more than ~250 ms apart. +tuning: + heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations + electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000 + # etcd keeps every superseded revision until told otherwise, so compaction is + # required rather than optional and cannot be switched off here. + autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count) + autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count + quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB +``` + +The timer defaults detect a dead leader in about 5 seconds across an AWS us-east ↔ eu-central ↔ us-west triangle. Locations further apart (US ↔ Asia-Pacific is 350–400 ms) need both raised in proportion; both bounds are enforced at render time. + +Compaction is what keeps the backend from growing forever, and **revisions accumulate with time alone** — a client that renews a lease on a timer (Patroni renews its leader lease every ~10 s) writes new revisions whether or not any application data changes, measured at ~151k revisions and ~19 MB per day on an otherwise idle cluster. The 1-hour default keeps the backend flat and is proven here. Raising `autoCompactionRetention` to `24h` or more buys a longer history window at the cost of a larger backend; `revision` mode keeps a fixed number of revisions instead of a time window. Disabling compaction is not offered — a retention of `0`, an unrecognised mode and a negative quota are all rejected at render time. + +Compaction frees pages for reuse *inside* the backend file, so `dbSize` plateaus rather than shrinking — that is expected, and it is sufficient to stay under quota (defragmentation only returns free pages to the filesystem, and this template does not automate it). Leave `quotaBackendBytes` at `0` unless the keyspace genuinely outgrows 2 GiB; 8 GiB (`8589934592`) is etcd's own suggested maximum, above which it warns at startup. + +### Storage + +```yaml +volumeset: + capacity: 10 # initial capacity in GiB (minimum is 10) +``` + +### Access + +```yaml +internalAccess: + type: same-gvc # options: same-gvc, same-org, workload-list + workloads: [] # only used when type is workload-list; the etcd workload is added automatically + #- //gvc/GVC_NAME/workload/WORKLOAD_NAME +``` + +The members reach each other over the same internal firewall, so `workload-list` would cut +the cluster off from itself. The chart therefore always adds its own workload to the list — +you only list the clients. + +Cross-location traffic inside one GVC is same-GVC traffic, so the default covers a stretched cluster with no extra rule. There is deliberately **no public access**: etcd here has no TLS and no authentication. + +### Disaster recovery + +```yaml +recovery: + # EMERGENCY ONLY. Set to the surviving location's name to restart that member + # as a new single-member cluster after permanently losing quorum. Read the + # "Recovering from a lost location" section of the README first. + forceNewClusterInLocation: "" +``` + +## Connecting + +| Target | Address | Credentials | +|---|---|---| +| Client API, load-balanced across locations | `http://{release}-etcd.{gvc}.cpln.local:2379` | none | +| A specific location's member | `http://replica-0.{release}-etcd.{location}.{gvc}.cpln.local:2379` | none | +| Raft peer traffic (members only) | port 2380 | none | + +Point a client at one endpoint per location so it can fail over. From inside the cluster: + +```bash +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl member list -w table +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl endpoint status --cluster -w table +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl put /demo/key hello +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl --command-timeout=5s get /demo/key +``` + +Every member is named `{release}-etcd-{location}`, so `member list` maps one-to-one onto your location list. + +## Migrating from 1.x + +**Never `helm upgrade` a 1.x release onto 2.0.0.** Versions before 2.0.0 created their own GVC. Upgrading in place drops `kind: gvc` from the manifest, and Helm deletes what a chart no longer declares — which destroys that GVC and **every workload, volumeset and identity inside it**, in seconds, while printing `upgraded successfully`. The chart refuses to render if your values still carry the 1.x `global.gvc` key, so a values-carrying upgrade fails safely; an upgrade run with *no* values at all sees only the new defaults and cannot be caught, which is why this is a migration and not an upgrade. + +Migrate to a new release instead: + +1. Pick an existing GVC (or create one) whose locations are exactly the ones you want members in. `cpln gvc get {gvc} -o yaml` shows `spec.staticPlacement.locationLinks`. +2. Snapshot the old cluster's keyspace: + + ```bash + cpln workload exec {old-release}-etcd --gvc {old-gvc} --container etcd -- etcdctl get "" --prefix -w json > keyspace.json + ``` + +3. Install 2.0.0 as a **new release** into the chosen GVC, with `global.locations` set to that GVC's locations. Replace `global.gvc.name` / `global.gvc.locations` from your old values with `global.locations`. +4. Confirm the new cluster is healthy — `etcdctl member list -w table` must show one member per configured location — then write your keys into it. +5. `cpln helm uninstall {old-release} --gvc {old-gvc}`, which removes the old release **and the GVC it created**. + +Do not point the new release at the GVC the old release created: that GVC is still owned by the old Helm release and uninstalling it will take the new workloads with it. + +## Recovering from a lost location + +Only needed when quorum is **permanently** gone — with two locations, that is the loss of either one. Writes will hang rather than fail, and `etcdctl member remove` cannot help because it needs quorum itself. + +1. Confirm the loss is permanent. If the location is coming back, wait: a member that returns with its volume intact rejoins on its own in well under a minute. +2. Set `recovery.forceNewClusterInLocation` to the **surviving** location's name and `helm upgrade`. That member restarts with `--force-new-cluster`, rebuilding a single-member cluster from its own write-ahead log. It serves writes again immediately. +3. Set `recovery.forceNewClusterInLocation` back to `""` and `helm upgrade` again. Leaving it set means the flag fires on every future restart of that member. +4. Before the lost location's member can rejoin, **reset its volume** — it still holds the old cluster ID and will be rejected. Uninstall and reinstall, or delete its volume, so it bootstraps fresh. + +Never set this to more than one location, and never leave it set: two members both forcing a new cluster produce two divergent single-member clusters with no way to merge them. + +The render-time check only compares this value against `global.locations`. If you name a location that the **GVC** does not have, no member runs there and the flag is silently never applied — so every surviving member logs a `[etcd] WARNING: recovery.forceNewClusterInLocation is set to '…', which GVC '…' does not have` line. Check the logs if a recovery appears to do nothing. + +## If the backend quota is already full + +Enabling compaction stops further growth but does not rescue a cluster that has already hit the quota: it cannot shrink an existing backend file, and once etcd has raised a `NOSPACE` alarm, writes stay rejected until an operator disarms it. + +The symptom usually shows up in the **client, not in etcd**. A Patroni replica that cannot renew its DCS lease exits cleanly, so it restart-loops with `exitCode: 0` / `reason: Completed` and a climbing restart count — which reads as healthy and gets misdiagnosed as a Postgres problem. Check etcd first. + +Both inspection commands are read-only: + +```bash +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl endpoint status --cluster -w table +cpln workload exec {release}-etcd --gvc {gvc} --container etcd -- etcdctl alarm list +``` + +A `DB SIZE` near 2.1 GB on every member, plus `NOSPACE` in `alarm list`, confirms it. Recovering from there — compacting to a revision, defragmenting each member, then disarming the alarm — is an operator procedure that this template deliberately does not perform, because each step is disruptive and the order matters. Follow etcd's [maintenance guide](https://etcd.io/docs/v3.6/op-guide/maintenance/) and plan it as a maintenance window. + +## Important Notes + +- **Do not `helm upgrade` a 1.x release onto this version.** It deletes the GVC the old release created and everything inside it. See "Migrating from 1.x". +- **`global.locations` must match the GVC's locations.** A location the GVC lacks makes a fresh member refuse to start with a named `[etcd] FATAL` line rather than bootstrap a cluster that reports healthy with zero fault tolerance. A GVC location you did not list runs nothing. +- **Quorum arithmetic decides your failure tolerance, not the template.** Two locations survive zero losses. If you want automatic failover, use three. +- **`IS LEADER: true` is not proof of leadership.** An isolated survivor keeps reporting itself leader for about **6 seconds** (measured) while unable to commit anything. Health-check with a linearizable read (`etcdctl get `, no `--consistency=s`), never with `endpoint status`. +- **Under quorum loss, writes time out rather than failing fast**, and serializable reads keep succeeding against stale data. Always give clients a short `--command-timeout` or the equivalent, or they will pile up connections against a cluster that cannot commit. +- **Never suspend a location on this workload.** Suspending and resuming a location permanently withdraws its endpoints from the other locations' service discovery while every status surface still reads healthy. This template therefore exposes no suspend knob; add or remove locations by editing the GVC and `global.locations` together. +- **Allow about two minutes of convergence after a cold install** before concluding a member is unreachable — cross-region service discovery can lag `ready: true` by well over a minute. +- **Changing `global.locations` reprovisions the cluster.** Every member restarts with a new cluster list; this is not etcd's graceful `member add`/`member remove` path. Plan it as a maintenance window. +- **No TLS, no authentication.** Anything permitted by `internalAccess` has full read/write on the keyspace. Scope it with `workload-list` if the GVC holds workloads that should not have it; list only your clients, since the chart adds the etcd workload itself. Firewall changes take up to a couple of minutes to take effect. +- **Every `helm upgrade` restarts all three locations together — a ~105 s rollout with roughly 20-30 s of lost quorum** (measured, 3 locations). This includes a **no-op** upgrade that changes nothing, so there is no "safe" upgrade to plan around. Writes time out while quorum is gone and the cluster recovers on its own; data survives. Nothing serializes the restart: the field that would limit it (`rolloutOptions.maxUnavailableReplicas`) is **not retained by the platform**, so the chart deliberately does not set it. Plan upgrades as a short planned outage. Whether more members per location would shorten it is untested — do not assume it helps. +- **Auto-compaction cannot be turned off**, only retuned (`tuning.autoCompactionMode` / `.autoCompactionRetention`). Revisions accumulate with time alone, so an uncompacted cluster hits etcd's 2 GiB backend quota and goes read-only weeks after install, with no warning. + +## Links + +- [etcd documentation](https://etcd.io/docs/v3.6/) +- [Tuning for cross-region latency](https://etcd.io/docs/v3.6/tuning/) +- [Disaster recovery](https://etcd.io/docs/v3.6/op-guide/recovery/) +- [Maintenance and compaction](https://etcd.io/docs/v3.6/op-guide/maintenance/) +- [Clustering guide](https://etcd.io/docs/v3.6/op-guide/clustering/) diff --git a/etcd-multi-location/versions/2.0.0/templates/_helpers.tpl b/etcd-multi-location/versions/2.0.0/templates/_helpers.tpl new file mode 100644 index 00000000..8b6b1cd8 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/_helpers.tpl @@ -0,0 +1,186 @@ +{{/* Resource Naming */}} + +{{/* +etcd Workload Name + +CROSS-CHART INVARIANT: this must stay `{release}-etcd`, identical to the `etcd` +template's `etcd.name`. `postgres-multi-location` builds its Patroni `etcd3.host` +list from `{{ .Release.Name }}-etcd`, exactly as `postgres-highly-available` does +with `etcd` today. Renaming this helper silently breaks the parent's DCS wiring. +*/}} +{{- define "etcd-ml.name" -}} +{{- printf "%s-etcd" .Release.Name }} +{{- end }} + +{{/* +etcd Secret Startup Name +*/}} +{{- define "etcd-ml.secretStartup.name" -}} +{{- printf "%s-etcd-startup" .Release.Name }} +{{- end }} + +{{/* +etcd Identity Name +*/}} +{{- define "etcd-ml.identity.name" -}} +{{- printf "%s-etcd-identity" .Release.Name }} +{{- end }} + +{{/* +etcd Policy Name +*/}} +{{- define "etcd-ml.policy.name" -}} +{{- printf "%s-etcd-policy" .Release.Name }} +{{- end }} + +{{/* +etcd Volume Set Name +*/}} +{{- define "etcd-ml.volume.name" -}} +{{- printf "%s-etcd-vs" .Release.Name }} +{{- end }} + +{{/* +GVC-read Policy Name +*/}} +{{- define "etcd-ml.policy.gvc.name" -}} +{{- printf "%s-etcd-gvc-policy" .Release.Name }} +{{- end }} + + +{{/* Validation */}} + +{{/* +The chart stopped creating a GVC in 2.0.0. Refuse to render if the values still +carry the 1.x `global.gvc` key — an in-place `helm upgrade` from 1.x would drop +`kind: gvc` from the manifest, and Helm deletes what a chart no longer declares, +taking the GVC and every workload, volumeset and identity inside it. Measured: +6 seconds, while printing `upgraded successfully`. +*/}} +{{- define "etcd-ml.validateNoLegacyGvc" -}} +{{- if hasKey (.Values.global | default dict) "gvc" -}} +{{- fail "etcd-multi-location 2.0.0: the `global.gvc` values key was REMOVED. This chart no longer creates a GVC — it deploys into the GVC you install into, and `global.gvc.locations` moved to `global.locations`. DO NOT `helm upgrade` a 1.x release onto 2.0.0: the upgrade drops `kind: gvc` from the manifest and Helm deletes what a chart no longer declares, which DESTROYS that GVC and every workload, volumeset and identity inside it. Install 2.0.0 as a NEW release against an existing GVC, copy your keyspace across, then uninstall the old release. See `Migrating from 1.x` in the README." -}} +{{- end -}} +{{- end -}} + +{{/* +Validate locations, replicas and raft timers. +*/}} +{{- define "etcd-ml.validate" -}} +{{- include "etcd-ml.validateNoLegacyGvc" . -}} +{{- if not .Values.global.locations -}} +{{- fail "etcd-multi-location: global.locations is required — it is the cluster roster, one etcd member per location, and every entry must already exist in the GVC you install into." -}} +{{- end -}} +{{- if lt (len (.Values.global.locations | default list)) 2 -}} +{{- fail "etcd-multi-location requires at least 2 locations in global.locations. For a single-location cluster, use the etcd template instead." -}} +{{- end -}} +{{/* +A duplicate no longer produces a duplicated locationLinks entry (the GVC is +gone) — it produces duplicated localOptions entries, which the platform accepts +without validating, and a duplicated `--initial-cluster` entry, i.e. the same +member name declared twice. etcd rejects that at boot, but only after the whole +cluster has been provisioned. +*/}} +{{- $seen := dict -}} +{{- range .Values.global.locations -}} +{{- if not .name -}} +{{- fail "etcd-multi-location: every entry in global.locations needs a `name`." -}} +{{- end -}} +{{- if hasKey $seen .name -}} +{{- fail (printf "etcd-multi-location: location '%s' is listed more than once in global.locations. A duplicate declares the same etcd member name twice, which etcd rejects at boot. List each location exactly once." .name) -}} +{{- end -}} +{{- $_ := set $seen .name true -}} +{{- end -}} +{{/* +Standalone mode is "this chart is the top-level chart", which .Chart.IsRoot +answers directly. A parent (postgres-multi-location) carries `replicas` in the +same shared list for ITS OWN tier; etcd always runs exactly one member per +location and ignores it. Gating on createGvc was wrong (a standalone user +pointed at an existing GVC also set it false, silently disabling the guard), +and a parent-set values flag was a weaker version of the same idea. +*/}} +{{- if .Chart.IsRoot -}} +{{- range .Values.global.locations -}} +{{- if and (hasKey . "replicas") (ne (int .replicas) 1) -}} +{{- fail "etcd-multi-location runs exactly one member per location, so global.locations[].replicas must be 1. A second member in a location adds cost and reduces fault tolerance: it makes that location's loss a quorum loss." -}} +{{- end -}} +{{- end -}} +{{- end -}} +{{- $heartbeat := int .Values.tuning.heartbeatIntervalMs -}} +{{- $election := int .Values.tuning.electionTimeoutMs -}} +{{- if lt $election (mul $heartbeat 10) -}} +{{- fail (printf "tuning.electionTimeoutMs (%d) must be at least 10x tuning.heartbeatIntervalMs (%d), i.e. >= %d. A shorter election timeout makes members campaign over ordinary heartbeat jitter." $election $heartbeat (mul $heartbeat 10)) -}} +{{- end -}} +{{- if gt $election 50000 -}} +{{- fail (printf "tuning.electionTimeoutMs (%d) exceeds etcd's hard maximum of 50000 ms." $election) -}} +{{- end -}} +{{- include "etcd-ml.validateCompaction" . -}} +{{/* +A typo here is a silent no-op, and it is read for the first time during an +outage — so it must fail at render, not at 3am. Note this can only check the +VALUES list: a location that is in the values but not in the GVC passes here and +never applies the flag, because no member runs there. The startup script warns +about exactly that case from the members that DO run. +*/}} +{{- with .Values.recovery.forceNewClusterInLocation -}} +{{- $names := list -}} +{{- range $.Values.global.locations -}} +{{- $names = append $names .name -}} +{{- end -}} +{{- if not (has . $names) -}} +{{- fail (printf "recovery.forceNewClusterInLocation (%s) is not one of the configured locations (%s)." . (join ", " $names)) -}} +{{- end -}} +{{- end -}} +{{- end -}} + +{{/* +Validate the compaction and quota knobs. + +etcd never discards a superseded revision on its own, and a Patroni DCS renews +its leader lease every ~10 s, so revisions accumulate with TIME ALONE — an idle +cluster measured ~151k revisions and ~19 MB per day, reaching the 2 GiB default +quota in roughly 110 days, at which point etcd raises NOSPACE and goes read-only +cluster-wide. Compaction is therefore mandatory here: the mode enum has no "off" +member and a retention of 0 (etcd's own way of disabling compaction) is refused. + +Two traps this catches that etcd itself accepts silently: + - `autoCompactionRetention: 30` in periodic mode means 30 HOURS, not minutes — + etcd multiplies a bare integer by time.Hour. An explicit unit is required. + - a negative `quotaBackendBytes` disables the quota outright, removing the only + backstop against unbounded growth. +*/}} +{{- define "etcd-ml.validateCompaction" -}} +{{- $mode := .Values.tuning.autoCompactionMode | toString -}} +{{- if not (has $mode (list "periodic" "revision")) -}} +{{- fail (printf "etcd-multi-location: tuning.autoCompactionMode must be \"periodic\" or \"revision\", got %q. Auto-compaction cannot be disabled — an uncompacted cluster fills its backend quota and goes read-only." $mode) -}} +{{- end -}} +{{- $retention := .Values.tuning.autoCompactionRetention | toString -}} +{{- if regexMatch "^0+([a-zA-Z]*)$" $retention -}} +{{- fail (printf "etcd-multi-location: tuning.autoCompactionRetention of %q disables auto-compaction, which fills the backend quota and takes the cluster read-only weeks later. Use a duration such as \"1h\" (periodic) or a revision count (revision)." $retention) -}} +{{- end -}} +{{- if eq $mode "revision" -}} +{{- if not (regexMatch "^[0-9]+$" $retention) -}} +{{- fail (printf "etcd-multi-location: with tuning.autoCompactionMode \"revision\", tuning.autoCompactionRetention must be a plain revision count such as \"10000\", got %q." $retention) -}} +{{- end -}} +{{- else if not (regexMatch "^[0-9]+(\\.[0-9]+)?(ns|us|ms|s|m|h)$" $retention) -}} +{{- fail (printf "etcd-multi-location: with tuning.autoCompactionMode \"periodic\", tuning.autoCompactionRetention must carry an explicit unit, such as \"1h\", \"30m\" or \"24h\", got %q. etcd reads a bare number as HOURS, so an unsuffixed value is almost never what was meant." $retention) -}} +{{- end -}} +{{- $rawQuota := .Values.tuning.quotaBackendBytes | toString -}} +{{- if not (regexMatch "^-?[0-9]+$" $rawQuota) -}} +{{- fail (printf "etcd-multi-location: tuning.quotaBackendBytes must be a plain byte count, got %q. etcd takes no size suffix, so \"2Gi\" would be rendered verbatim and rejected at boot — write 2147483648 instead." $rawQuota) -}} +{{- end -}} +{{- $quota := int64 $rawQuota -}} +{{- if lt $quota 0 -}} +{{- fail (printf "etcd-multi-location: tuning.quotaBackendBytes (%d) must not be negative — etcd reads a negative value as \"no quota at all\", removing the backstop against unbounded backend growth. Use 0 for etcd's 2 GiB default." $quota) -}} +{{- end -}} +{{- end -}} + + +{{/* Labeling */}} + +{{/* +Common labels — delegated to cpln-common +*/}} +{{- define "etcd-ml.tags" -}} +{{- include "cpln-common.tags" . }} +{{- end }} diff --git a/etcd-multi-location/versions/2.0.0/templates/identity.yaml b/etcd-multi-location/versions/2.0.0/templates/identity.yaml new file mode 100644 index 00000000..33371cdf --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/identity.yaml @@ -0,0 +1,5 @@ +kind: identity +name: {{ include "etcd-ml.identity.name" . }} +description: etcd identity +gvc: {{ .Values.global.cpln.gvc }} +tags: {{- include "etcd-ml.tags" . | nindent 4 }} diff --git a/etcd-multi-location/versions/2.0.0/templates/policy-gvc.yaml b/etcd-multi-location/versions/2.0.0/templates/policy-gvc.yaml new file mode 100644 index 00000000..cbe423f4 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/policy-gvc.yaml @@ -0,0 +1,20 @@ +{{- /* +The etcd container reads its own GVC at boot to confirm the GVC really has every +location this release declares. The platform does not validate localOptions +locations: an entry naming a location the GVC lacks is accepted, stored and +inert, so the declared member never starts and the cluster forms on a bare +majority while reporting healthy. Scoped to the ONE install GVC; never +`target: all`, which would grant view on every GVC in the org. +*/}} +kind: policy +name: {{ include "etcd-ml.policy.gvc.name" . }} +description: etcd GVC read policy +tags: {{- include "etcd-ml.tags" . | nindent 4 }} +bindings: + - permissions: + - view + principalLinks: + - //gvc/{{ .Values.global.cpln.gvc }}/identity/{{ include "etcd-ml.identity.name" . }} +targetKind: gvc +targetLinks: + - //gvc/{{ .Values.global.cpln.gvc }} diff --git a/etcd-multi-location/versions/2.0.0/templates/policy.yaml b/etcd-multi-location/versions/2.0.0/templates/policy.yaml new file mode 100644 index 00000000..03f73c13 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/policy.yaml @@ -0,0 +1,12 @@ +kind: policy +name: {{ include "etcd-ml.policy.name" . }} +description: etcd policy +tags: {{- include "etcd-ml.tags" . | nindent 4 }} +bindings: + - permissions: + - reveal + principalLinks: + - //gvc/{{ .Values.global.cpln.gvc }}/identity/{{ include "etcd-ml.identity.name" . }} +targetKind: secret +targetLinks: + - //secret/{{ include "etcd-ml.secretStartup.name" . }} diff --git a/etcd-multi-location/versions/2.0.0/templates/secret.yaml b/etcd-multi-location/versions/2.0.0/templates/secret.yaml new file mode 100644 index 00000000..e4d40902 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/secret.yaml @@ -0,0 +1,218 @@ +kind: secret +name: {{ include "etcd-ml.secretStartup.name" . }} +description: etcd startup script +tags: {{- include "etcd-ml.tags" . | nindent 4 }} +type: opaque +data: + encoding: plain + payload: |- + #!/bin/bash + set -euo pipefail + + # Runtime env provided by Control Plane. GVC comes from the built-in, not + # from Helm: this chart no longer creates or names a GVC, so a render-time + # name could only ever drift from where the workload actually runs. + GVC=${CPLN_GVC} + LOCATION=${CPLN_LOCATION##*/} + HOSTNAME=${HOSTNAME} + + # Extract replica index from HOSTNAME (always 0 — one member per location) + REPLICA_INDEX=$(echo "${HOSTNAME}" | awk -F'-' '{print $NF}') + WORKLOAD_NAME=$(echo "${HOSTNAME}" | sed 's/-[0-9]*$//') + + # Self FQDN for peer and client URLs + SELF_FQDN="replica-${REPLICA_INDEX}.${WORKLOAD_NAME}.${LOCATION}.${GVC}.cpln.local" + + # One member per location; the member name is location-qualified so it is + # stable across restarts and reschedules. + ETCD_NAME="${WORKLOAD_NAME}-${LOCATION}" + LOCATIONS=({{ range .Values.global.locations }}"{{ .name }}" {{ end }}) + CONFIGURED_LOCS="${LOCATIONS[*]}" + TOTAL=${#LOCATIONS[@]} + INITIAL_CLUSTER="" + for loc in "${LOCATIONS[@]}"; do + peer="replica-0.${WORKLOAD_NAME}.${loc}.${GVC}.cpln.local" + entry="${WORKLOAD_NAME}-${loc}=http://${peer}:2380" + if [[ -z "$INITIAL_CLUSTER" ]]; then + INITIAL_CLUSTER="$entry" + else + INITIAL_CLUSTER="${INITIAL_CLUSTER},$entry" + fi + done + + # --- Check A: never form a member identity in an undeclared location --- + # defaultOptions.minScale/maxScale are 0, so the platform should never place + # a replica in a location this release did not ask for. This is the last + # line of defence if it does: ETCD_NAME is derived from CPLN_LOCATION, so + # such a member starts with an --initial-cluster that does not contain + # itself. etcd's own message for that state is obscure; name the real cause. + # Unconditional — fresh or not, this member can never be a cluster member. + case " ${CONFIGURED_LOCS} " in + *" ${LOCATION} "*) : ;; + *) echo "[etcd] FATAL: this member is running in location '${LOCATION}', which is not in the configured location list (${CONFIGURED_LOCS})." >&2 + echo "[etcd] Its member name '${ETCD_NAME}' does not appear in --initial-cluster, so it can never join. Remove this location from GVC '${GVC}', or add it to global.locations in your values." >&2 + exit 1 ;; + esac + + # etcd warns that the mounted data dir is world-readable and recommends 0700. + # The mount point is created by the platform, so tighten it at boot. + chmod 700 /var/lib/etcd 2>/dev/null || true + + # Determine cluster state from the retained data directory + if [ -d "/var/lib/etcd/member" ] && [ "$(ls -A /var/lib/etcd/member)" ]; then + INITIAL_CLUSTER_STATE="existing" + FRESH="false" + else + INITIAL_CLUSTER_STATE="new" + FRESH="true" + fi + + # --- GVC location reconciliation --- + # The platform accepts a localOptions entry naming a location the GVC does + # not have: it is stored verbatim and is simply inert. For etcd that is the + # worst possible failure, because --initial-cluster is the DECLARED set and + # a `new` bootstrap only needs a majority of it: 3 declared / 2 present + # forms a healthy-looking cluster with ZERO fault tolerance, and nothing + # anywhere says so. Helm cannot see a live GVC, and peer unreachability is + # not usable as the signal (surviving a peer outage is the product). So ask + # the GVC itself. + # + # Bounding, measured in THIS image against a blackholed address + # (192.0.2.1 — an NXDOMAIN control returned in 6 s, so a fast control would + # have proved nothing): curl's own `--retry 3 --retry-delay 2 --max-time 10` + # took 46 s, because --max-time is per attempt and not a retry cap. The + # retry is therefore done in shell, each attempt bounded twice: --max-time 8 + # is what normally fires, and `timeout 12` is the backstop if curl itself + # hangs. Worst case 3x12 + 2x2 = 40 s; measured 28 s of guard time on a + # blackhole and 7 s on an HTTP 403. Both `timeout` and curl 8.12 are in the + # image. Do not tighten further: a timeout short enough to trip on a slow + # API turns this check off exactly when it is needed. + GVC_LOCS="" + GVC_READ_OK="false" + GVC_HTTP="000" + GVC_CHECK_TMP=$(mktemp) + for _try in 1 2 3; do + GVC_HTTP=$(timeout 12 curl -sS -o "${GVC_CHECK_TMP}" -w '%{http_code}' \ + --max-time 8 \ + -H "Authorization: ${CPLN_TOKEN:-}" \ + "${CPLN_ENDPOINT:-http://api.cpln.io}/org/${CPLN_ORG:-}/gvc/${GVC}" 2>/dev/null) || GVC_HTTP="000" + if [ "${GVC_HTTP}" = "200" ]; then + GVC_LOCS=$(tr -d ' \n' < "${GVC_CHECK_TMP}" \ + | sed -n 's/.*"locationLinks":\[\([^]]*\)\].*/\1/p' \ + | tr ',' '\n' | sed 's#.*/##;s/"//g' | tr '\n' ' ') + if [ -n "${GVC_LOCS}" ]; then + GVC_READ_OK="true" + break + fi + fi + sleep 2 + done + rm -f "${GVC_CHECK_TMP}" + + MISSING="" + PRESENT=0 + if [ "${GVC_READ_OK}" != "true" ]; then + # A control-plane hiccup must never be the reason etcd refuses to start. + # Skip the API-backed checks; Check A above still applies. + echo "[etcd] WARNING: could not read the location list of GVC '${GVC}' (HTTP ${GVC_HTTP}) — skipping the GVC location check." >&2 + else + for loc in ${CONFIGURED_LOCS}; do + case " ${GVC_LOCS} " in + *" ${loc} "*) PRESENT=$((PRESENT+1)) ;; + *) MISSING="${MISSING}${loc} " ;; + esac + done + EXTRA="" + for loc in ${GVC_LOCS}; do + case " ${CONFIGURED_LOCS} " in + *" ${loc} "*) : ;; + *) EXTRA="${EXTRA}${loc} " ;; + esac + done + if [ -n "${EXTRA}" ]; then + echo "[etcd] GVC '${GVC}' also has locations this release does not use: ${EXTRA% } — no etcd member runs there (minScale/maxScale are 0 outside localOptions)." + fi + + # --- Check B: quorum must be arithmetically possible --- + # Deliberately BEFORE check C: when locations are missing AND that costs + # the cluster its majority, both are true and "quorum can never be + # reached" is the more specific and more actionable of the two. It also + # turns what would otherwise be an unbounded wait for peers that can never + # exist into an immediate, named failure. + if [ $((2 * PRESENT)) -le "${TOTAL}" ]; then + if [ "${FRESH}" = "true" ]; then + echo "[etcd] FATAL: quorum is unreachable. ${PRESENT} of ${TOTAL} configured locations (${CONFIGURED_LOCS}) exist in GVC '${GVC}' (which has: ${GVC_LOCS% })." >&2 + echo "[etcd] --initial-cluster declares ${TOTAL} members and a 'new' bootstrap needs floor(${TOTAL}/2)+1 of them to elect at all. Refusing to wait for an election that can never complete." >&2 + echo "[etcd] Add the missing location(s) to the GVC, or remove them from global.locations in your values." >&2 + exit 1 + fi + echo "[etcd] WARNING: only ${PRESENT} of ${TOTAL} configured locations (${CONFIGURED_LOCS}) exist in GVC '${GVC}' — below the majority this cluster needs to commit a write." >&2 + echo "[etcd] This member is already initialised, so it keeps running rather than taking a live cluster down. See 'Recovering from a lost location' in the README." >&2 + # --- Check C: every declared location must exist in the GVC --- + # Asymmetric on purpose: hard-fail ONLY on a fresh data directory. A fresh + # member is being asked to bootstrap a cluster on a BARE MAJORITY of a + # roster whose remaining members can never exist — it would report healthy + # while one location from permanent quorum loss. An already-initialised + # member is a different situation: a location removed from the GVC is + # indistinguishable from one that is down, and hard-failing would turn + # "lost one member" into "lost the cluster". It also must not block the + # --force-new-cluster recovery path, which by definition runs on a + # non-fresh data directory. + elif [ -n "${MISSING}" ]; then + if [ "${FRESH}" = "true" ]; then + echo "[etcd] FATAL: locations declared in global.locations are not in GVC '${GVC}': ${MISSING% }" >&2 + echo "[etcd] GVC '${GVC}' has: ${GVC_LOCS% }" >&2 + echo "[etcd] ${PRESENT} of ${TOTAL} declared members can actually start. That is a bare majority: the cluster would form, report healthy, and have ZERO fault tolerance — losing any one location would lose quorum permanently. Refusing to bootstrap it." >&2 + echo "[etcd] Add the missing location(s) to the GVC, or remove them from global.locations in your values." >&2 + exit 1 + fi + echo "[etcd] WARNING: locations declared in global.locations are not in GVC '${GVC}': ${MISSING% }" >&2 + echo "[etcd] GVC '${GVC}' has: ${GVC_LOCS% }" >&2 + echo "[etcd] This member is already initialised, so it keeps serving — but members in the missing location(s) will never appear, and only ${PRESENT} of ${TOTAL} declared members exist." >&2 + else + echo "[etcd] GVC location check OK — '${GVC}' has every configured location (${CONFIGURED_LOCS})" + fi + fi + + EXTRA_FLAGS=() + {{- if .Values.recovery.forceNewClusterInLocation }} + # EMERGENCY: rebuild a single-member cluster from this member's own WAL after + # a permanent loss of quorum. Applies to one location only. + # + # The render-time check can only compare this against the VALUES list. If the + # named location is in the values but not in the GVC, no member runs there, + # the flag is never applied, and the operator sees nothing happen during an + # outage. Every surviving member says so instead. + if [ "${GVC_READ_OK}" = "true" ]; then + case " ${GVC_LOCS} " in + *" {{ .Values.recovery.forceNewClusterInLocation }} "*) : ;; + *) echo "[etcd] WARNING: recovery.forceNewClusterInLocation is set to '{{ .Values.recovery.forceNewClusterInLocation }}', which GVC '${GVC}' does not have (it has: ${GVC_LOCS% }). No member runs there, so --force-new-cluster will NEVER be applied. Set it to a location this GVC actually has." >&2 ;; + esac + fi + if [ "${LOCATION}" = "{{ .Values.recovery.forceNewClusterInLocation }}" ]; then + echo "RECOVERY: --force-new-cluster is set for location ${LOCATION}" + EXTRA_FLAGS+=(--force-new-cluster) + fi + {{- end }} + + echo "Starting etcd node '${ETCD_NAME}' with cluster state: $INITIAL_CLUSTER_STATE" + echo "Initial cluster: $INITIAL_CLUSTER" + + # Run etcd + exec etcd \ + --name "${ETCD_NAME}" \ + --data-dir /var/lib/etcd \ + --listen-client-urls "http://0.0.0.0:2379" \ + --advertise-client-urls "http://${SELF_FQDN}:2379" \ + --listen-peer-urls "http://0.0.0.0:2380" \ + --initial-advertise-peer-urls "http://${SELF_FQDN}:2380" \ + --initial-cluster "$INITIAL_CLUSTER" \ + --initial-cluster-state "$INITIAL_CLUSTER_STATE" \ + --heartbeat-interval {{ .Values.tuning.heartbeatIntervalMs }} \ + --election-timeout {{ .Values.tuning.electionTimeoutMs }} \ + {{- if .Values.tuning.quotaBackendBytes }} + --quota-backend-bytes {{ .Values.tuning.quotaBackendBytes }} \ + {{- end }} + --auto-compaction-mode {{ .Values.tuning.autoCompactionMode }} \ + --auto-compaction-retention {{ .Values.tuning.autoCompactionRetention }} \ + ${EXTRA_FLAGS[@]+"${EXTRA_FLAGS[@]}"} diff --git a/etcd-multi-location/versions/2.0.0/templates/volumeset.yaml b/etcd-multi-location/versions/2.0.0/templates/volumeset.yaml new file mode 100644 index 00000000..f9d1fe83 --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/volumeset.yaml @@ -0,0 +1,14 @@ +kind: volumeset +name: {{ include "etcd-ml.volume.name" . }} +description: etcd volumeset +gvc: {{ .Values.global.cpln.gvc }} +tags: + {{- include "etcd-ml.tags" . | nindent 2 }} + workload: {{ include "etcd-ml.name" . }} +spec: + fileSystemType: ext4 + initialCapacity: {{ .Values.volumeset.capacity }} + performanceClass: general-purpose-ssd + snapshots: + createFinalSnapshot: true + retentionDuration: 7d diff --git a/etcd-multi-location/versions/2.0.0/templates/workload.yaml b/etcd-multi-location/versions/2.0.0/templates/workload.yaml new file mode 100644 index 00000000..21a2349a --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/templates/workload.yaml @@ -0,0 +1,115 @@ +{{- include "etcd-ml.validate" . -}} +kind: workload +name: {{ include "etcd-ml.name" . }} +description: etcd multi-location +gvc: {{ .Values.global.cpln.gvc }} +tags: + cpln/publishNotReadyAddresses: "true" + {{- include "etcd-ml.tags" . | nindent 2 }} +spec: + type: stateful + containers: + - name: etcd + command: "/bin/bash" + args: + - "/etcd/start.sh" + cpu: {{ .Values.resources.cpu | quote }} + image: {{ .Values.image }} + inheritEnv: false + memory: {{ .Values.resources.memory | quote }} + ports: + - number: 2379 + protocol: tcp + - number: 2380 + protocol: tcp + volumes: + - path: /var/lib/etcd + recoveryPolicy: retain + uri: cpln://volumeset/{{ include "etcd-ml.volume.name" . }} + - path: /etcd/start.sh + uri: cpln://secret/{{ include "etcd-ml.secretStartup.name" . }} + identityLink: //gvc/{{ .Values.global.cpln.gvc }}/identity/{{ include "etcd-ml.identity.name" . }} + defaultOptions: + {{- /* + minScale/maxScale are 0 so a GVC location this release did not ask for runs + NOTHING: an undeclared location gets desiredScale: None. localOptions below + supplies the real count (always 1) for each configured location. Without + this, an extra GVC location silently starts an etcd member whose + --initial-cluster does not contain itself. + */}} + autoscaling: + maxConcurrency: 0 + maxScale: 0 + metric: disabled + minScale: 0 + scaleToZeroDelay: 300 + target: 100 + capacityAI: false + debug: false + suspend: false + timeoutSeconds: 5 + localOptions: + {{- range .Values.global.locations }} + - location: //location/{{ .name }} + autoscaling: + maxConcurrency: 0 + maxScale: 1 + metric: disabled + minScale: 1 + scaleToZeroDelay: 300 + target: 100 + capacityAI: false + debug: false + suspend: false + timeoutSeconds: 5 + {{- end }} + rolloutOptions: + {{- /* + maxUnavailableReplicas is deliberately NOT set: the API does not retain it + (confirmed on a live workload — it is absent from the stored spec while its + neighbours survive), so sending it bought nothing except permanent drift and + a false impression that the rollout was limited. It is not. + terminationGracePeriodSeconds is declared because the API backfills 90. + */}} + maxSurgeReplicas: 0% + minReadySeconds: 60 + scalingPolicy: OrderedReady + terminationGracePeriodSeconds: 90 + firewallConfig: + external: + inboundAllowCIDR: [] + inboundBlockedCIDR: [] + outboundAllowCIDR: [] + outboundAllowHostname: [] + outboundAllowPort: [] + outboundBlockedCIDR: [] + internal: + inboundAllowType: {{ .Values.internalAccess.type }} + {{- if eq .Values.internalAccess.type "workload-list" }} + {{- /* + This list governs ALL inbound internal traffic, INCLUDING raft peer traffic + on 2380 between the members of THIS workload. A list that omits this + workload blocks the members from each other and quorum is lost + cluster-wide, while Helm prints `upgraded successfully` (measured + 2026-08-27: every member unhealthy within ~2 min by following the README; + re-adding this workload recovered the cluster in 116 s). No legitimate + configuration denies the members to each other, so the chart appends + itself rather than requiring the user to know. + */}} + {{- $self := printf "//gvc/%s/workload/%s" .Values.global.cpln.gvc (include "etcd-ml.name" .) }} + inboundAllowWorkload: + - {{ $self | quote }} + {{- range .Values.internalAccess.workloads }} + {{- if ne . $self }} + - {{ . | quote }} + {{- end }} + {{- end }} + {{- else if .Values.internalAccess.workloads }} + inboundAllowWorkload: {{ .Values.internalAccess.workloads | toYaml | nindent 8 }} + {{- end }} + loadBalancer: + direct: + enabled: false + ports: [] + replicaDirect: true + supportDynamicTags: false diff --git a/etcd-multi-location/versions/2.0.0/values.yaml b/etcd-multi-location/versions/2.0.0/values.yaml new file mode 100644 index 00000000..35c4992d --- /dev/null +++ b/etcd-multi-location/versions/2.0.0/values.yaml @@ -0,0 +1,53 @@ +# ─── Locations ──────────────────────────────────────────────────────────────── +# This chart deploys into the GVC you install into — it does NOT create one. +# Every location listed here MUST already exist in that GVC: a location the GVC +# lacks contributes a declared etcd member that can never start, which shrinks +# the cluster to a bare majority while every status surface reads healthy. +# The members refuse to bootstrap in that state rather than form it. +# +# Lives under `global` so a parent chart (postgres-multi-location consumes this +# one as a subchart) sets the list once and Helm propagates it here — the two +# lists can then never be edited apart. + +global: + # One etcd member per location. Minimum 2; 3 survives losing one location, + # 5 survives losing two. See the quorum table in the README — 3 is the + # recommended production shape. This default is the smallest cluster the + # chart allows, because no location list can be right for every GVC: set it + # to YOUR GVC's locations before installing. + locations: + - name: aws-us-east-1 + - name: aws-us-west-2 + +# ─── etcd ───────────────────────────────────────────────────────────────────── +image: controlplanecorporation/etcd:0.1 + +resources: + cpu: 500m + memory: 512Mi + +# Raft timers, tuned for cross-region round trips (measured 63-140 ms). +# Raise both if your locations are more than ~250 ms apart. +tuning: + heartbeatIntervalMs: 250 # ~0.5-1.5x the worst round trip between locations + electionTimeoutMs: 5000 # must be >= 10x heartbeatIntervalMs; maximum 50000 + # etcd keeps every superseded revision until told otherwise, so compaction is + # required rather than optional and cannot be switched off here. + autoCompactionMode: periodic # periodic (retention is a duration) or revision (a revision count) + autoCompactionRetention: 1h # periodic needs an explicit unit (1h, 30m, 24h); revision takes a count + quotaBackendBytes: 0 # backend size limit in bytes; 0 = etcd's own default of 2 GiB + +volumeset: + capacity: 10 # initial capacity in GiB (minimum is 10) + +internalAccess: + type: same-gvc # options: same-gvc, same-org, workload-list + workloads: [] # only used when type is workload-list; the etcd workload is added automatically + #- //gvc/GVC_NAME/workload/WORKLOAD_NAME + +# ─── Disaster recovery ──────────────────────────────────────────────────────── +recovery: + # EMERGENCY ONLY. Set to the surviving location's name to restart that member + # as a new single-member cluster after permanently losing quorum. Read the + # "Recovering from a lost location" section of the README first. + forceNewClusterInLocation: ""